While trying to use empirical evidence to evaluate theories is an important way to advance knowledge in a given domain, I worry about how much weight is given to early empirical results found in fields that study chaotic systems. Case in point, look at the past 100 years of study in nutrition science. An entire generation was taught that fats are bad, food policy was designed around this premise, and now we are learning this may not have necessarily been the case. I don't have a solution to this, because studying empirical evidence allows us to identify what variables are missing, expand our scope of study, and evaluate what variables are/aren't relevant. However, designing policies based on empirical evidence from a field that is in its infancy when it comes to data collection is dangerous.
I can understand your point, but isn't the answer just "more science"? We want to have a theory based on early data (since there will be a policy, whether explicit or not, so better to base it on our best ideas to date), but it's early and we don't really know what we're doing. Then we get better data and better theories and voila! New consensus, better theory, better policies (in theory ;-).
Maybe advocate more open-mindedness so paradigm shifts don't require extinction of the proponents of the previous theory? Or strive for more policy responsiveness to scientific discoveries (I guess it's pretty clear this would be a Good Thing)?
I think structurally encouraging policies to be built around actual scientific consensus and data (and to keep up with the field) could go a long way.
In general I agree that yes, the answer is "more science", although my intuition tells me that some domains are too complex to ever be modeled accurately by humans...meaning writing and managing the codebase to accurately model some domains, even with infinite computing power, would be impossible due to humans limited mental capacities. What those domains are, however, I don't know and don't want to guess. I just suspect they exist.
In regards to your point about policy, if it is implicit (as in findings reported by the media), that's fine. However, as soon as policy is made explicit, either through law, or through national policies such as the food nutrition pyramid, interest groups immediately begin forming to protect said explicit policy. The rules our policymakers have to follow were explicitly designed in a way that makes changing policy a slow, arduous process to protect swift changes that could have dangerous, unknown side effects that every citizen would have to deal with. When we prematurely start explicitly making policies based on empirical results in domains of chaotic systems (i.e., those domains where unknown, new variables appear often and change the course of things), those policies have a higher chance of being wrong, and it becomes (by design) very hard to change those policies later on. This is where my concern is rooted.
