Saturday, September 11, 2004

Biased sampling

Whatever you believe about the world, I hope it's connected to experience and facts and such.

People want to believe that the experience they've had is representative of the potential experience available to them. Usually it isn't. This causes trouble when you want to predict the future based on your experience with the past. Or to predict, well, anything.

Here is a story. Some MDs thought that diet affected heart attacks, and they had ideas about just which kinds of food promoted heart attacks. So they went to a nearby hospital and gave questionnaires to people who had recently had heart attacks, and also to people who were in the hospital for broken legs and such. The people with the broken legs were a control group. Nobody thought diet made people break their legs. So if the two groups of people ate obviously different food, that would say something. It turned out they did eat significantly different foods and so the researchers published a paper about their work.

But then doubters appeared. One of the doubts went like this: The people who had had heart attacks were all people who had *survived* their first heart attack. Many do not. Maybe the food that they ate didn't give them heart attacks. Maybe that food helped them survive heart attacks. So they were still around to fill out questionnaires.

Both explanations fit the data very well. Eventually researchers set up the Framingham Study, where they looked at a whole town and kept track of a variety of things on everybody in the town of Framingham Massachusetts and then waited to see which of them would develop heart disease. It was the only way to be sure. And even that doesn't prove cause. It could be that the people who eat the "bad" foods have an inherited disposition to heart disease that also predisposes them to eat those particular foods.

There is no easy way to avoid biased data. My best suggestion is to look for te possibility of bias, and try to avoid being too certain of your conclusions.

Sunday, September 05, 2004


If you haven't read _Systemantics_ by John Gall, I highly recommend it. Here is a review.

Here is a publisher.

It covers fundamentals of dealing with systems, particularly large systems. It's written in the style of C Northcote Parkinson, with grandiose language. Here are some valuable things from it:

A complex system that works is invariably found to have evolved from a simple system that works

A complex system designed from scratch never works and cannot be patched up to make it work; you have to start over, beginning with a working simple system.

This fits my experience, does it fit yours?