Thursday, October 14, 2004

Yet more biased sampling

There is a story that during WWII Allied Bomber Command noticed they were losing too many bombers over germany. They set up a team to study the problem. The team requisitioned a collection of squads of soldiers who went all over southern britain looking at shot-down bombers. They recorded every bullet-hole and flak-hole in the shot-up airplanes, and eventually produced a map of the bomber showing where the planes were hit the most. They recommended armor at those spots. The armor would make the planes heavier so they would have shorter range or less payload or less speed, but it was agreed that this price was worth paying if fewer bombers were lost. The recommendations were followed.

Another team kept track of bomber losses. The missions changed, and there were more missions, so it was complicated, but this other team eventually concluded that losses were at least as high as before and maybe higher. The armor was useless or worse than useless. So the first team was assigned to repeat their work and if possible find out what had gone wrong.

While the repeat study was in progress, one of the soldiers they had counting holes in aircraft pointed out to them, "You're only making me count the holes in the planes that made it back." Sure enough, when they looked at the data they found that the spots that were shot up the most often were spots that had no vital function. The most vital spots (like the pilot's seat) were hardly shot up at all. Because those planes didn't make it back across the channel to get measured.

More biased sampling

A writer once decided to write a book about how self-made millionaires did it. He used his contacts to find a sample of self-made millionaires and fourteen of them agreed to tell him how they made their money starting from essentially nothing. He cleaned up the interviews a little and put them together into a book. He recommended their methods as ways for others to make money. The book turned into an easy $15,000 for him, once the royalties came in.

When I read the book I noticed that every one of those men had, early in the game, done something that was likely to get them jailed and they got away with it.

To actually get an idea how likely those methods were to make money, the author should have started with a bunch of people just like the self-made men and watched how many of them became millionaires versus how many of them became inmates. He was only counting the ones who made it.

Of course it's impractical to do that kind of study. That's why such books give mosty-useless advice.