Monday, August 13, 2012

Would you brush all but one tooth just to know how much brushing has helped?

Thou Shalt Have a Control Group is a popular refrain in analytics. But is a controlled experiment always worth it?

As William Coyne said, "[A]fter you plant a seed in the ground, you don't dig it up every week to see how it is doing."

You may want to spend an amount of money optimally, but you have a control group designed not to do anything differently. That control group has some elements that you would want to spend money on. This lends itself to a modified Heisenberg uncertainty principle:  you cannot optimize and be sure you have optimized simultaneously. Or more familiarly:  if you're the best, you'll never know it. A setback for narcissists.

No comments:

Post a Comment