My exploratory work tends to fall in one of three categories:
Diagnose what went wrong with this system.
Estimate if this project is worth pursuing.
Try to poke holes in a model before deploying it.
It's all goal directed against a mental model of what I'm doing. For (1), it's trying to eliminate swathes of the control flow and data flow of the system as quickly as possible. For (2), it's getting rough estimates of the economic impact of an idea. For (3), it's trying to find situations that will break a model, rather like the adversarial process a security researcher goes through.
Instead of writing up a huge list of things to explore, treat it the way Tukey did in his book Exploratory Data Analysis that started the field. You start with a data set, and sequentially ask questions about it and answer them as fast as possible, since often you can cut off a whole line of questioning at the very beginning with a quick check. (It's a great book, by the way...especially to see what he developed to do this kind of work by hand.)
My exploratory work tends to fall in one of three categories:
It's all goal directed against a mental model of what I'm doing. For (1), it's trying to eliminate swathes of the control flow and data flow of the system as quickly as possible. For (2), it's getting rough estimates of the economic impact of an idea. For (3), it's trying to find situations that will break a model, rather like the adversarial process a security researcher goes through.
Instead of writing up a huge list of things to explore, treat it the way Tukey did in his book Exploratory Data Analysis that started the field. You start with a data set, and sequentially ask questions about it and answer them as fast as possible, since often you can cut off a whole line of questioning at the very beginning with a quick check. (It's a great book, by the way...especially to see what he developed to do this kind of work by hand.)
Thanks Fred,
I have seen references to John Tukey's book in my reading. It does look interesting.
Hi Fred, I am curious. What other Data Science blogs or sites do you follow?
I don't.
Hi Fred,
I am curious. What other Data Science blogs or sites do you follow?