Open Letter to (one) Data Scientist

*Photo by MervC

Dear Dear Friend, 

As you take on the world of analytics and data science make me one promise... well maybe a few.

1) Don't quantify that which cannot be quantified. Specifically qualitative data. The motivation will be to try to turn human behavior into actionable numbers. You will argue that at a macro level, given many transaction/iterations of a particular behavior, you will be able to draw some sort of grand conclusion based on people's propensity to take particular actions given a controlled set of stimuli. I know this temptation and I have and still fall victim, but these generalizations can be exclusionary and if they are institutionalized (i.e. an organizations adds them to the standard list of KPIs) can eventually become obsolete and wrong and can even end up doing damage to the population in question. I don't have a solution but having worked in analytics before I understand the urge and I thought it was always important to be aware of the weakness of my assumptions and I always tried to downplay the significance of my conclusion to allow for error that I was unable to see/measure/understand.

2) Scientists love building models but they are always wrong. The plethora of variables that influence a biological or natural system are broad an varied. Trying to model these is impossible. The inclination is to limit variables to only the important ones. By doing this an unrealistic "world view" is created. And because you are excited about answers and understanding things more completely it easy to want to broadcast the results and try to apply them to other situations. I get it. But this is a mistake. Your conclusion work in a very limited scenario.

3) People will misuse you ideas. You may understand the limitations of your conclusions but others won't. They invariably want science to give an answer and they will apply your model to scenarios that aren't remotely related.

4) Participant observation has to be part of the question. Anthropologists use this a lot. As an observer they are actually changing what is actually being observed. Instead of controlling for their presence they completely embrace the manner in which they change the observed outcomes and contextualize their presence. It is not uncommon for the opening line of an ethnography to read, "As a white protestant upperclass woman..."

5) When at all possible mingle with hoi polloi. Your numbers and observations are about people. You can build models about their behavior or their click through rate, etc. Your numbers couldn't possibly capture everything that is going on. So roll up your sleeves and mingle with your users as often as you can manage. At the end of the day this is all about relationships, behaviors, and people. As much as you may not want to admit it (I don't know if you do or don't) analytics is a marketing effort that tries to get data that is better than what the respondent can give. Combine your hard numbers with qualitative data to get a better picture. Its also more fun this way.

6) Someone once said that if you want to create a better user experience give your product to a toddler. If they can figure navigation out then your product is intuitive to use. The purpose also has to be really straight forward and obvious enough that a toddler gets it. This is good. You have more going on on your site than a toddler can understand? Maybe things should be scaled back. This is reductionist (as cautioned above) but it helps guide your approach, I think.

7) Have fun. The numbers are hat they are. Metrics are what they are. Behavior maybe what you think it is. This effort is all about getting better. There will be very few big wins especially in a big data environment. Its more about small victories.