So what?!

There is a big demand for analysis of data in all companies. We love data. We love analysis based on fact, and that means data. But that’s not enough.

The method of doing analysis in an environment where developer and analyst overlap goes like this:

  1. Find the data, make sure it is clean and consistent to the nth degree; no approximations.
  2. Discover the important questions
  3. Answer the important questions
  4. Think about what would be the perfect answer to those questions
  5. Provide the “so what?” analysis

My experience recently has been with a few large datasets where we would hope that the application of a few GROUP BYs and some graphs would let the data speak for itself.

However, I have generally found that is not the case.

You can ask specific questions and get specific answers but dashboards don’t just appear out of thin air. And in the manner of “Zen and the art of motorcycle maintenance” there are far more questions than you have time to answer them, just that not all of the questions are interesting and not all of them will produce a result that you can use to change your business.

So how do we get to the “so what?” answers?

I think that an essential part has to be a great tool that allows you to quickly try out ideas. That means the tool must be

  1. Quick to use, low friction on getting and manipulating data
  2. Plenty of “batteries included” that will make use of the work of others
  3. Must have a REPL or something like it that allows and encourages you to “play” with data, rather than static processes like ETL
  4. Must have plotting and graphics.

So I looked and saw this: and really liked it

http://nbviewer.jupyter.org/

It’s a way of turning python code (notebooks) into interactive documents with living code.

So you put your python notepad into Git and then feed the git address to this thing and bingo. It’s the cloud, innit. And the notebooks look great. Very much in the manner of Magic Ink. Also looks very familiar from my days using Mathematica (great days they were too).

There are many other notebook like systems of course, and different tools support different languages. One of the most polyglot is Beaker which allows you to mix python, R and JavaScript. Which might not mean much to your analysts, but to the devs it looks awesome (and a train wreck too).

Just remember, you aren’t making software here! This is the new whiteboard: you throw up some ideas, ram the data in and see what happens. Just because it’s made of code and attached to Git, it doesn’t mean that this is software you want to keep.

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s