Creating a Metrics-Driven Organization

James G. Robinson, Director Web Analytics, The New York Times

Intro

• Mission for our group two years ago: put web analytics in our DNA
• Make decisions based on data, not (just) gut
• So,
• What is a metrics-driven organization?
• If we are not a metrics-driven organization, is it even possible for us (as web analysts) to transform the organization as a whole?
• If so, how?
• If not, what can we do?

• Metrics: Not just numbers; insights – what’s really happening
• Driven: Not just dedication; focus – movement towards something
• Organization: Not just the company; the people – social network, individual motivations

• Needs top-down buy-in, metrics-based goals, across depts (CEO)
• Needs powerful champion – get all the way to the top
• People acting in self-interest (want bonuses!)
• "You don’t get measured – you don’t get a budget
• internal "measurement board" = consensus for key measurements
• Metrics-driven org not goal: $ = goal
• Can only ecommerce sites be metrics-driven?


• "A journey that continues – never ends"
• Web analytics requires culture of iterative development (?)
• Old-school software development needs a start and an end
• Some don’t like continuous things
• Create a "swim lane" for optimization


• Metrics-driven org requires perfect data?
• Web analytics is statistics, not accounting
• There is no perfect data
• Sometimes we don’t even believe our own data
• Industry has touted measurability = expectation of perfection
• Do caveats = uncertainty?


• Danger in being TOO data-driven (data spam?)
• Role of analyst is to subtract data, not add it
• Ask the right questions – give the right data at the right time
• "Tyranny of KPIs"


• "Part of analytics is courage"
• #1 reason for web analysts leaving a job: "what they say is not valued"
• Turn them loose


• Task: Discover and deliver bad news
• See that something’s wrong: give the person a chance to fix it
• Defense mechanism: attack the numbers


• Task: Discover and deliver good news (even if it’s not good)
• "Metrics as a blunt instrument"
• Data to prove a point


• Analyst as salesman
• Build the business case
• Tell a good story
• Get decision-makers involved
• Sell them on the #’s throughout the whole process
• Tell them how much it’s worth
• Avoid jargon
• Move quickly


• Need to earn trust – not just handed to you
• Earn the permission
• Get buy-in, then deliver
• Show why it’s important (and that we are not just "report people")
• Schwab: be competent, skilled, and deliver deliver deliver
• Otherwise others will do it themselves


• Changing mentality of a company (vs. building web analytics group)
• One report at a time, one person at a time
• From "metrics-aware" to "metrics-driven" (i.e. Nokia)
• "Nothing changes unless it has to"
• an external force is always the catalyst
• Almost impossible to drive change internally (?)
• But you can change thinking
• A lot depends on a company’s attitudes towards "failure"
• "Fail fast, then fail better"
• Maturity: metrics focus stays with the company even if people leave


• Should analytics "own the insight"?
• In a metrics-driven org, analytics and data are decentralized
• Are you a service group or a business partner?
• Create a demand for the right data, not just data