Not all organizations are the same when it comes to data. There are many factors that determine the what/when/how data is collected, analyzed, and presented. However, I think after working with a number of companies over the years, that they fall into four basic levels. For a variety of reasons it can be difficult to move from one level to another.
In this post I’m going to describe these levels, as I see them, for Digital Marketing & Ecommerce, why an organization would be at that level and what it takes to move to the next level.
Organizations at Level 0 have virtually no Analytics capabilities. They may have a Google Analytics tag on their site and their website platform may provide some Analytics capabilities. If they are using a media agency, they are trusting them to deliver performance metrics for media (if you think that might be like the fox guarding the hen house, you would be right and I have stories).
They are usually only looking at a few KPIs and they don’t really know if the data collection is accurate or if the KPIs are correct since there are no internal Analytics resources. Also, the KPIs they look at are seldom used in decision making, as they should be.
Organizations at this level don’t really grasp the power of data and how it can positively impact their business. I have found that it is rarely due to budget constraints, they simply don’t find data a factor in business decision making (that’s why I called this Level 0, zero Analytics). Instead, they use gut instinct to make decisions and the limited Analytics is an FYI.
Organizations at this level sometimes believe in the next big thing: CRM, Big Data, AI, Machine Learning, Data Lakes, etc. They often trust platforms more than people because they don’t really understand how Analytics works and they have no one to help them (or they don’t care to listen). Many of the digital marketing platforms that have been developed over the years are targeted to organizations at Level 0. They sell in the platform as easy to use, it does all the Analysis and sometimes is named or marketed with the flavor of the month – AI That, Machine Learning This, etc. Often these systems are very expensive. However, no system will ever replace the insights of a good Analyst.
For an organization to move to the next level they need to understand the power of data. This is very difficult, since it requires a rethinking of how the business is run. However, it’s not impossible. A new CEO or other high level executive that sees the value in data or a cleaver employee that can demonstrate how Analytics can positively impact the bottom line may be enough to allocate resources (for a full-time Analyst).
An organization at Level 1 understands the value of some of the data. They have limited internal resources, a single Analyst or two and do not have an Analytics department. Data collection will be more sophisticated with an internal database, AWS or GCP setup and would use a tag management system.
A Level 1 organization would not be relying on media agencies for media analytics. They might even have media handled internally. Reporting would be through a web interface and could be anything from a home-grown system to Tableau.
A Level 1 organization would have a limited set of KPIs that they are interested in. They use these KPIs to make business decisions & react to the up and down changes in these KPIs.
They also incorporate A/B testing, since they understand the power of data.
However, at this level, A/B testing is very limited and often not implemented correctly. A/B testing is often used just to confirm a hypothesis. Without letting the data tell the story, it is tortured to try to confirm the hypothesis.
Using a limited set of data, be it aggregated or segmented, means that opportunities can be missed and the numbers can be misleading (see Simpson’s Paradox, for example).
To move to the next level is easier at Level 1 than Level 0. Creating an Analytics department would likely lead to additional Analytics resources and the power to dig deeper into the data, to determine what is important, how to run proper A/B tests and develop reporting.
An organization at Level 2 has an Analytics department. So there is likely an experienced Director or Manager of Analytics. They have the infrastructure of a Level 1 organization, but it is more fully utilized and additional resources are incorporated to fill gaps in data collection.
For example, since Google Analytics aggregates data, either a paid version of Google Analytics or something like Snowplow would be implemented to gain access to raw data.
Programming and stats would play a bigger role in a Level 2 organization. The use of R and/or Python (or Julia, etc.) would be likely. Using something like Shiny would allow Analysts to better dig into the data and share with stakeholders. This would be the level at which statistics would be commonly used and incorporated into reporting.
Since the data gaps would be mostly closed, end-to-end analysis could be done and it would be expected for deeper insights. A/B testing would follow a more rigorous step-by-step plan and it would be broadly in use.
Not too much is missing at this level. However, it can still be improved. To move to the final level, Data Science is needed. It’s the easiest to move from Level 2 to Level 3. The organization knows the value of their data and the power of Analytics.
This is the final level. This is the level where an organization is fully utilizing their data. At this level, there is likely a separate Data Science department. The Analytics & Data Science departments work together to analyze past data and use it to build predictive models which are shared with stakeholders.
At this level, media is fully run internally and predictive algorithms are used in addition to standard reporting to optimize spending, placements, creatives, etc.
A/B testing is widely utilized, however, it is augmented by the use of predictive models that can apply results to a wider range of web elements. For example, a test showing two different products can be turned into a predictive model by adding variables related to the users to predict future preferences.
I think it’s important to understand where your organization’s Analytics maturity is, or, if you are a consultant, where your client’s Analytics maturity is. In either case, knowing this can reduce the frustration you may feel in trying to move the organization forward.
I used to think only about how I thought Analytics should be done, but not whether the organization was ready. At times, I could not understand why they would not follow my recommendations. Understanding an organization’s Analytics level allowed me to better understand what was doable and what may be beyond their capabilities at that time.
Understanding an organization’s Analytics level will allow you to adjust your expectations and get on with the Analytics that are possible and avoid pushing the Analytics that are simply out of reach (at least for the time being).