I spend a lot of time working with data. One thing I see is that organizations often grow increasingly confident in their decisions as they collect more data. They build dashboards, establish key performance indicators (KPIs), and track changes over time. These are useful practices, but there is an assumption underlying them that deserves a closer look: that the numbers we measure provide an accurate representation of what is actually happening.
Measuring business performance is obviously a good thing. Without some way of observing outcomes, it becomes difficult to distinguish between a decision that produces results and one that merely feels productive. Analytics give us a way to identify patterns, compare performance, and test our assumptions against observable evidence. However, the relationship between a measurement and the thing being measured is more complicated than it initially appears.
The Difference Between Measurement and Understanding
Consider a company that measures website traffic as one of its primary marketing KPIs. During a particular month, traffic increases by 40 percent. The company might interpret this as evidence that its marketing strategy is working, particularly if the increase follows a new campaign or a significant investment in search engine optimization. The problem is that the observed increase does not, by itself, establish that the marketing strategy has become more effective.
Website traffic is a measurement of activity. Marketing effectiveness is a much broader concept involving the relationship between that activity and the outcomes the business wants to produce. An increase in traffic could indicate that a company is reaching more potential customers. Still, it could also reflect an increase in irrelevant visitors, automated traffic, or people who have no interest in purchasing its products or services. The same measurement can therefore be consistent with very different explanations of what is happening.
This distinction is important because analytics systems generally measure variables that we have selected as representations of something else. We cannot directly measure concepts such as marketing effectiveness, customer trust, or organizational health in the same way that we count website visits or calculate revenue. Instead, we identify observable indicators that provide evidence about these less directly observable phenomena.
In statistics, an observed variable can serve as a proxy for a construct that we want to understand. A proxy is useful when it provides information about the underlying construct, but its usefulness depends on how closely the two are related and under what conditions that relationship holds. Website traffic can be a reasonable indicator of audience reach, for example, while being a relatively poor indicator of revenue growth. The problem arises when we interpret a measurement beyond what it can reasonably tell us.
A Business Is a System, Not a Collection of KPIs
One reason individual metrics can be misleading is that business outcomes emerge from the interaction of multiple variables. Revenue, for example, depends on more than the number of leads entering a sales pipeline. It also depends on lead quality, pricing, conversion rates, customer retention, operating costs, market conditions, and the organization’s ability to deliver its products or services. These variables influence one another, which means that changing one part of the system can produce consequences elsewhere.
Suppose a company invests in a new marketing campaign that doubles the number of leads it generates. If the sales team has sufficient capacity and the additional leads are qualified, the campaign might produce a substantial increase in revenue. However, if the additional leads are poorly qualified, the sales team may spend more time pursuing prospects who are unlikely to convert. The company has improved its lead generation KPI, but the additional activity may produce little financial benefit while increasing operating costs.
The relationship becomes even more complicated when we account for feedback. An increase in lead volume can affect the workload of the sales team, which can influence response times and conversion rates. Lower conversion rates might then lead management to increase marketing expenditure in an attempt to compensate for the apparent decline in sales performance. That additional expenditure can generate more leads, placing further pressure on the same sales process. A decision that initially appeared to address one problem can therefore reinforce the conditions producing another.
This is a systems problem. The behavior of the organization depends not only on the values of individual variables but also on the relationships among them. Consequently, optimizing a single metric does not necessarily optimize the performance of the system as a whole.
This becomes particularly important when businesses introduce automation. Automating a process can reduce the time required to complete a task, but the resulting improvement depends on how that process interacts with the rest of the organization. If the original process is poorly designed, automation may simply allow the organization to perform the wrong process more quickly. The measured efficiency improves while the underlying operational problem remains.
KPIs Are Proxies for Objectives
There is another reason we need to be careful with metrics: the act of measuring something can influence the behavior of the people being measured. Once a KPI becomes a target, people begin making decisions based on how their actions affect that indicator. This is a predictable consequence of using measurements to evaluate performance, and it can produce results that differ from the original objective.
Imagine a customer support department whose employees are evaluated according to the number of tickets they close each day. The metric is easy to calculate, and it provides some information about productivity. However, closing a ticket and resolving a customer’s problem are different events. If employees are rewarded primarily for closing tickets, they have an incentive to prioritize speed over resolution, particularly when difficult cases require substantially more time.
The organization might observe an increase in tickets closed per employee and conclude that productivity has improved. Yet customer satisfaction could decline, repeat contacts could increase, and the total amount of work required to resolve each problem could rise. The metric has influenced the behavior of the system, producing an outcome that the original measurement was not designed to capture.
This problem is closely related to Goodhart’s law, commonly summarized as the observation that when a measure becomes a target, it often ceases to be a good measure. The underlying mechanism is relatively straightforward: a metric is initially useful because it correlates with an outcome we care about, but optimizing directly for the metric changes the conditions under which that relationship holds.
This does not mean KPIs are inherently problematic. It means we need to understand what they represent and how people are likely to respond when their performance is evaluated against them. A useful KPI should be connected to a meaningful organizational objective, and its interpretation should account for the other outcomes that might be affected when people optimize for it.
For example, ticket closure rates become more informative when considered alongside resolution rates, repeat contacts, customer satisfaction, and the complexity of the cases being handled. These additional measurements do not eliminate every possible distortion, but they make it more difficult to mistake activity for effectiveness.
The Problem of Causality
Another common mistake in analytics is assuming that a change in one variable caused a change in another simply because the two occurred together. This is particularly common when evaluating marketing campaigns, software implementations, and operational improvements because organizations often introduce a change and then observe whether their KPIs improve.
Suppose a company launches a new email marketing campaign in January and experiences an increase in sales during February. It is reasonable to investigate whether the campaign contributed to that increase, but the temporal relationship alone does not establish causation. Sales might also have increased because of seasonal demand, changes in pricing, improvements in the sales process, or opportunities that had entered the pipeline before the campaign began.
The difficulty is that several mechanisms can produce the same observed outcome. If we attribute the increase entirely to the email campaign, we may overestimate its effectiveness and make future investment decisions based on an incorrect explanation. The campaign may have contributed to the increase, but determining the extent of that contribution requires additional evidence.
Causal inference is the process of distinguishing genuine causal relationships from associations that can be explained by other factors. Depending on the question, this might involve controlled experiments, comparisons with an appropriate control group, statistical adjustment for confounding variables, or analysis of how outcomes change under different conditions. Each approach has limitations, but the important distinction is between observing a pattern and establishing a mechanism that explains it.
Business decisions are usually based on expectations about what will happen if we repeat or modify an intervention. A correlation may help us predict an outcome under certain conditions, but a causal explanation gives us a stronger basis for deciding whether changing a particular variable will produce the desired result. Without that distinction, we risk treating coincidental improvements as evidence of successful interventions.
Every Analytics System Has Blind Spots
The limitations of analytics also depend on the information available to the system. A dashboard can only represent the variables that are collected, recorded, and incorporated into its calculations. Anything outside those boundaries is either invisible to the system or represented indirectly through other measurements.
Consider a company that evaluates its website using a conventional analytics platform. The platform records visits, page views, traffic sources, and tracked conversions. A potential customer visits the website, reads several pages, and then calls the company two days later to discuss a project. If the phone call and subsequent sale are not connected to the original website visit, the analytics platform may record the initial activity without recognizing its contribution to the eventual outcome.
The resulting report is not necessarily inaccurate. It is incomplete with respect to the question the company is trying to answer. A measurement can be technically correct while still producing a misleading interpretation when used outside its intended context.
The same issue occurs in customer relationship management systems. A CRM might record that a prospect declined a proposal, but unless the reason is captured, the organization cannot distinguish between a pricing problem, a timing issue, a lack of trust, or a decision to purchase from a competitor. Each explanation suggests a different response, even though all of them produce the same recorded outcome.
Missing information can therefore affect more than the precision of a measurement. It can affect the explanation we construct from that measurement and the decisions that follow. This is why data quality is not simply a technical concern involving missing fields or inconsistent records. It is also an epistemological concern: a question about what we can reasonably know from the information available to us.
Adding more data does not automatically resolve the problem. If the additional data measures the same limited aspects of a system, it may increase the volume of information without improving our understanding of the underlying process. What matters is whether the data provides evidence relevant to the question we are investigating.
Time Changes the Meaning of a Metric
Business performance is also dynamic, which means that the significance of a measurement can depend on when it is taken. Some metrics reflect outcomes that have already occurred, while others provide information about processes that may influence future outcomes. These are commonly described as lagging and leading indicators, respectively.
Revenue is a lagging indicator because it records the results of activities that have already taken place. The number of qualified opportunities in a sales pipeline, by contrast, can provide information about potential future revenue. If a company’s current revenue is increasing while its qualified pipeline steadily declines, the two indicators describe different aspects of its performance. Current revenue may reflect the success of earlier activities, while the declining pipeline may indicate a problem that will become visible only later.
The reverse situation is also possible. A company might introduce a new operational process that initially reduces output because employees must learn a different way of working. If the process subsequently reduces errors and improves the organization’s capacity, the initial decline in productivity may be part of a longer-term improvement. Evaluating the intervention solely through short-term output would miss the relationship between the initial adjustment and the later outcome.
This is one reason comparisons across time require careful interpretation. A metric can change because the underlying system has changed, because the environment has changed, or because the measurement captures different stages of a process. Without accounting for those possibilities, we may interpret a temporary fluctuation as a structural change or mistake a delayed consequence for an unrelated event.
The relevant question is therefore not simply whether a KPI increased or decreased. We also need to understand the time horizon over which the outcome develops and the processes that connect present conditions to future results.
Quantitative Data and Qualitative Evidence
There is a tendency to treat quantitative data as objective and qualitative information as subjective. The distinction is understandable because numbers allow us to perform calculations, compare observations, and evaluate patterns in ways that are difficult to achieve through anecdotal evidence alone. However, the apparent precision of a number does not guarantee that it provides an adequate explanation of the phenomenon being measured.
A customer satisfaction score, for example, can reveal differences between customers, products, or periods of time. It does not necessarily explain why those differences exist. Two customers may assign the same score for entirely different reasons, and a change in the average score may conceal substantial variation among different customer groups. The number summarizes observations, but the aggregation can remove information that is important for interpreting them.
Qualitative evidence can help identify mechanisms that quantitative measurements do not directly capture. Interviews, customer feedback, observations of work processes, and discussions with employees can reveal why a particular outcome occurred or why a metric is failing to represent the underlying experience. These methods have their own limitations, including selection effects and the difficulty of generalizing from a small number of observations, but they provide a different kind of evidence.
The two approaches are most useful when they inform one another. Quantitative analysis can identify patterns that deserve further investigation, while qualitative research can help explain the processes that may have produced those patterns. The explanations can then be examined against additional data to determine whether they remain plausible.
This approach is particularly valuable when a metric changes unexpectedly. Rather than immediately assuming that the change reflects improved or deteriorating performance, we can investigate the conditions under which it occurred and identify possible explanations. The objective is to construct an explanation that accounts for the evidence, rather than simply assigning a positive or negative meaning to the observed number.
Designing Analytics Around the Questions We Need to Answer
These limitations have practical implications for how analytics systems should be designed. The starting point should be the decision or question the organization needs to address, rather than the data that happens to be easiest to collect. Once the objective is clear, we can identify the measurements that provide useful evidence about it and determine what additional context is needed to interpret them.
For example, a company investigating declining profitability might initially focus on revenue because it is one of the most visible measures of business performance. However, profitability depends on both revenue and costs, and the relevant costs may be changing at different rates. Revenue could be increasing while margins decline because customer acquisition has become more expensive, fulfillment costs have risen, or the organization is spending more time supporting each customer.
A useful analysis would therefore examine the relationships among revenue, gross margin, acquisition costs, retention, and operating expenses. Depending on the business model, it might also investigate differences between customer segments, since an increase in total revenue can conceal a shift toward customers who are less profitable to serve. The purpose of these additional measurements is to identify the mechanisms behind the change, allowing the organization to distinguish between several explanations that would otherwise appear similar.
However, there is a limit to how many metrics are useful. Collecting and maintaining data has costs, and every additional indicator creates another potential source of ambiguity if it is not connected to a clear analytical question. An excessively complicated dashboard can make it harder to identify the variables that actually matter.
The objective should be to develop a model of performance that is sufficiently detailed to support the decisions being made, while remaining explicit about what it does not capture. For a relatively simple operational process, a small number of KPIs may provide adequate information. For a more complicated business problem, we may need multiple indicators, qualitative evidence, and a better understanding of how the underlying processes interact.
In either case, the quality of the analysis depends on the relationship between the measurements and the questions they are intended to answer.
Analytics as a Way of Reducing Uncertainty
I do not think these limitations diminish the importance of analytics. If anything, they make a stronger case for using it carefully. Without measurements, organizations have little basis for evaluating their assumptions, identifying patterns across large numbers of observations, or determining whether an intervention has produced the expected result. Analytics provides a structured way to investigate the world, but the conclusions we draw must remain connected to the evidence the system actually provides.
The difficulty is that business decisions rarely involve complete information. We usually observe only part of the system, and the relationships among its variables may change over time. We therefore need to treat our understanding of performance as something that develops through observation, analysis, and the evaluation of competing explanations. When new evidence becomes available, our interpretation may need to change.
This also explains why a good analytics system should support investigation rather than simply report whether targets have been met. A KPI that falls outside its expected range can be useful because it identifies a condition that requires attention. However, the metric does not determine the appropriate response. That requires an explanation of the processes that produced the result and an assessment of how different interventions might affect the broader system.
Ultimately, KPIs provide partial observations of a much larger set of relationships. Their value comes from the information they provide about those relationships, not from an assumption that the numbers represent the entire system. The more complicated the system, the more important it becomes to distinguish what we have measured from what we have inferred, and what we have inferred from what the evidence actually supports.
Analytics can tell us a great deal about how a business is performing, but understanding why it is performing that way requires more than reading a dashboard. It requires an understanding of the system that produced the data, the limitations of the measurements, and the mechanisms connecting the variables we observe to the outcomes we care about.
