Introduction
Everything we do is intended to move us towards achieving some set of goals ranging from just satisfying our immediate hunger to achieving a revolution. These actions are executed within a strategy (or hierarchies of strategies) to varying extents of sensibility and organization. Presumably the effectiveness of our efforts towards those goals and the rhyme and reason behind why be do them are constantly assessed. After all, the world is ever-changing. It is a complex world where everything we do doesn’t take place in a vacuum and it is rife with the constraints of finite resources (costs – time, money, materials, other opportunities) that force us to find creative alternatives. Further, unlike our lives on the living room couch, activities in our work lives usually involve many people and other moving parts, and failure comes with consequences.
So we operate the execution of our enterprise strategies (activities in our work life) within a formal Performance Management framework in order to maximize our chances for success. The intent of Performance Management is to align the efforts of everyone and everything within an enterprise towards its goals. Properly implemented, we ensure we understand our purpose, our goals, develop coherent strategies, identify, assess and mitigate risks, understand how to measure the effectiveness of our efforts, and develop sets of contingencies.
The problem is that development of the strategy and properly thinking through it seems more like an enterprise chore than the primary key to success that it is. Strategy development tends to place too much faith in trends of the recent past and that the actions of our strategies will not trigger cascading, unanticipated effects. Cutting intellectual/analytical corners by failing to validate the strength of beliefs in the business (relationships between tactics and outcomes) or by failing to place more realistic effort into risk mitigation (too optimistic about things that haven’t yet happened) can easily derail the Performance Management effort.
The strategies also need to be novel in order to compete. I’d love a world where I knew exactly what everyone else is going to do because it’s a “best practice” (and therefore the responsible course of action). In such a world, that means our statistics-based predictive analytics models would work surprisingly well. Well, they sometimes do, then quickly stop working after everyone learns about the phenomenon and alters the statistics from which the model is based.
Developing strategy relies on our arsenal of cause and effect built up in our heads over the years. We draw on those experiences to imagine what will happen if we put a series of actions into play towards a goal. It’s when we execute on that strategy that we find the world will not simply go through katas (choreographed fighting) with us. I think the way Performance Management is implemented today doesn’t possess the necessary focus on agility required to fight in a randori world. In this blog, I wish to introduce an aspect to address this in a way that avoids analysis paralysis but also serves as a better warning about the sanity of our strategies.
This blog is intended to simply introduce this concept of an “Effect Correlation Score”, something I try to include as part of performance management projects I engage in. I do think that comprehensive coverage of this concept could take up an entire book as the purpose of the ECS is to profound: ensure our efforts are not just going as planned, but are still valid. The primary take-away is that we must be cognizant of whether our endeavors towards “good” KPI statuses are still valid and sensible in a world that is constantly changing and for which “business” is usually a competition.
Current State of KPIs
The basic elements of a Performance Management implementation are the Key Performance Indicators (KPI). KPIs are just measures of values. A good everyday example of a very basic, objective KPI is the speed of your car. The value, the easily obtained value of 55 mph, in itself only tells us what we’re doing. But in our minds we know that the value also helps us estimate the arrival time at our destination, it tells us whether we are breaking any rules (the posted speed limit), it tells us how capable we are of passing a car within a given distance. Such concepts are conveyed to us through other “perspectives” of the KPI. Currently, those other perspectives include these three things:
- Target – This is the value we’re shooting for; the figure that we should be reaching for. In this example, if the posted speed limit is 55 mph, we should shoot for that speed.
- Status – This is a formula that provides a score quantifying how well we’re doing or not doing, as opposed to the Value itself, which would be what we’re doing. In this case, if the posted speed limit (the Target) is 55 mph and we’re traveling at that speed, the status is good; 45 or 65, not exactly bad, but you’re open to getting horns tooted at your or a ticket; 25 or 95, and you’re looking at a very dangerous situation, maybe involving getting rear-ended and more than just a ticket. The KPI status is often represented by a “traffic light icon”, where the green light is good, the yellow light is warning, and the red light is bad.
- Trend – This tells us how we’ve been doing lately. It may be that the status is currently good at 55 mph, but maybe we’ve been following a wide-load for the past hour, so that in reality, we haven’t been doing well lately. Interestingly, trends normally look at the recent past, but we can also look at the predicted near-future trend. Maybe we see the flashing lights of the wide load truck up the freeway.
I’m proposing another “perspective” of KPIs I call the Effect Correlation Score (ECS). This is a measurement of the correlation between the measure of our efforts and the effect we hope to achieve. For example, happy employees correlate to superior product quality. I will explain how strategies can lose their integrity “in flight” and hard-headedly following through on pursuing good statuses of the KPIs for which conditions have changed can be detrimental if not devastating. However, unlike target, status, and trend, the ECS is really an attribute of the relationship between two KPIs.
KPIs and Strategies in the Complex and Competitive World
Making good decisions involves a balance of learning from the past and being cognizant (even paranoid) about our assumptions. Applying what we’ve experienced in the past to similar situations today is only sensible, and for the most part, it works. However, the world is complex and things change. Additionally, reacting on what statistics tells us is the best answer is precisely what our competitors (or enemies) want us to do (see my blog, Undermined Predictive Analytics).
Strategies that we build to achieve goals are made up of webs of cause and effect. For example, at an extremely high level, happy employees leads to higher quality products, which leads to happier customers, which leads to bigger revenue. Additionally, lower costs along with the bigger revenue leads to bigger profits. Each of these links (edges) such as happy employees leads to higher quality is just a theory. It’s not always true. Maybe sometimes happy employees also leads to complacent employees. Maybe there is only so much that happiness can do to continue raising quality; even if happiness can be infinite, a product can only become so good.
There are many reasons why strategies stop working during the course of execution, beyond simply poor execution:
- Our competitors adjust to counter our strategies. Unfortunately in business, one company’s gain is more often than not another company’s loss. If a company notices the weak points of a competitor and builds a strategy to capitalize on those weak points, the competitor will probably adjust. Therefore, that weak point, a major assumption of that strategy, fails to correlate with success.
- Diminishing returns. Often, a tactic works because it is filling a void. For example, Vitamin C works wonders when our bodies are short of it, but the benefits of increasing Vitamin C diminishes to practically nothing very quickly.
- Desensitization – A variant of diminishing returns. A strategy could stop working because the driving force of incentive can sometimes disappear. Would all workers initially driven to over-achieving simply from being happy to have a job continue to over-achieve?
- Non-linear relationships. This can often be the same as diminishing returns, but this is more complicated. For example, what if Linus Pauling were correct and that mega doses of Vitamin C kick in other mechanisms supporting health in other ways than avoiding scurvy? We would see a rising benefit, a tapering, then a rise again.
- People learn how to game the KPIs. An example is a call center agent goaled on successful resolutions who hangs up as soon as the call seems to be very difficult.
- Managers mistakenly “blow out” the KPI, thinking if X is good, X*2 is great. In this case, managers run up the score to blow away the KPI. Hopefully, that impressive performance would show they’ve outgrown their current position. What often happens is this stellar performance, beyond any wildest dream, often has a side effect.
I realize it’s tough to even suggest that the components of a strategy should be changed in flight (between performance review periods) when conditions change. I’m sure we’ve all made a few snide remarks about a re-org of some sort that sometimes seems like just action for the sake of doing something (I think sometimes it is). There is an inertia that builds up, and every time we adjust course there are the costs of context-switching and taking at least one step back to take two steps forward. But if a theory (a correlation) ceases to be true, we can continue to plug away anyway, attempt to destroy the cause of the change, or blend in with the change.
Continuing to plug away at a strategy in the face of empirical evidence that a correlation no long is valid is a cousin of the “definition of insanity” (Einstein’s doing the same thing over and over and expecting different results). Sometimes we can catch an agent of adverse change (at least adverse for us) early on and nip it in the bud; or catch it late and destroy it at great expense. But very often, victoriously vanquishing the enemy only delays the inevitable. What makes most sense is for us to embrace the change, trying as best as we can to guide the change to our favor.
I want to be clear that by changing the strategy, I don’t mean making wholesale changes. I mean at least starting incrementally by identifying the KPIs for which a “good” status indicator no longer correlate to desired outcomes. This is as opposed to just finding KPIs exhibiting “bad” status and manipulating your activity such that the status becomes good. That good status means nothing if that goodness doesn’t correlate to success anymore. Hopefully, like a good software design, components of the strategy are loosely coupled so that we can replace isolated poorly performing business processes with superior business processes with minimal side-effects.
Note too that it’s not just a weakening of a correlation that could hamper the success of a strategy. Even a strengthening of a correlation could have adverse system-wide effects. For example, if gasoline were suddenly reformulated to convert chemical energy into heat more efficiently, that’s a good thing, but many components of a car would need to be redesigned to handle that change.
The Effect Correlation Score
This Effect Correlation Score of a KPI measures the relationship between a cause and its intended effect. It would most simply be implemented as the linear relationship Pearson Product-Moment Correlation Coefficient (PPMCC) described in my prior blog, Find and Measure Relationships in Your OLAP Cube. The PPMCC is the same calculation used by Excel and the CORRELATION MDX key word. My blog describes what is required for such a calculation, although it describes the implementation in MDX, not SQL. (I just Googled sql pearson correlation and saw many good sites demonstrating the PPMCC with SQL.)
Of course, the PPMCC is just a simple implementation of an Effect Correlation Score. Another of my blogs, Nonlinear Regression MDX, describes a score for more complicated but more accurate non-linear relationships. For very complex relationships, we could implement involve predictive analytics models (regressions are just very simple Predictive Analytics models), particularly the probability of a decision tree query.
The ECS should result in a value ranging from -1 through +1. That is the range of the PPMCC and we could use the 0 through 1 probability of a predictive analytics model. This range of -1 to +1 is the same conventional figures of a status and trend. For the ECS, -1 should mean there is an inverse correlation (one goes up, the other goes down). +1 should mean there is a direct correlation (they go up and down together). Anything in between are grades of correlation, particularly the value of 0 which means no correlation at all.
The ECS in the Performance Management Planning Process
When we’re sitting with our team in a conference room in front of a whiteboard for a Performance Management strategy planning session, we develop our strategies of webs of arrows pointing from one box (a cause) to another box (an effect) where each of those arrows implies an existing correlation. Again, happier employees leads to higher product quality. Or better upstream testing leads to lower costs.
Normally during the PM planning process, at some point before setting the strategy in stone, we need to determine how we can measure the values of each cause and each effect (KPI). If we can’t measure our actions, we’re running blindly. We also need to define what good and bad performance means (the status of the KPI) and the targets (which are really the goals of sub-objectives).
(As a side-note, some effects are causes as well – ex, higher product quality leads to better customer satisfaction which leads to higher return customers which … etc, etc.)
For the ECS, what we should be sure to do is work on pairs of related KPIs as opposed to determining how to obtain the values of all KPIs first, then obtain the status of all KPIs, etc. For each pair of related KPIs, we should validate that there is indeed a correlation between them (the cause and effect). Because if there isn’t a correlation, our logical reasoning down the line could be faulty.
The correlation could be tested between the status or direct value of the KPIs. I tend to think it makes more sense to test the correlation between the statuses of the two KPIs since that is semantically richer. However, the status could be calculated such that the correlation (or lack of correlation) could be misleading. So I usually run a correlation on both to see which makes more sense.
A problem could be that the values for KPI statuses over time are not recoverable. For example, there may not be a record of target values (usually a component of a KPI status calculation) for sales for all time periods (months, quarters), just the latest. But KPI values such as sales over time should always be recoverable to run a correlation.
Whichever is chosen, this is validation of at least part of the overall theory. For the future (during the course of the execution of the strategy), we would want to periodically check if the correlation continues to exist, which is the main idea of this blog.
Where Does the ECS Go?
On a Scorecard, there is one row for each KPI. Each row shows the KPI name, the value, the target, and a cute icon for the status and trend. They fit nicely one a single row because there is a one to one relationship between a KPI and its value, target, status, and trend. In reality though, most causes effect more than one thing.
However, in a Performance Management system, there is probably one primary effect for each cause. If we want to preserve the nice matrix display of one KPI per row, we could select the primary effect and add to each row, the name of the primary effect and a cute icon for the ECS. For any actions we take, there is usually a primary reason. For example, I know right away that I need to earn money to pay the bills, although there are many other minor reasons.
I like to denote strong correlations (ECS value closer to 1) with green, no correlation (ECS value nearer to 0) with cyan, and an inverse correlation (closer to -1, one value goes up, the other goes down) with blue.
Performance Considerations
This PPMCC calculation should be not much more expensive that a typical trend calculation commonly implemented as a slope of recent values. That would require the SUMs of many facts over say the past few time periods (days, weeks, months, quarters, years, etc) and a few squares and square roots of this and that. The PPMCC is really just a step beyond calculating a slope. In a relational database, the heavy work will be the IO involved in loading the same data required for the trend (slope), but will just do a few more calculations. Predictive Analytics models should be even quicker since they are really distilled into a set of rules (from a large amount of data) as opposed to calculating off that raw data itself.
Which KPI relationships Needs an ECS?
Implementing these correlation scores is not as unselective as implementing the PPMCC calculation between KPIs. For example, not all KPIs would benefit from an ECS. For that matter, not all KPIs require a target, status, or trend. In fact, an ECS for some KPIs could be completely meaningless or worse yet, misleading. For example, it would not be entirely true to say that a great status for the revenue KPI results in a correspondingly improved status for the profit KPI. That’s because revenue is just one factor of profit.
An ECS doesn’t need to be constantly monitored either. At the least, we should periodically validate that relationships incorporated into our strategies are still valid and haven’t slipped into mythical status. We need to also ensure that the granularity and seasonality are taken into account. Correlations may not exist at a day to day level but may appear at the monthly level. There may be seasonality involved that do not exist at certain times of the years.
Correlations are Not Silver Bullets
If correlations such as those found in calculations like PPMCC values or in data mining models worked very well in reality, it would certainly be a different world. After all I wrote above, I do need to include a disclaimer about the folly careless analysis of data. There are so many ways to misinterpret correlations, so many reasons why strong correlations could be meaningless, how a higher level lack of a correlation could mask a lower-level one of great consequence.
Nonetheless, when someone at work tells me something such as an MBA does or does not equate to higher project management success, I know I can run a few queries to help prove or disprove such a statement. I know I need to think through any possible misinterpretation. At some point I need to either be satisfied that the numbers do back up the statement, go against it, or it’s still inconclusive. If I do come to the conclusion that there is a positive correlation, then the possession of an MBA goes into the plus column for my strategy to hire superior project managers. My point is to continue checking that correlation to ensure that it is true. Otherwise, as I mentioned above, our actions become a case of the definition of insanity.
Other KPI Ideas
There are many aspects of KPIs that are not sufficiently addressed beyond trends, status, and the ECS. For example, I mentioned earlier that trends normally show the recent activity allowing us to infer a future state of the KPI. But a predicted trend would help us better infer that future state. In fact, multiple future trends dependent on a range of possible inputs would be even better. Another example is that statuses could be different from different points of view. Could a sales KPI drastically exceeding expectations be the same for the sales manager and the inventory manager who must deal with the unexpected demand? Would it help for a sales manager to know that the sales KPI for the inventory manager is bad?
By the way, Map Rock does a lot of this stuff …
Soft Coded Logic 