Levels of Pain – Refining the “Bad” side of the KPI Status

Everything we do towards achieving goals involves costs – sacrifice, investment (a positive way to look at it). We purposefully put things we already have (time, money, our career) to be consumed or at risk in the hope of achieving a goal. For the sake of this short blog, I’ll call that investment “pain”. I present this blog for several reasons:

  1. KPI statuses are usually too simple. Most are a percentage of an actual value towards a target value. But in real life, there are real events that happen at certain thresholds which completely change the relationships between things.
  2. There really are no free lunches. Even if a cost is not readily evident, it will remain a credit in the account of the Complex World to be cashed in sometime, somehow.
  3. Life isn’t really as rigid with artificial boundaries. The systems of life mostly allow for some leeway, even though there may be an optimal value.

With that in mind, I’d like to explore the definition of a KPI’s “bad” status. Measures of a KPI status are not simply on smooth continuum of values (-1 to 1) ranging from terrible to bad to OK to good to great. This simple take on a KPI status was necessary at first because of limited data and processing capability. But in the Big Data era, we can do better by recognizing conditions as a status, not just a calculation.

Events start to happen at thresholds. The bad side of the continuum is really a set of progressively severe stages, each with its own continuum. A nicely graphic example is how the degradation of a car’s tires affect the car’s ability to move along a road progresses through these stages (along with the colors I use):

  1. Warning (pink)- The car tire is balding. This continuum starts with significant wearing of the treads to complete disappearance of it.
  2. Pain (red)- The tire pops, however, a car can still physically move. The tire will progressively shred as the tire rolls along, but you can still get the car off the freeway.
  3. Major Pain (maroon)- The tire breaks off the rim. The car can still move even though it will be sparking on the asphalt.
  4. Broken (black)- The axle breaks. At this point, barring gravity or a tow truck, the car cannot move.

Investment can also demonstrate these stages of pain:

  1. Warning – An investment of money that doesn’t impede cash flow or the ability to keep the “doors open”. The Warning goes away once we’ve recouped our investment.
  2. Pain – We’ve missed payments on a bill. At the high end of the continuum, we begin to receive Dunning letters and even court orders.
  3. Major Pain – Cash flow is impeded, we’re forced to sell of things, our wages are garnisheed.
  4. Broken – The doors are closed on us. We’re out on the street.

Most healthcare procedures such as curing cancer involves trading one Major Pain (death would be “Broken”) for one or more lesser Pains (side-effects).

When taking a 7am flight, but deciding to sleep until 4am, we can:

  1. Warning – Cut it close getting in time to board early.
  2. Pain – Get there boarding late missing out on overhead luggage space.
  3. Major Pain – Needing to run clear across the airport while your name is announced over the PA.
  4. Broken – Missing your flight.

The stages could be different, but the point is that clearly different sets of events are triggered at certain thresholds. Things don’t just become progressively bad from a balding tire to a broken axle. We can live with a balding tire while we ensure our water and electricity stays on.

In the implementation of a KPI capable of reflecting these events, the status calculation requires a CASE-WHEN-THEN statement for each level of pain. To keep with the convention of returning -1 through 1 (bad to good) for a KPI status, I return (-1 and <-.75)=Broken, (>=-.75 and <-.5)=Major Pain, (>=-.5 and <-.25)=Pain, (>=-.25 and <0)=Warning, 0=OK., and anything greater than 0 is Good to Great.

If KPI statuses are set up with these stages in mind, we will have a better idea of the real consequences as we prioritize. Being cognizant of these thresholds helps us prioritize the many pains that we need to address throughout an enterprise or just our lives at any given moment. Prioritization really is deciding what is most important at the moment. But that importance depends on the consequences for failing to address the pain.

If KPIs are the nerves of the “intelligence” of an enterprise, the KPIs should be linked to whatever consequences (effects) that may be triggered. Keep in mind too that there probably isn’t just one progression of pain, but varying progressions depending upon other circumstances. However, that gets into another story, so for now, let’s just start with recognizing that pain isn’t a smooth continuum. (Note: It was this reason that lead me to develop SCL to define relationships in a very robust manner.)

Consider too that warning isn’t necessarily a “bad” thing. As I’ve defined warning in this blog, it very often can be thought of as an investment or a sacrifice that we can live with. As I mentioned, every endeavor towards a goal, the resolution of a problem, requires some sort of investment. In other words, we purposefully put certain things in a level of pain we can live with in order to fix worse pain, which is sacrifice.

This notion of purposefully putting certain aspects of our lives or enterprise into pain (investment, sacrifice) opens the door for a twist on optimization. It is a technique I developed back in 2008 that I call Just Right Targets. The idea is to set up the KPI bad statuses as described here, select a threshold we’re willing to tolerate, select values we can manipulate (in what-if fashion), and run a genetic algorithm finding a set of those values whereby we do not exceed any of the pain thresholds.

For example, the purpose for Project Management is to deliver a product with the features required to provide intended value (scope), on time, and within budget – hopefully without having trashed the “resources” (burning out the workers and wrecking the equipment) for the next project. It’s the project manager’s job to balance the competing goals of scope, time, and resources:

  • If it’s discovered that new features are required, the project manager must negotiate a reduction in a combination of time and resources.
  • If the project deadline must be delivered earlier, say for competitive reasons, scope is reduced and/or resources are added.
  • If the key developer is hit by the proverbial bus, and there is no replacement, need to extend the timeline and/or reduce the scope.

Just Right Targets accepts the maximum acceptable pain bucket for each KPI (at the highest level for this example, scope, time, and resources), runs what-if forecasts on a wide range of scenarios, and scores each scenario based on the minimum pain it will cost, with preference to scenarios that do not cross any pain margins.

Posted in BI Development, Cutting-Edge Business Intelligence, Data Mining and Predictive Analytics, Map Rock | Tagged , | Leave a comment

The Magic of the Whole is Greater than the Sum of Its Parts


If businesses were people, they would lumber about in a vaguely purposeful manner like zombies. That’s due to the top-down, military-style hierarchies of modern corporations that result in integration of information only at the top and only to a limited extent below. Imagine Gepetto the CEO (external puppet master) of Pinocchio. Pinocchio is managed through strings from the CEO manipulating each of Pinocchio’s parts. The movements are jerky and not very lifelike. When Pinocchio becomes a real person managed through his completely integrated brain, his movements are smooth and lifelike. He can control and grow his life more effectively this way than through the indirect, latent, imperfect information-driven command of Gepetto.

This isn’t a criticism of how businesses are run today. Business enterprises are well beyond the capability of a single person to control it to the level that the enterprise appears “lifelike”. But taking Performance Management and Process Management to the next level supplements what is needed to achieve that “lifelike” movement in an enterprise. Businesses obviously have succeeded executed top-down from a command center (Gepetto’s brain) as opposed to the distributed, networked intelligence of the parts (Pinnochio’s brain). Businesses have produced valuable goods for their customers, met targets, supported the livelihood of employees and investors, and innovated. But most businesses don’t make it and for the ones that do, there was a lot of luck along the way, sometimes they made it in spite of themselves, and eventually do die.

Business Intelligence is supposed to provide the information required for decision makers to make better decisions. Although BI made significant impacts towards that goal, it still hasn’t quite made businesses look more like a world-class scientist or athlete than the lumbering zombie. So Big Data comes to the rescue … or does it?

At the time of this writing, one of the major reasons for why a Big Data project may yield underwhelming results is that it’s simply more data. There is no question that the availability of more data is beneficial, but most businesses still don’t know how to effectively analyze the data it already has and to implement actions off of it. On the other hand, more data can lead to counterproductive information overload.

So what happened to the huge buzzword of about ten years ago (circa 2005), Semantic Web? It’s still around, but it’s taken a seat far in the back of the buzzword bus. Yes, dealing with the complexity and mess of webs of relationships is more difficult than dealing with the tidy matrices of data (tables/spreadsheets) or even objects (aggregates in NoSQL terms). But we need to have relationships between data at least keep pace with the growing amount of data, or we just end up with information overload. Sure, we can find a needle in a haystack with Big Data, but so can a magnet.

In this blog, I present some of my thoughts on the feasibility and value of purposefully pursuing and measuring the level of integration of intelligence within a business, even if such an effort doesn’t address a clear and present need.

This blog is perhaps too philosophical for the tastes of the primarily “technical” audience. But as I look through the zoo of Apache projects around Big Data and all the skills required of a data scientist, it innately seems much too complicated. So at least for me, I need to take a few steps back in an attempt to see the single big picture of Big Data which cannot be captured in merely a thousand words or a catchy marketing phrase such as “the three Vs”, but can be felt by my consciousness.

Measuring a Continuum of Consciousness

On my flight home for Christmas Vacation some thoughts reminiscent of Map Rock were triggered by an incredibly intriguing article in the Jan/Feb 2014 issue of Scientific American Mind titled, Ubiquitous Minds, by Christof Koch. The article discusses the question of what things have consciousness. Is consciousness strictly a human phenomenon (at least on Earth today) or do other animals and even things possess it albeit to lesser degrees? The article suggests that it’s more of a continuum for which we can measure the degree.

That article includes introductions to two concepts, one called panpsychism and another called Integrated Information Theory. For the former, panpsychism, it would be too much of a stretch for this blog to place it in a business context.  However, I can for the latter. In particular, a crucial part of Integrated Information Theory is this notion of a measurement of consciousness referred to as Φ (phi). From a philosophical point of view, the notion of measuring consciousness in non-human, even things seeming completely non-sentient would drastically change the world view of many. Growing up Buddhist, especially in Hawaii, that notion isn’t so foreign to me. From a BI practitioner’s point of view, this is very compelling since I’ve always thought of each business as individual organisms competing in various levels of ecosystems.

The Scientific American Mind article  is actually an excerpt from Christof Koch’s book, Consciousness: Confessions of a Romantic Reductionist. I downloaded it onto my Kindle as soon as I could and blasted through it over Christmas Eve and Christmas.

Whether or not something is “conscious” as people are conscious, the notion of measuring the factors of consciousness as a measure of an enterprise’s ability to proactively evolve and get itself out of messes could be extremely compelling for a business. This notion is an extension of the foundation of my approach to BI, that businesses are like organisms. This metaphor has always been helpful in guiding me through the progression of BI over the past 15 years or so, as well as my earlier days with expert systems 25 years or so ago.

I know that sounds like useless BS, that I had too much time and maybe “Christmas cheer” over Christmas vacation. We don’t really even know what consciousness is in people, much less in an entity such as a business. Further it’s extremely difficult for most people to accept that anything but humans could possibly be conscious. Please bear with me a bit as thinking through the question of whether a business is conscious is related to the question of whether improving the aspects of sentience can serve a business as well as it has served humans.

So throughout this blog, I’ll stop apologizing every other sentence and assume that consciousness is something by far most highly developed in humans and that it is in fact the primary factor for our success as a species. Take this all with a grain of salt. This is a reset of how to approach BI, stepping out of the weeds.

As the developer of Map Rock, this means almost everything. Map Rock is about the integration of rules across an enterprise (a complex system) as is integration of the brain across regions what our symbolic-thinking consciousness is all about.

Businesses as Organisms

Like organic creatures, whether individuals or groups of creatures (hive, pride, species, or some higher level individuals such as dogs and great white sharks), businesses share major characteristics such as goals, strategies, complexity, intelligence, the ability to evolve, and maybe even “feel”. Those characteristics range widely in level; some businesses are more complex, some less, some more intelligent, some less.

The first four characteristics, goals, strategies, evolution, and complexity, are rather easy to understand and buy into. Businesses exist for a reason, therefore they have goals. This usually means to turn a profit for the owners or to improve the condition of a cause for non-profits. Reaching these goals means achieving some desired state in a complex system. It is accomplished through a strategy of taking in raw materials and converting them into something leading it towards its goals. Strategies involve a hodgepodge of many moving parts (usually complex, at least complicated) such as employees, software applications, machines, partners, and of course customers orchestrated in workflows.

Eventually strategies cease to work due to relentless changes in the world, and the business must adapt. Or sometimes the business is attacked at a vulnerable point, whereby it defends itself, and makes adjustments to strengthen that point. It evolves. In the case of the humans (highly conscious symbolic thinkers) we can proactively adapt in the face of predicted change.

The Magic of the Whole is Greater than the Sum of Its Parts

The intelligence of a business is mostly tied up in the heads of decision makers, mostly “bosses”. However, with the aid of Business Intelligence it’s increasingly feasible to de-centralize that intelligence, delegating more and more decision making to non-manager information workers. Additionally, certain aspects of BI-class applications such as predictive analytics models create non-human intelligence as well, encoded as objects such as regression formulas and sets of IF-THEN rules.

The sum of these individual intelligent components (the employees of the business and “smart” applications) does not equate to the intelligence at the business level. Even though this is an apples and oranges comparison of intelligence (like comparing the intelligence of your immune system to the intelligence from your brain), unfortunately the sum of the intelligence of the parts, inadequately integrated, is still greater than the whole (real intelligence doesn’t currently emerge in businesses today). In other words, businesses currently lacking adequate integration of intelligent parts are generally stupider than the collective intelligence of the parts. Genuine magic happens when the whole is greater than the sum of its parts.

To expand on the italicized term in the previous paragraph, inadequately integrated, the integration must be sensible as well. For example, a “bag o’ parts” is not the same as a properly assembled and tuned automobile that actually works. For a business, integration is the level of cross-entity reach within the collection and assemblage of the web of validated cause and effect throughout the business. “Cross entity” means not just cause and effect amongst team members or inter-department, but team to team across departments, role to role across departments, etc. “Validated cause and effect” refers to the data-driven proof of the validity of theories underlying the strategies that dictate our actions. I wrote more about this in my blog, The Effect Correlation Score for KPIs.

Unfortunately, I’ve experienced in the BI world an addiction to quantified values, values must be deterministic to be valid. It isn’t difficult to see why this is the case as BI is based on computers and quantitative is what computers excel at. Quantitative values are appealing because they are easier to understand than ambiguous qualitative “stories”. The phrase “gut decisions” (intuition) are the sworn enemy, inferior to data-driven decisions, because there is no elucidated process consisting of verifiable, deterministic values.

The thing is, neither quantitative nor qualitative (data-driven versus intuition) analysis is superior to the other. They are the crests and troughs of the iterative analytical process. They form a yin and yang cycle: Qualitative values address the fact that the world is complex and objects are constantly in flux. However, it’s difficult to make decisions based on such ambiguity. Therefore, we discretize objects into unambiguous entities so we’re able to execute our symbolic what-if experiments.

The problem with these discretized objects is that they are now a sterilized, hard definition of something, stripped of any “flair”, stripped of any fuzziness (reminds me of particle-wave duality). These hard values are now carved into square pegs for square holes, poorly able to deal with the ambiguity of real life. Everything is its full set of relationships, not a subset of what at some point seems like the most salient. What we’ve done is to define objects as particular states of a finite set of relationships (snapshots). Long story short, eventually we begin trying to shove rhomboid pegs into square holes.

It’s also interesting to consider that the results of Predictive Analytics models come with a probability attached to them. This is because a PA model only sees things from a small number of angles. In the real world, anything we are attempting to predict can happen from an infinite number of angles. In this context, we can think of an “angle” as a point of view, or a fixed set of inputs. This is the result of quantifying the inputs, stripping what appear to be non-salient relationships, reducing it to an unqualified, non-ambiguous object. We can consider those supposedly non-salient relationships as “context”, that unique set of circumstances around the phenomenon (the object we’re quantifying) that makes an instance in time unique even if the same object otherwise appears identical some time later.

Quantification really is a double-edged sword. On one hand, it is that very thing that enables symbolic thinking, the ability for us to play what-if games in our heard before committing to a physically irreversible action. On the other hand, that quantification strips out the context leading us to only probable answers. The real world in which businesses live isn’t a science lab where we can completely isolate an experiment from variance, thereby declaring that the definition of insanity is to keep doing the same thing expecting different results. In the real world, that is the definition of naïve.

Quantitative is the norm in the BI world. For example, I’ve become too used to the OLAP context of the word “aggregate”. These are sums, averages, maximum or minimum values, or counts over a large number of facts. These values still compare apples to apples values (ex, sum of all sales in CA to sum of sales in NY). Another is the knee-jerk rejection of the use of neural networks in predictive analytics because their results cannot be readily understood.

So it’s not always the case that the whole is greater than the sum of its parts. It’s more that the whole is at least very different from the sum of its parts. It is more like the chemistry context of a compound – water has very different properties than hydrogen and oxygen. Think of the great rock bands such as the Eagles, Fleetwood Mac, or the Beatles where none of them individually are the greatest.

Addressing the development and management of goals, strategies, complexity, evolution, and intelligence are where we BI/Analytics folks make our contributions. These are aspects of business that we can wrap our human brains and arms around towards improvement. I still wouldn’t go so far as claiming a business is alive like an organic creature, even though it acts like one; albeit, again, maybe a little on the zombie side. Setting aside that “alive” issue for now, do businesses possess some concept of “feeling”?  It’s that elusive thing that’s hardly even considered in the context of a business. And if it is, it is readily dismissed as just an irrelevant, indulgent exercise.

Our feelings and the feelings of others (people we deal with) do matter (well, for the most part) in the course of our daily grind. But the feelings of animals, if we were to consider they have some level of feelings, with the exception of our pets, don’t matter as much and often not at all. We’ve all anthropomorphized inanimate things (such as stuffed animals, cool cars, laptops) attributing them with feelings, but it’s really our feelings towards those things that matter, not the imagined feelings of those inanimate things.

At least, businesses can be in anthropomorphized states of “feeling”. For example, businesses can be in a state of pain when its losing money, in an ill state when the equipment and/or employees are wearing down, in a state of happiness when goals are met, and in a state of confidence when it is secure and will reach goals in the near future. But being in a state isn’t the same thing as feeling it. A KPI on a scorecard can express a state of pain with a red icon, but is it felt?

Certainly the humans working in the business are conscious, but that doesn’t mean there is a consciousness at the business level. Even if there is one, that doesn’t mean we (the human parts) can converse with it. Similarly, whether my organs such as my liver and heart have a consciousness of their own, I seriously doubt they are aware of or could even comprehend my life as a complete human being.

I call the properties emerging from the interaction of things (the whole is greater than the sum of its parts) “magic” because it really is a gift that in some sense defies all the conservation of energy, zero-sum thinking permeating our lives. It’s like the “magic” of compounded interest. We’re all aware of this concept and hear someone mention it almost daily, but probably don’t appreciate it and strive for it as much as we should.

We also see it very often at the level of the teams we work on, whether it’s a rock band, a basketball team, or a BI development team. It’s fairly easy at the team level because we as individuals are directly a part of it. It becomes trickier at higher scales (teams of teams) because then we’re not directly observed.  The point of this blog is to explore whether we can measure and the value of measuring this at the scales way bigger than teams; teams of teams of teams …

The Value of Φ to a Business

Does it matter if businesses have consciousness? Admittedly, the question of consciousness, whether people or whatever, was to me just an academic exercise. I thought of it as a very tough problem that wasn’t standing in the way of making technical progress towards smarter software.

I don’t think any business today is conscious, but that businesses could be driven towards consciousness and that progress in that direction would be beneficial. It is just an academic exercise from the standpoint of how measuring a business’ level of consciousness can help the business resolve current problems. However, to continue the analogy of a business of being an organism, I first ask if consciousness has provided benefits to humans. For the sake of argument let’s say that we humans are the most successful creatures and we are the most conscious. Is that directly linked?

Consciousness, self-awareness, enables us to imagine and perform symbolic what-if experiments in our heads before we commit to a physically irreversible action. That’s the game-changer because the world is complex and Black Swans, big and small, will constantly bombard businesses – like strokes and mini-strokes. Without the symbolic thinking afforded to us by consciousness and self-awareness, we would be at the mercy of the mathematical outcomes of the algorithms in our heads.

Would it be sensible to say that if a business were conscious, it would have the superior capability of generating strategies in a complex, only semi-predictable world as humans can? To me, that is the skill of skills. Consciousness is the secret sauce that makes humans resilient to the ambiguities the complex world in which we live during the course of our relatively short-term existence. That is the ability to fish; even if you happen to lose a fish, you’re not dead, you can still fish. When (not “if”) a business’ strategy starts to fail, it need not worry too much if it has superior capability to develop and implement new strategy.

If a business were indeed conscious, even if we individual humans were not able to interact with that consciousness (the voice of the CEO is not the same thing), would it still be valuable to attempt to nurture that consciousness (i.e. maintain a good value for the Φ KPI)? If that answer is yes, how could we do that?

The Term, “Actionable”

Before moving on to what it takes to calculate Φ for a business, I need to clarify that this is not what is thought of as an “actionable” KPI. I still read very many articles, mostly Big Data articles, insisting that the only good information is actionable. That is, we can readily do something with that information, otherwise, it’s merely interesting. I don’t disagree. However, I just think we need to consider that life throws curve balls and that it’s the human ability to deal with the unknown and ambiguous that sets us apart from other species, where we can imagine something not there right in front of us. It’s survival of the most adaptable and lucky, not survival of the fittest (see Note 1). “Fittest” is a definition that is constantly in flux. Regarding the luck component, as they say, “we make our own luck” through our ingenuity and agility.

When I read articles describing “actionable information”, the context usually implies that it means we can apply it to a clear and present problem. What can you do for me now regarding the problems that kept me up all last night? This mentality is dangerous because it strips imagination, the most prized human intellectual asset, out of the workflow. Remember, it’s often the un-actionable but still interesting pieces of information that allow us to step outside of the box and say, “but what if …”. We then avoid the side-swipe from that nasty startup.

Being a stickler for actionable information is seductive because it’s easier to deal with than the worrying about problem that could happen (see Note 2). It’s scary to get up in the morning needing to invent a new, un-Google-able (no one has yet figured it out) way to do something. Focusing completely on actionable information is smart as long as we’re in evolutionary, not revolutionary mode (see Note 3). The length of these periods is unpredictable. Periods of evolution could be decades or it could be days. But at any given moment,  surprises big and small, good and bad, deadly or inconvenient, show up. The complexity of the world guarantees that.

Calculating Φ for a Business

Consciousness, at least in the context of this blog, is a complex system’s ability to have a big picture feeling of its state. What we’re measuring is the level of quality of that “feeling” from that complex system.

Calculating Φ, in all honesty and as an extreme understatement, isn’t easy. Our own individual consciousness is built up over a long period time using input and processing equipment that in most ways greatly exceeds computers and sensors submersed in an environment (society) vastly richer than sterile databases (even Big Data databases).

Unfortunately, there are no direct formulas already developed for calculating Φ for any consciousness; people, animals, or a business enterprise. There is though high-level guidance, even though it is tedious at this time. In an ideal world, the idea is to inventory the actors, the relationships between the actors, and every possible state of each actor in the enterprise resulting in a very messy graph of cause and effect. To make it worse, actors don’t stop at the level of individual people or software applications. There are lower-level actors with those actors.

I know very well how difficult it is to build such a graph even if we were just talking about a family tree because I’ve built a few. Mapping out such relationships in the SQL Server engine was extremely daunting. Documenting a heterogeneous mix of relationships among the large number of agents in even a small enterprise is unfeasible if not impossible. Just for the sake argument, let’s say we did manage to do this. The Φ value would roughly speaking measure the level of integration across all aspects of the graph. To put it in terms of our own consciousness, we know that there is a difference between just knowing a list of historical facts and integrating those facts into a story serving as a metaphoric scaffolding for a problem we currently face.

A feasible but still very non-trivial process would involve specialized software and is dependent upon the maturity of the business’ IT, especially BI, Business Process Management, Collaboration (such as SharePoint) and Performance Management. Such systems have captured much of the essence of an enterprise and can even be thought of as the IT side of a business’ neo cortex. However, these systems are not yet purposefully integrated as a mainstream practice, meaning the knowledge exists in silos. In fact, integrating knowledge is the purpose of Map Rock. But before diving a little more, I’ll just mention for now that the third level is really a set (“set” implies un-integrated) of baby steps that I’ll describe soon.

For example, creating a graph of communication (email, IM, meetings on the calendar) between employees shows connections between people. Aggregating it on attributes of employees, we can determine the levels of inter-department communication. Also identify groupings of people on the communications; these groups are “virtual” teams. Based on titles or roles, we can determine cross-functional communication. This acts as a reasonable facsimile to needing to catalog everything everyone knows. Mining a collaboration software such as SharePoint, we could find relationships such as people working on the same document.

For that graph of communication, what we’re looking for are relationships. We may not even know the nature of the communication, but if we know the Inventory department is talking to the Sales department, chances are good that each department is making decisions with consideration to the other department’s needs. We would know a senior engineer from IT regularly talks to the CIO.

Map Rock

The idea is that we’re building a rich, extensive, highly-connected web of cause and effect. This “database” of cause and effect is the experiences of a business. Collecting these experiences is only one side of the coin. The equally important other side of the coin is the maintenance of the web; the expiration, updating, deletion, or consolidation of the rules and statistics.

I designed and developed Map Rock from 2010 through 2013. I consider Map Rock a child of the “Big Data” buzzword age, but I was shooting downstream along the analytics workflow from Hadoop and a couple years into the future. At the time, as a long-time BI practitioner, I well understood that most companies haven’t even come close to exhausting what they could do with the data already at their disposal.

Would simply more data (Big Data) drastically improve the quality of insights from analytics? Certainly the incorporation of very granular data from all sorts of sources and semi-structured or even unstructured data trapped in documents and pictures would greatly enhance analytical capabilities. But more data doesn’t mean greater understanding. It’s relationships, strings or webs of cause and effect, that are the molecules of intelligence.

Departments within an enterprise are usually headed by a director or VP. Because of the hierarchical nature of the organization, the tendency (not necessarily the intent) is for each department to operate as if they are an independent company serving their “customers” (other departments). Orders are given from above and that drives an information worker’s actions. Even an awareness of a pain in another department is placed down the priority list, as the worker has enough problems of their own to deal with.

The result in the end is that this dis-integration leads companies behave like puppets, as I mentioned at the beginning, where the actions are controlled loosely from above. They work well enough to hobble down the street, not quite like world-class athletes. As entities, businesses aren’t very smart and thus have a mortality along the line of crabs where for every fully grown crab tens of thousands of  larvae perished.

Intelligence exists at a relationship level, not at the data level. Data is simply the state of an agent living in system or relationships. Whenever we reason our way through a problem or create a solution, it is all based on a web of cause and effect slowly charted into our heads over the course of our lives (our experiences).

Catalogs, lists, and tables of data in our OLTP systems consisting of unrelated rows are not relationships. They are simply repositories placed together for convenience, no more than a list of alphabetically ordered names in a phone book says much about the community. The row in a typical customer database table for Eugene Asahara may store relationships about attributes of Eugene Asahara, but there are no relationships between another row for someone else. However, the fact that I am a customer may have relevance to the person who sold me something.

Relationships are rules. These rules are experiences in your head associating all sorts of things, predictive analytics models which precipitated relationships out of a slurry of data. In a business these rules are distributed amongst the brains of every employee and model (such as predictive analytics or relational schemas), and each is more island than a supercontinent where species roam wherever they can (see Note 4).


Before we can catalog relationships, we need to catalog the things we are relating – agents in a system. Agents are the players, the tangible and intangible things in a complex system. They can be as elemental and tangible as a document to as complex and intangible as a supply chain or major event. It’s important to realize too that agents can be part of many different macro agents; for example, a person can be an individual, part of a team, part of a department. Conversely, agents may not be as elemental as you would think either.

The direction of the communication matters as well. Obviously line employees receive communication from a chain spanning from their manager all the way to the CEO. But one way communication doesn’t promote informed decisions.

Events are really objects in a sense as well. I think of an “event” as an object that is ephemeral and consists of a loose set of characteristics. The attributes of a car or building are tightly tied together in terms of time and space (my arm is definitely a part of me and barring unforeseen circumstances always a part of me).

Machine-Stored Relationships

As mentioned, the intelligence of a business doesn’t only lie in the heads of humans. At most enterprises, there are many BI/Analytics objects possessing a level of intelligence.

Predictive Analytics models are the prime example of units of machine intelligence – even though humans do have a hand in their creation to varying extents. PA models are machine or human “authored” clusters of IF-THEN rules and calculations taking inputs and splitting best guesses either conveyed to a human or in many cases executed automatically. They can range from “white box” components where we can clearly see what goes on, to “dark gray  boxes” (not “black box” where the internals are completely unknown) such as neural networks where it’s very difficult to figure out what actually goes on.

The many Predictive Analytics models scattered across enterprises form an un-integrated web of knowledge that is very reminiscent of the structure of our brains. Neurons are not simple gates but more complex mechanisms in themselves. Like neurons, PA models take many varied sets of inputs and spit out an some answer. That answer, the resulting mash from many inputs is in turn the input for a subsequent decision. Integration of these models, not just the storage of the metadata of all the models, is what the Prediction Performance Cube is about.

OLAP cubes and departmental data marts hold some level of intelligence even though they are primarily just data. They store what has been vetted (during the requirements gathering phase of the project) as analytically important at a departmental level (a subset of a full enterprise). For example, OLAP attributes provide a clue as to how agents are categorized and the hierarchies show taxonomies. In other words, OLAP cubes and Data Marts don’t store the entire enterprise universe, so we select data that is probably analytically useful. Browsing through the OLAP cubes shed light on what is important to the departments that built them.

In Business Problem Silos I describe how departments create department-level BI (cubes or data marts), which integrates data from two or more data silos to resolve departmental business problems, but we end up with a different kind of silo, a business problem silo

Naturally something with the word “rule” in it such as Business Rules Management must be rules. IBM’s ILOG is a great example of  rules management software. Such Business Management Rules are usually intimately tied into a Business Process Management implementation as the decision mechanisms determining the next steps in a process flow.

Human-Stored Rules

Of course, the vast majority of the intelligence of a business lies in the heads of the employees. Machine-stored rules cannot even begin to match the nuances, flexibility, richer relationships, richer store of information that is in a human brain. The problem is that even though the intelligence of a human is richer, it can’t capture everything in an enterprise. Not even the enterprise-wide scope of a CEOs can match the collective intelligence of the narrowly-scoped “line workers”. Instead, they see a higher-level, more abstract, wider, shallower picture than the narrower but deeper intelligence of a line worker.

Not every employee can know everything and no one should need to know everything. But we must be cognizant of the side-effects of our actions. My discussion on KPIs below addresses that.

Human and Machine Interaction

In most enterprises, Performance Management, along with every workers’ assigned KPIs is a very well-socialized part of our work lives. KPIs are assigned (related) to people or machines measuring our efforts. They are like nerves informing us of pleasure or pain and the direction. I’ve written a few other blogs on KPI relationships that have the common theme of integrating KPIs into a central nervous system:

  • KPI Cause and Effect Graph Part 1 and Part 2. These old blogs take the notion of KPIs as nerves to the next level, attempting to integrate them to at least some degree as a nervous system, by decomposing the calculations of the target, value, and status and mapping KPIs into a web of cause and effect at that more granular level.
  • Effect Correlation Score for KPIs. This blog proposes another component to the KPI (besides value, target, status, and trend) that measures the intended effect meeting a KPI target is supposed to have. For example, does increasing employee satisfaction still lead to an increase in customer satisfaction?

Realistically, the KPI is the best bet for even beginning to attempt a Φ calculation. As KPIs are well socialized at the workplace, prevalent throughout an enterprise, they already define what is important to just about every breakdown of workers from departments to teams to individuals.

Business Process Management workflows are another fairly well-socialized program of an enterprise which documents the sequence of events that happen in business processes. Workflows include prescriptions for procurement, proposals, filing taxes, performance reviews, manufacturing, distributing. These are usually inter-department processes which will relate cause and effect across the departments. However, the workflows that are formalized probably account for just a small fraction of what actually goes on. The vast majority exist in the heads of the people involved in the workflow. This could be worse than it sounds as many workflows don’t exist in one person’s head, but pieces of it are distributed among many people (meaning people are only aware of their part of the workflow).

Map Rock is designed to integrate, both in automatic and manual fashion, such sources of relationships into a single repository. For example, a component of Map Rock known as the Prediction Performance Cube, automatically collects PMML metadata, training data, test data, and metadata of the training and test data sources. Another component of Map Rock, called the Correlation Grid, is a UI for exploring correlations among dimensions in different cubes.

Map Rock converts these rules, from whatever sources, into my SCL language, which is used as the lowest common denominator. SCL is based on the old Prolog language of the AI world. Beyond being a .NET implementation of Prolog, it is modified for the distributed world of today as opposed to the more monolithic world of the 1980s.

Putting it All Together

A suggestion for a calculation of Φ is a graph of communication ties between decision makers down the traditional hierarchical chains of command as well as across the departmental silos. These ties are the relationships described in the sections above from email, meetings, KPIs, workflows, etc.

Those are the ties that do exist, but we also need to figure out what should exist. In a nutshell, the Φ calculation is a comparison what exists and what should exist. I need to mention that what should exist is different from what could exist. What could exist is a Cartesian product of every agent in the system, a number so great it’s unusable. Determining what should exist is extremely difficult akin to “we don’t know what we don’t know”.

There will be four categories of ties:

  1. Known ties that should exist. These are the ties that we’ve discovered from KPIs, workflows, email, etc that should exist because there is information that must be passed.
  2. Known ties that do not need to exist. These are ties for which there is no required formal communication. Examples could be email from amongst employees who are friends but never work with each other or someone subscribing to updates on something for which he is not involved. These are not necessarily useless ties. The interest shown by an employee subscribing to updates unrelated to him demonstrates a possible future resource.
  3. Ties that should exist, but don’t.
  4. Ties that could exist, but don’t. This should make up the vast majority of conceivable ties. #1, #2, and #3 account for such a minority that this is considered “sparse”.

Ties between should and could, whether they exist in our enterprise or not, is interesting since these are the ties that can lead to innovation. The ties that should exist are in existence because they are part of formal processes already in place. But when we need to change processes to tackle a competitor or simply make an incremental improvement, we’ll need to explore these other ties. So ideally, our graph would consist of some level of ties that aren’t currently required. A good place to start would be ties involving agents from #2, those agents with ties that aren’t mapped to a formal process.

What would the Φ calculation look like? Imagine a graph of the agents of our enterprise and the relationships. Imagine that each relationship is a color ranging along a spectrum of green to yellow to red, where the extreme of green represents excellent communication and the extreme of red represents no communication (and that there should be communication).

The calculation of Φ would be range from greenish (score of close to 1 for generally good communication where there should be communication) to yellowish (score of around 0 for OK communication) to reddish (score close to -1 for generally poor communication where there should be communication) of the entire graph. If your graph ends up greenish, but that doesn’t correspond to your experience that the enterprise is dysfunctional, it would mean that there is a deficiency at identifying the ties that should exist.

A good starting place for what ties should exist would be in the KPI Cause and Effect graph I mentioned above. Going further with the KPIs, Map Rock includes a feature whereby we can view a matrix of the KPIs where each cell displays the level of correlation between the two. This will not necessarily identify all ties that exist, but it may show some surprises.

Back to the graph. We could start this graph of ties with the chain of command, the “org chart” tree, from the CEO at the top to the line workers with no one reporting to them. It’s a good starting point because every company has a chain of command, which forms a natural route of communication.

The next step is to identify the decision makers. Although decision makers and non-decision makers require information to operate, the decision makers are the ones that initiate changes to the system. However, in reality everyone makes decision to some extent, so this is more of a fuzzy score than a binary yes and no.

As a first pass it’s safe enough to assume that anyone with direct reports makes decisions. But for non managers, there are many who primarily operate under temporary project cross-functional teams consisting of members from across departments. In these cases, they may not necessarily have formal report relationships, but do make decisions in the team. For example, a “BI Architect” may not be a manager, but may decide on the components to be utilized. It’s tougher in these cases to automatically infer these decision makers.

In addition to identifying decision makers, we need to measure the strength of communication between the decision makers and those affected by and changes. This is an important aspect since we know a tie should exist, but it is poor. This is in some ways worse than not realizing a tie should exist. If a decision maker issues direction to someone and thinks the direction was received, the decision maker may go on assuming her instructions are carried out.

In the end, we should end up with a graph illustrating strong connections between many parts and little in the way of weak ties or ties that should exist but don’t. The thought processes behind such a graph is the bread and butter of companies like Facebook and LinkedIn. But the networks of enterprises are the soul of corporations.

For studying networks, particularly the social network aspects, there is no better place to start than with the work of Marc Smith. Marc also has lead the development the Excel add-in known as NodeXL, a visualization for displaying graph structures.

Baby Steps Towards Φ Calculation

All that is what Map Rock is about. Duplicating what I’ve developed in Map Rock will be daunting at the time of this writing. However, there are baby steps that should be very informative in helping to improve the agility of an enterprise. For the most part, these baby steps consist of the pieces I described above such as studying social communication networks, developing a KPI cause and Effect graph, implementing the Effect Correlation Score, and creating a Predictive Analytics Metadata repository. Still, though these parts won’t be integrated with each other, without Map Rock, they are an un-integrated set of tools.


So there it is, my attempt to step far back, looking at what it is we’re really trying to do with Big Data. From my point of view, we really are trying to apply to our businesses the same technique that made humans the dominant species at this time. That is, we’ve built the mechanisms for dealing with ambiguity. Therefore, with the proper implementation of Big Data we can much better deal with the roadblocks along the way.

The NoSQL world, particularly Neo4J the NoSQL graph database offering, should finally break down some walls in better embracing the complexity of the world. Graphs, the true representation of the world, the structure of workflow, semantics, taxonomies, etc, will finally take its place as the center of our “data processing”. Perhaps with Neo4j, the visions of the Semantic Web can finally help us make real progress.

Lastly, I apologize that this blog isn’t a step by step cookbook – that would indeed require a book. This blog ended up way larger than I had intended. My intent is to toss out this idea at a high level description, and drill down further in future blogs.


1. I asked my favorite judo instructor, What is the most important skill, if you could pick one? I expected him to tell me that is a useless question completely missing the point, that it’s a balance of many things; I’ve failed to think in a systemic way. But surprisingly, without a blink he said flexibility. Not technique, not strength, not size, not endurance, not balance, not tenacity. Certainly, they are all factors. He said, “With flexibility you can do all sorts of things your opponent had not thought of.”

2. I know it’s out of fashion to worry about problems that haven’t yet popped up; that we should focus on the now. However, there is a difference between being in the now because we’re:

    1. Incapable of predicting what will come next. So like honey bees we put faith in the math behind the rules. This is similar to how casinos put faith in the math behind black jack. In the end they will win.
    2. Deluded into thinking that if it hasn’t yet happened (i.e. the experience isn’t in my brain) it couldn’t possibly happen. A variation is, “No one is asking for that.”
    3. So busy with what we already have on the plate that anything not currently in your face goes down the task list.

and that we’re confident that we can get ourselves out of a jam if the time comes because we’ve packed our brains with an extensive library of experiences from which we can draw analogies to future problems, we are fully aware of our surroundings and at the same time fully engaged with the task at this instant in time, and we fully accept the outcome not dwelling in the past. In other words we’re satisfied that we’ve prepared as best as be could for whatever comes about.

3. I don’t like the phrase, “It’s evolution, not revolution”, uttered by the VP of the department I worked in a while back. The sentiment seemed to be the antithesis of high tech. Progress involves cycles of long periods of evolution and short bursts of revolution (even though I’m pretty sure that wasn’t the message of that VP). It’s a balancing act where progress requires some level of focus towards a goal and another where the evolution actually leads to some kind of stasis (leading to proactive revolution) or crisis (leading to reactive revolution). During evolution, distractions from elsewhere impede progress. However, when stasis or crisis arrives and there is a need to move on, integration of knowledge fertilizes the system.

4. The extremes of Island or supercontinent isn’t good. Rather there is something in between. There needs to be some level of isolation to allow diversity to first build, then for small, diverse populations to mature. Either a supercontinent separates into little islands allowing diversity to take shape, then converging to let the best one win, or there is some natural barrier whereby weak links can occasionally travel to other populations infusing some new ideas. Too much integration is really just chaos, too little will eventually lead to stasis.

Posted in Cutting-Edge Business Intelligence, Data Mining and Predictive Analytics, Map Rock, SCL (Soft Coded Logic) | 1 Comment

Micron’s Automata – A Rare Big Thing Out of Boise

Not many huge things come out of Boise and that is the very reason I live here. But something that should disrupt the semiconductor industry (the world in fact) is Micron’s Automata. It’s been occupying much of my thoughts since it was announced a few weeks ago. The Idaho Statesman’s front-page headline today is Big Data, Big Move for Micron with New Processor.

In a nutshell this is hybrid memory that is capable of having a large number of custom operations programmed directly on it that can be run in massively parallel fashion. From a software developer’s point of view, the parallelism is the key value. But from an IT guy’s point of view, not needing to depend on a trip to the CPU and back for every operation drastically cuts out a ton of processing overhead.

When I’ve told colleagues about this, many see this as just faster throughput which would be beneficial for a class of software applications – niche. This is like the dawn of the jet airplane; it’s not just a faster way to get from New York to Los Angeles. Automata isn’t ultimately a niche product only useful for research organizations or the NSA. We all live in a complex world where outcomes are dependent on things happening in massively parallel fashion.

Really, the way we use computers today is what is niche. They are useful for a subset of relatively simple (still within the realm of “complicated” as opposed to “complex”) problems that are deterministic, well-defined, limited in volume, and pretty linear. We’ve taken these well-defined applications, propagated and socialized them to severe extents, and forgot that these are really reduced, simplified versions of the real world. Then we consider the applications attempting to embrace real-world complexity as niche.

It will be incredibly interesting to see what players jump onto Automata first. Probably Hadoop with a retrofitted Automata version. But it could be Microsoft’s CEP (complex event processing) offering, StreamInsight, something from Oracle, maybe IBM’s ILog, or a yet unknown startup.

Regarding that yet unknown startup, it probably won’t be SCL. Ever since I wrote the SCL language, interpreter, base test-harness UI (Map Rock is in part a rich SCL UI), I’ve taken several approaches to writing a bad-ass “SCL Server”. They actually worked fairly well, but I knew I needed superior parallelism to what was offered by the current methods, lots of servers with lots of CPUs, in order to achieve what I really envisioned. Then Hadoop and all the associated Big Data things came along. Anyway, weighing everything, I decided to aim way downstream (Map Rock).

However, keep in mind that the kind of massively parallel processing that Hadoop can do is different from the sort I’m thinking of. I’m thinking of Hadoop as a homogeneous MPP where a long list of like tasks is divided amongst like processors. The sort I’m thinking of is a much more compelling heterogeneous sort of MPP where thousands of points of view work on streams of data in order to promote robust recognition.

The key question is, Can Automata-based Big Data servers resolve problems that currently are just too expensive (in terms of cost and time) to resolve today? Apparently it can, according to some of the sparse literature on Automata (see slide 12, titled Breakthrough Performance, of this file). If so, that’s disruptive.

Micron readily admits that the concepts of Automata-like technology has been around for a while. Micron showed a lot of courage sticking to the development of something so different instead of the sort of incremental improvements we’ve been seeing lately. It would be unfair to call technologies like multicore processors and NUMA merely incremental, but Automata is truly fresh and timely.

On a light note, related to not much coming out of Boise, on my flight back to Boise from my client site the other day, I overheard something very funny in a conversation behind me. This woman was talking to a guy about how people react when we say we’re from Idaho. Of course, it’s always something about potatoes. She said she tells people she’s the Bubba (from Forest Gump) of potatoes; French fried, au gratin, baked, mashed, … Then others started throwing in their potato styles too; twice-baked, potato skins, potato pancakes …

Posted in Uncategorized | Tagged | 2 Comments

The Effect Correlation Score for KPIs


Everything we do is intended to move us towards achieving some set of goals ranging from just satisfying our immediate hunger to achieving a revolution. These actions are executed within a strategy (or hierarchies of strategies) to varying extents of sensibility and organization. Presumably the effectiveness of our efforts towards those goals and the rhyme and reason behind why be do them are constantly assessed. After all, the world is ever-changing. It is a complex world where everything we do doesn’t take place in a vacuum and it is rife with the constraints of finite resources (costs – time, money, materials, other opportunities) that force us to find creative alternatives. Further, unlike our lives on the living room couch, activities in our work lives usually involve many people and other moving parts, and failure comes with consequences.

So we operate the execution of our enterprise strategies (activities in our work life) within a formal Performance Management framework in order to maximize our chances for success. The intent of Performance Management is to align the efforts of everyone and everything within an enterprise towards its goals. Properly implemented, we ensure we understand our purpose, our goals, develop coherent strategies, identify, assess and mitigate risks, understand how to measure the effectiveness of our efforts, and develop sets of contingencies.

The problem is that development of the strategy and properly thinking through it seems more like an enterprise chore than the primary key to success that it is. Strategy development tends to place too much faith in trends of the recent past and that the actions of our strategies will not trigger cascading, unanticipated effects. Cutting intellectual/analytical corners by failing to validate the strength of beliefs in the business (relationships between tactics and outcomes) or by failing to place more realistic effort into risk mitigation (too optimistic about things that haven’t yet happened) can easily derail the Performance Management effort.

The strategies also need to be novel in order to compete. I’d love a world where I knew exactly what everyone else is going to do because it’s a “best practice” (and therefore the responsible course of action). In such a world, that means our statistics-based predictive analytics models would work surprisingly well. Well, they sometimes do, then quickly stop working after everyone learns about the phenomenon and alters the statistics from which the model is based.

Developing strategy relies on our arsenal of cause and effect built up in our heads over the years. We draw on those experiences to imagine what will happen if we put a series of actions into play towards a goal. It’s when we execute on that strategy that we find the world will not simply go through katas (choreographed fighting) with us. I think the way Performance Management is implemented today doesn’t possess the necessary focus on agility required to fight in a randori world. In this blog, I wish to introduce an aspect to address this in a way that avoids analysis paralysis but also serves as a better warning about the sanity of our strategies.

This blog is intended to simply introduce this concept of an “Effect Correlation Score”, something I try to include as part of performance management projects I engage in. I do think that comprehensive coverage of this concept could take up an entire book as the purpose of the ECS is to profound: ensure our efforts are not just going as planned, but are still valid. The primary take-away is that we must be cognizant of whether our endeavors towards “good” KPI statuses are still valid and sensible in a world that is constantly changing and for which “business” is usually a competition.

Current State of KPIs

The basic elements of a Performance Management implementation are the Key Performance Indicators (KPI). KPIs are just measures of values. A good everyday example of a very basic, objective KPI is the speed of your car. The value, the easily obtained value of 55 mph, in itself only tells us what we’re doing. But in our minds we know that the value also helps us estimate the arrival time at our destination, it tells us whether we are breaking any rules (the posted speed limit), it tells us how capable we are of passing a car within a given distance. Such concepts are conveyed to us through other “perspectives” of the KPI. Currently, those other perspectives include these three things:

  1. Target – This is the value we’re shooting for; the figure that we should be reaching for. In this example, if the posted speed limit is 55 mph, we should shoot for that speed.
  2. Status – This is a formula that provides a score quantifying how well we’re doing or not doing, as opposed to the Value itself, which would be what we’re doing. In this case, if the posted speed limit (the Target) is 55 mph and we’re traveling at that speed, the status is good; 45 or 65, not exactly bad, but you’re open to getting horns tooted at your or a ticket; 25 or 95, and you’re looking at a very dangerous situation, maybe involving getting rear-ended and more than just a ticket. The KPI status is often represented by a “traffic light icon”, where the green light is good, the yellow light is warning, and the red light is bad.
  3. Trend – This tells us how we’ve been doing lately. It may be that the status is currently good at 55 mph, but maybe we’ve been following a wide-load for the past hour, so that in reality, we haven’t been doing well lately. Interestingly, trends normally look at the recent past, but we can also look at the predicted near-future trend. Maybe we see the flashing lights of the wide load truck up the freeway.

I’m proposing another “perspective” of KPIs I call the Effect Correlation Score (ECS). This is a measurement of the correlation between the measure of our efforts and the effect we hope to achieve. For example, happy employees correlate to superior product quality. I will explain how strategies can lose their integrity “in flight” and hard-headedly following through on pursuing good statuses of the KPIs for which conditions have changed can be detrimental if not devastating. However, unlike target, status, and trend, the ECS is really an attribute of the relationship between two KPIs.

KPIs and Strategies in the Complex and Competitive World

Making good decisions involves a balance of learning from the past and being cognizant (even paranoid) about our assumptions. Applying what we’ve experienced in the past to similar situations today is only sensible, and for the most part, it works. However, the world is complex and things change. Additionally, reacting on what statistics tells us is the best answer is precisely what our competitors (or enemies) want us to do (see my blog, Undermined Predictive Analytics).

Strategies that we build to achieve goals are made up of webs of cause and effect. For example, at an extremely high level, happy employees leads to higher quality products, which leads to happier customers, which leads to bigger revenue. Additionally, lower costs along with the bigger revenue leads to bigger profits. Each of these links (edges) such as happy employees leads to higher quality is just a theory. It’s not always true. Maybe sometimes happy employees also leads to complacent employees. Maybe there is only so much that happiness can do to continue raising quality; even if happiness can be infinite, a product can only become so good.

There are many reasons why strategies stop working during the course of execution, beyond simply poor execution:

  • Our competitors adjust to counter our strategies. Unfortunately in business, one company’s gain is more often than not another company’s loss. If a company notices the weak points of a competitor and builds a strategy to capitalize on those weak points, the competitor will probably adjust. Therefore, that weak point, a major assumption of that strategy, fails to correlate with success.
  • Diminishing returns. Often, a tactic works because it is filling a void. For example, Vitamin C works wonders when our bodies are short of it, but the benefits of increasing Vitamin C diminishes to practically nothing very quickly.
  • Desensitization – A variant of diminishing returns. A strategy could stop working because the driving force of incentive can sometimes disappear. Would all workers initially driven to over-achieving simply from being happy to have a job continue to over-achieve?
  • Non-linear relationships. This can often be the same as diminishing returns, but this is more complicated. For example, what if Linus Pauling were correct and that mega doses of Vitamin C kick in other mechanisms supporting health in other ways than avoiding scurvy? We would see a rising benefit, a tapering, then a rise again.
  • People learn how to game the KPIs. An example is a call center agent goaled on successful resolutions who hangs up as soon as the call seems to be very difficult.
  • Managers mistakenly “blow out” the KPI, thinking if X is good, X*2 is great. In this case, managers run up the score to blow away the KPI. Hopefully, that impressive performance would show they’ve outgrown their current position. What often happens is this stellar performance, beyond any wildest dream, often has a side effect.

I realize it’s tough to even suggest that the components of a strategy should be changed in flight (between performance review periods) when conditions change. I’m sure we’ve all made a few snide remarks about a re-org of some sort that sometimes seems like just action for the sake of doing something (I think sometimes it is). There is an inertia that builds up, and every time we adjust course there are the costs of context-switching and taking at least one step back to take two steps forward. But if a theory (a correlation) ceases to be true, we can continue to plug away anyway, attempt to destroy the cause of the change, or blend in with the change.

Continuing to plug away at a strategy in the face of empirical evidence that a correlation no long is valid is a cousin of the “definition of insanity” (Einstein’s doing the same thing over and over and expecting different results). Sometimes we can catch an agent of adverse change (at least adverse for us) early on and nip it in the bud; or catch it late and destroy it at great expense. But very often, victoriously vanquishing the enemy only delays the inevitable. What makes most sense is for us to embrace the change, trying as best as we can to guide the change to our favor.

I want to be clear that by changing the strategy, I don’t mean making wholesale changes. I mean at least starting incrementally by identifying the KPIs for which a “good” status indicator no longer correlate to desired outcomes. This is as opposed to just finding KPIs exhibiting “bad” status and manipulating your activity such that the status becomes good. That good status means nothing if that goodness doesn’t correlate to success anymore. Hopefully, like a good software design, components of the strategy are loosely coupled so that we can replace isolated poorly performing business processes with superior business processes with minimal side-effects.

Note too that it’s not just a weakening of a correlation that could hamper the success of a strategy. Even a strengthening of a correlation could have adverse system-wide effects. For example, if gasoline were suddenly reformulated to convert chemical energy into heat more efficiently, that’s a good thing, but many components of a car would need to be redesigned to handle that change.

The Effect Correlation Score

This Effect Correlation Score of a KPI measures the relationship between a cause and its intended effect. It would most simply be implemented as the linear relationship Pearson Product-Moment Correlation Coefficient (PPMCC) described in my prior blog, Find and Measure Relationships in Your OLAP Cube. The PPMCC is the same calculation used by Excel and the CORRELATION MDX key word. My blog describes what is required for such a calculation, although it describes the implementation in MDX, not SQL. (I just Googled sql pearson correlation and saw many good sites demonstrating the PPMCC with SQL.)

Of course, the PPMCC is just a simple implementation of an Effect Correlation Score. Another of my blogs, Nonlinear Regression MDX, describes a score for more complicated but more accurate non-linear relationships. For very complex relationships, we could implement involve predictive analytics models (regressions are just very simple Predictive Analytics models), particularly the probability of a decision tree query.

The ECS should result in a value ranging from -1 through +1. That is the range of the PPMCC and we could use the 0 through 1 probability of a predictive analytics model. This range of -1 to +1 is the same conventional figures of a status and trend. For the ECS, -1 should mean there is an inverse correlation (one goes up, the other goes down). +1 should mean there is a direct correlation (they go up and down together). Anything in between are grades of correlation, particularly the value of 0 which means no correlation at all.

The ECS in the Performance Management Planning Process

When we’re sitting with our team in a conference room in front of a whiteboard for a Performance Management strategy planning session, we develop our strategies of webs of arrows pointing from one box (a cause) to another box (an effect) where each of those arrows implies an existing correlation. Again, happier employees leads to higher product quality. Or better upstream testing leads to lower costs.

Normally during the PM planning process, at some point before setting the strategy in stone, we need to determine how we can measure the values of each cause and each effect (KPI). If we can’t measure our actions, we’re running blindly. We also need to define what good and bad performance means (the status of the KPI) and the targets (which are really the goals of sub-objectives).

(As a side-note, some effects are causes as well – ex, higher product quality leads to better customer satisfaction which leads to higher return customers which … etc, etc.)

For the ECS, what we should be sure to do is work on pairs of related KPIs as opposed to determining how to obtain the values of all KPIs first, then obtain the status of all KPIs, etc. For each pair of related KPIs, we should validate that there is indeed a correlation between them (the cause and effect). Because if there isn’t a correlation, our logical reasoning down the line could be faulty.

The correlation could be tested between the status or direct value of the KPIs. I tend to think it makes more sense to test the correlation between the statuses of the two KPIs since that is semantically richer. However, the status could be calculated such that the correlation (or lack of correlation) could be misleading. So I usually run a correlation on both to see which makes more sense.

A problem could be that the values for KPI statuses over time are not recoverable. For example, there may not be a record of target values (usually a component of a KPI status calculation) for sales for all time periods (months, quarters), just the latest. But KPI values such as sales over time should always be recoverable to run a correlation.

Whichever is chosen, this is validation of at least part of the overall theory. For the future (during the course of the execution of the strategy), we would want to periodically check if the correlation continues to exist, which is the main idea of this blog.

Where Does the ECS Go?

On a Scorecard, there is one row for each KPI. Each row shows the KPI name, the value, the target, and a cute icon for the status and trend. They fit nicely one a single row because there is a one to one relationship between a KPI and its value, target, status, and trend. In reality though, most causes effect more than one thing.

However, in a Performance Management system, there is probably one primary effect for each cause. If we want to preserve the nice matrix display of one KPI per row, we could select the primary effect and add to each row, the name of the primary effect and a cute icon for the ECS. For any actions we take, there is usually a primary reason. For example, I know right away that I need to earn money to pay the bills, although there are many other minor reasons.

I like to denote strong correlations (ECS value closer to 1) with green, no correlation (ECS value nearer to 0) with cyan, and an inverse correlation (closer to -1, one value goes up, the other goes down) with blue.

Performance Considerations

This PPMCC calculation should be not much more expensive that a typical trend calculation commonly implemented as a slope of recent values. That would require the SUMs of many facts over say the past few time periods (days, weeks, months, quarters, years, etc) and a few squares and square roots of this and that. The PPMCC is really just a step beyond calculating a slope. In a relational database, the heavy work will be the IO involved in loading the same data required for the trend (slope), but will just do a few more calculations. Predictive Analytics models should be even quicker since they are really distilled into a set of rules (from a large amount of data) as opposed to calculating off that raw data itself.

Which KPI relationships Needs an ECS?

Implementing these correlation scores is not as unselective as implementing the PPMCC calculation between KPIs. For example, not all KPIs would benefit from an ECS. For that matter, not all KPIs require a target, status, or trend. In fact, an ECS for some KPIs could be completely meaningless or worse yet, misleading. For example, it would not be entirely true to say that a great status for the revenue KPI results in a correspondingly improved status for the profit KPI. That’s because revenue is just one factor of profit.

An ECS doesn’t need to be constantly monitored either. At the least, we should periodically validate that relationships incorporated into our strategies are still valid and haven’t slipped into mythical status. We need to also ensure that the granularity and seasonality are taken into account. Correlations may not exist at a day to day level but may appear at the monthly level. There may be seasonality involved that do not exist at certain times of the years.

Correlations are Not Silver Bullets

If correlations such as those found in calculations like PPMCC values or in data mining models worked very well in reality, it would certainly be a different world. After all I wrote above, I do need to include a disclaimer about the folly careless analysis of data. There are so many ways to misinterpret correlations, so many reasons why strong correlations could be meaningless, how a higher level lack of a correlation could mask a lower-level one of great consequence.

Nonetheless, when someone at work tells me something such as an MBA does or does not equate to higher project management success, I know I can run a few queries to help prove or disprove such a statement. I know I need to think through any possible misinterpretation. At some point I need to either be satisfied that the numbers do back up the statement, go against it, or it’s still inconclusive. If I do come to the conclusion that there is a positive correlation, then the possession of an MBA goes into the plus column for my strategy to hire superior project managers. My point is to continue checking that correlation to ensure that it is true. Otherwise, as I mentioned above, our actions become a case of the definition of insanity.

Other KPI Ideas

There are many aspects of KPIs that are not sufficiently addressed beyond trends, status, and the ECS. For example, I mentioned earlier that trends normally show the recent activity allowing us to infer a future state of the KPI. But a predicted trend would help us better infer that future state. In fact, multiple future trends dependent on a range of possible inputs would be even better. Another example is that statuses could be different from different points of view. Could a sales KPI drastically exceeding expectations be the same for the sales manager and the inventory manager who must deal with the unexpected demand? Would it help for a sales manager to know that the sales KPI for the inventory manager is bad?

By the way, Map Rock does a lot of this stuff …

Posted in BI Development | Tagged , , , , | Leave a comment

Cluster Drift

A deficiency I notice in practically every implementation of clustering (segmentation) is the snapshot mentality. For example, a vendor of a product would segment their customers in an attempt to isolate the ones who would be most likely to buy their product. This captures a snapshot of the groups of similar customers right now, but it doesn’t capture how the groupings and salient points of similarity have changed over time. It’s change that triggers action.

The clustering technique of data mining mimics our brains’ constant categorization process. Whenever we encounter something in life, such as we’re about to be caught in the rain, our brains notice sets of characteristics (dark clouds, far from home) and tries as hard as it can to match it to something encountered in the past, even if that often means pounding a square peg into a round hole. Once we find the closest match to what we are currently seeing to a group of similar phenomenon from the past, we can reasonably think that what is associated to that past phenomenon applies to this current one. If those associations do hold true, that grouping gains strength, otherwise, we formulate a new grouping. That’s learning, the basis of our ability to apply metaphor.

Figure 1 illustrates one of the simplest forms of clustering, the Magic Quadrant. Here we see a bubble chart of countries in 2006 clustered on two dimensions – the barrels of oil consumed per 1000 people per year and the GDP/Capita. The size of the bubble actually conveys a third dimension of population.

USA and Mexico clusters.
Figure 1 – Magic Quadrants are a simple clustering technique.

The very simple bubble chart clusters countries into four distinct groups. On the lower-left corner are the countries with low GDP/Capita and low oil consumption. The huge circle in the lower-left is actually two countries, China and India (the huge populations). The upper right are the high GDP/Capita, high oil consuming countries. The relatively large circle is the USA. Generally, we attempt to cluster (categorize) for a reason. In this case, the primo quadrant is the lower-right, high GPD/Capita, low oil consumption.

Guess that country with the relatively large circle, the higher GDP/Capita, lower oil consumption?

I love Magic Quadrants on reports as a graphic. In fact, when implementing BI I’ve trained myself to think in terms of Magic Quadrants as my default view as opposed to a bar chart or line graph. But they are limited to two dimensions even though we could further the clustering to 3D taking into account the bubble size (population). We can easily see the lower-right is easily further clustered into those three larger circles and the other small ones.

However, the real world is more complex than that and a magic quadrant quickly becomes an inadequate tool for helping us to perform effective categorization. We need to consider many more factors when determining what something such as a potential customer or competitor is like. Arrays of magic quadrants help, but clustering will works through the manually process automatically – well, once you’re able to provide the necessary factors.

Clustering as it’s usually implemented these days discovers these categories from scratch, usually as part of the lowest hanging fruit application of target marketing. In those implementations of today, the clustering disregards the aspect of time, forgetting the categories of the past as though they are completely obsolete.  Or more importantly, that the evolution of clusters doesn’t have any meaning or value.

People change over time. They get married, have children, are promoted at work, suffer injuries and disease, become weary from work, change careers, become religious or unreligious, heath conscious, become older and wiser or cynical. As with any change, these changes are driven by each person’s sequence of life events. Sometimes there is a typical progression, sometimes there are unexpected setbacks or windfalls. Companies and even countries go through changes as well.

It may seem all that matters is what the customer is right now. The past is the past. What does it matter how we got here, we’re here? That’s like saying a photo of two baseballs crossing paths at roughly the same place in an instant of time is all that is important. But the past dictates the future. The planes have a trajectory that represent more value than the simple fact that they are in relatively close proximity. Along similar paths, customers are more likely to behave similarly.

Normally, this trajectory problem (acknowledgment that things change) is handled by “refreshing” the clusters periodically; for example, before a targeted marketing campaign. Refreshing the clusters using the current state of the customers means we know what they are right now. This may work well for an immediate marketing campaign, but what if I’m attempting to develop a product that will take months to get to market. Could I predict my audience for that time?

Sometimes change is not progressive. For example, we become completely different people in the presence of parents, co-workers, or a club where we have authority. In this case, change is dependent upon circumstance, not just a natural progression driven by time.

Whether changes are predictably progressive or not, people, countries, and companies change, thus how they react to our efforts to service or engage them change as well. So when I apply clustering during a predictive analytics engagement, I take the clustering down to a deeper level than just the entities we’re clustering. For example, how do these customers behave during recessions or boom times, or under fierce competition or where they are secure? How do people behave on vacation versus being at work?

For this blog, I simply created clusters for each country for each year. Instead of simply clustering countries using their current states, I consider each country during each year from 1971 through 2006 as separate cases. What I’m trying to identify are the levels of “development” among the countries. How do countries progress from tyranny or poverty to democracy and/or wealth? The cluster model incorporates four measures that seem like plausible measures of such terms as freedom, poverty, and wealth:

  • Enrollment Rate is the percentage of school-aged children enrolled in grade school. Education is certainly a measure of wealth, whether it causes it or is a result of it.
  • GDP per Capita has two forms. “GDP Capita” is the GDP divided by the population of the country for each year. I chose to use a figure per capita to eliminate the factor of the sheer size of a country. For example, the USA will would always be in a cluster of its own. Part of what I want to see what countries live at a level of the US citizens.
  • The other “GDP/Capita” figure is “GDP/Capita Rate”, the change in GDP/Capita from the average of the past four years to the current year. Moods are very different when things are improving or degrading, even if things are currently good or bad.

Change, Categorization, Correlation and Map Rock

The clusters and graphs shown in this blog were created from a hodge-podge of SQL, SSAS data mining models, DMX, and Excel charts. I call this technique Cluster Drift. Cluster Drift is actually one of the techniques inherent to Map Rock. Map Rock’s core theme is that:

  • Change is the primary trigger that gets our brains’ attention. Change is also used as a factor in the same sense as a key diagnostic question is “What has changed since … ?”
  • Categorization is recognition. We recognize things by categorizing them into things from the past. Categorization gives us the ability to create a fuzziness about things. We then don’t require direct hits in order to consider something recognized.
  • Correlation. Once we are alerted to something through change, recognize what is present through categorization, we then attempt to notice when these things or just a subset of these things were present together in the past. If these things did happen together in the past, it’s reasonable to consider that anything else associated back then as well applies here.

I chose to present this technique without Map Rock as I like to present techniques I’ve developed and implemented in its raw form. “Raw” means that my data prep involved importing data into SQL Server, performing a significant amount of transforms, generating data mining models until I found a set that yielded somewhat coherent results, and ran the models and pasted the results into Excel for analysis.

Data Mining Disclaimer

Before continuing, I need to digress and present this disclaimer, especially with this blog since I’m writing it during the time that all the NSA data mining crap is front-page news. So at this time, I want to be extra careful about anything said about data mining.

This blog is not about presenting any findings. It is about presenting a predictive analytics technique for which I’ve had significant success. Unfortunately, as predictive analytics provides a strategic advantage to my customers, I’m never at liberty to present the real data and results of my customers. Thus, the data I use here is downloaded from “free” sites and I really have no way (or time that I would invest if this blog were about the findings) to adequately validate it for the purposes of research. I certainly would have wanted to share both a data mining technique and research results, but there is only so much time. My hope is that the data is at least coherent enough to communicate this technique.

This is a good a time as any to mention that the data shown here only goes to 2008 because the quality of the data beyond 2008 was surprisingly just too strange.

Additionally, and I should include this paragraph in every blog I do on predictive analytics, predictive analytics should never become a substitute for thinking. It should be considered something like glasses or contact lenses, a tool that enhances our vision, nothing more. It is true that numbers don’t lie. But the human brain exists to resolve problems and with the pressures we are all under, we’re only too eager to buy into something that seems even remotely plausible so we can move on to resolving another problem on our plates.

There is so much data out there that I’m positive we could throw together a set of graphs to support anything conceivable. If one were to switch between Fox News and MSNBC a few times, one would immediately see what I mean. Therefore, the biggest discipline for helpful predictive analytics is the ability to not jump to conclusions.

Data Mining is not easy. It is extremely difficult. But most people who ask me about it think it is easy for the wrong reasons and hard for the wrong reasons. Many think it’s easy because they only know about the data gathering part and building the data mining models. They aren’t really aware of the less glamorous (doesn’t have a cool market name like “Big Data”), extremely tedious “data prep” and validation stages. Validation is usually given short-shrift. In fact it’s a much more developed concept in Map Rock where validation goes beyond testing the results of a test set, but providing tools to “stress a correlation” discovered through analysis. Building the models is by far the easiest part, not much more than executing an algorithm. Procuring the data is still very difficult, but the tools and techniques are well-developed.

So focus on the notion that change is the key to engaging our attention and that change requires time. The data is just to support our conversation here. Phew, that disclaimer rivaled any big pharma commercial.

Cluster Drift

Back to my presentation. Using the Analysis Services Cluster algorithm I generated a cluster model of ten clusters from data that includes about 120 countries from 1971 through 2006. As with most data mining, the vast majority of the results were uninteresting, very obvious, and sometimes confusing or contradictory. After perusing the results, I chose to focus on comparing the USA and Mexico for this blog since the resultant clustering of the two countries illustrate my main point.

Figure 2 shows a sample of the clusters. The selected clusters are the ones that the USA and Mexico belonged to at some points from 1971 through 2006.

USA and Mexico clusters.
Figure 2 – Clusters of the USA and Mexico. 

It’s easy to see the variance of values for each measure (look at the values for each measure from left to right). For example, Life Expectancy varies among the clusters in a range of about 30 years (51 through 80) and Enrollment Rate from about 37% to about 88%. However, the clusters aren’t too interesting in that for each cluster, the measures tend to go up and down together. It would have been very interesting to find a cluster where Life Expectancy is very high, but GDP/Capita is very low. We intuitively already know that high enrollment rate, high life expectancy, and high GDP would usually go hand in hand. Nothing new there.

Nonetheless, the clusters help measure when countries break thresholds that place them into different brackets. Figure 3 shows how the clusters of the USA and Mexico have drifted from 1971 through 2006.

USA and Mexico Cluster Drift.
Figure 3 – Cluster Drift of the USA and Mexico.

The row axis is the probability  that the USA or Mexico belongs to the color-coded cluster. Looking at the USA in 1971, we see that the USA most closely fit to Cluster 4 with about a .83 probability. What stands out about Cluster 4 is the higher GDP/Capita rate. We also see a small probability for fitting into Cluster 5 in 1971 and see that Cluster 5 trends upwards while Cluster 4 trends downwards until around 1976, the USA became more Cluster 5 than Cluster 4. The main differentiator between Cluster 4 and Cluster 5 is that for Cluster 5, the GDP/Capita is much higher, but the GDP/Capita Rate slowed a bit. During the early 1980s, we see the USA start to resemble what the USA is today, Cluster 7, firing on all cylinders.

From the Cluster Drift of the USA, notice that around 1976 and 1985, the  USA was in Cluster 5 at .7 probability. However, these were very different Cluster 5s, one on the away from the lower but rapidly growing GDP/Capita of Cluster 4 and the other giving way to very wealthy but more even GDP/Capita of Cluster 7.

Looking at Mexico, we see it starting out in Cluster 10 moving onto Cluster 2 with its higher Enrollment Rate, Life Expectancy, and GDP/Capita. Interestingly, variance of the GDP/Capita Rate of Cluster 10 is wider than it is for Cluster 2. My first thought would be the transition from Cluster 10 to Cluster 2 may hint of signs of stabilizing.

What is really interesting is that around the mid-1990s, Mexico started to fall into Cluster 4, where the USA was in 1971. And as of 2006, Cluster 5 began to trend upwards against Cluster 4. Does that observation hold in real life? Over the past ten years I’ve been to practically every corner of the USA, but to only Monterrey in Mexico. So I don’t know, but it’s precisely because I don’t directly know that I would resort to predictive analytics to arrive at a best guess from an indirect angle.

Figure 4 shows the USA against Japan. It’s interesting to see how what were the two biggest economies for a long period of time mirrored each other.

USA and Japan Cluster Drift.
Figure 4 – Cluster Drift of the USA and Japan.

Contrast Japan’s mirroring of the USA to another “Cluster 7 2006” member, Portugal in Figure 5.

Portugal Cluster Drift.
Figure 5 – Cluster Drift of Portugal.

Although the chart for Portugal suggests the people enjoy a similar quality of life to that of folks in the USA, Portugal’s grasp of Cluster 7 status isn’t very tight. It drifts between Cluster 5 and Cluster 7, whereas the USA and Japan are tightly in Cluster 7 despite Japan’s stagnant economy.

Figure 6 shows China. What is most interesting is that unlike the USA, Japan, and Mexico, from about 1975, it couldn’t fit very well (probability of say over .85) into any cluster. China certainly is unusual due to its sheer population size alone. But the USA’s portion of the world’s economy is similarly disproportionate to China’s portion of the world’s population. Yet, the cluster algorithm could fit the USA nicely into clusters much of the time.

China Cluster Drift.
Figure 6 – Cluster Drift of China.

China may be the 2nd biggest economy in the world today, but as of 2006, it hasn’t started to resemble the USA at any point as Mexico has (at least as the USA was in the 1970s – which was still very good). Remember, I didn’t include any cluster factors that speak to volume such as the GDP as itself (not per capita) or population size. China may be the 2nd largest economy today, but the GDP/Capita is still very small. Notice though in the lower right corner of the graph in Figure 6, that in 2003, Cluster 4 (as the USA was in 1971) starts to rise to a .1 probability.

Figure 7 shows the clusters China has bounced between, plus Cluster 7 (the USA in 2006) for comparison.

China clusters.
Figure 7 – China’s clusters plus Cluster 7 (current USA).

So What?

If it looks like a duck and quacks like a duck, it’s probably a duck, but there’s a chance it is a duck decoy. Predators getting prey to believe they are something they are not and prey getting predators to believe they are something they are not shaped life on Earth as it is today as much as the nature of water and the average temperature.

Whether we’re going to war, interviewing for a job, or making our pitch to a customer, our odds for success are greatly improved if we know more not just about the customers, but effectively differentiating from the competitors as well. If you’re a software vendor, you would want to know things such as:

  • How is the shape of my client-base changing?
  • How is the shape of my clients’ clients changing?
  • What are my competitors doing to outflank me?
  • What are the warning signs of the “death” of my customer?

For a software vendor, Cluster Drift is even more useful if the death and birth of the domain served is high, such as with restaurants. Could we answer something like, “Are we the choice software vendor for is now the walking dead?” If we found the proper clustering model, we could study the cluster drift of restaurants about to die and either help prevent it or move on to something else.

At the end of the day, what we want is intelligence on the entities we deal with. We want to know the nature of our relationship with these entities and how they are changing so we don’t interact inappropriately. Therefore, our cluster models must consist of measures that characterize those relationships. This includes things such as the number and type of contacts, number of successful and failed encounters, and volume of business; all divided by years so we can study the change in relationship.

The most compelling predictive analytics are implemented within a Darwinian meritocracy where business is a competition. For example, one of the big criticisms I’ve heard about Microsoft as a software company is that there are many groups working on the same thing. One example would be something like the workflow aspects of SSIS, BizTalk, Workflow Foundation, and even Visio developed redundantly throughout. It is true that it isn’t an optimal way to run a business, but it’s also beautifully Darwinian as well – attacking the same problem from different angles where each group is fighting for it’s own survival, eventually converging into a richer solution than had one group been charged with the problem. I know that sounds like heresy these days, but for a high-tech business that sort of Darwinian aspect seems to be a defining characteristic of an innovative entity, not a fault.

More Data Mining Caveats

It’s important to keep in mind as well that these clusters alone do not define a country or whatever is being clustered. This sounds obvious, but it’s fairly easy to get caught up in the fun of playing with these results. In reality, in our brains we cluster things from many angles. Most things operating in real life are too complex to be effectively captured via a single cluster. Some of these clusters are valuable, some not, either because they were misinterpreted from the start or have become obsolete.

The reason I love data mining is that it should force us to honestly reflect how we think and what drives us. But there is also a huge audience looking to data mining as alchemy; magic math that will reveal the secrets of the world providing riches grossly disproportionate to the effort invested (like winning the lottery). That magic math does indeed exist. The problem is that when we live in a world where intelligent creatures do not passively flow through life like trees and insects, the math will not calculate wins for everyone. Meaning, there is no way everyone can be the winner. Can a lion and gazelle both win? Life on Earth is based upon competition, an endless sequence of actions and changes.

Posted in Cutting-Edge Business Intelligence, Data Mining and Predictive Analytics | Tagged , , , | 1 Comment

Is OLAP Terminally Ill?

Someone told me yesterday that “OLAP is dead”. “Everyone is choosing tabular/in-memory.” I know it’s not dead, maybe at least sick. But did I underestimate the time of the tipping point, the curve in the hockey stick, where the vast majority of users will “sensibly choose” the tabular/in-memory option over OLAP?

I realize some, including me, think this topic is beaten to death. From the point of view that OLAP is my bread and butter, my top skill (I should have become a cop like my dad wanted), of course I took it to the heart, and take things (including “official” word from MSFT) with a grain of salt. But I also realize the person who told me this is very bright and knows the market. So I had to take a time-out today to revisit this issue as a reality check on my own professional strategy; a good thing to do every now and then.

When I first became aware of the OLAP is dead controversy a little over two years ago, I wasn’t too afraid of this since 256 GB of RAM was still really high-end. Today, 2 TB is “really high-end” (a few Moore’s Law iterations), well beyond the size of all but a few OLAP cubes I’ve dealt with (not even considering in-memory compression!).  And there were a couple of issues I still had not fully digested at that time.

One of those issues was not fully appreciating the value and genius of the in-memory compression. At first, I was only thinking that RAM with no IO is just simply faster. But the compression/decompression cost that occurs in the CPUs, which results in a whole lot more CPU utilization, isn’t really much of a cost since those cores were under-utilized anyway. Another was the volatility issue of RAM. At the time solid state memory was still fringe and my thought was that even though volatility wouldn’t be much of an issue in the read-only BI world, but would be an issue in the OLTP world. Well, that doesn’t seem to be the case with Hekaton.

After thinking for much of the night, here are two key questions I came up with that will determine whether OLAP (specifically SQL Server Analysis Services OLAP) will die:

  1. Is the question really more will hard drives (the kind we use today with the spinning wheels and all those moving parts) become obsolete? RAM and/or flash could neutrailize all the advantages of disks (cheaper, bigger, non-volatile) relatively soon.
  2. Will OLAP become minor enough in terms of utilization and product pull-through that Microsoft will no longer support a dev team? I can quickly think of a few Microsoft products with a strong but relatively small following that just didn’t warrant an infrastructure and were dumped.

An important thing to keep in mind is that there are really two separate issues. One is the underlying structures, OLAP versus in-memory, and tabular versus multi-dimensional. The first issue, the underlying structures, is a far more powerful argument for the death of OLAP. The underlying structure really will be seamless to the end-user and it won’t require any guru-level people to implement properly, messing with all those IO-related options.

However, I still don’t completely buy the “tabular is easier to understand than multi-dimensional” argument. I buy it to the point that, yes, it is true, but I don’t think this is the way it should be. My feeling is that the multi-dimensional concepts encapsulated in MDX and OLAP are more along the lines of how we think than what is encapsulated with SQL and relational databases. What comes to mind is the many times I’ve engaged a customer with “thousands of reports” that were really variations of a couple dozen and were mostly replaced with a cube or two.

As a side note, one exercise I use to demonstrate the elegance of MDX is to think about the syntax of how Excel handles multi-dimensions. Excel is multi-dimension, but just two dimensions. With a cap on dimensionality, it’s easy to use the A1 (column A, row 1) syntax. But what about three dimensions? A sheet (Sheet1$A1). Four dimensions? A different xlsx document. Five? A different directory. That’s not at all elegant. But MDX elegantly “scales” in the number of dimensions; it looks the same from zero through 128 dimensions.

The tabular model reminds me of when I started my OLAP career in 1998 as a developer on the “OLAP Services” (SQL Server 7.0 version of Analysis Services) team at Microsoft. OLAP for SQL Server 7.0 was really the just core OLAP, no frills, just strong hierarchies and aggregations. It was very easy to understand, but users quickly hit walls with it. That reminds me of how VB was so easy to learn. One could learn to build pretty good applications quickly, but would run into problems venturing beyond the 80/20 point. Eventually .NET (C#/VB.NET) came along, still relatively easy to use (compared to C++), but still a quantum leap in complexity. For OLAP Services, that was SQL Server 2005 Analysis Services with the weak hierarchies, many to many relationships, MDX Script, KPIs, etc.

I guess what I’m saying is this is a case of taking a step backwards to take two steps forward. The spotlight (tabular) isn’t currently on the high-end where I normally make my living. However, it doesn’t mean there isn’t a high-end. The high-end as we know it today (OLAP) will eventually die or at least severely morph, but requirements of yet unknown sorts on the low-end will push the complexity back up. How will Big Data affect the kinds of analysis that are done? Will 2 TB of RAM then be sufficient for the “masses”?

At the moment, I do believe that in terms of raw new BI implementations, tabular is giving a whooping to OLAP. It should since the idea is to expand the accessibility of BI to a much broader audience. I’ve lived through the rise of Windows 3.1 and the dot-com crash. This is a minor disruption; it’s not like I haven’t begun moving on years ago – in fact, skill-wise, I learned to always to be moving on to some extent.

BTW, he also told me that “no one makes money on Big Data and that Predictive Analytics is limited to one or two people”. Those are in fact the two skills I’ve been shifting towards the past few years in light of the sickness of OLAP. While I really don’t know about the former claim (and would find it short-sighted to base strategy on that), I do have a couple of opinions on the latter:


Figure 1 – Even my cat wonders whether cubes could be on the way out.

Posted in SQL Server Analysis Services | Tagged | 12 Comments

Map Rock Problem Statement – Part 5 of 5

This is Part 5 of 5, the last of the Map Rock Problem Statement series. Part 4 closed with a discussion on the limitations of logic, in particular how the quality of our decisions are limited to whatever information we have available to us. No matter how smart we are, what we know is a drop in the bucket. We’ll often encounter situations for which an answer “does not compute” due to this imperfect information. I close this series with a short essay on Imagination. The previous installments can be viewed here:

  • Part 1 – Preface to the 5-part series.
  • Part 2 –  I describe Map Rock’s target audience and the primary business scenario for Version 1. It is not just a tool for quants, wonks, and power-users.
  • Part 3 – We delve into a high-level description of the Map Rock software application, where it fits in the current BI framework, and how it differentiates from existing and emerging technologies. This is really the meat of the series.
  • Part 4 – We explore strategy, complexity, competition, and the limitations of logic.

Imagination, the Key to Human Success

The more we know, the more we know we don’t know. That is the thinker’s equivalent of the “Serenity Prayer”. I try to remember that every day, but often fail to in the heat of troubleshooting a brainteaser of a bug or performance issue in my applications. My Western, Aristotelian, reductionist science upbringing naturally forces me towards employing increasingly intense logic and tighter grasp over the situation, to no success. With billions of my brain’s “CPUs” pegged at 100% (our brains are massively parallel), I finally resort to the counter-intuitive action of taking a long walk or going home to sleep while the problem is still very much there wreaking havoc on the world.

However, as hundreds of thousands of clusters of neurons in my brain wind down their Gatling Gun fire, the very faint signal of a “weak tie” still lingers. It is a faded event from my childhood so unrelated to this situation and illogical in this context that it’s “sensible” to dismiss as ludicrous. But I still playfully think, “What if … ?” I begin to imagine relationships that I had never thought of or learned. These imagined relationships plugged into the web of what I know would result in a diagnosis of my problem, from which I can then engineer a solution! I run back into the building, to my laptop, to test if these imagined relationships do indeed exist. After resolving the problem, with my room service Mobley Burger as a reward, I marvel again at how the reliance on just pure logic applied to what I know I know (at least what is at the forefront of my thoughts) failed me.

The weak memory served as a metaphor, a template from which I can creatively substitute similar pieces with their corresponding current pieces or even add or delete pieces. Old ideas may not apply in exactly the same way in a new situation. They are a starting point towards a solution. Meaning, that old memory is not a “direct hit” recognition of what we are currently experiencing. It’s technically a mistake, like how I keep mistaking David Proval for Al Pacino. Our brain has to support us in a complex world where nothing is certain. But it’s these fuzzy, “close, but no cigar” recognitions that are the basis for imagination, our primary weapon. 

It’s our ability to imagine that raises us above all other creatures on Earth in terms of our ability to manipulate the environment to our liking. The options our brains are capable of producing aren’t limited to hard-wired instinct or what relatively little we’ve experienced (in comparison to all there is to experience). Being bi-pedal, having opposable thumbs and a large brain are all just characters within the bigger story of how our imagination propelled humanity to being the current lords of this planet.

If something hasn’t yet happened, our knee-jerk reaction is that we believe it could never happen. Our brain doesn’t have that relationship, so any inferences we make are computed without that relationship. This is the limitation of logic. Logical conclusions are based on the facts that are available. The problem is that the world is a complex system and thus we never really have all of the facts required to make fool-proof predictions.

Imagination means we are able to draw a relationship between two or more things that were never explicitly stated to us or so weakly connected that we normally disregard it as a silly thought or even paranoia. This is how humans have our cake and eat it too. We have the power to apply logic through symbolic thinking, enabling us to manipulate the environment to our benefit (within a limitation of space and time), yet overcome the limitations of logic in a complex world.

When we think, we do more than just process values in a Bayesian manner. We also audition ideas by playing what-if experiments in our mind. We safely experiment in our heads before committing to a physically irreversible action. We can change our mind when we suspect that the cost of being wrong is too great. This ability means that the results of our thinking are not as mechanically pre-ordained as that of any purely logical device such as a machine, rigid equation, or set of best practices. Imagination breaks us out of the rather Newtonian condition of reptiles, birds, and insects that are lives riding out a deterministic fate.

Armed with imagination, each human is a force of the universe, cognizant of our own deaths, thus not really willing to settle for just the greater good of the species – even though that would be admirable. We are capable of subverting the laws of physics through our contraptions. We can imagine an outcome other than what the vectors of each object dictate. We can manipulate other objects towards a more desirable outcome.

However, imagination is being expunged from our feature-list as a flaw. We are goaded into “best practices” and chastised for not following procedure, even if that insubordination resulted in a superior outcome. Imagination is seen as a childish characteristic, something that should be out-grown.

Why is this? Going back to the beginning of the Problem Statement series, the world is spinning faster and faster. There are a lot more people with a sufficient amount of wealth, information about all sorts of things, and wanting more things faster. This equates to chaos. Now, Map Rock isn’t about how to fix this chaos. I honestly don’t have an answer for that. What I am offering is how to better compete in this chaos.

Map Rock addresses imagination by guiding the user through brainstorming activity, integration of rules from many sources (particularly Predictive Analytics models), and the SCL mechanism for inference from multiple perspectives.

The Will to Compete

Thinking is hard and usually associated with some sort of pain or problem we need to address. We engage the act of thinking to resolve problems. Similarly, hunger is painful. Our ancestors prior to agriculture walked many miles and risked their lives engaging much larger and stronger animals to alleviate the pain of hunger. If hunger weren’t unpleasant, I’d probably be thin since I wouldn’t care to go through the trouble of shopping and cooking. We’d often rather make ourselves believe that a solution currently in play and what our beliefs resolve to will in the end succeed than face a string of burdensome fire drills – ie keep telling yourself that, eventually you’ll believe it.

Thankfully, we can invent or learn a procedure to easily alleviate pain without needing to re-invent it the next time. These procedures (tips and tricks, best practices) can be followed to alleviate pain with a chance for success ranging from good to great without needing to engage thinking. These procedures are incorporated into our lives, programmed into our psyche. Any time someone tries something different, it can upset the delicate machine that is serving us well, forcing us to think.

But here is the kicker. Here are two major thoughts upon which life on Earth has evolved:

  1. Every creature makes its living devouring other creatures and every creature tries to reproduce (or grow) as much as it can, exerting constant pressures driving other things to take action to survive. “Grow or Die” as they say.
  2. The world is a complex system with so many moving parts that predicting anything beyond a very limited volume of space and time is at best unreliable.

Putting aside the bacteria that consume raw minerals off rocks and dirt, every creature hunts and devours another creature. We are all both predator and prey, the hunters and the hunted. There are two sides to our intellectual lives; how to be better hunters and how to better avoid our hunters. Add to that as well that there is also the competitor relationship, peers in contention for the same resources. This predator and prey relationship, both trying to outdo each other, means that for life to keep going, creatures must be able to evolve to adapt to new tactics applied by the “enemy”. Creatures all try to over-reproduce because there is a constant pressure upon it from its predators as well as peer-competitors.

We reactively adapt to change (forced upon us by our predators and competitors) and we proactively engineer and mold conditions to our favor. Players at the top are at the top because conditions favor their strategies. The conditions were either already there yet unseen or they were imagined and forged into place. In either case, the company at the top intuitively defends these current conditions since different conditions may not favor them anymore. They don’t know for sure if different conditions would indeed favor them because the world is too complex to really predict such things. That’s why we still run the experiments and we still “play the game” even if we’re pretty sure about who will win.

We can’t just sit idly by not doing anything because we don’t know for sure. We can’t stubbornly hold out for a “just one number” answer or refuse to take “It depends …” as an answer. We take a best guess by running our thought experiments safely in our head, weeding out scenarios with the possibility for unacceptable outcomes, before we decide to take a physically irreversible action in the real world. Even if we’re wrong, meaning during the execution of our action we recognize a negative intended or unintended condition is developing, our second line of defense is that we can make on the fly adjustments.

The thing is, an action isn’t just an isolated action. It sets into motion lines of series of events that change things forever, even if just subtly at first. And the complexity of things make it impossible to predict what the results will be even just a few steps down the line.

Competition Is a Good Thing

I feel we have more than just a right to compete or to defend ourselves against “aggression” but an obligation as creatures of this planet in order to keep driving evolution. If we all simply “go with the flow”, offering no resistance to a force (of competition or even aggression), eventually everything will settle into a dull static pattern. It’s the resistance offered by the countless interacting things in the world that result in the dynamic system we call life, the action we live in. In note #1, I mention bowing before judo matches.

I get this feeling competition and aggression are dirty words these days. There are strongly driven criminals (of many sorts) who have been publicized for cheating in one way or another resulting in harm to others. There are bullies. I certainly don’t like cheaters as those minority of people impose a cost on all of us as we all go through the inconveniences of the processes to hinder their efforts (like TSA screenings and passwords on all web sites we use). But we cannot lump the sportsman-like competition of commerce in with the criminals and bullies.

Even when we do say “competition”, I think we’re trained to think only of the ridiculous sort on reality TV such as American Idol and Shark Tank. They are competitions, but what I’d think of as games where the rules are pretty set. To me, a game is something like that, where set rules very much dominate and there is specific criteria. Surely, landing a contract or a job is like one of those shows in that the “judges” (customer or employer) as looking for specific qualities and the winner will be the one to fit those traits the best. We don’t think of the sort of competition that goes hand in hand with a stress that moves things in a different direction; avoiding that dull, lifeless pattern.

What I’m trying to get at (dancing around so as not to sound paranoid), as tongue-in-cheek as I can is, yes, “they” are out to get us. Life is an eternal struggle between predator and prey – and most creatures are both to some extent. It’s usually “nothing personal, just business”, as they say. Everyone (every creature) is driven to survive, which in turn means seeking resources which are both consumed by and taken from the creature. This drive to survive is the churning, live action that fuels evolution.  Problems, mostly some form of contention for resources, pops up at every turn. Humans are problem solvers, better at it than anything else on the planet except nature herself – who will still kick our ass in the end.

Keep in mind that “evolution” doesn’t necessarily imply “improvement”, that things are now superior to how it used to be. For example, plants that made it across oceans to the sterile lands of Hawaii initially had thorns, but lost them as there was no longer any need for them in an environment devoid of reptiles and mammals. Is it better or worse? Neither.  Evolution means to adapt to changing conditions. In the short term, adapting may usually seem like improvement since we are then more comfortable after the adaptation.

This mechanism of evolution, simultaneous destruction and creation, is the reason why increasing complexity doesn’t completely destroy our planet. Our planet takes hits, but the force of evolution eventually will heal the wound and things will be vibrant again, even though things will not heal exactly as it was before. For me, the big lesson is that neither the extreme of resistance to change nor the passivity of going with the flow (no resistance to change) are good. The secret is to blend in with the system, a collection of relationships, in a jujitsu-like manner.

Map Rock’s central unit is strategy. Strategies are to Map Rock as DNA is to the various forms of life. It is through designed strategies by which humans excel. For the most part though, strategies don’t come out of a person’s head fully-baked. It involves a great deal of trial and error and an ability to recover from error. This is what Map Rock is all about.


From the heart, my primary motivation for developing Map Rock is to fight for “humanness” as the variance from our lives is efficiently and systematically purged in the name of optimization. But what else can we do? The way of life as it was a couple hundred years ago with only a few hundred million people isn’t scalable to seven billion. We need to be more efficient and sacrifice some things. However, as I mention whenever I’ve previously described two sides of a coin, neither is “bad”.

The way we think reflects both our unique sets of individual experiences through our lives as well as “harder-coded” functions we more or less share, which served us well when we were still living in caves. Flaws and all, our way of thinking has done pretty well for us. Our imperfections are not just quaint or nostalgic. It’s these imperfections that break the tyranny of myopic logic..

Coming Up:

  • Map Rock Proof of Concept – This blog, following the Problem Statement series, will describe how to assess the need for Map Rock, readiness, a demo, and what a proof-of-concept could look like.


  1. I didn’t fully understand bowing to our opponent before a judo match until I was much older. I had taken it as simply a polite gesture or maybe even just telling each other it’s nothing personal. But it’s also a promise to give you my best effort and in all sincerity ask that you do the same so that we do not waste each other’s time. In fact, poor practice doesn’t just waste your time, it makes you worse.
Posted in Map Rock | Leave a comment