I applied to speak at the inaugural TEDx Boise 2015 event, but I was not selected. However, I would still like to share what I had intended to say had I been chosen. The Synopsis is what I submitted to the Selection Committee and the Speech is what I subsequently started to put together just in case.
Last month (Nov 2014) I took a two-week sabbatical in the Zion area to think through concepts that I began to write about in a previous blog, Embracing Complexity – Part 1 of 6. It involves breaking through the limited paradigm under which analytics is currently implemented at mainstream enterprises, which I think are remnants of a time when the hardware and supporting software could not support true Business Intelligence. And now we’re too used to BI systems being more reporting tool than accentuating our human analytical capabilities.
After over 100 miles of hikes, over 50,000 words of notes, and maxing out my iPhone with voice recordings over those two weeks at Zion, I’m attempting to digest those concepts of moving analytics in the mainstream forward into a book and some supporting software. I suppose this speech I would have made will serve as a fair abstract to the book.
It’s worth mentioning that Boise actually has quite an impressive community of Business Intelligence professionals. ProClarity played a large part, attracting top talent as well as nurturing much local talent. Additionally, the outdoors lifestyle is attractive to the creative sorts who migrate to Business Intelligence. In fact, it was the purchase of ProClarity by Microsoft that brought me to Boise almost exactly eight years ago. So my speech was in large part intended to help further the nurturing of this already impressive community.
Before reading the (would be) Speech, keep in mind, that it is certainly not exactly what I would have ended up delivering. Had I been selected, I would have run it by many friends, cleaned and tightened it up much more based on feedback, and adjusted some things that I’d forgotten would be obvious only to BI and/or software sort of folks. Meaning, adjusting to a more generalized audience. Consequently, for a generalized audience, I take some liberties with definitions and gloss over some messiness towards the goal of getting my point out in 18 minutes.
Additionally, I think the TEDx folks provide some level of guidance. And besides, if you’ve ever attended one of my presentations or workshops, I hardly ever stick to script anyway … hahaha.
Lastly, the Speech is written as a speech. I wrote it imagining me presenting it as I would with certain inflections. So, some sentences may seem rambling because of the limitations of punctuation, some things perhaps redundant because I’m trying to drive home an important point.
Title: How to Think in 4 Dimensions
Synopsis: The latest generation of analytics tools driven by Big Data drastically accentuates our ability to navigate a truly 4D world. However, it’s a skill that must be learned for that power to be fully appreciated. We’re good at recognizing 3D things such as faces, places, and food, from any angle, up close or at a distance, in full view or obscured. But we’re actually bad at making predictions, which involves the chaotic interaction of many independently moving things over time. More often than not, we get things wrong, sometimes for millennia. And it only gets worse as the complexity of the world increases with more people, the Internet of Things, and the farther reach of everything. We ourselves are mere 3D objects faking our way through a 4D world, via our intelligent use of information. It’s unnatural for us, so we struggle to fight chaos through oversimplification and tightening control instead of counter-intuitively embracing complexity.
The Speech (what I would have said):
We live in a four-dimensional world, three spatial dimensions and time. We readily understand the three spatial dimensions, depth, width, and height, but most of us probably don’t fully comprehend the 4th dimension of time. Or at least, we forget about the true nature of the 4th dimension as we struggle through our daily lives addressing problems after unrelated problems, plural intended, that hit us as fast as snowflakes as we walk through a snowstorm. That relentless multi-tasking shatters our energy into unrelated silos of effort that don’t and can’t add up to the grander things we’re striving for. Addressing so many simple things makes us good at handling simple things resulting in the atrophy of our ability to think into that realm that’s bigger than our brains.
The problem is that we’re actually 3D creatures living in a point in time, along with countless other things existing with us at that point, faking our way through the 4th dimension with our uniquely human super powers of memory and symbolic thinking. Oversimplifying, memory and symbolic thinking allow us to perform what-if experiments about what could happen, in the safety of our heads before committing to physically irreversible actions. For a 3D creature faking its way through the 4th dimension, the 4th dimension means the exploration of all possibilities. So what does that mean?
Imagine a weather site in your Web browser with one of those 3D visualizations of your region, a fairly large area. It’s tilted so you can see the places and topography of your region and you can also see the height and thickness of the clouds, the density of the rain or snow. At the lower left of that page is a slider bar showing time, the 4th dimension, from the past, say four hours ago, to even the future predictions several hours from now. You slide that bar from far left towards the right watching the clouds and rain as they had moved over the past few hours along in that 3D visualization. You reach the present on that slider and continue to sliding towards the right, but now into the future seeing the clouds and rain move through space as the sophisticated predictive models have calculated.
Is that 4D? It is, to a limited extent. We can see all the 3D clouds and rain moving through time. But the future we see is just one possibility. One possibility is fine if we knew it’s a certainly, not merely one possibility among many others. We all know that’s not the case. We know weather predictions are often wrong, sometimes very wrong. Predictions are often wrong because the changes through time are calculated step by step, say every minute. At any given point in time, the calculated future of those clouds and rain is never exact. The most likely position is selected, but we treat it as though it is exact, ignoring the possibilities calculated to be less likely. Over subsequent steps, those approximations accumulate and exacerbate errors until a few hours out, the prediction looks nothing like what actually happens.
In the 4th dimension of time, as far as a 3D thing like we humans are concerned, the 4th dimension encompasses all possibilities. We generally think of more than one possibility, worrying about a less likely outcome, and are sometimes called paranoid for considering unlikely bad outcomes or delusional for thinking about very unlikely success. Certainly, we cannot consider all possibilities, not even a small fraction of them, even if our brains held magnitudes more capability. And that’s OK since most things would be so unlikely it’s not worth thinking about. For example, a rock on a planet in a distant solar system not will appear out of the blue, hit us on our head, with our head turning into asparagus.
Humanity’s unsurpassed capability with the 4th dimension is the secret sauce of our Earthly dominance. Our memories and symbolic thinking, though not flawless, are enough power for us to beat out every other species on this planet. But now that we don’t worry about a grizzly bear running into our camps and mauling one of us for dinner, we now have moved up to beating other equally intelligent humans with that same level of intelligence that gave us mastery over bears, wheat, cows, and chickens. It turns out that the smarts to beat the grizzly bears isn’t enough to beat other humans on the same playing field, so we learn to strategize in an arms race of technology. That is, we’re capable of knitting together a set of steps, plans, towards a goal. Our brains happened to have excess capacity and our ability to form teams helped scale up our capabilities further.
Here’s the catch. Eventually our brains reach their limits, and they become the bottleneck, even with teamwork and technology. The world we’ve created for ourselves is at or even beyond our ability to fully master it. Today, the world is so complex that as we joke in the software industry, we fix one bug and create two new ones. Another analogy from the software industry is that everything is easy until scalability becomes an issue.
To remedy that, we seek the comforts of simplicity and tighter control, taming the complexity of life back under the limits of our brains. We actively train ourselves to just one possibility. Our decisions are becoming formulaic heuristics, not thoughtful actions. We really have no choice. Seven billion and growing sentient beings, each with their own intelligence and desires, result in a messy web of demands set upon all of us. There are tens of thousands of government regulations, many which contradict each other, thousands of sensitive points to avoid, hundreds of mini decisions off-loaded by companies to us in the guise of self-service convenience. There are so many decisions we must make that we shun the answers that begin with “It depends …” or “Just one number please …”. There are so many factors that we can’t afford the time to consider the multitudes of “it depends” scenarios. We don’t even have time to consider even just a tiny fraction of the infinite possibilities, or we quickly freeze through analysis paralysis.
We never have time to deeply learn, to reflect, to let it all sink in, to let the fundamental “what fires together wires together” mechanism of our intelligence fully digest our recent experiences. We rush through tightly-scoped Scrum sprint projects at work in two weeks and before the current one is even over we’re dipping into the next one. Sure, there is a post-mortem phase, but it’s usually the first thing to be cut when time is short, which is usually the case, and it’s too little anyway. Additionally, there’s also no real time to celebrate, to let our brains equate the success with something good so that we’re driven to succeed during subsequent sprints.
So much is thrust upon us in this literally mind-bogglingly complex world we’ve created for ourselves that our power to see into the 4th dimension overloads our brains and we keep failing. Consequently, we retreat to worshiping simplicity, best practices, conformity; the comfort of “now”.
For the sake of simplicity and conformity, we disregard the diversity of our genes by developing drugs, cars, and airplane seats that are optimal for few, even detrimental for some, but work merely OK enough for the vast majority of the people. We insist on shoving pegs of all shapes into square holes. I chose squares because they easily stack into nice, compliant piles.
Our diversity of experience is swept under the rug through standardized testing. Our unique experiences form the boundaries of our logic, so the more collective experience we have, the better our logic. There are multiple ways to attack a problem, some more optimal than others, but those are usually under assumed, rigid circumstances. Meaning, that shortcut won’t work all the time, and the real value of human intelligence is when we need to come up with an alternative. Anyone can learn a standardized answer or well-defined process, even a computer, hint hint, but it’s a completely different thing to engineer a novel solution. Those novel solutions exist out there in the 4th dimension.
We’ve lost our capability for delayed gratification, arguably the single most important skill. That is, the ability to see something that the bears and alligators cannot see because they can only see the possibilities where an action leads to immediate improvement.
Everyone panics if there isn’t immediate progress, not considering that sometimes things go one way before they go the other; sometimes worse before getting better, sometimes better before getting worse. Answers are given to us without forcing us to explore webs of paths. Most of those paths don’t solve our problem, but they expose us to much more that will in part or whole be of use to us someday. Those currently fruitless, seemingly dead-end paths enrich our treasure trove of experience, the experience from which the quality of our logic is derived. It strengthens the skills required to engineer; perseverance, imagination, resourcefulness. We instantly disregard anything but the perfect 5-star choices, even though the choice with the checkered past, failing through all of those fruitless paths, may indeed now be the choice for a world that is all too complex.
To recap, our ability to consider many possible paths, at least more than other species, is our human shtick. But we’ve created a world of such complexity that it requires us to look at a daunting number of possibilities that has overloaded the capacity of our brain, resulting in chronically making bad decisions. So we’ve retreated to simplicity and tighter control, conformed heuristics. It strips from each of us the capacity to be the far-sighted forces of the universe that each of us are, devolving towards becoming short-sighted, push-button automata, relinquishing our sentience to the rule-based, single-step intelligence of our pre-neo-cortex ancestors.
Even so, the ability to think deeply still exists in areas where it must. Great chess and Texas Holdem players look several steps out, where each step along the way doesn’t necessarily take them closer to the ultimate goal. They understand the notion of delayed gratification as sacrifice and investment. They consider the consequences for being wrong as well as giving weight to plans with the most “outs”.
Over the years I’ve met many brilliant software debuggers and troubleshooters. They have this knack for beginning in the end, where all the pieces have converged into an incident, an error code or murder, and working backwards, with all those clues dispersing into what appears to be a mess of unrelated facts. Working backwards as such helps us to prune out the virtually impossible paths without tossing out the interesting outliers and weak links which may be the signs of things to come. Maybe that’s why “visualizing” a successful outcome, a technique taught in sports psychology, often works. Seeing that end, our subconscious works backwards ultimately to where we are now.
Geologists, like mystery writers, start with the end product, what they are currently seeing, such as a strange piece of sandstone with silver in it, and work backwards to unravel how it became what it is. They wonder, “How did this silver get here?” They must work through events that involve the effects from combinations of moving continents, volcanos, erosion, chemical and physical processes, the effects of biological activity. Every rock has a complicated and somewhat unique story, a story we can Google, but someone has to have figured it out first.
Experienced, grizzled doctors, geologists, mystery writers, detectives, software debuggers, and the elderly, know that for every outcome there are countless combinations of contributing factors, and so there is hardly really just one “major cause”; and that every time we attempt to never let something bad happen again by outlawing the causes for that one incident, there are countless other ways for it to happen.
So with all that said, now that Big Data has laid the foundation for gathering and accessing practically any sort of data, the next step in analytics will mature beyond the reactive, quick answer machines, instant gratification, asking questions of the “What is the ‘blank’ ?” nature we’ve conditioned ourselves to ask, to “What are the steps towards achieving ‘blank’ ?” The mainstream analytics space has punted the issue of what it takes for the level of analytics that can promote orderly design and execution of strategies but not at the expense of our personal individuality or ignoring the fact that our competitors aren’t going to cooperate. That is, we need to move back to focusing on relationships and the complex nature of things, as opposed to simply more data, the illusion of doing something while avoiding the hard problem. And by “back to” I mean before the instant gratification days when every day was a unique puzzle unanswerable by Google and food didn’t appear in five minutes.
Instead of accentuating specific “powers”, such as obtaining the accuracy of an eagle’s eyes, as facial recognition system do, or the speed of rather dumb machines, new technologies will help accentuate our ability to explore the infinite frontiers of the 4th dimension, in truly massively parallel, massively recursive, and massively hierarchical fashion. It will accentuate our logical capability rather than just being a glorified index, leaving the heavy intellectual lifting to our three pounds of brains. Genetic Algorithms will find solutions not by progressively selecting from sets of possibilities with immediate improvement, but looking at multiple steps towards improvement. With that expanded capability to explore the 4th dimension, we can then be braver about considering the unlikely, the black swans and weak ties, and not freeze through analysis paralysis or over-react to paranoia.
Our business intelligence systems will finally focus on the relationships between data and not just improving analysis with yet more data. Such an improvement will enable real integration, of the “whole is greater than the sum of its parts” variety, of our collective experiences and expertise, allowing for a genuine move from the limitations of top-down control at our institutions towards a distributed , scalable intelligence.
Until then, with or without that new technology accentuating our ability to explore the 4th dimension, here are a few things to keep in mind.
Keep it in the forefront of your mind that anything is possible. I mean anything. I don’t mean this in a “new age, positive outlook” or even a “quantum physics, uncertainty principle, multiverse” way. I mean that, firstly, the world is a complex system and we’re always making decisions on imperfect information, nothing is certain, even those weather forecasts stating there is a zero-percent chance for rain. Secondly, we are just 3D creatures faking our way through what is probably even more than a four dimensional world. Each dimension doesn’t merely add another element to a tuple, but almost god-like powers; read “Flatland” and “Spaceland”. And thirdly, our brains, though massive in capacity are still limited to the experiences we can shove into them, in what is a very short time and being trapped to one place at a time, so our logic is innately limited. Meaning, our logic could be infallible, but it’s based on our limited and unique set of experiences; so we could still be wrong. Don’t be one of those, “If it hasn’t yet happened, it couldn’t possibly happen” or “That’s not the way it’s been done” people.
Lastly, visit Zion, Canyonland, and Bryce Canyon National Parks; someplace bigger than you, bigger than humanity, that cannot be tamed. Hike the trails, even the scary ones; but don’t do anything crazy. Say hello to every single person you pass on the trail and always yield to them on the narrow, scary parts. Find one of those views that cannot be put into words, beyond just a poetic sense, because it requires integration of all of your senses; Angel’s Landing and Observation Point are my favorite. Take off your backpack, pull out a bottle of water and a Cliff Bar, and sit. Feel the vastness of the past written in every wall of those parks. Try to take a picture of it, and realize that 2D photo faking a 3D scene will never match what your five senses, consciousness, and subconscious as a whole are feeling at that instant. Ponder that the distance between humans and the dinosaurs is 65 million years, but the distance of strata from the top of Bryce Canyon to the lower reaches of Zion spans over 200 million years. And know that in a short time what you see will also be gone, replaced by something else. Think about how those crevices and even the entire canyons were formed by many processes, one of which goes something like, a random ball of plant matter inside that sandstone long ago submerged under the water table leached out and attracted the iron in the sandstone around it into a hard, walnut-sized iron concretion; that iron concretion eventually was exposed through erosion and fell out leaving a pock mark, which grew to a larger dent, merging with other dents, exacerbating into a crevice and even a canyon. Stay there long enough to let that feeling etch into you. Then begin your journey back. Take your time so as to be careful about your knees and not twist your ankle. And think of all the ways Zion could have been or even not have been had even the most miniscule of things been different.