In the quest for data-driven excellence, the ability to make informed decisions quickly and with more than superficial thought is more crucial than ever. Enterprises are increasingly seeking ways to harness the power of data to gain insights, drive innovation, and maintain a competitive edge. One of the key technologies that facilitate this process is Online Analytical Processing (OLAP). In this blog, we’ll explore the role of OLAP in enterprise intelligence and how it simplifies the journey from data to insights.
OLAP cubes play a significant role in my new book, Enterprise Intelligence (available at Technics Publications and Amazon). In the tradition of SQL Server Analysis Services (Multi-Dimensional), Kyvos Insights, can play a significant role in building a platform worthy of being called “The Intelligence of a Business”. However, like Sehrah Connah in the Terminator movie, we don’t know how important she really is until the end when we find out she is the mother of Chon Connah, the main protagonist and future leader of the resistance.
Note that if you purchase the book from the Technics Publications site, use the coupon code TP25 for a 25% discount off most items (as of June 25, 2024).
I do want to be clear that although I highly recommend the incorporation of pre-aggregated OLAP cubes, it isn’t required for what I discuss in my book. However, this is a crazy time where AI is in the hockey-stick part of the s-curve (even though there are signs it might be hitting the point of diminishing returns). I discuss the renewed need for OLAP cubes in my blog, The Effect of Recent AI Developments on BI Data Volume.
This blog is written with me wearing my “Principal Solutions Architect at Kyvos Insights” hat. That is, as opposed to the book announcement wearing my general Data/AI Engineer hat–the hat I usually wore to write my book.
What is OLAP?
OLAP is a category of data processing that enables users to interactively analyze multidimensional data from multiple perspectives. The core idea behind OLAP is to pre-aggregate data into cubes, allowing for rapid query performance and the ability to analyze large volumes of data efficiently. This pre-aggregation means that complex calculations are performed in advance, so users can retrieve results quickly without waiting for on-the-fly computations.
The Evolution of OLAP
Historically, OLAP has been synonymous with tools like SQL Server Analysis Services (SSAS), which provided powerful capabilities for data analysis. However, with the advent of big data and cloud technologies, traditional OLAP systems faced challenges in scalability and flexibility. This led to a temporary decline in the popularity of OLAP as newer, more scalable solutions like Hadoop and in-memory processing gained traction.
But the sheer volume and variety of data generated by modern enterprises—driven by AI advancements, the proliferation of IoT devices, and the explosion of unstructured data—necessitated a return to the optimized capabilities of OLAP, albeit in a more scalable and cloud-friendly form.
OLAP in the Context of Enterprise Intelligence
My new book, Enterprise Intelligence, is a discussion of how integrating BI structures into an Enterprise Knowledge Graph (EKG) can transform businesses into intelligent, agile entities. A key component of this transformation is the role of OLAP in managing and querying vast amounts of analytical data efficiently.
Following are a few bullet points on the value of OLAP today, in an AI world.
Speed and Efficiency
One of the primary advantages of OLAP is its ability to deliver sub-second query responses. This speed is essential for business intelligence, where decision-makers need to access insights rapidly to respond to market changes, customer demands, and operational challenges. By pre-aggregating data, OLAP systems reduce the computational burden during query time, allowing for faster data retrieval and analysis.
I discuss how this speed and efficiency is accomplished in my blog, All Roads Lead to OLAP Cubes … Eventually.
Scalability with Kyvos
Modern OLAP platforms like Kyvos have taken the traditional OLAP model and scaled it to meet the demands of today’s data landscape. Kyvos’ scale-out architecture allows enterprises to handle terabytes to petabytes of data, providing high concurrency and responsiveness. This scalability is crucial as the number of BI consumers and data sources grows, thanks to data mesh and the increasing quest to be data-driven.
Please see my blog, OLAP is Back as Kyvos Insights.
Enhanced BI Structures
OLAP plays a pivotal role in the two BI-derived structures within the EKG I describe in my book: the Insight Space Graph (ISG) and the Tuple Correlation Web (TCW). These structures passively capture insights from the BI activities of analysts across the enterprise, charting points of interest and correlations across an expansive space of insights. OLAP’s ability to pre-aggregate data ensures that these insights can be captured and queried efficiently, supporting a more dynamic and responsive BI environment.
I offer an introduction to the ISG in my blog, Knowledge Graph BI Component: Insight Space Graph.
Pruning with Minimal Penalty
One of the significant benefits of using OLAP in conjunction with the ISG/TCW is the reduced penalty for pruning data from what will be a massive graph. As the ISG captures salient points from visualizations and the TCW charts correlations between tuples, enterprises can efficiently manage and prune these insights without losing valuable information. Since OLAP cubes provide rapid query performance, any pruned insights can be re-queried and retrieved with minimal delay. This ensures that the EKG remains lean and efficient while maintaining the ability to quickly recover any needed insights, providing a flexible and resilient analytical environment.
Facilitating Complex Analysis
The multi-dimensional nature of OLAP cubes allows for sophisticated analysis across various dimensions, such as time, geography, and product lines. This capability is essential for identifying trends, patterns, and correlations that drive strategic decision-making. For example, a BI analyst can quickly drill down into sales data to uncover regional performance variations or analyze customer behavior across different time periods.
Handling High-Concurrency Scenarios
As the user base for BI data expands beyond traditional analysts and managers to include a diverse array of knowledge workers, the need for systems that can handle high-concurrency scenarios becomes critical. OLAP’s pre-aggregation capabilities mitigate the performance impact of simultaneous queries, ensuring that all users can access the data they need without delays.
I discuss this in more detail in my blog, The Role of OLAP Cube Concurrency Performance in the AI Era.
Pre-Aggregations are Resource-Preserving Cache
Pre-aggregations essentially function as a form of caching. By storing the results of previously computed processes, such as aggregations, we avoid the need to recompute data from scratch for each query. This allows the system to preserve compute resources, making it more efficient at handling concurrent queries.
In effect, the energy saved from avoiding redundant computations can be redirected to manage higher concurrency, reducing the strain on system resources. This becomes increasingly important in a world where concerns are growing about the sheer computational power required to sustain AI advancements. There are fears that the energy demands of future AI systems will outpace available resources within a few years.
While OLAP pre-aggregations won’t single-handedly solve this energy challenge, they represent a small but important step toward more efficient data processing, helping to reduce the overall computational burden in high-demand environments.
Bridging Structured and Unstructured Data
Enterprises must deal with both structured data (e.g., transactional databases) and unstructured data (e.g., text, images). OLAP’s ability to integrate with modern data platforms and leverage large language models (LLMs) facilitates the incorporation of unstructured data into the analytical process. This integration allows for a more comprehensive view of the enterprise, combining structured insights with the richness of unstructured data.
The Future of OLAP in Enterprise Intelligence
The resurgence of OLAP in the context of enterprise intelligence underscores its enduring value in the data analytics landscape. As enterprises continue to generate and consume data at unprecedented rates, the need for efficient, scalable, and responsive analytical tools will only grow. By leveraging the advanced OLAP capabilities of Kyvos, businesses can ensure that they are well-equipped to navigate the complexities of modern data environments and harness the full potential of their data.
OLAP isn’t a relic of the past but a critical component of the future of enterprise intelligence. Its ability to deliver fast, scalable, and sophisticated data analysis makes it an indispensable tool for any organization looking to thrive in today’s data-driven world. As we continue to explore new frontiers in BI and AI, the role of OLAP will remain central to unlocking the insights that drive innovation and success.
Kyvos isn’t sitting still on just being a DW accelerator with pre-aggregations. Kyvos has recently implemented AI-related features of several varieties. And it has taken measures to take the role of a bonafide semantic layer, which is great for a customer-facing layer.
OLAP is a form of caching. Caching is the preservation of compute, whether it’s web pages or database query plan. It’s always what we do when resources are constrained, but not when we’re in the land of milk and honey. Compute takes resources whether that’s GPUs, electricity, network bandwidth, or just time (which we know is money). So at this time when discussion of where we’re going to get the electricity for the massive data centers to power AI, and the sudden increase of data thanks to AI (and IoT), looking to Kyvos’ GenAI-Powered Semantic Layer promises analytics scalability.
Soft Coded Logic 