The Art of Crafting Competency Questions for Knowledge Graphs

Applying self-service BI concepts to knowledge graph development.

This blog is from the “deleted scenes” of my book, Semantic Webs of Meaning, Technics Publications, 2026. It is also a continuation of the competency questions discussion begun in Where Should an Enterprise Knowledge Graph Start?

Competency Questions: TL;DR

Competency questions …

• Are natural-language questions that a knowledge graph or ontology is expected to answer. They guide modeling by revealing which concepts, relationships, rules, and supporting facts the graph must represent. They can later be translated into queries and used to verify that the graph actually supports the intended reasoning.
• Are a test of competence, not the competence itself. It does not supply the knowledge or prescribe the graph’s structure. Instead, it provides a concrete way to determine whether the required knowledge and connections are present. A question such as “Why are SQL Server queries taking a long time?” may expose the need to connect symptoms, workloads, configurations, mechanisms, possible causes, evidence, and remedies.
• Should not be treated as a fixed and exhaustive requirements list. Answering one question often produces further questions, reveals missing knowledge, or crosses into domains that were not considered initially. The first set provides a manageable starting point, while the graph should remain capable of expanding toward important questions nobody knew to ask beforehand.
• Can be approached through the familiar idea of self-service BI and should be conceptually familiar to my core BI audience. We build a semantic layer that lets users explore combinations of dimensions and measures without specifying every report or question beforehand. Apply that same thinking to a KG: build connected knowledge that people and reasoning tools can explore through relationships, meanings, and rules. Competency questions guide and test that structure, while leaving room for inquiries beyond the initial requests.
• Have a longer history than their current visibility suggests. Michael Grüninger and Mark S. Fox introduced the term in work dating to 1994, published in 1995 as The Role of Competency Questions in Enterprise Engineering. Competency questions may sound new to many people because ontology engineering remained a specialized field for decades. As knowledge graphs rapidly move into broader enterprise use, this seminal idea is reaching a much wider audience.
• Involve an art of specification. We may recognize the kind of competence we want before we can fully express what it entails. The art is moving a step of levels from that broad expectation toward increasingly concrete (less abstract) questions that can guide modeling and testing, without mistaking those questions for the full extent of the competence.
• Aren’t exotic. Most instructions we’re handed aren’t exact concrete steps—otherwise we can build a machine to do it. Think of it this way, all of us are consultants to some extent (which I discuss in the Premise of Semantic Webs of Meaning). When our manager asks us to do something, she rarely tells us step by step how to do it. How cumbersome that would be for both of you. The requested result may be specific and novel to varying degrees, but exactly how to arrive at it is up to you—within the unique contexts we find ourselves immersed within.

Bottom line: If you’ve worked on BI systems, you’re familiar with building products that are expected to answer questions you didn’t think to ask (self-service BI). If you’ve worked in Performance Management, you’re familiar with hierarchies of abstract to concrete requirements (goals, objectives, and tasks).

Competency Questions: Purposefully Abstract Requirements

Competency Questions are the requirements for building a structure that facilitates the nebulous capability of answering questions we haven’t thought about—within a reasonably defined context and scope. Competency questions therefore tend to begin with open scope and may initially be vague, ambiguous, nebulous. The art is to iteratively work downward from a motivating scenario toward questions concrete enough to guide modeling and testing without mistaking those concrete questions for the full competence we sought.

That’s somewhat counter-intuitive of what we think of as requirements. Until the level of AI we’re engaging with at this time came along, outside of the art world, virtually everything we’ve built is built to blueprint-level specifications and a defined purpose. That includes software systems that support business processes, machines that go onto an assembly line, buildings housing people and businesses, and any infrastructure (roads, dams, power plants, etc.).

However, there are a few exceptions. For a few decades, before ChatGPT (Nov 2022), there was/is business intelligence (BI). A few decades ago (ca. late 1990s through mid 2000s), it too wasn’t like other software projects. Its requirements were about the nebulous job of assisting in decision support, not fulfilling tasks. It was about building a structure (an OLAP cube) to retrieve values in a robust but simple, user-friendly, and highly-performant manner.

But the real differentiation is downstream from the OLAP cube— what emerged as the notion of self-service BI—placing exploration directly in the hands of business users. They could slice, filter, dice, combine, and visualize data as questions emerged, without turning every new question into another report project. Instead of a business analyst or manager asking an IT resource to build a report for yet another need—and it may take hours to days to weeks—that analyst could fulfill at least a fair chunk of those requests on her own in seconds to minutes.

The deeper point is not that self-service BI teaches someone how to produce the same report. It gives them an environment for continuing the investigation in an independent and highly-responsive manner. To paraphrase great wisdom:

Write a report for an analyst, and you answer the question they asked. Give them self-service BI built on a well-designed semantic layer, and you give them the means to follow that answer into the questions it raises—including questions nobody knew to specify beforehand.

We never listed every report. We built a semantic layer capable of answering countless such questions, so the next question didn’t become another project.

So it is for the AI tools we’re building:

Answer a question for a SME, you’ve answered one question. Give a SME an AI tool and the SME can receive answers to countless questions within the tool’s scope.

We certainly wouldn’t test an LLM by specifying every question it must be able to answer. We can only sample its competence with a finite set of questions and judge from those samples whether it has the broader capability we expect. The same logic applies to self-service BI and KGs. The questions we test are evidence that the underlying structure is capable of answering a wider variety of questions, including some we didn’t know to ask when we built it.

Today, AI introduces reasoning capability into the things we build. If we know exactly what we want to do, the best thing is to engineer traditional software or a reliable machine, as we’ve always done. If we’re conceptualizing a path, trying to understand a phenomenon, etc., we reason about those things we haven’t yet encountered and solved.

This blog is about differentiating the nature of requirements for traditional projects versus requirements for the new class of applied AI projects—specifically, BI, knowledge graphs (KG), and LLM-based AI applications.

I should clarify:

• I consider KGs and LLMs to be components of AI. I describe this extensively in Chapter 5 of Semantic Webs of Meaning, as well as in the Assemblage of Artificial Intelligence. This blog is targeted at the KG audience. So please understand I’m talking about KGs (and BI) in that context.
• By “LLM”, I’m referring to more than the LLM model itself. Rather, the system in which the LLM plays a substantial role within an AI application, such as RAG or the chat framework of ChatGPT or Grok.

Grüninger and Fox‘s Original Work

In ontology engineering, Michael Grüninger and Mark S. Fox introduced competency questions (CQs) in 1995 as a foundational method for determining the scope and requirements of an information model. For my core BI audience, that’s like the KG counterpart to Bill Inmon and Ralph Kimball. Essentially, competency questions are a set of natural language questions that a knowledge base or ontology must be capable of answering based on its underlying data and logic. They serve as both the initial requirements checklist and the final evaluation criteria for the model.

Grüninger and Fox developed CQs as part of their formal Tove (Toronto Virtual Enterprise) methodology for ontology design. They approached ontology building not as a purely philosophical exercise, but as a practical software engineering problem.

Their process follows a structured pipeline:

• Identify Scenarios: Engineers look at informal descriptions of a business or domain problem (e.g., “We need to track why a project is delayed”).
• Extract Questions: From these scenarios, they extract informal questions that a user would ask the system (e.g., “What tasks are on the critical path?”).
• Formalize into Axioms: These natural language questions are then translated into formal logic (like first-order logic or SPARQL today) to ensure the ontology’s vocabulary can express the concepts.
• Determine Completeness: The questions define the minimum vocabulary needed. If the ontology can answer all CQs, it is considered complete and fit for its intended purpose.

However, enterprises were kind of uninterested in knowledge graphs since then. There are many reasons for that. But the emergence of LLMs (ChatGPT November 2022) changed all that—through a symbiotic relationship where KGs ground LLM in reality and LLMs help ontologists build and maintain KGs.

The Contemporary Role of CQs in Today’s Knowledge Graphs

EKGs and semantic technologies have shot up to higher prominence post 2022 in a world different from 1995. Competency questions have evolved from design guidance for things most people never heard of (knowledge graphs and ontology) in 1995 into agile, user-centric tools.

While the fundamental concept remains the same, the contemporary application of CQs has shifted in several distinct ways:

Feature	Classic Grüninger & Fox Approach (1995)	Contemporary Knowledge Graph Approach (2020s)
Logic & Formalism	Rooted deeply in rigorous First-Order Logic and strict axiom validation.	Focused on Graph Queries (SPARQL/Cypher), SHACL shapes, and graph analytics.
Development Style	Heavy, upfront, top-down design before building the system.	Agile and iterative. CQs act as the “user stories” for semantic web sprints.
Automation	Questions manually translated to code by specialized knowledge engineers.	Driven by Large Language Models (LLMs) that can auto-generate CQs or convert them directly to graph queries.
Integration	Used primarily for self-contained, isolated ontologies. Integration wasn’t that big a deal in 1995, although data warehousing was on its way up.	Used to define data fabric boundaries and map disparate, federated data sources.

Today, instead of just testing if an ontology is logically sound, CQs are used to bridge the gap between business stakeholders and ontology engineers. They ensure that complex graphs are built to effectively aid in achieving business goals rather than just accumulating data for its own sake.

In their foundational 1995 paper, “Methodology for the Design and Evaluation of Ontologies”, they laid out a highly structured, multi-tiered process. It’s important to understand that competency questions are not a flattened monolith of checked boxes. They exist at different layers of abstraction, moving from real-world business headaches down to specific requirements.

Here is how Grüninger and Fox defined those tiers in 1995.

1. The Motivating Scenario (The Real-World Root)

Everything begins with a motivating scenario. This is a detailed, narrative description of a specific problem or story from the business or domain.

• What it looks like: A story about a factory floor. “Machine A broke down, which delayed Order X, and the manager needs to know who is qualified to fix it and what parts are missing.”
• Purpose: It provides the real-world context, justification, and scope for the ontology. It ensures engineers aren’t building a model in a vacuum.

Note that all projects, traditional and knowledge-based (BI, machine learning, KG), have motivating scenarios. Something instigated the move.

2. Informal Competency Questions (The Conceptual Layer)

From that narrative story, we extract informal competency questions. These are written in plain, natural language.

• What they look like:
  • High-level/Abstract: “What are the components of a given product?”
  • Low-level/Specific: “Which tasks are currently delayed because of a missing part?”
• Purpose: This is where the abstraction layers happen. Engineers group these questions. A high-level question (e.g., “How is a project progressing?”) is systematically broken down into more specific, concrete sub-questions (e.g., “What is the start date of task X?”, “Is task X dependent on task Y?”).

3. Formal Competency Questions (The Logic Layer)

This is the final step in their methodology. Once the informal questions are stable, they are translated into formal competency questions using first-order logic (or today, SPARQL/Cypher).

• What it looks like: Converting “Is Task X delayed?” into a predicate logic expression or a graph query that evaluates to True/False or returns a specific set of nodes.
• Purpose: If the formal ontology vocabulary cannot express these formal questions, or if the system cannot logically derive the answer, it proves the ontology is incomplete.

However, since 1995, business intelligence, performance management, data science and machine learning have dominated the “analytics/knowledge” realm. Knowledge graphs are coming out of that dormancy into a world that has a different shape. This multi-level abstraction is treated more like an Agile Software Pipeline:

1. Motivating Scenario → Epic / User Journey (The big business goal).
2. Informal Competency Questions → User Stories & Acceptance Criteria (The abstract and granular questions a business user wants to ask the graph).
3. Formal Competency Questions → Integration Tests / SPARQL Queries (The code that proves the graph data model actually works).

By separating the informal (abstract thoughts of Tiers 1 and 2) from the specific items (Tier 3), Grüninger and Fox created a system where business experts can dictate the requirements without needing to understand the underlying graph structure. Instead of motivational scenario, conceptual questions, and encoding logical rules, we still have motivational scenario but instead abstract questions the KG is expected to answer, and specific queries used as tests.

Please note that this is described as the “The Logic Layer”. What this means for this blog is that when I refer to “developing the KG”, this includes not only the ontology (Eugene is a Programmer, a Programmer knows programming languages, C# is a programming language, etc.) but also the logic rules related to or incorporated into the graph—particularly, SHACL and SWRL.

Example 1: Dental Practice Management Firm

Before continuing, let’s look at example of how this is applied in a contemporary KG project. We’ll use a topic that I was deeply involved long ago—a dental practice management firm (DPM), which purchases dental practices and centrally manages them.

Motivating Scenario (The Business Context)

After acquiring a set of dental practices in a region that is new to the DPM, it seems new factors were introduced that weakens their optimization tactics. They need to understand why for future acquisitions. Executive leadership needs to optimize operations, ensure compliance, and maximize profitability across all clinics. They want to identify which clinics are underperforming, why they are underperforming (ex. staff shortages, high overhead, low patient retention), and if dentists are practicing safely and efficiently.

Note that all of the terms are not strictly defined. What constitutes staff shortages or high overhead? Rules and constraints need to be developed.

Tier 1 (Abstract Semantic Definition)

• The Question: “Which practices are at risk of an operational bottleneck?”
• Why it’s abstract, not just “high granularity”: The graph does not have a column called is_bottlenecked = True. Instead, “bottleneck” is an abstract concept defined by a set of logical relationships that can mutate.
• The KG Power: You might define a bottleneck today as “high chair utilization + low staffing.” But tomorrow, a new regulation passes or a new supply chain issue happens. Because Tier 1 is defined as an abstract concept rule in the ontology, the graph can flag a bottleneck caused by a pattern you never explicitly hardcoded or planned for.

Tier 2: (Contextual & Relational Traversal)

Instead of Tier 2 being “average wait time per clinic” (which is just BI granularity), a KG Tier 2 question looks at transitive relationships and hidden paths:

• The Question: “Are there unknown relationships between clinical anomalies to our supply chain?”
• The Unexpected Answer: You didn’t write a query to look for this. But because the graph links Dentist→Uses→Material Brand→Supplied By→Vendor, the graph’s inference engine automatically flags that Clinic A and Clinic C are suffering from an unusual drop in patient satisfaction scores. Why? Because both clinics happen to share a specific batch of dental composite material from a sub-vendor that is currently facing a recall.

Tier 3: Low-Level Concrete Competency Questions (The Data Mapping Layer)

These are the highly specific questions that novices mistake for the entire CQ process. They deal with raw data entities, attributes, and explicit relationships. These map 1:1 to graph queries (like SPARQL or Cypher).

• Deconstruction of CQ 1.2 (Chair Utilization Data):
  • CQ 1.2.1: What is the total operating hours for Clinic X’s chairs this month?
  • CQ 1.2.2: How many unique appointments were checked into Clinic X’s chairs during those hours?
  • CQ 1.2.3: What is the average duration of a standard cleaning appointment versus a crown placement?
• Deconstruction of CQ 3.2 (Credentialing Data):
  • CQ 3.2.1: What is the expiration date of Dentist Y’s state dental license?
  • CQ 3.2.2: What procedure codes (CDT codes) did Dentist Y log in the last 30 days?
  • CQ 3.2.3: Does Dentist Y have a recorded HAS_PERMIT relationship to the specific sedation procedure code they performed?

Why This Hierarchy Prevents Knowledge Graph Failure

If you skip Tier 1 and Tier 2, you will just build a graph that links Dentist to Appointment to Invoice. That’s already handled by a traditional BI implementation.

By maintaining this hierarchy:

• The domain experts can look at Tier 1 and say: “Yes, if the graph can tell me which practices are meeting benchmarks, I can run this firm.”
• The developers can look at Tier 3 and say: “Yes, I know exactly which data fields from our practice management software (like Dentrix or Open Dental) need to ingest into the graph to make this happen.”

The Black Box Consultant Function

When I engage an attorney, I want someone who can help me navigate a subject I don’t understand well enough to handle myself. I have some idea of my problem, maybe even know a bit about it—enough to be dangerous. But I expect the attorney to recognize things I missed, don’t really understand as well as I think, ask questions I wouldn’t think to ask, and explain what my options mean and the trade-offs. The same goes for a doctor. I want someone with a good reputation who can advise me on health issues, including things I might not recognize as issues.

I don’t arrive at my meeting with a comprehensive checklist of the knowledge and reasoning I expect them to possess. How would I know what belongs on that checklist? I’m engaging them for their ability to gather salient points, diagnose my problem, and engineer a treatment plan.

In goes my problem and properties and out pops a solution. The attorney is a function of law, the doctor is a function of healthcare.

An expertly crafted “recommended course of action” (attorney) or “treatment plan” (doctor) is presented to me as a checklist—step by step instructions in a linear form that’s easy for me to execute upon. It’s crafted from the experts’ perspective of understanding how the items relate and understanding and acceptance of the trade-offs. It is composed with consideration for my unique context involving the current state of the world, the particular players, the events leading up to my issue, available resources. It is custom-tailored for me and this situation.

Note: In reality, the plan is often a more robust decision tree, more complicated than a checklist. But a decision tree is still too rigid of a structure. For simplicity, I’ll stick to checklist for this blog.

Paths vs. Patterns

We routinely express what we want from consultants in the form of a generalized pattern.

A consultant is someone who can ease my workload by filling in a gap or two in the required stack of skills for a project or enterprise. We have an idea of what may be entailed, but we don’t know exactly how it will be done. We can interview candidates and test them with particular situations. Those situations provide evidence of something much broader.

However, they are a test of competence, not the competence itself. Most candidates have been around the block a few times and know the routine test questions—or know how to search the web for them. The test questions are a path.

I think this distinction deserves more attention when developing knowledge graphs. We are building something intended to support understanding and answer questions across a subject. Much of its value should emerge from questions we haven’t yet thought to ask.

For deductive reasoning, we can build a knowledge graph by linking specific paths of experience—like weaving threads into a fabric. The process is simple. We find the thread that most closely associates with a set of characteristics.

What is more robust is linking domains of knowledge into a graphical constellation, a system of relationships. Within that constellation of linked domains are answers to many, not all, questions we didn’t know could be asked.

Taken as a whole, a checklist (a flat structure) isn’t guaranteed to work for someone else or even for me in a future novel situation. However, each item individually could still be applicable elsewhere in a “fool me once, shame on you; fool me twice, shame on me” kind of way.

During my younger days when I moved quite often (usually for work), I rented an apartment and quickly experienced an unforeseen gotcha. For example:

1. I first learned that if there are just two or three levels, take only the top-level apartment. Rookie advice.
2. Then I learned not to take the apartment next to the elevator. Still rookie advice.
3. Then I learned to not take an apartment above a parking stall that isn’t your parking stall. My checklist grew, but every rental got something new by me. OK, that one got me.
4. Then I learned to not take an apartment under an apartment that is leased by a big corporation for a group of summer interns. OK, “caught me looking” again.
5. Then I learned that the garbage dumpster can be hidden behind a pretty wall, so you don’t know you’re close to it.

I did find one good apartment from about a dozen moves. So, I love when people are interviewed and asked why they chose some product, vacation destination, job, etc., and they say, “… it checked all the boxes …” Hahaha!

In the context of knowledge graphs (KG), performing a deductive reasoning query is mostly a matter of path finding—this leads to that, and that leads to that—linear and specific, like a checklist. But developing KGs is more a matter of generalized pattern matching—linking related entities then linking related domains. Its structure is sophisticated webs of relationships, generalized in scope, operating beyond the realm of data/information/knowledge, to the edges of the realm of understanding.

Semantic Layers and Knowledge Graphs

Both semantic layers and KGs are about discovery:

• Semantic layers, with roots in BI and OLAP cubes, are about discovery of information from within multi-dimensional aggregations of millions to trillions of facts, sliced and diced by dozens to hundreds of facets (attributes, properties).
• KGs are about discovery of information, knowledge and even a level of understanding hidden in webs of relationships.

KGs support the deductive reasoning efforts of:

• People. Humans reason over a KG by reading explicit classes, relations, and constraints—following links, asking competency questions, and checking whether a conclusion follows from shared definitions rather than from memory or tribal knowledge. The graph is the shared whiteboard: it makes premises inspectable, disagreements localizable, and deductions explainable in domain language.
• SWRL/OWL reasoners (ex. Pellet and HermiT, and Prolog processors such as SWI-Prolog). These are the formal engines. OWL reasoners classify individuals, check consistency, and compute entailed subclass and property relationships from TBox axioms; SWRL adds Horn-style if–then rules over that ontology. Prolog does the same job with unification, recursion, and backtracking over facts drawn from (or mirrored in) the graph. The KG supplies the premises; the reasoner returns only what those premises entail—deterministic, auditable, and independent of wording.
• LLMs. An LLM is a statistical analogical reasoner, not a theorem prover. A KG supports it by grounding retrieval (GraphRAG / SPARQL context), constraining generation to named entities and allowed relations, and handing hard entailments off to OWL/SWRL or Prolog. The model proposes; the graph and the symbolic reasoner verify. That split keeps fluency without treating probability as proof.

Prior to the LLM-driven era of AI we currently “enjoy”, particularly for BI/OLAP, the first item, People, were the only practical reasoning option. KG reasoning did exist since the early 2000s, but building and maintaining robust KGs was much too difficult for many reasons. BI/OLAP supported the reasoning of people by supplying highly computed, highly-curated values, at the “speed of thought”, that plug into the “models” in their heads.

Today SWRL/OWL reasoners are as good as the KG they reason over. LLMs are actually pretty good reasoners (and are continually improving), but they are still prone to the same kinds of errors people make. Even a 200 IQ person and even my YouTube hero, the average Redditor make reasoning mistakes all the time. Why? Because we never step into the same river twice (Heraclitus) and information is never complete, never fully current—our world is an unforgivingly complex system.

Now, returning to the great wisdom we paraphrased above, in regard to KGs, we can say:

Give a reasoner an answer, you’ve provided an answer to one question. Give a reasoner a KG and it can deduce answers to countless puzzles within its competence.

This is a good place to reiterate that the KG is a knowledge base for a reasoner to have the opportunity to deduce responses to queries that weren’t anticipated.

Requirements Aspects

The following aspects compare how we define, build, and evaluate each kind of capability from the points of view of building traditional software, BI system, EKG, and even an LLM. All four remain necessary, but what we expect from them changes how we approach their requirements.

Table 1 compares requirement aspects (Column 1):

• What we are asking for. The fundamental capability we want to obtain. We may need something that performs specified functions, provides figures across a business subject, connects explicit knowledge, or applies broad knowledge to an unfamiliar situation.
• What we know beforehand. How much we can reasonably specify before development begins. We may understand the required behavior in detail, or know the subject and desired competence while expecting further questions and needs to emerge through use.
• How requirements are expressed. The form we use to describe what success means. This may be detailed specifications, the dimensions and measures available for exploration, a hierarchy of competency questions, or expectations for the assistance a system should provide.
• How it accommodates an unforeseen need. Whether an unexpected request can be addressed with existing capabilities or requires additional development. A new need might fit within an established structure, require new knowledge, or expose limitations imposed by earlier design choices.
• What we can directly govern. The parts of the system we can explicitly define, inspect, and change. This includes behavior, calculations, meanings, relationships, rules, and the information available when producing an answer. The degree of control affects how we establish trust and correct problems.
• Why we can’t boil the ocean. The constraints that force us to choose what to include and how deeply to develop it. Resources, scope, time, data volume, integration effort, and the work of maintaining meaning all shape the boundaries of the capability we can provide.
• What testing demonstrates. The evidence that selected behaviors work and that the intended competence is present. For systems intended to support exploration, the tested cases represent only part of what users should be able to do. They are a test of competence, not the competence itself.
• Why we still need it. The contribution each makes alongside the others. Operational functionality, governed figures, explicit connected meaning, and flexible interpretation serve different needs. Their combination expands what we can accomplish; none automatically supersedes the others.

Aspect	Traditional projects: software, buildings, bridges	Self-service BI	Knowledge graphs	LLMs
What we are asking for	Build something that performs specified functions under defined conditions.	Make figures available across a business subject so we can investigate it ourselves.	Organize and connect knowledge so we can inquire into a subject and its relationships to the wider world.	Bring broad knowledge and flexible interpretation to situations we describe.
What we know beforehand	We develop a sufficiently detailed account of what must be built and how it must behave.	We know the subject areas and establish dimensions, measures, grain, and relationships. We don’t know every tuple users will request.	We know the subject and the competence we want. Discovering the knowledge and connections needed is part of the work.	We know the kind of assistance we want, often without knowing what knowledge or approach will resolve the situation.
How requirements are expressed	Functional and technical specifications, constraints, and acceptance criteria.	The scope and structure of an analytical space users can explore.	A hierarchy of competency questions, beginning with broad capabilities and becoming more specific.	Broad model capabilities, with more specific instructions, constraints, and success criteria for particular applications.
How it accommodates an unforeseen need	Through existing flexibility or a change to the design and implementation. Earlier choices affect the cost.	A new combination of existing dimensions and measures may already answer it. New subjects or incompatible grain may require development.	Existing knowledge may support a new query, inference, or path of investigation. Missing knowledge or distinctions require extending the graph.	It can attempt an interpretation using learned knowledge and supplied context, even when no dedicated solution was implemented.
What we can directly govern	The design, implementation, and specified behavior.	Data transformations, definitions, calculations, and access.	Explicit identities, meanings, relationships, rules, sources, and access.	Instructions, supplied context, tools, and surrounding processes; control over learned knowledge and generated responses is less direct.
Why we can’t boil the ocean	Resources, scope, time, and engineering constraints limit what we build.	Integration, transformations, reconciliation, and massive data volumes limit coverage.	Discovering, agreeing on, connecting, and maintaining meaning limits coverage.	Training and operating resources are finite; broad familiarity does not guarantee accurate or current knowledge of a particular situation.
What testing demonstrates	Whether the delivered thing meets specified requirements under tested conditions.	Whether selected calculations and analytical operations work correctly.	Whether selected inquiries and reasoning tasks provide evidence of the intended competence.	How reliably it performs across sampled tasks and conditions.
Why we still need it	It supplies the operational systems and structures on which we depend.	It supplies governed figures quickly and consistently.	It supplies explicit, connected meaning that can be inspected and reasoned over.	It supplies flexible interpretation and synthesis across a breadth difficult for people to hold in mind.

Consider a company investigating declining sales. Its operational software continues to process orders, payments, shipments, and returns. Self-service BI establishes the figures: when the decline began, which products and regions are affected, which customer segments are disappearing, and whether discounts, returns, or inventory changed at the same time. Those figures give analysts dependable points from which to investigate.

The KG extends the inquiry by connecting those customers, products, regions, suppliers, campaigns, organizational responsibilities, and relevant knowledge outside the operational schemas. It might reveal that the affected products share a supplier, that disappearing customers are concentrated near newly opened competitors, or that responsibility for related signals is divided among departments.

An LLM can help interpret the figures and relationships, formulate possible explanations, and propose further questions. Each capability contributes something the others do not automatically provide: the software operates the business, BI establishes the figures, the KG supplies explicit connected meaning, and the LLM helps reason across the resulting context. Testing particular cases gives us evidence that these capabilities work, but each case remains a test of competence, not the competence itself.

The Amorphous Requirements of Business Intelligence

That sounds familiar to my fellow BI architect/engineers who have built self-service BI systems. A good BI architect interviews stakeholders, SMEs, workers, and takes a lot of notes. Within the jumble of notes are mostly specific articulated issues. But those specific issues are not the whole requirement of the BI system. The real requirements are at a higher level of abstraction. The individual requests might be used as test scripts, but the requirements are not limited to them.

When we build traditional software (ex. a system to automate a workflow), a building, or a bridge, we are accustomed to taking requirements down to specific levels. It may take considerable time to figure out what is needed. That’s what the conceptual and design phases are for. Once we’ve fleshed out what we’re building, we can manage the familiar balance of the project management knobs of resources, scope, and time.

An exploratory knowledge capability complicates that process. We need enough definition to build and manage it, while preserving its ability to help with inquiries that haven’t yet occurred to us. In that respect, I think of knowledge graphs, and even BI, as being closer to human knowledge workers than the machines and structures we normally associate with construction projects.

Strictly speaking, the BI and KGs require a reasoning mechanism. That had been solely the realm of humans, but today, that is trickling down to AI.

Goals, Objectives, and Tasks from Performance Management as another Analogy

This is where I want to take the idea of competency questions. Questions about the competence we expect a KG to provide can begin broadly and become more specific. They can form a hierarchy, much like a scorecard from the Performance Management world, with goals, objectives, and tasks at the bottom.

Goals and objectives have a different nature from the tasks that contribute to them. We need to keep those differences in mind as we work down the hierarchy:

• Goals and objectives describe what we want to achieve.
• Competency questions identify what we must be able to answer to support those aims.
• Implementation tasks build the required capability.

That also distinguishes a question the finished system answers from work the development team performs.

Expectations for Adapting to Change

We have different expectations about how quickly a constructed thing must change and how quickly intelligence must reflect change.

We’ve already socialized the cost of modifying traditional systems and structures. People understand that an update takes money and time, that there are dependencies and restrictions, and that yesterday’s decisions constrain today’s options. Much of that work is retrofitting an existing thing to accommodate a newly recognized need.

With intelligence, the expectation is less tangible and more immediate. We expect advice to take account of what is happening now. We also use intelligence strategically: to recognize developing problems, consider possibilities, and prepare before a change becomes an urgent requirement. The maintenance itself helps us discover what needs to change.

Maintenance aspect	Traditional projects	Self-service BI	Knowledge graphs	LLMs
What must stay current	The constructed system’s ability to meet operational needs and constraints.	The figures, source coverage, measures, and dimensions available for investigation.	Knowledge of entities, relationships, rules, intentions, and relevant changes inside and outside the enterprise.	The information available to the model when interpreting a situation, alongside its broader learned capabilities.
Expectation of change	The time, expense, dependencies, and restrictions involved in modification are generally understood.	Users expect sufficiently fresh figures to recognize changes while they can still respond.	Users expect enterprise meaning and relationships to reflect the situation in which decisions are being made.	Users expect advice to incorporate recent developments and the specifics of the current situation.
Proactive purpose	Adaptations often follow a recognized need; planned maintenance also preserves service and prevents failures.	Reveal developing patterns and provide additional ways to investigate emerging concerns.	Add and connect knowledge that may expose dependencies, consequences, or possibilities we had not considered.	Help identify emerging issues, formulate hypotheses, and consider what may happen next.
How maintenance happens	Modify, retrofit, retest, and deploy within the constraints of the existing design.	Refresh data and extend analytical coverage; structural changes still require engineering.	Update assertions and their validity, connect new knowledge, and revise meanings or rules when needed.	Supply current context and retrieved information, update tools and instructions, and update the underlying model when necessary.

BI, KG, and LLM maintenance should support a short response cycle because delayed intelligence can lose its value. That doesn’t mean every change is technically easy. Refreshing figures differs from changing their grain; updating a relationship differs from revising an ontology; supplying current information differs from retraining a model.

This adds a temporal dimension to competency questions: Can the system maintain and extend its competence as the subject changes—and help us recognize changes we should prepare for?

That is a continuing obligation, well beyond passing the test scripts at rollout.

Self-Service BI

Consider what BI actually provides. At its core, it is a robust way to obtain figures very quickly. Of course, there is much more involved: making the data accessible in a user-friendly, integrated, conformed, and cleansed manner. A great deal of engineering goes into making a figure appear almost effortlessly on a screen.

But there is a distinction between asking IT for a figure and asking IT to provide the means to find figures ourselves.

If I know what I need, I can ask IT to create a report I can run whenever I need an updated answer. That report may have many parameters. It may call a stored procedure capable of returning a wide variety of results. It doesn’t have to be one fixed page with one fixed set of numbers. Nonetheless, IT has implemented the request I knew how to make.

Eventually, the IT person grows weary of needing to provide someone with reports “yesterday” and that person requesting the report becomes tired of the delays. The larger idea of self-service BI is something like this: “I’m not sure of all the values I need. But could you reformat the data so I can find them by specifying tuples and sets of tuples in a very robust way.”

A tuple identifies a particular combination of dimensions, such as a product, location, and time period. Sets of tuples let us compare, aggregate, and explore combinations. We know the subject areas we need to investigate. Those subject areas drive what goes into the cube. We establish the dimensions, measures, grain, and relationships so users can navigate a huge space of potential figures.

Nobody needs to list every tuple beforehand. That would defeat much of the purpose. Once the structure is in place, a user can ask for combinations that no developer individually implemented. One figure may prompt a comparison, which prompts a different grouping, which sends the investigation into another subject area.

The cube provides figures. How those figures fit into the situation remains largely the realm of human intelligence. The analyst brings experience, recognizes something odd, considers what else could matter, and decides where to look next. The cube can tell us how sales changed. Understanding the significance of that change may require knowledge well beyond what is in the cube.

Agile Project Management in BI

This is also why I’ve always thought BI fit reasonably well with agile development. The framework of the system (ETL pipeline, star/snowflake data warehouse, OLAP cubes) can be fairly stable. Much of the work consists of adding blocks: sources, domains, attributes, measures. Fittingly, like the term “cube”, pieces can fit together—just snap on another block, like we do with Legos.

That doesn’t mean every addition is easy. A different grain or an incompatible definition can require substantial work. But many future requests already have somewhere to fit within the model, even though the requests themselves weren’t known when it was designed.

My concern about agile is that choices made in earlier sprints can leave a system unable to readily accommodate a requirement that appears later. Thus, the phrase, “evolutionary dead end”. Welcoming change doesn’t undo those choices. Each sprint might successfully deliver its functionality while the accumulated system becomes increasingly difficult to move in an unforeseen direction. The relatively stable structure of a BI model helps within the range of questions that structure supports. Broader architectural intent still matters.

In fact, the rise of KGs itself could be thought of as an unexpected emergence that addresses a kind of brick wall a BI “platform” might hit as the world has become substantially more connected, integration between enterprises rose, and LLMs sped “speed of thought” from like the speed of a fastball pitch to that of a bullet. For an agile BI sprint series that has been merrily moving along for years on its block-like platform, that can toss in quite a monkey wrench into the upcoming sprint series.

The Boundaries of BI

Of course, we can’t boil the ocean with BI. There can be hundreds of data sources and dozens of domains in an enterprise, but only a small minority are fully onboarded into the BI system. That has always been a big problem. Integrating sources requires transformations, reconciliation of meanings, and decisions about detail. Then there is the massive volume of data itself.

Consequently, much of the widely integrated potential of BI cubes isn’t widely experienced. A person’s BI experience may center on a few sources, perhaps only one. A data product published by a domain in a data mesh may have that same domain as its biggest customer. The broader enterprise has access to it, but the people closest to the subject may remain its principal users.

That limit also makes the requirements more manageable. We know which subject we’re dealing with and have some idea of the figures people will need. Even within a bounded subject, the major entities present a huge space of tuples to explore, with a reasonable likelihood of providing figures relevant to an investigation.

From Figures to Interpretation

LLMs introduce an ability to shift a portion of the interpretive work from human to machine. The ambition is to have that know-it-all friend who knows a lot about a lot of things, but one that can bring together more material than I or any person can keep in their head at once. Broad training supplies knowledge across subjects, while the information supplied during an interaction makes particular situations available for consideration.

At that broad level, we want the LLM to become smarter. Training data, model capacity, and processes built around the model are among the ways people pursue that ambition. But we don’t directly control its learned knowledge the way we control a cube’s definitions and calculations. Its broad familiarity with business doesn’t mean it knows what our enterprise means by a particular term, which rules we actually follow, or which apparent exceptions are intentional.

Where the Enterprise Knowledge Graph Fits

An enterprise knowledge graph has a place between those capabilities. In the architecture I have in mind, detailed observations and large volumes of facts remain in the BI systems. The KG represents and connects knowledge about the enterprise and the world in which it operates. Its meanings and relationships are more directly inspectable and controllable than the knowledge implicit in an LLM.

It can ground an LLM in enterprise reality and extend BI beyond the information held in its schemas. That extension won’t always produce another actual value from a cube. It may explain a relationship, identify a dependency, or reveal something outside the cube that deserves investigation. The LLM can help interpret and pursue those connections, with the KG supplying explicit knowledge and BI supplying the figures.

The scope I envision for an EKG is broader than the BI cubes it connects, while remaining far more bounded than the breadth sought by a general-purpose LLM. But we can’t boil the ocean with KGs either. We still need requirements that guide what belongs and how far we should go.

From Broad Competence to Concrete Tests

One overarching competency question could be: “Can you ground LLMs in the reality and specificity of our enterprise?”

Another could be: “Can you extend knowledge beyond our schemas to the rest of the world?”

These questions express substantial expectations. They also leave much to discover. Grounding an LLM in explicit enterprise knowledge may entail representing our terminology, identities, business rules, intentions, and differing perspectives across departments. Extending beyond our schemas may entail connecting our suppliers to their parent organizations, our facilities to their surroundings, or our products to knowledge about how people use them.

We don’t have to pursue all of those directions to equal depth. The competence we’re seeking helps us decide which ones matter. Procurement and product development may need very different portions of that wider world.

The hierarchy can continue downward. Under grounding an LLM, we might ask whether the KG can explain enterprise terminology in the context where it is being used. Beneath that, whether it can represent and relate the different meanings departments assign to the same term. Each level adds definition while remaining connected to the original purpose.

Eventually, we get to implementation work: identifying definitions, resolving identities, modeling relationships, and connecting assertions to their sources. We also develop test scripts. We might test whether the system can explain why Sales and Finance classify a particular customer differently.

That test is valuable. BI systems also use test scripts before rollout. We verify calculations, filters, aggregations, and combinations of dimensions. We need confidence that the machinery works before people depend on it.

But the test script doesn’t specify the full competence of the cube. We expect users to explore many combinations outside the scripts. Likewise, correctly explaining one customer classification provides evidence toward a KG’s competence. The broader expectation remains its ability to support understanding across the agreed subject.

Competency questions are used in ontology engineering to guide what an ontology must represent and what reasoning it should support. Grüninger and Fox progressed from motivating scenarios through informal competency questions toward formal questions used to characterize a necessary and sufficient axiom set. In BI terms, I think of the formal questions much like sample reports: concrete probes of a structure whose possible uses we cannot enumerate. Detailed questions can guide modeling and testing without becoming the full measure of what we intended to build.

Mapping Territory for Future Questions

In Semantic Webs of Meaning (page 215), I use Lewis and Clark as an example of a job with very ambiguous goals. It’s a project more of exploration and research, not so much tangible goals. The aspect of their mission that matters here is mapping what lay along the way to the Pacific. Jefferson’s instructions combined a broad aim of finding a practicable route for commerce with observations about geography, people, plants, animals, and resources. They specified kinds of knowledge to acquire while leaving much to discover for subsequent expeditions. See, Jefferson’s instructions to Meriwether Lewis

I think of the overarching question this way: “Can the product of your mission enable us to execute more specific objectives?”

Their findings could inform later journeys and decisions whose details hadn’t been worked out when they departed. A map gives those later efforts something to work with. Its value extends beyond tracing the expedition’s own path. For the KG analogy, that map includes the flora, fauna, people, resources, and passable paths that help us understand what is out there and how things relate.

Further, we don’t know what we don’t know. It takes subsequent information that sheds light on new doors we can open in pursuit of a more concrete answer—kind of Bayesian.

Building the Enterprise’s Map

We can similarly map an enterprise’s knowledge with some idea of the territory, the purpose, and the detail worth capturing. Existing schemas, code, and documents give us places to begin. They contain evidence of what the enterprise does, how it has organized itself, and which distinctions have already mattered. Connections beyond them expand the territory we can investigate.

LLMs capable of deciphering code, text, and schemas can help considerably with that work. They can propose concepts, identify candidate relationships, and bring possible inconsistencies to our attention. We still have to judge what those things mean and whether the proposed representation expresses our beliefs and intentions correctly. AI-assisted, not AI-generated! The responsibility for meaning stays with us, even when AI does some of the heavy lifting.

The art of competency questions involves moving between that broad purpose and the detail required to make progress. Too much breadth without boundaries leaves us trying to boil the ocean. Allowing the first few requested answers to define the entire system can leave us with something that has little room for further inquiry. We need to build enough knowledge of the subject, with enough connections, that users can continue asking questions as their understanding develops.

Self-service BI has already taught us something about this. We build and test a structure that makes a large space of figures accessible. We don’t require users to exhaust their curiosity and ability to anticipate before development begins. A KG can carry that idea into a space where relationships, meanings, rules, and connections to the rest of the world become available for inquiry as well.

When I engage an attorney, I expect their competence to remain valuable after they answer my first question. I want to bring that same expectation to a knowledge graph.

Example 2: Competency Questions for an Idaho Plant Rescue

The intent of Example 1 was to offer a preview of the ideas, just so you have an idea of what we’re talking. Example 2 is another after all that has been explained.

The fictional Idaho Plant Rescue (IPR) example I introduced in Machine Learning Models in Knowledge Graphs works particularly well because a plant can thrive at its destination and still be the wrong plant to put there. Survival, ecological compatibility, conservation value, and operational feasibility are different questions—and their answers cross domain boundaries.

The motivating scenario is that good intentions can result in disastrous outcomes. Conservation is full of actions taken for sound reasons that produce consequences beyond the problem they were intended to solve:

• Introducing one species to control another. A predator, parasite, or pathogen introduced to suppress an invasive species may attack native species, spread beyond the intended area, or become another invasive problem.
• Reintroducing a locally extirpated species. Restoring a species that once lived in an area may repair part of an ecosystem, but the habitat, food web, climate, and human use of the land may have changed during its absence.
• Relocating a threatened species. Moving organisms away from development, fire, drought, or habitat loss may save them immediately while introducing disease, genetic disruption, competition, or unwanted spread at the destination.
• Suppressing wildfires. Preventing fires protects people, property, and forests in the short term, but decades of suppression can accumulate fuel and interfere with ecosystems that depend on periodic fire, eventually contributing to more destructive fires.
• Removing predators to protect livestock or game animals. The immediate losses may decline, while prey populations increase and place greater pressure on vegetation, waterways, and other species.
• Planting trees to restore habitat or capture carbon. Large-scale planting can help, but poorly chosen species or monocultures may consume scarce water, reduce local biodiversity, introduce disease, or create a forest poorly suited to future conditions.
• Feeding struggling wildlife. Supplemental feeding may help animals survive a difficult season, but it can concentrate them unnaturally, spread disease, create dependency, attract predators, and alter migration behavior.

The species, sites, records, and policies below are fictional; they illustrate the knowledge structure rather than provide planting advice.

Suppose IPR relocates plants threatened by development and other land disturbances. It needs to connect knowledge across many complicated domains. For example:

• Botany: species identity, growth habits, reproduction, and propagation.
• Ecology: habitat requirements, relationships with other organisms, and potential for invasive spread.
• Geography and environment: climate, elevation, soil, water, and connections between habitats.
• Conservation: population status, provenance, genetic diversity, and conservation priorities.
• Operations: access permission, disturbance dates, volunteer skills, equipment, and nursery capacity.
• Monitoring: survival, reproduction, spread, maintenance, and the evidence behind earlier decisions.

Each domain contributes something that the others cannot supply alone. Knowing a species’ preferred soil does not tell us whether we can reach the property before construction begins. Having enough volunteers does not tell us whether moving the plant could create an ecological problem.

If you’ve architected and/or built a semantic layer for self-service BI, much of what follows should look familiar. We identify the subject areas, connect their concepts, and give users a structure they can explore without specifying every question beforehand. A BI semantic layer organizes access to underlying databases through shared dimensions, measures, and calculations.

Here, in the KG, we place more emphasis on the relationships themselves: how a plant depends on its environment, how a destination connects to a vulnerable habitat, or how a revised assessment affects an earlier decision. Those relationships extend the investigation beyond retrieving figures to examining how things fit together and what those connections imply.

Level 1: Open scope—the overall competence

How should we rescue and place threatened plants so that we preserve their conservation value without creating ecological problems we must solve later?

This question establishes the purpose of the KG. It leaves room to discover what matters: which plants deserve priority, what makes a destination suitable, which dependencies must survive the move, and what consequences might follow.

It also leaves room for an answer such as “do not relocate this plant” or “preserve it through propagation instead.” Rescue does not predetermine the intervention.

Level 2: Open scope—more focused lines of inquiry

These questions develop the overarching question without enumerating every species, destination, or situation.

Line of inquiry	Competency question	Knowledge that must connect
Rescue priority	Which threatened plants should receive our limited rescue resources, and why?	Conservation value, population representation, threat dates, rescue feasibility, and capacity
Destination suitability	Where could a rescued plant establish, and which conditions would make a seemingly suitable destination unsuitable?	Species requirements, local soil and climate, seasonal water availability, and maintenance commitments
Ecological consequences	Where could successful establishment become unwanted spread or harm to an existing community?	Reproductive traits, regional risk assessments, dispersal routes, neighboring habitats, and vulnerable species
Ecological dependencies	What relationships must be preserved or restored for the rescued population to persist?	Pollinators, host plants, soil associations, flowering periods, and surrounding land management
Rescue execution	Which interventions are feasible before the threat occurs, and what could prevent them?	Site permission, timing, plant condition, required skills, equipment, transport, and temporary holding facilities
Learning and reassessment	What new observations or changes should cause us to reconsider an earlier placement or rescue decision?	Decision history, monitoring results, changing site conditions, revised species identification, and updated evidence

Each question is narrower than the mission question, but still opens a space of investigation. “Where could successful establishment become unwanted spread?” can lead to questions about waterways, neighboring properties, seed movement, future irrigation, or a newly discovered reproductive trait.

Level 3: Concrete questions suitable for test cases

For testing, we supply particular records and explicit decision rules. These should be readily translated into SPARQL queries or more sophisticated reasoning queries. The expected answers below follow from those records and rules. They are not claims that the graph can guarantee a biological outcome.

Parent inquiry	Concrete competency question	Test facts and expected result
Rescue priority	Which of rescue lots A and B receives priority under our policy for preserving unrepresented populations?	Both lots face disturbance on October 10 and require equal resources. A’s source population is already represented in the collection; B’s is not. The policy prioritizes unrepresented populations when these other factors are equal. Expected: B, with the policy and collection records supporting the result.
Destination suitability	Which of sites North, South, and East meets all recorded placement requirements for species P?	P requires well-drained soil and first-season irrigation. North has both; South has poorly drained soil; East has suitable soil but no irrigation record. Expected: North meets the recorded requirements; South fails the drainage requirement; East remains unresolved.
Ecological consequences	Does site North pass our invasive-risk screening rule for species P?	North lies in Region R. An assessment marks P as high risk in R, and our rule excludes high-risk species from open-ground placement there. Expected: North fails the screening, even though it meets P’s growing requirements.
Ecological dependencies	Does site Meadow meet the recorded pollination requirement for establishing a self-sustaining population of species Q?	Q’s recorded requirement identifies pollinator group G during May and June. Meadow has documented G activity only in July and August. Expected: the available records do not establish the required seasonal overlap; further evidence is needed.
Rescue execution	Is the October 8 rescue of lot B ready to proceed?	The crew, transport, and nursery space are confirmed. Our policy also requires valid landowner permission, but the recorded permission expires October 7. Expected: the rescue is blocked pending renewed permission, with that dependency identified.
Learning and reassessment	Which placements require review after assessment E replaces assessment D?	E changes P’s risk classification in Region R from low to high. Placement records identify the species, destination region, and assessment used. Expected: identify the P placements in R that relied on D, preserving the earlier decision evidence.

Notice that site North passes one test and fails another. A result from the habitat domain cannot settle a question that also depends on ecological risk. Likewise, missing irrigation information is not evidence that irrigation is absent. The graph must preserve those distinctions.

Important takeaway: The tests do not define the limit of the capability. It’s just a smoke test (a quick, broad test that verifies a system’s essential functions work before more detailed testing begins).

Once these relationships are represented, someone might ask:

“Which plants already in our nursery have destinations that meet their growing requirements but now fail our ecological-risk policy, and how much holding capacity will we need while we investigate alternatives?”

Nobody had to specify that exact question when the graph was designed. It combines nursery inventory, species requirements, destination conditions, current risk assessments, policies, and capacity. The KG supplies the connections and explanations; an inventory query or BI calculation can supply the totals.

There is still a boundary: a new question may expose missing knowledge. The point is that not every new question should require a separately engineered answer.

The goal is responsible plant rescue. The objectives include preserving populations, finding suitable destinations, and preventing ecological harm. Competency questions express what we need to know to pursue those objectives. Tasks—surveying a site, obtaining permission, recording an assessment, or arranging transport—supply knowledge or carry out the resulting decisions.

Passing the concrete tests provides evidence that we have built some of the required competence. The broader expectation is that this connected knowledge remains valuable when the next rescue presents a combination we have never encountered.