Knowledge graphs 'overcome the shortcomings of large language models'

A data catalog built on a knowledge graph addresses three of the limitations associated with large language models (LLMs), according to the principal scientist at data.world, Juan Sequeda.

An LLM is a type of artificial intelligence (AI) program that can recognise and generate text, among other tasks. Despite their capabilities, LLMs present significant challenges when it comes to storing and retrieving facts because they function as statistical pattern-matching systems.

In an interview with Insurance Day, Sequeda says LLMs analyse vast quantities of data to generate statistically likely responses to prompts, but they do not store facts in the traditional sense. Instead, they produce the most probable response based on patterns detected during training. This can create difficulties when the inquiry is business-critical and nuanced, such as, “What is the amount of total loss claims this quarter?”

In contrast, knowledge graphs are a collection of inter­linked descriptions of entities that put data into context and enable data integration, analytics and sharing.

Sequeda says LLMs have three main shortcomings – a lack of accuracy, a lack of explainability and a lack of governance. A data catalog built on a knowledge graph overcomes all three of these, by creating a map of what data exists in an organisation and tying that data to meaning – semantics and context.

The transformation from rigid tables to a flexible graph structure allows for more complex relationships between data points, providing a “richer and more comprehensive” understanding of the data, people and processes that drive a business, Sequeda says. Once that data is mapped and connected, it can then be shared with the LLM.

Best of both worlds

The result, he says, is the best of both worlds – the language reasoning power of the LLM and more accurate, explainable and governed responses. Increased accuracy comes from sharing enterprise context, increased explainability comes from directly tracing the LLM responses to enterprise knowledge, and increased governance comes from limiting what LLMs can access.

Data.world, whose risk management customers include Clinician Nexus, Tokio Marine and CareOregon, is a cloud-native software-as-a-service (SaaS) data catalog platform that “combines an intuitive user experience with a powerful knowledge graph to deliver enhanced data discovery, agile data governance and actionable insights”.

Its knowledge graph architecture leverages web standards such as Resource Description Framework (RDF) to model data as a graph, Web Ontology Language (OWL) to represent the ontology, and R2RML, a language for expressing customised mappings from relational databases to RDF datasets. This makes it possible to transform traditional relational data into a flexible and interconnected format. Finally, SPARQL is the graph query language for RDF.

In short, data.world describes its model-and-catalog approach as a “foundational platform” for any organisation seeking to drive value with AI.

Origin of the benchmark

Sequeda says the seed of this idea was planted during the June 2023 summit organised by Snowflake, the data cloud provider. 

There were demonstrations of instructing LLMs to answer a structured query language (SQL) question, but with no indication of whether they could produce accurate responses to more complex questions.

Sequeda explains: “I’m like, ‘That’s a cute demo for a simple question on a database that has two or three tables, but what about an enterprise data warehouse that has thousands of tables and tens of thousands of columns?’ The product folks at Snowflake set a challenge: ‘Ok, why don’t you do a benchmark?’”

Data.world began research into whether knowledge graphs could increase the accuracy and explainability of answers LLMs provide in response to “enterprise questions over enterprise data” on SQL databases. It structured its benchmark into three parts.

The first was a “truly enterprise” schema, Sequeda says, using the property and casualty data model of the Object Management Group Standards Development Organization. Data.world avoided producing a “laundry list” of business questions and instead created a spectrum, from easy to hard.

The second part was creating a technical framework that turned this spectrum into a quadrant, with easy questions over complex ones. 

The third was creating semantics, using context for table columns so that, for example, it was clear that a “claim identifier number” was not the same as a “claim number”.

With these three parts, data.world now had a “benchmark”, Sequeda says, and turned to the “setup”, using Generative Pre-trained Transformer GPT4, the multimodal LLM created by OpenAI.

"Companies need to start treating their metadata and business semantics as first-class citizens"

^{Juan Sequeda}

“We had about 10 questions in each part of the quadrant, so 40 questions, and we manually looked at what would be the correct answer, and we matched it to the question. The research showed a three-times uptick in LLM response accuracy using a knowledge graph over only SQL database accuracy,” he says.

The “eureka” moment had come from the use of more than five tables, which showed that using SQL without a knowledge graph produced zero accurate answers.

“The more complicated we got by using more tables, the greater the risk the AI would hallucinate,” Sequeda says, using the term for when an AI model generates an incorrect or misleading result.

The main conclusion, he says, is that investing in knowledge graphs provides higher accuracy for LLM-powered, question-answer systems over SQL databases. Companies, including insurers, must use knowledge graphs for their adoption of AI to be successful, he adds.

The correct starting point for that journey is investment in the best fuel.

Sequeda concludes: “Companies need to start treating their metadata and business semantics as first-class citizens. If they don’t, then their vision of what they want to accomplish with AI will fail, and it will be back to basics for them – a manual spreadsheet.”

Insurance Day will be publishing a special report on market modernisation later this month