“ChatGPT can make mistakes. Check important info.” This caveat appears with every ChatGPT response, a red flag that should give all users pause. It is like walking into a hospital with a sign over the door that reads: “Doctors can make mistakes. Check your vital signs.” The allure of artificial intelligence, particularly generative AI, is that it has gained an almost mystical ability to speak the truth, couching its responses in natural language that immediately connects with the user and establishes trust. But remember, when you are interacting with a chatbot, you are providing inputs to an algorithm that performs pattern recognition across a large language model trained on a mishmash of topics, a jack of all trades but master of none. The output from such interactions should be treated no differently than the first draft of an internet search result that still requires your own judgment to verify.
Despite ongoing accuracy issues with generative AI, many enterprises are rushing to tap into perceived efficiency gains, with 78 percent of organizations having deployed AI in at least one department and more than 50 percent in two or more, while only 1 percent have achieved full AI maturity, according to McKinsey. IBM research shows that 67 percent of oil and gas executives want to reach full AI maturity to materially improve revenue and organizational resilience.
So, what is holding back full AI maturity? Over the course of this blog series, we’ll delve into the root causes preventing AI scale-up, the hidden risks, and the practical, actionable steps your team can take to unlock AI’s full potential. The goal is not to throw AI under the bus, but to acknowledge its transformative power for energy companies and dispel the assumptions that have created hidden risks and costs. By understanding these risks, your team will be better positioned to leverage AI for improved business performance and lower costs.
First, let’s talk about the elephant in the room: ungrounded content. By now, almost every chatbot user can share stories of prompt responses that were off, misleading, or completely fabricated. Also known as AI hallucinations, these issues stem from a design choice by OpenAI and other LLM developers to let their models narratively fill in gaps in training data with half-truths and guesswork instead of replying with “I don’t know.”
Ungrounded content is so pervasive and currently unsolved that Microsoft and others have dedicated entire initiatives to measuring it. Microsoft Azure’s AI Content Safety service attempts to detect and correct ungrounded content using AI, though this introduces a clear conflict of interest since the system relies on the same type of model to police its own errors.
Absent meaningful metrics from AI model developers, third parties are stepping in through independent research. The Hugging Face Vectara Hallucination Leaderboard objectively ranks leading AI models by asking each LLM to summarize long, multi-page records. Claude and Google models perform the worst, with error rates above 10 percent. Even best-in-class models like OpenAI’s o3-mini-high-reasoning, designed for STEM tasks, still show an 8 percent error rate. In other words, they produce 8 percent ungrounded content even when given a relatively small amount of information to summarize.
That is like giving a pop quiz to a classroom of the brightest students and seeing most earn a B while the Ph.D. students score an A-minus. It may sound acceptable, but this was a best-case scenario with no energy domain knowledge involved. In the oilfield, where operational safety and million- or even billion-dollar decisions depend on data accuracy, our industry cannot settle for passing grades.
There are two sides to the AI data accuracy problem. On one side is ungrounded content and on the other is energy data structure, quality, and availability. In the next blog, I’ll show why everyone is ready for AI except their data. Stay tuned.
Have questions? Please reach out. I’m happy to discuss how Pepper Energy Partners can help prepare for and accelerate AI across your organization.
