Bridging Trust in AI Systems: The Importance of Explainability and Thinking Traces

OpenAI’s New O1 Model – The Quest for Explainability and Thinking Traces in

In the past 24 hours, we’ve posted extensively on the new OpenAI O1 / Strawberry / Q* model. We generally avoid using marketing-sounding phrases like “evolutionary” or “qualitatively different,” but this release does stand apart from everything that came before it in the last 12 months.

We’ve witnessed interesting and increasingly capable model updates and new architectures, like the Mixture of Experts (MOE) models from Mistral, but none have truly shifted the way the Large Language Models (LLMs) behave and perform compared to what was originally available in ChatGPT’s original interface.

Why Explainability of the O1 Model Matters

What’s particularly intriguing with the O1 model and the introduction of Strawberry planning functionality is how it brings up questions of observability and explainable AI. Specifically, when a model is performing extensive planning and thinking, being able to examine those thinking traces will be crucial for gaining business users’ trust.

The ability to see and understand how a model arrived at a decision is likely to be just as important as the decision itself, especially for high-stakes industries, such as legal and accounting, where accountability and transparency are non-negotiable.

Admittedly, at present, there are no requests for detailed insight into the matrix multiplication operations occurring while generating a response to your prompt. However, this is not where the issue lies.

Unfortunately, with the current release of O1-preview and -mini, OpenAI opted not to provide users with access to the thinking traces, explaining it as concern around allowing competition to gain access to valuable training data to aid in training of their models. I hope that, over time, OpenAI will find a way to make those thinking traces observable. If the concern is that competitors could exploit this data to get a leg up, then implementing safeguards like rate limiting could be a reasonable middle ground. Business leaders want to trust the systems they’re deploying, and making the thought process more visible is the correct direction to bridge that gap.

What Happens When Systems Scale Beyond What We See Today?

Stepping away from this particular product matter for a moment, which I’m sure OpenAI will address—and if not, their competition certainly will—the larger question is: what happens when systems like this scale beyond what we see today? Once the feasibility of advanced planning systems is proven, it’s only a matter of time before companies like Anthropic, Google and Meta release comparable features.

The real challenge comes when these new reasoning models can generate and explore not just hundreds, but thousands or millions of thinking paths simultaneously. Even if we’re given access to those thinking traces, parsing through that amount of data becomes an impractical exercise for business users who need actionable insights. This is the territory that some AI researchers have warned about—where even with full observability, we might still struggle to independently reason through the outputs produced by these systems.

Much of the comfort over the last few months has come from the concept of Co-intelligence championed by Ethan Mollick, i.e. the idea that humans work in collaboration with the AI as part of a decision-making loop. Business users feel that they are still driving the process, with AI serving as a sophisticated assistant (ahem, Copilot). But as AI systems evolve, exploring many more possibilities than a team of humans could reasonably keep up with, that sense of control and partnership will begin to erode.

This is where the higher-level abstractions and new tools to examine and interact with the model’s thinking process will be critical. Without them, the concept of Co-intelligence begins to feel threatened. Business leaders will need to see how these models can still operate within a framework that they understand and trust, especially as the systems themselves become more autonomous in their planning and decision-making.

Currently, we face limitations due to the scale, throughput, and soon, upcoming cost increases of the O1 model. However, the direction is clear. The next few months are crucial as OpenAI and competitors refine these systems. I hope they prioritize interpretability. I’m eager to see its impact on the business community’s large-scale AI adoption.

Takeaways

As we navigate this rapidly evolving landscape of AI capabilities, staying informed and adaptable is crucial for businesses. The potential of these advanced planning systems is immense, but so are the challenges they present.