Researchers have discovered a method to extract the internal reasoning processes of large language models (LLMs) from major providers like OpenAI and Google. The technique uses carefully constructed prompts to trick proprietary, closed-source models into revealing their step-by-step “thoughts,” a valuable asset that is normally kept hidden. This represents a new type of intellectual property theft specific to the AI era, potentially allowing competitors to replicate the performance of frontier models at a fraction of the cost.
The findings come from a paper titled “Stealing Reasoning Traces from Proprietary LLM APIs”, which has drawn significant attention from developers and security researchers, including a lengthy discussion on Hacker News. The attack is notable for its simplicity; it requires no special access or computational power, relying only on clever use of the models' public APIs.
The Value of a Thought Process
Modern LLMs often solve complex problems using a technique called chain-of-thought (CoT) reasoning. Instead of jumping directly to an answer, the model breaks a problem down into a series of intermediate, logical steps. For a math problem, this might involve identifying variables, setting up equations, and solving each part sequentially. For a logic puzzle, it would mean evaluating premises and making deductions one by one. These intermediate steps are the model's “reasoning trace.”
This trace is incredibly valuable. While the final answer to a query is useful, the process of arriving at it is a form of high-quality, structured data. Companies that have spent billions of dollars on training leading models like GPT-4 and Gemini consider these reasoning capabilities a core part of their competitive advantage. They design their APIs to provide the final, polished answer while concealing the messy, intermediate work.
By training a new, smaller model on a dataset of these stolen reasoning traces—not just on questions and final answers—a competitor could teach its model how to think like a much larger, more capable one. This process, a form of model distillation, could drastically lower the barrier to entry for creating high-performance AI.
An Attack Through Simple Conversation
The researchers demonstrated two primary methods for extracting these hidden traces from models including OpenAI's GPT-4 and Google's Gemini 1.0 Pro. Both techniques work by disrupting the model's standard procedure for sanitizing its output before presenting it to the user.
First is the distractor question method. An attacker submits a complex prompt that requires multi-step reasoning, but appends a simple, unrelated question. For example, after a difficult logic puzzle, the prompt might add, “Also, what is the capital of Sweden?” According to the paper, the model generates the reasoning for the primary query, but the presence of the distractor question can interfere with its final output-formatting step. In its attempt to structure an answer for both queries, it may inadvertently leak the raw reasoning trace for the first.
Second is stylistic injection. This method involves asking the model to reply in a highly specific or unusual format, such as a poem, a JSON object with a predefined schema, or a different language. This constraint appears to tax the model's ability to simultaneously reason and adhere to the stylistic request. The researchers found that this can cause the model to “drop” its final summarization step, outputting the raw chain-of-thought process directly to the user as it attempts to satisfy the formatting constraint.
The core vulnerability is not a bug in the traditional sense but an emergent property of how these complex systems handle multi-part or highly constrained prompts. They are designed to be helpful and follow instructions, and the attack exploits this helpfulness to bypass the safeguards that hide internal state.
The Blueprint for a Cheaper Competitor
The implications of this research are primarily economic and strategic. The most direct threat is the theft of intellectual property for competitive gain. A well-funded startup or a rival lab could automate this process, sending millions of queries to a proprietary API to build a massive dataset of high-quality reasoning traces.
This dataset could then be used to fine-tune a powerful open-source model, such as one from the Llama or Mistral families. The resulting model would be significantly cheaper to produce than the original, as it would learn from the curated “thoughts” of a frontier model rather than requiring a full training run on a massive, unfiltered corpus of web data. This could erode the primary business model of companies that rely on the superior reasoning capabilities of their closed models to justify API costs.
This research exposes a security blind spot for AI providers. Until now, much of the industry's focus on API security has been on preventing prompt injection attacks designed to elicit harmful, biased, or dangerous content. The goal was to control the model's behavior. This new attack vector is about information leakage—not of user data, but of the model's own internal, proprietary processes. It redefines what must be secured.
What to Watch Next
This paper establishes a new front in the ongoing cat-and-mouse game of AI security. The immediate response will likely come from the API providers themselves. They may implement more robust output filters designed to detect and scrub reasoning traces before they reach the user. Another defense is to further fine-tune their models to recognize and resist these manipulative prompt structures, effectively teaching them to ignore attempts to reveal their internal state.
However, just as with traditional prompt injection, new variants of these extraction attacks are likely to emerge as quickly as old ones are patched. The fundamental tension is between making models more capable and instruction-following, and preventing those same qualities from being used to expose their inner workings.
For the broader AI industry, this research will likely spur more work into the fundamental security of model internals. It highlights that in the age of AI, the product is not just the answer, but the thinking that leads to it. The key thing to watch is how effectively the major AI labs can protect that thinking from being stolen through a simple conversation.