A Novel Method Unveils the Thinking Processes of AI Models

Recently, computer scientists uncovered a method to unveil the underlying “thinking” that advanced AI models engage in while resolving intricate problems.

The results yield some evidence—albeit not definitive proof—that specific Chinese models might have been trained by “distilling” reasoning insights from US models, as certain reasoning patterns appear closely aligned. The researchers also indicated that this method could potentially extract personal data, such as passwords and API keys, from an AI model’s internal reasoning, though this vulnerability has since been addressed.

“Every major frontier model provider we examined has this vulnerability,” states Alexander Panfilov, a computer scientist at the University of Tübingen in Germany involved in this research. “It poses risks of personal information leakage and opens doors to large-scale reasoning distillation attacks.”

Panfilov, alongside colleagues from the University of Tübingen, the Max Planck Institute, the AI safety organization MATS Research, and the security firm Snyk, identified the same problem in frontier models from OpenAI, Anthropic, and Google, which are accessed via an application programming interface (API).

In a paper detailing their findings, the researchers demonstrate that the open-weight Chinese model Kimi K3 from Moonshot AI produces remarkably similar outputs to the hidden reasoning traces—the documented reasoning steps used in problem-solving—of Claude Opus 4.8 and GPT 5.6 Sol for specific prompts. Despite these similarities, they caution that the findings “cannot causally establish distillation.” They also observed that two other open-weight models, China’s DeepSeek and Inkling from the US company Thinking Machines, did not show this reasoning similarity with Claude Opus.

Moonshot AI and Z.ai did not respond to requests for comment by the time of publication.

Distillation is a recognized and commonly utilized method for efficiently replicating the capabilities of existing models into new ones, particularly prevalent in the development of open-weight or fully downloadable models.

Recently, however, distillation has sparked controversy due to allegations that Chinese AI firms exploit it to effectively replicate the top US models. In February, OpenAI informed US lawmakers that DeepSeek appeared to have imitated one of its models to construct a reasoning model named R1. In June, Anthropic notified lawmakers that Alibaba had systematically distilled its models to develop its own, known as Qwen.

There is no evidence suggesting that Chinese AI companies employed this specific technique to distill US-based AI models. However, Panfilov and his team assert that their method would enable the extraction of more information from closed models than was previously understood.

Mini-Me Models

Sophisticated AI models tackle challenging problems by deconstructing them into smaller parts, which are analyzed sequentially in a form of artificial reasoning or “chain of thought.” Companies often keep a proprietary model’s reasoning private to prevent others from leveraging it to train new ones. However, they commonly send an encrypted version of that reasoning to a user’s computer, offloading some computational work.

The researchers’ attack exploits the fact that most AI companies offer related models of varying sizes. Larger models boast greater capabilities, but are also more resource-intensive to operate and pricier to access. Consequently, users may opt for smaller, less powerful models for specific tasks to reduce costs.

Panfilov and his colleagues discovered that feeding encrypted reasoning traces to a smaller version of the same model could expose the hidden reasoning within. The smaller models, having undergone less alignment training, are less likely to refuse to disclose their internal thoughts compared to their larger counterparts.