For the First Time, AI Matches Human Experts in Language Analysis

The original version of this article was published in Quanta Magazine.

Among the countless capabilities that humans have, which are truly unique to our species? Language has consistently been a leading contender since the days of Aristotle, who asserted that humans are “the animal that has language.” Even as advanced language models like ChatGPT seemingly mimic everyday conversation, researchers are keen to identify whether there are particular elements of human language that lack equivalents in animal communication systems or artificial intelligence.

Specifically, scholars have been investigating the degree to which language models can reflect on language itself. Many in the field of linguistics believe that language models not only lack reasoning skills, they are unable to acquire them. This perspective was articulated by Noam Chomsky, a renowned linguist, and two collaborators in 2023, who claimed in The New York Times that “the proper explanations for language are intricate and cannot be learned merely by immersing oneself in vast datasets.” While AI models show proficiency in using language, these researchers argued, they fall short of analyzing language in a nuanced manner.

Gašper Beguš, a linguist at the University of California, Berkeley.

Photograph: Jami Smith

This perspective was recently contested in a study by Gašper Beguš, a linguist at the University of California, Berkeley; Maksymilian Dąbkowski, who recently earned his Ph.D. in linguistics at Berkeley; and Ryan Rhodes from Rutgers University. The researchers subjected several large language models (LLMs) to a variety of linguistic tests—one of which required the LLM to generalize the rules of an invented language. Although most LLMs struggled to parse linguistic rules similarly to humans, one demonstrated notably impressive capabilities, far surpassing expectations. It was able to analyze language in a manner akin to a linguistics graduate student—diagramming sentences, resolving ambiguities, and utilizing complex linguistic features like recursion. According to Beguš, this finding “challenges our understanding of what AI is capable of.”

Tom McCoy, a computational linguist at Yale University not involved in the research, labeled this new study as both timely and “very important.” He emphasized that as society becomes increasingly reliant on technology, it’s crucial to understand the strengths and weaknesses of these systems. He stated that linguistic analysis serves as an ideal testing ground for assessing the extent to which these language models can emulate human reasoning.

Endless Complexity

One of the obstacles in administering a thorough linguistic assessment to language models is ensuring they do not already possess the answers. Typically, these systems are trained using vast arrays of textual information—not only sourced from the internet across multiple languages but also including materials such as linguistics textbooks. Thus, in theory, the models could simply memorize and reproduce information encountered during their training.

To circumvent this issue, Beguš and his colleagues devised a four-part linguistic examination. Three of the parts required the model to interpret specially designed sentences using tree diagrams, originally introduced in Chomsky’s groundbreaking 1957 work, Syntactic Structures. These diagrams decompose sentences into noun phrases and verb phrases, further subdividing them into nouns, verbs, adjectives, adverbs, prepositions, conjunctions, and other elements.

One segment of the examination concentrated on recursion—the capacity to embed phrases within larger phrases. For instance, “The sky is blue” represents a straightforward English sentence. However, “Jane said that the sky is blue” embeds the original statement into a slightly more complex structure. Notably, this recursive embedding can continue indefinitely: “Maria wondered if Sam knew that Omar heard that Jane said that the sky is blue” remains a grammatically correct, albeit somewhat clumsy, recursive sentence.