New paper: LLMs don’t do formal reasoning.

Well, of course.

LLMs don’t do informal reasoning, either.

Humans are great at pattern recognition. We even recognize faces in clouds and tree trunks. We recognize wheelbarrows, bears and crabs and archers in the stars. We make things that have meaning, and we communicate through speech and text and art.

LLMs are great pattern-generators. They are extremely well-tuned to make patterns that look like they might have meaning. A human trying to communicate may be bad at it, but they have an underlying model of the world that they are referencing and updating. An LLM is not trying to communicate anything. An LLM has a model of language, not a model of the world.

The map is not the territory. All models are wrong, but some are useful.

The situations in which it is reasonable to use an LLM are exactly the situations in which it is reasonable to roll some dice and use that to read the table of random monster encounters; to pull a card from an Oblique Strategies deck; to twirl the knobs on the synth and see if you can get a cool sound. In years past, you could type in a good list of keywords to Google and hit the I’m Feeling Lucky button.

Attempts to use LLMs for more than this fail. Often they do not fail in such an obvious way that the results are immediately discarded, which is where most of the danger resides.

-30-

I had thought that ‘pareidolia’ referred exclusively to the human tendency to see human faces everywhere – any time you see two dots or circles and a line below them, boom :) it’s a face. It can be triggered by rocks, clouds, shadows, uneven browning on a tortilla or a pancake.

It turns out, though, that pareidolia covers any kind of overactive pattern matching. Humans look for meaningful patterns all the time, without thinking about the search, and when something trips the right thresholds in our brain, we seize upon it. The result can be faces, voices, music or just a cloud that looks like a moose.

Here’s my thesis: the large language models (LLMs) and similar generative software processes that people currently refer to as “AI” – those are exceptionally well-tuned to produce patterns that will trigger threshold recognition in human brains. Not by accident – that’s the goal. They are pareidolia-circuit stimulators.

The problem is that LLM output is exactly as meaningful as any other kind of pareidolia. Every well-formed paragraph is a false-positive error.

And when people repeatedly mistake their overactive pattern recognition for reality, we call that a mental disorder: psychosis.

-30-

It is essential to realize that these technologies [large language models, currently called AI] do not answer questions about a model of reality, but about a model of language.

The utterances that they produce are statistically good at being plausible conversation related to the input prompt. Any relationship to reality is basically coincidental. Our tendency to view it as meaningful is a product of anthropomorphism and the social conventions of our culture.

People who use tools are responsible for their actions.

(I first wrote a slightly altered version of the above on the XBBN mailing list on January 30, 2023, in response to a query from Candy Sidner.)

-30-

I am aware that this trick never works (cf ‘Hacker’). Still, I need to try.

Please stop using the terms AI, Artificial Intelligence, and even ML and Machine Learning. In the same spirit as “the cloud is just someone else’s computer”, please regard all these systems as Statistical Correlation Networks, SCNs.

When you read, hear, or say AI you carry the baggage of a hundred years of science fiction stories. You will then make mistakes in reasoning about SCNs.

Yes, it is amazing what SCNs can do, but they aren’t people you can teach and reason with.

-30-