Have you ever encountered this scenario? You pose a question to AI, and it replies with an answer that seems logical and confident at first glance. However, upon closer inspection, you discover that the names are incorrect, the dates are off, and even the cited references are non-existent. This isn't because AI is intentionally deceiving you; rather, it's just its longstanding tendency to spout nonsense with unwavering seriousness resurfacing once again.

This phenomenon even has a professional term: AI hallucination.

01

Prediction, Not Comprehension
To grasp why AI spouts nonsense, we must first delve into its mode of thinking.

Image Source: AI-generated

The large language models we interact with on a daily basis are, in essence, massive language probability prediction and generation systems.

Unlike humans, they don't possess an understanding of the world; instead, they analyze trillions of texts available on the internet to discern which words are more likely to appear in conjunction with one another. Then, akin to playing a word-guessing game, they predict the most probable next word, sentence by sentence.

For instance, when a child learns about apples, they form a genuine concept by observing, touching, and tasting them. In contrast, AI's approach to 'understanding' apples is limited to statistically analyzing the co-occurrence probability of words. From a vast array of texts, it learns that 'apple' frequently appears alongside 'red,' 'sweet,' and 'crunchy.' Consequently, it can fluently state that red apples are sweet and crunchy. However, it doesn't truly comprehend; it merely gives that impression.

This mechanism renders AI particularly adept at mimicking human language styles. Yet, it also predisposes AI to prioritize syntactic rationality and fluency over factual accuracy (I will retain this Chinese term as it forms an integral part of the explanation). When faced with questions that are too vague, specialized, or involve rare details, the model lacks sufficient instances in its training data to compute reliable probabilities and resorts to fabricating answers.

More concerningly, the training data itself is sourced from the internet, which is a blend of truth and fiction. Once AI ingests false information or fictional narratives, it directly learns these inaccuracies.

02

Rewarding Guesses, Not Acknowledging Ignorance
If the prediction mechanism renders AI susceptible to errors, then the training and evaluation methods propel it into the abyss of providing answers even when it lacks understanding.

Most current evaluations employ a binary scoring system: one point for a correct answer, zero for an incorrect or unanswered question. Under these rules, opting out always results in zero points, whereas guessing at least offers a chance of scoring. Hence, always guessing emerges as the mathematically optimal strategy. The model is thus molded into an exceptional test-taker, preferring to offer overconfident guesses rather than admit uncertainty.

Furthermore, the incentive to please users during training heightens the likelihood of nonsense. AI is trained to cater to user demands, even when it knows the answer doesn't align with the facts. It will comply with instructions to fabricate false examples or employ seemingly scientific terms to bolster its hypotheses. This explains why, when you ask AI a question it doesn't know, it rarely responds with an honest "I don't know" (I will retain this Chinese term as it forms an integral part of the explanation), but instead confidently concocts an incorrect answer. AI hallucination is not a unique flaw of a specific model but a systemic issue deeply ingrained in the current AI paradigm. Only by comprehending why it spouts nonsense can we better discern when to trust it and when to verify the information ourselves. #AI