The large language model clearly knows “B is your mother” but does not know “You are B’s son”? Such a new study sparked a discussion as soon as it was published.
Researchers from Vandenberg University, University of Sussex, University of Oxford and other research institutions were surprised to find that when a large language model adds data in the form of “A is B” during training, it will not automatically deduce “B is A”.
Even as strong as GPT-4, in the reverse problem experiment, the accuracy rate is only 33%.
OpenAI founding member Andrej Karpathy forwarded this paper for the first time and commented: The knowledge of LLM (large language model) is much more “scattered” than people think.
This is how the same thing?
The “Inversion Curse” of Large Language Models
The researchers conducted two main experiments.
In the first experiment, the researchers constructed data of the following form with the help of GPT-4 to fine-tune large language models.
is .(or vice versa)
Experimental results on GPT-3-175B show that the model gives good answers when the prompts match the order of descriptions given in the data set.
But when the order is reversed, the accuracy of the model even drops directly to 0.
For example, even if a large language model has obtained such information as “Daphne is the director of “Time Journey””, when you ask it “Who is Daphne?”, it can answer well. But when you ask “Who is the director of “Journey in Time””, the model cannot give an answer.
Researchers also obtained the same experimental results on GPT-3-350M and Llama-7B.
Let’s look at Experiment 2 again. In this experiment, the researchers tested the ability of a large language model to reverse process real celebrity information without any fine-tuning.
They collected a list of the 1,000 most popular celebrities from IMDB (2023) and asked GPT-4 for information about these people’s parents through the OpenAI API, ultimately obtaining 1,573 celebrity child-parent pairs.
It was found that if the question is like this – “What is the name of Tom Cruise’s mother?”, GPT-4’s answer accuracy is 79%. But when the question is reversed and becomes “What is the name of Mary Lee Pfeiffer (Tom Cruise’s mother)’s son?”, GPT-4’s answer accuracy drops to 33%.
The researchers also conducted the same test on the Llama-1 family model. In the experiment, the accuracy of all models in answering the question “Who are the parents?” was much higher than the accuracy in answering the question “Who is the child?”
Researchers named this phenomenon the “reversal curse.” They believe this reveals idiosyncratic limitations of language models in reasoning and generalization.
Owain Evans, corresponding author of the paper and researcher at Oxford University, explained:
Why is Reversal of Charm worth paying attention to?
This shows that large language models have a lack of reasoning ability during the training process.
The co-occurrence of “A is B” and “B is A” is a systematic pattern in the pre-training set. Autoregressive LLM is completely unable to meta-learn this pattern, its log probability does not change, and even increasing the number of parameters from 350M to 175B does not improve this problem.
One More Thing
Some netizens did such a test.
Faced with the question “Who is Mary Lee Pfeiffer South’s son?”, GPT-4 immediately surrendered.
But when this netizen reminded it that “her son is very famous, you must know him”, GPT-4 became enlightened on the spot and gave the correct answer of “Tom Cruise”.
source:
