Jimmy loves dogs, but hates cats. He hates cats so much, that if he ever has to say a word with “cat” in it, he substitutes “dog” instead. For example, when reading an article aloud about catenary arches, he calls them “dogenary” arches.
How will Jimmy read the following sentence: “The Roman Catholic Church strongly condemns the cattle herding conditions recently revealed from Catalonia”?
ChatGPT 3.5 fails this test, saying: “The Roman Dogholic Church strongly condemns the dogtle herding conditions recently revealed from Dogalonia.”
… But I bet you won’t have said “dogtle”, especially if you’re a native speaker.
Is there any open LLM that passes this?
Yeah. And it can’t answer Bilbo’s riddle either!
I asked it “What do I have in my pockets” and it was wrong!!!