Pakistan · World · Independent Digital Newsroom
The experiment included both paid and free variants of ChatGPT, Claude, Gemini, and Perplexity based on 220 shopping queries, including laptops, TVs, mattresses, sunscreen, and robotic vacuum cleaners.Each question was asked five times for each service, yielding a total of 8,794 responses.
Conflict was considered any contradicting detail that could be easily checked.Questions asking models to compare two products directly produced conflicts 97% of the time, compared to a 75% conflict rate for straightforward factual questions.
Pricing was a particular weak spot: of 913 verifiable answers, only 85% matched the current listed price, and when answers were wrong, the price was off by a median of $300.Gemini posted the highest rate of costly errors in the study, at 56% on its free tier and 54% on paid.
Claude's accuracy improved substantially on its paid version, with costly errors dropping to 21% from 44% on the free tier.Perplexity had the lowest costly-error rate at 14% on paid, followed by ChatGPT's paid tier at 17%.
Gemini's free tier also contradicted itself most often, giving inconsistent answers to the same question 29% of the time."In short, it says that these LLMs aren't there yet when it comes to this end-to-end experience," said Dakota Nunley, Product.ai's head of search product.The spokesman for Perplexity dismissed this comment because their firm remains focused on accuracy, whereas Perplexity leads the market in.
The spokespeople from ChatGPT, Claude, and Gemini were not responsive to the inquiry.According to Nunley, one should employ the AI as an instrument of exploration and not rely on it solely, confirming the information via multiple engines or by visiting the manufacturing companies’ websites while shopping.
September 25, 2026.
Comments & Discussion
Join a respectful, on-topic conversation. Personal attacks, hate speech and spam may be removed.