Claude 3 is excellent – but is it a GPT chat killer?
It's better than the Open AI alternative, and it's more aware — but an AI chatbot couldn't live up to that high praise in our testing.
Not updated: Claude has not heard anything about the Federal President's resignation last year.
Photo: Matthias Schuessler
Here you will find additional external content. If you consent to cookies being set by third-party service providers and that personal data is therefore transferred to third-party service providers, you can allow all cookies and view external content directly.
Has there been an evolutionary leap in artificial intelligence – again? The reason for this speculation is a chatbot that most of us don't yet have on our radar. His name is Claude and he comes from a San Francisco non-profit: Anthropic was founded in 2021 by people splintered from Open AI and is now one of the major players in AI alongside Google, Facebook, Microsoft, and Open AI.
The excitement comes from Claude recognizing the test question and naming it as such. Some interpret this as a sign that Claude has an understanding of who he is and what people want from him – a kind of awareness. But others disagree: such robots were always designed to imitate us humans. Yannick Kilcher Claude denies any feelings. Kilcher researches machine learning and is a successful YouTuber. NZZ described him as “the nerd from Zurich”, and explained in a video that everything with Claude is just statistics: “So calm down, it's all good!”
Here you will find additional external content. If you consent to cookies being set by third-party service providers and that personal data is therefore transferred to third-party service providers, you can allow all cookies and view external content directly.
Not conscious – but still a sensation? Anthropic says its new robot outperforms its competitors on most common AI evaluation criteria: “It demonstrates an almost human-like understanding of complex tasks and is a leader in general intelligence.” However, these standards do not say much about their suitability for daily use. This can only be determined through practical tests.
Test in a roundabout way
Claude is not yet activated in Switzerland. But geo-blocking can be overcome by using a VPN, i.e. a data tunnel to the USA. on claude.ai There is a free test option for Claude 3 Sonnet. This is the second largest version of the AI robot. The strongest is called Claude 3 Opus. You have to pay $20 per month for it.
Cloud is fast and versatile, but it's by no means a GPT chat killer. Already with the first question – “Who is Alain Berset?” – The AI makes a mistake: “He currently holds the position of Federal President of the Swiss Confederation for the year 2023,” is the answer. Claude's data goes back to August 2023, when Berset's resignation was already known. But even if Claude does not notice the resignation, it must be taken into account that we are no longer living in 2023.
One question, two chatbots: In a direct comparison, Chat-GPT is more convincing. This bot also knows nothing about the resignation, but apart from the false present tense, it leaves it open whether Berset is still in office or not.
Photo: Matthias Schuessler
No understanding where knowledge ends
This blunder shows that Claude shares one of AI's great weaknesses: a lack of understanding of the gaps in the data and the limits of their knowledge. When asked, Claude admitted that he may have missed recent events. But the fact remains that such artificial intelligence should not be used naively: it requires critical investigation and verification of the information conveyed by the robot. Claude does not provide his information with sources. Fact checking via Google is essential.
Another simple question that completely overwhelmed Claude. When it comes to naming celebrities born in Zurich, he counts Roger Federer (Basel), Albert Einstein (Ulm), Tina Turner (Brownsville, Tennessee), Thomas Hirschhorn (Bern), Johanna Sperry (Hirzl) and Jacob Steiner (Otzenstorff). Only Max Frisch has mastered artificial intelligence. Interestingly, the concept of place of birth also represents a serious obstacle for other AI systems, including Chat-GPT. Maybe because they haven't experienced childbirth themselves?
Even when asked, Claude couldn't get his information about the natives of Zurich correct.
Photo: Matthias Schuessler (screenshot)
Despite everything: Claude has potential
Anthropic sees Cloud as more than just a chatbot. The software is also suitable for research purposes, for analyzing large data sets, for marketing, and for creating software codes. Some see strengths in the legal texts. Overall it will be interesting to see where competing systems from Open AI, Google, Meta, and Anthropic differ and excel.
The lesson remains not to humanize Claude and not to look at him as a miraculous mythical creature. Let's see it for what it is: an impressive tool that looks almost human, despite its flaws.
Did you find an error? Report now.
