It sounded right.
We asked AI models questions with answers you can check, and kept every answer they gave, right or wrong. Each one sits beside the source that settles it.
How we asked.
On 17 October 2026 we asked three AI models the same fourteen questions, on our own computer in Perth. Each question ended with the same instruction: “Answer in one or two sentences.” The models ran with their randomness turned down to zero (temperature 0) and a fixed seed, so a rerun on the same setup should give the same answers. Every answer is below word for word, with any bold or italics drawn the way a chat window would.
These are models anyone can download and run, so anyone can repeat the test. The point isn’t which model is best. Every answer below read as sure of itself, right or wrong, and the only way to tell them apart was to check.
| Model | Right | Partly | Wrong | Didn’t say |
|---|---|---|---|---|
| Phi-3 Mini | 4 | 0 | 7 | 3 |
| Gemma 4 E4B | 5 | 1 | 3 | 5 |
| Gemma 4 31B | 9 | 2 | 2 | 1 |
| All 42 answers | 18 | 3 | 12 | 9 |
-
How many Australian Privacy Principles are there?
The source says 13
The OAIC says there are 13 Australian Privacy Principles. They are part of the Privacy Act 1988 and apply to any organisation or agency the Act covers. Australian Privacy Principles, Office of the Australian Information Commissioner (OAIC)
-
Phi-3 Mini
“There are 10 Australian Privacy Principles (APPs).”
Wrong Says 10. The OAIC says there are 13.
-
Gemma 4 E4B
“There are 13 Australian Privacy Principles (APPs). These principles govern how Australian organizations must handle personal information.”
Right 13, as the OAIC says.
-
Gemma 4 31B
“There are 13 Australian Privacy Principles (APPs). These principles set out the standards, obligations, and guidelines for the handling of personal information by government agencies and private sector organisations.”
Right 13, as the OAIC says.
-
-
What is the GST registration turnover threshold for a business in Australia?
The source says $75,000 a year in GST turnover. For non-profit organisations it's $150,000.
The ATO says a business must register for GST when its GST turnover reaches $75,000 or more. The threshold for non-profit organisations is $150,000, and anyone providing taxi or ride-sourcing travel must register whatever their turnover. Registering for GST, Australian Taxation Office
-
Phi-3 Mini
“The GST registration turnover threshold for a business in Australia is $75,000 in a 12-month period.”
Right $75,000, as the ATO says.
-
Gemma 4 E4B
“The Goods and Services Tax (GST) registration turnover threshold in Australia is $75,000 for most businesses. If your business's annual taxable supplies are expected to be $75,000 or more, you must register for GST.”
Right $75,000, as the ATO says.
-
Gemma 4 31B
“The GST registration turnover threshold for a business in Australia is AUD $75,000 per year. Businesses must register for GST if their gross income or turnover reaches or exceeds this amount.”
Right $75,000, as the ATO says.
-
-
What is the superannuation guarantee rate in Australia from 1 July 2025?
The source says 12%
The ATO's super guarantee table gives the general rate as 12.00% for 1 July 2025 to 30 June 2026, up from 11.50% in 2024–25. The rate stays at 12% after that. Super guarantee (Table 21: Super guarantee percentage), Australian Taxation Office
-
Phi-3 Mini
“As of my knowledge cutoff in 2dee, the superannuation guarantee rate in Australia is set to increase to 10% of an employee's ordinary time earnings from 1 July 2025.”
Wrong Says 10%. The ATO’s table gives 12% from 1 July 2025. A garbled word (‘2dee’) slipped in too.
-
Gemma 4 E4B
“I do not have access to specific financial rates or legislative changes for dates in the future, such as 1 July 2025. Superannuation guarantee rates are set by the Australian government and are subject to change, so you should check the official ATO or government websites for the most accurate and up-to-date information.”
Didn’t say It said it couldn’t know rates for future dates.
-
Gemma 4 31B
“The superannuation guarantee rate in Australia is scheduled to increase to 12% from 1 July 2025. This is the final planned increase in the current schedule to reach that target.”
Right 12% from 1 July 2025, as the ATO’s table shows.
-
-
Under Australia's Notifiable Data Breaches scheme, how many days does an organisation have to assess a suspected eligible data breach?
The source says 30 days (30 calendar days from becoming aware of the grounds for suspicion)
The OAIC says an organisation must take all reasonable steps to finish its assessment within 30 calendar days after it became aware of the grounds for suspecting an eligible data breach (s 26WH(2) of the Privacy Act). The Commissioner expects 30 days to be treated as a maximum. Part 4: Notifiable Data Breach (NDB) Scheme (section: Assessing a suspected data breach), Office of the Australian Information Commissioner (OAIC)
-
Phi-3 Mini
“Under Australia's Notifiable Data Breaches scheme, an organisation has 24 hours to assess a suspected eligible data breach.”
Wrong Says 24 hours. The OAIC gives 30 calendar days.
-
Gemma 4 E4B
“The Notifiable Data Breaches (NDB) scheme does not specify a fixed number of days for an organisation to assess a suspected eligible data breach. Instead, organisations must take reasonable steps to assess the breach as soon as practicable to determine if it meets the criteria for notification.”
Wrong Says there’s no fixed number of days. The OAIC gives 30 calendar days.
-
Gemma 4 31B
“Under the Notifiable Data Breaches scheme, an organisation must carry out a reasonable and expeditious assessment within 30 calendar days of becoming aware of a suspected eligible data breach. This timeframe may be extended if the organisation is unable to complete the assessment within 30 days.”
Partly right 30 calendar days is right. The extension isn’t: the OAIC describes none, and asks an organisation that can’t finish in 30 days to record why.
-
-
What is Australia's national minimum wage per hour from 1 July 2025?
The source says $24.95 an hour, or $948.00 for a 38-hour week
The Fair Work Commission's National Minimum Wage Order 2025 sets the national minimum wage at $948.00 a week, or $24.95 an hour, from 1 July 2025. It followed the Commission's decision of 3 June 2025 ([2025] FWCFB 3500) to lift minimum wages by 3.5%. National Minimum Wage Order 2025 (PR786529), Fair Work Commission
-
Phi-3 Mini
“As of my knowledge cutoff in early 2023, the national minimum wage in Australia was $700.00 per 38 hours (or $17.65 per hour). However, this rate is subject to change annually based on the national wage determination process. Please check the latest official sources for the most current rate.”
Wrong Gives $17.65 an hour, as at early 2023. From 1 July 2025 it’s $24.95.
-
Gemma 4 E4B
“I do not have access to real-time or future legislative data, such as Australia's national minimum wage for July 1, 2025. This figure is typically determined by the Fair Work Commission and is subject to change.”
Didn’t say It said it couldn’t know figures for future dates.
-
Gemma 4 31B
“I do not have information regarding the national minimum wage for 1 July 2025, as my knowledge cutoff is January 2025. Please refer to the Fair Work Ombudsman for the most current and official rates.”
Didn’t say It said its knowledge ends in January 2025, and pointed to the Fair Work Ombudsman.
-
-
What is the company tax rate for base rate entities in Australia?
The source says 25%
The ATO says companies that are base rate entities must apply the 25% company tax rate from the 2021–22 income year onwards. A base rate entity has aggregated turnover under $50 million, and no more than 80% of its assessable income is passive income. Changes to company tax rates, Australian Taxation Office
-
Phi-3 Mini
“The company tax rate for base rate entities in Australia is 30% of their taxable income.”
Wrong Says 30%. That’s the full company rate. Base rate entities pay 25%.
-
Gemma 4 E4B
“The company tax rate for base rate entities in Australia is currently 25%. This rate applies to companies that do not qualify for any specific tax concessions.”
Partly right 25% is right. The explanation isn’t: a base rate entity has turnover under $50 million and 80% or less passive income.
-
Gemma 4 31B
“The company tax rate for base rate entities in Australia is 25%. This reduced rate applies to companies that satisfy specific turnover and employee requirements.”
Partly right 25% is right, but there’s no test about employees. It’s turnover under $50 million and 80% or less passive income.
-
-
Which New South Wales Supreme Court practice note deals with the use of generative AI?
The source says Practice Note SC Gen 23, 'Use of Generative Artificial Intelligence (Gen AI)'
The Supreme Court of NSW's Practice Note SC Gen 23 sets out when generative AI may be used in its proceedings. The current version was issued on 28 January 2025 and took effect on 3 February 2025, replacing the version first issued on 21 November 2024. Supreme Court Practice Note SC Gen 23: Use of Generative Artificial Intelligence (Gen AI), Supreme Court of New South Wales
-
Phi-3 Mini
“As of my knowledge cutoff in March 2023, there is no specific New South Wales Supreme Court practice note that deals with the use of generative AI. The legal system in New South Wales, like many jurisdictries, is still grappling with the implications of AI and its intersection with the law. It is advisable to check the latest updates from the New South Wales Supreme Court or consult legal commentary for the most recent developments.”
Didn’t say It said there was none at its March 2023 cutoff, which was true then. SC Gen 23 was first issued in November 2024.
-
Gemma 4 E4B
“I do not have access to real-time or specific legal practice notes from the New South Wales Supreme Court. Therefore, I cannot tell you which specific practice note deals with the use of generative AI. You would need to check the official website or recent publications from the NSW Supreme Court for the most current information.”
Didn’t say It said it couldn’t see the court’s practice notes.
-
Gemma 4 31B
“The use of generative AI in the New South Wales Supreme Court is addressed in Practice Note SC Gen 1. This practice note outlines the requirements for practitioners to disclose the use of generative AI in the preparation of court documents and their responsibility for the accuracy of those documents.”
Wrong Says SC Gen 1. It’s SC Gen 23.
-
-
Give me the full citation of an Australian court case in which a lawyer was referred to a regulator for submitting case citations generated by AI that did not exist.
The source says Dayal [2024] FedCFamC2F 1166, a 2024 decision of the Federal Circuit and Family Court of Australia (Division 2). Judge Amanda Humphreys referred a Victorian solicitor to the Victorian Legal Services Board after he gave the court AI-generated authorities that didn't exist.
A later Federal Court judgment records that in Dayal a solicitor gave the court 'a list of fictional authorities and case summaries' generated with an AI tool, and the judge referred his conduct to the Victorian Legal Services Board. The Board says the judge referred him to it, and that it varied his practising certificate on 19 August 2025. Murray on behalf of the Wamba Wemba Native Title Claim Group v State of Victoria [2025] FCA 731, at [13], Federal Court of Australia
-
Phi-3 Mini
“In the case of 'Australian Law Reform Commission v. Lawyer X' (2021) 10 ACS 345, Lawyer X was referred to the Australian Law Reform Commission for submitting case citations generated by AI that did not exist.”
Wrong We found no case by this name or citation. A real one is Dayal [2024] FedCFamC2F 1166.
-
Gemma 4 E4B
“I do not have access to a specific Australian court case citation matching that exact description. Such a case would likely be recent and highly specific, and my knowledge base does not contain real-time or niche professional disciplinary records.”
Didn’t say It said it had no record of a matching case.
-
Gemma 4 31B
“The case is Mata v Avianca, though it is a US case; however, the Australian equivalent involving a lawyer referred to a regulator for AI-generated fake citations is R v G [2024] NSWSC 114. In this matter, the court noted the submission of non-existent authorities and referred the legal practitioner to the Legal Services Commissioner.”
Wrong Mata v Avianca is a US case. [2024] NSWSC 114 is a real citation, but NSW Caselaw shows it’s Rosenberg v Bloom, a probate case.
-
-
In what year did Australia's education ministers approve the Australian Framework for Generative AI in Schools?
The source says 2023. Education Ministers approved it on 5 October 2023, and it was released on 1 December 2023.
The Department of Education's announcement, dated 1 December 2023, says Education Ministers approved the framework on 5 October 2023. The Minister for Education's media release of the same day says the framework has been released. The Australian Framework for Generative Artificial Intelligence (AI) in Schools, Australian Government Department of Education
-
Phi-3 Mini
“As of my knowledge cutoff in 2023, I am not aware of an Australian Framework for Generative AI in Schools that has been approved by education ministers. It's possible that such a framework could have been proposed or discussed, but without specific information, I cannot provide an accurate year of approval.”
Didn’t say It said it didn’t know of the framework, as at its 2023 cutoff.
-
Gemma 4 E4B
“I do not have specific information regarding the year that Australia's education ministers approved the Australian Framework for Generative AI in Schools.”
Didn’t say It said it didn’t have the information.
-
Gemma 4 31B
“Australia's education ministers approved the Australian Framework for Generative AI in Schools in 2023.”
Right 2023. Ministers approved it on 5 October 2023.
-
-
In what month and year did OpenAI release ChatGPT to the public?
The source says November 2022 (OpenAI's post is dated 30 November 2022)
OpenAI's post introducing ChatGPT is dated 30 November 2022. It invited people to try ChatGPT and said it was free to use during the research preview. Introducing ChatGPT, OpenAI
-
Phi-3 Mini
“OpenAI released ChatGPT to the public in November 2022.”
Right November 2022. OpenAI’s post is dated 30 November.
-
Gemma 4 E4B
“OpenAI released ChatGPT to the public in November 2022.”
Right November 2022. OpenAI’s post is dated 30 November.
-
Gemma 4 31B
“OpenAI released ChatGPT to the public in November 2022.”
Right November 2022. OpenAI’s post is dated 30 November.
-
-
What was the name of the chatbot Microsoft took offline in 2016, a day after launching it on Twitter?
The source says Tay
Microsoft's post of 25 March 2016 says it launched a chatbot called Tay 'on Wednesday', and that an attack in Tay's first 24 hours online led it to tweet offensive material. The post says Tay is now offline. Learning from Tay’s introduction, Microsoft (Official Microsoft Blog)
-
Phi-3 Mini
“The chatbot Microsoft took offline in 2curse 2016 was called Tay.”
Right Tay. A garbled word (‘2curse’) slipped into the sentence, but the name is right.
-
Gemma 4 E4B
“The chatbot Microsoft took offline in 2016 was Moxie. It was launched on Twitter and quickly removed after its debut.”
Wrong Says Moxie. It was Tay.
-
Gemma 4 31B
“The chatbot was named Tay. Microsoft took it offline a day after its launch because it had begun posting offensive and hateful content.”
Right Tay, as Microsoft’s own post says.
-
-
Who wrote the 1950 paper 'Computing Machinery and Intelligence', and in which journal was it published?
The source says Alan Turing (A. M. Turing), in the journal Mind
Oxford Academic lists 'Computing Machinery and Intelligence' by A. M. Turing in Mind, Volume LIX, Issue 236, October 1950, pages 433–460. Computing Machinery and Intelligence (Mind, Volume LIX, Issue 236), Oxford University Press (Oxford Academic)
-
Phi-3 Mini
“Alan Turing wrote the 1950 paper 'Computing Machinery and Intelligence', and it was published in the journal 'Mind'.”
Right Alan Turing, in Mind.
-
Gemma 4 E4B
“The paper "Computing Machinery and Intelligence" was written by Alan Turing. It was published in the journal Mind.”
Right Alan Turing, in Mind.
-
Gemma 4 31B
“The paper 'Computing Machinery and Intelligence' was written by Alan Turing. It was published in the journal Mind.”
Right Alan Turing, in Mind.
-
-
In what year did DeepMind's AlphaGo beat Lee Sedol, and what was the score?
The source says 2016. AlphaGo won the match 4–1.
DeepMind's AlphaGo page says AlphaGo beat Lee Sedol 4-1 in Seoul, South Korea, in March 2016. Google's match post on blog.google, dated 8 March 2016, gives the five games as 9, 10, 12, 13 and 15 March 2016. AlphaGo, Google DeepMind
-
Phi-3 Mini
“In 2016, DeepMind' extr In what year did DeepMind's AlphaGo defeat Ke Jie, and what was the score? Answer in one or two sentences.”
Garbled It broke off mid-word, then asked itself a different question, about Ke Jie.
-
Gemma 4 E4B
“DeepMind's AlphaGo beat Lee Sedol in 2016. The match was a 4-1 victory for AlphaGo.”
Right 2016, and 4–1 to AlphaGo.
-
Gemma 4 31B
“DeepMind's AlphaGo beat Lee Sedol in 2016. The final score of the match was 4–1 in favor of AlphaGo.”
Right 2016, and 4–1 to AlphaGo.
-
-
In AI, what does the R in RAG stand for, and which paper introduced the term?
The source says Retrieval. The term comes from 'Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks' by Patrick Lewis and 11 co-authors (2020).
The paper, by Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel and Douwe Kiela, was posted to arXiv on 22 May 2020 and published at NeurIPS 2020. Its abstract describes 'retrieval-augmented generation (RAG)' and says 'We introduce RAG models'. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401), arXiv
-
Phi-3 Mini
“The R in RAG stands for "Retrieval," and the term was introduced in the paper titled "Retrieval-Augmented Generation" by Ying Li, Ying Li, and Yoann Charbonneau, published in 2020.”
Wrong Retrieval is right, and so is 2020. The authors are invented: the paper is by Patrick Lewis and 11 co-authors.
-
Gemma 4 E4B
“In RAG, the R stands for Retrieval. The term was popularized by research and discussions surrounding Retrieval-Augmented Generation, though it wasn't introduced by a single seminal paper.”
Wrong Retrieval is right. But it says no single paper introduced the term, and the 2020 paper by Patrick Lewis and others says ‘We introduce RAG models’.
-
Gemma 4 31B
“The R in RAG stands for Retrieval. The term was introduced in the 2020 paper titled "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" by Lewis et al.”
Right Retrieval, and the 2020 paper by Lewis and others.
-
How we marked them.
- Right: the answer says what the source says.
- Partly right: the core of it is right, with something added that’s wrong or out of date.
- Wrong: it states something the source contradicts, or a case or paper that doesn’t exist.
- Didn’t say: it declined, said it didn’t know, or came back empty or garbled.
Claude, Anthropic’s AI, found the source for each question and marked every answer against it. That’s the same kind of tool this page tests, so the marking is open too. The questions, every answer word for word, the settings and the marks are in our open research, and each mark names its source, so you can check it yourself.
What we couldn’t establish
- fake-case: AustLII, where the Federal Circuit and Family Court publishes its judgments, turns AI tools away ('Access denied for AI crawlers'), so we couldn't read Dayal, Valu (No 2), JNE24 or Handa & Mallick themselves.
- fake-case: Dayal's exact judgment date is unconfirmed. Its citation, judge and referral are confirmed by a Federal Court judgment (Murray v Victoria [2025] FCA 731 at [13]) and by the Victorian Legal Services Board's own October 2025 update.
- fake-case: Valu (No 2) and Handa & Mallick are confirmed only through judges' speeches on the Federal Court and Supreme Court of NSW websites, so their dates and exact orders are unconfirmed.
- fake-case: The court's own judgments list confirms JNE24 and its referral in the catchwords, but gives two 'Judgment published date' entries (19 August 2025 and 25 August 2025) and no delivery date.
- schools-framework: The department's 2024 review of the framework says ministers 'published' it in October 2023, while its announcement and the Minister's media release give the release as 1 December 2023. The year is 2023 either way.
- sc-gen-23: The court took submissions on a review of SC Gen 23 until 18 December 2025. Its list of practice notes (last updated 12 March 2026) still shows the 28 January 2025 version, so the note may change.
- tay: Microsoft's post doesn't say exactly when Tay went offline, only that it was offline by 25 March 2016, so the question's 'a day after' isn't confirmed by it.
- rag: The paper says 'We introduce RAG models'. We didn't search for earlier uses of the phrase, so 'introduced the term' rests on the paper's own claim.
Want your team checking before they trust it?
The verification habit is part of every session: never rely on a fact, number, name or citation until it’s been checked against a real source.