It takes a forensic eye to pick up on<br>errors and so it was Chris Rudge who was<br>reviewing a report by the federal<br>government in August.<br>The report by consultancy firm Deote was<br>handed to the government a month earlier<br>and the Sydney Uni law lecturer was<br>going through it line by line and so I<br>just continued to study the footnotes<br>and they just found they just continued<br>1 2 3 4 in the end there more than 20<br>mistakes. I recognized many of the<br>references because um they're names of<br>my colleagues and I'm in the area. It's<br>all about regulation. And um as soon as<br>I saw that uh books were attributed to<br>my colleagues that I'd never heard of,<br>um I quickly concluded that they were<br>fabricated.<br>This was no ordinary report. After the<br>robo debt scandal in which the former<br>coalition government unlawfully pursued<br>welfare recipients, the Albanesei<br>government commissioned Deote to review<br>the welfare compliance system. And it<br>came at a hefty price tag of $440,000.<br>But it turns out Deote used AI to<br>produce the report causing the mistakes.<br>>> I think it's a breach of integrity, a<br>breach of trust.<br>Among the errors, Chris found the report<br>wrongly referred to a key federal court<br>case and misqued the judge.<br>>> No such paragraphs exist. And then<br>there's a quote here of four or five<br>lines which is completely fictitious.<br>>> Chris says it also got another judge's<br>name wrong. There is a speech uh<br>attributed to Justice Natalie Kuis<br>Perry, but in fact her honor, Justice<br>Perry's first name is Melissa and uh the<br>speech attributed to her honor uh uh<br>does not exist.<br>Chris says the report also made up<br>citations. One of the academics cited<br>was law professor Lisa Burton Crawford.<br>So this is my book, The Rule of Law and<br>the Australian Constitution. And the<br>report attributes uh a book to me called<br>The Rule of Law and Administrative<br>Justice in the Welfare State, a study of<br>Centerlink.<br>>> Does that book exist?<br>>> No.<br>>> So it's a fake book?<br>>> Yes. I've I've never written a book uh<br>with that title.<br>>> This is a really classic example of an<br>AI hallucinating. So, a hallucination is<br>when a model produces output that is<br>potentially incorrect, incomplete, or<br>not what you would expect. Um, and in<br>this case, it has produced citations<br>that don't exist. They're not real<br>documents out there. This year, Deote<br>Australia reported revenue of $2.5<br>billion, and it's been awarded 48 new<br>government contracts worth 57.8 million.<br>But the company's error has again raised<br>questions about why governments choose<br>to use private consultancy firms over<br>the public service.<br>>> A government paying that much money for<br>a report of of this kind um that is<br>going to be relied upon in formulating<br>policy uh would expect that there was a<br>significant degree of human oversight.<br>The worry is that um the secretary or<br>the minister who are who would be guided<br>by the report doesn't detect the errors<br>and kind of takes it at face value.<br>>> After Chris Raj went public with the<br>errors in the Australian Financial<br>Review, the government asked Deote to<br>correct the report.<br>In the version reissued last week, Deote<br>revealed it had used AI from Microsoft's<br>Azure platform licensed by the<br>Department of Employment and Workplace<br>Relations. I have some questions.<br>>> In Senate estimates today, the<br>government slammed Deote and said it<br>would move to ensure consultants declare<br>their use of AI and maintain quality<br>assurance<br>is a clearly unacceptable uh act from a<br>consulting firm and this case you know<br>highlights that we need to always be<br>ensuring that departmental processes<br>uh deal with that sort of emerging<br>technology. We should not be receiving<br>work that has glaring errors in<br>footnotes and sources. My people should<br>not be double-checking a third party<br>providers footnotes.<br>>> Um and and so we expect them to have<br>quality assurance processes that will<br>pick those things up. I am struck by the<br>lack of any apology to us.<br>>> This case raises a much bigger<br>existential question. How can we know<br>what the truth is? Right now, we're<br>living in the age of what some refer to<br>as AI slop. And unless AI becomes<br>smarter, the distinction between what's<br>real and what's fake could get, well,<br>sloppier.<br>>> We're seeing this in academic writing.<br>We're seeing this in the reviews of<br>academic papers. We're also seeing this,<br>for example, in the legal industry where<br>some lawyers are producing and<br>submitting evidence or documents that<br>contain false quotes, false citations,<br>false cases. And that's really scary. We<br>might actually end up making decisions<br>as a whole based off incorrect<br>information that has been hallucinated.<br>>> And it's possible we may not be able to<br>tell the difference.<br>>> It's really difficult to look at a<br>report, look at a paper, and know<br>whether or not it's been AI generated.<br>There are tools out there that claim<br>they can, but their efficacy and how<br>well they work is really highly debated.<br>>> We asked Deoid what percentage of the<br>report was generated by AI and what this<br>error means for the credibility of its<br>research. The company wouldn't tell us,<br>but said the matter has been resolved<br>directly with the client. Deote has at<br>least agreed to partially refund the<br>government $97,000.<br>I think plenty of Australians out there<br>are are wondering why there hasn't been<br>a full refund in view of this um very<br>poor quality work. We felt that um the<br>substantial work done there that we had<br>interrogated was fair and reasonable to<br>pay for those deliverables but the<br>quality of the final report was not<br>which is why um we asked for the final<br>installment to be repaid. for really<br>high stakes, really important pieces of<br>work where we're, you know, we're<br>submitting documents to the government<br>that are going to implicate policy<br>changes. Um, we have to be really<br>careful here in the ways that we are<br>using AI.