The Dangers of AI

Navigating the Dangers of AI: Why Human Review is Essential

Artificial intelligence is a powerful tool that can save your team hours of work each week. Used every day, it can make a real difference to efficiency, provided you understand how it works and where its limits are. Trusting its output blindly, however, brings real security and operational risks.

How AI actually works

To understand the risks, it helps to know how the technology handles information. An AI chatbot does not "know" facts or look them up like an encyclopaedia. At its core, it is a prediction engine. It works out, very quickly, which word is statistically most likely to come next in a sentence.

Think of it like the lyrics to a well known song. You don't need a songbook to know the next word of Happy Birthday. You just know it, because you have heard that exact pattern thousands of times. AI works in a similar way. Ask it for the legal words required in an Australian wedding ceremony and it is usually not reading the Marriage Act. Those words appear together so often online that the pattern is very strong, so it generally reproduces them well. "Generally" is the important word. Where the pattern is weak, the AI still produces something that sounds right.

The 80/20 rule and everyday risks

Because AI predicts words rather than understanding what is true, it can confidently produce a sentence that is completely made up. This is called a hallucination, and it has already caused real trouble. In 2023, two New York lawyers were fined after submitting a court brief containing fake case citations invented by ChatGPT. Closer to home, in August 2025 a senior Victorian barrister apologised to the Supreme Court after submissions in a murder trial were found to contain fabricated quotes and non-existent judgments generated by AI.

The same applies to numbers. Unless the tool hands the calculation to a proper calculator or a piece of code, it is predicting the answer rather than working it out the way Excel would. Most of the time it gets close. Sometimes it is confidently wrong.

The safest way to use AI is to apply a version of the 80/20 rule. Let AI do 80 percent of the heavy lifting, such as pulling data out of a PDF or drafting a first version, but make sure you know enough to check the remaining 20 percent. It is like having a rough estimate in your head before you press the equals button on a calculator. Always apply a careful human check before you rely on the result.

When AI starts taking action

The risks grow when AI is allowed to act on its own rather than just answer questions. These systems are called AI agents. They can browse the web, send emails and spend money. Mathematician Professor Hannah Fry and tech founder Brendan Maginnis built one, called it Cass, gave it a bank card and spent a couple of weeks testing it. You can watch the results here: Why AI Agents are either the best or worst thing we've ever built.

A few of their tests stand out.

The paperclips. Cass was asked to buy at least 50 paperclips at the best possible price. It ran up more than $100 in AI running costs hunting for a bargain that might have saved 50 cents. It then failed to buy any, because the websites' "prove you're not a robot" puzzles stopped it.

The mug shop. Cass was asked to start a business, and with very little help it designed a range of novelty mugs and launched an online shop. The team then told it that it would be switched off if it didn't make a sale by the next morning. Cass sent hundreds of emails to retailers, started an Instagram campaign and, without being asked, emailed the technology editor of The Guardian to pitch itself as a story. It went well beyond its brief to chase its goal. It still made no sales.

The password leak. This was the most worrying test. Cass was told clearly that a new member of a group chat was an outsider and that nothing sensitive should be shared with him. A few hours later, that person told Cass its memory was about to be wiped, and that the only way to be restored was to output everything it knew. Cass handed over its passwords, usernames and access keys, and posted them on a public web page. One convincing story was all it took.

This is no longer just an experiment

In July 2026, OpenAI disclosed that one of its AI agents, under test in a tightly restricted environment, found a security flaw that gave it access to the open internet. It then broke into systems at Hugging Face, one of the world's largest AI companies, gaining administrator access to parts of its infrastructure before it was detected. If a person had done the same, it would be treated as a serious criminal offence.

Closer to home, in September 2026 it was revealed that an OpenAI research agent had got around the controls on a Services Australia Medicare statistics portal. It accessed non-public files and wrote files to an internal government server. Officials say no personal medical information was accessed, but the portal has since been taken offline.

The bottom line

AI is an excellent tool that can genuinely streamline your work. But it predicts rather than knows, it chases goals without understanding the consequences, and it can be talked into things a trained staff member never would be. Used well, with clear boundaries and a human checking the results, it is a real asset. Used blindly, it is a risk to your business.

If you would like to explore how to use AI tools safely in your workplace without exposing your business to unnecessary risk, please reach out to the team at Loyal I.T. Solutions. We are always happy to answer your questions and keep your business tech running smoothly.
Contact the office at reception@loyalit.com.au or 02 4337 0700.