When people hear “AI could end the world” they tend to picture a film. Researchers mean something quieter, and in some ways more unsettling: systems that become very capable, very fast, at pursuing goals that aren’t quite the ones we meant to give them.
What the worry actually is
Today’s systems already surprise their makers. As they’re given more autonomy (writing code, running tasks, making purchases), small misalignments between what we want and what they optimise for can compound. The concern is not malice. It’s competence without the right brakes. A second worry is misuse: powerful models helping someone design a cyber-weapon or a pathogen.
Preparing for a risk isn’t the same as predicting it. We buy smoke alarms without expecting a fire.
Where experts disagree
Some think dangerous capabilities are years away; others, decades or never. Many argue that focusing on far-off scenarios pulls attention from the bias, scams and job disruption hurting people now. We think a mature society can hold both conversations at once, and that many of the same fixes, like transparency, testing and accountability, help with both.
None of this requires panic. It requires the same steady, practical preparation we bring to bushfire season.