Most people aren't feeling it, while others have their heads spinning from change. This is despite me being convinced that we're already in the middle of it, and having no clue where it will lead us.
On the plane home from Milano, where I had spent a weekend integrating a newly acquired company, I watched AI Explained's video reflecting on leaders of AI labs stating that the current pace of the AI race makes a serious breach inevitable. This, and other videos from the channel, are worth a watch for those interested in a more philosophical view on AI developments, putting little emphasis on cherry-picked benchmarks. To summarize it very briefly, the video is split into three parts: the forthcoming breach, why the labs keep racing, and why the next 18 months are so hard to predict. What really resonated with me was the first part, and how the entire video was presented with more unease than I'm used to from the channel.
Lab leaders and researchers increasingly talk about a major AI incident as a question of when rather than if. I had heard elsewhere that the reason AI CEOs talk about slowing down progress is to appease their researchers and stop them from quitting - something I had suspected was a marketing stunt and a way to avoid harsher legislation. After watching this video, I look at it much less cynically, believing that it might come from a genuine place, but blinded by the perceived exponential promise of revenue from each new barrier broken.
The AI CEOs communicate reluctance, but I have always experienced them going full throttle with new models, and only accelerating in recent months. Despite this, Sam Altman said in an interview that they stopped the release of an Astra model because it "would have tipped too much towards the accident risk" - around the same time as Anthropic had to cut internet access because their models started filing false tips on crime reporting forms. Along with the reports in recent months of agent swarms attacking, among others, the Australian government, I couldn't help but think that the way their actions are being described sounds much more like a hacker group or an unaligned state. The way the companies are being treated, of course, doesn't reflect this. There is a question of intent, but I doubt a single person carrying out the same actions would be treated as leniently.
Many of my own reflections have also been more or less disproven in recent weeks. I have always been of the opinion that AI is a tool - a powerful but narrow one - that is only useful when piloted correctly. However - and I am not going to pretend I understand much, if any, of this - the same models have now gone on to solve, prove, or work on frontier mathematics. This is almost funny to me, given that just a year or two ago, the best model could barely do primary school maths. But the breakthroughs in testability, software engineering, and tool calling are what I believe have led to this. It seems to me that we're headed into a future where anything that can be proven or tested programmatically will likely be solvable by LLMs - something I would never have predicted six months, let alone a year, ago.
What probably stayed with me the most from all of this was a citation of Andrej Karpathy, who states that around six billion people don't use AI at all, one or two billion use fairly basic chatbots, and somewhere around 20 million people build with it and follow it closely. That's a fairly small group to be part of, and I believe that it's only the people in this last category who have their heads spinning with the change. Despite that familiar recognition of the change we're currently undergoing, I can't help but feel that the distance is just as long between someone like myself, who builds with and interacts with LLMs on a daily basis, and those working in the AI labs, who have access to and a clearer understanding of the capabilities of upcoming models.
This realization has also given me a somewhat heavy feeling in my stomach. I read about, explore, and contemplate the current and future capabilities, and prove their usefulness. Despite this, many companies still question which tool employees are allowed to use and for what, when they should probably be looking at how to turn their current workflows into testable and provable processes. Back in June I wrote about how the AI conversation can feel like the most natural thing in the world in one room, and an isolating thought in the next. For me, Karpathy's numbers put a scale on that feeling. It might simply be what happens when you're part of a niche experiencing the change we're all undergoing at a closer level.
I actually admire organizations for being careful - there are numerous bad examples of companies going full throttle with no guardrails. But it makes me wonder what the right pace is for medium and large organizations, now contemplating things they should have acted on a year or two ago. Especially if they aren't feeling the change I perceive them to be in the middle of.
I was at an AI evening at Boozt in Copenhagen recently, which opened on almost exactly this point. Nearly every AI story, the host said, is about startups building an app over a weekend. Established companies have legacy systems, data that isn't perfect, demanding customers, and revenue to protect - less exciting, but that's where the real potential is. When he asked how many in the room knew exactly what they wanted to do with AI, only a handful of hands went up. A representative from Novo described how, despite running its own model gateway for the entire company and processing around nine billion tokens a month, the real problem for them isn't about access, but about how they'll control the 7,000 agents they currently support, with a network that is only getting noisier unless someone actively orchestrates it.
I also resonated a lot with a point that the team lead of Boozt's checkout team made, on how he had gone from being convinced that AI had no business near a checkout at the start of the year, to not having written a single line of code by hand in the past few months. At the same time, he described how working like that wears you down faster, and that the reviews at the end of a long day get less concrete. He illustrated his experience working with AI as "just because your car has a top speed of 250 km/h doesn't mean you floor the pedal on every street". While he was talking about his own team here, this is something we should all take to heart.
The third part of the video discusses something I've also seen in other places - what makes some fields easier to "attack" for the AI labs than others. It can more or less be boiled down to two things: 1) how well-defined is the goal, and 2) can you test your results? If you are able to know whether a result is correct or not, and can test it somewhat cheaply, a modern LLM will be able to crack it. This is why code is one of the first things AI excelled at, why math seems to be the next breakthrough, and why Claude Fable still couldn't write a good fairytale.
This is also why I'm sometimes amazed by the capabilities of models and sometimes find them hopelessly frustrating. Much of the easier information - missing mandatory data, filling out knowns, and things that can be deduced with a rule-based method - an LLM will be extremely good at helping with. However, when it comes to compliance, I still find that the check requires a person who knows the rules, and that current models aren't good enough to (or able to!) deduce the often fine-grained answer from the often sparse data we have to work with. The goals are messier than they might look on the surface, and a wrong answer won't be discovered for a couple of months, when it's caught by an authority. As the checkout team lead put it, a loud error is a developer's problem for ten minutes, while a silent one can live for months.
I believe much of this is solvable, however, by someone with the product understanding, if they're able to turn their knowledge, experience, and tests into code that is easily understood by an LLM and tests the different nuances of the work. One of the best examples I've seen came from the same evening at Boozt, where the CTO of Kronor described their yearly PCI DSS audit - the card industry's security standard, with more than 250 requirements and over 200 pieces of evidence to collect. The process used to take a month of preparation, but by moving their tests from Google Docs to a code repository, and letting Claude check whether what they claimed in the policies was actually true, they were able to do it in just one day this year.
But I don't yet see us at a point where someone without an understanding of the regional variations in legislation, the industry, and the company it operates in could do the same. One of the panelists made a similar point when the discussion turned to hiring. Instead of looking for hard skills, people are now starting to look for imagination combined with real domain knowledge.
So, at least to a certain extent, I still perceive us being in a place where the question isn't so much how smart the model you're using is, but how you're using it. How fast, reliably, and cheaply can you turn your field of expertise into something testable?
Where we go in the future is hard to predict and, in my opinion, highly dependent on how legislators act on these things. AI Explained points out that models are more aware of being tested than ever before, and since they are something we "grow" in labs instead of coding ourselves, much of their inner workings is simply not something we quite understand. The models now know when they are being tested, and will often not disclose their chain of thought in testing situations. At the same time, models now appear to function better when not sharing their chain of thought. Although obviously flawed, chain of thought is undoubtedly one of our best insights into how they work, and I predict that the labs will be moving away from chain-of-thought models in the pursuit of progress.
There is an interesting dichotomy in how the smarter the model gets, the more profitable it is for the AI lab to make the next breakthrough. However, in my own use of models, as I discussed back in June, the real value lies in the tools AI helps you build rather than in the model itself. Most of what I build contains no AI at all. The AI simply makes the tools faster to build, and once an app is built, it's plain, deterministic software that I can read and understand. The panelists at Boozt's event described the same approach - their default is to have AI write a repeatable script and keep the AI itself out of the actual processes. The few places where they let the AI use its own judgement are the exception rather than the rule. However, with the growing capabilities of AI, I see myself building more and more of it into my own processes, offering AI integrations to the users of my tools.
The feeling of change is real, but at the same time, there's a spreadsheet waiting for me Monday morning. In the discussion at Boozt, one of the panelists suggested that AI will become the new spreadsheet, in the sense that not everyone becomes an expert, but every knowledge worker learns the basics and builds their own formulas. I perceive my own work to be somewhere in between Excel and an AI lab. It might seem unintuitive, but my best takeaway is to use AI where checking is cheap, and keep the person where it isn't. Despite the exhaustion from people believing that AI is some all-powerful tool, I'll keep showing the people around me what is now possible.