When generative AI first entered the mainstream, it felt as though software testing had reached one of those rare moments that fundamentally changes the way we work. Almost overnight, our feeds filled with demonstrations of AI writing automation, generating test cases, analysing logs, reviewing code and producing documentation in seconds. Every week seemed to bring another breakthrough, another assistant and another company claiming to have reinvented testing.

Like many others, I found it difficult not to be excited. The promise was compelling: repetitive work would disappear, productivity would increase and testers would finally have more time to focus on the creative, investigative side of quality engineering. It wasn't just the technology that captured our attention; it was the possibility of fundamentally changing how we approached our work.

When I recently spoke at EuroSTAR in Oslo, I didn't want to revisit the familiar debate about whether AI would replace testers or speculate about what the future might hold. Those conversations have become commonplace, and most of us have already formed an opinion. Instead, I wanted to share something much more practical: what happened after my team stopped talking about AI and started living with it every day.

For the past eighteen months, AI has become part of our normal workflow. We use it to help write automation, generate exploratory testing ideas, review code, analyse defects, draft documentation and solve countless smaller problems that arise throughout the day. We didn't introduce it because we wanted to prove that AI was revolutionary, nor because we wanted to demonstrate its limitations. We simply wanted to understand what would happen if it became another tool that testers reached for without thinking twice.

Initially, the experience felt remarkably close to everything we had been promised.

The immediate improvements were difficult to ignore. Tasks that had previously required significant effort could now be started in minutes, and the intimidating blank page that often accompanies automation or documentation simply disappeared. AI rarely produced a perfect first attempt, but perfection was never really the point. It offered momentum, and momentum is often the hardest thing to create. Rather than beginning with nothing, we began with something that could be refined, challenged and improved.

Across the team, the effect was noticeable. Routine work became less repetitive, automation moved faster and documentation required less effort to produce. It genuinely felt as though we had found a way to spend less time on mechanical tasks and more time solving interesting problems. Looking purely at our productivity, it would have been easy to conclude that AI had delivered exactly what the industry had been promising.

Yet as the months passed, I realised that we had quietly started asking ourselves the wrong questions.

Much of the conversation surrounding AI focuses on what it is capable of doing. Can it generate Playwright tests? Can it write API tests? Can it explain unfamiliar code or analyse production logs? These are useful questions because they help us understand the technology, but after working with AI for long enough they became far less interesting than a different question that gradually emerged during our own discussions.

Were we actually becoming better testers, or were we simply becoming faster ones?

At first glance, the distinction feels almost meaningless. Faster delivery, increased automation and shorter lead times are all things we naturally celebrate, largely because they are easy to measure. We can point to numbers, dashboards and completed work. The qualities that define exceptional testers, however, are much more difficult to quantify. Curiosity, judgement, healthy scepticism and an instinct for risk rarely appear in metrics, yet those are precisely the qualities that have distinguished the best testers I have worked alongside throughout my career.

The most valuable people on any testing team are rarely the ones who write the largest number of automated tests. They are the people who notice the assumption that everyone else overlooked, who ask uncomfortable questions during refinement sessions, who recognise subtle risks before they become production incidents and who understand that successful testing has always been about learning rather than simply executing predefined checks.

That realisation caused us to look more closely at how AI was changing the way we worked.

Interestingly, nothing appeared to be going wrong. If anything, every visible indicator suggested success. Automation was being written more quickly, documentation improved in quality and repetitive work was disappearing exactly as we had hoped. However, every so often a small moment during a review would interrupt that reassuring picture.

Someone would present an automated test that had been generated with the help of AI, and technically everything looked correct. The code compiled, the test executed successfully and nothing immediately appeared questionable. During the discussion, however, somebody would ask why a particular assertion had been chosen or what specific risk that test was intended to cover. Sometimes the explanation came immediately, but sometimes it didn't.

Initially, we dismissed those occasions as isolated incidents. Every team has moments where someone struggles to explain a particular implementation. Over time, though, we realised that these conversations were occurring often enough to reveal a pattern rather than a coincidence.

What surprised us even more was who seemed to benefit the most from AI.

Before we introduced it into our daily workflow, I had assumed that less experienced testers would gain the greatest advantage. After all, having access to an assistant capable of explaining unfamiliar concepts, generating examples and suggesting approaches sounded like the perfect way to reduce the gap between junior and senior engineers.

What we observed was almost the opposite.

Experienced testers extracted significantly more value from AI than anyone else. The reason became obvious only after watching people work over an extended period. They rarely accepted AI's suggestions without question. Instead, they challenged the generated code, rewrote sections they disagreed with, recognised weak assertions, identified missing edge cases and questioned assumptions that AI had silently made. Their experience allowed them to use AI as an amplifier for knowledge they already possessed.

For less experienced engineers, the interaction often looked very different. Because AI could solve many of the immediate technical problems, it occasionally removed the need to work through those problems independently. The answers arrived before the understanding had time to develop.

One particular code review has stayed with me ever since.

A junior engineer had been asked to create a Playwright test. They described the scenario to AI, copied the generated solution, made a few small adjustments and produced a perfectly functional test within minutes. From a delivery perspective, everything had gone well. The task was completed quickly, the implementation worked and nobody could argue with the result.

During the review, however, somebody asked why one of the assertions had been chosen.

The room became quiet.

It wasn't because the engineer lacked ability or intelligence. Quite the opposite. The problem was that they had never been forced to reason through the decision themselves because AI had already done the difficult thinking. The generated solution was perfectly acceptable, yet an important part of the learning process had quietly disappeared along the way.

That moment fundamentally changed how I thought about AI.

Looking back over my own career, I realised that very little of what eventually became experience was acquired through effortless success. The lessons that shaped my judgement almost always came from struggling with a difficult problem, debugging a defect that refused to reveal itself, writing an assertion that turned out to be wrong or spending hours investigating behaviour that initially made no sense. Those experiences were often frustrating, but they were also where intuition developed.

AI excels at removing exactly that kind of friction.

In many situations, that is precisely why it is such an extraordinary productivity tool. Yet it also raises an uncomfortable question. If learning often emerges from working through difficult problems, what happens when those difficult problems increasingly disappear?

That question eventually led us to change the way we approached AI within our team.

Rather than treating AI as a source of answers, we began treating it as a collaborator. We introduced one simple principle that now applies regardless of where code originates: if you cannot explain it, you cannot use it.

The rule applies equally whether the code comes from ChatGPT, GitHub Copilot or even another member of the team. Understanding remains the responsibility of the person committing the change. AI is free to suggest ideas, generate implementations or challenge assumptions, but ownership of the reasoning cannot be delegated.

That single change transformed our conversations. Reviews became less focused on where code had come from and more focused on whether the author genuinely understood what they were trying to prove. In many ways, AI stopped being an answer engine and became what I believe it is at its best: an exceptionally capable thinking partner.

So, after eighteen months, did AI deliver?

Without hesitation, I would say yes. It has reduced repetitive work, accelerated automation and removed countless small obstacles from our daily workflow. I would struggle to imagine returning to the way we worked before it became available.

At the same time, I no longer believe that speed is the most important question AI asks of us.

The more interesting challenge is whether we continue to develop testers who understand systems deeply, question assumptions instinctively and recognise risk before anyone else does. AI can generate code remarkably well, but testing has never really been about producing code. It has always been about understanding why that code exists in the first place.

Perhaps that is the real lesson our team learned over the past eighteen months. AI is at its most powerful when it amplifies human expertise rather than replacing the opportunity to develop it. Used thoughtfully, it can make experienced testers exceptional. Used carelessly, it can allow understanding to be replaced by convenience.

As our industry continues to embrace AI, I suspect that distinction will become one of the most important conversations we have.

I'd love to hear whether your own experience has been similar. Has AI fundamentally changed the way your team thinks about quality, or has it mainly changed the speed at which you deliver it?