I'm a fan of testing, but however good my test suite is, there's always that nagging question - how good is it? Does it miss anything important? Mutation Testing is a way of probing that. Each run makes a small modification to the code, such as reversing a conditional or removing a line. We then run the test suite. If the tests pass, then we've found a problem.

I first ran into mutation testing a couple of decades ago. My colleague Ivan Moore built Jester, an open source mutation testing tool. But it wasn't something that saw a lot of use, because it was very time-consuming to use. Every mutation required compiling everything, and re-running the test suite. It also ran into the problem that for its high cost of usage, it didn't find many problems because those that used it also did good Test Driven Development, so had a solid test suite.

Since then, more sophisticated mutation testing tools have appeared (Pitest and Stryker have appeared in the Thoughtworks Technology Radar). Some speedups are possible. Languages like JavaScript and Python don't need a recompile to make test runs. Tooling can assess where recent changes occurred in the repository and concentrate mutations in those areas. Tests can be analyzed to see which parts of the code they exercise so that only relevant.

Mutation Testing can also be done manually. If I'm working with some code, where I'm concerned about its test coverage, I'll quickly mutate a few things and see how the tests do.

The rise of LLMs have increased the value of mutation testing. We can ask an LLM to use TDD, but that doesn't mean they will, or that they will do a good job. Mutation testing is one way to assess how good their tests are. LLMs increase the value of having a good test suite, because they do a better job of building or restructuring code when they have a strong test suite to act as a computational sensor. LLMs make it easier to run a mutation testing loop and can also be used to generate mutations more efficiently.