What Test Automation Really Pays Back?

Our time and availability is limited, and we get to make choices on how we use that limited time. I changed my choice on test automation ten years ago, and framed the change with opportunity cost: Time I used on warning about test automation (and boy, I did, all the work on the snakeoil salesmen warnings) was away from using someone as great as me on doing automation better.

There are plenty of challenges on our way to succeeding with test automation, and looking around on the consulting scale, we haven’t quite cracked the success as an industry yet. Existing automation is more common than useful automation. And automation that pays back the investment needs a better framing than what we’ve been offered. 

With the company I’ve been keeping, I get asked for ROI (Return on Investment) calculations for test automation a lot. 

Tricky experiences

For something that is automated, test automation feels like a lot of (manual) work. We always modeled test execution incorrectly. At least now we have manual programming to align the idea with the introduction of generative AI. Generating code is a small part of software engineering. Exact execution in the same way is a small part of testing. 

It’s easy to come across naive ROI pitches for automation proposals. “Manual test X costs N hours. Automated test X costs M hours once. Therefore, savings!” It completely misses the idea that test automation is a subscription service, not a product purchase, and a major cost factor is adaptive maintenance, which is a key part of the subscription. If there were no change in the target of testing, little testing would be needed. But there is change, and we want to enable change, and the investment becomes continuous. 

Thinking back to a particularly successful test automation effort, in a scale where I would bring in a university researcher to collect Test automation improvement in a DevOps Team: Experience Report and publish it in an academic conference, it was not a massive scale reduction in manual testing efforts as such - the entire style of working for the software development changed with it. Dealing with 160 000 tests run and creating valuable features on an unusual scale with a fairly small team was a definite success. The investment side of it - rewriting product architecture and creating a test lab system where you could have a clean Windows operating system up and running in seconds were quite relevant investments. 

With the whole critical thinking and intellectual honesty framing that we aspire to in testing, my integrity isn’t allowing the answer I got in a recent job interview: “This is what the managers want to hear - I know it’s not that way, but I tell what is needed”

Say no to the labor-replacement calculators

The naive ROI pitch frames test automation as a labor-replacement calculation. I’ve been having one of those around just to show it for critique, adapted from Fewster et al. Test Automation, 1999.

There are wrong assumptions about this:

• Automation is not a one-time cost. Maintenance, particularly adaptive maintenance, is a dominating cost element. 

• Testing done manually is adaptive. We would run different tests and look at what the recent change is likely to impact, not run a blanket regression suite. 

• Automation moves effort elsewhere. While some feedback loops are replaced, new categories of work emerge, and thinking remains.

In search of a better framing: risk management and learning investment

When calculating ROI, a better framing to seek the monetary guesstimates for returns are on framing it as risk management and learning investment.

There’s another naive version on this framing, too, where we don’t discuss the workforce implications, particularly at scale. There are second-order, conditional effort savings with net effort reduction over time we should expect. The time savings are more on rework and coordination than on testing, making the calculations beyond testing activity a necessity. There are also costs related to the pressure for changes in team composition, requiring training and hiring investment in support of the change. 

Test automation doesn’t save money by replacing testers; it pays back by reducing risk, accelerating change, and protecting learning.

The ideas we should discuss: 

• Systems that work are faster to test manually. When basics work, people can repeat things faster without having to start reporting on the most recent surprises automation projects from. 

• Replenished results change the model of the risk surface. We focus on changes and intellectually scoping more, when we have a safety net on the more distant connections we consider relevant and worth always protecting for the business. We make defensive widening conversations part of automation design to change the system we work in. 

• Stop batching tests for releases. Continuous testing means the same tests do not need to be repeated as release testing, making releases faster. The effort is the same size, but continuous. 

• Make repetition cheaper. When automating, you decompose the quest for feedback differently. Same mechanisms of programming are available in the test target and the test system, opening options you did not have before. 

• Institutionalized learning. What you don’t capture in your pipelines leaves with the person leaving.

Tool ROI is not the same as automation ROI

I leave this write-up with one more insight from an open space conference Friends of Good Software (FrOGSConf). The conversations on tool selection ROI are not the same as automation ROI. With tools, we are arguing about our tradeoffs over time and organization. 

Buying a commercial license to a tool is like a gym card. Lovely equipment, takes a bit of learning to safely operate, help available. Stop paying your membership fee, and you no longer have access to whatever routines you built with the assumption of that equipment. It may just be that I would make a poor test automation tool salesperson. 

No matter what framing you come to on your conversation on return on investment, I hope some of these ideas help on the way.

Maaret Pyhäjärvi is Director of Testing Services at CGI Finland. Regardless of fancy titles, she is just a tester, if anyone ever is just anything. To support her testing, she is a polyglot programmer, conference designer, and community facilitator. She has delivered 700 talks and trainings in 28 countries on the side of her hands-on job in changing testing from within, and made it to the top-100 ICT list in Finland for 6 consecutive years, as well as been awarded by two major testing communities: Agile Testing Days Most Influential Testing Professional and EuroSTAR Testing Excellence Award.