Tech Trends & Future of Work

Deployment Testing Tools and Challenges

One-Minute Insights:

To keep the organization running, decision-makers must ensure that business-critical apps and services are successfully deployed from development to end users. Deployment testing is one way to limit crucial errors that lead to deployment failure.

In this report, Pulse asked over 400 tech decision-makers about deployment testing in their organization. Decision-makers can use this report to benchmark against their peers.

Data collection: Aug 26 - Nov 29, 2021

Respondents: 441 tech decision-makers

Access a downloadable library of hundreds of peer-sourced One-Minute Insight Reports, Q&A, and polls by joining the Pulse executive community.

Join Now

Human error is the most widely cited source of deployment failures, with many decision-makers seeing up to a quarter of deployments fail initial testing

Most decision-makers attribute their team’s deployment errors to human error (54%) and a lack of testing time (51%).

Most (63%) decision-makers report that 1-25% of their deployments fail testing the first time.

“Testing is really necessary at every point of deployment, there is no alternative.”
- VP, large professional services company

While most tests reveal functional errors, most decision-makers still report errors after testing

During testing, most errors are found in functional tests (38%).

However, even after testing, most (57%) occasionally see errors in deployment, though more than a quarter (27%) report that this happens frequently.

“Although we are using ‘best practices’ to test code and deployments, we are not at a 100% ‘pass’ rate yet (I don’t think anyone will be at 100%) so we are continuing to improve the processes and drive deeper automation to drive down the human errors.”
- C-suite, large software company

Most use third-party tools for both desktop and mobile testing with a wide range of tools cited for both

For desktop testing, most (72%) turn to external tools.

Of those who use external, third-party tools, Selenium (15%) is the most popular (the top 5 most popular tools are listed).

However, decision-makers turn to many different desktop testing tools, shown by the pool of responses in all other tools outside of the top 5 most popular (77%).

When it comes to testing for mobile, there’s a similar story. Most use third-party tools (62%).

Appium (21%) was the most cited tool among decision-makers (the top 5 most popular tools are listed).

Though, again, 77% of respondents cited other tools outside of the top 5 most popular.

“No matter how good the testing tools [are], debugging and testing are necessary and need to be repeated over and over again in different environments and ways to get a bug free app.”
- Director, small software company

Most feel confident in their deployment testing tools, with many using automated testing

On the accuracy of deployment testing tools most feel ambivalent, with 40% feeling neither confident nor unconfident, though more feel positively (43%) than negatively (16%).

Most (38%) report that 1-25% of their deployment tests are automated.

“Automation will reduce the errors related to deployment testing.”
- Director, large manufacturing company

Decision-makers want their testing tools to feature GUIs and simple integrations

“Security is one of the most important factors in deployment testing.”
- Director, medium-sized software company

Many decision-makers report that deployment testing has become more challenging during the COVID-19 pandemic due to remote work

Almost two-thirds (62%) of decision-makers describe their app deployment as more challenging during the COVID-19 pandemic due to the remote work situation.

“COVID has negatively influenced our ability to retain competent employees.”
- Director, small education company

Respondent Breakdown

Join the executive community

Make and shape business decisions with tried-and-true advice and benchmarks from technology leaders

Executives powering Pulse

“With its survey data, Pulse skips the anecdotes and provides deep context and real numbers for the topics that are top of mind for my organization.”
Julie Cullivan photo
Julie Cullivan
Chief Technology and People Officer, Forescout
“Pulse beats any other platform, research company, Slack groups, etc. at getting me the most relevant advice and content. I rely on Pulse for all knowledge and insights. The answers are consistently exactly what I need.”
Roberto Torres photo
Roberto Torres
CTO, Taimingo
“What the IT community has needed is a vendor free, agenda free platform which encourages discussion and debate amongst peers. Pulse has nailed that in both their Q&A and timely reports.”
Lee's headshot
Lee Vorthman
CSO, Oracle
“I love that Pulse is a one-stop shop for all the peer conversations and insights that are presently super scattered and disconnected among various Slack channels and other CIO groups.”
Enrique Jenkins photo
Enrique Jenkins
Head of IT, DoorDash
“Being able to drive discussions on new tech with my peers and getting immediate feedback is exactly what has been missing until Pulse.”
Manjit Singh photo
Manjit Singh
CIO, Toyota
“For the past two weeks, the first news source I check [every morning] is Pulse. I look at Focused Five everyday. Pulse first, then Twitter, etc. You're that good.”
Miguel Borbolla Olea photo
Miguel Borbolla Olea
Director of IT, OCESA
“I’m excited for what the Pulse team have built to better connect the CIO community. It’s been exceptional for many of us in the community to get clarity and aid decision making as we develop our strategy.”
Yusuf Khan image
Yousuf Khan
CIO, Automation Anywhere
"Transformative change and real-time insights can only come from the people who are doing it day to day in an innovative way. I get a wide variety of that insight from Pulse."
Malcolm Harkins photo
Malcolm Harkins
Chief Security and Trust Officer, Cymatic