How to Pick a Test Management Solution: 5 Mistakes to Avoid

There are many test management solutions, and they share essential functionality. They all offer test cases & scenarios, reporting, and integrations with other QA and/or software development tools. How do you choose one and not regret it later? Simple: go over less obvious features.

For years, we have been seeking a perfect solution for our enterprise clients. The one that would unify testing in a huge team. Our clients needed to plan better, reuse test cases & resources, and automate testing. The critical requirement was the need to simplify preparation for regulatory audits because many of our clients are BFSI companies. At one point, we understood that it would be easier to create our own all-in-one test management solution taking into account the clients’ needs instead of juggling those tools that were on the market but couldn’t fulfill the needs of enterprise clients. The few available options at the time had too many drawbacks, so we had to walk this road ourselves. Our own test management tool was launched in 2013.

Almost a decade later, we still talk to in-house quality assurance specialists, clients, and clients’ software testers to see what they miss in modern test management solutions. Below are just some of their musings.

On-Premise vs Cloud Versions

The deployment model may not be your primary criterion, but it can become a deal-breaker. In most cases, you will be just fine running a test management solution on the cloud. In fact, it is preferable for the following reasons:

• Upkeep (and associated man-hour costs) lie with the solution provider.

• Prices start low and can be scaled depending on the project.

• Online availability enables work-from-home, both as a policy and emergency Covid containment measure.

  
  

Unfortunately, it’s the online availability that can be a problem. FinTech companies and banks, let alone government agencies can’t risk personal data getting leaked online. Sensitive industries have to use on-premise software, which entails:

• Restricted access (office-only or corporate VPN).

• Higher start fee for an enterprise license.

• Various maintenance options that inevitably involve your IT resources.

• Tailored solution to address your company’s unique needs.

  
  

Large companies can often use cloud software for some projects yet be forced to on-premise products for others. Even if you’re not a large company (yet!), the safest option is to pick a test management solution that runs both on cloud and on-premise. Surprisingly, Jira + a test management plugin for it is not the answer: they are retiring the on-premise offering.

From my experience, on-premise is a very much in-demand model. This year, we have had one On-Premise client for every two Cloud clients.

Don’t take it as a dig on the competition: the distribution is a natural reflection of our client portfolio. For an evangelistic tool like Jira with a lot of SaaS companies as clients, maintaining the on-premise offering might indeed be unfeasible.

Desktop Client vs Web

The client vs web choice is about both preference and performance. Some people prefer to have everything they use at work as separate apps. After all, it makes sense to split tools where you actually work, talk about work, and show your progress. Desktop apps also have better offline functionality, although some web applications handle internet outages well.

The desktop convenience, however, is a double-edged sword. Some test management solutions may lack versions for individual operating systems, such as QTest lacking a Linux version. Desktop-only products can also frustrate management: you can’t simply use your phone to glance at unresolved issues or stability dashboards.

Performance-wise, both desktop and web apps can get things done at scale. Modern software architecture offloads as many calculations from the user as possible. When either cloud or on-premise server does the hard labor, web apps run virtually as well as desktop apps.

On the other hand, some solutions can become slower (no matter how you load them) due to architectural compromises. Here’s a review of Polarion from a seasoned user.

The test management tool of my company (aqua ALM) has both a desktop and web version and most clients prefer the web one. Desktop aqua adepts are power users of the few features that we have not moved to the web yet. The company’s workspace synchronizes changes done in both versions, so every employee is free to use the one they like better.

Neglecting User Permissions

Modern test case management solutions target more than just testers. You have the project manager who checks if things are going well. Sometimes, a product owner might get curious if any bugs would delay a new feature. Finally, you have software developers that need to know what bugs to fix first.

The use scenarios go beyond basic “read/write permissions”, and things only get harder when you run multiple projects.

Luckily, dedicated test case management solutions handle user permissions much better than general-purpose task boards. Advanced products come with more than just several presets of permissions (e.g. Developer, Tester, Manager).

When the functionality is available at all, you can usually create roles with more granular permissions or even change them for each individual user. Here are just some scenarios:

• Junior Project Manager can see/comment on bug reports, but not edit them.

• Senior Developer can mark a bug as done, but only Testers can verify that it is fixed and archive the ticket.

• Members of the Agile team can access issues and tests from their projects only.

  
  

If you would appreciate such control, run possible scenarios in your head before picking a tool. Here’s an abstract from a review on Tuleap from a user who didn’t enjoy this particular aspect:

Apart from granularity, the difference in user permission often comes down to price. Some cloud solutions allow role-based access only if you buy a more expensive package (Qase’s charges at least 50% extra for that). Individual user permissions might be restricted to enterprise (see TestRail) and/or on-premise licenses only. Aqua offers custom user permissions on all cloud and on-premise plans.

Underdeveloped Filters

Now that you’ve decided what users can access, let’s see how they can access it. Even for a QA specialist, test management solutions give more information than one can handle or even need at the moment.

This is an even bigger issue for managers.

The common answer is functionality to see only items assigned to somebody and/or having a certain deadline. Such filters, however, usually lack flexibility and require too much clicking. Even when you usually cycle through similar sets of filters, you have to set them up every time.

So, it would be great to have all items in one place, but also have a neat and fast way to slice through them. Our solution was Views, which are advanced filters and presets of filters.

You can choose a number of filters that you would like to apply at the same time, save them as a View, and switch to it whenever you like. They save a few minutes every day, which adds up to hours and workdays.

Views are also a powerful management tool to assess an individual’s workload and mid-sprint progress. Here are some examples:

• QA Lead can see which Software Tester has the least tickets to assign them a reproduction of a recently reported critical bug.

• QA Lead can verify whether the Software Tester re-assigned items before leaving for vacation.

• Product Owner can see if there are any unresolved defects that caused a recent spike in the abandonment rate.

  
  

Last but not least, Views can be a great collaborative tool. QA Leads can set them up for testers to help them see their area of responsibility. Testers can share their views to speed up sprint planning and daily standups.

You save time by creating views and increase transparency by sharing them — looks perfect to me!

Subpar Logging

In industries with mandatory external audits, software development and testing should be traceable. Regulatory compliances require that you know who deployed a new version of the software and how well it was tested beforehand.

This functionality is often what separates test management solutions from regular task boards.

On the surface, logging is a non-issue. Practically every cloud tool shows you the history of changes to individual items/tickets. This information helps even if you are not legally required to store it.

But what happens if the cloud solution shuts down? How do you access the changes if a card was permanently deleted?

Simple: you decouple public change history from regulatory-compliant logging.
In my case, aqua logs all actions of users. I have information on how all present and past employees interacted with my workspace, projects in it, and even individual items.

The record is unaffected by items and even entire projects removed from the workspace. The log cannot be edited, so it always displays a genuine timeline of changes.

Now that you know the extra criteria for picking the right test management solution, it’s time to scout them. Good luck!

First published here and reposted with permission.