Application development has ceased to be the domain of large software houses and has become the domain of companies from various industries. Currently, most corporations that provide customers or employees with any IT solutions, create them themselves. Therefore, the growing demand is not so much for systems that enable software development as for solutions that automate and accelerate the entire process.
One of the strategic stages of software development work is testing – every single “package” of code should be tested in order to identify and fix bugs as quickly as possible. In addition to automation, it’s important that the environment bridges the silos between the development and operations teams (in line with DevOps principles).
Both of these needs are met by Jira Software in conjunction with the Xray application and additional software development tools.
Xray – benefits
The application has been tested by thousands of users – according to the manufacturer, it is used by more than 5000 customers from 70 countries around the world.
- Lower price than competitive applications. Recommended: Xray vs Zephyr
- Compatibility with Atlassian tools – allows all tests to be managed as tickets (Jira issues) in Jira Software.
- Provides test automation according to BDD methodology and integration with frameworks such as: JUnit, NUnit, Robot, Behave, Selenium
- Xray interacts with DevOps solutions (e.g. Jenkins, Bamboo, GitLab) and fulfills the concept.
- Advanced reporting, team-specific views (dashboards), gadgets.
Recommended- Xray Test Management – Product Walkthrough for Jira Server/DC:
Who is it for?
- for any company that develops or collects software.
- Software Development House, Telecommunications, Banking and Finance, Energy, Shipping, among others.
What is the solution made of? Main components
- Jira Software
- Xray
- Cucumber
- GitLab/ Bitbucket
- Jenkins / Bamboo
+ additional tools for running specific types of tests (e.g. API, Mobile, UI, security, performance tests as well as reporting)
Individual tools may vary depending on what solutions a company uses.
What is it like? A sample test automation process and its key features
Example process for automatic tests
- A common standard for developing manual and automated tests using Cucumber.
- Links between requirements, tests, results, and defects.
- Full CI/CD automation using Jenkins Pipeline, allows for earlier and faster testing (shift left).
- Full integration with test automation tools, making it possible to run tests from within Xray and automatically retrieve results from their execution.
- Easy test scheduling, report generation and creation of custom JIRA dashboards to track project progress.
- Improved team communication.
- Ability to integrate with other systems using REST API.
What does it enable?
Combining the functionality of Jira Software with selected applications enables preparation, execution, monitoring, management of automated tests and analysis of their results in a centralized way. The software development team receives a coherent environment for agile task management based on requests in Jira Software system and for automatic testing by using BDD (behavioural-driven development) methodology. The platform also includes Jenkins functionality for CI/CD activities, as well as additional plugins and scripts, thus streamlining the work of the entire team. Despite using many different tools, it is a coherent, integrated and comprehensive solution.
Test organization
Testing process
The testing process consists of several steps that include the following:
- Test execution planning
- Test design
- Running the tests
- Monitoring progress
- Generating reports and analyzing the results
The platform, built with Jira Software + Xray + Cucumber + Jenkins + (other test management apps), ensures that each step goes smoothly so that teams can run the entire process seamlessly without having to perform repetitive activities; automation mechanisms make the whole thing happen in a well-organized order of activities. Below are brief descriptions of each of them.
1. Test execution planning
Test planning can take many forms, depending on the criteria adopted and the design needs. The necessary factors to determine are:
- The type of tests planned.
- Identifying the requirements we want to test.
- Specifying how we want to check the requirements: automatic/manual.
- Allocation of resources.
- Who, where and when will execute the tests.
The solution we are offering has a functionality (Test Plan object provided by Xray) that allows collecting tests for a particular version, inside its test plan; a test plan thus created shows results of all activities, i.e. it displays current statuses of all running tests within the plan, regardless of the number of test iterations. This can be best illustrated by an example of created test plan view:
Test Plan Board
The “Test Plan” object provides:
A clear view of tests for a specific version with the most important information such as: person assigned to the tests, number of tests executed, most recent status (given in real time).
- The ability to run one or more Test Executions for all or just a portion of the tests included in this plan.
- Ability to assign one or more test plans to a specific version.
- Ability to monitor the progress of these tests within each test plan.
Additional functionality – the test repository – organizes tests hierarchically in folders; if multiple test cycles (test executions) are planned and run, the dashboard will group the results of those tests and show the current status for each; if you need to check the status of a specific subset of tests, simply select the appropriate folder:
Test Repository, Xray, Jira, Automated Testing, Transition Technologies PSC, Atlassian Partner, Jira solution
2. Test design – analysis, design and implementation of tests
Test analysis, design, and implementation is about moving from strategic, core assumptions to specific technical aspects. Simply put, test design requires an analysis of “what” and “how” to test, including risk-based testing.
Using the platform built in the Jira Software environment with Xray, you should create specific objects that contain the steps, test data, and expected result to illustrate how the tests are supposed to run. Individual test cases should be labeled (using labels), which will make it much easier to monitor progress later on.
Test design is:
- Organizing tests into test sets (Test Set).
- Writing out the test cases specified in the test conditions or documentation.
- Creating test data. Note!: giving objects labels will allow you to effectively monitor the progress of your work later on.
3. Test execution – running the tests
Test Execution is the activity that occurs between creating test scripts and reporting and analyzing test results.
Test Execution (visible in the environment as “Test Execution”) is an object that aggregates a user-defined collection of tests, and monitors and verifies that the tests work in the target context and environment.
With the “Test Execution” object, the user (e.g. Test Manager):
- assigns the test execution task for a particular group of tests (manual or automatic) to a tester; it is also possible to assign individual test executions to several testers.
- can see the progress of the tests through the continuously updated status of each test (e.g. “pass”, “fail”, “executing”, “to do”),
- has an overview of the test results generated by the program.
Overall Execution Status
Xray enables creating custom statuses for automated and manual tests (so you can e.g. create additional status names to distinguish e.g. failure causes – “Exception” caused by a code error/ “Not-implemented” – lack of implemented step). Test execution is complete when all tests verifying the specified system functionality are finished.
4. Monitoring progress
The platform built from the solutions we suggest offers many different ways to monitor the progress of the work. These include:
4.1 Test Sets Overview
Test Set Overview is a view dedicated to the Test Manager – it contains information on all tests assigned to a given function: their status, type (manual/automatic), and whether all functions are planned to be tested by a specific test set.
Test Set Overview
4.2 Test Execution Overview
Test Execution is the set of tests that is required to test a particular functionality. With this view, the manager has an overview of all tests for the particular system functionality that is being tested. It shows the types of tests; whether they are manual or automatic, to whom the execution of the selected test is assigned, to which test set the selected tests are assigned, and also indicates the statuses of each test shown (successful / failed / to do / or other statuses configurable by the administrator). With two simple clicks, you can access information about each step of the test, as well as the comments that were added to each test during its execution.
Test Execution Details
4.3 Checking the results and planning tests (Test Plan)
Another way to have a quick and clear view of tests being executed is to use the Test Plan board. Similarly to the „Test Set” object, it allows to filter tests based on e.g. the date of their creation, specific components or other information entered into the tests and groups them into a list. Apart from basic information, it also shows their statuses; the role of the Test Plan object is to enable quick retrieval of specific tests and checking and appropriate grouping of their results, as well as scheduling the execution of those tests that have not yet been launched.
Test Plan can be created for a given version or a given sprint so that you can track results related to different tests, e.g. you can create one Test Plan for regression testing and another for functional testing.
Test Plan Board
5. Generating reports and analyzing the results
Analyzing results and having a clear view of the desired information gathered in one place is one of the key features a good test management solution must have. Xray has built-in modules for creating reports and analyzing data so that a Product Owner, Scrum Master or Test Manager can quickly identify bottlenecks and improve the efficiency of their teams. Thanks to Jira Software’s flexibility, it is possible to configure additional, dedicated views (dashboards) that will gather the desired summaries in the form of charts.
5.1. Traceability Report
Often it is necessary to verify, even before – or during – the launch of tests, whether planned activities are consistent with the client’s/project’s requirements. Traceability Report shows the introduced requirements, tests and defects, collecting data in the form of graphs, which well reflect the connections between activities. This makes it easy to identify potential deficiencies and assess the “readiness” of the tests to be launched.
Traceability Report, source: Xray Ebook “How to measure and report quality metrics”
5.2. Overall Requirement Coverage Report
Estimation of time to complete testing is possible thanks to the “Overall Requirement Coverage Report”, which collects data related to the progress of testing – a glance at this report is enough to assess whether the product is ready for release. This module allows to check the test statuses per sprint and verify which testing areas need to be improved; Overall Requirement Coverage Report allows to narrow down the scope of tests to be checked by selecting specific requirements (requirement issues)/user stories /overarching tasks (epics)/ or by using more advanced JQL filtering.
- Helps estimate the time to release a product or individual parts of a product.
- Facilitates identification of areas for improvement.
- Gives QA Managers insight into the progress of testing.
- Allows communication of current test statuses to the development team so that everyone is up-to-date.
Overall Requirement Coverage Report, source: Xray Ebook “How to measure and report quality metrics”
5.3. Historical Requirement Coverage
Another way to check the progress is to use the Historical Requirement Coverage module – this is a functionality created for Test Managers, Product Managers, Project Managers and people on the customer side who are involved in the release of the application and want to have an overview of the workflow. Historical Requirement Coverage presents statuses of particular tests in the desired time range – so it is possible to trace the whole path from the beginning of the project or select e.g. the last month; the report illustrates the testing progress very well and helps to analyse the whole scope of activities and estimate whether the work will be completed within the assumed time.
- Generates information that provides evidence of test completion.
- It can serve as a document, showing the progress of the release.
- Great for companies that do testing/development for external parties.
Historical Requirement Coverage, source: Xray Ebook “How to measure and report quality metrics”
5.4. Test Plans Report
Test Plans Report is linked to Test Plan module and gives a progress overview of all tests that are accumulated in it (Test Plan) – it displays how many tests have what status in a clear table; it allows to estimate the progress of activities before the product release; it also gives a clear picture of defects that appear in the report in real time. From Test Plans Report you can go directly to the described defect and check its status.
*If a test fails, the tester creates a bug report in Jira so developers can see where the defect occurred.
With Test Plans Report you can analyze tests by version/sprint/selected label/status/components. You can also select the time interval within which the program should display data (by specifying the start date and end date).
Test Plans Report, source: Xray Ebook “How to measure and report quality metrics”
5.5. Test Executions Report
Test Executions Report aggregates all test executions in the context of test plans; provides insight into which tests have failed and need to be relaunched; also shows the progress of ongoing tests and their success rate. The user can generate the report by selecting only the test types of interest.
*For a test that failed, you can create and run a new test from Test Executions instead of changing its status.
- With the view in Test Executions Report you can:
- Check the progress of the tests.
- Determine the time spent performing the tests.
- Measure test progress – the report shows the percentage of test completion.
- Identify and eliminate process bottlenecks.
Test Execution Report, source: Xray Ebook “How to measure and report quality metrics”
5.6 Individually configurable dashboards
Atlassian’s solutions are highly flexible and highly configurable. Xray is a native application for Jira Software and, in addition to the usual test management functionality, it offers additional facilities for creating dashboards. Each user – tester, developer or Project Manager – involved in the testing process can have their own dashboard which will suit their needs – e.g. presenting all tasks (Jira tasks) they are working on and those they have planned to complete. The Project Manager, on the other hand, can get a view showing the work of the teams and information about the progress of testing in real-time. Setting up your own custom dashboard is easy with available widgets (such as the Test Evolution Gadget, which shows which tests succeeded and which failed).
Our specialists will advise you which dashboards will work well for each team on a daily basis and configure views to suit all needs.
Dashboard example, source: Xray Ebook “How to measure and report quality metrics”
The Jira +Xray+… automated testing platform. Benefits:
- Easy installation and configuration if your company already uses Jira Software.
- Fast and efficient test management.
- Centralized test management.
- Automatically generated results and reports.
- Test sessions executed automatically or according to schedule.
- Use of CI/CD in test execution.
Why should you cooperate with us?
The automated test management platform is a powerful environment that we customize to meet your needs. We have qualified teams that have experience in implementing this solution. We start our work from needs and requirements analysis in order to adjust particular ecosystem components to company’s requirements.
- We have over 10 years of experience.
- We are a Platinum Partner of Atlassian.
- We are a Partner of Xpand IT, the developer of the Xray application.
- We have implemented the platform for automated tests for organizations in such industries as: telecommunications, energy and industry.
- We have more than 50 application testers working in our company.
- We offer services related to Quality Assurance.
- Implementations for business customers are handled by: business analysts, Test Managers, Jira administrators and developers.
- We care about good partner relations with our customers.
If you had any questions while reading the description of the automated test management platform, feel free to contact us. The purpose of this article is to present the most important elements, features and advantages of the platform built with Jira Software + Xray + (…) tools. During conversations with our customers, we will be happy to discuss the various possibilities and clarify any doubts that may arise and together we will find the most optimal set of tools that will meet your company’s needs. You can rely on our experience and the expertise of TT PSC specialists.