Application development has ceased to be the domain of large software houses and has become the domain of companies from various industries. Currently, most corporations that provide customers or employees with any IT solutions, create them themselves. Therefore, the growing demand is not so much for systems that enable software development as for solutions that automate and accelerate the entire process.

Atlassian changes, Jira migration, important information

One of the strategic stages of software development work is testing – every single “package” of code should be tested in order to identify and fix bugs as quickly as possible. In addition to automation, it’s important that the environment bridges the silos between the development and operations teams (in line with DevOps principles).

Both of these needs are met by Jira Software in conjunction with the Xray application and additional software development tools.

Jira and Xray environment, test tools, test apps, test platform

Xray – benefits

The application has been tested by thousands of users – according to the manufacturer, it is used by more than 5000 customers from 70 countries around the world.

  • Lower price than competitive applications. Recommended: Xray vs Zephyr
  • Compatibility with Atlassian tools – allows all tests to be managed as tickets (Jira issues) in Jira Software.
  • Provides test automation according to BDD methodology and integration with frameworks such as: JUnit, NUnit, Robot, Behave, Selenium
  • Xray interacts with DevOps solutions (e.g. Jenkins, Bamboo, GitLab) and fulfills the concept.
  • Advanced reporting, team-specific views (dashboards), gadgets.

Recommended- Xray Test Management – Product Walkthrough for Jira Server/DC:

Who is it for?

  • for any company that develops or collects software.
  • Software Development House, Telecommunications, Banking and Finance, Energy, Shipping, among others.

What is the solution made of? Main components

  • Jira Software
  • Xray
  • Cucumber
  • GitLab/ Bitbucket
  • Jenkins / Bamboo
    + additional tools for running specific types of tests (e.g. API, Mobile, UI, security, performance tests as well as reporting)

Individual tools may vary depending on what solutions a company uses.

What is it like? A sample test automation process and its key features

Jira and Xray, tests in Jira and Xray, automatic tests, manual tests

Example process for automatic tests


  • A common standard for developing manual and automated tests using Cucumber.
  • Links between requirements, tests, results, and defects.
  • Full CI/CD automation using Jenkins Pipeline, allows for earlier and faster testing (shift left).
  • Full integration with test automation tools, making it possible to run tests from within Xray and automatically retrieve results from their execution.
  • Easy test scheduling, report generation and creation of custom JIRA dashboards to track project progress.
  • Improved team communication.
  • Ability to integrate with other systems using REST API.

What does it enable?

Combining the functionality of Jira Software with selected applications enables preparation, execution, monitoring, management of automated tests and analysis of their results in a centralized way. The software development team receives a coherent environment for agile task management based on requests in Jira Software system and for automatic testing by using BDD (behavioural-driven development) methodology. The platform also includes Jenkins functionality for CI/CD activities, as well as additional plugins and scripts, thus streamlining the work of the entire team. Despite using many different tools, it is a coherent, integrated and comprehensive solution.

Test organization

Testing in SDLC infographic, Transition Technologies PSC, Atlassian Partner

Testing process

The testing process consists of several steps that include the following:

  1. Test execution planning
  2. Test design
  3. Running the tests
  4. Monitoring progress
  5. Generating reports and analyzing the results

The platform, built with Jira Software + Xray + Cucumber + Jenkins + (other test management apps), ensures that each step goes smoothly so that teams can run the entire process seamlessly without having to perform repetitive activities; automation mechanisms make the whole thing happen in a well-organized order of activities. Below are brief descriptions of each of them.

1. Test execution planning

Test planning can take many forms, depending on the criteria adopted and the design needs. The necessary factors to determine are:

  • The type of tests planned.
  • Identifying the requirements we want to test.
  • Specifying how we want to check the requirements: automatic/manual.
  • Allocation of resources.
  • Who, where and when will execute the tests.

The solution we are offering has a functionality (Test Plan object provided by Xray) that allows collecting tests for a particular version, inside its test plan; a test plan thus created shows results of all activities, i.e. it displays current statuses of all running tests within the plan, regardless of the number of test iterations. This can be best illustrated by an example of created test plan view:

Test Plan Board in Xray, Jira Xray, testing in Jira, Transition Technologies PSC, Atlassian Partner

Test Plan Board

The “Test Plan” object provides:

A clear view of tests for a specific version with the most important information such as: person assigned to the tests, number of tests executed, most recent status (given in real time).

  • The ability to run one or more Test Executions for all or just a portion of the tests included in this plan.
  • Ability to assign one or more test plans to a specific version.
  • Ability to monitor the progress of these tests within each test plan.

Additional functionality – the test repository – organizes tests hierarchically in folders; if multiple test cycles (test executions) are planned and run, the dashboard will group the results of those tests and show the current status for each; if you need to check the status of a specific subset of tests, simply select the appropriate folder:


Test Repository, Xray, Jira, Automated Testing, Transition Technologies PSC, Atlassian Partner, Jira solution

2. Test design – analysis, design and implementation of tests

Test analysis, design, and implementation is about moving from strategic, core assumptions to specific technical aspects. Simply put, test design requires an analysis of “what” and “how” to test, including risk-based testing.

Using the platform built in the Jira Software environment with Xray, you should create specific objects that contain the steps, test data, and expected result to illustrate how the tests are supposed to run. Individual test cases should be labeled (using labels), which will make it much easier to monitor progress later on.

Test design is:

  • Organizing tests into test sets (Test Set).
  • Writing out the test cases specified in the test conditions or documentation.
  • Creating test data. Note!: giving objects labels will allow you to effectively monitor the progress of your work later on.

3. Test execution – running the tests

Test Execution is the activity that occurs between creating test scripts and reporting and analyzing test results.

Test Execution (visible in the environment as “Test Execution”) is an object that aggregates a user-defined collection of tests, and monitors and verifies that the tests work in the target context and environment.

With the “Test Execution” object, the user (e.g. Test Manager):

  • assigns the test execution task for a particular group of tests (manual or automatic) to a tester; it is also possible to assign individual test executions to several testers.
  • can see the progress of the tests through the continuously updated status of each test (e.g. “pass”, “fail”, “executing”, “to do”),
  • has an overview of the test results generated by the program.

Overall Execution Status, Xray Jira, Automated Testing, Transition Technologies PS, Atlassian Partner Jira

Overall Execution Status

Xray enables creating custom statuses for automated and manual tests (so you can e.g. create additional status names to distinguish e.g. failure causes – “Exception” caused by a code error/ “Not-implemented” – lack of implemented step). Test execution is complete when all tests verifying the specified system functionality are finished.

4. Monitoring progress

The platform built from the solutions we suggest offers many different ways to monitor the progress of the work. These include:

4.1 Test Sets Overview

Test Set Overview is a view dedicated to the Test Manager – it contains information on all tests assigned to a given function: their status, type (manual/automatic), and whether all functions are planned to be tested by a specific test set.

Test Set Overview Xray, Xray and Jira, Automated Testing,Transition Technologies PSC, Atlassian Partner, Jira solution

Test Set Overview

4.2 Test Execution Overview

Test Execution is the set of tests that is required to test a particular functionality. With this view, the manager has an overview of all tests for the particular system functionality that is being tested. It shows the types of tests; whether they are manual or automatic, to whom the execution of the selected test is assigned, to which test set the selected tests are assigned, and also indicates the statuses of each test shown (successful / failed / to do / or other statuses configurable by the administrator). With two simple clicks, you can access information about each step of the test, as well as the comments that were added to each test during its execution.

Test Execution Details, Jira Xray, Jira solution, testing in Jira, Transition Technologies PSC, Atlassian Partner

Test Execution Details

4.3 Checking the results and planning tests (Test Plan)

Another way to have a quick and clear view of tests being executed is to use the Test Plan board. Similarly to the „Test Set” object, it allows to filter tests based on e.g. the date of their creation, specific components or other information entered into the tests and groups them into a list. Apart from basic information, it also shows their statuses; the role of the Test Plan object is to enable quick retrieval of specific tests and checking and appropriate grouping of their results, as well as scheduling the execution of those tests that have not yet been launched.

Test Plan can be created for a given version or a given sprint so that you can track results related to different tests, e.g. you can create one Test Plan for regression testing and another for functional testing.

Test Plan Board in Xray, Jira Xray, testing in Jira, Transition Technologies PSC, Atlassian Partner

Test Plan Board

5. Generating reports and analyzing the results

Analyzing results and having a clear view of the desired information gathered in one place is one of the key features a good test management solution must have. Xray has built-in modules for creating reports and analyzing data so that a Product Owner, Scrum Master or Test Manager can quickly identify bottlenecks and improve the efficiency of their teams. Thanks to Jira Software’s flexibility, it is possible to configure additional, dedicated views (dashboards) that will gather the desired summaries in the form of charts.

5.1. Traceability Report

Often it is necessary to verify, even before – or during – the launch of tests, whether planned activities are consistent with the client’s/project’s requirements. Traceability Report shows the introduced requirements, tests and defects, collecting data in the form of graphs, which well reflect the connections between activities. This makes it easy to identify potential deficiencies and assess the “readiness” of the tests to be launched.

Tracebility Report in Xray, Jira and Xray, Testing in Jira and Xray, Transition Technologies PSC, Atlassian Partner

Traceability Report, source: Xray Ebook “How to measure and report quality metrics”

5.2. Overall Requirement Coverage Report

Estimation of time to complete testing is possible thanks to the “Overall Requirement Coverage Report”, which collects data related to the progress of testing – a glance at this report is enough to assess whether the product is ready for release. This module allows to check the test statuses per sprint and verify which testing areas need to be improved; Overall Requirement Coverage Report allows to narrow down the scope of tests to be checked by selecting specific requirements (requirement issues)/user stories /overarching tasks (epics)/ or by using more advanced JQL filtering.

  • Helps estimate the time to release a product or individual parts of a product.
  • Facilitates identification of areas for improvement.
  • Gives QA Managers insight into the progress of testing.
  • Allows communication of current test statuses to the development team so that everyone is up-to-date.

Overall Requirement Coverage Report Xray Jira, testing in Jira, Jira solution, Transition Technologies PSC

Overall Requirement Coverage Report, source: Xray Ebook “How to measure and report quality metrics”

5.3. Historical Requirement Coverage

Another way to check the progress is to use the Historical Requirement Coverage module – this is a functionality created for Test Managers, Product Managers, Project Managers and people on the customer side who are involved in the release of the application and want to have an overview of the workflow. Historical Requirement Coverage presents statuses of particular tests in the desired time range – so it is possible to trace the whole path from the beginning of the project or select e.g. the last month; the report illustrates the testing progress very well and helps to analyse the whole scope of activities and estimate whether the work will be completed within the assumed time.

  • Generates information that provides evidence of test completion.
  • It can serve as a document, showing the progress of the release.
  • Great for companies that do testing/development for external parties.

Historical Requirement Coverage, source: Xray Ebook “How to measure and report quality metrics”

Historical Requirement Coverage, source: Xray Ebook “How to measure and report quality metrics”

5.4. Test Plans Report

Test Plans Report is linked to Test Plan module and gives a progress overview of all tests that are accumulated in it (Test Plan) – it displays how many tests have what status in a clear table; it allows to estimate the progress of activities before the product release; it also gives a clear picture of defects that appear in the report in real time. From Test Plans Report you can go directly to the described defect and check its status.

*If a test fails, the tester creates a bug report in Jira so developers can see where the defect occurred.

With Test Plans Report you can analyze tests by version/sprint/selected label/status/components. You can also select the time interval within which the program should display data (by specifying the start date and end date).

Test Plan Report in Xray and Jira, Transition Technologies PSC, Atlassian Partner, testing in Jira, Jira solution

Test Plans Report, source: Xray Ebook “How to measure and report quality metrics”

5.5. Test Executions Report

Test Executions Report aggregates all test executions in the context of test plans; provides insight into which tests have failed and need to be relaunched; also shows the progress of ongoing tests and their success rate. The user can generate the report by selecting only the test types of interest.

*For a test that failed, you can create and run a new test from Test Executions instead of changing its status.

  • With the view in Test Executions Report you can:
  • Check the progress of the tests.
  • Determine the time spent performing the tests.
  • Measure test progress – the report shows the percentage of test completion.
  • Identify and eliminate process bottlenecks.

Test Execution Report in Xray, Xray and Jira, Testing in Jira with Xray, Transition Technologies PSC, Atlassian partner

Test Execution Report, source: Xray Ebook “How to measure and report quality metrics”

5.6 Individually configurable dashboards

Atlassian’s solutions are highly flexible and highly configurable. Xray is a native application for Jira Software and, in addition to the usual test management functionality, it offers additional facilities for creating dashboards. Each user – tester, developer or Project Manager – involved in the testing process can have their own dashboard which will suit their needs – e.g. presenting all tasks (Jira tasks) they are working on and those they have planned to complete. The Project Manager, on the other hand, can get a view showing the work of the teams and information about the progress of testing in real-time. Setting up your own custom dashboard is easy with available widgets (such as the Test Evolution Gadget, which shows which tests succeeded and which failed).

Our specialists will advise you which dashboards will work well for each team on a daily basis and configure views to suit all needs.

Dasboards in Xray and Jira, testing in Jira, testing in Xray, Transition Technologies PSC, Atlassian Partner

Dashboard example, source: Xray Ebook “How to measure and report quality metrics”

The Jira +Xray+… automated testing platform. Benefits:

  • Easy installation and configuration if your company already uses Jira Software.
  • Fast and efficient test management.
  • Centralized test management.
  • Automatically generated results and reports.
  • Test sessions executed automatically or according to schedule.
  • Use of CI/CD in test execution.

Why should you cooperate with us?

The automated test management platform is a powerful environment that we customize to meet your needs. We have qualified teams that have experience in implementing this solution. We start our work from needs and requirements analysis in order to adjust particular ecosystem components to company’s requirements.

If you had any questions while reading the description of the automated test management platform, feel free to contact us. The purpose of this article is to present the most important elements, features and advantages of the platform built with Jira Software + Xray + (…) tools. During conversations with our customers, we will be happy to discuss the various possibilities and clarify any doubts that may arise and together we will find the most optimal set of tools that will meet your company’s needs. You can rely on our experience and the expertise of TT PSC specialists.


How useful was this post?

Click on a star to rate it!

Average rating 0 / 5. Vote count: 0

No votes so far! Be the first to rate this post.

If you violate the Regulations , your post will be deleted.

    _All posts in this category

    Document Management System at Jira - digital transformation for government, accounting, office and more

    Document System Management is the basis for digitizing business processes. See how it is possible to transfer and archive documents in the Jira…
    Read more

    Migrating Jira Server to Jira Cloud

    20 years after entering the IT market, Atlassian is still one of the world's leading providers of enterprise software solutions - from test…
    Read more

    Customer Portal in Jira Service Management: notifications, automation, knowledge base, SLA

    The Customer Portal is easy to use, and at the same time it offers advanced configuration possibilities. It is a solution dedicated to…
    Read more

    Jira Service Desk becomes Jira Service Management. New name, more possibilities.

    The year 2020, and especially its end, is the time of changes announced by the producer of Jira software. Recently, we have heard…
    Read more

    Jira Cloud vs Jira Server

    What does Jira Cloud have that Jira Server doesn’t? What benefits does it offer, what functionalities are worth noting, and do both these…
    Read more

    BigPicture 8.0 – a new, better version of the PMO add-on for Jira Software

    BigPicture by SoftwarePlant, a new version of the project management and PMO development addendum in Jira, is coming. What changes will it bring?…
    Read more

    Agility. All you need to know about the agile methodology

    What actually is agile? How did it all start? What are the principles that characterise this working method, how does it differ from…
    Read more

    How do Jira and Confluence help you build your Information Security Management System?

    The turn of June and July 2019 was an intense time for us. Due to the increasingly dynamic development of companies in the…
    Read more

    What software should you choose to work remotely?

    At a time of the coronavirus threat, the ability to work from home is no longer a privilege, but a necessity. The employees…
    Read more

    Jira Core – a tool for effective work and project management for the marketing team

    A good marketing team consists of people who have different, complementary skills: content, SEO, paid campaigns specialists, graphic designers, developers, analysts, PR or…
    Read more

    Atlassian Summit 2020 – we’ll be there!

    After a two-year break we return to the Atlassian Summit as sponsors of the event. In Las Vegas, Tomasz Pabich, Project Manager, will…
    Read more

    Jira as CRM system

    Jira by Atlassian is software well-known all over the world; it is associated mainly with solutions for developer teams. The tool can be…
    Read more

    Program for business trip expense reports – Jira

    Settling local and overseas business trip expenses is a process that companies would like to go through in possibly the least complicated and…
    Read more

    Logging work time - timesheets in the Jira system

    Timesheet processing, time tracking in Jira? No problem! If your company uses software by Atlassian, you don't have to invest in new solutions…
    Read more

    Vacation management in Jira. Explore the possibilities of Vacation Manager

    A vacation management program that is easy to use, affordable, and enables the finalization of the entire process without the need to print…
    Read more

    Atlassian Open 2019 – news, thoughts

    A strong emphasis on cloud solutions, customer-focused narration as well as promoting the idea of being “open”. What happened during Atlassian Open 2019…
    Read more

    Confluence by Atlassian – opinions and the most important functionalities from users' point of view

    Based on implementation in Transition Technologies Capital Group currently hires nearly 1000 employees working in broadly understood advanced technology field.
    Read more

    Atlassian Summit 2019 – summary, trends, changes

    Atlassian Summit is an annual even organized by an Australian producer of software for work management, among which the most popular ones are…
    Read more

    DevOps – from trend to standard

    DevOps idea appeared in IT field already in 2009 and since then, it has been systematically entering global business world as a method…
    Read more

    Jira Data Center – solution which does not let down. What is it and why is it worth investing in?

    Regardless of implemented tools, every company needs an environment where they can embed their own processes and manage them. A perfect tool for…
    Read more

    _Let’s get in touch

    Contact us