Most modern manufacturing companies use PLM systems to document and manage product data. PLM systems support many important operations in a company: first of all – administration of CAD files, documents, and data related to change management. All these functions are permanently updated with new requirements. A small error that occurs in one area can negatively affect whole processes. This is why software verification and quality control are so important. In this article, we will present the benefits of PLM software testing.
PLM software testing from a specialist’s perspective
“Engaging QA specialists in the project is just an additional cost” – unfortunately, we often hear such opinion from people responsible for PLM system in companies. Software testing is still treated as an unpleasant necessity. Hardly anyone pays attention to real benefits of this process. However, during the implementation or development of PLM software, lack of testing is only a false economy.
Customisation – the answer to an organisation’s specific needs
PLM is the process of managing the end-to-end product lifecycle, from launch, through development, maturity, and decline. Implementing a PLM system is an investment that allows project teams to work faster and more accurately and to efficiently share information about the current status of the project.
PLM systems producers guarantee that their OOTB (out of the box) PLM software will be ready to use immediately after installation. And that’s how it is, but it requires basic configuration. Only then such a system will facilitate and optimise the work in an organisation implementing PLM system.
“In fact, such implementation based on configuring should take a few weeks to a few months for SMB companies (Small and Medium-Sized Business). This implies conducting several workshops to identify aspects related to the configuration of the data model, user accesses based on defined roles and the alignment of digital business processes with company policies. In comparison, in the case of Enterprise class companies, the realisation of requirements and implementation of a PLM system connected with the Digital Thread paradigm involves several years.”
-explains Leszek Zborowski , Business Unit Manager at TT PSC, responsible for the PLM area. A further change is customisation. It covers everything apart from configuration and requires modification or extension of the source code (new actions for users or integration with other systems used by the company, which cooperate with PLM). You can read more about the differences related to PLM systems implementation in the article:
Customisation versus testing
Verifying all the changes is crucial to the proper functioning of the entire system. PLM Software testing is used to verify its correct operation and to gather key information about it. The greater the correspondence between verification and validation tests, the better the quality of the application will be. It means less risk of errors, faults, and downtime in the future. And if a failure does occur, by using a ready-made set of tests to check the basic functionality of the system, you can make sure that the system works perfectly after repair.
As part of basic testing activities, we prepare test plans and describe key scenarios. We execute tests, document their results, and report errors. Thanks to a well-organised testing process, system imperfections are detected at the earliest possible stage. What is more, correction of defects or removing inconsistencies in documentation is connected with relatively low expenditure of work and resources. This allows us to significantly optimise costs.
Why is PLM software testing so important?
PLM systems are complex software, consisting of many interdependent modules. To guarantee the quality expected by the customer, testing must be divided into several levels, such as:
- Unit tests – verify any method, function, class, module, or component;
- Module integration tests – verify that the components interact correctly with each other;
- System tests – check if the system works as a whole. At this level e2e (end-to-end) tests are conducted, which check the compliance of software operation with requirements and specifications.
Testing of both individual parts and the software as a whole is essential to:
- check if everything works correctly;
- improve places where tests showed imperfections;
- anticipate the possibility of future problems.
PLM systems are hundreds of code lines, written by many developers, often working in distributed teams. This code is used by many different components. It is characterized by complex communication and a multi-level compilation system. Therefore, the key to success is testing, which ensures that despite the number of processes, the system works as it should. This significantly reduces the risk of future potential errors and the need to fix them. The later a bug is discovered, the higher the cost of fixing.
Testing and access control and data security
PLM systems have very complex access control rules. Some data will be accessible for an engineer, others for a production worker, and still others for managers of different levels. So it is important to check the access very carefully during testing the whole PLM process so that everyone can see only the data they are entitled to. PLM systems provide a very sophisticated access control structure, but it takes a team of testers to make everything work properly.
Performance can cause problems. Test it!
As part of PLM software testing, it is also necessary to check non-functional requirements of our system. It is important to make sure that a sudden increase in the number of users or actions does not make the system unavailable and financial losses for the company. As part of the system performance tests, it is examined how long the system performs individual actions, especially with key or long-lasting tasks.
For example, as part of performance testing, we can check how the login operation works for different numbers of users. Let’s use the code from automatic login tests, written earlier in Selenium, and simulate the different number of users logging in, using JMeter.
We can visualise the results of such load tests on the PLM system by sending the data from JMeter to InfluxDB (a toolkit for time series), Chronograf (a tool providing a dashboard for data from InfluxDB) and Grafana (an application for analysis and interactive visualisation). In this way, we will present the number of user logins per second, for example, for the last 5 minutes.
The data obtained as a result of such tests can also be help make decisions about the PLM system infrastructure. In this way, it is possible to find out how many users can work on one server, or how to scale the number of nodes, depending on the number of users.
PLM software testing as part of business systems integration
PLM software usually does not work independently, but cooperates with other systems – it receives from their actions and then sends them further. In this way, using dedicated architecture and integration technologies, we create integrations between systems.
PLM, ERP, IoT
An example can be integration of PLM solution with SAP (ERP system allowing to direct, plan, manage and control all activities in a company). It is crucial for full utilisation of PLM functions. Also it is worth mentioning the integration of PLM with IoT systems (as an example with PTC solutions, integration between Windchill system and ThingWorx platform). Internet of Things platforms have been created to connect, aggregate, analyse and display data – from devices spread around the world, machines on the production line in the factory, through other corporate systems. Very often IoT and PLM systems support each other.
Integration that worries the whole team
Checking of integrations business systems with PLM platform is a task for the testing department. The synergy of many systems can cause problems, and fixing integration defects is crucial for the software to work properly. We can verify whether the systems have been integrated correctly by performing end-to-end tests (which we will discuss in more detail later in this article). Thanks to them it is possible to check how the PLM system behaves from the very beginning till the end of its work.
Upgrade that can spoil a working system
The pace of technology development and the number of new IT solutions make PLM system upgrade inevitable. During such a process, systematic tests are necessary. They are a guarantee that after the upgrade, everything will work correctly. An example can be the recent upgrade of the Windchill system from version 11.0 to 12.0. After such a significant update, it is necessary to test the system as a whole and check whether any new defects have appeared. As the PLM system is very complex, it is possible to do the tests manually (it will take a lot of time), using a regression test set, but it is much faster and more efficient to automate them.
Continuous integration and contiunous delivery for efficient deployment
Changes in PLM system are added not only during the upgrade. In some projects, they are an everyday part of code development with new functionalities (within continuous integration and continuous delivery). In such situations, it is worth using automation with a tool like Jenkins. CI/CD pipeline automates the entire software delivery process. It builds code, runs tests, and implements the new version of the application. Automated pipelines remove errors that could arise if the deployment process was performed manually and a developer or DevOps specialist accidentally failed to perform some necessary action.
Among the many stages of testing performed by the Jenkins tool (pipeline), is the checking of all software at all levels of testing (unit, module integration, system). If changes caused any defects in the PLM system, it will be visible in the Jenkins regression report. It is possible to take corrective steps quickly. The faster a software defect is detected, the lower the cost of fixing it.
Benefits of PLM software testing
Thanks to the various types of tests (performed as part of PLM software development) the client gains a number of benefits that later translate into profits for the company.
1) Risk identification
As part of the testing activity, it is possible to identify and manage risks. Thanks to that, the customer is aware of what can cause trouble in PLM system in the future and what should be done to avoid potential pitfalls or deal with the trouble effectively. PLM, as well as other Enterprise systems, are highly developed and complex, so the risk of error is high. Proper preparation will prevent such a situation or quickly eliminate its effects.
2) PLM software testing and meeting customer expectations
The main role of all testing activities is to improve software quality. It is not only a significant number of cases written, tests performed or bugs found. It is also determining the extent to which a given module or process meets the needs and expectations of the customer. And this is the most important thing in the whole process of software development. Additionally, in this way it is possible to significantly reduce the occurrence of change requests, which are submitted by the client when they conclude that something should work differently, have more functionality, etc.
3) Identification of defects and failures
Two issues need to be distinguished – what the PLM system is and whether it works as the customer expected. Checking all the requirements step by step, catching the gaps or inconsistencies between them – all this is achievable in the testing process, during writing test cases. An experienced tester will catch any inconsistencies and deficiencies, which are verified and then completed or changed.
4) Compliance with required legal regulations
Testing makes it easier to achieve safety compliance. In industry, especially in sectors that are critical to human life and health, there are a multitude of regulations (EU, national, quality) and good practices that must be met for a system to be recognised as safe and allowed to operate.
5) Cost savings
Thanks to a well-organised testing process, system errors are identified at an early stage of software development. Correcting a defect or removing inconsistencies in the documentation entails a relatively low expenditure of forces and resources. Every project connected with software development can be implemented without tests. However, it is a matter of business risk that an organisation is able to take. Unfortunately, the scenario of system incompatibility with the customer’s expectations is very probable, and the costs of error repair will make such an approach unprofitable. Therefore, it is worth bearing in mind that the later a bug is discovered, the more resources are needed to fix it.
The whole PLM software testing process is organised in such a way that the customer receives real-time information about what is happening, why, and how potential problems can be prevented. The customer is aware which elements will be tested at what time. In addition project milestones are defined. The contractor also knows when he can expect the whole process to be completed and the PLM system to be ready for use. Therefore, transparency and measurability of IT processes at each stage is an important aspect from the customer’s point of view
6) Increasing trust between the customer and the system provider
Transparency is closely related to trust. If the customer knows what and why is going on in the software development process and all the steps are consulted with him, the trust in the PLM solution as well as in the supplier increases. Good relations and trust are the basis of successful cooperation that results in a valuable product that meets the customer’s expectations.
Transparency is closely related to trust. If the customer knows what and why is going on in the software development process and all the steps are consulted with him, the trust in the PLM solution as well as in the supplier increases. Good relations and trust are the basis of successful cooperation that results in a valuable product that meets the customer’s expectations.
Test automation in a business context
PLM systems are very complex, so testing absolutely all elements manually would take a long time and require a large team of testers. This is where automation comes to the rescue. Covering the most error-prone paths and the most frequently performed actions by the system users allows to avoid critical errors.
The Behavior-Driven Development (BDD) approach works very well here: it is worth writing PLM system tests using Cucumber + Java tools, using the relatively easy syntax of the Gherkin language, where we have “given” (initial context), “when” (description of the occurring event) and “then” (confirmation of the occurrence of the expected result). By using the BDD process to create test cases, customer confidence increases. Describing the use case from the end user’s perspective helps to understand the customer’s needs. Thanks to that, he gets what he needs and what he paid for. PLM system tested in this way delivers real business value.
PLM – test automation
When automating regression tests that are executed periodically, testers do not have to perform the same repetitive operations over and over again. Detailed reports with test execution results provide specific information about what stopped working. What is more – tests are executed in the same way every time, which eliminates possible mistakes (in case of manual testing of software).
In the long run, time savings translate into cost reductions. Automatic tests are much faster than manual tests and do not require additional work of many testers as during
the tedious process of manual testing. A set of tests can be run in the evening and conclusions can be drawn the next day.
Example of automated tests in PLM systems is not only regression but also smoke or sanity tests, like those which initially check if the application/environment is at all suitable for deeper exploration by testers. Thanks to automatic tests we receive information whether the system is ready for thorough testing without involving employees.
End-to-end testing – we test your system from the beginning to the end.
Automation is an indispensable element in all kinds of end-to-end tests in PLM. They are long-lasting and labour-intensive, because they cover the whole business process, including cooperating systems and often a large number of input data.
The work on end-to-end testing starts with a thorough specification of the test scope with the customer. We create test scenarios under BDD principles. Then we prepare documentation for each step, including a description of verifications to be checked in the system. Such a prepared set of data (test scenarios in Gherkin and documentation) is given to the client once again for verification and acceptance to make sure that the topics are well understood.
Accepted test cases for end-to-end testing are placed in the code repository. The automation of each step is then started using Java and JUnit or Selenium. When the automation process is complete, Jenkins is configured to run the automated tests as the test environment is built.
PLM software testing – permanent quality assurance
The development of complex software requires the creation, implementation, and maintenance of a large number of test cases. For over 15 years we have been systematically implementing test processes for medium-sized and global companies. Besides the practical experience we have gained, we always try to incorporate new technology solutions into our services. We have completed more than 1000 Product Lifecycle Management projects for customers from all over the world and in various industries: aerospace & defence, automotive, customer goods, energy, telco, and machine builders.
In our projects we implement automation, supported by continuous integration and continuous delivery. So that all errors are detected as soon as possible and the PLM software we develop is under constant control of a set of tests, with the lowest possible cost for the client and involvement of employees.
If you have questions about how to ensure software quality or need support for the testing process in your company, write to us or check our PLM software testing offer.