The previous entry of my quality management series introduced quality management methods, which can be used during a software project to reduce or eliminate quality problems. Since this is the last part of that series, it is only just that I will describe a quality management method, which is used to verify that the delivered software fulfill the given requirements. The last stand of the quality management of software projects is called acceptance testing.
Acceptance testing has also got a juridical meaning. Typically project contracts state that a part of the price is paid after the software has passed acceptance testing. This practice is used as a carrot, which motivates the subcontractor to do its very best to fulfill its responsibilities. If the delivered software passes the acceptance testing phase, the customer of the software project has agreed that the subcontractor has fulfilled its responsibilities as agreed in the project contract. That is why this phase is very important to both parties.
It is often said that the goal of acceptance testing is to verify that the functions of the delivered software are working as the end users expect. This is a very important part of acceptance testing phase, but in some situations it simply is not enough. In these cases the acceptance of the software delivery might also depend on the outcome of
- Performance testing. The goal of performance testing is to ensure that the performance of the system is acceptable. It is not obligatory to do performance testing in the acceptance testing phase. Actually it is often done by the subcontractor before the software is delivered. However, if the software has got strict performance requirements, it is a good practice to do performance testing in this phase.
- Security testing. Security testing aims to verify that the delivered software does not contain security holes. The scope of the security testing depends from the type of the delivered software. A normal web application does not require so thorough security testing than for instance an internet banking application. Security testing does not have to performed during the acceptance testing. It is also a viable option to let the subcontractor to take care of it after the development phase of the application is over. In cases, where the application has got strict security requirements, it is often mandatory to do security testing in the acceptance testing phase.
The next question is how acceptance testing should be planned, and how the criteria for the acceptance of the software delivery should be selected. This process consists of following parts:
- Specifying the test cases. The test cases are often derived from the requirements of the software. Thus, the requirement specification or other available requirement documentation is used as an input of the planning process. In an ideal situation test designers have not participated in the previous testing activities of the software, because participation in testing activities often leads into biased understanding concerning the scope of the needed testing.
- Selecting the criteria for the acceptance of the delivery. This part is started by specifying severity levels for the possible findings. The principle is that the severity of each finding is specified by using these severity levels. Severity levels are used as a tool for deciding whether the software delivery can be accepted or not. It is a common practice to agree that only problems, which severity level exceeds a specified threshold level, can prevent the acceptance of the software delivery. Other problems are typically fixed later.
- Creating an acceptance test plan. A test plan must be created for following reasons: 1) Without a test plan users of the software do not know what they should be testing. 2) It is only fair (and often required by the project contract) that the acceptance criteria is clear to the customer and to the subcontractor.
After the test plan has been created, it is time to start the actual testing phase. The most important practices concerning this phase are given in following:
- Functions of the software are usually tested by actual end users, who have got the best understanding of their needs. If the end users cannot be used, the second best option is to use persons, who have got previous experience from software testing. Usually, using developers of the software is a bad idea, because their participation to the project often prevents them from being objective.
- If security or performance testing is needed, they can be done by using customer’s technical personnel or external consultants. The selection between these options often depends on the schedule of the acceptance testing and the technical knowledge of the customer’s employees.
- The results of each performed test run must be entered to a test report, which must also contain the outcome of the test run (passed or failed).
After the criteria for the acceptance of the software delivery are met, the acceptance testing phase is officially ended. The delivered software is deployed in the production environment, and the subcontractor is paid. It is time to concentrate on new challenges and new projects.