User Acceptance Testing
Learn what is User Acceptance Testing (UAT), along with its definition, types, steps, and examples:
My rule number one when trying to understand a new concept is that: the name is always going to be relevant and mostly a literal meaning (in the technical context).
Finding out what that is, will give an initial understanding of it and help me to get started with.
We know what testing is, acceptance means approval or agreement. The user in the context of a software product is either the consumer of the software or the person who requested it to be built for him/her (client).
So, following my rule – the definition will be:
User Acceptance Testing (UAT), also known as beta or end-user testing, is defined as testing the software by the user or client to determine whether it can be accepted or not. This is the final testing performed once the functional, system and regression testing are completed.
The main purpose of this testing is to validate the software against the business requirements. This validation is carried out by the end users who are familiar with the business requirements.
UAT, alpha and beta testing are different types of acceptance testing.
As user acceptance test is the last testing that is carried out before the software goes live, obviously this is the last chance for the customer to test the software and measure if it is fit for the purpose.
This is typically the last step before the product goes live or before the delivery of the product is accepted. This is performed after the product itself is thoroughly tested (i.e after system testing).
Users or client – This could be either someone who is buying a product (in the case of commercial software) or someone who has had a software custom built through a software service provider or the end user if the software is made available to them ahead of the time and when their feedback is sought out.
The team can be comprised of beta testers or the customer should select UAT members internally from every group of the organization so that each and every user role can be tested accordingly.
Developers and functional testers are technical people who validate the software against the functional specifications. They interpret the requirements according to their knowledge and develop/test the software (here is the importance of domain knowledge).
This software is complete according to the functional specifications but there are some business requirements and processes that are known only to the end users are either missed to communicate or misinterpreted.
This testing plays an important role in validating if all the business requirements are fulfilled or not before releasing the software for market use. Use of live data and real use cases make this testing an important part of the release cycle.
Many businesses who suffered big losses due to post-release issues know the importance of a successful User Acceptance Test. The cost of fixing the defects after release is many times greater than fixing it before.
After performing loads of system, integration and regression testing one would wonder about the necessity of this testing. Actually speaking, this is the most important phase of the project as this is the time at which the users who are actually going to use the system would validate the system for its fit to purpose.
UAT is a test phase that largely depends on the perspective of the end users and the domain knowledge of a department that represents the end users.
As a matter of fact, it would really be helpful to the business teams, if they were involved in the project quite early, so that they can provide their views and contributions that would help in effective usage of the system in the real world.
The easiest way to understand this process is to think of this as an autonomous testing project – which means, it will have the plan, design and the execution phases.
The following are the pre-requisites before the planning phase begins:
#1) Gather the key Acceptance Criteria
In simple terms, Acceptance criteria is a list of things that are going to get evaluated before accepting the product.
These could be of 2 types:
(i) Application Functionality or Business Related
Ideally, all the key business functionality should get validated, but due to various reasons, including time, it is not practical to do it all. Therefore, a meeting or two with the client or the users who are going to be involved in this testing can give us an idea on how much testing is going to be involved and what aspects are going to be tested.
(ii) Contractual – We are not going to go into this and the involvement of the QA team in all this is almost nothing. The initial contract that gets drawn up even before the SDLC begins is reviewed and an agreement is reached upon whether all the aspects of the contract have been delivered or not.
We are going to focus only on the application functionality.
#2) Define the scope of QA involvement.
QA team’s role is one of the following:
(i) No Involvement – This is very rare.
(ii) Assist in this testing – Most common. In this case, our involvement could be training the UAT users on how to use the application and be on standby during this testing to make sure that we can help the users in case of any difficulty. Or in some cases, in addition to being on standby and assisting, we might share their responses and record the results or log bugs etc., while the users perform the actual testing.
(iii) Perform UAT and present Results – If this is the case, the users will point the areas of the AUT that they want to evaluate and the evaluation itself is performed by the QA team. Once done, the results are presented to the clients/users and they will make a decision on whether the results that they have in hand are sufficient or not and in accordance with their expectations in order to accept the AUT. The decision is never that of the QA team.
Similar to system testing, effective governance is enforced for UAT to ensure that strong quality gates along with the defined Entry and Exit criteria
The gathered acceptance criteria from the users are used in this step. Samples could look as shown below.
(These are excerpts from CSTE CBOK. This is one of the best references available about this testing.)
User Acceptance Testing Template:
Based on the criteria, we (QA team) give them the users a list of UAT test cases. These test cases are not different from our regular system test cases. They are just a subset as we test all of the application as opposed, just to the key functional areas.
In addition to these, the data, templates to record test results, administrative procedures, defect logging mechanism, etc., has to be in place before we move to the next phase.
Usually, when possible, this testing happens in a conference or a war room sort of a set up where the users, PM, QA team representatives all sit together for a day or two and work through all the acceptance test cases.
Or in case of the QA team performing the tests, we run the test cases on the AUT.
Once all the tests are run and the results are in hand, the Acceptance Decision is made. This is also called the Go/No-Go decision. If the users are satisfied it’s a Go, or else it’s a No-go.
Reaching the acceptance decision is typically the end of this phase.
Typically, the type of software tools that are used during this testing phase is similar to the tools used while performing functional testing.
As this phase involves validating the complete end to end flows of the application, it might be difficult to have one tool to automate this validation completely. However, to some extent, we would be able to leverage the automated scripts developed during system testing.
Similar to system testing, users would also use test management and defect management tool like QC, JIRA etc. These tools can be configured to cumulate data for the User Acceptance phase.
Though conventional methodologies such as specific business users performing UAT of the product is still relevant, in a truly global world like today, User Acceptance Testing sometimes has to involve varied customers across countries based on the product.
For Example, an e-commerce website would be used by customers across the globe. In scenarios like this, crowd testing would be the best viable option.
Crowd testing is a methodology where people from all over the world can participate and validate the usage of the product and give suggestions and recommendations.
Crowd testing platforms are built and are being used by many organizations now. A website or a product which needs to be crowd tested is hosted in the platform and the customers can nominate themselves to do the validation. The feedbacks provided are then analyzed and prioritized.
Crowd Testing methodology is proving to be more effective as the pulse of the customer across the globe can be easily understood.
The agile environment is more dynamic in nature. In an agile world, business users will be involved throughout the project sprints and the project would be enhanced based on the feedback loops from them.
At the beginning of the project, business users would be the key stakeholders to provide requirement thereby updating the product backlog. During the end of each sprint, business users would participate in the sprint demo and would be available for providing any feedbacks.
Moreover, a UAT phase would be planned before the sprint completion where the business users would do their validations.
The feedbacks which are received during sprint demo and sprint UAT, are collated and added back to the product backlog which is constantly reviewed and prioritized. Thus in an agile world, the business users are more close to the project and they evaluate the same for its use more frequently unlike the traditional waterfall projects.
A Typical UAT organization would have the following Roles and responsibilities. The UAT team would be supported by the project manager, development and testing teams based on their needs.
|Business Programme Manager||• Create and maintain Programme Delivery plan
• Review and Approve UAT Test Strategy and Plan
• Ensure the successful completion of the programme on schedule and budget
• Liaise with IT programme Manager and monitor the progress of the programme
• Work closely with business operations team and equip them for Day 1 operation
• Sign-off Business Requirement Document
• Review the e-learning course content
|• Programme progress report
• Weekly status report
|UAT Test Manager||• Crete UAT Strategy
• Ensure effective collaboration between IT and Business BA and PMO
• Participate in requirements walkthrough meetings
• Review Effort Estimation, Test Plan
• Ensure Requirement Traceability
• Drive metrics collection to quantify the benefits derived out of the updated testing methodology, tools and environment usage
|• Master Test Strategy
• Review & approve Test Scenarios
• Review & approve Test Cases
• Review & Approve Requirement Traceability Matrix
• Weekly Status report
|UAT Test Lead & Team||• Verify & Validate Business Requirement against Business process
• Estimation for UAT
• Create & Execute UAT test Plan
• Participate in requirement JAD session
• Prepare test scenarios, test cases and test data based on Business Process
• Maintain Traceability
• Execute test cases and prepare test logs
• Report defects in test management tool and manage them throughout their lifecycle
• Produce UAT End of test report
• Provide Business Readiness Support and Live proving
|• Test Log
• Weekly Status Report
• Defect Report
• Test Execution Metrics
• Test Summary Report
• Archived Reusable Test artifacts