UAT guide

The Ultimate Guide to User Acceptance Testing​ (UAT)

Introduction to UAT

User Acceptance Testing (UAT) is inevitably near the end of any software change cycle, and it performs a number of key objectives in the overall software quality process. It serves to ensure that the software is usable, safe, understood, and meets the objectives laid out in the first instance. 

  The word ‘acceptance’ has become less important in most current forms of UAT as it is not typically used to ‘accept’ the software or system against a set of acceptance criteria, more it is an additional set of checks and, at the same time provides familiarization and training so that the implementation is smooth and successful.

Organizing, managing, and executing any type of testing or testing project can be a substantial and daunting task, and UAT is no exception. For every project that UAT is implemented in, there are unique challenges and factors to consider. However, the effort is worth it, and in many cases, essential for the success of the project and saves time and money in the long run.

However, these results are entirely dependent on the appropriate implementation of testing and good input from users. Without this, the software’s residual errors will not be discovered, negatively impacting the change of systems or causing costly disruption and discontent internally and externally.

Any UAT phase is likely to be the most costly type of testing undertaken due to the number of people, timescales, and repeated cycles. Hence, it is well worth optimizing this activity to get the best outcome and do it in the most efficient and cost-effective way.

This handbook looks at the potential activities that might apply to a UAT project. It provides the knowledge for putting in place frameworks that give a project the best chance of success and maximize positive outcomes from each activity and the overall project.

As with any testing process, not all phases need to be implemented within every project, but each should be considered to achieve the best possible outcome.

User Acceptance Testing Scope

Initially, you need to determine the scope of your UAT testing through three stages.

Determine timescales

Whilst implemented at the end of projects; it is advisable to start early and engage with all teams involved to ensure UAT is on the radar and considered in their planning. Often, departments consider the challenges and defects that will arise but do not realize that UAT identifies defects for fixing and overcoming.

When determining timescales, you also need to consider where resources will need to be implemented and the overall strategy which the UAT is being implemented to achieve. Part of the planning process will involve determining why the project is being carried out and the benefit the organization hopes to gain from the change.

Once the benefit is realized, it must be effectively communicated to all UAT participants so they understand what they need to look for while testing to achieve success.

Things to consider

• What does the whole project look like?
• What are the business’s goals?
• What previous phases will the project go through?
• Who is involved in the whole project?
• The methodology being implemented, and where UAT will be used?
• The strategy and content used in previous test phases, and what can be reused?
• The resources set aside for fixing issues and bugs
• The planned date to go live, and is it flexible to allow changes?
• The extent of the changes, and how will they impact the business?
• Are there any known challenges and issues the change is bringing?

Analyse the change

It’s necessary to determine the scope of the project to understand the changes that the business is hoping the software will bring and the resources and time needed to test it sufficiently.

When analyzing the change, if it is a unique software a business is developing, you can communicate expectations from previous teams and phases, including development and quality assurance, to understand the software’s expectations.

If it is a cloud-based software or package being implemented, obtain data from other users via forums and other channels to gauge the scale of the testing and resources needed.

Things to consider

• Significance of change
• The scale of the change
• Systems which will be impacted
• Business teams affected and the number of people who will be using the software
• The existing assets affected
• Input from the provider and reviews from clients
• Determine the impact of changes and risks.

By conversing with those involved with the system, that being the system architects if developing software internally or with the software provider, determine the impact of changes on the business.

It is always important to remember that whilst the scale of a project and impacts can be closely related; it is not for certain that small changes only result in small impacts but rather result in disproportionately large impacts to the business.

Therefore, assessing each change accordingly and determining its impact is important to efficiently plan User Acceptance Testing.

Things to consider

• Impact to current systems
• Impact on the overall business and how affected systems can impact further impact the business
• Impact to IT and other infrastructure
• The risk to the business from undetected errors
• Risk of timescales not being met
• The scope of the testing required

Strategy

Once a plan is put in motion, creating a comprehensive and holistic strategy is important for successful UAT implementation.

Refer to test policies.

Different organizations have different cultures and business practices; some may have developed from historic need and others from dynamic approaches and may have pre-existing test policies.

Having these, however, do not mean they will be appropriate for every testing, and reviewing them to determine if they can be implemented needs to be determined early on. However, if you’re going to adapt these policies for the test, this also needs to be outlined early in the process.

If your organization does not have test policies, take the opportunity to outline a policy for future guidance and best practice but be mindful of how testing is dynamic and policies may not fit every scenario.

Things to consider.

• Does your already existing policy for UAT work for this project?
• Does the policy need altering?
• Does a policy need to be written?
• If you’re not following the policy, outline why not and get secondary feedback.
• How is the project risk impacted by policies?
• Determine the strategy

Once the plan and initial preparations have been completed, determine a strategy that accounts for everything discovered from the preparations.

Things to consider

• Do you need a fully test case driven, documented approach?
• How much exploratory testing is appropriate?
• What resources will be used and in what way?
• Who will write the test cases?
• Level of detail needed vs SME knowledge
• Are detailed test scripts required?
• Who will carry out the tests?
• How will they do them?
• How many rounds of testing will be required?
• Will it all be manual?
• Is there an exploratory testing phase?
• Can you build in automation as you go?
• How will you determine readiness?
• What metrics will guide you in this and how will you gather these along the way?
• Determine the UAT tools to be used

As with every test, using specific technology can make life considerably easier. There are 4 key areas to consider

4 KEY AREAS TO CONSIDER

1.Project management

1) You may need a platform to manage the projects’ assets combined with communication, resources assignment and scheduling. Fundamentally, it will act as a test management solution to support the users and the project’s needs. 
The working and strategy undergoing may have different facets depending on the approach being taken. For example, you may have ‘issues’ that do not fall under ‘defects’ or ‘feedback’ segmented by positive, negative and neutral results. Regardless of your approach, your test management platform will need to be designed to support the style of UAT being planned and the users’ needs.

2. Capturing User actions

 Technology capturing user actions unobtrusively and accurately with little to no effort, especially if you plan on implementing exploratory testing. Historically, the capturing of results is one of the most frustrating aspects of UAT, with users getting tired of documenting via print screen and pasting to Word when errors occur and manually detailing what happened. 
This historic method is unreliable as it depends on users being honest and meticulous in documenting the process. There are inexpensive solutions for capturing the process, which can be easily implemented and take the pressure off users.

3.Automation

 UAT needs to be repeated to ensure the results are as accurate as possible; however, after several testing cycles, automation can be implemented to seamlessly run the software, checking for issues that manual testing has overlooked.

4. Test Data Management

The testing process needs a test environment and ideally consistent data to go with it. Test data management tools help handle the data collected, ensuring it is easy to validate when integrated with automation and capturing user actions technology.

Factors to consider for technology

Before implementing technology for the project, there are factors that need to be considered:

Factors to consider

• How will you organise test assets?
• Requirements
• Test cases
• Test scripts
• Test results
• Issue Reports
• Defects

Surveys

• How will you ensure traceability for test coverage?
• How will tasks be allocated?
• How will users know what to do and when to do it?
• How will progress and KPI’s be measured?
• How will test results be captured and analyzed?
• Can aspects be automated?
• What automation solution fits team, skills, timescales, approach and business objectives?
• Are there existing tools or will you need to acquire them?
• How will you create consistent test environments?
• How will you communicate with product teams, business users, stakeholders and buy-ins?

Planning

Once the scope and strategy have been determined, map it out by a time unit that best suits the UAT manager, users (participants) and business objectives. It will need to be flexible to change as factors will inevitably impact it.

• Outline the test plan
• Outline the strategy into weekly segments
• How realistic is this approach?
• What are the risks?
• Where are the compromises?
• Get feedback from product teams, business users and stakeholders for the time process.
• Determine resources needed
• Assess the resources needed
• Where do they need to be implemented?
• Skills associated with the resources
• How available and accessible are the resources
• Can they be supplemented?
• Assess the role and impact of interfaces and other systems

Any capable and meaningful application and software undergoing UAT will connect to other applications, internally and potentially externally. This can complicate the change, which extends the coverage and nature of test cases and resources you may need to assess the impact of changes as many businesses operate over several integrated solutions.

Things to consider

• Will the impact on other software mean more teams need to be involved?
• Do you have existing linked test environments to support this end-to-end testing?
• The impact of changes on test cases and data

Preparation and Initiation

There are many things to consider in developing a plan; this covers the majority of considerations but may vary from project to project.

Understand and capture business processes

• Know how the business operates to be able to test a change in an area. Is this information to be gained via documentation or key users?
• The business processes may change, and as the UAT is implemented in a separate environment, you will need to know the new business process changes, including inputs, outputs, and outcomes.
• You may want to document this understanding and associated data capture using a classic structure. However, the downside of this is it does not lend itself to multiple scenarios. In this case, you will need to support any process definition with the data variations that are required, both for input and output. 
• If the project is largely revisions and additions to an already existing system, you may want to capture the existing process from production if the knowledge is not already available. You can use this to create text style instruction cases or even how-to videos, which users can refer to alongside data scenarios.

The UAT management tool should be used here to provide a) a repository for these assets and b) the ability to organize and hence reuse them.

Create or adapt test cases

It is important to look at Test Case coverage, identifying if you have covered all the processes and scenarios that need to be tested. This analysis is critical, and it may require considerable input from business users to create.

Some organizations opt to delegate this task entirely to the users, which is often the best route; however, if you take this approach, you will need inspection and reconciliation to ensure that you have achieved the necessary coverage level.

Check coverage

Two simple things you’ll need to consider for the coverage:

1. Do the test cases provide full coverage of the test scope?
2. Maintain traceability between changes/requirements and test cases

Detail in test cases may vary

It is possible to perform effective UAT with very lightweight cases, but this depends on the system knowledge level of the users performing tests. If you don’t require extensive detail, there is no reason to put it in the test case.

Historically, detailed test cases and scripts also helped define what the tester was doing so any issues could be described in the context of that series of events. Modern UAT, however, uses capturing technology to show precisely what the user was doing and document their concerns and feedback.

Regardless of the approach, your UAT will take, test cases have become valuable assets that we want to re-use, and hence they need to be stored and organized in a way that supports that.

Things to consider

• Organize test case repository and classification
• Depending on who is preparing test cases, what help and guidance do they need?
• Amend existing test cases, if available, to match the changes/requirements
• Create test cases where they are needed
• Consider the sequence and dependencies
• Create test scripts

The same principles as outlined for test cases apply to any scripts which take instructions to the next level. The coverage and level of detail in relation to the skills and knowledge of the persons executing the tests. A purpose-designed organizational repository that makes scripts accessible, maintainable, referenceable, and reusable is fundamental for increasing organizational efficiency and avoiding frustration.

Things to consider

• Create a standard script template if there is not an existing one.
• Consider if detailed scripts for each test case are required.
• Base the scripts on captured existing processes, if available.
• The UAT management platform should support a script repository and linked to test cases.
• Consider what format to hold the scripts in.
• Are training aids required to assist in script execution?Determine test data needs and strategy

There are two aspects of data that need to be initially considered:

Data aspects to consider

1. What data is needed to be processed
2. What supporting data is required

You’ll need to consider where the required data will come from, and this is inevitably based on what data is already available. This data needs to be treated as an asset; you should aim not to rebuild it or recreate it every time you go through a test cycle. Automation is sometimes used in this scenario to automatically create the data needed for subsequent testing and to carry it out when it is difficult to manage the condition of supporting data. However, the use of automation can be avoided in these scenarios by using a solution to manage data, this will allow for automation to be implemented in other areas which would benefit from it more.  

You can avoid this by utilizing available technology and virtualized environments, which will save considerable time and effort.

Things to consider

• Plan where and how to provide test environments(s).
• Can virtual environments be used?
• Consider the backup and reset strategy for the data.
• What supporting technology is required?
• You will probably need to coordinate with IT, infrastructure teams.
• Test the test scripts

If you have created test scripts to be given to the users, then you will need to check, for at least some of them, that they are workable and understandable.  If the users have created them you probably won’t need to.

Things to consider

• Check both the content and format
• Liaise with users to check their needs and the script or the scripts they have developed

Risk and priority assessment

There are times when a project doesn’t go to plan. To successfully achieve a competent and effective UAT, it is important to understand the critical testing areas and those that can be deemed as lower-risk. If the situation arose, it could be cut from the test. Through understanding these risks, you will need to prioritize important tests first.

Things to consider

• Which tests are most important?
• Are any tests able to be skipped?

Detailed Plan

The nature of the strategy you are taking will determine the level of detail required.  If you are allocating test tasks at an individual or perhaps department level, you will need a granular plan combining the task, test case, user, and timescale.

 If you’re adopting a looser ‘crowd’ style strategy, the allocation of individual tasks becomes less of an issue in the first instance, but the monitoring of who has done what (or has not been done) becomes more important.

The crowd approach broadly involves making test cases available to anyone who has the time and skills to carry them out and then monitoring the results with a pre-determined execution and pass level. The advantages of this are that it is more dynamic and resource availability determines the progress with a greater sense of visibility of the whole project creating greater accountability.

Collaborate with User department heads

Collaboration will allow for faster access to resources as and when needed through early planning of who will do what, when and what is needed.

Things to consider

• Publish goals, approach and timescales company-wide
• Determine skills, names and availability of seconded resources
• Establish communication methods and frequency
• Use a central platform to enhance visibility and remove the reporting burden.
• Plan the sequence and dependencies of tests.
• Allocate time, resources, and areas for exploratory testing.
• Consider timescales.
• Determine resources.
• Plan communication and access to the central platform.

Exploratory testing

Exploratory testing is essential for ensuring high quality and successful UAT. The UAT’s value as a whole may depend on many factors, not least the influence you have over the final users. The less influence, the more exploratory testing you should conduct to understand the results and their accuracy better.

For example, suppose you have a small group of in-house users who are dedicated and will be committed to following the UAT process rigorously. In that case, exploratory testing will not be a major factor or relevant to your test. However, suppose the application is to be rolled out on a mass scale publicly and contains sensitive or valuable information. In that case, internal exploratory testing may be critical for ensuring your testing process is functional.

Exploratory testing may have been included in earlier quality assurance phases if the software is being developed internally. Still, it is also good practice for the end-user to test the software too before distribution.

Communicate the plan to managers and individuals

• Hold briefing session as a group
• Inform individual business units
• Consider how they will be notified of changes and progress
• Provide access to key stakeholders to central dashboards and metrics

Execute the plan

Depending on the team’s level of skills and knowledge, it may be beneficial to have workshops before executing the plan to get the team ready, understand the process, and mitigate their issues and concerns. They will need to understand how the process works and their expectations and set the standards for producing and gathering feedback.

The feedback will need to be specified to the UAT’s objective; if you are interested in understanding issues, you will need to communicate this efficiently to all participating stakeholders. Using a capture tool is essential to make the gathering of this data accurate and support the users’ experience.

Competent capturing tools should capture the inputs and screenshots against the target applications with the ability for users to add comments, feedback, and issue information. UAT best practices have managers seeking feedback beyond the initial issue reports; encouraging this helps improve engagement and ownership whilst also reducing the possibility of conflict with development and configuration teams.

Surveys are another way for increasing user engagement by providing the opportunity for a broader perspective on acceptance and readiness. It also helps highlight the differences between broadly accepted concerns against individual ones which do not have broad support. Ideally, the UAT management platform will support the concept of user surveys and sharing of the results.

Train users

It is important to train all users to ensure the success of the UAT.

Things to consider

• Teach the testing process you want
• How to access the central system
• How to access the work allocation
• How to execute task and capture test steps
• Feedback process and standards
• Execute the plan and monitor

Once the plan has been executed, ongoing monitoring is essential to ensure the UAT works efficiently and plan.

Things to consider

• Ensure the central system is updated.
• Assess progress and rate of progress
• Deal with issues and concerns
• Maintain team motivation.
• Ensure required results achieved
• Encourage good feedback
• Communicate status with business users and stakeholders.
• Monitor completion rates
• Monitor forecast completion.
• Monitor level of issues reported.
• Provide a feedback loop to users of issue status.

Triage test results

Whilst a full-time quality assurance team may correctly identify and raise defects, it is often preferred with users testing to distinguish between their issues raised and defects of the system, combined with a triage process to determine which issues should be promoted to defects.

Triaging results can avoid swamping development teams with incorrect or duplicated defects reports, making it easy for users to report what they find. To be effective, good evidence and audit trails of the test result are needed, and this requires complying and easy-to-use capture technology.

Things to consider

• Review feedback and issue reports.
• Examine the collected evidence
• Update central system with new issue status
• Document and communicate defects to the product team.
• Ensure the central system is updated.
• Provide a feedback loop to users of issue status.

Review and manage

In an ideal testing process, the tests will have been developed to provide 100% coverage of the areas and scenarios needing testing. These will all have been executed correctly and passed, even if the process has undergone several cycles.

 At the same time, users will not have significant questions or concerns and will have provided positive feedback about the software’s quality and readiness. Upon completing this, all shareholders will have agreed that the changes will positively impact the organization and achieve project goals.

Reaching this position is largely dependent on the factors previously considered and implemented before this point; however, you will also need clarity of the project metrics to determine when this project status is approaching or arriving.

This data will need to have been collected as the project has progressed and cannot be realistically constructed in retrospect, so it needs to be reinforced into the management platform and methods from the start.

Assess overall status and readiness

Once the test has concluded, we recommend following these 6 steps:

6 steps

1. Are there any outstanding defects?Are there any outstanding issues?
2. Has there been progress made against key metrics?
3. What are the test passes against failures? Any outstanding tests?
4. Assess the state of the surveys and content and review accordingly
5. Communicate the status of the UAT to stakeholders
6. Automation build strategy

Automation of tests can be extremely useful for UAT. It provides the most value in areas that have already ‘passed’ user inspection but will need retesting or re-execution either as part of a subsequent cycle in this project or a future release cycle where it forms the basis of regression testing. Automation can also provide value in data loading to support testing activities.

Whilst manual testing is essential for successful UAT, having users repeatedly test the same software will eventually lead to less accurate results. Users will become tired of repeating the same hypothetical task over and over again. However, if automated, UAT managers can still validate and accept the results without users repeating tests and still gain valuable test assets for future use.

To be successful, automation will need to be relatively easy to create, maintain and understand. By delegating some repetitive aspects of the testing scope to automation time, users’ time is freed up substantially. Through planning for automation at the start of the testing process, this freed-up time can be redirected to exploratory testing.

Things to consider

• Can past manual tests be converted to automation?
• Where is the greatest gain?
• Is the process knowledge available, captured?
• What future requirement for regression testing can be addressed now?
• Consider intensive, repetitive manual tests or tasks (such as data loading) for early automation.
• Ensure you have selected an automation solution to match team skills and rate of progress.
• Are there configuration and environment tasks that will need to be repeated when moved to production, captured in automation for consistency?

Repeat

 If your UAT, Conference Room Pilot (CRP), or Model Office Testing (MOT) project has completed successfully with only one cycle, you are very lucky, or very good, or both!  It is most likely that a number of cycles will be needed, perhaps with reducing effort and coverage as the testing progresses and confidence in some areas solidifies. 

But this is also a time to be careful as spotlighting only the areas of latest concern may allow errors to sneak past in previously cleared tests that have not been repeated.  This is where automation of these areas can pay dividends, providing confidence without the hard work

However, during this time, is it important to still be careful in reviewing the whole software and not only areas that have previously been concerned. This may result in undetected errors not being flagged up; using automation in these scenarios can provide confidence without committing more resources. Having got successfully through the first cycle of testing, the remaining tests will be easier and the plan considerably shorter.

Replan for testing

• Determine the approach you wish to take
• Ensure that you are testing this holistically and not just for repeated failures
• What regression testing will be taking place – Whole, part, manual, automated?
• Provide a data environment matching the tests needs

Apply automation to areas not needing manual testing

• Re-baseline scripts to include changes
• Review the results with users

Once all preparation for repeating the tests has been conducted, you will need to repeat the execute and review and manage sections of the handbook.

Retrospective

There will always be things that could have gone better, and during this phase, it’s important to reflect on these for improving future projects. Hopefully, the project will have been a triumph, and with the successful implementation, you will need to positively feedback the project’s success to all stakeholders to encourage future UAT participation within the organisation.

Analyze and critique the project

• What was the level of issues found, and was it acceptable?
• What was the level of defects found, and was it acceptable?
• Where was time lost or gained?
• How effective was the communication process?
• How was the quality of the delivered results perceived?
• Consider the level of questions and concerns after/during the UAT.
• Praise users and stakeholders where appropriate

Organize assets from the plan and results for future use

Collate assets and organize for future use

Adapt test policies if necessary.

Once the reflection has been completed, adapt your test policy for future UATs

• What strategic changes would have created a better framework?

Conclusions

The nature of User Acceptance Testing is bound to vary according to the project’s needs, the business, the users, and the overall risk.  In some cases, it is the only form of testing an organization will experience, but even if it is not, UAT is the most expensive type of testing that can be performed. 

Many Quality Assurance theorists and academics will be unhappy that cost is even mentioned in the context of testing. They will consider it irrelevant compared to the project’s goals and will argue this by making the comparison to the cost of not testing—a fair point.  However, the cost is a real factor and a real constraint for many projects, and given that UAT is the most costly type of testing that can be performed, it is worth considering how the cost can be controlled and reduced. Thankfully, a cost reduction is a by-product of other more inspiring objectives and is achieved by efficiency, reuse, improving communication, and better testing approaches that drive out more issues in less time, with less effort.   In the end, delivering an issue-free system, which the users know how to use and gain benefit from, will be the most valuable factor in the project’s success.

This guide has been created to help identify many of the topics and considerations which should be examined when looking to achieve the above goals.  The use of technology is a recurring theme because it plays a valuable, indeed essential, part in making the improvements (compared to a manual process) to enable these goals to be achieved.  In some organizations which are too IT and development-focused, testing is deemed to be only worthy of attention within the IT team, and once in the hands of the business users, it can be forgotten.  Perhaps because by then, it is someone else’s budget, but the reality is that it impacts the business as a whole by increasing cost and reducing project success if not done well.  The business case for investment in improving this process is usually easy when considering the amount of time that can be saved and reducing issues that hit production systems. 

No organization wants to be the next ‘Software testing failure’ headline, and UAT is a crucial component to maximize the quality of the delivered solution and stay out of the news for the wrong reasons.

About Original Software

The Ultimate UAT Guide has been brought to you by Original Software. Our mission is to deliver quality software, that is easy to buy, has an intuitive UI, and is loved by our clients. Original Software enables organizations to meet their testing objectives rapidly, by delivering innovative solutions to support application quality.

Our software provides the fastest way to capture and share business processes, validate application functionality, and manage projects in real-time.

If you’re only just starting out with UAT or have been conducting it for years, get in touch with us today to discover how our approach can dramatically improve the efficiency of your process.

Our clients have reported time and effort savings of 60%