So, what does it mean to test on agile terms? Agile teams are required to release new features to the market in a matter of weeks, not months or years. Quality is essential to the agile method, because its main focus is on satisfying customer requirements. The challenge is how to test features rapidly and comprehensively as development velocity increases.
Learn how modern development teams are testing in parallel to development, combining testers into the development team, and creating a continuous feedback loop within each product iteration to “bake” quality into their code.
In this page you will learn:
- What is agile testing?
- 3 key attributes of agile testing
- The 8 agile testing principles
- How to use the agile testing quadrants
- 7 agile testing challenges and how to master them
What is Agile Testing?
Agile testing begins at the onset of a development project and involves the ongoing integration between testing and development. Traditionally, testing was a separate activity that came after the coding phase; in agile, testing is continuous, putting testers between product owners and developers. This creates an ongoing feedback loop which helps developers improve their code.
Key attributes of agile testing:
- Communication with product owners—testers interact with product owners to clearly establish project expectations, so they can help developers align with the product roadmap and satisfy customer needs.
- Close interaction with developers—testing is linked to the development process. Testers are part of the development team, they report on quality issues that can affect users, and suggest how to improve the solution.
- The entire team is involved in quality assurance—the whole team is passionate about quality, developers build unit test cases for a better testing process and to enhance the quality of audits. Developers also follow the recommendations of testers for test requirements and code improvements.
Agile Testing Principles
There are eight main principles that guide agile testing:
- Continuous testing—agile teams perform tests regularly to make certain that the product is continuously progressing. Testing is done in conjunction with development.
- Continuous feedback—testers provide continuous feedback to team members. Members regularly receive feedback regarding quality rather than requirements.
- Involving the whole team—testers, developers and business analysts all test the software.
- Quick feedback—the business team participates in each iteration; ongoing feedback reduces the time it takes to get feedback on development work.
- High-level software quality—teams test the software to ensure the code is clean and tight. Through regular testing of the software, issues and vulnerabilities can be easily detected and fixed in the same iteration as they are developed.
- Less documentation—teams use a reusable checklist. Agile development focuses on current customer needs rather than comprehensive, documented requirements and instructions.
- Test-driven—testers evaluate the product at the time of implementation, rather than after implementation (as is the case with traditional testing methods).
- Customer satisfaction—customers are exposed to their product during development. They can adapt and update requirements as development progresses. Tests can be modified to updated requirements.
What are the Agile Testing Quadrants?
Agile testing can be simplified using a system of quadrants, created by Janet Gregory and Lisa Crispin. The quadrants provide a taxonomy for tests, which can help testers answer questions like “which test to run?”, “when to run a test?”, and “how to run tests?”
- Quadrant 1—tests that relate to code quality, including automated tests like unit and component tests.
- Quadrant 2—tests that focus on the business-related aspect of the product, typically manual and automated functional tests. These includes prototypes, functional tests and testing examples of scenarios.
- Quadrant 3—this quadrant provides feedback for tests in quadrants 1 and 2. The team, business owners and even customers use the product in a realistic way to test the user experience and measure business results.
- Quadrant 4—tests of nonfunctional requirements, including security, compatibility, and stability. Tests used in quadrant 4 include stress, performance, and infrastructure testing.
Using Agile Testing Quadrants to Define Your Testing Strategy
When planning a new release or sprint, here is a process you can use to determine which tests to focus on:
- Go through each quadrant as a team and identify which types of testing is needed based on the sprint plan and product roadmap.
- Talk to customers about quality criteria – what is most important for them at this point of development? For example, if the product has the needed functionality but is not sufficiently stable, focus on Quadrant 4. If there are critical features missing, focus on Quadrant 2.
- Understand who can conduct the tests needed. Do you have the required expertise as part of your Scrum team? If not, do you need to recruit expertise from other teams or consultants? For example, performance and security testing require special expertise.
- Do you have the tools and infrastructure necessary to carry out the tests? For example, you may need a test environment on the cloud which you can spin up automatically. You’ll need to build this environment as part of the sprint, or understand who can provision it for you.
- Do you have the required data for testing? If not, build a detailed user story with product owners or users, and ask them to provide realistic test data you can use.
7 Testing Challenges in Agile and How to Master Them
The continuous nature of agile development processes raises a few serious testing challenges:
1. Changing Requirements
It can sometimes happen that management changes requirements or drops stories during a sprint, even though this is not encouraged in an agile/Scrum framework. This means that work already half-done needs to be discarded or modified, which changes the scope of testing unexpectedly.
How to master:
Testers should be able to react and modify their processes according to changing conditions, because in agile projects, change is common. When requirements change, testers should share as much information as they can about what tests they have conducted, and which areas of the application have not been tested yet. This can help the team understand how to make the required changes to the sprint without hurting the quality of the release.
2. Not Enough Information
Product owners, who are responsible for developing user stories, may have an idea about a new feature but may not be aware of the specifics. This means they can’t write a good set of acceptance criteria. If there is missing information about requirements, testers can’t build comprehensive test cases.
How to master:
Testers do not need in-depth requirements to begin testing, they can begin by coming up with high level scenarios that test the idea of the story, confirming them with the product owner. Testing can be done without the complete details about a feature. You can create high level test scenarios, even when particulars change.
3. Continuous Testing
Testing is not restricted to one part of the development process, rather it’s an ongoing activity that starts before the development phase. This creates a major challenge because testers are expected to start building tests for features before coding has even started, or while coding is taking place.
How to master:
To make life easier for testers, user stories in the backlog should be expanded during sprint planning. Testers, developers and product owners should jointly define the details of each story and then write effective acceptance criteria.
The team should ensure that each story has sufficient acceptance criteria and that the context of the story is universally understood before work on development begins. This makes it possible to start creating tests early on, which can be implemented when the code for the feature is complete.
4. Technical Skills
Testers who work in an agile environment need to be technically savvy, helping developers with API testing, integration testing, and scripting UI automation checks with Selenium or similar frameworks. Testers with an exploratory or manual testing background entering the world of agile will encounter a steep learning curve.
How to master:
Testers can and should learn programming or scripting languages such as Javascript and Ruby. Testers who are familiar with programming but lack practical experience can ask for help from developers. Testers can also learn automated testing tools like Selenium tool and JMeter.
For specialized testing areas, such as performance, security, or compliance testing, teams should have dedicated testers with the relevant professional background, or leverage consultants with deep experience in these areas.
5. Frequent Regression Cycles
Developers frequently and continuously add features to the product. This can cause regressions in previous features. Testers use regression tests to identify this problem and overcome it, but manual regression testing is impractical in a fast-paced agile environment.
Another challenge is that modern web applications behave differently when viewed on different devices or browsers. This creates a complex matrix of compatibility testing scenarios, which need to be tested to ensure that the application functions correctly for all users.
How to master:
Agile testers rely on automation. They use unit testing to ensure recent changes have not broken the code, and tools like Selenium and JMeter to verify there is no regression in basic functionality. Testers can use Docker or Selenium Grid to manage and run their automated test, in parallel on various browsers and machines.
6. Lack of Communication
If communication between developers, testers and the product owners is lacking, agile testing will simply not work.
How to master:
Direct communication within the team should be strongly encouraged. Developers, testers and product owners should talk face-to-face on a regular basis to ensure everyone is on the same page. Scrum ceremonies such as stand-up meetings, sprint planning, and retrospectives are instrumental in creating a common understanding of the sprint scope and goals.
7. No Quality Measurement
Agile teams today have no single measurement of quality, which they can use to optimize and plan testing efforts.
There are numerous metrics like unit test code coverage, lines of code (LOC) and code complexity, but none of them provides a clear picture as to “where we stand” with quality at each point in a sprint or release – which areas of the product are working well, which are less stable and at higher risk of quality issues.
Most agile teams are flying blind, responding to production failures or bugs, but unable to proactively focus on product areas which have the biggest quality issues.
Establishing a Measurement of Quality to Guide Agile Testing
A new category of tools called Quality Intelligence Platforms has emerged, to help agile teams understand where to focus testing efforts.
For example, SeaLights is a platform that collects data about test execution across all testing quadrants (code quality, functional testing, acceptance testing and non-functional testing), code changes, and usage of product features in production. It analyzes the data and creates a visualization of “testing gaps”: areas of the product which have recently changed, are used in production, but are not adequately tested.
By visualizing test gaps, agile teams can immediately understand where to focus testing efforts. Instead of over-testing, or building tests to react to previous production faults, they can precisely target areas of the product which are at high risk of quality issues.
To learn more, read our white paper: Reactive Software Maintenance: The Silent Killer of Developer Productivity
