When initially learning about test-driven development, it’s tempting to want to test everything using every kind of test. However, when building a valuable test suite, over-testing isn’t the best approach. This guide is primarily targeted at production applications with problematic test suites, but it also provides information that will be useful in all stages of development.
An effective test suite allows you to make changes to your code with minimal worry, yet without hindering the development process. Having too few tests will make it difficult to expand or refactor your codebase without introducing regressions. Having too many tests causes a lengthy feedback loop between committing code and being able to deploy. Over-tested code can be difficult to refactor, when the tests are coupled too tightly with the implementation.
Acceptance tests
Acceptance tests, sometimes referred to as “system tests” or “end-to-end tests,” drive your application or library using its external interface, rather than testing internal logic. For web applications, this is often accomplished using a testing library that can automatically drive a web browser and make assertions against the HTML on the page. An end-to-end test can consist of many steps. For example, a test of the account registration process for a website could consist of driving the browser through:
- navigating to the account registration page
- the creation of the account
- the newly created user’s successfully logging into their account
These tests are often the first tests you will write in a new application, and they are a good place to start when adding test coverage to existing applications, because they line up closely with user stories. However, they tend to be very slow and therefore should be used sparingly. Test only the paths you expect normal users to encounter frequently. Avoid duplicating code coverage. For example, you need to test the login process once, but any other acceptance tests that require a user to be logged in should set the user’s session directly rather than using the UI to log in.
What to test
- Common user interactions with your application
- Error cases that you expect to happen frequently (such as entering an incorrect password when logging in)
- User interface logic that is not encapsulated in any particular unit of your application
- Integration with third-party services (but stub the interactions according to the third party’s documentation)
What not to test
- Edge cases
- Sections of the application with no user-accessible behavior, such as APIs
What to stub
- Third-party services; you don’t want your test suite failing because an external service is not responding
Unit tests
What they are
A unit test is a specification of the nooks and crannies of a particular unit of an application. In languages having a module system, a single module is generally considered a testable unit, and in object-oriented languages a class is a single unit. Many modern programming languages have built-in or official frameworks for unit testing, and a unit test should have no dependencies other than the unit being tested and the test framework.
Unit tests are designed to be comprehensive; the entire public interface of a unit should be tested. Each test should be small, covering one aspect of the unit’s behavior. Since unit tests tend to be fast, accumulating large numbers of unit tests that follow the aforementioned rules is not a concern.
What to test
- Public methods of the unit
- All branches of code execution within the unit
- For units that have direct user exposure (such as controllers in an MVC framework), unexpected inputs that a user may use to break the system
What not to test
- Interaction between the tested unit and another unit
- Correct behavior of underlying libraries*
- Unmodified boilerplate code (including generated scaffolds)
What to stub
- Behaviors of other units (including external applications and libraries) that are necessary for this unit to function
* If a unit depends on a library that behaves in a manner contrary to the documentation, and your unit needs to work around that library’s defect, you should test the workaround to ensure your code doesn’t break if the library is fixed.
Integration tests
What they are
Integration tests resemble unit tests. They are designed to test the cooperation of multiple units. Integration tests should be comprehensive for interactions between the units, but they generally do not need to cover cases of bad input from one unit to another. Bad or unexpected messages from one unit to another should be covered in each unit’s tests, as well as in higher-level tests. An integration test can use the same tooling as a unit test, but it needs to include several units as well as the testing framework.
What to test
- Interactions between units that were stubbed in the unit tests
- Interactions between units in this application and other applications under your control
- Edge and failure cases in interaction between units
What not to test
- Interactions that do not cross the boundaries between units
- Interactions that require the user interface to be driven from outside the application
What to stub
- External applications
- Third-party libraries
Performance tests
The idea of a performance test is to ensure that changes to the application do not cause it to run more slowly. This can be a tricky thing to test, because there are so many different aspects of an application that can affect performance. The tests can be written using unit testing tools, but rather than running them and expecting a pass/fail result, the tests should be run on one machine against the current production version of the code and against the new code. If the time to run the tests increases in the new version, that is a regression that may need to be corrected.
Not every application needs performance tests; in fact, most applications don’t. You need performance tests if:
- Parts of your application perform large database queries, complex math, or other computationally expensive operations, and those operations are not able to be precomputed and cached
- Your application has had performance problems in specific areas
What to test
- Code that needs to run quickly in order to maintain a responsive user interface
- Code that runs in the background but could cause problems if it takes too long or gets a problematic set of inputs
- Anything that has previously caused performance problems in the application
What not to test
- Code that isn’t performance-sensitive
What to stub
- Third-party services
Adam Trilling
