On Testing – Part 2

Unit & Integration Testing

In part 1 of this series, I talked about the test pyramid, and how to approach applying it in your testing process. In this post, I’m going to focus on the lower levels of the pyramid, where you should ideally be spending most of your time.

Revisiting the test definitions

The problem with defining categories for different types of tests is that words like “unit” and “integration” are very loaded terms. For example, some people refer to many kinds of tests as a unit test. Anything is some sort of unit, isn’t it? Integration tests often refer to just about any kind of test that isn’t a unit test. Here are my definitions.

Unit tests

As mentioned in part 1, the most important thing about unit tests is that they are fast. But how fast is fast? Well, that depends on your language and tools of course, but ideally a single test should run in well under a second (the lower the better). Some other key things about unit tests:

• Unit tests should not perform any kind of I/O. This includes talking to a database, the file system, or any form of network communication.
• Unit tests should not invoke any framework code ¹. Framework code is plagued by expensive bootstrapping time which accumulates quickly across a large suite of tests. You should write tests which use framework code in component or integration tests.
• Unit tests should not have any environment or execution-context dependencies. e.g. a file present on the path, variable exists in an environment, etc.

If you look at these points, most of them pretty much follow directly from the fact that unit tests need to be fast. Unit tests are the least likely kind of tests to be brittle, because of how self-contained they are. Unit test suites are fast enough for a developer to run locally, ideally on every push to source code control.

Integration Tests

Component Integration vs. Integration Tests: what’s the difference?

In part 1 of this series, I drew the distinction between component tests, which are concerned with the integration testing of a single component; and integration tests, which are concerned with testing the interaction between components.

But aren’t they all just integration tests? Here’s why I think the distinction is important:

• Because they are more focused, component tests are easier to write than integration tests. From my experience, any additional friction to writing tests will result in fewer tests being written. This is just human nature. Anything that can be done to make test writing easier should be encouraged.
• Like unit tests, component tests will run and fail faster than integration tests because they have less setup and overhead. They test more specific things and thus provide more specific information on failures.
• There is a temptation to just write an integration test for a whole sub-system, instead of more specific tests. This often stems from a desire to ensure basic coverage with limited time. Since there’s typically a lot of machinery involved in setting up these tests, there tends to be few of them. This often misses important edge cases which are configuration or integration-related. By encouraging the practice of writing focused component-level integration tests, it’s more likely that corner cases will be given proper treatment in the test suite. Corner-cases are just as important in integration testings. Don’t give them short shrift.

If you only have so much time to write tests, prefer writing component tests over integration tests. You’ll probably write higher-quality tests, find more bugs and get faster feedback.

Multi-component Integration Tests

In part 1 of this series, I talked about integration tests being about interaction.

Since the interaction is what is key in an integration test, you might be tempted to think it makes sense to mock the underlying services. But you should always prefer using actual service implementations² so that you aren’t missing important integration scenarios. In my experience, that is really where the value of integration tests lie. If the integration logic between components is sufficiently complex, you should consider seeing whether any unit tests can be written for this logic with some re-factoring. The key point is:

Avoid writing an integration test which is just masquerading around as a really slow unit test

An integration test should exercise all configuration code wiring your components together. Bugs hide there. This is the stuff integration tests should test. The complexity at the boundaries between code and configuration are non-trivial in most applications.

The Mocking Craze

Mocking is great, but it’s not a panacea. Have you ever worked with a test that made such heavy use of mocking that it frequently failed due to mocking issues?! How about a test that makes so much use of mocking it basically doesn’t test anything at all!

The problem with mocking is that it’s really powerful, which makes it very tempting to use. everywhere. When you first discover it, it’s almost like magic. It’s like having a hammer and everything looks like a nail. It makes testing gnarly code appear easy, but it’s often just masking code smells in the underlying application.

Mocking is fragile. Extensive use of mocking makes tests more sensitive to actual code changes because they will frequently need to be updated to mock more or different things depending on what in the code under test changed. The more your tests can leverage the actual application code in driving their setup and execution, the better. It’s makes them more resilient to change.

Use mocking sparingly and judiciously. Try to restrict your use of mocks to only where absolutely necessary. If you have a component in your integration test whose behavior is irrelevant to the test, consider whether you could write a component integration test instead. Or just use a real implementation.

Investing in a test support library

I touched on the idea of friction with respect to writing tests. The excellent book XUnit Test Patterns discusses the idea of investing in a robust test library for your code-base. This means factoring out test-support API’s and treating them as you would real production code. Having solid API’s to flexibly and easily construct test fixtures (the builder pattern is great here), you can give developers the tools to eliminate boilerplate when possible. This can make a huge difference on your team in terms of enabling more test writing, and making the test writing process more enjoyable.

In the next post in this series, i’ll cover browser-based testing. Stay tuned!

---

¹ Common framework component code includes things like: the spring framework, an ORM tool like hibernate or an MVC framework like ruby on rails.

² A real implementation meaning the actual code. The underlying configuration should still be test-context specific, of course.