Tests you love to read, write and change

Introduction

Tests play a really important role in our application. They convince us our code works as we expect it to, and won't break production. They stand in as guard rails as our application changes. They protect us from ourselves as we forget the all the details about why and how things are implemented. They provide context for new members as they ramp up their contributions. When written well, tests provide a lot of great benefits, and live a long useful life.

Unfortunately, often tests become a real impediment, slowing our comprehension and ability to change them. Often we forget that tests are read many more times than they're written, and don't optimize for that reality. Furthermore, most tests are read and changed sometime after by a different author. If we're not careful, our tests will harden the softness of software.

Following are several tips to make your tests easier to read, maintain and change. I'll talk about using centralized abstractions to simplify changes and using focused tests to improve legibility.

Simplify test changes with factories

Pattern

Test data is bootstrapped in a central method, shared by multiple tests.

class TestSubscriptionCreation:

    def setUp(self):
        # The shared method where we setup ALLLLL the things

        # Create a user, business and associate them
        user_with_free_subscription = User.objects.create(...)
        first_business = Business.objects.create(...)
        add_user_to_businesss(user_with_free_subscription, first_business)

        # TONNES of other declarations... 
        # e.g. more businesses, premium subscription, etc...

    def test_free_subscription_is_created_properly(self):
        # test that uses first business and free subscription

    def test_premium_subscription_is_created_properly(self):
        # test that uses second business and premium subscription

In this example, I'm bootstrapping test data in a shared setUp method. The setUp method is called before each test. Each test has access to the data created in the setUp method.

In the setUp method I'm creating users, businesses, plans and subscriptions. I'm creating data for a free and premium subscription. For each subscription, there's a user, business and plan that must be created first. Once I've set that up, all tests can access this pattern.

Why it's problematic

Dissimilar use cases depending on the same data complicates changes.

When the data is near identical across tests, this can be an easy way to avoid repetition and centralize logic across tests. As more functionality is added and tested, the shared central method (setUp) gets really crowded and the data needs of each test slowly diverge. The breaking point comes when you're forced to change the test data in an incompatible way.

Once the test suite grows arbitrarily so does the friction for making changes. Imagine having a base class and there are hundreds of tests. Changing the test data can end up breaking several tests. Also, the rationale for the breakage isn't always obvious. This is especially compounded when we're bootstrapping complex objects. It becomes that much harder to change the test data when it involves lots of objects. Comprehension is harder and duplication requires surgical precision. Often we're forced to make inelegant additions, or pile unto the already crowded variables. Ultimately, the tests are harder to read, understand and change.

The test data is declared too far from where it's used -- this hinders legibility and comprehension

Often the test data is defined at a site that's quite far from where it's needed, and this slows the process of reading and understanding the code. As this pattern is extended, the bootstrapped data ends up being several hundreds of lines away, or even in a different file. This distance makes for lots of jumping back and forth, and increases the cognitive overhead. You have to try to remember what something means. Bad variable names makes this harder. It's also several compounded if the test data is declared in other files in a sequence of base classes.

Instead, create a factory

Instead of declaring all the data in a central place, use factories that can create the test data on demand.

class TestSubscriptionCreation:

    def test_free_subscription_is_created_properly(self):
        user_with_free_subscription = UserFactory()
        business = BusinessFactory()

        free_subscription = SubscriptionFactory(
            user=user_with_free_subscription,
            business=business,
            subscription_type=FREE)


    def test_premium_subscription_is_created_properly(self):
        user_with_paid_subscription = UserFactory()
        business = BusinessFactory()

        paid_subscription = SubscriptionFactory(
            user=user_with_paid_subscription,
            business=business,
            subscription_type=PAID)

In the snippet above, I've gotten rid of the setUp method and I'm using factories to declare the bootstrap data within each test.

Using factories improves legibility and maintainability. With factories, the test data is closer to where it's being used. This greatly improves readability, as you get answers quickly about the test data you've created. Factories also provide a great opportunity to define a clean, simple API for bootstrapping test data. If you use a single set of interfaces repeatedly, you start to see patterns in the use cases which inspires thoughtful evolution of the interfaces. It also greatly resolves the coupling issues mentioned earlier. Changing test data in one test doesn't affect any other tests. The factories also help you accelerate nuanced use cases that call for complex object structures. Using factories will increase your test legibility, accelerate new additions and simplify changes.

On the flip side, this pattern can increase your test length. A distinct advantage of centralizing test data in a shared method (e.g. setUp as shown earlier) is test size reduction. It also means sometimes test can actually take longer to write and run, because you're re-declaring test data. In the long run, however, as your test size grows this pattern will still provide net benefits. Use these challenges as an opportunity to think thoughtfully about your test factory interfaces, and creating elegant abstractions that reduce the boilerplate. Finally, test run time can be reduced with some parallelization of test runs.

Enable large scale changes through centralized abstractions

We're bound to change things in our software and our test design can help or hinder that process. Often there's some functionality we need in our tests (e.g. mocking) and we don't bother to centralize it.

Pattern

We've got some functionality (e.g. mocking) to provide the test and we mock it directly. In the following example, I want to mock the repository and confirm I'm calling it properly.

class TestSubscriptionCreation:

    @patch('SubscriptionRepository.create')
    def test_free_subscription_is_created_properly(
            self, subscription_repository_mock):
        # mock the repository that creates the subscription
        subscription_repository_mock.create.return_value = mock_subscription   

        # create the subscription
        subscription_service.create_subscription(...)

        # assert on the repository
        subscription_repository_mock.create.assert_called_with(
            business_id=...,
            user_id=...,
        )

In the example above, I'm testing that subscriptions are created properly. The repository is responsible for subscription creation. I want to mock the repository so it doesn't call the database. I used Python's patch functionality to directly patch the create method and let it return what I want.

Why it's problematic

The above pattern works well, if the thing you're mocking, or functionality you're providing doesn't change much. However, revisit this pattern applied over a couple years, and you have a real problem when you need to change how it works. It becomes a real chore to update all the places that's using the mock. Imagine that the repository now returns a tuple of (created, subscription) instead of just a subscription? Imagine changing hundreds or maybe more tests that need to make that change.

The other issue is that it becomes kludgy and cumbersome to write tests that involve multiple calls to the thing you're mocking. Imagine your use case calls create and list. You would need to manually mock those methods in your test. This creates a real danger for distance between your tests and application code.

Instead, centralize the mock

What you can do instead is to centralize the mock and provide an elegant API for configuring the behaviour of the component in question.

class TestSubscriptionCreation:

    def test_free_subscription_is_created_properly(self):
        # tell the mock you want it to return this subscription on create
        MockSubscriptionRepository().with_created_subscription(
            subscipription=subscription)

        # create the subscription
        subscription_service.create_subscription(...)

        # assert on the repository
        MockSubscriptionRepository().assert_create_called_with(
            business_id=...,
            user_id=...,
        )

In the example above, I'm using a MockSubscriptionRepository to configure the repository's behaviour. I use the with_created_subscription method to control what it returns. Then later, I'm calling assert_create_called_with(...) to verify the call signature.

Centralizing your logic within an abstraction will really enable you to make large-scale changes easily. Doing so keeps your test unaware of the underlying structure and implementation of the things you're trying to mock. This means when that structure changes, your test doesn't need to care, or change. So, changing the structure doesn't need to mean changing hundreds or thousands of tests. The abstraction shields your test from changes. Also, just like factories, this centralization enables you to build a well thought-out test API since it's used across several use cases. You can gradually extend the API in a fashion that really benefits test writing and modification.

Increase legibility, comprehension and maintainability by avoiding assert fatigue

Pattern

Often when we're writing our tests we assert on everything! The sky is blue, the grass is green, and everything in between! Hmm... nice rhyme.

class TestSubscriptionCreation:

    def test_free_subscription_is_created_properly(self):
        # create test data

        # assert on test data
        user_has_subscription = \
            Subscription.objects.filter(user=user).count() == 1
        assert user_has_subscription

        # create the subscription
        subscription = subscription_service.create_subscription(...)

        # check the output
        assert subscription == expected_subscription

        # check the repository was called
        assert repository_was_called_properly

        # oh... check logs too
        assert actual_logs == expected_logs

        # metrics, are important too, let's check that!
        assert actual_metrics == expected_metrics

        # did I miss anything
        assert that_i_checked_everything_i_need_to_check

In the example above, I want to verify that my subscription creation works properly. I'm checking a whole list of different things! I'm verify test data creation, serialization of the created subscription, subscription repository interaction, logs, metrics and maybe more.

Why is it problematic?

These spurious assertions muddy the test's focal point. Understanding the test becomes harder. What's the primary goal of the test. When the test breaks for no obvious reason, it can be a real pain to debug. The test is overwhelmed, and so is the reader. In the example above, the test could break because of a bug in any number of different places. Including all these assertions in the test can really hinder our understanding of the test's purpose.

When tests are written in this form, it can really hurt the maintainability of the codebase. It tests have lots of reasons to break, then they will break for lots of different reasons. Given this pattern is applied several times, over many years, and we change one thing, say the metrics. We end up with a mammoth of a task because we need to go change a tonne of different places. We end up having to change the tests for multiple reasons which really tanks productivity.

Instead, write focused tests

Undo the alert fatigue by writing focused tests. Each of those assertions deserve their own test. We can write tests that verify that we're publishing metrics correctly, or logs. We can have a focused test that verifies the structure of the subscription that's returned. We can yet another focused test to check our interaction with various components. This will increase the test footprint, but it will provide much more clarity for readers and improve discoverability of use cases.

Conclusion

By adopting these strategies, you can significantly enhance the maintainability and readability of your tests. Simplifying test changes with factories, centralizing abstractions, and writing focused tests not only makes your codebase more robust but also accelerates development and reduces the cognitive load on your future self and other engineers. These practices foster an environment where tests can evolve gracefully with the application, ensuring they remain valuable tools rather than obstacles. Embrace these tips to create tests that support continuous improvement and innovation, empowering your team to deliver high-quality software with confidence!

Thanks for stopping by! I'd love to hear about testing best practices you have! Drop some wisdom in the comments!