Why frozen test fixtures are a problem on large projects and how to avoid them

To me, fixtures are a code smell. If you need so much common setup to test your application, the code under testing is doing too much. It's unfortunately quite common in Rails or Django projects. You need to pass the Foo model to your function, but it will lookup foo.bar.baz, so you need to wire up these as well, which again need further models. Of course everything also talks with the database.

Instead, if you're able to decouple the ORM from your application, with a separate layer, and instead pass plain objects around (not fat db backed models), one is much freer to write code that's "pure". This input gives that output. For tests like these one only needs to create whatever data structure the function desires, and then verify the output. Worst case verify that it called some mocks with x,y,z.

I find inheritance in tests leads quickly to hell. Striving for every last bit of reuse seems like the right thing to do but it hurts in subtle ways that compound over time. If you must, use composition and spend the time on a DSL that clearly documents the setup in each test.

I feel like the elephant in the room in this post is property-based testing. I dislike using fixtures for all the reasons stated in the post, and when it seems like I might need really them, I reach for property-based testing instead.

"Generators" for property-based testing might be similar to what the author is calling "factories." Generators create values of a given type, sometimes with particular properties, and can be combined to create generators of other types. (The terminology varies from one library to another. Different libraries use the terms "generators," "arbitraries," and "strategies" in slightly different and overlapping ways.)

For example, if you have a generator for strings and a generator for non-negative integers, it's trivial to create a generator for a type Person(name, age).

Generators can also be filtered. For example, if you have a generator for Account instances, and you need active Account instances in your test, you can apply a filter to the base generator to select only the instances where _.isActive is true.

Once you have a base generator for each type you need in your tests, the individual tests become clear and succinct. There is a learning curve for working with generators, but as a rule, the test code is very easy to read, even if it's tricky to write at first.

Example 1 bothers me. It says

> This test has just made it impossible to introduce another active project without breaking it, even if the scope was not actually broken. Add a new variant of an active project for an unrelated test and now you have to also update this test.

And then goes on to test that the known active projects are indeed included in what the call to Project.active returns.

However, that doesn't test that "active scope returns active projects". Rather, it tests that

- active scope returns _at least some of the_ active projects, and

And it does not test that

- active scope returns _all_ of the active projects

- active scope does not return non-active projects

Which, admittedly, is only different because the original statement is ambiguous. But the difference is that the test will pass if it returns non-active projects, too; which probably is not the expected behavior.

I prefer to set things up so that my test fixtures (test data) are created as close to the test as possible, and then test it in the way the article is saying is wrong (in some cases)... ie, test that the call to Project.active returns _only_ those projects that should be active.

Another option would be to have 3 different tests that test all those things, but the second one (_all_ of the active projects) is going to fail if the text fixture changes to include more active projects.

I've been doing one thing for many years that forces me to be smart about test fixtures in my integration tests and test data: I test concurrently with many threads.

This means that my test can't depend on the database to be in some known state or assume to have exclusive access to that database. And for example modify anything that might be used by another test. They can only modify things that are specific to that test.

Most of my tests work around this limitation by either just creating their own teams, users, and other objects they need with randomized ids; or in some cases deferring their execution until some bit of logic with lock has created some shared data that then is never modified.

Instead of hard coded IDs, I tend to use randomized ids (UUIDs typically). I have a person data generator that gives me human readable names, email addresses, etc. Randomized data like this avoids test modifying each other's data.

As an example, we have a few tests for an analytics dashboard that locks on a bit of expensive code that creates a lot of content via our APIs to do analytics on. The scenario is quite elaborate and uses a few factories, known timestamps, etc. If I refactor my data model, my factories are also refactored. Using a lock ensures that data is initialized only once. Once that is done, there are a bunch of test that that test different queries against that.

You might think that all this is slow. It's not. I have about 380 integration tests like this that run in under 30 seconds on my laptop (which has a lot of CPU cores). Having this as a safety net is very empowering. I've been on teams that had less tests where running them took ten or more minutes. This I can do quickly before committing.

Testing like this has many advantages. But one includes easy to maintain tests. I put some effort into usable test data factories. The "when" part of a BDD style integration tests is usually most of the work. So, by making that as easy as I can, I lower the barrier for writing more tests. And using all my cpu cores, minimizes the impact new tests have on execution time to the point where I don't worry about that.

Another is that for big structural changes my tests continue to work if I just fix their shared factories to do the right thing usually.

I feel this is all contentious because, like so much in coding, we have people taking Thing That Works For Me on their current project or in their experience over time and declaring it to be The One True Way while other people are in completely different codebases with different priorities around shipping v quality v cost or what have you and are complaining "That doesn't work for me, Y has always been my go to".

The answer is most definitely, 100%, with no room for argument, to not speak so assuredly, acknowledge other people have the right to think differently and find synthesis and/ or a set of heuristics that apply for given cases.

But this is the Internet, and we need to be arguing PS2 vs X-Box for the rest of our lives, so have at it.

(Me? Factories are great until they aren't, which may not happen if a project or a team is small enough. Generators are great but do have some footguns and I would love to hand over everything to property-based testing, but I _feel_, without any experimenting or trying, they resist anything other than the purest of pure unit tests and can't help with integration tests that much.)

factories definitely are the cause of my test suite being slower than I'd like, I've been thinking of switching to fixtures. Good to see some context of what challenges I might be dealing with instead if I do.

I think fixtures generally work fine. If a change to one breaks many tests, introduce a new one and start using that. I also think it's okay to make some manual changes to them in the test and it's distinct from wanting factories; needing factories only in test code feels like a waste.

100% agree with "Test only what you want to test".

These two suggestions are fine, but I don't think they make fixtures really that much better--they're still a morass of technical debt & should be avoided at all costs.

The article doesn't mention what I hate most about fixtures: the noise of all the other crap in the fixture that doesn't matter to the current test scenario.

I.e. I want to test "merge these two books" -- great -- but now when stepping through the code, I have 30, 40, 100 other books floating around the code/database b/c "they were added by the fixture" that I need to ignore / step through / etc. Gah.

Factories are the way: https://joist-orm.io/testing/test-factories/

For a database-driven application with sqlalchemy, I've found mixer[0] to be pretty helpful. It gives you an easy way to generate an object, and it automatically creates dummy-objects that your object depends on.

You can also supply defaults and name schemes for individual columns.

For business logic, I prefer to have it structured in a way that it doesn't need the database for testing, but loading and searching stuff from the DB also needs to be tested, and for those, mixer strikes a really good balance. You only need to specify the attributes that are relevant for the test, and you don't need shared fixtures between many tests.

[0]: https://pypi.org/project/mixer/

I’ve found that golden master tests (aka snapshot testing) pair very well with fixtures. If I need to add to the fixtures for a new test, I regenerate the golden files for all the known good tests. I barely need to glance at these changes because, as I said, they are known good. Still I usually give them a brief once over to make sure I didn’t do something like add too many records to a response that’s supposed to be a partial page. Then I go about writing the new test and implementing the change I’m testing. After implementing the change, only the new test’s golden files should change.

They are also nice because I don’t have to think so much about assertions. They automatically assert the response is exactly the same as before.

The title was a bit confusing, what's frozen is the fixture definition (and shouldn't be).

The data created by the fixtures shouldn't be touched, or factories are being used, like the author suggested

This makes me think of https://github.com/AutoFixture/AutoFixture

I missed the "frozen". Ok, i understand the text better now. I think the issue is only with the frozen though, i understand why people would think it is necessary, but i think it should be avoided as much as possible, and fixtures, like data should be rewritten each time a data model change.

We have a solution. Not sure if it is elegant, but use it as an inspiration: it works.

When our project run its test, it will generate its database json representation itself (only using its models) with a file that contain fake/test data. That database representation will be loaded in the dev environment, and also in the database fixture that then run our tests. If our tests pass and we have an issue in dev, that mean our test missed something (that happen waaaaaay more often that i like to admit) and we have to add them.

Forcing every test to use this representation also force us to have a dev environment that contain enough items to run the test, and we can't forget to generate an item in the dev database, since that would mean our new feature isn't tested.

Author here, thanks for posting. :)

Your test fixtures are introducing tighter coupling between the tests, than the code they are testing! In this scenario (a mock DB with data that a test relies on, which is incompatible with a new test you want to add), duplicating the fixture is correct. Different tests with incompatible requirements on state should use different mock data. In this specific case, however, it's also true that one of them can be modified to make the tests compatible. If you do red-green-refactor then you can duplicate first, and then coalesce by changing the first test.

"assert_equal names, names.sort" is a wrong answer. It would accept an empty collection.