When is a unit test not a unit test

Working with people who have a good attitude to testing is great. Every new piece of code has some tests associated with it, not a huge number, but a few. And a little test is better than nothing especially when they get run many times a day in CruiseControl.net. It gives you a place to put more tests when you need them. What is bad about it, is that these tests are referred to as unit tests. These aren’t unit tests and people don’t seem to realise a) why they aren’t genuine unit tests and b) why genuine unit tests are good.
Why they aren’t unit tests?

The biggest problem with people who think that unit tests don’t test anything is making them realise that their unit tests don’t test anything because they aren’t writing OO code. How can you tell that they aren’t doing OO? Easy: they aren’t using interfaces. If you aren’t using interfaces you aren’t doing OO; you are just doing strongly-typed coding based around groups of functions. That has a name in some languages: a module. In C# it is called a static class, but really it is the same thing (don’t start me on the evil overloading of the word “class” here). Of course, you can group a bunch of methods and test that bunch of methods and that will be testing a unit, but it won’t be OO code and it won’t be an OO unit test.

How does the lack of interfaces prevent OO unit testing? Well, nearly every class that you use has to talk to some other class. Simple classes consist of some logic working on primitive types (integer, char, maybe string etc). Complex classes contain logic that brings together different simple classes (or maybe multiple instances of simple classes, like a collection). So a simple class has a unit test that goes through its public interface and drives its logic, but a complex class may rely on having working instances of the simple classes that it wraps. The first time you see this, you think that is not so bad. You can still use the tests to see the errors, if the simple types pass and the complex types fail, then you can still get value from the tests. true enough.

Of course, as soon as you enter the world of “enterprise” applications the wheels come off this very quickly. Your “simple” classes are often resource wrappers like messaging middleware or a database connection or other external resource. When it is time to run the “unit” tests, you find that only one developer can run them at a time, they need to run on a special machine, you can’t just check the code out and run, you need to run a script file to register all the components locally, or a hardcoded connection to a test database. All these things are warnings that what you have written aren’t unit tests, but are integration tests that are trying to run in an inadequate unit/class test environment.

Why do you care?
Well, what is wrong with integration tests? Well, nothing. In fact, if only one set of tests could be written then you should be writing the integration tests. Only full system integration tests will verify application behaviour. This assumes that you consider unit testing to be a part of testing. Most agile development considers unit testing to be part of development work as a way of exploring class and interface design.

So, if integration tests are better tests, then why are we spending time working on unit tests? Well, unit tests are the key to improving design. If your tests are integrating half the application stack (i.e., database, messaging etc) and is in and out of all kinds of resources, can you assess the test quality? If you are writing a GUI application that throws some data in and out of a database, how confident are you that when you make a change that your “unit” tests will catch it? You are confident? Good. I’m glad that you are confident, but you probably get that same confidence by starting the app and clicking the buttons. Now, what if your applications have dozens of threads running in multiple, dynamically-loaded AppDomains in multiple processes on multiple machines; all caching and updating data and state and communicating with each other. Now, should I be confident that my half-stack integration “unit” tests are catching all the bugs?

Somewhere in between these extremes is a point at which you need to make changes very carefully, the only way to make careful change is to have tests that operate at all levels.

Composable, flexible, testable
You often hear that you shouldn’t design for testability. There is some truth in that… if you make your systems testable by adding more public members on the classes! However, good design – and by “good” I mean a design that allows the code to be maintained and modified – is often very testable. If the units are decoupled, then the units can be tested.

This makes an impact on the other big criticism of using unit tests: “Every time I change my class, I need to change all my unit tests. This is a waste of time”. If you need to do this often, you should think about the public members on your class. A really common kind of change is adding an extra field to a GUI; then that field needs to be added to the business logic, the service layer, the database and – of course – the tests. If you are doing this often then it is not an indication that unit tests are bad, it is a indication that your class is likely not well encapsulated and is exposing too much of itself.

This work, breaking an application up into really composable units, is often unnecessary in a small application. An intellligent person can mostly manage the complexity, but as soon as the application becomes large – and my rule of thumb is something like 5000 lines – you will start to see the benefits of this interface-based, testable design.

How to change
So how to you fix this?

Well if it a big system – and if it isn’t, then don’t bother – then nothing is simple. It isn’t just a case of getting your refactoring tool and hitting Extract Interface. But how else could you do it? Well there are no silver bullets and it will be hard work. “More work” might not look like the right thing to do, but you are going to change your application to be more composable that means hiding implementation behind interfaces, and that means refactoring. And for refactoring to succeed it means more testing and coding carefully and baby-stepping towards the goal. And to do that you need to know what the goal is and how you will know when you get there.

The point is to make your complex classes that integrate simple classes work with interfaces not instances, so that you can replace external resources with fake resources. So instead of making a call out the MSMQ messaging API via COM and waiting for a response, you insert a stub class that just returns “true” or whatever. So you remove the dependency on the messaging system or database when you are testing. This isn’t just useful for classes that wrap external resources, replacing instances simple classes with interfaces allows unit testing to concentrate on only one class – system under test – and not all classes under it. Reducing the class coupling, reduces the complexity of verifying test behaviour and also, as a nice side effect, can speed up your test execution by an order of magnitude.

Adding unit tests is not at all simple for a large system, and it certainly won’t be free. It might be worth doing for a complex system if you can break the system into good parts that can hide behind interfaces. If you can’t hide implementation behind interfaces then you should be afraid, very afraid; even if you are doing some testing it isn’t unit testing and if your system can’t be unit tested, you should spend a lot of effort on automating the tests you do have, you will certainly need them!

See also:


One thought on “When is a unit test not a unit test”

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s