For some reason I came across a rule of thumb that says something like:
You should have only one mock per test; all other interactions must be stubbed.
That sounds like good advice but it’s not. This rule of thumb doesn’t really help writing good tests and it can easily lead to testing the wrong thing.
I’m not terribly strict with nomenclature but for this article I’ll try to stick to the generally accepted differences between mocks and stubs:
- Stubs If called during a test provide canned responses. If not called doesn’t do anything. Never break a test.
- Mocks Expect the test to interact with it in a specific way. Fails the test if the actual interaction is not what was expected.
It seems like the rule above was first published by Roy Osherove in his book The Art of Unit Testing. This article is not a review of Osherove’s book, I haven’t read it yet, just bought the ebook and read enough to understand the way he presents the rule.
I think it’s fair to summarize it like this:
In a test where you test only one thing […] there should be no more than one mock object. All other fake objects will act as stubs. Having more than one mock per test usually means you’re testing more than one thing, and this can lead to complicated or brittle tests. […] Once you’ve identified [your single mock object] you can leave the others as stubs and not worry about assertions against them.
Wrong Focus
For me, tests are much like paragraphs. If you read about how a good paragraph is structured you will often find some advice like this:
A paragraph is a piece of writing that consists of several sentences. A paragraph should always have complete, correct, and concise sentences. As well it should be easy to read and well organized. The paragraph itself should focus on one subject, theme, or central idea.
[…]
When examining a paragraph you can always ask yourself, what is the main idea in this paragraph? If you see two ideas […] you might have to create two paragraphs
So Roy is right when he says that “testing more than one thing […] can lead to complicated or brittle tests”. For me the problem is that it focuses on the wrong way to solve this.
The one subject, theme, or central idea a test should focus on should not be object interactions. This can be the case for Integration Tests –where the focus is on how different parts of a system get along– but it not for Unit Tests.
When writing Unit Tests we should not be concerned about specifying how a class does its job or which other classes it talks to. Those lead to heavy coupling between test and implementation and this is just as bad as testing many things at once, if not worse.
A Unit Test should focus on specifying what a class does. We should focus on describing what behaviour we expect from that class. How it achieves that is not the unit test’s problem, it’s design.
As a (completely biased) example, let’s say we want to write some code that lets us post status updates to multiple social networks. Following the one-mock-per-test rule we will probably end up with something like this:
[Test()]
public void ShouldUpdateFacebook ()
{
var message = "Having a beer.";
var twitter = MockRepository.GenerateStub<Twitter>();
var facebook = MockRepository.GenerateMock<Facebook>();
facebook.Expect(f => f.Update(message));
var updater = new StatusUpdater(facebook,twitter);
updater.Update(message);
facebook.VerifyAllExpectations();
}
[Test()]
[ExpectedException(typeof(StatusUpdateException))]
public void ShouldThrowExceptionIfFacebookUpdateFails ()
{
var message = "Having a beer.";
var twitter = MockRepository.GenerateStub<Twitter>();
var facebook = MockRepository.GenerateMock<Facebook>();
facebook.Expect(f => f.Update(message)).Throw(new FacebookException());
new StatusUpdater(facebook,twitter).Update(message);
}
[Test()]
public void ShouldUpdateTwitter ()
{
var message = "Having a beer.";
var twitter = MockRepository.GenerateMock<Twitter>();
var facebook = MockRepository.GenerateStub<Facebook>();
twitter.Expect(t => t.Update(message));
var updater = new StatusUpdater(facebook,twitter);
updater.Update(message);
twitter.VerifyAllExpectations();
}
[Test()]
[ExpectedException(typeof(StatusUpdateException))]
public void ShouldThrowExceptionIfTwitterUpdateFails ()
{
var message = "Having a beer.";
var twitter = MockRepository.GenerateMock<Twitter>();
var facebook = MockRepository.GenerateStub<Facebook>();
twitter.Expect(twit => twit.Update(message)).Throw(new TwitterException());
new StatusUpdater(facebook,twitter).Update(message);
}
This test provides good code coverage. It’s hard to introduce a bug that won’t break it somehow. This is all good but let me remind you again that what we should be automating here is not tests; we should be automating specifications.
Even if the test above covers all scenarios for execution it is a terrible spec for what a class should do. It doesn’t even talk about the most important expected behaviour for this class, the fact that it must update all social networks at once.
And why it doesn’t specify this? Because it’s not easy (possible?) to do that without using more than one mock! And this is no exception, is a very common scenario for most systems.
Verifying Side-Effects isn’t the Goal
If we want to finally add a proper test that verifies if the class does what it should do (send the same status update to all systems) we have to break the one-mock-per-test rule.
The reason we have to break it is very simple: mocks verify side-effects, and having more than one side effect during a test doesn’t mean testing more than one thing. Writing good tests is about asking yourself “why should I bother?” all the time; and instead of bothering about interactions we should bother about good specifications.
I don’t use mocks a lot, and I even for stubs I spend a lot of time thinking which collaborations should be tested using a double/mock/stub and what are just part of the object. The only case when I use one is when I have to test some side effect.
Having to use too many mocks in a test is definitely a bad smell; something is funny with your code. That doesn’t mean, though, that reducing the number of mocks in a test to one is a good thing.
Using mocks and interactions as drivers for test cases you away from good specifications; always keep in mind that we should only bother about what, not how.
Indeed. Writing good tests is an art form. It’s easy to be mislead by such rules of thumb and missing the essential. In the same page, some managers are befuddled by coverage metrics and send the wrong signal to teams who are not very yet test-savvy. The result? 100% coverage with tests who are nothing more than a rewrite of the test methods.
For sure such teams are using fancy BDD tools with nice natural language specs, but the tests themselves are not about expected behaviour, but instead about the inner workings of the methods.
In consequence, this kind of bad test, littered with mocks and assumptions about call order and arguments, become a hindrance to development.
It’s kind of ironical: Bad tests instead of helping change, become a hindrance to change. BDD tools, instead of encouraging behaviour testing, are taken as a silver bullet and as an excuse to badly thought test that test anything but behaviour. Also, those tests, that look like some sort of insane code rewrite, are not something that someone could look at and thus understand what the code is supposed to do, because this kind of test is too much busy describing HOW the code does what it does, instead of telling why and what it should do.
Mocks are useful tools for what they were intended to be used for. They are not to be used with abandon. Everytime you use a mock, you should instead think about how you are dealing with your dependences. Instead of mocking, try going back and refactoring, try making it simpler, try making it more direct. After you refactor as much as you can, you now can honestly use a mock, but with care. Double thinking should be exercised when using mocks with a dynamic language, try thinking why are you using a dynamic language with duck typing instead of Java: You don’t have fucking gargantuous Interfaces that you HAVE to implement to call a method, so, if you are mocking as much in python or ruby as you used to do in Java, there must be something wrong here.
Frameworks are useful chains, but they are chains. Some parts of some web frameworks like Django are really not very conductive to Unit Testing, in those cases, you’d be better using the functional testing facilities provided by the framework.
Last but not least. If your local prophet or guru proposes a style of testing that brings pain and sweat instead of making things better, you’d better be changing your guru. Phrases like “if it hurts do more of it” are the trademark of a marketidiot. Instead, I propose: “If it hurts, let’s use our brains to find a better way”
Phillip, I agree. The rule is very similar to the “One assertion per test” rule, and I suspect it’s related.
I like to think of a test as an example of some valuable behaviour the class does. The value’s usually in the outcome. However, sometimes you might have an aspect of behaviour which requires two or more outcomes to be valuable. Your status update example is brilliant. I’d like to offer another, just to show that it applies to more than just lists of things.
A banking transaction, when executed, should move money from one account into another.
var result = transaction.Execute();
accountToDebit.Verify(acc => acc.DebitWith(100, Currency.USD);
accountToCredit.Verify(acc => acc.CreditWith(100, Currency.USD);
Assert.IsTrue(result.Success);
If we only do one of these, that behaviour is not merely not valuable; it’s fundamentally wrong and probably expensive.
Despite the three verification / assertions, it’s still only one aspect of valuable behaviour. Thinking of it this way helps me, regardless of whether I’m using mocks or asserting equality.
This is a great blog post. Thank you!
Ok. I got it. But it could be interesting if you show how to rewrite the test code.