MARTYHAUGHT

So what do you do when you can’t test all the time? Let’s say your new project simply doesn’t give you the time to write 100% tests to your satisfication? I have been faced with this recently on a few of my projects and my answer is value-based testing.

The first thing I’d recommend is you get clear on why you are testing. Prioritize what is most important to you and your team about the tests that you’re writing. This will help you make the hard decisions on where to cut corners. Not every project/team will have the same answers. Some domains really need a lot more in depth testing such as software used in life/death matters or external APIs. Other domains may not need much, especially if you’re writing a prototype web app to try out a new idea.

For example, I’ll use my personal view that happens to work with 80% of my projects.

Things I really want to test:
Important business logic
Authentication and locking away access to valuable actions/resources
Main workflows through the system

Things that I care less about (or not at all):
Html
typical Rails stack testing and model associations
edge cases
parts of the system used by admins or used very infrequently

You might pick up a theme in these two lists. The things I test more of are of high value to the business or would have great consequences if they did not work. The less tested list includes things that I can not really test well or have less impact on the business if they fail/have bugs.

So what does this mean specifically in your typical Rails app? Though the following list is specific to Ruby on Rails, I think you can extract the basic ideas and apply it to other software stacks.

Unit Tests (model layer)

One good litmus test for how much testing I need is how long I would need to explain the code to your typical Rails developer. Can I make it so my tests/specs speak for the code? Code that is self explanatory or typical Railsfare won’t get as much testing. I would say that unit testing is where I start and comprises the most tests in the system. Hopefully you’re putting more of your business logic there so naturally your tests should follow.

Functional Tests (controller layer only)

If you are keeping your controllers skinny then all that should be there is delegation to models for the heavy lifting and thus you’re just testing that you wired everything up right. I would contend that testing here might not carry much value. I recommend you cover that in your integration tests and cut out some pork here.

View Tests (template/html layer)

This is the first to go. You can never really test that the view displays right and I’ve yet to see much testing of value here. Save your energy for testing your models and business logic in your helpers. You are putting all your logic in helpers, right?!

Helper Tests

The same standards on unit testing apply here (you can actually think of helper testing as unit testing only on the view layer).

Integration Tests

Beyond model testing, this is where most of my testing energy goes. Integration tests in Rails are full stack and preferrably won’t have any mocking/fake code in there. This is really important as all other testing layers previously mentioned are working in isolation and this is your first chance to make sure they all work together nicely. What’s even better is if you can hit a real application running in production mode such as via Selenium or Watir. I’ve encountered bugs that no amount of development integration tests would have caught. I would recommend you start with the most important page flows in your system first and continue to add use cases as you get time. Cucumber and Webrat are great tools to check out.

So now that we’ve gone over my quick overview on how I deal with things if I don’t have time to test everything. One question that might come to mind is should I test 100% even if I have plenty of time? That is a hard question the really comes down to looking at what sort of value are you getting from your tests. It is my opinion you should never write marginally worthwhile/worthless tests. When you phrase it that way it sounds obvious that you shouldn’t do that but I’ve found that some tests indeed do not add much value.

As you measure whether you’re getting sufficient value out of your tests consider the costs of your tests. First, it takes times to write tests (duh). That’s time you could be using to write more valuable tests elsewhere or moving onto a new feature that really delivers value to your customers. Second, more tests add additional time to any change in functionality as you’ll need to change those worthless tests when you do change things. Third, the more tests you have, the longer your test suite takes to run. Both of these last points may seem minor in a new app but wait until you have hundreds of tests and your opinion may change.

If you’re not doing this already, I encourage you to be more active in this line of questioning. What benefit are you getting from the tests that you’re writing? Are you just going through the motions? Make sure there’s business value in what you’re testing. Imagine if you were paying the hourly rate on you and your team. Are you getting good bang for the buck or are you essentially wasting money?

blog comments powered by Disqus