Every software developer knows the importance and significance of unit testing yet I am constantly surprised at how little we practice this very basic principle. Most unit testing I have observed is done manually with little to no repeatability or predictability. In a waterfall world, we may well have gotten away with it but as the world changes rapidly around us, this approach is no longer acceptable or even justifiable. Continuous integration and Continuous Development (CI/CD) have evolved and matured to a point where not automating the building and testing of the software we write is surely going to lead to failure either because of quality problems (if we focus purely on velocity of feature development) or because we're too slow to introduce features (if we focus only on releasing with reasonably high quality).
Like many things, automation is also a multi-faceted problem:
- Build process automation: In this stage we run automated unit tests with every commit that goes into our SCM systems. If we wrote good quality unit tests that cover a broad range of functionality, doing this should give us a high degree of confidence in making changes and ensure no regressions are introduced as a result. We cannot write software in a fast changing world, without constantly improving the quality of the code and that necessarily involves refactoring and improving the code that provides functionality. Tools like Maven, TestNG and JUnit are used (with Java code) make it relatively easy to achieve this automation;
- Integration/System Test Automation: In this phase of CI/CD, the entire system gets built using the build tool chains and a full set of automated integration tests are run against the system. In this phase, in addition to building the individual components (perhaps using #1 above), the entire system is built from these components and deployed usually in a virtualized environment. Once the system is deployed, the tests are run, results evaluated, and the systems are torn down if all the tests pass. If the tests fail, the system state is preserved for debugging purposes. Tools like Jenkins or Hudson are used to make automating system tests.
The focus of the next few blog articles will be on #1 above. There are several aspects to be discussed in writing good automated unit tests.
- Test driven development: There are many great articles written on this topic that are worth reading but the focus of my blog is beyond the dogma or religious fervor with which this topic is discussed. I'll focus more on some practical approaches which I have found useful in developing tests for my code while writing the code (instead of after the fact as it is usually done);
- Test independence: Tests must be written as much as possible to be independent and stand on their own. I have seen dependencies created between tests that result in essentially making the entire test suite run only one test at a time (#4 below). This not only significantly degrades build performance, but also makes it harder to change/add more tests because at some point we will be unable to track all the pre-requisites for a test;
- Randomize inputs: To write high quality tests there must be little to no hard coding of inputs. This topic has not received much attention, but I think it is critically important to do to produce tests that are robust and stand the test of time;
- Multi-threaded: This is a critically important aspect. In our current project we got to 5000+ unit tests in a relatively short period of time. However, because tests were not written to be independent (#2) we ended up forcing us to run the tests in a single-threaded manner. This significantly impacted our build times;
- Coverage (Branch/line): When we started out we focused exclusively on line coverage of code. While this was a good starting point for us, given the maturity of the organization and its capabilities, line coverage is not adequate in judging the quality of the unit tests and may in fact even be misleading. Branch coverage is a much better metric to use when judging the quality of the tests. Tools like SonarQube generate reports that provide visibility into unit test coverage. Using these metrics will significantly impact the quality of the code;
- Mock external service interactions: Many classes interact with external systems. It is sometimes believed that such systems cannot be unit tested. In this approach we demonstrate the use of Interface driven development to show how to unit test code that interacts with external systems; and
- Test classifications (groups): As mentioned in #3, we got to more than 5000 tests in a relatively short period of time. In order for automation to be effective, we need to group tests into logical units that serve different purposes (nightly, sanity, functional, full regression etc.). Doing this early will prevent time consuming and mind numbing classification later.
Hopefully at the end of these series of blog articles it will help us become better software engineers and write quality unit tests.