Metrics Are Unit Tests for Your Business
It recently occurred to me that the unit tests I wrote as a software engineer are very similar to the company metrics most startup teams track. Because metrics are so important for startups, I thought it might be useful to briefly describe unit testing, draw an analogy between unit testing and metrics, and discuss how some of the advantages of unit tests also apply to the world of metrics.
If you’re not familiar with the concept, a unit test is a small piece of code which verifies that another piece of code is correct and meets all expectations. For example, if you’re developing an address book website, one portion of your program might be a function called isPhoneNumberValid. This function takes a user-entered phone number and returns GOOD if the phone number is valid and BAD if it’s invalid. A unit test for isPhoneNumberValid might check that “310-555-1234” and “1-800-TMOBILE” are GOOD phone numbers while “310-555” and “TMOBILE” are BAD. If isPhoneNumberValid returns an unexpected result (e.g. it says “310-555” is GOOD), the unit test fails and the developer is notified.
The obvious purpose of unit tests is to ensure that your code does what you think it does. Checking if a phone number is valid requires considering many special cases, and it’s quite possible that your first draft of the code contains a bug – or several bugs. Some engineers stop here: they write a few unit tests to make sure their code works, then they go off to write the next piece of code. However, unit tests have several important benefits in addition to checking correctness:
Unit tests make it easier to experiment. If you have a large collection of existing tests, you can try code experiments without worrying about silently breaking things that currently work. For example, you can try a different algorithm for doing phone validation, knowing that if the algorithm has problems then your existing unit tests will reveal them.
Unit tests implicitly document the importance of different software components. If you had unlimited time, you could write unit tests for everything. Because your time is not infinite, you might test some of your code thoroughly, some code lightly, and some code not at all. For example, if you’re developing a social networking site, you might have a lot of unit tests for handling invitations (because you’re very concerned with spam and privacy), some tests for embedding photos and videos into status updates (because showing thumbnails of photos and videos is nice, but not critical), and very few tests for giving virtual gifts (because they’re a low-priority feature that you’re thinking of removing). Where you write tests is a reflection of which components you consider to be the most critical (or the most challenging to implement correctly).
Unit tests improve the design of your software. Sometimes writing unit tests is a pain because your software is hard to use. It’s easy to overlook design and usability when you’re just thinking about correctness, but unit tests force you to look at your software from the perspective of its end users. Some developers even write tests before writing feature code because it helps them come up with better software designs. (See Test-driven development for more info.)
In many ways, metrics are for companies what unit tests are for software. Companies keep track of all kinds of data: number of active users, average amount per purchase, average number of purchases per user, hours of website downtime per year, and so on. On the surface, these numbers are simple indicators of company health. You have 10 million active users? Great! The average purchase amount is only $4.37? Uh-oh. When you look beyond the surface, however, metrics also offer the secondary benefits of unit tests. Specifically:
Metrics make it easier to experiment. A metric is a series of points, not a single point in time, and looking at a metric’s trend enables better experimentation. Let’s say you want to change the business hours of your store. How do you know if your change had a positive effect? You can measure daily revenue after the change, but that doesn’t really tell you if your change was good. To know that, you need a baseline: the daily revenue before your experiment began. Keeping track of a metric over time gives you a standard to compare your experiments against.
Metrics implicitly document what’s important to your business. In a perfect world you’d measure and optimize everything simultaneously. In reality, your resources are limited, so you pick a handful of metrics (often called key performance indicators or KPIs), measure them regularly, and work on improving them. The metrics that you pick say a lot about your company and your priorities. When I worked at LinkedIn in its early days (2003-2005), I received daily emails that summarized key stats like how many new users were signing up, how many invites people were sending, and how many introductions were being made on the website each day. Even if you didn’t know anything about LinkedIn, the metrics we were tracking indicated that the company was concerned with growth and engagement and unconcerned with revenue. (On a side note, because people naturally assume that you’re measuring the things that are most important to you, make sure that your metrics correspond directly with your company’s objectives. Avoid vanity metrics at all costs).
Metrics help you focus on the right things. As the saying goes, what gets measured gets improved. If you’re not measuring something, it’s very hard to tell if it’s getting better. Did adding feature X improve revenue per user? “I think so” or “it feels that way” are not good enough – what you need is an objective number. Without metrics, you might work on ten things and assume that some of them move you in the right direction; with metrics, you work on ten features, see that 3 of them help, 5 hurt, and 2 have no effect, and so you get rid of everything except the 3 helpful features, making your product simpler and more focused.
Finally, in the same way that unit tests can reveal when new bugs are introduced in a piece of code, metrics can reveal when you make a misstep. If you are measuring the number of people who visit your site every week, then a sudden 15% drop indicates that one of your recent features is probably broken or not having its intended effect.
Metrics have many benefits beyond assessing the current health of a company, and as a results it’s hard to overstate their value. Figure out what’s important to your business, design a handful of metrics that correspond to your priorities, and then use those metrics to run focused experiments that move you in the right direction. Happy measuring!