Introduction
Software metrics aren’t just for fancy-pants NASA projects. Here’s a simple introduction to measuring IT project performance that even a space monkey can apply.
Feedback is a very useful thing. When we go on a diet, it’s useful to know how much weight we’re losing. When we train for a race, it’s useful to know how long it takes us to reach the finish line. In every aspect of our lives, it’s useful to know how we’re really doing.
IT projects are no different. If you want to improve the performance of your project team, feedback is essential.
The Four Pillars
Instead of talking about IT, let’s start by talking about golf:
Imagine you are the manager of a golf team. You have an important tournament coming up, and you want to have a reasonable chance of winning – especially as you plan to take a bet on it, too.
What kind of things might you need to know before you place that bet? Funnily enough, pretty much the same kind of things an IT manager needs to know.
- How long things take – how long does it take to play a certain number of holes, for example?
- How big or complicated things are – is this hole par 4 or par 5?
- How good things are – did the ball go where the player aimed to put it?
- How costly things are – how many shots did it take to get the ball in the hole?
These cover the four pillars of performance:
- Time
- Scope
- Quality
- Cost
How can we tell a good golf player from a bad golf player?
Well, what if told you that Bill took 5 hours to play 18 holes? And Ben took 3.5 hours to play the same course? And what if I told you that Bill took 6 shots to complete the 9th hole, and Ben took only 4? And what if I told you that Bill missed 6 out of 13 shots, and Ben only missed 3? Who is the better golfer: Bill or Ben?
Obviously, it’s Ben. If Bill wanted to improve and give himself a chance of beating Ben, he might sensibly organise his practice regime around a handful of golf metrics like the examples we’ve seen.
So if you were managing a golf team, practising for a tournament, metrics could provide you with very useful feedback on whether or not it would be worth taking a bet on them.
IT projects are no different. The four pillars of performance apply every bit as much, and IT metrics can be just as useful in helping you to improve your game – so to speak – as well as helping you to judge where your budget might have the best chances of bringing a return.
Let’s see how the four pillars might be applied:
- Time: how long does it take to deliver a feature? How long is a release cycle? What time elapses between a bug being reported and the fix being released? And so on.
- Scope: how complicated is a feature? How big is the software? How big is the team? And so on.
- Quality: how many known bugs does the software have? How maintainable is the code? How scalable is the application? And so on.
- Cost: how much does it cost to deliver the software? How much does a particular developer cost? And so on.
As with golf metrics, the real magic happens when we start to compare the four pillars. If team X can deliver 26 features in 12 weeks, and team Y can deliver 32 in the same time, what would you say? If team X delivered 50,000 lines of code with just 17 known bugs, and team Y delivered 28,000 lines of code with 213 reported bugs, what does that tell you? If Bill delivers twice as much as Ben, with half as many bugs, and Bill costs £500 a day and Ben costs £350 a day, who represents the best value for money?
Obviously, it’s Bill. But you’d surprised how many people would hire Ben instead of Bill. That’s because their understanding of performance is unbalanced.
Balancing Performance
The reason why it’s “the four pillars”, and not “the three pillars”, or “the one pillar”, is because it’s important to strike a balance when measuring performance. The saying goes that you should be careful what you wish for, and if you wish for the cheapest developers, for example, then you may regret not taking the other three pillars into account.
You can think of the four pillars as ropes attached to your project. When you pull harder on one rope, it causes your project to move more in that direction, and away from the others. If you pull too hard on cost, for example, you often do so at the expense of quality. Many teams focus almost entirely on deadlines, and are willing to sacrifice everything else to meet them. It’s very rare that teams pull hard enough on the quality rope, which turns out to be a false economy.
Stay Out Of the Long Grass!
An important lesson IT managers learn when they introduce metrics is that there is a direct relationship between quality and productivity. Let’s go back to the golfers:
When Bill hits the ball, he intends for it to end up in a specific place that will make his next shot easier – on the fairway or on the green. If the ball ends up in the rough, Bill’s next shot will be harder. It may cost him an extra shot or two to get back to where he needs to be. If it ends up in a bunker, it will cost even more shots. If it goes in the water, then he really is in trouble!
The same principle applies to software. Bugs take time to fix. That’s time that could have been spent adding valuable new features. The longer it is before bugs are discovered, the more time it takes to fix them. If you rush to meet a deadline, you may end up taking twice as long in the next phase to fix the bugs as it would have taken to avoid them in the first place by doing things properly.
Design quality has an equally direct effect on productivity. The less maintainable the code is, the harder it will be to add new features.
The best IT professionals know that to improve productivity, they need to beef up their quality assurance efforts. Throwing them out of the window always turns out to be the most expensive option.
Metrics Facts & Figures
If you’re thinking of applying metrics to your IT projects, but need to sell it to your boss first, here are a few facts about software metrics to help build your business case:
- Some of the most successful software organisations in the world use metrics to continuously improve their productivity. These include Google, IBM, Microsoft, Sun, Oracle, PeopleSoft, Sony Eriksson, Amazon, Yahoo! and – of course – NASA.
According to studies by the Software Engineering Institute, IBM and the Butler Group:
- The average productivity gain after metrics-driven software process improvement is more than 35% in the first year
- The average return on investment for metrics-driven SPI is more than 500%
Metrics Tools & Resources
Over the years, a plethora of tools and technologies have emerged that can help IT organisations implement and control their metrics programs. For programmers interested in code and design quality, there are tools that can analyse and measure programs written in almost every popular programming language:
- Java metrics tools
- JDepend – measures quality of packaging of Java classes by analysing dependencies between packages
- Semantic Designs – Java source code metrics
- PowerSoftware – very comprehensive suite of Java source code metrics
- NET metrics tools
- Semantic Designs – small suite of C# metrics
- NDepend – not just package dependencies like JDepend, but a whole suite of metrics
- C++ metrics tools
- CMT++ - complexity measures for C/C++
- Visual Basic metrics tools (yes, they do make them!)
For quality assurance managers, most commercial test management and bug tracking software offer some kind of defect analysis and reporting tool.
Project, programme and IT managers looking for an at-a-glance overview of multiple IT projects might be interested in investing in business intelligence “dashboard” software, like:
Other metrics resources can be found at:
- Software Engineering Institute
- Function Points
Summary
Exactly what you choose to measure for each pillar will depend on your specific circumstances. The following is just an example of 4 metrics that could be used to steer an IT project:
- Function Points (FP) / Iteration
- Bugs / Thousand Lines of Code (KLOC)
- Software Maintainability
- Cost / FP (or cost / KLOC)
Some enlightened IT managers, who have responsibility for governance over many projects, set up project dashboards that give them an at-a-glance summary of how each project is going, and warns them if any project seems to be “going into the red” on any aspect of performance.
Of course, metrics can’t tell you everything. As Mark Twain famously decried “there are lies, damn lies, and statistics”. You need to apply your own judgement as well as considering the numbers. Metrics, if applied with common sense and humanity – more carrot than stick – can be a very powerful tool in the struggle for better IT services and greater accountability between IT and the business.
Jason Gorman is an independent consultant specialising in lightweight, metrics-driven approaches to software process improvement. He has helped a diverse range of IT organisations to “up their game”, using metrics to steer their efforts and aid in strategic IT governance. His web site www.parlezuml.com is one of the most popular UML tutorials on the Internet, and he also works coaches developers, analysts, architects and managers in all aspects of software development.