Break Points
On design metrics
Jack Ganssle
7/18/2011 1:56 PM EDT
Good friend George Dinwiddie has a blog about software development, and George's latest posting caught my attention.
George talks about the balance between people and process. He faults organizations that find the most productive developer, and then tries to clone him across the team by duplicating whatever processes he uses.
I agree that even though this approach is seductive to some managers it’s doomed to failure. But there’s a more fundamental problem: without productivity metrics it’s impossible to know if the team is getting better or worse.
In my travels I find few firmware groups that measure anything related to performance. Or quality. There are lots of vague statements about “improving productivity/quality,” and it’s not unusual to hear a dictate from on-high to build a “best-in-class” development team. But those statements are meaningless without metrics that establish both where the group is now, and the desired outcome.
Some teams will make a change, perhaps adopting a new agile approach, and then loudly broadcast their successes. But without numbers I’m skeptical. Is the result real? Measurable? Is it a momentary Hawthorne blip ?
Developers will sometimes report a strong feeling that “things are better.” That’s swell. But it’s not engineering. Engineering is the use of science, technology, skilled people, process, a body of knowledge – and measurements – to solve a problem.
Engineers use metrics to gauge a parameter. The meter reads 2.5k ohms when nominal is 2k. That signal’s rise time is twice what we designed for. The ISR runs in 83 μsec, yet the interrupt comes every 70. We promised to deliver the product for $23 in quantities of 100k, and so have to engineer two bucks out of the cost of goods.
Some groups get it right. I was accosted at the ESC by a VP who experienced a 40% schedule improvement by the use of code inspections and standards. He had baseline data to compare against. That’s real engineering. When a developer or team lead reports a “sense” or a “feeling” that things are better, that’s not an engineering assessment. It’s religion.
Metrics are tricky. People are smart and will do what is needed to maximize their return. I remember instituting productivity goals in an electronics manufacturing facility. The workers started cranking out more gear, but quality slumped. Adding a metric for that caused inventory to skyrocket as the techs just tossed broken boards in a pile, rather than repair them.
Metrics are even more difficult in software engineering. Like an impressionistic painting they yield a fuzzy view.
But data with error bands is far superior to no data at all.
Jack G. Ganssle is a lecturer and consultant on embedded development issues. He conducts seminars on embedded systems and helps companies with their embedded challenges. Contact him at jack@ganssle.com. His website is www.ganssle.com.
|
|
|
|
|
cdhmanning
7/18/2011 8:27 PM EDT
"To measure is to know". Certainly, but it only helps if we know what to measure and how to measure it. Sometimes all we have are feelings.
To torture your resistor analogy. If we only had a scale, does it help to find out that the resistor has a mass of 0.3 grams?
Managers like to oversimplify both people and tasks because that is all management tools and methods can cope with. Everything is just reduced to Man Hours and Function Points etc so that it is easy to fit the numbers to Gantt charts and spread sheets.
We all know that Fred is wizard at writing interrupt service routines and John is good at GUI, so when we just treat them as interchangeable Man Hours, is it any wonder that the outcome is pure fiction?
Every person is different and every task is different. Every potential assignment of person to task will give a different outcome.
Most management systems are based on industrial widget making principles where it is very uncommon for one person to be more than twice as productive as another and the tasks can be controlled and refined.
Engineering is not like that. There is wide variety in capability and each task is different.
Sign in to Reply
krwada
7/19/2011 9:04 PM EDT
It is far easier to get a good result if your developers are excited and believe in what they are doing. I believe this is called "ownership" no?
Sign in to Reply
cdhmanning
7/19/2011 11:11 PM EDT
So how do you measure excitement and ownership to determine whether your group is getting better or not?
The only metric I've ever seen to measure commitment to a project is divorce rate. One place I worked they pushed the programmers harder if the divorce rate dropped below 10% per year.
Sign in to Reply
krwada
7/20/2011 12:08 PM EDT
The short answer is you can't. I suppose it is these emotional intangibles that separate the great managers from just decent managers.
I have heard of the 'divorce rate' metric. Usually, companies that use metrics like this also like to enforce a forced RIFF twice a year. The belief here is to instill the Fear component to motivate the work force.
In all, most of this does not matter much to me. Money, is money ... and spends just as well if one earns it from a nice person ... or earns it from a not-so-nice person.
Sign in to Reply
GordonScott
7/22/2011 4:43 AM EDT
People are at their most productive when they enjoy what they do.
So finding out what people best like doing and least like doing, allows one to offer tasks to get the best out of them.
Putting people under high pressure to deliver, for most people reduces enjoyment, reduces their morale, increases the number of risks on which they take a gamble `if we're to have a chance of meeting that deadline' and ultimately hits either productivity, quality or both (IMHO).
Even in engineering, I believe there is an important place for subjectivism.
Sign in to Reply
lwriemen
7/22/2011 11:56 AM EDT
Capers Jones has some really good books on the subject of software metrics. The biggest problem isn't really how to measure, but why to measure. Measurement exposes weaknesses as well as strengths. Some people want to be able to keep the illusion of "best in class" and not take risk of not "measuring up".
Some companies have a culture where any management action is viewed as effective. They don't want to know the truth. The same can go for programmers, who don't want to challenge their beliefs. To borrow from Capers Jones' data, C++ is 2.4x more productive than C, and SMALLTALK is 2.5x more productive than C++. Of course, Capers points out that those numbers are averages and actual numbers will be dependent upon many other factors.
Sign in to Reply
cdhmanning
7/24/2011 5:09 AM EDT
The problem with any numbers is that they are far too easily picked up as sound bites and used out of context.
I can easily see "C++ is 2.4 times as productive as C" being used to change projects from C to C++ without regard for the cases where this might or might not be true.
While software metrics are very useful for tracking within individual projects, they are - in general - poor at comparing between projects.
Is writing 10 function points per day in a GUI application more productive than writing 5 function points per day of device drivers?
When we're clearing up bugs, some are trivial and some are not. Eliminating typos in GUI buttons is easy. Finding a problem with some interrupt handler is a bit more tricky.
Sign in to Reply
lwriemen
7/27/2011 10:15 AM EDT
Capers discusses the differences between types of projects as well, and you are right about needing to understand that there are various factors to take into consideration when comparing projects.
However, you qualify your statement, "poor at comparing between projects", with an "apples to oranges" comparison. It could be used out of context to say you can't use metrics to compare one device driver project with another. I'm not sure if you are trying to use that as an argument for not using metrics, but it would certainly be an invalid one.
One of the benefits about function points is that they are used to capture the whole development cycle, not just writing code, so the productivity definition has to account for verification and design effort. If the metrics are used well, one can say that the 10 FP/day GUI and 5 FP/day device driver have equivalent productivity.
Sign in to Reply
MichelT
8/8/2011 8:55 AM EDT
Hi Jack,
I agree that metrics of all sorts are needed, along with detailed design practices that engineers should comply to. And as far as one can see, there is a pretty long history of checklists that design teams have been using to report on their quality progresses and achievements. Typically in form of Excel files to be filled in manually, plus home-grown scripts in option.
In my view, a good QA environment must add automation (“fact-based reporting, rather than manual”) and instant sharing (“web-based dashboards”). That’s the only way to turn raw design data into valuable information at project level.
You may want to have a look at this short video if you have 2 minutes : http://www.satin-tech.com/videos.html.
Michel Tabusse, Satin Technologies.
Sign in to Reply