Measuring the Unmeasurable

In a previous post, I wondered how you measure yourself as a developer. The available metrics are pretty poor, overall. Consider all the ways a developer can be "good"--fast, good at translating requirements into functionality, good at the key algorithm, good at clear, concise code, good at predicting when she'll be done with a given bit of code... on and on. How do you measure that? The short answer seems to be: you don't. I haven't found a single metric which addresses developer quality which is in any way objective. Then I cast back to my earlier life as an engineer--and surprise! Exactly the same situation exists there.

At the best large companies I've worked for -- Babcock & Wilcox, GPU Nuclear, Oracle -- jobs I look back on with fondness -- individual performance was evaluated by senior people, and job descriptions were carefully thought-out lists of subjective performance metrics. When I was promoted from Engineer 2 to Engineer 3 at GPU Nuclear, my boss and I went over the metrics for that job to see which I hadn't quite mastered yet, and that became my development plan. When I mastered Engineer 3, I would start on the list for Senior Engineer 1, and when my superiors felt I had mastered enough of that list, I'd be promoted again.

No such list has governed my development as a software developer, designer, and architect, except at Oracle. I've been promoted several times, but there were never any objective criteria against which I could say "see? I have done this", or "boy, here's an area I should work on". More than once, I've been involved in projects where the "senior" architect, designer, or developers were among the least-accomplished on the team. This can only happen in a situation where measures of capability--even if they are evaluated subjectively--are not present.

The chief architect at Sabre Airline Solutions and I spent some time a year or two back discussing job descriptions for various programmer levels and career paths. While we didn't discuss specifics, I now wish we had. In my career as a developer, I have seen only one set of specifics for different developer levels: the job descriptions in Oracle Consulting Services. I wish I had those descriptions now; my recollection is that they were concise, complete, and compelling. I'll try to collect some job descriptions for developers over the next couple of months. Perhaps there's something available that could be used to address the whole "what's a good developer" question.