Have We Lost Our Ability to Estimate Software Size

I can clearly remember that day I arrived at work - towards the end of the year 2003 – it was easily before 6am. I was leading a small team tasked with prototyping a test environment for a NASA proposal. I stood there amazed as I watched my two programmers demonstrate a completely reengineered satellite simulation environment. Wait – let me be clear - within only a few days – they rewrote close to 30,000 lines of FORTRAN and another 4,000 lines of assembly code. How did they do it? They called it “goop.” “The hand cleaner?” I asked – rather befuddled. No, they were referring a new development language by LabVIEW (National Instruments) called “GOOP” – short for Graphical Object Oriented Programming.

That was the day I decided to stop being a programmer. I was, at my ‘age,” no longer really interested in keeping up with the latest programming paradigms (and vernacular – such as “paradigm”). I decided to abandon the past and embrace my role as a program manager. But now, looking back I wish I would have asked a few more questions.

Measuring Failure

In 1995 the Boston, Mass. - based IT project management research and consulting firm The Standish Group released their first CHAOS Summary report. The report quickly became an industry score card for measuring the success or failure of IT projects; due mostly in part to the astounding percentage of failed projects disclosed in the report. The report served as a wake-up call that appears to have been heard - the 10th anniversary CHAOS report announced that the percentage of failed projects had been reduced by more than half. But alas, within a mere five years, the number of failed projects is back on the rise; the 2009 Standish Group CHAOS report indicates that nearly 1 in 4 projects are doomed. But why?

According to the original 1995 CHAOS report, to improve the probability of success projects should be reduced in complexity and the software “grown.” The recommendation was to reduce software into smaller, more manageable segments, and develop it outward. If this reported reduction in failed projects is to be believed then it appears the software industry was diligent in “growing” projects using smaller elements (1). However, what should also be understood is that those projects were comprised of many smaller pieces that were easier to “size.”

In 1995 the most common approach to sizing software was to count the source lines of code (SLOC) or count Function Points (though less prevalent). Software sizing was an established and mature methodology spanning over twenty years – with a myriad of tools available to automate the process and additional metrics available to measure software complexity and probability of defects (bugs). If the size of a project was understood, then the ability to estimate schedule and effort could easily be modeled by applying previous performance measures (and many other parameters). By forecasting a realistic estimate early in the development cycle there was a significantly higher probability of the project’s success (on time, within budget, at promised functionality) – and hence – less failure.

Back to the Future

It’s 2003 again, and I’ve just been told about a new methodology that allows anyone to build software using graphical components. This was not really all that new; in the mid-1990s through early 2000s “visual” and “portable” languages gained industry acceptance and began to dominate the development landscape. Within just a few years, languages that could be “produced” by an environment became the lingua du jour. After all, who could argue with the massive scale of economy that software manufacturing tools could generate using “Visual” programming? And for me in particular - after three days of watching my team crank out “GOOP” - I was a hero to my management.

But wait - notice the timeline in the Chaos study mentioned above and the resurgence of software failures. While I am not a big proponent of causality – let’s at least take a few moments and explore this potential contributor to the trend in software failure. First, a picture from the TIOBE Programming Community (2) – the unofficial keepers of what is popular in programming languages.

The languages with the most growth in popularity for five years have been: Java, followed by C#, JavaScript, and then Ruby. Some older languages also in vogue are Perl, C, and Visual Basic. Why? Perhaps it’s because most of these languages have become more sophisticated, are wrapped in integrated development environments, and are positioned with the sole purpose of increasing productivity. In a word – they’ve become more “visual.” While it may be easier to build the code, there is little consideration as to how the generated code should be “sized.” In fact, the environment builders boast that one barely needs to fiddle under the hood; Draw, click, and Poof – instant code that runs.

Size Matters

The most significant driver to how much time, cost and effort it takes to build software is the scope (or size) of what is to be built and therefore one of the biggest factors in accurate estimation. As the Godfather of software estimation has warned us (Barry W. Boehm) - “The biggest difficulty in using today’s algorithmic software cost models is the problem of providing sound sizing estimates” (3). How does an estimator measure “GOOP” and how many lines of code that a code generator inserts are really needed? What percentage of a C++ template can we remove (if we dare) and keep the Class fundamentally stable – yet concise. As Mark Twain once said – “the hardest part about writing is removing all the extra words.”

Is it possible to count software lines of code anymore? Even using the best code counting tools available – aren’t they really just counting lots of lines of code that may be unnecessary? Or in inverse – how much time did it take the programmer to remove all that code that should not have been counted – and was not? My suspicion is that all the code stays in (unless a standard with high rigor like FAA DO-178B verified the system).
Since I’m on the topic of counting code - what happened to Ada and FORTRAN; those stalwart languages of the 80’s and 90’s that were easy to count? Alas, they are now ranked number 24 and 25; again, no assumption of causality. (But yes my tongue is planted firmly in cheek). Estimates seemed so much easier then. Cue the music.

Hmm, there is something becoming clear in the Standish report – assuming I am not making what statisticians would call an “error of confirmation”; I would propose that perhaps the industry has made capturing the size of software too complicated – and as a consequence - our ability to accurately create a good cost estimate. Ultimately, if the industry is moving away from “countable” languages and migrating towards “visual” representations then some mechanism needs to be established that can accurately correlate effort to size or vice-versa.

Here’s a thought – remember those thousands of dollars used to purchase graphical requirements and design tools – such as the IBM Rational Rose, RSA Integration, and Rhapsody? Why not use the output of these use case models to calculate Use Case Points (unadjusted) which can then be fed into the parametric models. Are you building “Design Patterns?” Why not spend a bit more time and calculate the COSMIC Function Points and publish them along with the pattern – that way the cost of implementing the pattern can be calculated. At a minimum, before charging forward from requirements to code – attempt to calculate some “functional” size that the parametric model accepts and proceed – then go back later to see if your effort per function assumptions were correct.

The software community has made great progress in creating tools to improve productivity – but our estimates are wrong because we stopped half way! We need to regroup and identify software size as it relates to software cost and involve parametric tools to calculate accurate estimates. Until then – it’s purely guess work to estimate new product development and blind trust in tribal knowledge when modifying existing applications. After all – just how long does it take to make GOOP?

1 - Jim Johnson, chairman of The Standish Group, says he was so surprised to observe a dip in IT project success rates that he waited an extra four months before publishing the CHAOS report to make sure its findings were accurate. He attributes the increase in IT project failures to the recession, which according to economists began in December, 2007, and subsequent budget cuts.
2- http://www.tiobe.com/index.php/content/paperinfo/tpci/index.html
3- Software Engineering – Barry W. Boehm’s Lifetime Contributions to Software Development, Management and Research., Edited by Richard W. Selby , Wiley-IEEE Computer Society Pr; Reprint edition (June 4, 2007)