I’m looking for better methods of quality control; better estimation tools; and collecting reliable quantitative data on both productivity and quality

28/10/2011 |

Capers Jones, advisor to the Consortium for IT Software Quality (CISQ)

Wikipedia says that “Capers Jones is a specialist in software engineering methodologies, and is often associated with the function point model of cost estimation. He also collects data on software quality, software risks and software best practices. My first question is how do you define yourself and what is your contribution to the software engineering industry?

In the early 1970’s I was asked to collect quantitative data about productivity and quality within IBM. The data was interesting and I enjoyed collecting the information, so I never stopped. After more than 40 years of data collection I have information on about 13,000 software projects.

I collected internal data when I worked at IBM and ITT. After founding Software Productivity Research (SPR) in 1984 my team and I collected data from around 650 companies in 24 countries.

Aventia uses your metrics and predictive formulas for building a quantitative management base of software construction and testing process. You publish concrete ant tangible metrics. Let us know how these metrics are gathered, analyzed and how formulas were synthesized. What level of confidence or veracity can be attributed to your metric?

In 1984 I developed a questionnaire for collecting both assessment information and also quantitative data about quality and productivity. The questionnaire was developed about a year before the Software Engineering Institute (SEI) was first incorporated so I have been doing assessments longer than SEI. Unlike SEI, I collect quantitative data on quality and productivity as well as qualitative assessment data.

For about 15 years until I sold SPR and retired in 2000 I had more than a dozen consultants collecting data on between 25 and 75 projects every month. SPR still collects such data although I am now retired and no longer part of the company.

Both SEI and I use assessment questions but my questions have a different structure. My questions have a five-point scale and “1” is the best. With SEI, “5” is the best. My questions look like this:

PROJECT MANAGEMENT EXPERIENCE
1) All experts (implemented many priority projects)
2) Extensive (implemented some priority projects
3) Average (implemented at least one priority project)
4) Limited (participated in priority projects but did not manage them)
5) Novice (no experience or management of priority projects)

Because 3 represents the approximate U.S. average it is easy to show clients how the compare against U.S. norms.

In total I have about 375 such questions and I also collect quantitative data on productivity and quality.

I had several statisticians working for me so the accuracy of the historical data was within 5%. In fact one of the benefits of on-site data collection such as what my consultants did is to correct errors in historical data.

A very common error is that companies do not record unpaid overtime. Another common error is that they do not record project management effort.

A lot of your metrics are referenced to function points. In Spain this practice is seldomly used and frequently is viewed as an exhaustive and difficult method. How could you encourage our market to use it and appreciate the benefits of this technique? Could you provide some positive experiences to promote the extensive use of function points in our market?

Function point metrics are the most widely used metric in the world. The International Function Point Users Group (IFPUG) and the COSMIC users group have chapters and affiliates in about 30 countries.

Several countries such as Brazil and South Korea require that function points be used for all government contracts. I know that some Spanish companies also use function points and in fact have developed function point counting tools.

In the past function points were fairly expensive to calculate. A certified function point analyst only counts function points at a rate of about 400 functions per day. Since an “average” project counted with function points is around 1,000 function points in size, it takes more than two days to count function points by hand.

However several companies and researchers have developed much faster methods. In January I filed a patent for a high-speed sizing method that can produce sizes using both function points and logical code statements. The same method can also produce sizes using story points and use-case points.

My method can size a new project in about one minute and 15 seconds. It takes about one minute and 35 seconds to size an enhancement to a legacy application because it is usually necessary to size both the original application and also the enhancement. This double sizing takes a few seconds longer.

A very old method developed in 1975 by the inventor of function points, Al Albrecht, is called “backfiring.” Al and his colleagues at IBM measured size in both function points and logical code statements. This allowed them and other colleagues to publish data on the number of code statements needed to implement one function point.

For example COBOL required about 106 statements per function point in the procedure and data division. There are catalogs available that show the number of source code statements per function point for about 800 programming languages. For newer languages Java takes about 53 statements to implement one function point.

With regard to the software development operative model, we are attending to the extensive use of agile models versus cascade traditional models. We would like to know if the application of a quantitative and predictive model makes sense with agile development. How to use function points in agile scenarios?

Some Agile projects use function points but many do not measure much of anything. Some Agile projects use story points, but these are undefined and there are no benchmark data collections for story points so there is no way to compare such projects against international benchmarks such as those collected by the International Software Benchmark Standards Group (ISBSG).

If you compare Agile against other methodologies using function points, you will see that Agile is much better than waterfall development, but about equal to several other methods for productivity.

Agile projects seldom use pre-test inspections which are more effective than testing, so Agile lags in quality control. You need a combination of inspections, static analysis, and formal testing to get above 95% in total defect removal efficiency.

From the software engineering point of view, application software quality management is a key point; in fact, its economical dimension is the focus of your most recent publication “The economics of software quality”. What economical return benefits can we use to convince the organizations to invest in software quality? What useful guides can we find in your new book?

The software industry is embarrassingly bad in quality control. Most companies do not have any idea how to measure the economics of quality. The most common metric, “cost per defect” penalizes quality and achieves its best results for the buggiest applications. The “lines of code” metric is so bad for measuring quality that I consider it to be professional malpractice. Lines of code measures ignore bugs in requirements and design, and also penalize modern high-level languages.

If you measure quality costs using function points, you will find that there is a perfect correlation between high quality levels, low costs, and short schedules. Projects that can top 95% in defect removal efficiency levels all have shorter schedules and lower costs than “average” projects that only remove 85% of bugs.

AVENTIA is working actually with big relevant companies in the definition of testing outsourcing models to be used in testing factories. What are the operative and economical challenges of the testing outsourcing? What are the contributions of your measurement model to software testing in general and to testing factory management in particular?

From measuring a number of outsource projects, the better outsource vendors are about 10% better than their clients in terms of both quality and productivity.

Of course there are some outsource projects that fail, but in general the outsource community is better. This is why outsourcing is a growth industry.

Testing is a more complicated topic. The companies with the very best quality use a synergistic combination of inspections, static analysis, and formal testing.

The best testing companies also use certified test personnel and they develop test cases using formal mathematical models such as those based on the design of experiments. They also measure test coverage, cyclomatic complexity, defect removal efficiency, errors in test cases themselves, bad fixes or new bugs in attempts to fix old bugs, and a number of other topics.

Other key aspect of software quality is, undoubtedly, the definition and management of software requirements. Industry reports tell that software developments teams have practically not improved in this area. What is your opinion about the maturity of the software industry with regard to definition and management of software requirements and what is the contribution of your model in terms of benchmarking?

Software requirements are a weak link in the chain of software engineering technologies. Requirements are often less than 60% complete; they contain harmful or “toxic” requirements that should not be included; and they grow at more than 2% per calendar month. They are also hard to understand.

In general requirements defects cannot be found and removed by testing. They can only be found and removed by using formal inspections. Requirements defects can be prevented by using prototypes, joint application design (JAD), and quality function deployment (QFD). However not many companies use these methods.

There are also tools that can analyze requirements and find errors. These are similar to static analysis only they work text. There are other tools that can calculate the readability of requirements using the FOG index or other readability scores.

In general requirements are troublesome and most companies are not very good at handling requirements.

Lastly, what is your future investigation focus about software engineering? Are you working on your next book yet?

My future research will continue the same threads as my past research: I’m looking for better methods of quality control; better estimation tools; and collecting reliable quantitative data on both productivity and quality.

Back

Developed by BabSoft