Who's First - The CMMI Chicken or the Six Sigma Egg?

This post is abstracted from “Connecting Software Industry Standards & Best Practices: Lean Six Sigma and CMMI®”

Crosstalk Feb.2007 by Gary Gack and Karl Williams

Software professionals, especially those working in the Department of Defense environment, face a somewhat bewildering array of relevant standards and best practices. As awareness and penetration of Lean Six Sigma in this environment have increased significantly over the last several years, we find many organizations struggling to understand and leverage the relationships between Lean Six Sigma and several other approaches to software process improvement, including CMMI®. In our view, the cases described below and other industry experience answer in the affirmative four questions we hear quite frequently:

• We are already doing Six Sigma; does it make sense to do CMMI® as well? Clearly each of the cases below felt there was additional value to be gained from engaging CMMI® in addition to Six Sigma.
• We are probably CMMI® (staged) level 2 or 3; does it make sense to do Six Sigma before we get to level 4? None of these organizations had reached level 4, but all had realized benefits from Six Sigma at lower levels.
• Management wants us to get to level 5 as soon as possible; can Six Sigma help us get there quicker, or will it slow us down? Case 3 below, as well as other results reported in the literature, clearly demonstrates Six Sigma can reduce the time needed to move to higher maturity levels.
• We are already doing CMMI® ; does it make sense to do Six Sigma as well? In her article “Using Six Sigma in Software Development” in News@SEI Lauren Heinz wrote the following:
“Each year at Lockheed Martin, corporate management challenges its Integrated Systems & Solutions business unit (IS&S) to reduce total costs. Each year, IS&S uses Six Sigma tools to make it happen. Six Sigma has resulted in significant cost savings, said Lynn Penn, director of quality systems and process management at IS&S. It’s a structured approach that provides more than a checklist—it shows you what’s coming next, lets you look at data from different views, and gives you a big picture of your practices for making decisions.
Lockheed Martin is part of a growing number of organizations using Six Sigma to improve software quality and cycle time, reduce defects in products and services, and increase customer satisfaction. As Six Sigma evolves from an improvement framework for the manufacturing sector to one that can be applied across all levels of an enterprise, the SEI is looking at ways that Six Sigma has benefited software and systems development.”

Answering a fifth common question necessarily moves into the realm of opinion – the following is ours:
• We are just getting started on process improvement; should we do Six Sigma, CMMI®, or both? At the same time, or one or the other first? – Our experience convinces us that most organizations will get measurable business results more quickly with Six Sigma than with CMMI®, but ultimately will need the additional insights available from CMMI® to realize maximum benefit. Government contracting organizations will in most cases need to do both in parallel - CMMI® to satisfy government bid requirements and Six Sigma to ensure tight and near-term linkage to measurable business results. Even when CMMI® comes first, level 4 and 5 will invariably lead organizations to something that is very like Six Sigma, even if by another name.

Lean Six Sigma places primary emphasis on understanding and managing performance (outcomes) while CMMI® (often in practice if not in principle) places more emphasis on maturity/capability level. Many within the SEI have expressed concern about over-emphasis on the rating alone. Certainly level rating is important for government contracting organizations, but is not sufficient by itself to quantitatively demonstrate improved outcomes in terms of cost, quality, or cycle time.

CASE STUDIES

Williams provided CMMI® training and performed Class C Appraisals in three different organizations that were well into Six Sigma deployment. Black Belts, Green Belts, and Champions had been trained and all staff had some form of Six Sigma orientation. The results across the board were significantly different than the average initial appraisal. A significant number of processes were already documented and used, as opposed to the usual blank stares when procedures / templates are requested. All three had results more typical of a second or third round of appraisals than the normal results.

Especially noticeable was the difference in the quantitatively based Process Areas (PA), e.g. Measurement and Analysis (MA) at Staged Level 2 and the Level 4 and 5 Process Areas covering Quantitative Process Management (QPM) and Organizational Process Performance (OPP). In most initial Class C Appraisals, we do not even bother to look at Levels 4 – 5, sometimes not even Level 3. In all three cases, plans called for reviewing only Levels 2 and 3. As the appraisals progressed, the results were so startling that the Levels 4 – 5 were also reviewed.

Case 1 Results
• Measurement and Analysis Process Area – All goals and practices compliant with no improvement opportunities.
• Level 4 Process Areas (2) – All but two goals satisfied. All practices largely compliant with three improvement opportunities.
• Level 5 Process Areas (2) – All but two goals satisfied. All practices largely / partially compliant with two improvement opportunities.
• Follow-up accomplishments – The Group completed a Level 2 Appraisal as planned for calendar-year objectives. The Level 3 Appraisal was successful nine months later. Plans were halted when the organization was acquired by a large multi-national organization.

Case 2 Results
• Measurement and Analysis Process Area – All goals and practices compliant with one improvement opportunity.
• Level 4 Process Areas (2) – All but two goals satisfied. All practices largely / partially compliant with five improvement opportunities.
• Level 5 Process Areas (2) – All but one goal satisfied. All practices largely compliant with two improvement opportunities.
• Follow-up accomplishments – The Group was appraised at Level 3 three months later and is planning for a Level 5 Appraisal this quarter.

Case 3 Results
• Measurement and Analysis Process Area – All goals satisfied and practices compliant with one minor improvement opportunity.
• Level 4 Process Areas (2) – All but one goal satisfied. All practices largely compliant with three improvement opportunities.
• Level 5 Process Areas (2) – All goals and practices satisfied.
• Follow-up accomplishments – Group was appraised at Level 3 one month later and Level 5 six months following that event

All three case study organizations shared other attributes as well, including the following:
• All processes and procedures contained measurements, root cause analysis, and continuous improvement follow-up.
• All organizational results exceeded the data published by the Software Engineering Institute benchmarking activities.
• All levels of management and staff used the measurements on a daily basis.

The above results are phenomenal for first time Class C Appraisals. Upon further investigation, all stakeholders agreed that the Six Sigma tools and Six Sigma mindset had contributed greatly to such unparalleled results.