How to improve verification planning

Project management is all about planning and execution. If everyone properly plans their verification project, why do quality problems and schedule slips persist? It really comes down to the adage "Begin with the end in mind." A good plan contains detailed goals using measurable metrics, along with optimal resource usage and realistic schedule estimates.

When managers and their teams engage in planning, the results of that planning process typically do not address the common problems that cause schedule slips, resource productivity issues and product quality. Most verification plans focus solely on task performance rather than defining the verification problem, independent of its solution.

This almost guarantees gaps leading to bug escapes, schedule delays to fix those bugs, or significant resource strains and inefficiencies. With a bit of improvement in planning we can do much better.

Why verification planning is incomplete

Most teams don't do complete verification planning. They leap to verification environment development before design because they are stretched extremely thin.

This can lead to an unchanged or inflexible plan which becomes nearly irrelevant because there is no easy way to maintain the relationship with the actual project. In other words, the verification plan isn't really a plan, it's merely a set of incomplete discussion notes that atrophy as the project moves forward, as the team begins working and learns new things about what it must accomplish.

The solution is to make the verification plan an executable part of the verification process itself. A verification plan becomes executable when it is read by a verification process automation tool. A solution that is used to organize and generate reports about project status and becomes the basis for analyzing data to determine next steps.

The plan's value is maximized, serving as the genesis and touchstone of the verification process throughout the life of the project. This directly increases the return-on-investment of developing and maintaining the verification plan by automatically using it to measure verification completeness. Thus, when changes in the project are necessary, those changes are introduced, tracked and measured through the updated executable verification plan. This makes the plan a continuous valuable part of the verification project, from specification to closure.

Figure 1 — The verification productivity gap

Better planning, and use of the executable plan to measure project completion, leads to higher quality, increased schedule predictability and improved resource productivity. With a better plan, it's easier for the team to produce the desired results.

Using the plan all through the project helps surface issues earlier, which is a key to staying on schedule. Tracking progress against the plan metrics enables everyone on the team to self-manage and stay focused, which leads to improved productivity.

What is good planning?

Verification planning needs to change from a focus on "how" to a focus on "what." By starting with the important goals of what needs to be verified, the team can ensure the plan is complete and balanced. The plan in this case is more than an engineering specification of how the verification is to be accomplished.

It is an agreed upon statement of what the project must accomplish to be successful. It identifies success criteria in a prioritized fashion — for example, these features must work; these features should work in the first release; and these features are nice to have working in the first release. It also identifies the details of how the success criteria will be measured, indicating the project has reached closure.

Basics of verification planning

From a high level process perspective, verification planning is actually quite straightforward. The basic steps of verification planning are:

1. Analyze the device specifications
2. Scope the verification objectives
3. Identify the feature set of the design
4. Design detailed coverage models
5. Select aggregate metrics to track progress
6. Use historical metrics to estimate effort and schedules