The things functional verification engineers do may seem important, but they may not be useful to the company. If we want to be better engineers, then we should review our work from time to time and analyze whether what we did was worthwhile.
The goal of this article is to discuss which activities in verification actually provides value to the business. Let’s propose a maybe flawed metric and evaluate our activities against that metric. We can rank the activities against the metric in order to see which activities matter and which don’t. Informed by this ranking we can try to move away from relatively useless work to more useful work.
Finding a Metric
A successful tape-out to production is the primary indicator of business value in the ASIC projects. Before tape-out there is nothing to sell, after tape-out there are chips to sell. A failed tape-out often also provides no business value, but it is not all binary. For example: Customer A requires features X and Y to be working. Customer B requires features X and Z to be working. Customer C requires feature X to work, but is only willing to pay a lower price. The project manager wants some debug features, just in case.
You can already guess how project planning is affected. Feature X is a core feature that has the most business value. There are no customers without this feature. Feature Y and Z are high priority because customer A and B will pay for them. Debug features are low priority because there are no-one who will pay for those before tape-out. This is pretty normal and probably not new to you.
From a functional verification perspective this means that completing verification of feature X and then feature Y and Z add value to the business. If you are old school test driven, then writing the last test for feature X will add the most value. If you are metrics driven then closing coverage for feature X will add the most value.
Let’s use coverage closure as our metric for linking verification activity directly to business results. Other activities are only useful if they aid in coverage closure.
Finding Data
The Wilson Functional Verification Studies commissioned by Mentor Graphics have recorded the activities of verification engineers for the last decade. The activities are broken down into 4 different groups:
- Test Planning
- Testbench Development
- Writing and running tests
- Debug
Notice how debug is taking a larger share of effort over time.
Coverage closure is unfortunately not one of those activities, so we will have to estimate how all activities aid in closing coverage.
Ranking Activities
1. Test Planning
Test planning is most likely the highest value activity that a verification engineer can work on. Test planning defines what needs to be covered and what tests need to be developed thereby directly aiding in coverage closure. Sizing the needed coverage correctly will improve project schedule and cost. Missing a feature here could cause a respin and missed sales. Test planning matters. The features listed in the document are the features that will actually be present in the chip and the features that will actually work at tape-out.
Unfortunately, test planning is also easily boring, so engineers have a natural tendency to under invest in planning. Project Managers have often been able save weeks of effort and double work by combing through the test plan.
An up to date testplan also aids managers with business decisions on staffing, project schedule, priorities and cost.
2. Writing and running tests
Tests are needed to close coverage. Some tests may not close any coverage, but most of them do and thus the writing and running of tests is closely aligned with coverage closure. This is a valuable activity. The more time that you can spend in this activity, the more the business will generally benefit.
3. Testbench Development
Developing a good testbench will make writing and running tests much faster and aid in the closure of coverage. In this regard, testbench development can be a higher value activity than writing and running tests. The opposite is also true: A bad testbench makes writing and running tests harder and coverage closure harder. Normally, testbench development will help the project up to a point where the returns are diminishing.
Unfortunately, testbench development is often fun and some times testbenches get features that don’t really help the project instead of features or tests that do.
All in all, testbench development is often valuable, but soon not as valuable as writing tests that close coverage.
4. Debug
Debugging is an unpredictable task. It is only the last part of debugging where the bug is isolated and fixed that is useful. In the worst case debugging never finds any bug. The effect on coverage closure is minimal at best. All in all this makes debugging a relatively low value activity.
Unfortunately, debugging is the largest task of verification engineers in the Wilson Functional Verification Studies and it is increasing. Something needs to happen.
Discussion
We have ranked the verification activities based on their general usefulness. As a project progresses the relative value will change based on the current situation, but from a macro perspective this ranking is good enough to have an overall discussion.
As we see in the Wilson Verification studies, debugging is taking the largest share of verification work, it is the lowest value value activity and it is growing. This is the exact opposite of what we want. We want to have debugging be a smaller part of verification activity and be decreasing in share such that the focus is on higher value activities.
When you are debugging, you are looking for relevant information that can help you isolate a bug. This information can come in many different forms. It could be:
- logs
- assertions
- waveforms
- test suites
- test reports
- recent changes
The more relevant information that is readily available to you, the less time you have to spend finding it and the less time you will have to debug. If relevant information isn’t readily available or if there is information that you have to dig out repeatedly when debugging, then it is often better to add simpler tests or improve the testbench. This will give you more information now and help you in the future. Being stuck in a low value activity does not mean that you have to stay there.
References
- https://blogs.mentor.com/verificationhorizons/blog/2019/01/29/part-8-the-2018-wilson-research-group-functional-verification-study/