What activities in functional verification are actually useful?

The things functional verification engineers do may seem important, but they may not be useful to the company. If we want to be better engineers, then we should review our work from time to time and analyze whether what we did was worthwhile.

The goal of this article is to discuss which activities in verification actually provides value to the business. Let’s propose a maybe flawed metric and evaluate our activities against that metric. We can rank the activities against the metric in order to see which activities matter and which don’t. Informed by this ranking we can try to move away from relatively useless work to more useful work.

Finding a Metric

A successful tape-out to production is the primary indicator of business value in the ASIC projects. Before tape-out there is nothing to sell, after tape-out there are chips to sell. A failed tape-out often also provides no business value, but it is not all binary. For example: Customer A requires features X and Y to be working. Customer B requires features X and Z to be working. Customer C requires feature X to work, but is only willing to pay a lower price. The project manager wants some debug features, just in case.

You can already guess how project planning is affected. Feature X is a core feature that has the most business value. There are no customers without this feature. Feature Y and Z are high priority because customer A and B will pay for them. Debug features are low priority because there are no-one who will pay for those before tape-out. This is pretty normal and probably not new to you.

From a functional verification perspective this means that completing verification of feature X and then feature Y and Z add value to the business. If you are old school test driven, then writing the last test for feature X will add the most value. If you are metrics driven then closing coverage for feature X will add the most value.

Let’s use coverage closure as our metric for linking verification activity directly to business results. Other activities are only useful if they aid in coverage closure.

Finding Data

The Wilson Functional Verification Studies commissioned by Mentor Graphics have recorded the activities of verification engineers for the last decade. The activities are broken down into 4 different groups:

  • Test Planning
  • Testbench Development
  • Writing and running tests
  • Debug

Notice how debug is taking a larger share of effort over time.

Percentage of verification engineering time

Coverage closure is unfortunately not one of those activities, so we will have to estimate how all activities aid in closing coverage.

Ranking Activities

1. Test Planning

Test planning is most likely the highest value activity that a verification engineer can work on. Test planning defines what needs to be covered and what tests need to be developed thereby directly aiding in coverage closure. Sizing the needed coverage correctly will improve project schedule and cost. Missing a feature here could cause a respin and missed sales. Test planning matters. The features listed in the document are the features that will actually be present in the chip and the features that will actually work at tape-out.

Unfortunately, test planning is also easily boring, so engineers have a natural tendency to under invest in planning. Project Managers have often been able save weeks of effort and double work by combing through the test plan.

An up to date testplan also aids managers with business decisions on staffing, project schedule, priorities and cost.

2. Writing and running tests

Tests are needed to close coverage. Some tests may not close any coverage, but most of them do and thus the writing and running of tests is closely aligned with coverage closure. This is a valuable activity. The more time that you can spend in this activity, the more the business will generally benefit.

3. Testbench Development

Developing a good testbench will make writing and running tests much faster and aid in the closure of coverage. In this regard, testbench development can be a higher value activity than writing and running tests. The opposite is also true: A bad testbench makes writing and running tests harder and coverage closure harder. Normally, testbench development will help the project up to a point where the returns are diminishing.

Unfortunately, testbench development is often fun and some times testbenches get features that don’t really help the project instead of features or tests that do.

All in all, testbench development is often valuable, but soon not as valuable as writing tests that close coverage.

4. Debug

Debugging is an unpredictable task. It is only the last part of debugging where the bug is isolated and fixed that is useful. In the worst case debugging never finds any bug. The effect on coverage closure is minimal at best. All in all this makes debugging a relatively low value activity.

Unfortunately, debugging is the largest task of verification engineers in the Wilson Functional Verification Studies and it is increasing. Something needs to happen.

Discussion

We have ranked the verification activities based on their general usefulness. As a project progresses the relative value will change based on the current situation, but from a macro perspective this ranking is good enough to have an overall discussion.

As we see in the Wilson Verification studies, debugging is taking the largest share of verification work, it is the lowest value value activity and it is growing. This is the exact opposite of what we want. We want to have debugging be a smaller part of verification activity and be decreasing in share such that the focus is on higher value activities.

When you are debugging, you are looking for relevant information that can help you isolate a bug. This information can come in many different forms. It could be:

  • logs
  • assertions
  • waveforms
  • test suites
  • test reports
  • recent changes

The more relevant information that is readily available to you, the less time you have to spend finding it and the less time you will have to debug. If relevant information isn’t readily available or if there is information that you have to dig out repeatedly when debugging, then it is often better to add simpler tests or improve the testbench. This will give you more information now and help you in the future. Being stuck in a low value activity does not mean that you have to stay there.

References

  • https://blogs.mentor.com/verificationhorizons/blog/2019/01/29/part-8-the-2018-wilson-research-group-functional-verification-study/

Never miss an article



Modern testbench architecture

The aim of this article is to explain the what and why of modern testbench architecture. With the knowledge in this article, you should be able to understand how most of all modern verification frameworks are structured and how similar they really are. There are many hardware verification frameworks in this world, but the concepts have almost stayed the same for decades. If you squint hard enough, they all look the same.

Interface

The DUT and the Interface

We have a design that we want to test. We call this the “design under test” (DUT). The design has input and output pins. We group related pins on a design under test into interfaces. Interfaces are our primary means of accessing the functionality of our design, but interfaces is not our primary means of understanding. We would like some abstraction that only considers the functionality of our DUT, not the cycle by cycle nature of the interface.

Transaction

The DUT and the transaction

At the core of modern testbench architecture is the transaction. A transaction is a data structure that encapsulates the information that is needed to perform some functionality. Our transactions is our primary means of understanding when designing testbenches. It is also our primary means of abstraction from the cycle by cycle nature of the underlying hardware.

We think in sequences of transactions in order to stimulate our design into interesting scenarios.

The flow of transactions: Drivers and Monitors

Drivers and Monitors connect the transaction to the interface

We need to have a mechanism for connecting transactions and interfaces. We have one mechanism for each direction. A driver converts a transaction to cycle by cycle pin wiggles. A monitor samples cycle by cycle pin wiggles and convert these into transactions for us to understand and reason about. We some times call drivers and monitors bus functional models.

The flow of transactions: Sequencer

A generator or sequencer creates transactions that the driver converts into pin wiggles. The sequencer orchestrates one or more sequences of transactions. The sequencer is often simple. The complexity is often embedded inside sequences. Sequences can contain other sequences, so they can be composed into arbitrarily complex scenarios.

Transaction generation and sequencing is often where most frameworks differ.

Agent

The Agent wraps the driver, monitor and sequencer

For ease of use and understanding, we group the driver, monitor and sequencer into an agent. An agent handles all instantiation and connection of the components inside the agent. This makes reuse much easier

Checkers and Scoreboards

Checkers and Scoreboards verify the design

Checkers and scoreboards are our primary means of verifying our design. A checker observes an interface or the output of a single agent and looks for inconsistencies. A scoreboard connects two or more agents and compares their outputs. Depending on methodology, the scoreboard may be connected directly to monitors instead of their agents.

Checkers are placed in multiple places, depending on the needs of the testbench. Checks can be placed inside a monitor or in a subscriber to the monitor. It is possible that a generic instance of an interface does not guarantee a certain property, but the DUT does, so the check must go into a subscriber to the monitor insider the environment instead of placing the check directly in the monitor. Placement of checks is often based on the taste of the individual verification engineer.

Coverage

Coverage collection gives you a sanity check on whether all interesting use cases were hit

Coverage collectors are placed in multiple places just like checkers. While they often exist together with checkers, they are separated from checkers both conceptually and for ease of reuse. Coverage collection may appear inside monitors just like checkers, but they are often placed inside a subscriber as a generic interface may be able to perform some transaction that the DUT will never perform. Keeping the generic coverage collection inside the monitor would make it impossible to close coverage.

Environments

An environment is a collection of agents for a design under test. The environment also hooks up agents, scoreboards and subscribers. An environment can contain other environments enabling the ability to reuse an environment as a subsystem in a larger verification environment. This mean of combination allows for more complex testbenches to be built out of smaller, simpler testbenches. The goal is to put as much as possible into the environment in order to maximize reuse.

Ideally there is only one environment per DUT in order to maximize reuse.

Harness (top)

The harness is the top level module that instantiates the DUT and connects it to the environment. It is the starting point for the simulator to begin execution. The top level is ideally a relatively small module, but connectivity and corner cases often makes it much larger than anticipated.

Tests

The complete system architecture of a modern testbench

Tests control the environment and the DUT inside the harness to push the DUT into interesting behavior. The goal is to have many tests for each environment. The environment should only have to be compiled once and then have all the tests simulated in parallel.

Conclusion

This overview shows how most of the many verification methodologies are structured and how similar they are. If you see a new verification methodology, then the learning curve should not be too steep if you already understand the general structure.

References

Never miss an article



First silicon success is still a dream

Don’t beat yourself up if don’t get it perfect the first time. For the last 14 years, first silicon success rates have not improved. The expected number of tape-outs is still around 2.

I analyzed the results from the 2012 and 2018 Wilson Research Functional Verification Study to see if anything has changed in the last decade and a half with regards to ASIC respins. Unfortunately, it appears that the introduction of advanced verification methodologies such as UVM and Formal Verification have not affected first silicon success in any meaningful way. From 2004 to 2018 the expected number of tape-outs have hovered around 2 across the industry. The first silicon success rate has hovered around 31% except for 2018 where it sank a little to 26%. Interestingly, about 25% of all projects needed more than 2 tape-outs before succeeding.

Most projects get it right in two attempts.

Take-aways for engineers

Don’t beat yourself up, if you don’t get it right the first time. Few projects do. On the other hand, if you need 3 tape-outs to get it right, then you aren’t doing as well as your peers. A third tape-out can either come from a bad fix or from a bug that hid behind another bug, so do you best to make both the first and the second tape-out a success. Ask yourself whether you have the required verification quality to ensure that no bugs are big enough for other bugs to hide behind.

Take-aways for managers

Plan on having two tape-outs. There are two common ways to run these and one backup way which some times comes in handy.

If you have a relatively small chip or low confidence in a successful tape-out and 5-6 months of slack in your required time-to-market, then a multi-purpose wafer (MPW) shuttle makes the most economic sense as a MPW shuttle is 10% of the full tape-out price.

If you have a large chip that can’t fit in a MPW shuttle or if you have a no slack in your required time-to-market, that is, the lost revenue from delaying a product launch by 3 months would vastly exceed the tape-out cost, then you should go for a full maskset tape-out and follow up with a metal-only respin.

A metal-only respin has about half the turnaround time as a full mask tape-out since the foundry can process the first half of the fabrication steps from the first tape-out in advance of receiving new masks for the metal layers. A two layer metal change costs about 20% of a full tape-out. You should budget at least 3 months of calendar time to complete a metal-only spin with the time spent evenly between your end and your foundry. A metal-only respin plan requires high confidence in product quality, an experienced physical design team and ample spare or gate-array cells in the floor plan.

A multi-layer maskset (MLM) tape-out is the middle ground between MPW and full maskset. MLM costs A40% of the full maskset tape-out price. It is only really useful when you have low confidence in the product quality, but you can’t wait for an MPW shuttle.

References

Never miss an article