Constrained random stimulus generation is wasteful and misguided and when coupled with poorly devised coverage metrics gives a false sense of security. This patent makes the situation worse...
People who know my history and background know I spent most of my time working on simulation and other forms of dynamic verification. I covered just about every aspect of it from hardware/software co-simulation, emulation, mixed-signal simulation etc. But I was probably destined to be a formal person, just grew up in the wrong time period. Why do I say this? Because the verification methodologies we use today and coverage in particular just make my skin crawl. They are stimulus based methods and metrics and actually have nothing directly to do with verification completeness. Formal, on the other hand, uses response based metrics. Many people in the industry admit that going from coverage goals to the establishment of metrics is one of the most difficulty and time consuming aspects of verification, and to me it is no wonder – the dynamic metrics in use are misguided. But I can’t blame people for using them – it is what has been highly developed, marketed and sold by the EDA industry as the solution.
What got me going this week was a patent application that I came across. It was published this week 2013/0086538 (note this is not a patent, just a publication of a patent proposal under review by the patent office). It was created by an individual and this blog is not meant to be an attack on the individual at all – I merely use it to demonstrate my issue.
The application title is “Design Verification System and Method using Constrained Random Test Pattern Selection”. What it tries to do is to improve the process by adding coverage into the generation picture – a worthy goal given the waste that constrained random creates. It says if a value has been chosen for a random variable in the past, then don’t pick it again. I know there is a lot more to it, but that is the essence of it. This concept reminds me about cheating with code coverage. You look at a code coverage report, see a line or block of code that has not been executed and devise a test to execute it. The problem is that the test does nothing apart from filling in the coverage report to make your manager happy. It is fudging the metrics. Now, this patent is suggesting exactly the same thing. In order for this to work, the coverage goals are the inputs. So, we select stimulus based on the past stimulus provided just so that we can say that those coverage holes have been filled. Proper coverage metrics have nothing to do with inputs; they have to do with observed responses based on completed actions in the design. The really tough part is deciding what stimulus values and sequences will makes those functional aspects of the design be activated and the ways in which it can be verified that the right things happened as a result.
The only way I know how to do this is based on tracking backwards through the design, just like the old tester methods used to do. Those methods proved too compute intensive at the gate level and the patterns generated were too long as design sizes grew and so alternative methods were found, but for functional verification it is what needs to happen. Formal methods attempt to do this as does the graph-based approach being developed by Breker Verification Systems (disclosure: I do consulting work for Breker but this article is not paid for or endorsed by Breker). By tracking backwards through a design, at a high enough level of abstraction, it may be possible to ascertain the stimulus needed to make something useful happen and this is what constrained random should be doing, not trying to find unique combinations of random values.
Brian Bailey – keeping you entertained
If you found this article to be of interest, visit EDA Designline
where – in addition to my blogs on all sorts of "stuff" – you will find the latest and greatest design, technology, product, and news articles with regard to all aspects of Electronic Design Automation (EDA).
Also, you can obtain a highlights update delivered directly to your inbox by signing up for the EDA Designline weekly newsletter – just Click Here
to request this newsletter using the Manage Newsletters tab (if you aren't already a member you'll be asked to register, but it's free and painless so don't let that stop you [grin]).