Design Article
Comment
mdodgson
@hriedel there are a few good tools that are around for truely embedded designs. ...
hriedel
And in what does that article help the embedded developers on unit and code ...
The ‘8 commandments’ for choosing a unit testing solution
Gil Zilberfeld, Typemock Technology
8/19/2010 2:57 AM EDT
Teams who perform unit testing on a regular basis are perceived to be more reliable, professional and advanced. But what do you need to consider before choosing a unit testing solution? We have developed the ‘8 commandments’ below as a guide for ensuring you select a unit testing solution that is right for your development.
1. Thou shalt not waste time on the learning curve
When choosing a unit testing solution, you will want one that will require minimal time for implementation. It may be worthwhile to time a new developer within your team with the framework to get an accurate idea of how long it takes to get started. For example – how long will it take them to write the first three tests for some class in your system? Is the API clear and simple? Is there a single point of entry in the API? Is there clear guidance on what to do at each step of the way? How often, if at all, do you need to check the docs and tutorials? How easy is it to look for the next step when you're not sure what to do? Some tools offer guidance within the IDE, while some provide extensive help. Some don't do either.
2. Thou shalt not waste time fixing your tests
How resistant to change do you need your tests to be? Different frameworks provide varying levels of change resistance (when production code changes). Change resistance can be measured according to the number of tests needed to modify for a single piece (method) of changed production code. Does the tool support these production changes without affecting the tests? Do the frameworks support recursive fakes? Is it non-strict by default or does it throw exceptions on unexpected interactions by default? How does it handle method overloads (if your production code changes to call a different overload?). How are arguments verified? Are all arguments ignored? Are actual values used for expectations? All these things affect the fragility of your test. The more fragile it is, the less change resistant it is. If your production code changes a lot you will need to take this into account.
3. Thou shalt design your code the way you need it
Your project might be a greenfield project (fresh new code) or have a lot of legacy code. Not all isolation frameworks were designed, or have the same support for all (or any) legacy code scenarios. If your unit tests need to be written against legacy code (existing code without tests): See if your production code contains static constructors, internal or private classes that might need to be faked. Does the framework support faking them? Does your code instantiate objects directly all over the place, or does it use a dependency injection framework or factory of some kind that would need to be faked? Make sure the framework you choose supports these scenarios, or that you have the time needed to refactor your code for testing such cases.
4. Thou shall make your test code readable
How will your tests look when you use the framework? Test code is still code and needs to be readable. When you jump into it to debug, you need to be able to see at a glance what you’re testing.
Are your tests messy? Does the framework cause your tests to be longer than a page? Can you understand the expected behavior of the code under test if the framework is involved? Can your team members understand without a great deal of explanation what you are using the framework for in each test? Can you still write test code according to accepted industry best practices such as AAA (arrange-act-assert) when using this framework? Not all frameworks support the easier-to-understand AAA tests, for example. It is important that the solution you choose guides you in writing tests, (incorporating the best-practices in test writing), which will make the process of unit testing quicker and easier.
5. Thou shall not need to replace tools for the sake of the tool
How well would the tool integrate with your current coding environment and ecosystem? Can you use the framework from your various test runners (TestDriven.NET, Resharper, MS Test etc.) or in combination with profiling and code coverage tools? Some frameworks incorporate profiling, and it is important to see that they work effectively with other profilers and runners. Can you run your tests with code coverage or with other profiling technologies? How well does the framework integrate with various versions of Visual Studio? Do you need it to support VS 2005? VS 2008? What .NET versions should it support? Some of the frameworks require .NET 3.5 and up.
6. Thou shall not forsake interaction testing
Do you need the ability to verify interactions between objects in your tests? Some frameworks may not have the built in ability to verify interactions, requiring you to provide your own manual flagging mechanisms (hand rolled mocks). For complicated interfaces this can grow cumbersome and result in an un-maintainable piece of code. Can you verify method calls between the code under test and its 3rd party dependencies? For example, that a SharePoint method gets called at the end of a test.
7. Though shall not settle for an incomplete solution
A complete solution is not only one that meets all your requirements for the tasks it needs to carry out, but one that provides you with support from the moment you show an interest in seeing a demo until long after the product has been implemented in your system.
Ask yourself, what kind of technical support do you require? How fast do you need to get a response to a question if you are not sure how to accomplish a specific task with the tool? How many people are on staff at the vendor who makes the framework and at what hours of the day? Is support provided by a company or an independent individual? A company is often more invested in supporting you than an individual – or they stand to lose money. What is the company's main expertise? Are unit testing and agility at its core business? If so, the support team is more likely to well versed be on these subjects. There is nothing more frustrating for a development project manager who is trying to complete a project on time and in budget then being delayed because the team cannot get the support or answers they need from the solutions experts in real time.
8. Though shall consider the solution’s TCO
Often, one of the biggest factors in any development project is the total cost of ownership (TCO) for the tool. The TCO should consider price, time and effort required to work and implement the tool in your project or organization.
Compare the following:
About the author:
Gil Zilberfeld is Typemock's (Tel-Aviv, Israel) Technology Evangelist. With over 15 years of experience in software development, he has worked with a range of aspects of software development, from coding to team management, and implementation of processes. He presents, blogs and talks about unit testing, and encourages developers from beginners to experienced, to implement unit testing as a core practice in their projects
1. Thou shalt not waste time on the learning curve
When choosing a unit testing solution, you will want one that will require minimal time for implementation. It may be worthwhile to time a new developer within your team with the framework to get an accurate idea of how long it takes to get started. For example – how long will it take them to write the first three tests for some class in your system? Is the API clear and simple? Is there a single point of entry in the API? Is there clear guidance on what to do at each step of the way? How often, if at all, do you need to check the docs and tutorials? How easy is it to look for the next step when you're not sure what to do? Some tools offer guidance within the IDE, while some provide extensive help. Some don't do either.
2. Thou shalt not waste time fixing your tests
How resistant to change do you need your tests to be? Different frameworks provide varying levels of change resistance (when production code changes). Change resistance can be measured according to the number of tests needed to modify for a single piece (method) of changed production code. Does the tool support these production changes without affecting the tests? Do the frameworks support recursive fakes? Is it non-strict by default or does it throw exceptions on unexpected interactions by default? How does it handle method overloads (if your production code changes to call a different overload?). How are arguments verified? Are all arguments ignored? Are actual values used for expectations? All these things affect the fragility of your test. The more fragile it is, the less change resistant it is. If your production code changes a lot you will need to take this into account.
3. Thou shalt design your code the way you need it
Your project might be a greenfield project (fresh new code) or have a lot of legacy code. Not all isolation frameworks were designed, or have the same support for all (or any) legacy code scenarios. If your unit tests need to be written against legacy code (existing code without tests): See if your production code contains static constructors, internal or private classes that might need to be faked. Does the framework support faking them? Does your code instantiate objects directly all over the place, or does it use a dependency injection framework or factory of some kind that would need to be faked? Make sure the framework you choose supports these scenarios, or that you have the time needed to refactor your code for testing such cases.
4. Thou shall make your test code readable
How will your tests look when you use the framework? Test code is still code and needs to be readable. When you jump into it to debug, you need to be able to see at a glance what you’re testing.
Are your tests messy? Does the framework cause your tests to be longer than a page? Can you understand the expected behavior of the code under test if the framework is involved? Can your team members understand without a great deal of explanation what you are using the framework for in each test? Can you still write test code according to accepted industry best practices such as AAA (arrange-act-assert) when using this framework? Not all frameworks support the easier-to-understand AAA tests, for example. It is important that the solution you choose guides you in writing tests, (incorporating the best-practices in test writing), which will make the process of unit testing quicker and easier.
5. Thou shall not need to replace tools for the sake of the tool
How well would the tool integrate with your current coding environment and ecosystem? Can you use the framework from your various test runners (TestDriven.NET, Resharper, MS Test etc.) or in combination with profiling and code coverage tools? Some frameworks incorporate profiling, and it is important to see that they work effectively with other profilers and runners. Can you run your tests with code coverage or with other profiling technologies? How well does the framework integrate with various versions of Visual Studio? Do you need it to support VS 2005? VS 2008? What .NET versions should it support? Some of the frameworks require .NET 3.5 and up.
6. Thou shall not forsake interaction testing
Do you need the ability to verify interactions between objects in your tests? Some frameworks may not have the built in ability to verify interactions, requiring you to provide your own manual flagging mechanisms (hand rolled mocks). For complicated interfaces this can grow cumbersome and result in an un-maintainable piece of code. Can you verify method calls between the code under test and its 3rd party dependencies? For example, that a SharePoint method gets called at the end of a test.
7. Though shall not settle for an incomplete solution
A complete solution is not only one that meets all your requirements for the tasks it needs to carry out, but one that provides you with support from the moment you show an interest in seeing a demo until long after the product has been implemented in your system.
Ask yourself, what kind of technical support do you require? How fast do you need to get a response to a question if you are not sure how to accomplish a specific task with the tool? How many people are on staff at the vendor who makes the framework and at what hours of the day? Is support provided by a company or an independent individual? A company is often more invested in supporting you than an individual – or they stand to lose money. What is the company's main expertise? Are unit testing and agility at its core business? If so, the support team is more likely to well versed be on these subjects. There is nothing more frustrating for a development project manager who is trying to complete a project on time and in budget then being delayed because the team cannot get the support or answers they need from the solutions experts in real time.
8. Though shall consider the solution’s TCO
Often, one of the biggest factors in any development project is the total cost of ownership (TCO) for the tool. The TCO should consider price, time and effort required to work and implement the tool in your project or organization.
Compare the following:
Price of toolFollowing these 8 commandments can help ensure that the unit testing solution you choose will increase the quality of the final product, lower development costs and reduce time to market.
Time to get started and reach RTM test (readable, trustworthy, maintainable)
Time to write tests
Time to fix test because of production change (change resistance)
About the author:
Gil Zilberfeld is Typemock's (Tel-Aviv, Israel) Technology Evangelist. With over 15 years of experience in software development, he has worked with a range of aspects of software development, from coding to team management, and implementation of processes. He presents, blogs and talks about unit testing, and encourages developers from beginners to experienced, to implement unit testing as a core practice in their projects
|
|
|
|
|
Navigate to related information
t.alex
8/19/2010 5:14 AM EDT
Gil you have good points! Sometimes it is quite tempting to skip unit testing, due to time consuming process, or even some bugs in the unit test itself. Unit testing is always important but the hard part is how to make it part and parcel of the development process seamlessly.
Sign in to Reply
gil.zilberfeld
8/19/2010 2:10 PM EDT
Alex,
First, I'd like to give credit where credit's due: Roy Osherove, Typemock's CTO, was jointly behind this article.
Let me share a personal story:
Five years ago, as a team leader, I tried and then decided that TDD was something that was not going to work in my team - due to tooling, skills I needed to instill, and the overall effort of changing the organization. Like you said, it just looked too hard.
At Typemock, we have thousands of tests running with each check-in. Each test that fails - saves are butts every single time. It's that experience, that drives my belief in unit testing, and that developers should adopt it as a life saver.
And we've got much better tooling these days than what I had 5 years ago. It is much easier. We just need to drive the message that unit testing is important as I believe.
Sign in to Reply
t.alex
8/19/2010 9:19 PM EDT
So there is a system in place to run all the unit tests whenever there is a check-in. That's a great idea! In my development, most are commercial embedded products, and the challenge is how to automatically run the unit tests on these platforms whenever there is a check-in.
Sign in to Reply
hriedel
8/24/2010 12:22 AM EDT
And in what does that article help the embedded developers on unit and code coverage testing?
That is the reason why lots of embedded developers just don't test, don't use much C++ and keep on conservative tools, languages and style (e.g. the reason for MISRA C, no dynamic memory allocation, etc. pp). The unit testing can be well done on PCs or servers, but what the hell does .NET or Visual Studio have to do with Embedded Systems? I don't see anything useful to the EETimes community, or is that not anymore a forum also for deeply embedded systems development? I don't see much help of the unit test frameworks around to stub/wrap simulate HW register access, updating a system timer,,, that you can only write by hand. Windows, .NET do not run on microcontrollers, and do not resemble the actual conditions on the real target!
Sign in to Reply
mdodgson
9/3/2010 4:13 PM EDT
@hriedel there are a few good tools that are around for truely embedded designs. One such open source framework is Unity which I have used and love. I have used this on PIC up to ARM running linux. another useful part of this design is using CMock, a great mocking framework. These are being combined into a complete package which works straight out the box with a framework called Ceedling (still in beta). Hope this helps convince or at least encourage you to give it a try.
Sign in to Reply