Choices, choices, choices. The advantage of having a thriving open-source community in the .Net world is that you have choices…the disadvantage is that you have to actually make them. Selecting a unit-testing framework is an important step, and can have far-reaching effects across your organization and over the lifetime of your products. Complicating things further, you must also decide whether you want to trust an open-source solution, or go with what Microsoft offers out-of-the-box. Given that Microsoft only provides one solution for unit testing, MSTest is our candidate there. In the open-source world, NUnit is the incumbent candidate.
MSTest’s greatest strength is in who created it. Microsoft’s offering of a complete end-to-end solution for product development, testing, deployment, and source control is a boon to developers. Writing applications in .Net means paying the Microsoft tax up-front, so why not keep it all integrated in the same product suite and offer developers a superior experience? Getting up and running in MSTest is as simple as Add -> New Test Project. No other solution provides that level of simplicity, and no other solution can guarantee that they can support future versions of the .Net runtime and Visual Studio.
I don’t feel that one killer feature, or one strength, should be enough to sway anyone on such an important choice. This is especially so when some would argue that other choices like NUnit are the defacto solution. So what are MSTest’s strengths? Why should I pick it over NUnit? Let’s jump in and find out!
Drew: MSTest properly instantiates a new instance of the test class for each test method being executed. This provides a number of advantages. First, it allows for easy state management in your tests, your setup and teardown code will be run each time a method is called and your instance variables are automatically reset. There’s no need to reset them manually as it is in NUnit. Indeed, many of MSTest’s other strengths rely on this very principle, as we shall see.
MSTest is fully multithreaded. Each test runs on its own thread, and is completely isolated from other test methods being executed. If you need to rely on state provided by another test, as we’ll get into next, it’s no problem as you can also specify the order the tests are executed in. Again, this all goes back to MSTest doing the correct thing and instantiating a new test class for each test method that is executed.
At this point, some of MSTest’s detractors will point out that instantiating a new instance of the test class makes it difficult to work with test data, or rely on the state of some external object after the execution of a previous test method. I’m glad they did, because it will allow me to explain the benefits of the TestContext class.
The TestContext class is an abstract class which is part of MSTest. The TestContext class can store various bits of information about a test such as performance metrics, test outcome, test name, arbitrary data contained within a Dictionary object, as well as advanced information such as a DbConnection and DataRow. By passing along this information via a known interface (the TextContext abstract class), we get the best of both worlds. Isolated by default, and shared-state when we want.
Performance testing can be done by taking advantage of the timer properties of the TestContext class. This would even allow you to fail tests if timers exceeds a certain threshold.
By utilizing the DbConnection and DataRow properties of the TestContext class, we can build data-driven tests in MSTest. By using the correct MSTest attributes on our test method, we can specify that a test method run over each row from a table, view, or stored procedure. This allows us to not only test the logic in our database layer, but also that stored procedures are returning the expected output, but not at the same time of course. It’s important to remember to test as little as possible in a single test method. Regardless, the opportunity to test database logic along with your application logic exists. If we were to set up some test data in our test class setup, we could work with that test data in our test methods, and clean it all up in our teardown method when we’re done. But what if you want to share that test data between multiple test methods and not tear it down between each method? MSTest has an answer for that! In MSTest you can create setup and teardown methods for an entire assembly, meaning you can group multiple tests and test classes together into one assembly and work with the same data.
MSTest also integrates with Team System, which means you can get all sorts of fancy reports on your unit tests for free. What’s better, you can design custom reports based on performance metrics, certain test groups, whatever your heart desires. If you haven’t seen the kinds of reports that Team System can churn out, go take a look. They are impressive.
There are a few advanced tools that go hand-in-hand with MSTest, such as Pex, but it’s difficult to compare with other options once you start including all the extra tools built around the test framework.
Vijay:
NUnit is a unit-testing framework for all .Net languages. NUnit is written in C#. The best thing about NUnit is that it is open-source. This means that all of NUnit source is available for download. The source can be extended or customized according to the needs of your organization.
NUnit is free. However, the free edition of NUnit does not integrate well with Visual Studio. TestDriven.Net ensures that NUnit is well integrated with Visual Studio. A single license of TestDriven.Net costs about $160. In addition to supporting NUnit, TestDriven.Net supports other unit-testing frameworks like MbUnit, NCover, Reflector, TypeMock.
NUnit appeals more to the developers than the testers. Here is why. NUnit 2.4 introduces a constraint-based assert framework. Using the constraint-based assert framework, the code becomes very readable. For example, the syntax for asserting if an object is not null is as simple as: Assert.That(obj, Is.Not.Null).
There are many more examples of readable asserts. The following table shows this:
| Code | Explanation |
| Assert.That(collection, Is.Unique) | Asserts that the collection is composed of unique elements. |
| AssertThat(number, Is.AtMost(100)) | Asserts that the number is lesser than or equal to 100. |
| Assert.That(number, Is.TypeOf(typeof(int)) | Asserts that number is of integer type. |
| Assert.That(collection, Is.All.GreaterThan(0)) | Asserts that all items in the collection is greater than 0. |
| Assert.That(collection, Has.Some.GreaterThan(90)) | Asserts that the collection has some numbers greater than 90. |
| Assert.That(fruits, Has.Member(“Apple”)) | Asserts that the fruits collection has the fruit – “Apple” |
Another advantage of using NUnit is that it supports a lot of attributes. There are a lot of attributes that support providing input values to a test. As you know, a test can contain multiple input parameters. The TestCase attribute can be used to supply values to each input parameter. The TestCase attribute also has a named property called Result which is used to verify the value returned after executing the test. Consider a test method with two input parameters a, b as follows:
[Test, Combinatorial]
public void Add ([Values(1,2)] int a, [Values(3,4)] int b)
{
}
The above test is run four times with the following input parameters (a,b): (1,3), (1,4), (2,3), (2,4). NUnit also has a Sequential attribute. If the previous test is decorated with a Sequential attribute instead of a combinatorial attribute, the test method will be run two times with the following input parameters: (1,3), (2,4).
Input parameters can be decorated by a Random attribute. Using the Random attribute supplies a random value to the input parameter within a specific range. Alternatively, the input parameter can be decorated with a Range attribute. A Range attribute [Range(4,10,2)] provides the same input values as a Values attribute specified by [Values(4,6,8,10)]. The Range attribute can be used to run the test with a lot of input parameters to ensure that the tests run well under many circumstances. As you can see, NUnit is a good unit testing framework for systems that involve a lot of mathematical operations.
Sometimes, not all test cases need to be run. The Ignore attribute ensures that the tests decorated with the attribute will not be run. NUnit also has an Explicit attribute. Tests decorated with an Explicit attribute will not run unless specifically invoked from the GUI. The Category attribute categorizes tests. This allows tests specific to a category to be run. There is also another way that a tester can execute a set of tests. This is done with the help of the Suite attribute. The Suite attribute is more helpful for running automated tests using the command-line interface. This is especially useful if you want to run a set of tests after doing the build.
A test fixture can be decorated with a Culture attribute. Providing a value for the Culture attribute ensures that the set of tests within a test fixture is executed only for the specified culture. If the culture is set to fr-Fr, then the tests within the test fixture is run only when the current culture is french. Sometimes, it is required to run tests under a specific culture. To change the culture of the current thread in which the tests are running, NUnit provides two attributes – SetCulture and SetUICulture.
As you can see, NUnit is very appealing to developers who write the automated tests. NUnit provides an easy-to-code constraint-based assert framework. It also has a wide variety of attributes, especially attributes that set the input values to the tests. These developer-friendly features make NUnit a good alternative to MSTest.