Visual T# (pronounced "T Sharp") is a free unit tests development environment integrated within Visual Studio™, though it can also be used indepedently from it. It is composed of :

• T# : a programming language specific to unit tests. T# is based on the C#™ 2.0 language, which makes it a natural fit for .NET developers. New reserved keywords have been added to simplify the writing of unit tests. It puts a lot of emphasis on unambiguously defining tests' intentions.
• Tools for compiling tests, executing them, testing them and easily navigating amongst them.

Visual T# is a new generation of unit tests tool. Coding tests becomes simpler. And since tests better describe their intention, they also have a better quality!

Summary

1. Benefits
2. The Language
3. Best practices for unit tests
   1. Structure of a test
   2. Verification
4. 4 states for a test
5. What to test?
6. Contexts
   1. Context for every test
   2. Different levels of context
7. Which case to test?
   1. Missing cases
   2. Case Expressions
   3. Criterias
8. Testing exceptions
   1. Test that an exception was thrown
   2. Completely test an exception
9. Verify changes
   1. Verify the constancy of an expression
   2. Verify the constancy of an object
   3. Verify a change
10. Testing Events
    1. Verify that an even wasn't raised
    2. Verify that an event was raised
    3. Verifying an event thoroughly
11. Testing with 'Code Snippets'

Benefits

T# is a programing language for Microsoft .NET, compatible with C# 2.0 (except anything related with unmanaged code), and offers the following benefits compared with other existing tools such as NUnit and Visual Studio Team Test :

• Transparent usage of best practices : clearly identifies the three parts inherent to a test (Preparation, Execution, Verification).
• Fast problems identification : tests that are not well constructed are separated from those that actually failed.
• Easy validation specification : one keyword assert to handle all the cases.
• Effective debugging : tests concerning a working declaration can be executed without knowing where they have been declared.
• Usage of different contexts: reexecute tests for different contexts without rewriting them.
• Indication of missing logical tests: missing logical tests are indicated upon compilation of the test code.
• Simplified writing: each verification can be expressed in one line of code (this also holds for events, exceptions, etc.), use relative verifications.

The Language

Here's an example of a minimal test, written in T# :

testclass
{
  test
  {
    Calculator calc = new Calculator();
 
    runtest double sum = calc.Add( 1, 2 );
 
    assert sum == 3;
  }
}

Best practices for unit tests

T# is completely oriented towards best practices.

Structure of a test

A unit test is always composed of three parts:

• Preparation : creation of instances and dependencies needed by the execution of a test
• Execution : one instruction to use the declaration being tested in a given context
• Verification : verification of the expected results (directly or indirectly).

The most important part being Execution, it's for this part that you are writing the test.

T# clearly identifies the Execution part by beginning the instruction with the runtest keyword. The preparation is therefore whatever is before runtest, and the verification is whatever is after runtest.

Verification

The verification part makes sure that all effects (i.e., function returns, parameters modification, instances modification, static declarations modification, files modification, databases modification, etc.) accounted for during the usage of the declaration were properly made as expected.

T# supplies one keyword for this: assert. The message is automatically generated from the source code. Therefore, if the preceding test fails (if the Add function isn't well written and always returns 0), the test will fail with the following error message : "Expected: sum == 3 but was: 0 == 3". It is therefore easier to see that the sum is 0, though we expected it to be 3. This way of generating the error message makes it closer to the source code (easier to make the link with the code when the error occurs) and decomposes the various implicated values (if multiple variables were implicated, we'll know the value of each of them and not just the one from the final expression), which makes the test easier to debug.

Furthermore, natural conversions of the programming language are used (sum is a double, 3 is an integer). Therefore, there is nothing else to do when comparing both values, in contrast with Visual Studio Team Test in which you have to write : Assert.AreEquals( 3.0, sum );, to do the same verification!

4 states for a test

The test being code, it can also fail. Contrary to other tools, T# knows exactly the instruction which actually tests the declaration (runtest). Therefore, it is well positioned to determine whether the failure occurred prior or after this instruction :

• If the test fails prior to the runtest instruction, it's because we can't properly prepare the context needed to test our intention. The test is not good.
• If the test failes after the runtest instruction, it's probably because the tested code doesn't do what was expected of it. The tested code is not good.

T# has therefore 4 states for a test :

• Passed : the test passed.
• Ignored : the test is temporarily ignored.
• Failed : the test failed, the tested code is not good.
• Invalid : the test failed, the context for the test is not good, it is therefore impossible to verify if the tested code is right or not.

In order to take advantage of this difference and make the test easier to understand, use asserts before the runtest instruction :

testclass
{
  test
  {
    Product prod = new Product( "T#", 123 );
 
    runtest prod.Price = 0;
 
    assert prod.Price == 0;
  }
}

In this example, we want to give T# for free. The test passes. The code which encapsulates the set property of Price is correct. But is it really? If you look closer, even if neither the constructor nor the set property of Price are coded...the test still passes!

A good test for the price change could be :

testclass
{
  test
  {
    Product prod = new Product( "T#", 123 );
 
    assert prod.Price != 0;
 
    runtest prod.Price = 0;
 
    assert prod.Price == 0;
  }
}

Now, this case is excluded. If the constructor doesn't initialize the Price property, the first assert will fail and, given that it's before the runtest, the test is said to be 'Invalid' and not failed! Furthermore, in a business logic perspective, we can clearly see that setting the price to 0 makes it go from a non-null value to a null value and, thus, a null price is acceptable.

What to test ?

T# induces in expressing what is tested, not by specifying appropriate class and method names, but instead by clearly indicating it. That being said, the preceding test should have been written as the following:

testclass for Product
{
  test Price set
  {
    Product prod = new Product( "T#", 123 );
 
    assert prod.Price != 0;
 
    runtest prod.Price = 0;
 
    assert prod.Price == 0;
  }
}

The benefits are the following:

• The tested elements are validated by the compiler.
• We can easier find the tests that exercice a specific declaration.
• It is possible to execute all the tests for a declaration without the need of knowing where they are! Within Visual Studio, place your cursor inside the code of any declaration (class, method, etc.) and use the contextual menu to execute the 'Run T# Tests' command! This is particularly useful when you're refactoring your code, as you don't have to switch from the business code to the test code simply to run the latter.
• It is possible to extract all the tests for a given declaration to view them graphically within the 'Tests Runner'.

Contexts

For every system tests, there are potentially many redundancies amongst the tests. In fact, it is necessary to have many tests for every class declaration and, generally, a class has many declarations. In any case, a instance of the class to test will have to be instantiated.

Every system tests provides a method to invoke prior to every test and another method to invoke after every test. T#, instead, provides only one method.

By doing this, you get the following benefits:

• usage of local variables and not instances systematically.
• usage of these instructions to surround every test: using, try...catch, try...finally, etc.
• repetition of the runtest keyword to run each test multiple times! directly or in a loop. This is particularly useful to execute all the tests in various contexts.

Context for every test

The simplest form of context is the context of each test. It is used by default.

The tests are executed, but not directly. The context, introduced by a method declared with the testcontext keyword, is called for each test. The runtest keyword indicated the location where the test must really be executed.

Therefore, in our example, we would like to create an instance in a single line of code, but we also have to create an instance for each test :

testclass for Product
{
  Product prod;
 
  testcontext
  {
    prod = new Product( "T#", 123 );
    runtest;
  }
 
 
  test Price set // Minimum value
  {
    assert prod.Price != 0;
 
    runtest prod.Price = 0;
 
    assert prod.Price == 0;
  }
 
 
  test Price set // any valid value
  {
    assert prod.Price != 12;
 
    runtest prod.Price = 12;
 
    assert prod.Price == 12;
  }
}

Different levels of context

In T#, the context can be represented in three levels:

1. test : the code of the context is executed for each test. The test itself being run upon the call to runtest in the context. This is the context by default.
2. testclass : the code of the context is executed for the test class. The tests for the test class being run upon the call to runtest in the context.
3. testassembly : the code of the context is executed for the set of test classes in the assembly. The tests being run upon the class to runtest in the context.

In this example, the tests will be executed two times, without the need to writing them two times :

1. for a connection to a SQL Server database.
2. for a connection to an Oracle database.

testclass
{
  IDbConnection connection;
 
  testcontext
  {
    testclass
    {
      connection = new SQLConnection(...);
      runtest;
 
      connection = new OracleConnection(...);
      runtest;
    }
  }
 
  ...
}

Which case to test?

One of the most popular question that we ask ourselves when writing unit tests is : "Which case should I test?". In fact, one declaration should be tested in different cases. One of the preceding examples concerned the price of a product represented by a property. How many tests are required and which are those tests in any given case?

In classical system tests, it is once again the name of the test which determines which case is tested (or a comment, as our previous example). This often results in very long names and not necessarily explicit...and mostly not up to date.

T# introduces a new keyword to express the tested case : when followed by a case to test. Therefore, the example about the tests for the product's price should be :

testclass for Product
{
  Product prod;
 
  testcontext
  {
    prod = new Product( "T#", 123 );
    runtest;
  }
 
 
  test Price set
    when MinIncluded.IsMin
  {
    assert prod.Price != 0;
 
    runtest prod.Price = 0;
 
    assert prod.Price == 0;
  }
 
 
  test Price set
    when MinIncluded.IsAboveMin
  {
    assert prod.Price != 12;
 
    runtest prod.Price = 12;
 
    assert prod.Price == 12;
  }
}

Missing cases

In fact, what follows the when keyword is one case amongst many working together, described by a criteria. In our example, MinIncluded is the criteria which combines two normal cases (IsAboveMin and IsMin) and one error case (BelowMinCase).

That being said, we only need to identify that a product's price can have a minimum value (0) to know that we need to test it in accordance with the MinIncluded criteria. This criteria defining three cases, we'll need to write three tests to properly test this property, once for each defined case by the criteria.

For now, we only have two defined cases (the normal cases). As soon as you compile the example, T# will notify you the missing cases : MinIncludedCriteria.BelowMinCase.

Case Expressions

In reality, after a when, an expression case is used. This expression can be a simple case of a criteria or a combination of criterias.

The following operators exist :

• && (logical AND) : combines all the possibilities between two criterias, knowing that only the normal cases can be combined, the error cases needed to be tested separately.
• || (logical OR) : groups two cases in one same test. At first sight this doesn't look like a good idea, but it could be useful to express the situation of one test executing multiple tests with parameters.
• => (implication) : combines the left case with the different cases of the criteria in the right side. This is useful for when all the combinations aren't logical.

Finally, when a case doesn't make sense, it is possible not to consider it by declaring the !when case. In this scenario, the test must not be implemented (no code), therefore no curly braces, only a semi-colon.

Criterias

T# comes with a suite of pre-defined criterias, but chances are that they might not be sufficient for some scenarios. It is very easy to define your own criterias.

A criteria is like an enum type, but defined with the criteria keyword instead of enum. Error cases are distinguished by adding the [Error] attribute on each one of them.

The convention specifies that :

1. the name of the criteria ends with "Criteria", even if it's not necessary to specify it when used (much like "Attribute" for attributes)
2. a normal case starts with an 'Is' or a 'Has'
3. an error case ends with 'Case'

Therefore, the declaration of MinIncludedCriteria is :

public criteria MinIncludedCriteria
{
  [Error]
  BelowMinCase,
  IsMin,
  IsAboveMin,
}

Testing exceptions

As we have seen with criterias in the preceding paragraph, not only is it necessary to test normal cases, but also error cases.

Generally, an error case is reported by an exception.

We must therefore be able to test for exceptions.

Test that an exception was thrown

T# verifies exceptions like any other verification :

• with only one line
• with the assert keyword followed by thrown and the name of the exception

The benefits of this approach are :

1. guarantee that the exception did in fact occured within the execution of the business code (execution defined by runtest) and not before (preparation) or after (verification)!
2. add other possible assertions to guarantee that nothing unexpected occurred before the exception (like changing the price before triggering the exception!)

Therefore, in the previous example, it is necessary to test the case where the price affected to the product is negative. Since this scenario (having a product with a negative price) doesn't make sense, the Price property should generate an ArgumentOutOfRangeException exception. Test it this way :

testclass for Product
{
  Product prod;
 
  testcontext
  {
    prod = new Product( "T#", 123 );
    runtest;
  }
 
 
  test Price set
    when MinIncluded.BelowMinCase
  {
    runtest prod.Price = -12;
 
    assert thrown ArgumentOutOfRangeException;
    assert prod.Price == 123; // The price didn't change
  }
 
  ...
}

Completely test an exception

In fact, this will only simply verify that the exception was indeed generated in the runtest instruction. It's not bad. But it's not enough! It would be better to also validate the error message for example.

The assert thrown <exception-type> can also be followed by the name of a variable, just like in a catch instruction, and a code block to make as many verifications as needed when the exception is thrown. You can then use that variable to verify anything you want for that particular exception.

testclass for Product
{
  Product prod;
 
  testcontext
  {
    prod = new Product( "T#", 123 );
    runtest;
  }
 
 
  test Price set
    when MinIncluded.BelowMinCase
  {
    runtest prod.Price = -12;
 
    assert thrown ArgumentOutOfRangeException e
    {
      assert e.Message == "The price cannot be negative!";
    }
    assert prod.Price == 123; // The price didn't change.
  }
 
  ...
}

Verify changes

The problem with using a context is that it can be physically located too far from the test we're currently working with, and once it changes can have consequences on the set of tests.

Therefore, in our previous example, if the Product created has now a Price of 100 instead of 123, the assert prod.Price == 123; instruction fails because the Price will be 100!

The ideal would be to make the tests relative : keep the initial value of prod.Price in a local variable, then use it in the verification. The problem is that it makes us write more code than we actually need.

T# offers the possibility to write relative verification in one line of code.

Verify the constancy of an expression

The simplest form of relative verification is that of the constancy of an expression.

T# offers a new form of the assert instruction : assert !changed <expression>

The expression will be evaluated before the runtest instruction, and its value conserved, so that it can be compared with equality by the assert in question.

Therefore, in our example, instead of verifying that the product's price is actually 123, it would be much better to verify that it didn't change at all :

testclass for Product
{
  Product prod;
 
  testcontext
  {
    prod = new Product( "T#", 123 );
    runtest;
  }
 
 
  test Price set
    when MinIncluded.BelowMinCase
  {
    runtest prod.Price = -12;
 
    assert thrown ArgumentOutOfRangeException e
    {
      assert e.Message == "The price cannot be negative!";
    }
    assert !changed prod.Price;
  }
 
  ...
}

Verify the constancy of an object

The most sophisticated form of relative verification is that of the constancy of an object. In fact, there's nothing that tells us that our business code didn't modify the object prior to throwing the exception!

In the assert !changed <expression> instruction, the expression can reference an object and end with :

1. .* : indicates the T# compiler to verify each public property of the object under test. Therefore that the object didn't change in appearance.
2. .-* : indicates the T# compiler to verify each variable (regardless of their level of encapsulation) of the object under test. Thus that the object didn't really change.

Note : the .- operator is similar to the . operator, except that it accesses any declaration, private or not.

Therefore, in our example, instead of just verifying that the price didn't change, it'll be preferable to verify that the prod object itself didn't change :

testclass for Product
{
  Product prod;
 
  testcontext
  {
    prod = new Product( "T#", 123 );
    runtest;
  }
 
 
  test Price set
    when MinIncluded.BelowMinCase
  {
    runtest prod.Price = -12;
 
    assert thrown ArgumentOutOfRangeException e
    {
      assert e.Message == "The price cannot be negative!";
    }
    assert !changed prod.-*; // the product didn't change!
  }
 
  ...
}

Verify a change

Following the same principle, verify that a change has been made by a relative assertion.

A relative assertion for a change is done with the assert changed <assignment> instruction.

The assignment presents itself under three forms:

1. element = expression
2. element op= expression : therefore, element += 1 is equivalent to element = element + 1
3. element++ or element-- : therefore, element++ is equivalent to element += 1 thus to element = element + 1

The right part is evaluated before the runtest instruction, and kept, so that it can be compared for equality to the left part on the corresponding assert. Therefore, assert changed element++ doesn't increment element, but verifies that the value of element added by 1 before the runtest instruction is equal to the value of the element after the runtest instruction. Or, simply put, that the runtest instruction actually did increment the value of the element by one. Therefore, it's an expression equivalent to an assignment as said, but only read-only access are taken into account. It is thus possible to use them with read-only properties too.

If we continue our example with the Product class, we could add an Inventory class (a collection of Product) which would have an Add method and a Count property.

The test for this method would be :

testclass for Inventory
{
  Inventory inventory;
 
  testcontext
  {
    inventory = new Inventory();
    runtest;
  }
 
 
  test Add( Product p )
  {
    Product prod = new Product( "T#", 123 );
 
    runtest inventory.Add( prod );
 
    assert changed inventory.Count++; // one product has been added
    assert inventory[ inventory.Count - 1 ] == prod; // the product has been added at the end
  }
 
  ...
}

Testing Events

Exceptions put aside, events are neither easy nor simple to test correctly. Existing testing tools don't provide the mean to test events properly in an easy and simple manner.

T# offers once again a new assert instruction with the raised keyword.

For example, a class that implements INotifyPropertyChanged must trigger the PropertyChanged event if the value of a property changes. But it shouldn't trigger the event if the assigned value is the same as the current one!

Note : This case being a classic one, T# already provides the NotifyPropertyChangedCriteria criteria with three cases :

1. HasNoSubscriber : normal test, case represented in the previous examples.
2. HasSubscribersSetSameValue : case represented in the next paragraph.
3. HasSubscribersSetOtherValue : case represented in the next paragraphs.

Verify that an event wasn't raised

The simplest form is the verification that an event wasn't raised by the business code.

In T#, the verification of a non-triggered event is done as always in one line of code : assert !raised <event>;

The T# compiler generates an instance variable and a method compatible with the signature of the event. In the test, the variable is initialized to false, the method is registered (+=) to the event before the runtest instruction and unregistered (-=) after the runtest instruction. The generated method will reinitialize the variable to true. The runtest !raised instruction will verify that variable is always set to false.

Supposing that our Product class supports the INotifyPropertyChanged interface, we should include the following test : :

testclass for Product
{
  Product prod;
 
  testcontext
  {
    prod = new Product( "T#", 123 );
    runtest;
  }
 
 
  test Price set
    when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasSubscribersSetSameValue
  {
    runtest prod.Price = prod.Price;
 
    assert !changed prod.-*;
    assert !raised prod.PropertyChanged;
  }
 
  ...
}

Verify that an event was raised

The simplest form of verification of a raised event only verifies that the event was raised.

As always, T# verifies this in only one line of code : assert raised <event>;

The T# compiler generates exactly the same things as assert !changed, except that it verifies that the variable is set to true.

Therefore, in our example, we should have :

testclass for Product
{
  Product prod;
 
  testcontext
  {
    prod = new Product( "T#", 123 );
    runtest;
  }
 
 
  test Price set
    when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasSubscribersSetOtherValue
  {
    assert prod.Price != 12;
 
    runtest prod.Price = 12;
 
    assert prod.Price == 12;
    assert raised prod.PropertyChanged;
  }
 
  ...
}

Verifying an event thoroughly

The inconvenient to continue like the previous chapter, is that it only proves that the event was indeed raised, but not :

1. that the parameters associated with the event have been correctly passed. According to our example, is the sender really the modified product? Does the second parameter correctly references the expected property?
2. the correct state of the object at the moment of the event. According to our example, is the product's price correctly set to its new value once the event is raised?

A more sophisticated form for testing events exist : assert raised <event>( <parameters> ) { <verifications> } Where :

• <parameters> represents the parameters corresponding to the signature of the event.
• <verifications> represents the different assertions to verify in the method reacting to the event.

Therefore, the same tests as the preceding chapter, but complete would be :

testclass for Product
{
  Product prod;
 
  testcontext
  {
    prod = new Product( "T#", 123 );
    runtest;
  }
 
 
  test Price set
    when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasSubscribersSetOtherValue
  {
    assert prod.Price != 12;
 
    runtest prod.Price = 12;
 
    assert prod.Price == 12;
    assert raised prod.PropertyChanged( object sender, PropertyChangedEventArgs e )
    {
      assert sender == prod;
      assert e.PropertyName == "Price";
      assert prod.Price == 12;
    }
  }
 
  ...
}

Testing with 'Code Snippets'

Visual Studio offers the possibility to use 'Code Snippets' for most of Microsoft's languages. We have also added our own 'Code Snippets' for Visual T# (25 in version 1.0).

Using 'Code Snippets', all you have to do is :

1. identify which criteria to apply.
2. use the corresponding 'Code Snippet'.
3. replace any parameter of the 'Code Snippet' with your own values.

In our example, we want to test a product's price.

So, we have already established that we'll be using the following criterias:

1. MinIncludedCriteria : because the Price can be null (minimum value), but not negative.
2. NotifyPropertyChangedCriteria : because the Product class implements the INotifyPropertyChanged interface.

Now aren't we lucky? There already exists a 'Code Snippet' named NotifyMinIncluded which will generate the code in a fly.

You only need to specify the following information :

1. the name of the property to test (Price).
2. the name of the instance under test to use, which is the one created in the context (prod).
3. the acceptable minimum value (0).
4. the value above the minimum to use (12).
5. the value under the minimum to use (-12).

Here's the generated code :

    test Price set
      when MinIncluded.IsMin && NotifyPropertyChanged.HasNoSubscriber
    {
      assert prod.Price != 0;
 
      runtest prod.Price = 0;
 
      assert prod.Price == 0;
    }
 
 
    test Price set
      when MinIncluded.IsMin && NotifyPropertyChanged.HasSubscribersSetSameValue
    {
      assert prod.Price == 0;
 
      runtest prod.Price = 0;
 
      assert !changed prod.Price;
      assert !raised prod.PropertyChanged;
    }
 
 
    test Price set
      when MinIncluded.IsMin && NotifyPropertyChanged.HasSubscribersSetOtherValue
    {
      assert prod.Price != 0;
 
      runtest prod.Price = 0;
 
      assert prod.Price == 0;
      assert raised prod.PropertyChanged( object sender, PropertyChangedEventArgs args )
      {
        assert sender == prod;
        assert args.PropertyName == "Price";
        assert prod.Price == 0;
      }
    }
 
 
    test Price set
      when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasNoSubscriber
    {
      assert prod.Price != 12;
 
      runtest prod.Price = 12;
 
      assert prod.Price == 12;
    }
 
 
    test Price set
      when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasSubscribersSetSameValue
    {
      assert prod.Price == 12;
 
      runtest prod.Price = 12;
 
      assert !changed prod.Price;
      assert !raised prod.PropertyChanged
    }
 
 
    test Price set
      when MinIncluded.IsAboveMin && NotifyPropertyChanged.HasSubscribersSetOtherValue
    {
      assert prod.Price != 12;
 
      runtest prod.Price = 12;
 
      assert prod.Price == 12;
      assert raised prod.PropertyChanged( object sender, PropertyChangedEventArgs args )
      {
        assert sender == prod;
        assert args.PropertyName == "Price";
        assert prod.Price == 12;
      }
    }
 
 
    test Price set
      when MinIncluded.BelowMinCase
    {
      runtest prod.Price = -12;
 
      assert thrown ArgumentOutOfRangeException;
      assert !changed prod.-*;
    }