logo1

logo

Noninvasive Unit Testing in ASP.NET MVC4 – A Microsoft Fakes Deep Dive

59 Comments
Posted in Unit Testing

A lot of today’s unit testing technologies require significant invasive code changes in order to unit test appropriately. I’ve always been of the mindset that testing your code should be as noninvasive as possible to the system under test, regardless of how that system is designed.

The ability to test a system shouldn’t be dependent on whether or not that system was designed to be compatible with a certain set of testing tools. The design of systems should instead be driven by the needs of the problem domain, while complexity in applied patterns and concept count should only be escalated as it becomes necessary to do so.

In combining the KISS principle and YAGNI with agile architecture you get an architectural design, that at any given point in time, is the simplest to use, easiest to work with and the most maintainable as is allowable or possible in the problem domain in question.

The KISS principle states that most systems work best if they are kept simple rather than made complex, therefore simplicity should be a key goal in design and unnecessary complexity should be avoided. – Wikipedia

"You ain't gonna need it" or “You aren′t gonna need it” (acronym: YAGNI) is the principle in extreme programming that programmers should not add functionality until it is necessary. – Wikipedia

Up until recently the ability to test without escalation of certain architectural patterns was not possible. Even if you wanted to practice a noninvasive testing style, the means to do so as well as community support weren’t generally available.

Conceptually, noninvasive testing tools and methodologies represent a natural progression/evolution in unit testing practices and principles. This is most visible in the evolution of major commercial testing tools, such as TypeMock’s Isolator and Telerik’s JustMock, that now have features to test/mock everything, not just interfaces and base classes. Now, with the introduction of Microsoft Fakes in Visual Studio 11 (more specifically the ability to detour via shimming), we are given all the tools necessary to accomplish noninvasive unit testing built right in to our development environment.

See my earlier blog posts on Microsoft Fakes for more background information: Using Stubs and Shims to Test with Microsoft Fakes in Visual Studio 11 and Comparing Microsoft Moles in VS2010 to Microsoft Fakes in VS11

Additionally Fakes allows us to take the “mockist” approach of behavior verification described by Martin Fowler in his article Mocks Aren’t Stubs.

But as often as not I see mock objects described poorly. In particular I see them often confused with stubs - a common helper to testing environments. I understand this confusion - I saw them as similar for a while too, but conversations with the mock developers have steadily allowed a little mock understanding to penetrate my tortoiseshell cranium.

This difference is actually two separate differences. On the one hand there is a difference in how test results are verified: a distinction between state verification and behavior verification. On the other hand is a whole different philosophy to the way testing and design play together, which I term here as the classical and mockist styles of Test Driven Development.

Later on in his article Martin provides a more concrete example as he discusses the differences.

The key difference here is how we verify that the order did the right thing in its interaction with the warehouse. With state verification we do this by asserts against the warehouse's state. Mocks use behavior verification, where we instead check to see if the order made the correct calls on the warehouse. We do this check by telling the mock what to expect during setup and asking the mock to verify itself during verification. Only the order is checked using asserts, and if the the method doesn't change the state of the order there's no asserts at all.

With that said I’ll be using Microsoft Fakes to apply noninvasive and mockist testing techniques to test the AccountController of a default MVC 4 project created using the "Internet Application” template, making absolutely no changes at all to the project. This example will use a mixture of both shimming and stubbing from Microsoft Fakes in order to get the job done.

Getting Started

Let’s take a quick look at the class definition for AccountController.

MethodsToTest

Looking through the implementations, a few of the methods are trivial enough for us to skip as part of this example.

public ActionResult ChangePassword() { return View(); }

public ActionResult ChangePasswordSuccess() { return View(); }

[AllowAnonymous]
public ActionResult Login() { return ContextDependentView(); }

[AllowAnonymous]
public ActionResult Register() { return ContextDependentView(); }

Additionally, we’re going to forego testing ContextDependentView, GetErrorsFromModelState & ErrorCodeToString in favor of the more complex methods. That’s not to say you wouldn’t test these methods for appropriate coverage, just that we’re going to exclude them to keep this post somewhat reasonable in length.

Before we get started though, we need to do some basic project setup. 

  • Create a new ASP.NET MVC 4 Application project using the Internet Application template
  • Add a Unit Test Project, I renamed the default cs to AccountsControllerTests
  • Add references to the following items in the Unit Test project
    • the MVC 4 project
    • System.Web
    • System.Web.MVC
  • Additionally I’ll be using NUnit for assertions, so pull down NUnit from Nuget and add the following using statement to the top of the AccountsControllerTests file:
using Assert = NUnit.Framework.Assert;

After all that your solution should look something like this:

InitialSolutionConfig

LogOff Method

We’ll start off with the LogOff method (as seen below), since this is one of the simpler methods we’re going to be looking at.

public ActionResult LogOff()
{
    FormsAuthentication.SignOut();

    return RedirectToAction("Index", "Home");
}

First off, let’s review our goals here. Since our intent with mocking is behavior verification, we want to test both that the correct RedirectToAction was returned and that FormsAuthentication.SignOut() was called. Testing that the correct RedirectToAction was returned seems easy enough, so we’ll start with that.

using System;
using System.Web.Mvc;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using NoninvasiveMVC4Testing.Controllers;
using Assert = NUnit.Framework.Assert;

namespace NoninvasiveMVC4Testing.Tests
{
    [TestClass]
    public class AccountsControllerTests
    {
        [TestMethod]
        public void TestLogOff()
        {
            var accountController = new AccountController();
            var redirectToRouteResult = accountController.LogOff() as RedirectToRouteResult;

            Assert.NotNull(redirectToRouteResult);
            Assert.AreEqual("Index", redirectToRouteResult.RouteValues["Action"]);
            Assert.AreEqual("Home", redirectToRouteResult.RouteValues["controller"]);
        }
    }
}

We get the following results when running our new unit test:

FailedInitialUnitTest

Looking at the stack trace we can see that a NullReferenceException was thrown from FormsAuthentication.SignOut(). This makes sense as technically we’re not in the context of an actual web request and FormsAuthentication depends on a valid HttpContext to be available. This type of problem is common when testing web applications outside of the context of an actual request to a web server.

The traditional guidance on how to test something like this is as follows (see this StackOverflow post for more information):

  • Create a wrapping class around FormsAuthentication with a public method that runs the necessary method
  • Create an interface for this behavior
  • Use dependency injection in our controller to replace the direct call to FormsAuthentication with that of our wrapping class.

Using this formula, our controller code (not the test code mind you) would have to be changed as follows:

public interface IAuthenticationProvider
{
    void SignOut();
}

public class FormsAuthWrapper : IAuthenticationProvider
{
    public void SignOut()
    {
        FormsAuthentication.SignOut();
    }
}

public class AccountController : Controller
{
    private readonly IAuthenticationProvider _authenticationProvider;

    public AccountController(IAuthenticationProvider authenticationProvider)
    {
        _authenticationProvider = authenticationProvider;
    }

    public ActionResult LogOff()
    {
        _authenticationProvider.SignOut();
        return RedirectToAction("Index", "Home");
    }
}

As you can see this pattern is invasive to the system under test and with the purpose of this post being to apply noninvasive testing techniques, we’re going to consider a different way of testing using Microsoft Fakes. Surprisingly enough, it makes short work of these types of scenarios.

The Noninvasive Approach

Let’s start off by putting in what’s minimally necessary to get our test to pass as is. Right click on the System.Web reference in the test project and select Add Fakes Assembly. Once a Fakes assembly is added for System.Web we can use shims in Microsoft Fakes to detour the call to FormsAuthentication.SignOut() to an implementation of our choosing, hopefully one that won’t throw a NullReferenceException.

using System;
using System.Web.Mvc;
using System.Web.Security.Fakes;
using Microsoft.QualityTools.Testing.Fakes;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using NoninvasiveMVC4Testing.Controllers;
using Assert = NUnit.Framework.Assert;

namespace NoninvasiveMVC4Testing.Tests
{
    [TestClass]
    public class AccountsControllerTests
    {
        [TestMethod]
        public void TestLogOff()
        {
            var accountController = new AccountController();
            RedirectToRouteResult redirectToRouteResult;

            //Scope the detours we're creating
            using (ShimsContext.Create())
            {
                //Detours FormsAuthentication.SignOut() to an empty implementation
                ShimFormsAuthentication.SignOut = () => { };
                redirectToRouteResult = accountController.LogOff() as RedirectToRouteResult;
            }

            Assert.NotNull(redirectToRouteResult);
            Assert.AreEqual("Index", redirectToRouteResult.RouteValues["Action"]);
            Assert.AreEqual("Home", redirectToRouteResult.RouteValues["controller"]);
        }
    }
}

That’s simple enough and it does indeed pass.

TestLogOffInitialSuccess

We still have to test that FormsAuthentication.SignOut() was actually called. All we have to do is flip a bit inside of the detoured SignOut method (see lines 28 and 31) and assert it. Here’s the final method.

[TestMethod]
public void TestLogOff()
{
    var accountController = new AccountController();
    var formsAuthenticationSignOutCalled = false;
    RedirectToRouteResult redirectToRouteResult;

    //Scope the detours we're creating
    using (ShimsContext.Create())
    {
        //Detours FormsAuthentication.SignOut() to an empty implementation
        ShimFormsAuthentication.SignOut = () =>
        {
            //Set a boolean to identify that we actually got here
            formsAuthenticationSignOutCalled = true;
        };
        redirectToRouteResult = accountController.LogOff() as RedirectToRouteResult;
        Assert.AreEqual(true, formsAuthenticationSignOutCalled);
    }

    Assert.NotNull(redirectToRouteResult);
    Assert.AreEqual("Index", redirectToRouteResult.RouteValues["Action"]);
    Assert.AreEqual("Home", redirectToRouteResult.RouteValues["controller"]);
}

Testing that FormsAuthentication was called seems pretty trivial here, however we’ll build on this approach as we test more and more complicated methods.

JsonLogin

Moving on, JsonLogin is probably the next simplest method to test in order for us to ease our way into noninvasive testing with Fakes.

[AllowAnonymous]
[HttpPost]
public JsonResult JsonLogin(LoginModel model, string returnUrl)
{
    if (ModelState.IsValid)
    {
        if (Membership.ValidateUser(model.UserName, model.Password))
        {
            FormsAuthentication.SetAuthCookie(model.UserName, model.RememberMe);
            return Json(new { success = true, redirect = returnUrl });
        }
        else
            ModelState.AddModelError("", "The user name or password provided is incorrect.");
    }

    // If we got this far, something failed
    return Json(new { errors = GetErrorsFromModelState() });
}

Right off the bat, it’s pretty clear that MemberShip.ValidateUser and FormsAuthentication.SetAuthCookie will need to be detoured based on our prior experience with the LogOff method. We’ll additionally test that the correct parameters were passed into each.

[TestMethod]
public void TestJsonLogin()
{
    string testUserName = "TestUserName";
    string testPassword = "TestPassword";
    bool testRememberMe = false;
    string testReturnUrl = "TestReturnUrl";

    var loginModel = new LoginModel
    {
        UserName = testUserName,
        Password = testPassword,
        RememberMe = testRememberMe
    };

    var accountController = new AccountController();
    JsonResult jsonResult;
    //Scope the detours we're creating
    using (ShimsContext.Create())
    {
        //Sets up a detour for Membership.ValidateUser to our mocked implementation
        ShimMembership.ValidateUserStringString = (userName, password) =>
        {
            Assert.AreEqual(testUserName, userName);
            Assert.AreEqual(testPassword, password);
            return true;
        };

        //Sets up a detour for FormsAuthentication.SetAuthCookie to our mocked implementation
        ShimFormsAuthentication.SetAuthCookieStringBoolean = (userName, rememberMe) =>
        {
            Assert.AreEqual(testUserName, userName);
            Assert.AreEqual(testRememberMe, rememberMe);
        };

        jsonResult = accountController.JsonLogin(loginModel, testReturnUrl);
    }
}

Now on to the tricky part, testing JsonResult. JsonResult.Data is of type Object, but is filled with an anonymous type.

JsonResultProperties

return Json(new { success = true, redirect = returnUrl });

This makes it slightly more difficult to get at the properties we want to test.

Possible solutions

First off, we might try to cast JsonResult.Data out of Object into some type we could use to access the fields. This requires some runtime trickery and ends up being a bit of mess. See this StackOverflow post for more info.

private static T CastTo<T>(this Object value, T targetType) 
{ 
    // targetType above is just for compiler magic 
    // to infer the type to cast x to 
    return (T)x; 
}

Unfortunately, this only works if you’re working within the same assembly that defined the original anonymous type.

Next up, we could cleverly stuff a dynamic type with the value from JsonResult.Data and access the properties that way.

dynamic data = jsonResult.Data;
Assert.AreEqual(true, data.success);
Assert.AreEqual(testReturnUrl, data.redirect);

This fails as well, since anonymous types are declared as internal as described by the blog post: Anonymous Types are Internal, C# 4.0 Dynamic Beware!.

JsonLoginDynamicFail

Even though the dynamic data variable has the success property, we don’t have access to it. We could use the assembly attribute InternalsVisibleTo in order to give our testing project access to internal types.

[assembly: InternalsVisibleTo("NoninvasiveMVC4Testing.Tests")]

I don’t consider this to be a bad technique, however since we’re trying to be completely noninvasive, I’m going to opt for a slightly different approach.

We’ll use PrivateObject (MSDN Link) to get at the properties. PrivateObject’s MSDN description:

Allows test code to call methods and properties on the code under test that would be inaccessible because they are not public.

PrivateObject ultimately just uses reflection in order to expose the values we need to test.  The real value is in the fact that it abstracts the reflection code away from us. Here’s the code updated with PrivateObject:

var success = (bool)(new PrivateObject(jsonResult.Data, "success")).Target;
var redirect = (string)(new PrivateObject(jsonResult.Data, "redirect")).Target;

Assert.AreEqual(true, success);
Assert.AreEqual(testReturnUrl, redirect);

And with that we now have successful tests

JsonLoginTestsSuccess

Just for completeness, I’ve put together a test to validate the behavior of an invalid login.

[TestMethod]
public void TestInvalidJsonLogin()
{
    string testUserName = "TestUserName";
    string testPassword = "TestPassword";
    bool testRememberMe = false;
    string testReturnUrl = "TestReturnUrl";

    var loginModel = new LoginModel
    {
        UserName = testUserName,
        Password = testPassword,
        RememberMe = testRememberMe
    };

    var accountController = new AccountController();
    JsonResult jsonResult;
    //Scope the detours we're creating
    using (ShimsContext.Create())
    {
        //Sets up a detour for Membership.ValidateUser to our mocked implementation
        ShimMembership.ValidateUserStringString = (userName, password) => false;
        jsonResult = accountController.JsonLogin(loginModel, testReturnUrl);
    }

    var errors = (IEnumerable<string>)(new PrivateObject(jsonResult.Data, "errors")).Target;
    Assert.AreEqual("The user name or password provided is incorrect.", errors.First());
}

And there we go, easy as pie.

LogOff_ValidLogin_InvalidLogin_Tests

For the remainder of the post, I’m just going to focus on the “happy” path for brevity. Testing the other paths is relatively straight forward given what we’ve already done.

Login Method

Stepping up in complexity we move on to the Login method.

[AllowAnonymous]
[HttpPost]
public ActionResult Login(LoginModel model, string returnUrl)
{
    if (ModelState.IsValid)
    {
        if (Membership.ValidateUser(model.UserName, model.Password))
        {
            FormsAuthentication.SetAuthCookie(model.UserName, model.RememberMe);
            if (Url.IsLocalUrl(returnUrl))
                return Redirect(returnUrl);
            else
                return RedirectToAction("Index", "Home");
        }
        else
            ModelState.AddModelError("", "The user name or password provided is incorrect.");
    }

    // If we got this far, something failed, redisplay form
    return View(model);
}

MemberShip.ValidateUser and FormsAuthentication.SetAuthCookie are easy enough to test via Shimming. Under normal circumstances Url.IsLocalUrl would be simple to Shim as well. Unfortunately I ran into an issue when faking the System.Web.Mvc assembly containing it. Once you try to instantiate a controller (as part of your test project) after adding a Fakes assembly you get a System.Security.VerificationException: Operation could destabilize the runtime. See my Microsoft Connect submission for more info.

Fortunately enough, there’s a way to mock its implementation using the stubs portion of Microsoft Fakes as opposed to shims. This brings up an interesting dilemma, if a compatible stubbing technique is available should you use that instead of shimming?

I would say the answer is generally “yes” provided that these criteria are met:

  • It doesn’t significantly decrease the readability of the test
  • It doesn’t require excessive measures (such as reflection dumpster diving) to figure out how to do it

Stubbing Around System.Web.Mvc

The first problem we need to solve is that the Url property (of type UrlHelper) is null on our instance of AccountController. The ctor on UrlHelper requires a RequestContext. The ctor on RequestContext requires an HttpContextBase. Since HttpContextBase is an abstract class we can stub it easily and make our way back up the dependency hierarchy.

Decompiling UrlHelper with ILSpy shows us that we’ll need to stub one more item in order to avoid the dreaded NullReferenceException.

public bool IsLocalUrl(string url)
{
    return this.RequestContext.HttpContext.Request.IsUrlLocalToHost(url);
}

We need to make sure that the Request property on HttpContextBase returns a value. As luck would have it the Request property is of type HttpRequestBase and we can easily stub it as well.

[TestMethod]
public void TestLogin()
{
    string testUserName = "TestUserName";
    string testPassword = "TestPassword";
    bool testRememberMe = false;
    string returnUrl = "/foo.html";

    var loginModel = new LoginModel
    {
        UserName = testUserName,
        Password = testPassword,
        RememberMe = testRememberMe
    };

    var accountController = new AccountController();

    //Setup underpinning via stubbing such that UrlHelper 
    //can validate that our "foo.html" is local
    var stubHttpContext = new StubHttpContextBase();
    var stubHttpRequestBase = new StubHttpRequestBase();
    stubHttpContext.RequestGet = () => stubHttpRequestBase;
    var requestContext = new RequestContext(stubHttpContext, new RouteData());
    accountController.Url = new UrlHelper(requestContext);

    RedirectResult redirectResult;
    //Scope the detours we're creating
    using (ShimsContext.Create())
    {
        //Sets up a detour for Membership.ValidateUser to our mocked implementation
        ShimMembership.ValidateUserStringString = (userName, password) =>
        {
            Assert.AreEqual(testUserName, userName);
            Assert.AreEqual(testPassword, password);
            return true;
        };

        //Sets up a detour for FormsAuthentication.SetAuthCookie to our mocked implementation
        ShimFormsAuthentication.SetAuthCookieStringBoolean = (userName, rememberMe) =>
        {
            Assert.AreEqual(testUserName, userName);
            Assert.AreEqual(testRememberMe, rememberMe);
        };

        redirectResult = accountController.Login(loginModel, returnUrl) as RedirectResult;
    }

    Assert.NotNull(redirectResult);
    Assert.AreEqual(redirectResult.Url, returnUrl);
}

With that, let’s run our tests and make sure everything is working.

TestRunAfterLoginMethod

One thing to notice here, is that the stubbing we’re doing (starting line 20 and continuing to line 27) doesn’t exactly convey what we’re trying to accomplish. All we care about doing is getting Url.IsLocalUrl to return true. Additionally, we had to know quite a bit about the internals of a Controller, UrlHelper, HttpContextBase, HttpRequestBase just to get this behavior to work.

In this scenario it would be preferable, readability wise, just to set and detour Url.IsLocalUrl. In this case our hand was forced since Microsoft Fakes and System.Web.Mvc aren’t currently cooperating, so I’m more than happy that at least a fallback was available.

JsonRegister Method

Both JsonRegister and Register are very similar, so we’ll just hit one of them. There’s really no new concepts here, just reapplying the what we used to test earlier methods.

[AllowAnonymous]
[HttpPost]
public ActionResult JsonRegister(RegisterModel model)
{
    if (ModelState.IsValid)
    {
        // Attempt to register the user
        MembershipCreateStatus createStatus;
        Membership.CreateUser(model.UserName, model.Password, model.Email, 
            passwordQuestion: null, passwordAnswer: null, isApproved: true, 
            providerUserKey: null, status: out createStatus);

        if (createStatus == MembershipCreateStatus.Success)
        {
            FormsAuthentication.SetAuthCookie(model.UserName, createPersistentCookie: false);
            return Json(new { success = true });
        }
        else
            ModelState.AddModelError("", ErrorCodeToString(createStatus));
    }

    // If we got this far, something failed
    return Json(new { errors = GetErrorsFromModelState() });
}

For JsonRegister we’ll need to shim Membership.CreateUser, which is straightforward enough. We’ll need to add a reference to System.Web.Security.ApplicationServices to our testing project for to work with MembershipCreateStatus and we’re good to go.

[TestMethod]
public void TestJsonRegister()
{
    string testUserName = "TestUserName";
    string testPassword = "TestPassword";
    string testConfirmPassword = "TestPassword";
    string testEmail = "TestEmail@Test.com";

    var registerModel = new RegisterModel
    {
        UserName = testUserName,
        Password = testPassword,
        ConfirmPassword = testConfirmPassword,
        Email = testEmail
    };

    var accountController = new AccountController();
    JsonResult jsonResult;
    //Scope the detours we're creating
    using (ShimsContext.Create())
    {
        //Sets up a detour for Membership.CreateUser to our mocked implementation
        ShimMembership.CreateUserStringStringStringStringStringBooleanObjectMembershipCreateStatusOut =
            (string userName, string password, string email, string passwordQuestion, 
                string passwordAnswer, bool isApproved, object providerUserKey,
                out MembershipCreateStatus @createStatus) =>
            {
                Assert.AreEqual(testUserName, userName);
                Assert.AreEqual(testPassword, password);
                Assert.AreEqual(testEmail, email);
                Assert.Null(passwordQuestion);
                Assert.Null(passwordAnswer);
                Assert.True(isApproved);
                Assert.Null(providerUserKey);
                @createStatus = MembershipCreateStatus.Success;

                return null;
            };

        //Sets up a detour for FormsAuthentication.SetAuthCookie to our mocked implementation
        ShimFormsAuthentication.SetAuthCookieStringBoolean = (userName, rememberMe) =>
        {
            Assert.AreEqual(testUserName, userName);
            Assert.AreEqual(false, rememberMe);
        };

        var actionResult = accountController.JsonRegister(registerModel);
        Assert.IsInstanceOf(typeof(JsonResult), actionResult);
        jsonResult = actionResult as JsonResult;
    }

    Assert.NotNull(jsonResult);
    var success = (bool)(new PrivateObject(jsonResult.Data, "success")).Target;
    Assert.True(success);
}

Running our tests once again, we can see everything’s passing

TestRunAfterJsonRegisterMethod

ChangePassword Method

The ChangePassword method is slightly more difficult to work with, since we have additional items to fake and stub, but otherwise the concepts are pretty similar.

[HttpPost]
public ActionResult ChangePassword(ChangePasswordModel model)
{
    if (ModelState.IsValid)
    {

        // ChangePassword will throw an exception rather
        // than return false in certain failure scenarios.
        bool changePasswordSucceeded;
        try
        {
            MembershipUser currentUser = Membership.GetUser(User.Identity.Name, 
                userIsOnline: true);
            changePasswordSucceeded = currentUser.ChangePassword(model.OldPassword, 
                model.NewPassword);
        }
        catch (Exception)
        {
            changePasswordSucceeded = false;
        }

        if (changePasswordSucceeded)
            return RedirectToAction("ChangePasswordSuccess");
        else
            ModelState.AddModelError("", "The current password is incorrect or the new password is invalid.");
    }

    // If we got this far, something failed, redisplay form
    return View(model);
}

We need to make sure that User.Identity.Name returns properly. In order to do this, we’re going to have to make sure AccountController’s User property gets populated with an Identity object. Again, due to the MVC faking issue, we’re going to approach this via stubbing, which is slightly less readable and requires some framework dumpster diving, but still gets the job done.

Decompiling down into the Controller class in System.Web.Mvc to see what we need to stub show the following:

public IPrincipal User
{
    get
    {
        if (this.HttpContext != null)
        {
            return this.HttpContext.User;
        }
        return null;
    }
}

Drilling in this.HttpContext

public HttpContextBase HttpContext
{
    get
    {
        if (base.ControllerContext != null)
        {
            return base.ControllerContext.HttpContext;
        }
        return null;
    }
}

The ControllerContext property is settable, so that’s our way in and it has a public ctor taking elements we already have. Additionally, we already have a StubHttpRequestBase which we can set the User property on.

We’ll need to add a Fakes Assembly for mscorlib in order to stub an IPrincipal for the AccountController’s User property. To add a Fakes assembly for mscorlib, add one for the System reference.  System.Web.ApplicationServices needs a Fakes assembly as well in order to shim the ChangePassword method on MembershipUser.

[TestMethod]
public void TestChangePassword()
{
    string testUserName = "TestUserName";
    string testOldPassword = "TestOldPassword";
    string testNewPassword = "TestNewPassword";

    var changePasswordModel = new ChangePasswordModel
    {
        OldPassword = testOldPassword,
        NewPassword = testNewPassword
    };

    var accountController = new AccountController();

    //Stub HttpContext
    var stubHttpContext = new StubHttpContextBase();
    //Setup ControllerContext so AccountController will use our stubHttpContext
    accountController.ControllerContext = new ControllerContext(stubHttpContext, 
        new RouteData(), accountController);

    //Stub IPrincipal
    var principal = new StubIPrincipal();
    principal.IdentityGet = () =>
    {
        var identity = new StubIIdentity { NameGet = () => testUserName };
        return identity;
    };
    stubHttpContext.UserGet = () => principal;

    RedirectToRouteResult redirectToRouteResult;
    //Scope the detours we're creating
    using (ShimsContext.Create())
    {
        ShimMembership.GetUserStringBoolean = (identityName, userIsOnline) =>
        {
            Assert.AreEqual(testUserName, identityName);
            Assert.AreEqual(true, userIsOnline);

            var memberShipUser = new ShimMembershipUser();
            //Sets up a detour for MemberShipUser.ChangePassword to our mocked implementation
            memberShipUser.ChangePasswordStringString = (oldPassword, newPassword) =>
            {
                Assert.AreEqual(testOldPassword, oldPassword);
                Assert.AreEqual(testNewPassword, newPassword);
                return true;
            };
            return memberShipUser;
        };

        var actionResult = accountController.ChangePassword(changePasswordModel);
        Assert.IsInstanceOf(typeof(RedirectToRouteResult), actionResult);
        redirectToRouteResult = actionResult as RedirectToRouteResult;
    }
    Assert.NotNull(redirectToRouteResult);
    Assert.AreEqual("ChangePasswordSuccess", redirectToRouteResult.RouteValues["Action"]);
}

After running tests, we see that our new unit test is passing.

TestRunAfterChangePasswordMethod

Conclusions

Through the use of Microsoft Fakes and the idea of noninvasive testing, with the mockist approach, we’ve been able to test the AccountController quite thoroughly without any project modifications. I imagine we could of easily hit 100% coverage if that was our goal. The only real issues we ran into were related to beta software.

Oddly enough, I’m glad we ran into the System.Web.Mvc faking issue. This forced us to use stubbing, and ultimately exposed both negative effects on overall readability and increased complexity in terms of the amount of framework decompiling needed to figure out what stubbing was necessary. Shimming in these cases would of better conveyed our intent and abstracted us away from having to deal with the guts of the underlying framework.

With these results in mind, it’s evident that testing tools have truly reached a point where anything can be tested, regardless of design. We’re entering a time where ANY application with ANY architecture can be thoroughly unit tested without even the slightest change to code; a time when the ability to unit test a system is decoupled from the design and architecture of that system.

All of this is for good reason. Today’s testing patterns and practices have arisen from limitations in our capabilities to isolate dependencies when unit testing code. Those limitations have been addressed, it’s time to reevaluate our approaches and move on.

When you drive architecture with the goal of being structurally easier to test, the only thing you end up with is an architecture that is good at being tested. Let architecture be naturally shaped by the needs of the problem domain over time. Let complexity escalate only as needed and simplicity, maintainability and ease of use all be key goals in a system’s design.

From now on, we can definitively say that any constraints or limitations in our abilities to thoroughly test any system with any design, are entirely self imposed.

 

The code for this post is available on GitHub

  • 59 Comments

Comparing Microsoft Moles in VS2010 to Microsoft Fakes in VS11

0 Comments
Posted in Unit Testing

In my previous post: Using Stubs and Shims to Test with Microsoft Fakes in Visual Studio 11 I went over Microsoft Fakes (a mocking framework built into Visual Studio 2011) in detail.  During my research I ran into Microsoft’s Moles Isolation Framework, which is the prior version of Microsoft Fakes that’s been in Microsoft Research for a few years. 

Moles is a lightweight framework for test stubs and detours in .NET that is based on delegates. Moles may be used to detour any .NET method, including non-virtual/static methods in sealed types. Moles is freely available on Visual Studio Gallery or bundled with Pex.

Just as with Microsoft Fakes, Moles provides a way to detour any .NET method as well as generally accepted basic Mocking functionality. The real interesting part about Moles is that it happens to be compatible with both Visual Studio 2010 and 2008. As I mentioned in the previous post there are only a handful of products that can detour .NET methods in VS 2010: TypeMock’s Isolator, Telerik’s JustMock and Microsoft’s Moles, costing: $799, $299 and free respectively.

I've been looking into “future proofing” some of our new testing initiatives and perhaps using Moles to bridge the gap between today’s testing and tomorrow’s tools. The Moles roadmap points out that Microsoft Fakes is the next generation of Moles and that a Moles to Fakes conversion will require a few modifications.

The Fakes Framework in Visual Studio 11 is the next generation of Moles & Stubs, and will eventually replace it. Fakes is different from Moles, however, so moving from Moles to Fakes will require some modifications to your code. A guide for this migration will be available at a later date.

That raises a few questions:

  • How different are the two versions?
  • How many modifications will be required for conversion?
  • Is it “safe” to use Moles in existing projects?

Under the covers things appear to have been refactored significantly for inclusion in VS 11. It’s hard to get a feel of how much movement there’s been since compare tools tend to breakdown when namespaces and class names are changed dramatically. After looking through the structures of both Moles and Fakes with ILSpy it’s pretty obviousthat the dev teams on this have been quite busy.

Luckily enough though, after a decent amount of usage it looks as though Moles is nearly identical to Microsoft Fakes in terms of general usage. With that in mind, the migration, even if it is a manual one, doesn’t look to be too bad and appears relatively straight forward. Despite the simplicity of conversion I’ll be holding out hope for an automated conversion out of the box or as a separate tool. 

The main user facing changes from Moles to Fakes are:

  • Fakes refers to a detour as Shim, instead of as a Mole
  • Generated classes that were prefixed with “M” for Mole  or “S” for Stub are instead prefixed with “Shim” and “Stub” respectively
  • The HostType attribute is no longer necessary
  • Some assembly attributes may no longer be necessary
  • .moles configuration files are now .fakes configuration files
  • Some configuration file options have been removed

With that said here’s a quick overview of installation, usage and comparisons detailing the similarities, reusing examples from my previous post on Microsoft Fakes.

Note: this post is a little lighter on the details of what Microsoft Fakes and Moles are and why you’d want to use it. If you find yourself wanting a little more background information and/or haven’t read my previous post on the subject, then you may want to take some time and check it out first

Installing Moles

Moles is available through Visual Studio’s Extension Manager. This eventually leads you to the product page in the Visual Studio Gallery. 

ExtensionManager

Search for Moles.

ExtensionManager2

Download and start the installer. Note: not all steps are shown here; the install is rather straight forward.

 MolesInstall1

Pick the typical setup type as custom doesn’t really offer anything you’d want to change anyway.

MolesInstall1b

Quick, easy, done.

MolesInstall1c

Now we’re ready for some “Moling” action!  Hmmm… perhaps it was for the better that they renamed this in Visual Studio 11.

Moling Example & Comparison

First off we have a little bit of boilerplate code to setup. Aside from the different namespaces and 1 different class name, this is essentially the same trivial cart example as the previous post, so feel free to skip ahead if your interested in the more “juicy” bits.

In Visual Studio 2010 create a C# Class Library project called MolesExample.  Rename the default class file to CartToMole and modify the code to be as follows:

using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;

namespace MolesExample
{
    public class CartToMole
    {
        public int CartId { get; private set; }
        public int UserId { get; private set; }
        private List<CartItem> _cartItems = new List<CartItem>();
        public ReadOnlyCollection<CartItem> CartItems { get; private set; }
        public DateTime CreateDateTime { get; private set; }

        public CartToMole(int cartId, int userId)
        {
            CartId = cartId;
            UserId = userId;
            CreateDateTime = DateTime.Now;
            CartItems = new ReadOnlyCollection<CartItem>(_cartItems);
        }
 
        public void AddCartItem(int productId)
        {
            var cartItemId = DataAccessLayer.SaveCartItem(CartId, productId);
            _cartItems.Add(new CartItem(cartItemId, productId));
        }
    }
}

Add the CartItem class.

namespace MolesExample
{
    public class CartItem
    {
        public int CartItemId { get; private set; }
        public int ProductId { get; private set; }

        public CartItem(int cartItemId, int productId)
        {
            CartItemId = cartItemId;
            ProductId = productId;
        }
    }
}

Add the DataAccessLayer class. Again, don’t worry about the connection string, we’ll never end up hitting it anyway.

using System.Data;
using System.Data.SqlClient;

namespace MolesExample
{
    public static class DataAccessLayer
    {
        public static int SaveCartItem(int cartId, int productId)
        {
            using (var conn = new SqlConnection("RandomSqlConnectionString"))
            {
                var cmd = new SqlCommand("InsCartItem", conn);
                cmd.CommandType = CommandType.StoredProcedure;
                cmd.Parameters.AddWithValue("@CartId", cartId);
                cmd.Parameters.AddWithValue("@ProductId", productId);

                conn.Open();
                return (int)cmd.ExecuteScalar();
            }
        }
    }
}

Next add a Unit Test Project called MolesExample.Tests and rename the default unit test class file to CartToMoleTests.

In the MolesExample.Tests project add a reference to the MolesExample project.  You’re solution should look as follows:

InitialSolutionConfig

Right click on the MolesExample reference in MolesExample.Tests references and select the “Add Moles Assembly” option.  Note: Screen captures of the Fakes version are positioned either to the right or bottom depending on your available screen width.

AddFakesAssembly  AddFakesAssembly

This adds a couple of new items to the project. You may need to refresh to see the new items as they didn’t come up right away for me.

AfterFakes  AfterFakes

The new references add the following types. Also take note of how the namespacing between Moles and Fakes has changed.

NewTypes  NewTypes2

The generated .Moles/.Fakes assembly type naming has only changed slightly between versions, which is what we’ll be working with for the most part. 

VS 10VS 11
MCartItem ShimCartItem
MCartItem.AllInstances ShimCartItem.AllInstances
MCartToMole ShimCartToShim
MCartToMole.AllInstances ShimCartToShim.AllInstances
MDataAccessLayer ShimDataAccessLayer
SCartItem StubCartItem
SCartToMole StubCartToShim

The .moles configuration file has been appropriately renamed to .fakes and the internals renamed accordingly.  I’ll expand out the structure of the .moles file later on in the post and compare that to the expanded .fakes configuration file.  For now here are the differences between them in terms of our two projects:

<Moles xmlns="http://schemas.microsoft.com/moles/2010/">
  <Assembly Name="MolesExample" />
</Moles>
<Fakes xmlns="http://schemas.microsoft.com/fakes/2011/">
  <Assembly Name="FakingExample"/>
</Fakes>

Now, let’s create our first test method using a Mole.  Just as before we’re going to unit test the AddCartItem method. AddCartItem calls SaveCartItem on DataAccessLayer, which happens to be static.  We’re going to mock out that database call to isolate the logic in AddCartItem in our unit test. Add the code below to CartToMoleTests.

using Microsoft.Moles.Framework;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace MolesExample.Tests
{
    [TestClass]
    public class CartToMoleTests
    {
        [TestMethod, HostType("Moles")]
        public void AddCartItem_GivenCartAndProduct_ThenProductShouldBeAddedToCart()
        {
            //Create a context to scope and cleanup moles
            using (MolesContext.Create())
            {
                int cartItemId = 42, cartId = 1, userId = 33, productId = 777;

                //Mole SaveCartItem rerouting it to a delegate which 
                //always returns cartItemId
                Moles.MDataAccessLayer.SaveCartItemInt32Int32 = (c, p) => cartItemId;

                var cart = new CartToMole(cartId, userId);
                cart.AddCartItem(productId);

                Assert.AreEqual(cartId, cart.CartItems.Count);
                var cartItem = cart.CartItems[0];
                Assert.AreEqual(cartItemId, cartItem.CartItemId);
                Assert.AreEqual(productId, cartItem.ProductId);
            }
        }
    }
}

Here’s the class file we used in the Microsoft Fakes example for reference.  The only real differences aside from namespacing are on lines 9 (Moles HostType is no longer needed), 13 (context name change) and 19 (actual detour) and all are quite minor.

using Microsoft.QualityTools.Testing.Fakes;
using Microsoft.VisualStudio.TestTools.UnitTesting;
 
namespace FakingExample.Tests
{
    [TestClass]
    public class CartToShimTests
    {
        [TestMethod]
        public void AddCartItem_GivenCartAndProduct_ThenProductShouldBeAddedToCart()
        {
            //Create a context to scope and cleanup shims
            using (ShimsContext.Create())
            {
                int cartItemId = 42, cartId = 1, userId = 33, productId = 777;
 
                //Shim SaveCartItem rerouting it to a delegate which 
                //always returns cartItemId
                Fakes.ShimDataAccessLayer.SaveCartItemInt32Int32 = (c, p) => cartItemId;
 
                var cart = new CartToShim(cartId, userId);
                cart.AddCartItem(productId);
 
                Assert.AreEqual(cartId, cart.CartItems.Count);
                var cartItem = cart.CartItems[0];
                Assert.AreEqual(cartItemId, cartItem.CartItemId);
                Assert.AreEqual(productId, cartItem.ProductId);
            }
        }
    }
}

Now let’s run it in Unit Test Explorer and see it pass.

TestrunAfterMole

Just as before, our mole/detour completely skipped over the SaveCartItem method in DataAccessLayer avoiding the exception due to “RandomSqlConnectionString” clearly not being a valid connection string. Start to finish, Moles ends up being just as easy to setup and use as Fakes for our simple cart example.

Stubs Example and Comparison

As with the Mole example above, there’s some boilerplate we need to setup before we get to the stub test.  Again, there are very few changes here, so if you’ve seen these before feel free to skip ahead.

We’ll need to create an ICartSaver interface and inject/pass it into the CartToStub object.  At which point we can stub out ICartSaver and have it return 42 just as we did for our Moling example. 

Add a new interface ICartSaver to the MolesExample project.

namespace MolesExample
{
    public interface ICartSaver
    {
        int SaveCartItem(int cartId, int productId);
    }
}

Next up, for completeness (although not necessary, since we’ll be stubbing around it), add a CartSaver class to the MolesExample project so we can replicate the implementation of DataAccessLayer.

using System.Data;
using System.Data.SqlClient;

namespace MolesExample
{
    public class CartSaver : ICartSaver
    {
        public int SaveCartItem(int cartId, int productId)
        {
            using (var conn = new SqlConnection("RandomSqlConnectionString"))
            {
                var cmd = new SqlCommand("InsCartItem", conn);
                cmd.CommandType = CommandType.StoredProcedure;
                cmd.Parameters.AddWithValue("@CartId", cartId);
                cmd.Parameters.AddWithValue("@ProductId", productId);

                conn.Open();
                return (int)cmd.ExecuteScalar();
            }
        }
    }
}

Create a new CartToStub class as follows similar to CartToMole but allowing for manual dependency injection of ICartSaver in the constructor:

using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;

namespace MolesExample
{
    public class CartToStub
    {
        public int CartId { get; private set; }
        public int UserId { get; private set; }
        private List<CartItem> _cartItems = new List<CartItem>();
        public ReadOnlyCollection<CartItem> CartItems { get; private set; }
        public DateTime CreateDateTime { get; private set; }
        private ICartSaver _cartSaver;

        public CartToStub(int cartId, int userId, ICartSaver cartSaver)
        {
            CartId = cartId;
            UserId = userId;
            CreateDateTime = DateTime.Now;
            _cartSaver = cartSaver;
            CartItems = new ReadOnlyCollection<CartItem>(_cartItems);
        }

        public void AddCartItem(int productId)
        {
            var cartItemId = _cartSaver.SaveCartItem(CartId, productId);
            _cartItems.Add(new CartItem(cartItemId, productId));
        }
    }
}

Moving back over to the MolesExample.Tests project, add a new unit test file named CartToStubTests.  Create/Modify the AddCartItem_GivenCartAndProduct_ThenProductShouldBeAddedToCart method to take into account the new changes.

using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace MolesExample.Tests
{
    [TestClass]
    public class CartToStubTests
    {
        [TestMethod, HostType("Moles")]
        public void AddCartItem_GivenCartAndProduct_ThenProductShouldBeAddedToCart()
        {
            int cartItemId = 42, cartId = 1, userId = 33, productId = 777;
 
            //Stub ICartSaver and customize the behavior via a 
            //delegate, to return cartItemId
            var cartSaver = new Moles.SICartSaver();
            cartSaver.SaveCartItemInt32Int32 = (c, p) => cartItemId;
 
            var cart = new CartToStub(cartId, userId, cartSaver);
            cart.AddCartItem(productId);
 
            Assert.AreEqual(cartId, cart.CartItems.Count);
            var cartItem = cart.CartItems[0];
            Assert.AreEqual(cartItemId, cartItem.CartItemId);
            Assert.AreEqual(productId, cartItem.ProductId);
        }
    }
}

Here’s the class file we used in the Microsoft Fakes example for reference.  Differences are on lines 8 (Moles HostType) and 15 (generated Stub type name) and are quite minimal.

using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace FakingExample.Tests
{
    [TestClass]
    public class CartToStubTests
    {
        [TestMethod]
        public void AddCartItem_GivenCartAndProduct_ThenProductShouldBeAddedToCart()
        {
            int cartItemId = 42, cartId = 1, userId = 33, productId = 777;

            //Stub ICartSaver and customize the behavior via a 
            //delegate, ro return cartItemId
            var cartSaver = new Fakes.StubICartSaver();
            cartSaver.SaveCartItemInt32Int32 = (c, p) => cartItemId;

            var cart = new CartToStub(cartId, userId, cartSaver);
            cart.AddCartItem(productId);

            Assert.AreEqual(cartId, cart.CartItems.Count);
            var cartItem = cart.CartItems[0];
            Assert.AreEqual(cartItemId, cartItem.CartItemId);
            Assert.AreEqual(productId, cartItem.ProductId);
        }
    }
}

Run the unit tests once more to see that everything’s passing.

TestrunAfterStub

Just as with the Moling example, the code is very similar to what would be used in Microsoft Fakes.

.Moles and .Fakes Xml Configuration File Comparison

The changes in the .fakes xml configuration file may require more effort to convert depending on what features were used. Basic usage as in generated but never modified shouldn’t be a problem. However, there are a handful of attributes which could be problematic to convert, since they’ve been removed. I’ve included both files expanded out via intellisense, with a description of the removed elements.

<Moles xmlns="http://schemas.microsoft.com/moles/2010/">
  <Assembly Name="" ExportedTypes="" Location="" ReflectionOnly="" x86="" />

  <CodeStyle AssemblyAlias="" DisableUniqueAlias="" MaxIdentifierLength="">
    <Copyright></Copyright>
    <FileHeader></FileHeader>
  </CodeStyle>

  <StubGeneration Disable="" SkipVirtualIndexers="" SkipVirtualMethods="">
    <Types>
      <Clear />
      <Add AbstractClasses="" Classes="" FullName="" Interfaces="" Namespace="" TypeName=""/>
      <Remove AbstractClasses="" Classes="" FullName="" Interfaces="" Namespace="" Obsolete="" TypeName=""/>
    </Types>
    <TypeFilter /> <!-- Deprecated -->
  </StubGeneration>

  <MoleGeneration Disable="">
    <Types>
      <Clear />
      <Add FullName="" Namespace="" TypeName="" />
      <Remove FullName="" Namespace="" Obsolete="" TypeName=""/>
    </Types>
    <TypeFilter /> <!-- Deprecated -->
  </MoleGeneration>

  <Compilation AssemblyName="" Debug="" DisableCodeContracts="" KeyFile="" ProjectTemplate="">
    <COMReference Guid="" VersionMajor="" VersionMinor="" WrapperTool="" />
    <Property Condition="" Name="" />
  </Compilation>
</Moles>

Here’s the .fakes file intellisense expansion for reference. 

<Fakes xmlns="http://schemas.microsoft.com/fakes/2011/">
  <Assembly Name="" Location="" Version="" x86=""/>

  <StubGeneration Disable="" SkipVirtualIndexers="" SkipVirtualMethods="">
    <Clear />
    <Add AbstractClasses="" FullName="" Interfaces="" Namespace="" TypeName=""/>
    <Remove AbstractClasses="" FullName="" Interfaces="" Namespace="" Obsolete="" TypeName=""/>
  </StubGeneration>

  <ShimGeneration Disable="">
    <Clear />
    <Add FullName="" Namespace="" TypeName=""/>
    <Remove FullName="" Namespace="" Obsolete="" TypeName=""/>
  </ShimGeneration>

  <Compilation Debug="" DisableCodeContracts="" KeyFile="" ProjectTemplate="">
    <COMReference Guid="" VersionMajor="" VersionMinor="" WrapperTool=""/>
    <Property Condition="" Name="" />
  </Compilation>
</Fakes>

Here’s an overview of the changes from Moles to Fakes:

  • The ExportedTypes attribute (.moles line 2) was removed from the Assembly element.

    Per the schema ExportedTypes is described as “Reflect over exported (visible outside the assembly) types only”. The only examples I can find for its usage are in moling the System assembly here and here. We’ll have to wait for the migration guide to see what is a suitable replacement for this behavior.
  • The CodeStyle element (.moles line 4) was removed.

    Per the schema CodeStyle is described as “Code generation settings”. Aside from the Copyright and FileHeader elements the child attributes generally looked like debug switches.  From the docs available at Microsoft Research there isn’t a whole lot of detail on when to use these settings or what purpose they serve.  Nobody’s going to miss these.
  • The Types elements (.moles lines 7 & 16) have been removed and everything below has shifted up one level. Sounds good to me.
  • The Classes attributes (.moles lines 9 & 10) were removed from the Add & Remove elements in StubGeneration.

    Per the schema Classes is described as “An optional filter to filter non-abstract classes”. Why this one was removed but not the inverse (AbstractClasses) is anyone’s guess. I imagine you can achieve the desired results with the other filters, so this could require a little more effort to update if you make use of this it.
  • The AssemblyName attribute (.moles line 24) was removed from the Compilation element.

    Per the Schema AssemblyName is described as “Specifies the assembly name of the generated assembly”. This attribute only applies when compilation is disabled”. Again, no documentation available, no examples of usage, probably not going to miss it.

Additional Considerations for System & Mscorlib

Previously in Moles when adding a Moles assembly for mscorlib or System, Moles required a little bit of jury rigging.  Assembly attributes may be required in some cases.  See this post at social.msdn.microsoft.com (code below quoted from post) and this one from StackOverflow for more info.

using System;
using System.Moles;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using Microsoft.Moles.Framework;
[assembly: MoledType(typeof(System.DateTime))]

namespace Y2kMoles
{
    [TestClass]
    public class Y2kTest
    {
        [TestMethod]
        [HostType("Moles")] // add this attribute
        public void Y2k()
        {
            MDateTime.NowGet = () => new DateTime(2000, 1, 1);

            if (DateTime.Now == new DateTime(2000, 1, 1))
                throw new Exception("y2k bug!!!!");
        }
    }
}

Even with these changes I was still getting build errors. Supposedly there are issues related to the installation of SP1. See this post on StackOverflow for more info.

BuildErrors

I ended up modifying the mscorlib.moles file using the type filters to only include DateTime and this fixed everything up.

<Moles xmlns="http://schemas.microsoft.com/moles/2010/">
  <Assembly Name="mscorlib" />
  <StubGeneration Disable="true" />
  <MoleGeneration>
    <Types>
      <Clear />
      <Add TypeName="DateTime" />
    </Types>
  </MoleGeneration>
</Moles>

That’s a slightly suboptimal user experience for such a common case. Microsoft Fakes seems to have fixed these issues. Adding a Fakes assembly to System also adds one for mscorlib and everything works just fine out of the box, which is the expected user experience. So during conversion to Fakes there may be some “dehacking” involved.

Conclusion

It would seem that Microsoft Moles is quite capable and so similar in terms of usage to Microsoft Fakes that using it in the interim until Visual Studio 11 is released is a simple decision. The functionality gained as well as future proofing ongoing development is well worth any drawbacks related to migration or some of Moles’ current issues. There will be some conversion effort involved (as the only conversion plans mentioned so far, are a guide for manual conversion steps), but it will be relatively simple and straight forward. Learn it, use it, love it!

 

The code for this post is available on GitHub.

  • 0 Comments

Using Stubs and Shims to Test with Microsoft Fakes in Visual Studio 11

16 Comments
Posted in Unit Testing

Microsoft Fakes is a full featured mocking framework built into Visual Studio 11. Currently, the available documentation is limited and marked as “preview only”, however it does provide us with some very good descriptions.

Microsoft Fakes is an isolation framework for creating delegate-based test stubs and shims in .NET Framework applications. The Fakes framework can be used to shim any .NET method, including non-virtual and static methods in sealed types.

Additionally here is Wikipedia’s definition of Mock Object for reference

In object-oriented programming, mock objects are simulated objects that mimic the behavior of real objects in controlled ways. A programmer typically creates a mock object to test the behavior of some other object, in much the same way that a car designer uses a crash test dummy to simulate the dynamic behavior of a human in vehicle impacts.

Shim any .NET Method

The real interesting part here is that Microsoft Fakes can shim “any .NET method, including non-virtual and static methods in sealed types”.  Existing mocking frameworks work by providing “on the fly” implementations to preexisting interfaces with a bit of Dependency Injection to get them situated in the class/method under test.

Microsoft Fakes is going a level deeper by mocking objects with no preexisting interfaces and allowing statics to be mocked in the process. This type of mocking is quite complex to pull off.  So much so, that there are only a handful of products out there that do this.  TypeMock’s Isolator, Telerik’s JustMock and the Microsoft Research project Moles that ultimately lead to Microsoft Fakes. 

TypeMock’s Isolator has been a regular so to speak in unit testing circles for a while now and I’ve actively considered it before, but the starting price of $799 has always been a little steep for my tastes. Telerik’s JustMock is the new kid on the block (past 2-3 years). JustMock has freemium edition supporting general mocking features and a pay version which has support for static, sealed & non virtual mocking with a base price of $299.

The biggest surprise is the Microsoft Research project Moles, which seems to have about the same set of features as Microsoft Fakes, has apparently been available since 11/1/2010 and is compatible with both Visual Studio 2010 & 2008 (based on the Visual Studio 2010 Moles x86 - Isolation Framework for .NET download page).  Perhaps this is the internet’s best kept secret… well, maybe not.  Regardless, I’ll be creating a future blog post going over using Moles in Visual Studio 2010.

Architectural Implications

Without a commercial mocking framework or Microsoft Fakes, if you’re going to do unit testing and you want to do it right as in truly isolate the code under test then you’re going to need to mock dependent objects.  However, just mocking the objects isn’t enough, you also need to instruct the code under test to use the mocks as opposed to its normal implementation.  Typically this requires some form of Dependency Injection. 

For reference here is Wikipedia's definition of Dependency Injection:

Dependency injection (DI) is a design pattern in object-oriented computer programming whose purpose is to reduce the coupling between software components. It is similar to the factory method pattern. Frequently an object uses (depends on) work produced by another part of the system. With DI, the object does not need to know in advance about how the other part of the system works. Instead, the programmer provides (injects) the relevant system component in advance along with a contract that it will behave in a certain way.

To truly isolate code in unit tests you’re typically bound to the following practices as part of Dependency Injection:

  • Any object that needs to be mocked needs to be non static and have an interface
  • You’ll either need to have a Dependency Injection framework as part of the architecture or the right patterns in place to manually resolve and inject as needed
  • Constructors will have to be modified to take in interface types for the dependent objects

That’s a lot of Architectural commitment.  As with any design pattern there are “trade offs” involved in usage.  With Dependency Injection you get reduced coupling and the ability to unit test thoroughly, increasing complexity and reducing maintainability by some level in the process.  The question with any design pattern’s usage is: are the trade offs worth it based on the functionality provided for the problem domain in question? 

Microsoft Fakes changes the trade off evaluation for Dependency Injection.  Currently you trade increased complexity and reduced maintainability for increased code quality brought about by unit testing. 

Microsoft Fakes decouples Dependency Injection from unit testing, therefore changing the value proposition for Dependency Injection to instead be weighed on its own merits of whether or not it adds value to the problem domain in question.

Proper unit testing can now be integrated into any codebase, legacy or new, small or large, and using Dependency Injection or not.  With this in mind Microsoft Fakes is a game changer.  Why a game changer now, presumably years after these features have been available in the wild? 

The answer is widespread availability and acceptance.  Anyone running Visual Studio will be able to run this (hopefully this will be part of the bare SDK as well).  Additionally examples, documentation, blogs and training will be much more readily available.  Faking may very well become a recommended practice along with unit testing at which point it could become as pervasive as general unit testing in .NET solutions.  It’s a lot easier to justify its usage when everyone has access to it, not just those willing to shell out $299 to $799 a seat.

Shims vs Stubs

Let’s take a look at the current MSDN definition for Shims and Stubs

Stub types Stub types make it easy to test code that consumes interfaces or non-sealed classes with overridable methods. A stub of the type T provides a default implementation of each virtual member of T, that is, any non-sealed virtual or abstract method, property, or event. The default behavior can be dynamically customized for each member by attaching a delegate to a corresponding property of the stub. A stub is realized by a distinct type which is generated by the Fakes Framework. As a result, all stubs are strongly typed.

Although stub types can be generated for interfaces and non-sealed classes with overridable methods, they cannot be used for static or non-overridable methods. To address these cases, the Fakes Framework also generates shim types.

Shim types Shim types allow detouring of hard-coded dependencies on static or non-overridable methods. A shim of type T can provide an alternative implementation for each non-abstract member of T. The Fakes Framework will redirect method calls to members of T to the alternative shim implementation. The shim types rely on runtime code rewriting that is provided by a custom profiler.

Delegates Both stub types and shim types allow you to use delegates to dynamically customize the behavior of individual stub members.

Stub types appear to be what most free and/or open source mocking frameworks already do for us today, so there’s really no surprises here.  Shim types on the other hand are where the action is at allowing us to detour static or non-overridable methods. 

MSDN mentions that the detouring in shims does degrade performance slightly.  In the brief amount of testing I’ve done there’s about a 20ms difference between shimmed vs stubbed implementation of the same code, which isn’t a deal breaker in terms of keeping tests fast.  It would make sense the cost could be additive based on the amount of detouring you end up doing.  So if you have the infrastructure to do either or, then pick stubbing over shimming. 

Getting Started with Shims

First off, you’ll need Visual Studio 11, fire it up and create a new C# Class Library project called FakingExample.  For this example I’ve put together a very trivial cart example to play with.

Rename the default class file to CartToShim and modify the code to be as follows:

using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;

namespace FakingExample
{
    public class CartToShim
    {
        public int CartId { get; private set; }
        public int UserId { get; private set; }
        private List<CartItem> _cartItems = new List<CartItem>();
        public ReadOnlyCollection<CartItem> CartItems { get; private set; }
        public DateTime CreateDateTime { get; private set; }

        public CartToShim(int cartId, int userId)
        {
            CartId = cartId;
            UserId = userId;
            CreateDateTime = DateTime.Now;
            CartItems = new ReadOnlyCollection<CartItem>(_cartItems);
        }

        public void AddCartItem(int productId)
        {
            var cartItemId = DataAccessLayer.SaveCartItem(CartId, productId);
            _cartItems.Add(new CartItem(cartItemId, productId));
        }
    }
}

Add the CartItem class.

namespace FakingExample
{
    public class CartItem
    {
        public int CartItemId { get; private set; }
        public int ProductId { get; private set; }

        public CartItem(int cartItemId, int productId)
        {
            CartItemId = cartItemId;
            ProductId = productId;
        }
    }
}

Add the DataAccessLayer class. Don’t worry about the connection string, we’ll never end up hitting it anyway.

using System.Data;
using System.Data.SqlClient;

namespace FakingExample
{
    public static class DataAccessLayer
    {
        public static int SaveCartItem(int cartId, int productId)
        {
            using (var conn = new SqlConnection("RandomSqlConnectionString"))
            {
                var cmd = new SqlCommand("InsCartItem", conn);
                cmd.CommandType = CommandType.StoredProcedure;
                cmd.Parameters.AddWithValue("@CartId", cartId);
                cmd.Parameters.AddWithValue("@ProductId", productId);

                conn.Open();
                return (int)cmd.ExecuteScalar();
            }
        }
    }
}

Next add a Unit Test Project called FakingExample.Tests and rename the default unit test class file to CartToShimTests.

In the FakingExample.Tests project add a reference to the FakingExample project.  You’re solution should look as follows:

InitialSolutionConfig

Right click on the FakingExample reference in FakingExample.Tests references and select the “Add Fakes Assembly”.

AddFakesAssembly

This adds a couple of new items to the project.

AfterFakes

The new references add the following types.

NewTypes

The .fakes file created under the Fakes folder turns out to be an xml file.  The file provides a configuration file for the generation of the fakes assembly. I’ll cover more about what can be done in this file later on in the post.  For now here’s a look at the default code generated output.

<Fakes xmlns="http://schemas.microsoft.com/fakes/2011/">
  <Assembly Name="FakingExample"/>
</Fakes>

Now we’re ready to do some faking.  Let’s take a look at unit testing the AddCartItem method.  AddCartItem calls SaveCartItem on DataAccessLayer, which happens to be static.  We’re going to mock out that database call to isolate the logic in AddCartItem in our unit test. Add the code below to CartToShimTests.

using Microsoft.QualityTools.Testing.Fakes;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace FakingExample.Tests
{
    [TestClass]
    public class CartToShimTests
    {
        [TestMethod]
        public void AddCartItem_GivenCartAndProduct_ThenProductShouldBeAddedToCart()
        {
            //Create a context to scope and cleanup shims
            using (ShimsContext.Create())
            {
                int cartItemId = 42, cartId = 1, userId = 33, productId = 777;

                //Shim SaveCartItem rerouting it to a delegate which 
                //always returns cartItemId
                Fakes.ShimDataAccessLayer.SaveCartItemInt32Int32 = (c, p) => cartItemId;

                var cart = new CartToShim(cartId, userId);
                cart.AddCartItem(productId);

                Assert.AreEqual(cartId, cart.CartItems.Count);
                var cartItem = cart.CartItems[0];
                Assert.AreEqual(cartItemId, cartItem.CartItemId);
                Assert.AreEqual(productId, cartItem.ProductId);
            }
        }
    }
}

Line 13 creates a ShimsContext , which limits the scope of our shimming.  Line 19 defines our shim/detour for the SaveCartItem method.  Notice how we set the new behavior through a property named SaveCartItemInt32Int32. The Int32Int32 on the end is the type signature of parameters accepted by SaveCartItem. Microsoft Fakes has to keep the generated property names unique and predictable for methods, since they could have overloads or be refactored to have them someday.  Now let’s run it in Unit Test Explorer and see if it passes.

TestrunAfterShim

It does indeed pass.  Notice that our shim/detour completely skipped over the SaveCartItem method in DataAccessLayer, if it hadn’t then we would’ve received a nasty exception since “RandomSqlConnectionString” is clearly not a valid connection string . Start to finish, this ends up being pretty easy to setup and use for our simple cart example. 

Stubs Example

For the stubs example our code will be very similar to the shim example however we’ll need to create an ICartSaver interface and inject it into the CartToStub object.  At which point we can stub out ICartSaver and have it return 42 just as we did for shimming.  We’ll manually inject the dependency as opposed to pulling down a DI framework.

Add a new interface ICartSaver to the FakingExample project.

namespace FakingExample
{
    public interface ICartSaver
    {
        int SaveCartItem(int cartId, int productId);
    }
}

Next up add, for completeness (although not necessary) add a CartSaver class to the FakingExample project so we can replicate the implementation of DataAccessLayer.

using System.Data;
using System.Data.SqlClient;

namespace FakingExample
{
    public class CartSaver : ICartSaver
    {
        public int SaveCartItem(int cartId, int productId)
        {
            using (var conn = new SqlConnection("RandomSqlConnectionString"))
            {
                var cmd = new SqlCommand("InsCartItem", conn);
                cmd.CommandType = CommandType.StoredProcedure;
                cmd.Parameters.AddWithValue("@CartId", cartId);
                cmd.Parameters.AddWithValue("@ProductId", productId);

                conn.Open();
                return (int)cmd.ExecuteScalar();
            }
        }
    }
}

Create a new CartToStub class as follows similar to CartToShim but allowing for manual dependency injection of ICartSaver in the constructor:

using System;
using System.Collections.Generic;
using System.Collections.ObjectModel;

namespace FakingExample
{
    public class CartToStub
    {
        public int CartId { get; private set; }
        public int UserId { get; private set; }
        private List<CartItem> _cartItems = new List<CartItem>();
        public ReadOnlyCollection<CartItem> CartItems { get; private set; }
        public DateTime CreateDateTime { get; private set; }
        private ICartSaver _cartSaver;

        public CartToStub(int cartId, int userId, ICartSaver cartSaver)
        {
            CartId = cartId;
            UserId = userId;
            CreateDateTime = DateTime.Now;
            _cartSaver = cartSaver;
            CartItems = new ReadOnlyCollection<CartItem>(_cartItems);
        }

        public void AddCartItem(int productId)
        {
            var cartItemId = _cartSaver.SaveCartItem(CartId, productId);
            _cartItems.Add(new CartItem(cartItemId, productId));
        }
    }
}

Moving back over to the FakingExample.Tests project, add a new unit test file named CartToStubTests.  Create/Modify the AddCartItem_GivenCartAndProduct_ThenProductShouldBeAddedToCart method to take into account the new changes.

using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace FakingExample.Tests
{
    [TestClass]
    public class CartToStubTests
    {
        [TestMethod]
        public void AddCartItem_GivenCartAndProduct_ThenProductShouldBeAddedToCart()
        {
            int cartItemId = 42, cartId = 1, userId = 33, productId = 777;

            //Stub ICartSaver and customize the behavior via a 
            //delegate, ro return cartItemId
            var cartSaver = new Fakes.StubICartSaver();
            cartSaver.SaveCartItemInt32Int32 = (c, p) => cartItemId;

            var cart = new CartToStub(cartId, userId, cartSaver);
            cart.AddCartItem(productId);

            Assert.AreEqual(cartId, cart.CartItems.Count);
            var cartItem = cart.CartItems[0];
            Assert.AreEqual(cartItemId, cartItem.CartItemId);
            Assert.AreEqual(productId, cartItem.ProductId);
        }
    }
}

Run the unit tests once more to see that everything’s passing.

TestrunAfterStub

Just as with shims, start to finish, this ends up being pretty easy overall for our simple cart example.

The Fakes Xml File

The current MSDN library documentation details the nature of the .fakes xml file as follows:

The generation of stub types is configured in an XML file that has the .fakes file extension. The Fakes framework integrates in the build process through custom MSBuild tasks and detects those files at build time. The Fakes code generator compiles the stub types into an assembly and adds the reference to the project.

The .fakes file provides fine grained control of how stub and shim generation work for a particular assembly.  Luckily enough the file has intellisense and pretty good descriptions when hovering, so exploration of features is rather straight forward.  Expanding out all attributes and elements using intellisense provides us with the following structures.

<Fakes xmlns="http://schemas.microsoft.com/fakes/2011/">
  <Assembly Name="" Location="" Version="" x86=""/>

  <StubGeneration Disable="" SkipVirtualIndexers="" SkipVirtualMethods="">
    <Clear />
    <Add AbstractClasses="" FullName="" Interfaces="" Namespace="" TypeName=""/>
    <Remove AbstractClasses="" FullName="" Interfaces="" Namespace="" Obsolete="" TypeName=""/>
  </StubGeneration>

  <ShimGeneration Disable="">
    <Clear />
    <Add FullName="" Namespace="" TypeName=""/>
    <Remove FullName="" Namespace="" Obsolete="" TypeName=""/>
  </ShimGeneration>

  <Compilation Debug="" DisableCodeContracts="" KeyFile="" ProjectTemplate="">
    <COMReference Guid="" VersionMajor="" VersionMinor="" WrapperTool=""/>
    <Property Condition="" Name="" />
  </Compilation>
</Fakes>

Based on this we see the following features:

  • Either shimming or stubbing can be completely turned off
  • Shims and stubs can be filtered such that only specific items from an assembly are shimmed and/or stubbed
  • Strong signing can be overridden via the KeyFile attribute, based on the MSDN docs the Fakes framework will automatically sign the .Fakes assembly with the same key the source assembly was signed with unless overridden here
  • There’s a facility to reference/deal with a COM component with COMReference if necessary.  Hopefully it won’t be…

Aside from items within the compilation element everything is along the lines of what we’d expect to be available here.  I imagine in most circumstances this file won’t need to be touched, however if you’re faking a larger library like mscorlib, then it would seem almost mandatory to filter the types generated otherwise the compiler could be chewing on it for a while.  The classic DateTime.Now shimming examples should probably be doing this type of filtering.

Conclusion

All in all I’m really excited about Microsoft Fakes.  It has the capacity to drive change in the .NET testing landscape similarly to how adding unit testing to Visual Studio (back in the 2005 version I believe) did so.  Widespread inclusion, usage and education of faking/mocking will be very beneficial for the community and will help drive innovation of testing techniques. Projects which would’ve been considered difficult if not impossible to add some level of unit testing to, without large amounts of refactoring, can now be unit tested easily.  Microsoft Fakes is a definitive win in my book.

 

The code for this post is available on GitHub

  • 16 Comments