The best way to learn something is to jump right in.  Best practices don’t really emerge until you’ve found out what works and what doesn’t.  This is my list of things that didn’t work on my four SpecFlow and WatiN projects in the last two years.  Of course, I’ll also talk about what did work.

Acknowledgements

First off, almost everything I know about SpecFlow/Cucumber/Gherkin I learned from Paul Rayner and pairing with with him on two projects.  I’m grateful for the experience and the knowledge gained.

I’ve also tried to glean as much as I can from Marcus Hammarberg, Darren Cauthon, Richard Lawrence, and Brandon Satrom in their twitter feeds and articles.

What level of your app should you test?

One big issue that comes up around SpecFlow is what level of your application should you be testing – internal or external.  The answer has a big impact on how you write your feature files, what side benefits you get from the feature files, and the speed of your tests.

Developer-facing

You can use SpecFlow as a BDD-style tool and do unit or light integration testing with it.  I call these developer-facing feature files, because they are expressed in language internal to the application (“Then I post the form values to the AccountController”).  The nice thing about this is the readability of the test from the dev side and the speed of execution, since you are explicitly testing code internals.

Business-facing

But I think this misses an opportunity to have richer language in the feature files.  If the SpecFlow features are worded the way the product owner sees the app working, you are fostering a ubiquitous language between the business and the developers.  You have a clearer vision of your app’s external behavior expressed through concrete examples.

You end up with “Then I click the save button” or even better, “Then I save” instead of “Then I post the form values to the AccountController”.  The bad thing about this style of spec wording is you end up testing through UI test automation tools like WatiN.  This testing style is slow and can be buggy and fragile if you aren’t careful (see below for tips).

In my experience, one of the main problems in software development is getting the dev team to understand what the business wants built and why.  SpecFlow is a great tool for getting both sides together to define that behavior through examples.

Furthermore, UI testing becomes much more important as web applications take on more and more JavaScript and other client-side code.  Since the external behavior of the app is how the business and users will evaluate the app, I think it makes sense to test at this level.

How not to set up your test project

Acceptance tests that drive the browser are not unit tests.  They are closer to integration tests, but they are even slower, since your local web server and the browser have to spin up each time and fill out forms, click buttons, etc.

Because of this, don’t add SpecFlow to your current unit test project or even your current integration test project, if you separate those.  Let it live in its own project with a name like MyProject.AcceptanceTests.  For quick test running and CI, it’s much easier to omit an entire assembly of tests, so keep these guys separate. 

I find that I often want to run the faster unit tests and run the acceptance tests only around the parts of the application we are working on.  I run the full suite maybe 2-5 times a week, especially before a hand-off to a stakeholder or a big demo.

How not to install SpecFlow and WatiN

Don’t bother with DLL downloads or manually setting up file references.  Install the SpecFlow NuGet package and the Visual Studio Extension.  Let NuGet handle the assembly references and setting up the app.config for you.  The VS Extension adds some nice feature file editing and right-click features to run a single test.

You’ll need to decide on MSTest versus NUnit.  The first comes with .NET, the later is free open source and you can add it with NuGet.  If you’re in a shop that heavily favors one over the other, use that.  All you’re using these for in the SpecFlow world is the generated code and the attributes marking your test fixtures and methods.

If you’re using NUnit (the default), you’ll need to be sure ApartmentState is set to “STA” in the app.config for your test project like this.  If you use the NuGet download, it should take care of that.

Finally, you’ll need WatiN, which can also be installed with a NuGet package.  I’ve sometimes had to set some of the WatiN assemblies to Copy Local = true in the acceptance test project to get everything working.  Kind of a hassle, but once it’s set up right, you’re done forever with it.

How not to set up a WebBrowser class and app.config

WatiN can run UI tests with Internet Explorer or FireFox (not Chrome, Safari, Opera, or anything else).  The FireFox API didn’t have parity with the IE one last time I checked, so I’ve gotten used to using the WatiN IE class.

However, don’t reference this class directly, or all your code is IE dependent.  What if the next version of WatiN supports an iPad emulator you want to use?  Do you really trust your find-replace skills that much?

Instead, wrap the references with a static class, and reference that in your step definitions.  Here’s the one I’m using these days that gets an instance of IE in a static constructor:

namespace Helpers
{
    public static class WebBrowser
    {
        private static IE _browser;

        static WebBrowser()
        {
            _browser = new IE();
            _browser.ShowWindow(NativeMethods.WindowShowStyle.Maximize);
        }

        public static IE Current
        {
            get { return _browser; }
            set { _browser = value; }
        }

        public static string BaseUrl
        {
            get { return ConfigurationManager.AppSettings["BaseUrl"]; }
        }
    }
}

This is maximizing the IE window after it opens and adding a reference to a BaseUrl property, which is set in the app.config. 

<appSettings>
    <clear/>
    <add key="BaseUrl" value="http://localhost:1234/"/>
    <!--<add key="BaseUrl" value="http://myapp.apphb.com"/>-->
</appSettings>

I use the BaseUrl to switch out the location of the test target in the step definitions (not shown above).  So if I want to test locally, the BaseUrl is “http://localhost:1234/”.  If I want to hit the AppHarbor version, the BaseUrl “http://myapp.apphb.com/”.  Just change the app.config and run the tests.

How not to set up your SpecFlow Before and After attributes

SpecFlow has attributes that can be put over your step definitions for your setup and teardown code.  Here are the hooks you have:

• BeforeTestRun / AfterTestRun
• BeforeFeature / AfterFeature
• BeforeScenario / AfterScenario

I’ve struggled with where these go before, but the last couple projects I created a BeforeAndAfter.cs class (clever, huh?), set the SpecFlow [Binding] attribute to the top of the class, then added all my before/after code.

This is a great place to put database clean ups, security code if your app has a login, any custom SpecFlow tags you’ve created, etc.  Here’s the code I used on my last project:

using System.Configuration;
using TechTalk.SpecFlow;
using WatiN.Core;

namespace Helpers
{
    [Binding]
    public class BeforeAndAfter
    {
        [BeforeTestRun]
        public static void BeforeTestRun()
        {
            // Ignore all features and scenarios when running on AppHarbor
            if (ConfigurationManager.AppSettings["Environment"] == "Test")
            {
                Assert.Ignore();
                return;
            }

            // Always login first
            Security.ILoginAsAdministrator();
        }

        [AfterTestRun]
        public static void AfterTestRun()
        {
            // Logout to be ready for next session
            Security.ILogout();
            
            // Close the browser and dispose
            WebBrowser.Current.Close();
            WebBrowser.Current.Dispose();

            // Reset the data to be ready for the next session
            TestData.CleanUp();
        }
    }
}

I put all my helper methods like the WebBrowser.cs and BeforeAndAfter.cs in a /Helpers folder in the acceptance test project.

How not to organize your feature files

Feature files have one or more features in them.  You could put each separate feature in it’s on .feature file, but that’s overkill.

I used to create a new feature file for each view in the app.  That probably makes sense in some cases, but this approach often misses the value of those pages.  For instance, if there are 2-3 pages for an e-commerce checkout process, it’s probably not that helpful to have a feature file for BillingAndShippingAddress.feature, CreditCard.feature, and Confirmation.feature.  It might be helpful to developers, but it’s likely non-developers on the team see the checkout process more holistically, so why not start with a Checkout.feature file?  If this gets too big, split out pieces into their own logical .feature files later.

I put all my feature files in a /Features folder in the acceptance test project.  It’s mostly cosmetic, but it does help a little.

How not to organize your step definition files

Step definitions are the implementation of the steps in the feature files.  Since I was organizing feature files by web app page, I started out doing the same with step definitions.  When you try this approach, you quickly see  the problem, since step definitions can and should be reused across features. 

If there is a step definition “Then I go to /MyPage”, the step definition might be clicking a menu, a link in the page, or just opening a URL.  As you can imagine, this can be helpful in lots of different feature files, so it makes more sense to group these in a Navigation.cs file or something like that.  Maybe you have a separate Menu.cs with those step definitions, or maybe you lump them in with Navigation. 

How you group these is up to you.  SpecFlow sees all *.cs files with a [Binding] on them as places where your step definitions could live, and it uses regex matching to find the right method.  SpecFlow doesn’t care if you put your methods in one class or 100 classes, but you will.  So put some thought into how you group these, because it’s easier to see opportunities to refactor out similar code you find within the same .cs file versus across .cs files.

This makes the file names much more generic, but I’ve had the most success with this style.  On recent projects, my step definition files have included Forms.cs, Menu.cs, Messages.cs, Navigation.cs, Search.cs, Security.cs, and SEO.cs.

I put all my step definition code in a /Steps folder.  In the end, it looks like this:

How not to code step definitions

Step definition code in each method should be brief – like a couple lines.  If you’ve got large blocks of code, you’ve probably got too much going on. 

Maybe you have a giant form with tons of fields and a SpecFlow table feeding you the values.  You can probably get away with something like this:

        public void IFillOutTheForm(Table table) 
        {
            foreach (var row in table.Rows)
            {
                var field = row["Field"];
                var value = row["Value"];
                switch (field)
                {
                    case "Something Different":  // Put exceptions here for drop downs, date pickers, radio buttons, etc.
                        WebBrowser.Current.SelectList(Find.ByName("SomethingDifferentId")).Select(value);
                        break;
                    default:
                        WebBrowser.Current.TextField(Find.ByName(field)).Value = value;
                        break;
                }
            }
        }

Here, the “Something Different” case is where you would put non-textbox filling out code.  But if your forms are like most business apps, there are lots of text boxes, a handful of drop downs, and a smattering of radio, checkbox, date pickers, sliders, etc. that require some different WatiN code. 

Of course, if the exceptions outnumber the convention, make a separate method and see if you can factor out some common code to use somewhere else.  Remember, the key to being productive with SpecFlow is reusing the step definitions, so once you start seeing duplication, look for ways to refactor to a common convention in your feature files and step definitions.  It takes a while to get a critical mass of these built up, but once you do, you really start picking up speed.

If you're using SpecFlow and WatiN to automate your web testing, things work great on localhost, but tend to fall apart on the continuous integration server.  It's a "chicken or the egg" paradox.  If the new code hasn't been deployed yet, it can't be externally tested with WatiN, but it shouldn't be deployed until all the tests have passed.  What are you supposed to do?

TeamCity

If your SpecFlow and WatiN tests are in their own assembly (something like MyProject.AcceptanceTests.dll), and you're using TeamCity, it's easy to run your unit tests and integration tests, then deploy the code to a web server, then run your acceptance tests in a separate build step.  It's just a matter of breaking up the steps and designating which test assemblies will be run at which time.

This step in the screen shot below is running the unit tests and integration tests.  A separate, later step will run the tests in the **\*AcceptanceTests.dll assembly after the code has been deployed to the web server.

AppHarbor

If you're using AppHarbor to run your tests on check in with "git push appharbor master", (and you should - it's super easy), you don't want to run your web tests until the code is deployed, but AppHarbor doesn't deploy anything until all the tests pass.

What we want is for AppHarbor to ignore the web automation tests until the code is deployed.  AppHarbor flips a *.config appSetting key named "Environment" to "Test" when it's testing and to "Release" when the code is deployed.  So while your tests are running, your App.config in your test project should look something like this:

<appSettings>
  <add key="Environment" value="Test"/>
</appSettings>

We can take advantage of this appSetting test runner change to tell SpecFlow to ignore this set of tests, which allows AppHarbor to continue running other tests and deploying the code.

This is not be the most elegant way to do this, but I got this working today:

[Binding]
public class MyStepDefinition
{
    [BeforeFeature]
    public void BeforeFeature()
    {
        if (ConfigurationManager.AppSettings["Environment"] == "Test")
            Assert.Ignore();
    }

    // Rest of step definition code...
}

The BeforeFeature attribute tells the SpecFlow test runner to execute that method before each feature.  With a quick check on the Environment appSetting value, we can tell AppHarbor to ignore this test and continue processing.  This will ignore all scenarios in all feature files.  Since all my scenarios are web automation tests, that works for me.

But what if you have some scenarios that rely on web automation testing and others that don't?  A more targeted approach would be to set a SpecFlow tag, like @web over the SpecFlow scenarios that require web automation testing to pass.  Then in the [BeforeScenario("web")] handlers for that tag, you could set Assert.Ignore to skip those scenarios while executing all other scenarios.

[BeforeScenario("web")]
public void BeforeScenario()
{
    if (ConfigurationManager.AppSettings["Environment"] == "Test")
        Assert.Ignore();
}

Once your non-web automation tests have passed and the code is deployed, you can point your local test runner at the AppHarbor URL and test against the newly deployed web app.

I'm assembling a team to work with me on a green field ASP.NET MVC project.  We will be using ASP.NET MVC 3, an ORM (NHibernate or EF Code First), SQL Server 2008, dependency injection (Ninject or Castle Windsor), and ATDD and TDD (Specflow, Watin, NUnit, and Moq).  If you've done some but not all of these things, I'll teach you the rest to fill in the gaps.

This project will be run using lean methods, so we'll pull in work only when the current task is completely done, meaning the product owner has accepted it and it's ready to go live.  We'll spend a lot of time getting the acceptance criteria just right before we build anything, so there will be lots of small, just-in-time conversations with the product owner.

We'll have flexible hours and do a lot of remote work, but we'll get together for designing and pair programming several days a week, also.  This will be done face-to-face, ideally, but remote web-cam/desktop-sharing pairing is an option, too.  We'll have to tweak this as we go to be sure communication isn't slipping and the work is getting done to the customer's expectations.

For more details or to apply, please check out the careers section.

I have been using SpecFlow with WatiN for about 9 months now.  It's a great combination for driving the features you want in a web site and verifying they work as expected.  SpecFlow takes care of the BDD framework and application features side, while WatiN drives the browser to verify behavior.

I've gotten used to the the pattern of writing a SpecFlow scenario, then writing the WatiN code in the step definition to open a browser and perform the task or look for the expected HTML.  Once you get the hang of the tools and the process, it's a very sustainable workflow for the developer.

But today I ran into something that I couldn't test with WatiN.  I wanted to check that an XML sitemap was coming up correctly on a web site.  WatiN was able to open the XML document in the browser, but there was no Body tag because the XML is not an HTML document, so WatiN couldn't read the XML.  All I needed to do was verify that an expected value was in the XML sitemap.  If the value was found, the test would pass.

I spent about 15 minutes trying out different WatiN commands to find the text, but with no Body tag, there was nothing loaded into the WatiN object model, so nothing was working.  I also tried looking through the WatiN Elements collection and a few other dead ends, but it was only showing me the previous web page or null.

Finally, the obvious answer occurred to me.  I'm looking at an XML document and I want to verify that it has a value in it.  I'm in a SpecFlow step definition and I've got the entire .NET framework at my disposal, not just WatiN.  Five minutes later, I've got it coded to load and examine an XDocument for the value I needed so the test can pass.

This was a good lesson for me.  Being on a development workflow roll is no excuse to not keep your head up for alternative strategies that would better solve your problem.  SpecFlow isn't just a wrapper for WatiN calls.  It can be used to verify any behavior it can invoke.

He doesn't like to be mocked!

It's so easy to take a direct dependency on HttpContext and not even realize it.  If you're in the code behind in Web Forms or in a controller action in MVC, it's just right there, tempting you to use it to access session variables, application security, etc.

But don't.

Some little known facts about HttpContext:

• HttpContext is the largest object ever created by humans.
• If you printed out the code for everything in HttpContext, the pages could be stacked end to end to wrap around the Earth's equator 7 times.
• Mocking HttpContext is like trying to calculate the last digit of pi.  There is always a little more to it. 
• Chuck Norris gave up trying to mock HttpContext.  He was deep in HttpContext.Response and quit, curled into a ball on the floor, and started whimpering.

Sealed what?  Object reference what?

Let's say you've got some MVC security stuff you're trying to work with in a controller action.  You've got code like:

    public class OrderController : Controller
    {
        [Authorize]
        public ActionResult Process()
        {
            if (User.IsInRole("Admin"))
            {
                return View("SecretAdminStuff");
            }

            return View("NotAuthorized");
        }
    }

See that User.IsInRole() code?  That call is really to HttpContext.Current.User.IsInRole().  You just took the bait and are tied to HttpContext now.

So what? You have to use HttpContext to get that information, right?  Sure, but you don't want to be tied directly to it or you've created untestable code.

Let's try a couple tests to verify the branching in this action method is returning the correct view based on the user's role.  I'm using NUnit, Rhino Mocks, and the Should assembly below.  Rhino Mocks has .Stub() and .Return() extension methods to set the expected return values for whether the user is or isn't in the Admin role.

        [Test]
        public void Should_return_NotAuthorized_view_when_not_Admin_user()
        {
            // Arrange
            var mockContext = MockRepository.GenerateMock<HttpContext>();
            mockContext.Stub(x => x.User.IsInRole("Admin")).Return(false);
            var orderController = new OrderController();

            // Act
            var result = orderController.Process() as ViewResult;

            // Assert
            result.ViewName.ShouldEqual("NotAuthorized");
        }

        [Test]
        public void Should_return_SecretAdminStuff_view_when_Admin_user()
        {
            // Arrange
            var mockContext = MockRepository.GenerateMock<HttpContext>();
            mockContext.Stub(x => x.User.IsInRole("Admin")).Return(true);
            var orderController = new OrderController();

            // Act
            var result = orderController.Process() as ViewResult;

            // Assert
            result.ViewName.ShouldEqual("SecretAdminStuff");
        }

As soon as we run the tests we see "System.NotSupportedException : Can't create mocks of sealed classes".  Oh yeah, HttpContext is sealed.  That won't work.

Hmmm.  Aren't there some new classes in System.Web.Abstractions for just this kind of thing?  Let's change HttpContext

var mockContext = MockRepository.GenerateMock<HttpContext>();

to HttpContextBase

var mockContext = MockRepository.GenerateMock<HttpContextBase>();

and see if that helps.

The new problem is: "System.NullReferenceException : Object reference not set to an instance of an object."  Mocking HttpContextBase isn't helping because the test isn't passing the mock context into the controller.  Every call in the action method to User.IsInRole() will always call the real HttpContext, which isn't created because the test is not running in the ASP.NET process.

You can get around this with Typemock Isolator, which goes beyond mocking and can intercept the next call to HttpContext.Current.User.IsInRole() and swap out a result, but isn't there a simpler way to do this without buying another product or waiting for better abstract classes to work with?

A better way

We can't test something tied to HttpContext like this.  We need another approach.  The one I favor is wrapping the calls we care about, delegating out to the untestable code in the wrapper class, then passing an interface to the wrapper class into the constructor of the class using it.

Here's the wrapping class I've created to access a few HttpContext current user values and the interface for the wrapping class:

    public class CurrentUser : ICurrentUser
    {
        public virtual string Name()
        {
            return HttpContext.Current.User.Identity.Name;
        }

        public bool IsLoggedIn()
        {
            return HttpContext.Current.User.Identity.IsAuthenticated;
        }

        public bool IsGuest()
        {
            return HttpContext.Current.User.IsInRole("Guest");
        }

        public bool IsAdmin()
        {
            return HttpContext.Current.User.IsInRole("Admin");
        }
    }

    public interface ICurrentUser
    {
        string Name();
        bool IsLoggedIn();
        bool IsGuest();
        bool IsAdmin();
    }

Now I need to update the controller to inject this dependency into the constructor so it can be mocked, and use a private field set in the constructor for subsequent calls to _currentUser:

    public class OrderController : Controller
    {
        private readonly ICurrentUser _currentUser;

        public OrderController(ICurrentUser currentUser)
        {
            _currentUser = currentUser;
        }

        [Authorize]
        public ActionResult Process()
        {
            if (_currentUser.IsAdmin())
            {
                return View("SecretAdminStuff");
            }

            return View("NotAuthorized");
        }
    }

Now we've got a testable action method.  I mock the CurrentUser with MockRepository.GenerateMock<ICurrentUser>, set its return value with the Rhino Mocks .Stub() and .Return() extension methods, and check for the expected results:

        [Test]
        public void Should_return_NotAuthorized_view_when_not_Admin_user()
        {
            // Arrange
            var mockCurrentUser = MockRepository.GenerateMock<ICurrentUser>();
            mockCurrentUser.Stub(x => x.IsAdmin()).Return(false);
            var orderController = new OrderController(mockCurrentUser);

            // Act
            var result = orderController.Process() as ViewResult;

            // Assert
            result.ViewName.ShouldEqual("NotAuthorized");
        }

        [Test]
        public void Should_return_SecretAdminStuff_view_when_Admin_user()
        {
            // Arrange
            var mockCurrentUser = MockRepository.GenerateMock<ICurrentUser>();
            mockCurrentUser.Stub(x => x.IsAdmin()).Return(true);
            var orderController = new OrderController(mockCurrentUser);

            // Act
            var result = orderController.Process() as ViewResult;

            // Assert
            result.ViewName.ShouldEqual("SecretAdminStuff");
        }

Hey, wait a sec.

If you're thinking to yourself: I've just tested the code in the action method and not whether HttpContext really works, you're exactly right.  The tests should cover the branching in the controller action and are unit tests.  They are not integration tests that check to see if HttpContext is returning the right values.

I'm OK with that tradeoff.  I know HttpContext works.  It's been around forever and it's in a System.<something> namespace, so I shouldn't be testing it. 

The real question is, do my login and roles and authorization settings work as expected in my application.  That calls for integration testing, and the easiest way to do that is click through those screens and see if it's working.  You can also automate this clicking around with Watin, or SpecFlow (which uses Watin internally to drive the browser).