BDDing with SpecFlow

Unit testing and TDD have become standard development practices for some time. Unit Testing helps build loosely coupled software and it verifies code is functioning correctly. Where it comes up short is verifying if the software being built actually satisfies the business goals. MSTest, NUnit and most other test frameworks can be used for testing high-level business processes. Unfortunately the tests are written in code with a programming language like C# that is not easily communicated with stakeholders making it difficult to verify if the software is achieving the business goals.

Behavior Driven Design (BDD) attempts to solve this problem. Tests are written at a level business stakeholders can review and understand. Stakeholders, working with the development team write examples of how the software should behave and the examples are copied into code verbatim! The result is tests that match the business requirements because the business stakeholders wrote (or co-wrote) them. And the tests function as living documentation – the documentation does not go stale because the documentation is the tests and the tests can be run regularly as part of automated builds.

This post focuses on SpecFlow. SpecFlow is one of the more popular BDD frameworks for .Net. Most importantly, for my needs it has integration with Visual Studio and MSTest and it looks like it easily integrates with my existing project.

Getting started

1. Install the SpecFlow Visual Studio plugin:

http://visualstudiogallery.msdn.microsoft.com/9915524d-7fb0-43c3-bb3c-a8a14fbd40ee

2. Create a Visual C# – Unit Test Project

3. Install the SpecFlow NuGet package


Install-Package SpecFlow

4. Create an App.config file with the following:


    <!-- For additional details on SpecFlow configuration options see http://go.specflow.org/doc-config -->

5. Create a SpecFlow Feature File – Merge.feature in this example

specflow-featurefile

6. Write SpecFlow Specification

Feature: Merge
	Merging users from one repository to my local database.

Scenario: Merge one user
	Given database is empty
	Given database does not have a user with Id 1
    When Merge User with Id 1
    Then database has user with Id 1

Hitting Ctrl-S will save the Feature file and generate the MSTest stubs. The next step is to generate the Step Definitions which are invoked from the MSTest stubs.

7. Generate SpecFlow Step Definitions

Right-click on the specification to select the Generate Step Definitions option. This generates the SpecFlow C# Binding where we map from the Specification to standard C# code.

spec-flow-generate-step-definitions

8. Implement Step Definitions

   [Binding]
    public class MergeSteps
    {
        private Merger merger = new Merger();

        [Given(@"database does not have a user with Id (.*)")]
        public void GivenDatabaseDoesNotHaveAUserWithId(int userId)
        {
            var existingUser = merger.FindUser(userId);

            Assert.IsNull(existingUser);
        }

        [Given(@"database has User with Id (.*)")]
        public void GivenDatabaseHasUserWithId(int userId)
        {
            merger.Merge(new User() {UniqueId = userId});
        }

        ...
    }

9. Run the tests

The SpecFlow tests can now be run like normal unit tests!

SpecFlow – Custom Attributes

I would like to be able to add annotations to SpecFlow specifications and have it translate to C# Attributes on the generated test methods.

Given a specification like this…

 @Author:John.Smith
 @Jira:SMG-1374,SMG-223
 Scenario: Merge one user alias. Deactivate an alias that was deleted.
    Given database has User with Id 1
    Given database User with Id 1 has addresses [foo1@gr.com|Active, foo2@gr.com|Active ]
    When Merge User with Id 1 and addresses [ foo2@gr.com|Active, foo3@gr.com|Active]
    Then database has user Id 1 and addresses [foo1@gr.com|Inactive, foo2@gr.com|Active, foo3@gr.com|Active ]

I want the generate coded to have my custom attribute like this.

[Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
[MyAttribute( "Merge one user alias. Deactivate an alias that was deleted.", "John.Smith", "SMG-1374,SMG-223" )]
         public virtual void Test() {...}

Out of the box Specflow only allows adding Tags to specifications. The Tags generate MSTest TestCategoryAttributes. By default, my Gherkin spec above generates the following code. Using reflection I could probably get the information I need from this implementation, but there should be a way to make it do exactly what I want.

[Microsoft.VisualStudio.TestTools.UnitTesting.TestMethodAttribute()]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategoryAttribute("Author:John.Smith")]
[Microsoft.VisualStudio.TestTools.UnitTesting.TestCategoryAttribute("Jira:SMG-1374,SMG-223")]
         public virtual void Test() {...}

I see SpecFlow does have support for creating plugins. Unfortunately their documentation is somewhat lacking. Luckily, SpecFlow is open source and there are a few references out there such as this very helpful blog post – http://blog.jessehouwing.nl/2013/04/creating-custom-unit-test-generator.html.

How-To

First, create a Visual C# – Class Library Project and Import the SpecFlow Custom Plugin NuGet Package


Install-Package SpecFlow.CustomPlugin

Next create a create a derived class of MSTest2010GeneratorProvider.

    public class MyGeneratorProvider : MsTest2010GeneratorProvider
    {
        public MyGeneratorProvider(CodeDomHelper codeDomHelper)
            : base(codeDomHelper)
        {
        }
     }

For my purposes, I need the code generation to change in two ways:

  1. Add my custom attribute when “Author” and “Jira” tags are present
  2. Add custom code in the generated TestInitialize() and TestCleanup() methods

Adding Custom Attribute

The MSTest2010GeneratorProvider has a SetTestMethod to override. This seems like a logical place to add custom attributes to each of the test methods. My implementation searches for any Tags that start with Author or Jira for the given scenario. Then using CodeDom, I add my custom attribute to the code. I also remove the tags so that the base.SetTestMethod() call does not add TestCategoryAttribute’s to each of the methods.

    public override void SetTestMethod(TestClassGenerationContext generationContext, CodeMemberMethod testMethod, string scenarioTitle)
    {
        foreach (var scenario in generationContext.Feature.Scenarios)
        {
            if (scenario.Title == scenarioTitle)
            {
                string title = scenarioTitle;

                if (scenario.Tags != null)
                {
                    Tag Author = scenario.Tags.FirstOrDefault(x => x.Name.StartsWith("Author"));
                    Tag jira = scenario.Tags.FirstOrDefault(x => x.Name.StartsWith("Jira"));

                    if (Author != null && jira != null)
                    {
                        scenario.Tags.Remove(Author);
                        scenario.Tags.Remove(jira);

                        var authorText = Author.Name.Split(new[] {':'}, StringSplitOptions.RemoveEmptyEntries)[1];
                        var jiraText = jira.Name.Split(new[] { ':'}, StringSplitOptions.RemoveEmptyEntries) [1];

                        testMethod.CustomAttributes.Add(
                            new CodeAttributeDeclaration(
                                "TestLogConnector.TestCaseAttribute",
                                new CodeAttributeArgument(new CodePrimitiveExpression(title)),
                                new CodeAttributeArgument(new CodePrimitiveExpression(authorText)),
                                new CodeAttributeArgument(new CodePrimitiveExpression(jiraText))));
                    }
                }

                var text = string.Join("\n", scenario.Steps.Select(c => c.StepKeyword.ToString() + ":" + c.Text));
                testMethod.CustomAttributes.Add(
                        new CodeAttributeDeclaration(
                            "TestLogConnector.TestCaseComment",
                            new CodeAttributeArgument(new CodePrimitiveExpression(text))));
            }
        }

        base.SetTestMethod(generationContext, testMethod, scenarioTitle);
    }

TestInitialize() and TestCleanup()

I need to add 2 public properties and logic in the Initialize and Cleanup methods. SpecFlow has SetTestClass and SetTestClassInitializeMethod methods to override, but there is no SetTestClassCleanupMethod. I found that for my needs it didn’t matter where I put the code generation logic, so I put it all in the SetTestClassInitializeMethod.

Here’s the code I used to modify the TestInitialize and TestCleanup methods:

    public override void SetTestClassInitializeMethod(TestClassGenerationContext generationContext)
    {
        var field = new CodeMemberField()
                        {
                            Name = "testContext",
                            Type = new CodeTypeReference("Microsoft.VisualStudio.TestTools.UnitTesting.TestContext"),
                            Attributes = MemberAttributes.Private
                        };
        generationContext.TestClass.Members.Add(field);

        field = new CodeMemberField()
                    {
                        Name = "testCase",
                        Type = new CodeTypeReference("TestLogConnector.TestCase"),
                        Attributes = MemberAttributes.Private
                    };
        generationContext.TestClass.Members.Add(field);

        var codeMemberProperty = new CodeMemberProperty();
        codeMemberProperty.Name = "TestContext";
        codeMemberProperty.Type = new CodeTypeReference("Microsoft.VisualStudio.TestTools.UnitTesting.TestContext");
        codeMemberProperty.Attributes = MemberAttributes.Public;
        codeMemberProperty.HasGet = true;
        codeMemberProperty.HasSet = true;
        codeMemberProperty.GetStatements.Add(
            new CodeMethodReturnStatement(
                new CodeFieldReferenceExpression(
                    new CodeThisReferenceExpression(), "testContext")));
        codeMemberProperty.SetStatements.Add(
            new CodeAssignStatement(
                new CodeFieldReferenceExpression(
                    new CodeThisReferenceExpression(), "testContext"),
                new CodePropertySetValueReferenceExpression()));

        generationContext.TestClass.Members.Add(codeMemberProperty);

        codeMemberProperty = new CodeMemberProperty();
        codeMemberProperty.Name = "TestCase";
        codeMemberProperty.Type = new CodeTypeReference("TestLogConnector.TestCase");
        codeMemberProperty.Attributes = MemberAttributes.Public;
        codeMemberProperty.HasGet = true;
        codeMemberProperty.HasSet = true;
        codeMemberProperty.GetStatements.Add(
            new CodeMethodReturnStatement(
                new CodeFieldReferenceExpression(
                    new CodeThisReferenceExpression(), "testCase")));
        codeMemberProperty.SetStatements.Add(
            new CodeAssignStatement(
                new CodeFieldReferenceExpression(
                    new CodeThisReferenceExpression(), "testCase"),
                new CodePropertySetValueReferenceExpression()));

        generationContext.TestClass.Members.Add(codeMemberProperty);

        base.SetTestClassInitializeMethod(generationContext);

        generationContext.TestInitializeMethod.Statements.Add(new CodeSnippetStatement(
                                                                    @"            if (TestContext != null)
        {
            TestCase = new TestLogConnector.TestCase(GetType(), TestContext.TestName);
        }
"));

        generationContext.TestCleanupMethod.Statements.Add(new CodeSnippetStatement(
                                                                    @"           if (TestContext != null)
        {
            TestCase.WriteFinishedTestToTestLog(TestContext.CurrentTestOutcome.ToString());
        }
"));

Referencing the Plugin

The next step is to implement IGeneratorPlugin and register my custom unit test provider.

[assembly : GeneratorPlugin (typeof (MyGeneratorPlugin ))]

public class MyGeneratorPlugin : IGeneratorPlugin
    {
        public void RegisterDependencies( ObjectContainer container)
        {
        }

        public void RegisterCustomizations( ObjectContainer container, SpecFlowProjectConfiguration generatorConfiguration)
        {
            container.RegisterTypeAs< MyGeneratorProvider, IUnitTestGeneratorProvider >();
        }

        public void RegisterConfigurationDefaults(SpecFlowProjectConfiguration specFlowConfiguration)
        {
        }
    }

The final step is to compile the assembly (hopefully it compiles) and place the libary in a patch accessible from SpecFlow. They are loaded from several locations. GeneratorPluginLoader.cs contains the logic for how SpecFlow loads plugins.

I found the easiest place to put the plugin is in the /packages/SpecFlow.1.9.0\tools folder.

Finally reference the plugin in the main projects App.Config file:

    <plugins>
      <add name="MyGenerator" type="Generator"/>
    </plugins>

Next time the Feature file is saved the code will be generated using the plugin!

Wrap-up

I’m interested to see how well SpecFlow works over an extended period of time. My hope is that it provides long term value to my project in the form of better documentation and more easily maintained tests.

All source code for this can be found here:

https://github.com/marksl/Specflow-MsTest

Advertisements
This entry was posted in .Net, C#, Unit Testing and tagged , , , , , . Bookmark the permalink.

One Response to BDDing with SpecFlow

  1. Charles says:

    The section on Custom Attributes was awesome. I haven’t had a change to look into extending SpecFlow and your post sparked all kinds of things I would like to try. Thanks.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s