BDD with Cucumber vs Spock

In this post I compare Cucumber with Spock for BDD. To do this I consider two simple test cases of

• a book library that offers a 'search by publication year' feature, and
• a salary manager that modifies the salary of one or more employees. 

I take the cucumber code from this bitbucket repository. The Spock code can be found in the spock-example repository. The source code (domain model) is the same in both cases. Only the test code changes.

The Book Library Test Case

With Cucumber

Cucumber offers a DSL for writing feature specifications in English, called Gherkin, that is then associated with test code. A feature contains one or more scenarios. Each scenario consists of steps. A simple feature with one scenario for a book library could look like:

Feature, Scenario, Given, When, Then, And are cucumber reserved keywords, while the rest is English. Each sentence starting with Given, When, Then defines a step. If you run this test, cucumber will search for step method implementations associated with the steps defined in Gherkin. The association is done with annotations containing proper regular expressions. If no such implementation exists Gherkin will print in the console the signature of the methods you have to implement, like:

You need to copy paste this code into a test class and then provide implementations for each of them. Notice however that some methods can be merged with each other if we provide smarter regular expressions. For instance, the three methods annotated with @Given can be merged into one by providing regular expressions for the title, author and publishing date. The most concise implementation is:

This class name ends with Steps to denote that it provides step implementations and starts with BookSearh to denote that these steps are associated with the search_book.feature, but the association is not done with the class name, but solely relies on the regular expressions with which methods are annotated. Cucumber will search any class in the 'src/test' directory to find a method whose annotation matches a Gherkin step.

Cucumber also needs a test class that links to the Cucumber runner like:

With Spock

Spock is a DSL for BDD written in Groovy. The search by publication year scenario can be written in Spock as:

This is very similar to the Gherkin syntax and equally easy to read. The difference is that Gherkin is a text document that is parsed by the Cucumber framework and associated to Java (or other language) code, while a Spock method is at the same time a natural language specification and valid Groovy code. Therefore the implementation is done in the same Spock file as follows:

The code that implemented Cucumber step methods is mixed with the natural language specification in Spock. Spock allows the implementation to be done in Groovy or Java.

The Salary Manager Test Case

With Cucumber
A more complex test case is that of a salary manager of many employees that modifies employee salaries. Although the original example considered the modification of a single employee's salary, I change the test to allow for different test input data in the same manner as a DataProvider works in TestNG:

To run the same test with different input data, Cucumber offers the Scenario Outline and Examples keywords. Scenario Outline denotes that this scenario (test) needs to run multiple times and the table after the Examples keyword offers the input data. The first table row contains column headers which act as variables that can be used in the steps text. The scenario runs once per data line. Notice that the first table contains initialization data, e.g. the employees that this manager manages. The same set of employees will be created for each run.

The step method implementation follows:

Notice, that the initialization data table is mapped to a List<Employee> object and passed by Cucumber as an argument of the the_salary_management_system_is_initialized_with_the_following_data method.

With Spock
The same feature specification could be written with Spock as follows:

Spock offers data tables for test input data (the analog for Cucumber Examples table), but not for initialization data. This is a clear drawback. A workaround is to define the init data as a two dimensional array. The implementation then needs to create a list of employees out of the array, as shown:

Conclusion

Both Cucumber and Spock tight couple the natural language specification with the test code. This is a direct consequence of the BDD paradigm that both frameworks have been created to support. But Cucumber does this in a more strict manner. Changing the natural language will break the test code, e.g. with a missing step implementation if no method's regular expression matches the given step text. While for spock the text is an arbitrary string after the ':' symbol. It is not validated against the implementation that follows.

Also Cucumber offers a clear distinction between natural language specification and test code. This is an advantage for people other than developers having to write or read the specs. In the end, the very essence of BDD is the close collaboration of product owners, BAs, QAs, architects and developers so that the specification is agreed and understood by all before the development starts.

On the other hand, Spock offers a fast, concise and one file solution. Groovy's ability to use any string as a method name allows for concise test case names. It offers to developers a single point to read and understand the spec and the code that implements it, let alone the goodies with which Spock comes, like internal mock support or advanced data table capabilities (e.g. using closures in table cells).

To conclude, Cucumber looks to be a better fit for integration or end-to-end tests, that involve interaction of people of different skills and backgrounds, while Spock for unit tests, that are more or less solely developers responsibility.