How can we sweat those test assets?


How many of us have embraced the wave of free thinking over the last few years, and started implementing automated acceptance tests in our sleep?  I know that the teams I’ve worked with have, and we have all become increasing better at it at an encouraging rate.  On top of that, we no longer have to come with a huge investment in tools to get started on this journey these days.  I love the fact that I can download software in a few minutes and start building automated test assets in an equally short amount of time.  I did this recently with a demo.  A quick download of Java, Eclipse, Selenium 2 Jar… 15 minutes later we had some working code, in the form of some JUnit 4 tests for a web page, to talk around.  This is some distance away from the pain I remember from assessing automation even a mere 5 years ago (insert former Mercury products and consultancy blank cheques here).

However, in this age of quick start and easy assess to tools and knowledge don’t fool yourself.  You are still making in investment, and usually a huge one after a team have built automated acceptance tests that have matched an ever growing feature set of the system under test.  People time has been invested, and it has still cost you money.  It is just that the entry point into the investment has been made easier.  Don’t get me wrong, I know these test assets will have enormous value for a multitude of reasons.  However, with valuable assets in any type of business… you need to get the most out of them and maximise their value (if it helps you towards your overall goal).

Multi-Purpose Usage

Ok.  So I should have really entitled this article “reuse of functional automated acceptance tests for another purpose” because that’s where I’m coming from.  I suppose I wanted to give the idea a value context, as that is where my head lives when trying to communicate with others.  However, to drop down to the test and risk level again… What are your team doing to cover off potential risks (assuming the system-under-test is web site for example) around some of the following:

  • Stack performance bottlenecks under load, stress, etc.
  • Performance at page level (e.g. page weights)
  • Security (e.g. cross site scripting, SQL injection)
Unfortunately, in many cases that I have come across the answer is likely to be one of the following:
  • We haven’t thought about that yet as we are focusing on story acceptance
  • We do something ad hoc either manually or semi automated
  • We have another set of monolithic environments and tools to handle that
  • We want to cover those risks, but there is a bar from entry (e.g. setup, cost, knowledge)

So over the next couple of posts I’m going to look at what I’ve come across in this area, but I’m also interested in what you’ve come up with. Send me your thoughts, ideas, and blog posts.  Whether that’s on how you’ve got ShowSlow ( setup to collect data from your tests, or if you’ve got OWASP ZAP ( hooked in to the build.  I hope to hear from you.

Keep your eyes on the DSL prize with Twist

Lost and Confused Signpost


Learning a language can be a challenging task.  The absorption of a lexicon takes time and patience.  Twist can help keep this task achievable for consumers of acceptance tests by allowing the definition of confirmation language to be natural.  This is assuming that the consumers will be business customers or other non-technical people.  If that is not the case, I would suggest using a pure code test framework such as TestNG to achieve the technical benefits of the Twist runner design.

Assuming the later statement is the case, Twist has the potential to facilitate communication in and around software delivery teams on a dramatic scale.  It can help give teams a highly accesible language to express acceptance tests that give common understanding, and enable rapid continuous delivery through automation.  However, it does take discipline to keep that goal in mind, and not to get side tracked on the technicalities of coding the automation solution.  It’s not that those things don’t need consideration.  On the contrary, it’s good to be thinking about the execution giving the fastest possible feedback, producing maintainable automation code, and ideally even test data abstraction.  The fact is that those things won’t matter if the language for communication is neglected as the focus.

Content Assist

Twist has an excellent set of features for managing a scenario focused language.  In particular, it allows the following:

  • Creation of method signatures and code from workflow steps (in natural language)
  • Rephrasing of workflow steps across scenarios and the use of searching by content assist
  • Abstraction of multiple workflow steps in to concepts or code implementation (the first is preferred for language clarity)
  • Data driving of workflow step parameters
  • … and an ever growing list

Limit The Deviation and Limit The Confusion

The experience I’ve had is that it is easy to create a massive amount of confusion (and also code) if you don’t keep to some basic patterns of usage around the language used and some ground rules.

Usage Patterns

  • Break down workflow fixtures by pages / areas / services to give them context
  • Before creating a new workflow step search for an existing match
  • When defining a new workflow step ensure it can be easily searched for by others

ground rules

  • To support the searching of workflow steps allow the context of the area of the system under test to be visible
    • e.g. “… on the home page” or “… using google search”
  • Keep to a basic set of action words against each area so that it can display the interactions it allows easily
    • e.g. “Verify search suggestions for ‘Simon Reekie’ using google search”
    • e.g. “Set language to ‘English’ on the home page”
    • e.g. “Set currency to ‘British Pounds’ on the home page”
  • If you are going to get something off one part of the system under test and verify it in another, then keep it visible within the scenario language
    • e.g. “Get and Store breakfast price as “Reported Breakfast Price” on the price page
    • e.g. “Verify breakfast price is equal to Stored “Report Breakfast Price” on the summary page


It is difficult enough communicating with many individuals to a common understanding, and keeping up with managing the complexity of that communication.  Give yourself a chance by building a DSL for Twist scenarios that keeps to a few understood rules, and works within the boundaries of the tool capabilities.  There is nothing wrong with adding or changing the rules, but do it knowingly and collaboratively with those using them.

For those that are looking for the technical implications of this approach, I can only really give you these figures as food for thought.

  • Before this approach was taken in my current workplace, and scenario workflow step language spaghetti was everywhere… we had:
    • Thousands of fixture and test implementation classes built up with duplication (I know… refactoring mindsets could have helped)
    • It took days in some cases for all types of people to write a useful automated scenario
    • The debugging of test failures out of CI took an hours and became a specialist skill
  • After this approach was embedded we had the same coverage plus much more, but we also had:
    • A couple of dozen fixture classes
    • A whole range to skill levels could build new scenarios to point of automation within minutes to an hour