The measure of quality in software is a difficult thing to quantify. We have various tools which enable us to test various aspects of the system, but ultimately I think the measure of quality in a system is the extent to which the software meets the expectations of actual users.

The human perspective must be injected into any test framework which aims to meet this goal. Behavior Driven Development (BDD) fulfills this need by creating specifications modeled in the domain language of the end users. While programmers are adept at manipulating abstractions, this should not be a requirement of end users. They should be able to articulate their expectations in the language most familiar to them. By codifying these expectations as acceptance criteria, the tests become a living document of behaviors rooted in the experience of its users.

Currently there does not exist a single tool which allows us to model these expectations. Instead, we must construct a framework in which user expectations can be used to drive tests against an application. In addition, other factors must be taken into consideration in the design of the framework. Extensibility, modularity, and maintainability are also desirable features. If we assume that the Application Under Test (AUT) will evolve over time to better express the needs of the users, then tests written against the system should be robust to changes which do not affect their original intent.

The framework that I've constructed uses the latest versions of Selenium/WebDriver 2.0 in addition to JBehave 3.4. WebDriver provides the automation harness against which test specifications are run against using JBehave. Tests are executed via Jenkins when either the application has changed or new tests are introduced to the system. There are a lot of moving parts to this solution, so I'll break it down into a number of parts:

Part 1 - Plan for Change

All systems evolve over time. As a consequence, not accounting for change makes for brittle solutions. Fix this limitation with the right architecture.

Part 2 - Testing for the Common Man

The most prominent barrier to testing involves expressing tests without the requirement of programming experience. Having this capability allows business requirements to be expressed by those who will use the system the most.

Part 3 - Organization is Key

There are a lot of articles out there that show how to use WebDriver/JBehave functionality, but less that address organizing tests into workable groupings. Organization is paramount if we are to create a solution that limits duplication and allows for scalability.

Part 4 - The Automation Environment

BDD works best when we remove the foibles of human intervention from the equation. This however, requires tooling an environment to support continuous deployment. This dovetails well with the concepts of DevOps. Virtualization in particular is useful for creating these types of environments.

There are a few misconceptions regarding Selenium / WebDriver that should be clarified from the start. When I first began to use Selenium (pre 1.0) the mechanism for automation relied on the technique of Proxy Injection. While this worked well for applications which rendered largely static content, the wide-spread use of AJAX made this technique very unreliable from an automation point-of-view. Timing was difficult to manage and required client-side javascript to notify automation of the application's ready state. In addition, it was impossible to account for browser events that were not represented in the DOM (such as javascript alerts).

Enter WebDriver 2. Instead of using Proxy Injection, WebDriver automates actions using a browser's native events. This allows for more reliable timing mechanisms as well as the possibility for catching native browser events. Overall, this is great leap forward for automation.

Beware Selenium-IDE

There is just one other caveat with using Selenium from my perspective which should be avoided at all costs - the Selenium IDE. While useful for small, relatively static sites, Selenese tests result in massive duplication, brittle abstractions, and tests that do not encourage reuse. Capture and record tools such as the Selenium IDE are seductively simple to use, but extremely difficult to maintain. Changes to the application require changes to tests themselves. This quickly becomes unmanageable due to the violation of the DRY principle. As such, tools such as Selenium IDE can be classified as semi-automatic as they require a lot of manual intervention when the application changes. Ultimately, semi-automated testing methodologies are doomed to fail simply because the underlying system they depend upon to function ultimately changes. Any system which does not account for application evolution can only capture the requirements at a given point in time.

WebDriver is an API

By favoring programmatic automation, you can leverage general programming principles to create a framework which accounts for the natural evolution of the AUT.

Which brings us to the crux of the matter -WebDriver is an API, not the solution. What does this mean? WebDriver is the means by which automation is achieved, but by itself it does not necessarily give structure to the solution of automating applications in general. The relationships between application components are loosely defined and has no inherent structure in WebDriver. In some cases, the pre-package abstractions are insufficient to reflect the complex relationship between application components.

This is the role of the framework; to support automation using reusable abstractions against an evolving application.

Automation as a goal does not happen in a vacuum. Applications must be constructed to support automation. As such, coordination with application architects is crucial to support any effort. The framework must be flexible enough to support clearly defined tests whose implementations may change as the application evolves, but whose intent remains the same. To this end, I will describe some of the design decisions I made when creating my testing framework.

Web Application Automation Concepts

The Reusable Element

Most applications reuse GUI controls; this must be reflected any testing framework. While WebDriver supports finding generic WebElements and manipulating them via additional actions, this requires test code which rely on these methods to be aware of implementation details. In addition, this introduces duplication as soon as you have more than one of a single type of element. By creating an Element abstraction, you can define similar application components by their locators as well as the types of operations they can support for automation. For example, a Text Box is different from a Dropdown. One cannot select items from a Text Box but both may allow for text input. Common behaviors need to be defined in a single representation, while type specific behaviors need to be differentiated.

Having a reusable Element abstraction allows you to do nifty things like automatic validation of a control based on its type. This is particularly useful for smoke tests. As well, changes to an Element's behavior can easily be propagated throughout the entire testing framework if it is expressed canonically in a single representation. Elements also allow for functional composition. By taking two for more fundamental Elements, you can compose testable aggregate Elements with increasingly sophisticated behavior which still behave as a single functional unit.

You can also localize procedural abstractions such as when an Element is resolved. Ideally, you want to resolve any given just before it is used. This minimizes DOM inconsistencies which arise in applications which re-render output based on post-backs.

XPath is Regex for the DOM

Location Strategies are determined by application structure. Ideally, every element has a consistent element ID that is unique to the page where it is located and the same between application invocations. A lot of applications however, do not meet this criteria. Although there are a number of different location strategies supported by WebDriver, the most powerful by far is using XPath.

Tools such as XPather for Firefox allow you to select elements via XPath, but unfortunately use only positional element expressions (such as table[1]/tr[3]/td[2]). Not only are these expressions difficult to read, but they are heavily reliant on the ordering of the DOM. This makes them brittle.

What is need is a way to specify DOM path expressions which are rooted in the application's vernacular and disambiguate Elements effectively. By leveraging the expressiveness of XPath, you have the ability to specify Elements relative to other Elements. This is useful when labels are distinct from components they decorate. In addition, the annotation method of supplying locators for Elements in WebDriver precludes the ability to use templates to find Elements of a give type which vary only by their identifying characteristic. XPath allows the use of simple String templates. This allows for parameterized Element locators.

The expressiveness of XPath comes at a price. Generally, using XPath for element location is slower than other methods. In addition, not all browsers support XPath natively (Internet Explorer for example). That being said, XPath provides a strategy for taking an existing application and making it amenable to automation. As the application evolves to support unique persistent IDs, these changes can be made globally at the Element abstraction.

For an excellent tutorial on XPath, use the documentation at ZVON.org

The Reusable Page

Essentially a Page is a container for Elements which are manipulated via automation. Ideally, Elements associated with a Page should be lazy-initialized on use. In addition, a Page serves as a navigational component. To test something, you need to know where it is and how to get to it. Relationships between pages which define navigational structure depend on the type of application you are trying to automate.

The Hierarchical Application

In a Hierarchical Application, each page is located only once in the navigational structure. This type of structure is amenable to programmatic page traversal. In addition, if the application constructs page hierarchies programmatically (as they should), this information can be extracted from the application and the Page relationships can be created via code generation. The Generation Gap pattern is particularly useful in this regard. C#'s Partial Classes in addition to the ability to nest Classes makes it well suited to solve this problem.

The Process Oriented Application

By far the more difficult to automate, the Process Oriented Application has no clear notion of location; Pages relationships are defined in the context of a given process. A wizard-based application is the stereotypical Process Oriented Application. This type of application is not well suited to programmatic page traversal simply due the fact that Pages may have circular dependencies. In this case, it may be difficult to automate the creation of Page relationship.

The Case Against WebDriver.PageFactory

PageFactory relies on defaults for the Element lookup strategy that may not be appropriate for the AUT. The use of @FindBy annotation also makes it difficult to create dynamic Element lookups which are parameterized. The modification of annotations requires the use of reflection which is both cumbersome and expensive. In addition, it is questionable whether caching WebElements via @CacheLookup is useful given the possibility of StaleElementExceptions.

Instead of the PageFactory, Pages should express their dependencies explicitly in their constructors and hold a lazily-initialized dictionary of Elements with keys based on the language present in the application. If used in combination with the Element abstraction described previously, Element initialization is delegated back to the Element when accessed, not to Page. The Page's element dictionary provides a mechanism for finding Elements; nothing more. Pages constructed in this manner can be invoked directly or through the use of a dependency injector.

Putting it All Together

The Case For the Use of Fluent Interfaces

From a programming perspective, It is useful to think of the automation framework as serving different clients. There will be programmers responsible for wiring up the framework to the application as opposed to those responsible for wiring up tests to the framework.

These are two different tasks whose difficulty can be mitigated through the use of Fluent Interfaces. Page and Element definitions clearly express their requirements. Page navigation and Element access read more descriptively instead of a series of programmatic operations on application component primitives.

Degrees of Freedom

In the words of Einstein:

"Everything should be made as simple as possible, but not simpler."

All of these abstractions are not designed to introduce unnecessary complexity, but to manage the inherent complexity of automating an application. Application testing must be able to respond to various degrees of freedom which have the ability to destabilize test outcomes. The ultimate goal is reproducibility of the test's intent in the face of change. The following are the different changes which a framework must be resilient to.

The Application Changes : An Element is added/modified/removed

To add an element for automation simple requires associating it to a given Page. When an element is modified (such as when it is superseded with a new control with more advanced functionality), it need only be changed in a single location. Changes cascade throughout the entire framework with little work.

The Application Changes : A Page is added/modified/removed

If you programmatically determine page relationships, then simply running the code-generation component will create a stub for the new page or remove associated references to a deleted page. Most applications however undergo evolution more often; pages are modified. Elements are added/substracted from pages; this should happen independent of Element evolution. Pages should simply bind Elements that are part of their scope of responsibility.

The Automation Framework Changes

While the solution presented here hinges on the use of WebDriver, there is a case to be made for for framework isolation. All software evolves, and WebDriver is no exception. An automation framework built on WebDriver should also isolate changes to WebDriver itself. Leaking implementation details into tests by directly referencing WebDriver primitives results in fragility when the WebDriver API changes. Ideally, Page or Element bindings should not be directly impacted by changes to the framework itself.

The concepts of interface inheritance and implementation delegation to wrap primitive framework calls works well to isolate the automation framework from WebDriver changes. In essence, the Element object behaves much like a WebDriver WebElement without exposing any internal implementation details. This allows extension of the original WebDriver API with custom helper methods/interfaces.

The Target Web Browsers Change

There's a good chance that at some point, you will have to test you application against different browsers. To prepare for this eventuality, tests should be created in a web browser agnostic fashion. No test should depend on a specific browser; all automation operations should be done through the RemoteWebDriver/WebElement. By doing so, not only will you be able to run your tests against other browsers, but you will also be able to accommodate future browser updates as support for them is added to WebDriver.

What's Next?

Despite having the ability to automate testing, it is infeasible to test everything. Not all tests are equal. The most valuable tests reflect actual application usage. This is the role of specification testing. In the next article, I'll talk about how to use JBehave to fill this role.

What Are We Testing?

To be clear, the focus of this article is to discuss tests which involve various client web browsers which connect to the application. There are other kinds of tests that we will not cover here - unit tests and integration tests (at the service level). There are other tools which are suitable for those tests and they fall under the responsibility of the application developers themselves. More importantly, these types of tests are non-client facing.

What is important however, is that tests cover the behavior of different versions and kinds of web browsers. While WebDriver supports HTMLUnit via the WebDriver interface, we focus primarily on browsers supported by the RemoteWebDriver interface. This includes the following browsers:

• Firefox
• Chrome
• Internet Explorer
• Opera

The only unsupported major browser is Safari. Writing tests against the RemoteWebDriver allows us to keep them browser agnostic. In addition, Grid 2 currently supports only RemoteWebDriver clients.

Everyone seems to have a different definition of for particular kinds of tests. For the purpose of this discussion, I'd like to offer my own definitions; others however, will likely disagree with these categorizations.

Acceptance Tests

All tests which ensure that the application behaves as expected can be considered Acceptance Tests. Within this categorization are other sub-categories.

Smoke Tests

Smoke Tests ensure that the application is technically functional. Generally, these type of tests ensure navigational and Element behaviors are consistent through the application. Validation at the Element level could be done programmatically as part of these tests. Smoke Tests are capable of verifying technical requirements, but generally at the expense of providing user context to the operations. These types of tests are useful for programmers, but not so much for end users.

Specification Tests

Specification Tests use the language of the user domain to express application behavior. The value of using specifications is that these tests create a consistent vocabulary which is used to communicate end-user expectations to programmers. Essentially, these specifications form the foundation of acceptance criteria which is codified in a readable manner. There are many different specification frameworks, but I choose JBehave for the simple fact that specifications are written in plain text. This makes it easy for non-technical endusers to contribute to the creation of specifications. Features as well as bugs can be described via Stories and Scenarios.

Regression Tests

Regression Tests typically express defects within the application to which fixes have been applied. The difficulty with this type of test is usually a question about with reproducibility. By using the same language used for Specification Tests, preconditions, steps to simulate the behavior, and expected outcomes are clearly expressed.

Performance Tests

Performance Tests are generally designed to exert load on the application back-end. There are tools such as JMeter which apply the record-playback principle for generating load, but tools such as this have limitations. If your application uses dynamically generated assets, these types of tests are difficult to use. In addition, it is questionable whether tests based on JMeter measure actual application behavior against a client or the ability of the application server to respond to concurrent requests.

Synthetic browser tools remove a critical part of the equation - the browser itself. While HTMLUnit can be used in this scenario as well, it is not a browser that end users are likely to use. Different browser introduce variability (such as different javascript engines) that must be accounted for. This is more important for applications that are AJAX aware.

One and the Same

Code reuse is a fundamental principle in managing complexity in large software projects. As test coverage expands to cover more aspects of the AUT, leveraging this principle becomes crucial to manage the tests themselves.

Ideally, Specification and Regression Tests can be re-used for measuring performance. This allows for load testing using a single framework. This can be accomplished through the use of a concurrency demultiplexer that can take single Specification or Regression Tests and dynamically create multiple concurrent instances (determined at runtime) of the test to run. JUnit natively supports concurrency through the RunnerScheduler. You can effectively simulate group tests using this method.

This solution however, requires a number of machines to run the tests. Virtualization can be used to provision client browser machines, but has it's own costs. One must balance the costs of machine resources with human resources.

Domain-specific Languages (DSL) To The Rescue

Behavior Driven Development engages end-users in defining application behavior. Instead of simple technical requirements driving application development, the perspective of the enduser is given a central role. Specifications are written using the language of their domain; this provides the necessary context for considering the success or failure of a behavior. This is the role of the Domain-specific Language; JBehave is particularly useful in this regard.

JBehave allows the following:

1. Specifications can be written in plain text.
2. Specifications use a structured syntax that clearly expresses pre-conditions, steps, and expected outcomes.
3. Specification encourage stories to be written in the point-of-view of a User of an application. This allows allows for specifications to be defined for different perspectives given a User's role within an application.
4. Instead of tests expressing operations at the component (Element) level, instead they describe functionality in the context of individual user operations (Scenarios) grouped by functionality (Stories).
5. Stories are reusable a pre-conditions of other stories.
6. Parameterized steps allow for table-driven input. This effectively removes the need for tools like Fitnesse. Both input and output can be specified with the test.

Caveat Emptor

While JBehave is generally useful for BDD, it is not without its faults. My initial attempt to use the GuiceAnnotatedEmbedderRunner was a complete failure. The documentation surrounding its use and the associated documentation where insufficient to diagnose my issues. Barring a source-code evaluation of JBehave's implementation of this feature, I would recommend avoiding it.

In addition, I would suggest avoiding the JBehave-Selenium module. There are design flaws in its implementation which tightly couple test code to WebDriver. Pages created in the manner suggested by JBehave-Selenium inherit directly from WebDriver thus breaking encapsulation. Changes in WebDriver APIs have a direct impact on the Page implementation. Using Dependency injection via an injector framework, or manually via constructor injection is the only requirement for the Page. As per the Gang of Four, favor composition over inheritance.

In addition, having a typed WebDriverProvider exposes the browser implementation details into the test framework; this should be safely abstracted away into a Configuration object.

The Grid

Grid allows tests to distributed across machines for parallel execution. This is particularly useful for load testing as well as providing concurrent testing against various browser types and versions. When combined with different operating systems, this method provides an effective means of testing an application across a number of different platforms simultaneously. A single virtual machine can have multiple grid nodes which can consist of different browser types. The number of nodes per machine only limited by memory. By having a central repository of available browser resources, scheduling concurrent test execution becomes manageable.

The structure of the automation environment plays a direct part in how Grid is used. I'll cover this in more detail in Part 4.

What's Next?

The number of tests against a mature application only increase over time. This requires a method for organizing the tests themselves. The next article deals with issues of test organization and how to introduce modularity and versioning to your tests.

Modularity

While tests should be comprehensive, only the test automation usually is required to run the complete suite of tests; test and application developers generally work on a subset of the entire framework. This is more important when test runs become time consuming. Immediate feedback encourages the use of tests; delays in receiving test results interrupt workflow and make it less likely that will used concurrently with development. Ideally, testing can be initiated from within the IDE (Eclipse in this case) or the command line (which ever is preferable to the developer). Being able to support both modes of execution fulfills the different requirements of developers versus automation.

The design of the test execution encourages the use of java Properties; these can have local overrides which are used for development, but can be overridden when executed in the context of automation. This article focuses on scriptable execution via Ant; other build tools such as Maven can also be used.

Test configuration parameters that should be made into Properties include:

• application URL
• username & password
• browser type

Reference Datasources

In addition to having modular tests, a reference database is essential. A reference database should provide only enough functionality to verify application behavior. This allows separating out configuration dependent tests from core functionality. By being able to differentiate between core and configuration, you have better separation of concerns. Failures in actual application behavior can be distinguished for misconfiguration errors. In addition, for test behavior to be replicable, the same test pre-conditions should be present for every test execution run. Using VMWare snapshot functionality or Oracle's FlashBack capability allows you to ensure that your database is in a consistent known state when tests are run.

Test Organization

The organizational mechanism employed depends on the tools you use for test execution. Below, I discuss the use of JUnit and JBehave mechanisms for organizing tests into workable modules. In addition to using the tool's organizational support, it is also helpful to use package namespaces to modularize the physical test files.

JUnit

JUnit has the @RunWith(Suite.class) and @Suite.SuiteClasses class annotations which allow you to create a hierarchy of suites with one master suite (used by automation) to include other functionally distinct suites. Within these functional suites, I suggest separating out Reversible and Non-Reversible tests. Reversible tests do not make permanent database changes, while NonReversible tests do. Application developers can use Reversible suite for quick smoke tests. Ideally, you will the restore the reference database to its initial state after the execution of NonReversible tests. This ensures consistency between test runs. This is even more important if you are testing outputs based on fixed/known inputs and want to compare it with an expected output.

JBehave

JBehave's organization is based on the concept of a Story. A Story is a functionally cohesive set of scenarios. Scenarios are the actual behaviors which describe the story's functionality. In addition to using Stories, it is useful to also modularize the Stories themselves along functional application lines. Step classes and story files are required to run JBehave tests. Story files are simple text specifications which include everything necessary to describe the expected behavior given a set of pre-conditions, operational steps, and expected post-conditions. Step classes are responsible for mapping the scenario language to actual test automation commands. Both Steps and Stories can be reused. I suggest keeping them in separate directories. JBehave uses reflection to determine which stores to run via the storyPaths() method on JUnitStories. This uses an Ant-like path expression to find textual stories which JBehave inspects to invoke test methods. Using more specific path expressions in the various functional specific stories limits test execution. Conversely, a single JUnitStories class with a sufficiently general path expression should be able to find all stories in your project.

Versioning

Tests should be considered programmatic artifacts; like application code, they should be managed via a source code repository. Treating tests as code allows us to do all the normal things version control allows. Test file histories document the changes required as the application evolves. In addition, it is useful to be able to version test collections in relation to the application releases/branches. This allows for test development to proceed unhindered on development branches while supporting writing new test cases against the stable release (with the understanding that tests applicable to the development branch will be ported).

Test Context

Inevitably, you will have to reconstruct context to understand the reasons behind why a given test was written a certain way. Files that you used to troubleshoot the problem domain must be found again and inspected to fully understand the implications of a change. When this reconstruction happens is indeterminate; it might be tomorrow or a year from now. Unfortunately, your ability to reconstruct this context diminishes over time. In addition, there is a (high) probability that the person who may need to make sense of this context may be someone other than yourself.

What is need is a mechanism to associated this context with the test. If you are using Eclipse, Mylyn's saved contexts in conjunction with the appropriate adapter for your defect system (such as Redmine/Trac/Bugzilla) solves this problem. By having the ability to associate a context with a defect (via a zipped file which contains your project layout), you remove the limitation of an individual's memory and instead leverage the automation system's abilities.

What's Next?

Now that I've established some basic principles for creating a test automation framework, I'll describe how to put it all together to support continuous deployment using Jenkins/Ant/VMWare. The goal is to provide a feedback system for application development. The sooner developers are aware of potential regressions caused by application changes, the quicker they can respond to fixing it.

Continuous Deployment

Human error is fundamental to the endeavor of creating software. Our ultimate aim to not to completely remove error (an impossible task), but to be aware when errors are introduced. This awareness is what makes for better software, not simply automation itself. There are a large classification of behaviors where the computer does not know or care which behavior is correct. Being able to use relative judgement is still a characteristically human trait.

Automation is the 80/20 solution (give or take a few percent). It should cover a majority of the test cases, but depending on the complexity of the application, automation does not generally encompass all possible tests. What it does is remove the tedium from tests which are amenable to automation. This allows manual tests (which should still be done) to focus on the really hard 20 percent where no amount of automation (barring artificial intelligence) is sufficient to determine the correctness of behavior.

There are few key components to creating the automation environment - Continuous Integration, Virtualization, and Scripting. Each is discussed in detail.

Jenkins/Hudson

Continuous Integration is the cornerstone of any automation effort. If you have not integrated CI into your development practices, stop and invest the time to do so. Without CI, you do not have a means of coordinating the efforts of the entire organization to improve the quality of the produced software.

The original focus of CI was on developers. By generalizing this principle to the entire application stack, we move towards Continuous Deployment. Not only are applications tested (via unit / integration tests for code level validation) when built, but the built artifacts themselves are deployed without user intervention to run complete end-to-end tests. This type of testing ensures that the software will work when fully integrated and deployed, not simply tested in isolation.

Automated acceptance tests should be run when either a new version of the application is available or when tests against the application have changed. Test execution is coordinated through a dependent project which is aware of either application or test project changes. I strongly suggest to separate out executing end-to-end automation from the actual acceptance test framework. Should either change, it has minimal impact on the other.

The role of Jenkins in the Automation Environment is the that of the Coordinator. Built artifacts for both the application and test clients are moved to a Staging Area in the environment. The Staging Area contains assets for the client machines (which include the target web browsers), the application servers, and the database. In addition, it can contain the scripts which coordinate actions between the various nodes of the automation environment.

Caveat

Jenkins currently does not allow projects to reference other workspaces. In some cases (such as with the application artifacts), the copy operation may be coupled with the application project. Depending on the size of your published artifacts, this may add considerable time to the build process. Also non-build failures, such as the inability to push artifacts into the test environment result in build warnings.

VMWare

There are generally three classes of machine in a 3-tier application environment:

• Clients (consisting of different target OS/Browser combinations
• Application Servers (consisting of the supported OS/Application environments
• Database Servers (consisting of supported back-ends)

VMWare (as well as other virtualization platforms) enables having the entire application environment in a self-contained network. End-to-end testing can then use this environment to test against different combinations of the application stack. Tests which succeed in all combinations validate consistent behavior across the supported architectures.

The elegance of this solution is that it can easily scaled by simply adding more VMs. Hardware is typically the constraining factor in virtualization solutions which are highly dependent on memory and I/O. Modern multi-core processors are more than sufficient from the point of view of a client VM. CPU affinity and proper resource allocation can ensure that VMs are sufficiently provisioned for their role. There are few principles to follow when designing your virtual infrastructure:

1. Ideally, you have enough physical RAM to hold all VMs in memory without swapping.
2. Drives which have fast IOPs reduce disk contention (important for Windows VMs which like to swap)
3. CPU allocation should not exceed actual number of Cores/Threads

In addition, VMWare supports virtual snapshots.Snapshots allow for VMs to be restored consistently to a known state. This feature removes the need to physically start machines and wait for boot up. VMs are started and ready to run immediately. In addition, tests which make non-reversible changes to the database layer can have their changes undone by simply restoring the VM to its original snapshot. This ensures that tests can run against a known baseline environment. Variation between test runs is minimized, if not entirely eliminated.

Operations against VMs (such as starting and restoring snapshots) is supported by VMWare through the use of the vmrun utility. In addition, vmrun also allows commands to be executed within the guest VMs from the virtualization host. Scripted test execution from and external trigger is possible when combined with shell/Ant scripts on guest VMs. If these scripts are available on a common Staging Area as suggested in the previous section, all dependent guest VMs can use the same scripts between test executions.

In addition, vmrun can be used against snapshots. If testing against the same OS with slight variations, you can save space by taking snapshots of the core image with the variations applied. For example if you are testing Windows clients with various Service Packs, you can take a snapshot against each Service Pack state you wish to test against. When invoking a test, all you need to do is specify the appropriate snapshot.

Supplementary Services

Since the virtual environment is a full functional network, there are a few services which make managing VMs easier.

DNS/DHCP

Name resolution and network configuration can be unified using DHCP with dynamic DNS updates. This allows all machine IPs/names to be centralized to the dhcpd.conf file (assuming you are using a Unix/Linux host). The Workstation/Fusion/Server versions of VMWare natively support DNS/DHCP through their built-in virtual network interfaces for the NAT or Host-only networking configuration. In ESXi, this ability is not present as there are are no virtual network interfaces; a dedicated host for managing this service is necessary.

It is strongly recommended that you avoid using static IPs for hosts as well as providing static host mapping files. This solution quickly becomes difficult to manage as you add more machines to the system as changes to the network have to be propagated to all hosts participating in the same network.

NTP

VMs are sensitive to time differences, specially if they attempt to synchronize their clocks against the host's internal clock. To avoid time drift between VMs, it is suggested that you use NTP to synchronize all VM clocks. By removing this source for variation, you can ensure that time-sensitive operations, such as establishing secure connections, happens reliably within the virtual environment. Time drift may affect secure connections as most security mechanisms protect against replay attacks by having a sufficiently small connection windows.

HTTPS

There may be times when you are required to simulate connections over HTTPS. To ensure that invalid certificate messages do not interfere with client tests, create a self-signed certificate for the application server. Import this self-signed certificate as a root certificate authority on the client VMs. Connections should be established without security messages.

Ant

Scripts are the glue that binds the Coordinator (Jenkins in this example) to the VMs that participate in test execution. Automation scripts should be treated as code; include them with your version control system. In this example we use Ant, but certain tasks may be better served with DevOps tools such as Chef / Puppet.

Ant should be structured with a number of custom targets corresponding to the various phases of test execution. To kick off test execution, the Jenkins run automation project is started which invokes an ant script at the root of the Staging Area on the VM host responsible for managing the participating guest VMs. The phases are as follows:

Pre-automation

1. Set the test build to use for the clients.
2. Set the app build to use for applications.
3. Execute application config pre-conditions such as setting configuration information.
4. Start the VMs in the order of their dependencies; usually the database first, followed by the application server, and lastly the client VMs being used. Ant allows for the use of the elements and . Generally, database and application starts occur sequentially, but client VMs can be started in parallel.

Test Execution

1. Execute tests on client VMs.
2. Collect test artifacts and results and publish them to a known location.

Post-automation

1. Suspend all VMs in parallel.
2. Revert the VMs to their base snapshots. This step ensures that the VMs are ready for the next test execution run

There were a number of Ant/Ant-Contrib tasks that I used to

• propertyregex - Performs regular expression operations on an input string, and sets the results to a property.
• timestampselector - The TimestampSelector task takes either a nested element, or a path reference, and sets either a named property, or a path instance to absolute pathnames of the files with either the N latest or earliest modification dates
• sshexec - Runs a command on a remote machine running SSH daemon.

Miscellanea

As I suggested previously, the use of single Staging Area simplifies Continuous Deployment. When testing against different supported application servers, it allows a single copy to be used for configuration purposes. Each application server variant configures itself against the same build. Keeping a recent history of published artifacts in the Staging Area also allows for test run comparisons. By resetting a symbolic link to select the current build to use, you have the ability to troubleshoot between versions of the application/test builds.

TODO : Grid and Automation in the Cloud

To fully leverage the Grid, tests need the ability to run in parallel. There are a number of strategies that can be employed to achieve this, but I'll defer to a future article on Grid usage. There are a few experiments that I'd like to try first before committing to any design. In addition, the use of Grid should be consistent whether you are running it from your own environment (as above) or in the Cloud (via SauceLabs).