Continuous Deployment

Human error is fundamental to the endeavor of creating software. Our ultimate aim to not to completely remove error (an impossible task), but to be aware when errors are introduced. This awareness is what makes for better software, not simply automation itself. There are a large classification of behaviors where the computer does not know or care which behavior is correct. Being able to use relative judgement is still a characteristically human trait.

Automation is the 80/20 solution (give or take a few percent). It should cover a majority of the test cases, but depending on the complexity of the application, automation does not generally encompass all possible tests. What it does is remove the tedium from tests which are amenable to automation. This allows manual tests (which should still be done) to focus on the really hard 20 percent where no amount of automation (barring artificial intelligence) is sufficient to determine the correctness of behavior.

There are few key components to creating the automation environment - Continuous Integration, Virtualization, and Scripting. Each is discussed in detail.

Jenkins/Hudson

Continuous Integration is the cornerstone of any automation effort. If you have not integrated CI into your development practices, stop and invest the time to do so. Without CI, you do not have a means of coordinating the efforts of the entire organization to improve the quality of the produced software.

The original focus of CI was on developers. By generalizing this principle to the entire application stack, we move towards Continuous Deployment. Not only are applications tested (via unit / integration tests for code level validation) when built, but the built artifacts themselves are deployed without user intervention to run complete end-to-end tests. This type of testing ensures that the software will work when fully integrated and deployed, not simply tested in isolation.

Automated acceptance tests should be run when either a new version of the application is available or when tests against the application have changed. Test execution is coordinated through a dependent project which is aware of either application or test project changes. I strongly suggest to separate out executing end-to-end automation from the actual acceptance test framework. Should either change, it has minimal impact on the other.

The role of Jenkins in the Automation Environment is the that of the Coordinator. Built artifacts for both the application and test clients are moved to a Staging Area in the environment. The Staging Area contains assets for the client machines (which include the target web browsers), the application servers, and the database. In addition, it can contain the scripts which coordinate actions between the various nodes of the automation environment.

Caveat

Jenkins currently does not allow projects to reference other workspaces. In some cases (such as with the application artifacts), the copy operation may be coupled with the application project. Depending on the size of your published artifacts, this may add considerable time to the build process. Also non-build failures, such as the inability to push artifacts into the test environment result in build warnings.

VMWare

There are generally three classes of machine in a 3-tier application environment:

• Clients (consisting of different target OS/Browser combinations
• Application Servers (consisting of the supported OS/Application environments
• Database Servers (consisting of supported back-ends)

VMWare (as well as other virtualization platforms) enables having the entire application environment in a self-contained network. End-to-end testing can then use this environment to test against different combinations of the application stack. Tests which succeed in all combinations validate consistent behavior across the supported architectures.

The elegance of this solution is that it can easily scaled by simply adding more VMs. Hardware is typically the constraining factor in virtualization solutions which are highly dependent on memory and I/O. Modern multi-core processors are more than sufficient from the point of view of a client VM. CPU affinity and proper resource allocation can ensure that VMs are sufficiently provisioned for their role. There are few principles to follow when designing your virtual infrastructure:

1. Ideally, you have enough physical RAM to hold all VMs in memory without swapping.
2. Drives which have fast IOPs reduce disk contention (important for Windows VMs which like to swap)
3. CPU allocation should not exceed actual number of Cores/Threads

In addition, VMWare supports virtual snapshots.Snapshots allow for VMs to be restored consistently to a known state. This feature removes the need to physically start machines and wait for boot up. VMs are started and ready to run immediately. In addition, tests which make non-reversible changes to the database layer can have their changes undone by simply restoring the VM to its original snapshot. This ensures that tests can run against a known baseline environment. Variation between test runs is minimized, if not entirely eliminated.

Operations against VMs (such as starting and restoring snapshots) is supported by VMWare through the use of the vmrun utility. In addition, vmrun also allows commands to be executed within the guest VMs from the virtualization host. Scripted test execution from and external trigger is possible when combined with shell/Ant scripts on guest VMs. If these scripts are available on a common Staging Area as suggested in the previous section, all dependent guest VMs can use the same scripts between test executions.

In addition, vmrun can be used against snapshots. If testing against the same OS with slight variations, you can save space by taking snapshots of the core image with the variations applied. For example if you are testing Windows clients with various Service Packs, you can take a snapshot against each Service Pack state you wish to test against. When invoking a test, all you need to do is specify the appropriate snapshot.

Supplementary Services

Since the virtual environment is a full functional network, there are a few services which make managing VMs easier.

DNS/DHCP

Name resolution and network configuration can be unified using DHCP with dynamic DNS updates. This allows all machine IPs/names to be centralized to the dhcpd.conf file (assuming you are using a Unix/Linux host). The Workstation/Fusion/Server versions of VMWare natively support DNS/DHCP through their built-in virtual network interfaces for the NAT or Host-only networking configuration. In ESXi, this ability is not present as there are are no virtual network interfaces; a dedicated host for managing this service is necessary.

It is strongly recommended that you avoid using static IPs for hosts as well as providing static host mapping files. This solution quickly becomes difficult to manage as you add more machines to the system as changes to the network have to be propagated to all hosts participating in the same network.

NTP

VMs are sensitive to time differences, specially if they attempt to synchronize their clocks against the host's internal clock. To avoid time drift between VMs, it is suggested that you use NTP to synchronize all VM clocks. By removing this source for variation, you can ensure that time-sensitive operations, such as establishing secure connections, happens reliably within the virtual environment. Time drift may affect secure connections as most security mechanisms protect against replay attacks by having a sufficiently small connection windows.

HTTPS

There may be times when you are required to simulate connections over HTTPS. To ensure that invalid certificate messages do not interfere with client tests, create a self-signed certificate for the application server. Import this self-signed certificate as a root certificate authority on the client VMs. Connections should be established without security messages.

Ant

Scripts are the glue that binds the Coordinator (Jenkins in this example) to the VMs that participate in test execution. Automation scripts should be treated as code; include them with your version control system. In this example we use Ant, but certain tasks may be better served with DevOps tools such as Chef / Puppet.

Ant should be structured with a number of custom targets corresponding to the various phases of test execution. To kick off test execution, the Jenkins run automation project is started which invokes an ant script at the root of the Staging Area on the VM host responsible for managing the participating guest VMs. The phases are as follows:

Pre-automation

1. Set the test build to use for the clients.
2. Set the app build to use for applications.
3. Execute application config pre-conditions such as setting configuration information.
4. Start the VMs in the order of their dependencies; usually the database first, followed by the application server, and lastly the client VMs being used. Ant allows for the use of the elements and . Generally, database and application starts occur sequentially, but client VMs can be started in parallel.

Test Execution

1. Execute tests on client VMs.
2. Collect test artifacts and results and publish them to a known location.

Post-automation

1. Suspend all VMs in parallel.
2. Revert the VMs to their base snapshots. This step ensures that the VMs are ready for the next test execution run

There were a number of Ant/Ant-Contrib tasks that I used to

• propertyregex - Performs regular expression operations on an input string, and sets the results to a property.
• timestampselector - The TimestampSelector task takes either a nested element, or a path reference, and sets either a named property, or a path instance to absolute pathnames of the files with either the N latest or earliest modification dates
• sshexec - Runs a command on a remote machine running SSH daemon.

Miscellanea

As I suggested previously, the use of single Staging Area simplifies Continuous Deployment. When testing against different supported application servers, it allows a single copy to be used for configuration purposes. Each application server variant configures itself against the same build. Keeping a recent history of published artifacts in the Staging Area also allows for test run comparisons. By resetting a symbolic link to select the current build to use, you have the ability to troubleshoot between versions of the application/test builds.

TODO : Grid and Automation in the Cloud

To fully leverage the Grid, tests need the ability to run in parallel. There are a number of strategies that can be employed to achieve this, but I'll defer to a future article on Grid usage. There are a few experiments that I'd like to try first before committing to any design. In addition, the use of Grid should be consistent whether you are running it from your own environment (as above) or in the Cloud (via SauceLabs).