Case Study: Test Automation

Client: A leading provider of financial analytic application software

Application: Financial reporting product that interfaces with a large number of general ledger (GL) applications.

Issues: The client was manually testing each new release of the product to ensure that GL data was loading successfully. This manual test cycle was incredibly time-consuming, taking about nine months to complete. Compounding the problem, the team often had to test multiple releases (old and new) of partner GL products during the nine-month process. Finally, manual testing produced limited metrics to measure functional coverage or to support vendor certification, so it was difficult to communicate the results of this effort to partners or customers.

ProtoTest’s Solution: First, we worked with the manual test team to document and refine the QA process, adding metrics to support better decision-making. The client test team and QA manager attended ProtoTest's "Fundamentals Of Software Testing" course to ensure a baseline knowledge of QA process and procedures. Our consultants provided ongoing mentoring as the client team implemented our recommendations for changes to test planning and test case and procedures documentation. Next, we conducted a proof-of-concept and pilot for test automation to eventually replace most of the manual interface testing. Our goals were to:

• Establish a testing environment to support all automation needs
• Prove that a leading commercial tool would be able to successfully automate the tests
• Demonstrate the benefits of a leading commercial test management tool in the environment

A six month follow-on implementation project began in September 2003. Technologies selected
include QuickTest Pro and TestDirector from Mercury Interactive.

Results:

• Dramatic time savings. The old manual effort relied on a spreadsheet-based comparison of expected and actual results from a partner application data load. It was estimated that it would take 313 hours to verify every spreadsheet cell and perform a quick analysis of the errors. Due to time constraints -- each interface testing cycle is limited to 10 person-days -- the old manual testing process compared only a subset of the cells, leading to the potential for over-looked errors. The automated scripts that ProtoTest developed using QuickTest Pro perform this comparison, on all cells, in approximately 7 hours -- a 98% savings in time.
• Significant cost savings. Using an estimated loaded cost per tester of $30/hour, the automation is providing a savings of more than $9,000 every time the process is run. While the test executes, the tester is free to do something that can't be automated, or to perform expert analysis of test results.
• More complete results. Using automation means that all expected vs. actual results can be compared instead of "selective" comparison under manual effort, which reduces risk.
• Better metrics leads to better decision-making. Using TestDirector to hold and manage all the automated tests and test results provides much better information to management about test coverage and product readiness, and the potential for improved customer and partner communication.