Testing Systems Under Load By Jim Jenkins and Mike Burke | IQ Services

Today’s telecommunications and contact center infrastructures have become extremely complex.  They involve highly integrated combinations of hardware and software delivering business solutions to virtually every industry.  New technologies underlying these infrastructures are added every day, sometimes halfway around the world.  The design and reliability of these infrastructures is critical to delivering the level of customer service demanded by business-to-business and business to consumer business solutions.

In order to meet the strict Customer Satisfaction and ROI objectives associated with deploying complex infrastructures, companies must first know a business solution works as designed before it is put into production.  Testing systems under load is the only way to be confident a business system will work as designed.

What does it mean to test a system under load?  The simplest definition is to create a level of activity in a test environment that equals or exceeds the level of activity you expect to experience when actual users access and exercise the system. 
What do you expect to verify?  The objective is to demonstrate that the end user experience meets the business solution objectives at the designed level of activity.  The variables involved include system configuration/setup, number and types of access (i.e. network and long distance capacity, telephone lines, PBX capacity, IVR ports, Web server capacity, etc.)  router and server processing capacity, WAN and LAN bandwidth, database and host processing capacity, and more.  Verifying an entire business solution under load requires test scenarios that exercise the infrastructure from end to end, the way actual end users will really use it – from the outside in. 

Planning Load Test Activity

Set up is everything!  The establishment of well-defined test objectives is key to creating test scenarios that will result in the best demonstration of system performance under load.

Implementing Load Testing

There are many interesting issues and approaches to testing systems under load.  Over the years, we have learned several important things about effectively performing testing and evaluating the results.  One of the key lessons learned is the value of an automated, repeatable test methodology that provides clearly documented results, not just anecdotal reports of issues.  Another is having everyone who cares about the test results participate in the actual testing.  There is no substitute for real-time visibility of performance and issues.  Observing performance changes driven by turning the volume of activity up and down improves everyone’s understanding of how a business solution actually performs.  Issues can frequently be identified and corrected in minutes instead of days. 

One of the major challenges of both planning and implementing testing is scheduling.  Testing milestones in an implementation schedule certainly help to drive a project.  However it is important to keep the testing in perspective and not allow the testing to become an end in itself.  Systems need to be tested when they are ready for testing.  Load testing before appropriate unit level testing has been successfully completed and overall system setup has been properly verified can result in spending unnecessary dollars and chasing the wrong problems.  It is good practice to plan on two to three iterations of load testing before you expect to be complete.  The actual time required for load testing will depend on the complexity of the infrastructure and the test objectives.  Whenever possible, load testing should occur one to two weeks before cutover, to allow time for tweaks and re-testing, and should be performed on the actual production infrastructure to ensure all components have been integrated correctly.

Observing and Documenting Load Testing 

Simple, accurate and thorough documentation of test results that can be quickly accessed and analyzed on-line is a key element of effective load testing.  Recordings of test calls are invaluable.  Being able to listen to calls online immediately after they are completed adds major value to the process.  Recordings clearly document and illustrate issues so they can be addressed without having to attempt to replicate an issue based on anecdotal reports of unexpected system behavior.  It is amazing what a knowledgeable person who intimately understands the system infrastructure can determine from listening to a call that was not handled as expected.  The test data collected should be comprehensive and it should be possible to organize and present the data in the ways that best match the test objectives. Straightforward formal documentation of the test results provides confidence the system is ready for actual customers.

Benchmark and Regression Testing
Testing systems under load has uses beyond being ready for initial release to production. It is often valuable to document performance with load testing so as changes are made to the system, or when new systems are being installed, data will be available to make comparisons and document improvements to a business solution.  Periodic regression testing may also point out changes in performance or trends that proactively identify areas where action should or can be taken before the end user customer experience is negatively affected.  Regression load testing can also become part of documenting service level performance.

What can be learned by testing systems under load

Properly planned and implemented, a load test is a thorough system check-up. Load testing can answer questions such as: 

• Are all the pre-paid telco facilities provisioned? 
• Do rollover and hunting really work as expected? 
• Are FIFO queues still FIFO when they are maxed out? 
• What about converged/unified queues? 
• Are there enough speech recognition licenses available when all callers are uttering inputs rather than using DTMF? 
• Does host response time over the WAN degrade when load is maxed? 
• Does screen pop data arrive at the agent workstation at the same time as the call?
• Does screen pop data follow the proper call?  
• Does the faxback server send the information requested to the proper calling party?  
• Are VoIP calls intelligible at full load? 
• Are recordings for agent monitoring and coaching intelligible at full load?
• If the system is configured as a hot standby, can it handle a sharp wave of incoming traffic that fills all channels in 10 seconds?

Testing systems under load is a key requirement of meeting customer satisfaction and ROI objectives.  The consequences of not performing this testing can be disastrous to your business.  Proper planning and implementation will result in cost effective testing that gives you confidence your end users are having the experience you intend.

Testing systems under load at all phases of the business solution lifecycle

The following chart provides more examples of how load testing can fit into the life cycle of a business solution.

Back