Summer 2007 Newsletter: Contact Center Load Testing
Originally published June 21, 2007

Letter from the Publisher
by Jim Jenkins

Testing Systems Under Load:
by Jim Jenkins and Mike Burke

Fermentations
by Chuck Blethen
Evaluating a Wine - Part 1: Appearance (Robe & Legs)

Quality Testing Service Overview
A quick reference guide of our services

Letter from the Publisher

Summer 2007

The evolution of contact center and communications technology is continuing at a fast and challenging pace in 2007. Business solutions that optimize the customer experience are built on hardware and software infrastructures that are highly integrated.  In order to achieve the required service level and provide the desired customer experience, businesses must demand efficiency and reliability in both the design and performance of these solutions. Businesses that do not deliver the best possible customer experience fail to meet ROI and related business objectives.

Since 1996, IQ Services has demonstrated that customer experience testing is a key element of managing risk and achieving ROI and project objectives related to communication and contact center business solutions. In 2007, we continue to expand our offerings and capabilities by emphasizing the importance of proactive testing throughout the business solution life cycle.  We have rounded out our comprehensive quality assurance testing packages to include everything from risk assessment and test planning to post deployment availability monitoring and regression testing.  We also continue to deliver the same flexible, standalone-testing services that have been so valuable to businesses over the past 11 years.  IQ Services’ Quality Testing Service Overview is included with this newsletter for your reference.

In this issue of the newsletter we have also included a white paper on Testing Systems Under Load that will hopefully provide you with some valuable ideas related to testing your communication and contact center business solutions.

Thank you, as always, for reading our publication.

Sincerely,

Jim Jenkins
Publisher
President & CEO

Back to Top

Testing Systems Under Load:

Today’s telecommunications and contact center infrastructures have become extremely complex.  They involve highly integrated combinations of hardware and software delivering business solutions to virtually every industry.  New technologies underlying these infrastructures are added every day, sometimes halfway around the world.  The design and reliability of these infrastructures is critical to delivering the level of customer service demanded by business-to-business and business to consumer business solutions.

In order to meet the strict Customer Satisfaction and ROI objectives associated with deploying complex infrastructures, companies must first know a business solution works as designed before it is put into production.  Testing systems under load is the only way to be confident a business system will work as designed.

What does it mean to test a system under load?  The simplest definition is to create a level of activity in a test environment that equals or exceeds the level of activity you expect to experience when actual users access and exercise the system. 

What do you expect to verify?  The objective is to demonstrate that the end user experience meets the business solution objectives at the designed level of activity.  The variables involved include system configuration/setup, number and types of access (i.e. network and long distance capacity, telephone lines, PBX capacity, IVR ports, Web server capacity, etc.)  router and server processing capacity, WAN and LAN bandwidth, database and host processing capacity, and more.  Verifying an entire business solution under load requires test scenarios that exercise the infrastructure from end to end, the way actual end users will really use it – from the outside in. 

Planning Load Test Activity

Set up is everything!  The establishment of well-defined test objectives is key to creating test scenarios that will result in the best demonstration of system performance under load.

Implementing Load Testing

There are many interesting issues and approaches to testing systems under load.  Over the years, we have learned several important things about effectively performing testing and evaluating the results.  One of the key lessons learned is the value of an automated, repeatable test methodology that provides clearly documented results, not just anecdotal reports of issues.  Another is having everyone who cares about the test results participate in the actual testing.  There is no substitute for real-time visibility of performance and issues.  Observing performance changes driven by turning the volume of activity up and down improves everyone’s understanding of how a business solution actually performs.  Issues can frequently be identified and corrected in minutes instead of days. 

One of the major challenges of both planning and implementing testing is scheduling.  Testing milestones in an implementation schedule certainly help to drive a project.  However it is important to keep the testing in perspective and not allow the testing to become an end in itself.  Systems need to be tested when they are ready for testing.  Load testing before appropriate unit level testing has been successfully completed and overall system setup has been properly verified can result in spending unnecessary dollars and chasing the wrong problems.  It is good practice to plan on two to three iterations of load testing before you expect to be complete.  The actual time required for load testing will depend on the complexity of the infrastructure and the test objectives.  Whenever possible, load testing should occur one to two weeks before cutover, to allow time for tweaks and re-testing, and should be performed on the actual production infrastructure to ensure all components have been integrated correctly.

Observing and Documenting Load Testing 

Simple, accurate and thorough documentation of test results that can be quickly accessed and analyzed on-line is a key element of effective load testing.  Recordings of test calls are invaluable.  Being able to listen to calls online immediately after they are completed adds major value to the process.  Recordings clearly document and illustrate issues so they can be addressed without having to attempt to replicate an issue based on anecdotal reports of unexpected system behavior.  It is amazing what a knowledgeable person who intimately understands the system infrastructure can determine from listening to a call that was not handled as expected.  The test data collected should be comprehensive and it should be possible to organize and present the data in the ways that best match the test objectives. Straightforward formal documentation of the test results provides confidence the system is ready for actual customers.

Benchmark and Regression Testing
Testing systems under load has uses beyond being ready for initial release to production. It is often valuable to document performance with load testing so as changes are made to the system, or when new systems are being installed, data will be available to make comparisons and document improvements to a business solution.  Periodic regression testing may also point out changes in performance or trends that proactively identify areas where action should or can be taken before the end user customer experience is negatively affected.  Regression load testing can also become part of documenting service level performance.

What can be learned by testing systems under load

Properly planned and implemented, a load test is a thorough system check-up. Load testing can answer questions such as: 

• Are all the pre-paid telco facilities provisioned? 
• Do rollover and hunting really work as expected? 
• Are FIFO queues still FIFO when they are maxed out? 
• What about converged/unified queues? 
• Are there enough speech recognition licenses available when all callers are uttering inputs rather than using DTMF? 
• Does host response time over the WAN degrade when load is maxed? 
• Does screen pop data arrive at the agent workstation at the same time as the call?
• Does screen pop data follow the proper call?  
• Does the faxback server send the information requested to the proper calling party?  
• Are VoIP calls intelligible at full load? 
• Are recordings for agent monitoring and coaching intelligible at full load?
• If the system is configured as a hot standby, can it handle a sharp wave of incoming traffic that fills all channels in 10 seconds?

Testing systems under load is a key requirement of meeting customer satisfaction and ROI objectives.  The consequences of not performing this testing can be disastrous to your business.  Proper planning and implementation will result in cost effective testing that gives you confidence your end users are having the experience you intend.

Testing systems under load at all phases of the business solution lifecycle

The following chart provides more examples of how load testing can fit into the life cycle of a business solution.

Back to Top

Fermintations

Note: to make reading this article more enjoyable and instructive, you may wish to follow along with a glass of wine in hand!

In the world of wine tasting, there are four key elements involved in the evaluation of any wine: appearance (i.e., robe or color and legs), nose, taste, and finish of the wine.  The four evaluation steps of wine tasting, which focus on these four key elements, are:  Appearance (Color & Legs), Swirl, Smell, and Taste. Today, let’s discuss the first two steps – Appearance and Swirl.

Appearance
You can tell much about a wine simply by studying its appearance. The wine should be poured into a clear glass and held in front of a white background (a tablecloth or piece of paper will serve nicely) so that you can examine the robe or color.  Tilt your glass of wine over a white surface without spilling any of the contents and observe the color of the wine. The color of wine varies tremendously, even within the same type of wine.

You can guess the age of a red wine by observing its "rim" or rim color.  Again, tilt the glass slightly and look at the edge of the wine.  A purple tint may indicate youth while orange to brown tint indicates maturity. The range of colors that you may see depends, of course, on whether you are evaluating a white or red wine. Here are the appearance factors that you should observe:
           
COLOR DEPTH:        
Watery, pale, medium, deep, dark

COLOR HUE:(youngest to oldest wine)
WHITE WINE: Greenish, yellow, straw yellow, gold, amber
RED WINE: Purple-red, ruby, garnet, brick, brown
ROSÈ WINE: Pink, Salmon, orange

CLARITY:      
Clear, Slight Haze, Dull, Cloudy

Swirl
Swirling the wine serves many purposes, but visually it allows you to observe the body of the wine. "Good legs" may indicate a thicker body and a higher alcohol content and/or sweetness level. For normal wines it is possible to guess the alcohol content by counting the number of “legs” or teardrops falling from the top of the rim of the swirl mark in your wine glass. Three to four tears per linear inch around the glass indicates the wine has about 11 percent alcohol.  Six to eight tears per inch are usually indicative of a 14 to 15 percent alcohol wine.  It is fun to swirl, observe, and guess the alcohol content based on legs before looking at the label on the bottle to confirm.  You can guess quite accurately with a little practice.  Note that a dirty wine glass will not allow the legs to form properly for this particular evaluation.  Wait staff will sometimes handle wine glasses by the bowl when setting a table and this will also hinder your ability to observe the legs. Sometimes, the legs will begin to form in small tight groups that re-combine into larger legs about ¼ inch below the rim of the swirled top edge of the wine.  In this case, use the count that is below this level when evaluating the alcohol level of a wine. The swirl evaluation step is important in that it allows the bowl to fill with the vapor of the wine so that you can judge the nose (aroma)… that is coming up in the next article!

Next time: Evaluating a Wine – Part 2:  The Nose

- Chuck Blethen, Vigneron in Residence, Black Mountain Oasis Vineyard, Scottsdale, AZ

Back to Top