Stress Relief

Subjecting your e-commerce application to preproduction load testing may help you avoid business disaster

By Richard Ellison

The IT manager’s cell phone rang.The caller — the CEO of an e-commerce business whose Web site had just crashed under the Christmas shopper load — was frantic. All he could talk about was his entire company crashing along with the Web server.

His high-flying IPO was about to fade away….

How many times have you approved a software prototype for development, only to see it crash under a full production load? Have you ever gone to an e-commerce portal to buy something, waited in vain for a response, and then simply clicked over to another site to buy exactly the same thing?

A friend of mine told me recently how a vendor was demonstrating a prototype of some Web-based order-processing software for his company. This system was supposed to serve employees in his department, which is spread across the East coast, through the corporate intranet. The release date arrived with great fanfare; everyone in the department received the URL of the application. As they logged onto the new system at the appointed time, the load instantly crashed the Web server and the database server. It simply couldn’t handle the load. The outside vendor had to go back to its drawing board, having lost its reputation and a large customer as well.

All these scenarios are symptoms of the same problem: a lack of predeployment load testing. Load testing is a specialty within the software quality assurance area that addresses this need; it involves the use of packaged or home-grown tools that emulate thousands of logins to the subject application. This specialty is growing both in importance and popularity because e-business is making performance not only a more important issue, but also one that you need to address earlier in the development cycle.

That’s So “Yesterday”

In the old days, an application was limited to a department on one floor. One of the final tests before rolling it out would be to ask all the end users to stay a little late one evening to bang on their keyboards until the boss said “stop.” That was the “load test.”

The dynamics on the end-user side have changed dramatically. Today, the end users may not only be on separate floors, but in separate buildings in faraway cities around the world; furthermore, most of them may not even be employees. It’s not always economical, not to mention possible, to pay people to load the system down. Nor can you perform strategic testing to determine future computing needs if you don’t exceed the current user volume in your test.

The dynamics of the technical side have changed dramatically, too. Remember when the network comprised a dumb terminal or a powerful PC wired directly to a mainframe? In those days, performance was a rather simple issue. Now, hundreds or thousands of local PC users are connected to an Ethernet LAN, which is connected through a firewall to the Internet. The system processing the transaction may be anywhere from two to 12 “hops” across the country; it may comprise a firewall, a Web server, and one or several application servers and databases. Often, a legacy mainframe system is hooked to the back of the back end. That’s a lot of potential bottlenecks to worry about — many pieces of hardware and software sit between the customer and the back-end systems, and each one adds more time to the total response. If each piece adds only a half-second to the total performance, the minimum response time, even before any business logic processing, would be four to 30 seconds.

Distributed, multitiered architecture is only part of the problem, however. Whereas huge costs used to limit dedicated-line information transfer to only the largest of corporations, it is now cheaper to deliver data through the Internet, and end-user setup is measured in minutes and days, rather than weeks and months. Consequently, it’s now a given that huge numbers of remote users can access your systems.

Furthermore, e-commerce may be only a mouse-click away, but so is “e-competition.” Disgruntled customers don’t have to obtain, install, and learn another vendor’s software in order to take their business elsewhere; they just click away somewhere else. The propensity to jump brands is much higher — nearly instantaneous — in an e-business environment.

Examples of Failure

Recently I purchased some circus tickets online and received a confirmation. But when I arrived at the will call window with my excited five-year-old daughter, I was turned away because the box office had no record of the purchase. Infuriated, I resorted to buying scalped tickets from the nearest scalper. As it turned out, the scalped tickets actually cost less than they would have conveniently cost online. Old-fashioned ticket scalping saved the day.

In essence, the ticketing system’s front end outpaced the load capability of the back end and lost data in between. The system looked available, but actually wasn’t completing the order on the back end. This problem is a common one.

Many businesses also bring misfortunes on themselves by charging forward with new sales systems without developing the support systems first. For example, I recently did business with an online airline ticket bidding company. This company has a large, automated sales engine and a tiny, manual error handling process. I caught a snafu within two minutes of initiating the transaction, but it took me three hours on the telephone and two emails to get the company to refund my money minus a minor handling fee. The revenue producing part of the site could marginally handle the load, but the support part couldn’t keep pace — a scalability problem all in itself.

A casual scan of the trade media will reveal multiple headlines about failed transactions, sites crashing under a load, denial of service attacks, and disasters resulting in lost transaction data. Furthermore, as companies increasingly outsource Web-site hosting to external companies, a new kind of story is becoming increasingly common: those about customers suing hosting vendors when the Web site doesn’t perform up to specifications.

The Solution: Load Test

Many categories of quality assurance exist in the software development arena, but the type pertaining to our discussion here is called load (or stress) testing. Load testing emulates the presence of masses of customers on the system in order to find the performance “knee” and maximum capacity before causing a system outage. (See Figure 1.) The performance knee is the point where one additional unit of customer usage decreases the performance greater than one unit of performance. In economic terms, it’s known as the law of diminishing returns. It marks the volume where performance degrades more rapidly to the point of unsuitability. Performance testing, another quality assurance specialty, measures system response to a certain volume of customers and compares this response time against the design performance specifications.

FIGURE 1 The performance knee.

Some companies also use these types of tests in development, capacity planning, and disaster recovery planning. Three of the six largest U.S. banks (Bank of America, Citicorp, and First Union) apply these techniques, and the dominant database vendor, Oracle, uses them in its own product development process. I’ve also found load-testing programming hooks in Sun Microsystems’ NetDynamics application server platform.

The goal is to answer the bottom line question: “How much computing power do we need to run our business?” accurately and objectively. More specifically, as your business grows, you need to know the “economic order capacity” at which to buy more computing power before your business performance dips below a certain standard.

The Method

Developing scalable applications requires careful attention to performance at every stage of the software development lifecycle. It all starts with defining an acceptable response time, perhaps by using prototypes that gauge how users react to certain response times.

The practical application of this meth- odology is to first measure system performance with the anticipated number of customers involved. Let’s say that 1,000 customers have access to our internal system. As a business process solution, the system will probably be used heavily during business hours; the expected concurrent usage is 10 percent of the customer base, which is 100. (This percentage is a starting point based on the pilot or beta customer usage. You should monitor this number carefully as the customer base increases; a 1 percent change can mean a large change in real numbers.) Thus, we’ll design a test to run 10 test rounds in increments of 10: 10, 20, 30, and so on. We measure the time each process takes and compare it against the design standard — perhaps the standard states that a customer should be able to perform a query for one item in less than five seconds and 10 items in less than 30 seconds — and tally the results for evaluation. Let’s say the application performed up to standard at the current maximum anticipated load of 100.

The next step is to perform a load test. Based on the performance measurement results, now the goal is to find the performance knee bottlenecks, maximum capacity, and point of failure. Thus, we’ll ramp up the numbers of customers, to 90, 100, 110, 120, 130, 140, and 150. Let’s say that the system performs up to 100 just fine, but at 110, the performance falls below the standard. The server monitoring also shows that application server A is consumed with hard-disk paging and a high processor usage number. We have found that perhaps application server A needs more memory. We continue with the next round of 120, and the application starts logging errors in 50 of the 120 virtual customers. This particular project has defined these errors as points of failure because processing errors are unacceptable. Thus, we cease the testing, and the systems manager adds more memory to application server A. Now that we’ve identified and fixed one bottleneck, we can run the test again.

As you can see, this iterative process is designed to tune the systems, software, and database to the point of maximum efficiency. Through tuning and performance enhancements, this system can handle 140 customers. If we want to go to 280 or 420 customers, we need to add two or three times the computing capacity to accommodate this business requirement. The corporation in our example needs three months to order and install a new system, the sales staff is subscribing 100 new customers per month, and the current customer base is 1,000. Therefore, the computer equipment order process has to start within one month in order to up the capacity above 140 by the fourth month, when 1,400 total customers will translate into a load of 140 concurrent users.

For the sake of brevity, this simple example focuses on only a portion of the capacity calculations involved. It does not address many other aspects such as concurrent usage trends, disaster recovery, and the amount of business risk associated with this application.

Other Applications

Load testing has several other uses that testers have serendipitously found while preparing load test scripts. Running a load test magnifies the processing, making it easier and faster to find some problems.

One such problem is a memory leak, a software programming error in which every usage increases the allocated memory without releasing that memory resource when the program is finished with it. The amount of allocated memory increases until it exceeds the physical memory; at this point, performance degrades dramatically and ultimately crashes the system. During a “normal” testing process in which only a few people perform functional testing, it can take over a day to recreate a memory leak. Furthermore, during an iterative development process, the machine may be restarted before it crashes from a memory leak, thereby concealing the problem until it reaches a crisis level in production. In contrast, with a load test scenario, you can recreate a memory leak problem within an hour.

Load testing is also helpful in determining the optimal configuration of a load balancing solution. One person at the input end can “present” numerous customers to the system while the systems administrator tweaks the settings on a load-balancing tool.

Because you can configure a tool to run as fast as the application can respond to input, it can also find timing problems among the server-side components. These timing issues often exist in a production load but usually remain hidden in a conventional functional quality assurance process. These problems can crash a Web site and keep it crashing after each reboot until the issues are solved.

Rate This Article Rating: 1 (best) 2 3 4 5 (worst) Comments: Optional e-mail address:

Stay Vigilant

Performance is becoming a more important issue as systems become more complex. Your company can gain a competitive edge and acquire more customers more rapidly through a top performing e-commerce solution. Conversely, without a load testing strategy in place, reputations can be ruined and revenues lost at the speed of e-commerce.

Richard Ellison (richard@leesystems.com) is currently an independent consultant on a large development project involving Internet-based business banking applications. A performance developer, he writes load test programs and tests the applications.

RESOURCES Reddy, Ram. “Building the Unbreakable Chain.” Intelligent Enterprise, Feb. 9, 2000. Available at www.intelligententerprise.com/000209/feat3.jhtml. Van Cruyningen, Ike. “E-Business Everlasting.” Intelligent Enterprise, Oct. 26, 1999. Available at www.intelligententerprise.com/db_area/archives/1999/992610/feat1.jhtml.