The Struggle for
E-Quality
Designing an effective test
environment for an Internet application can be challenging
by Alex Buccino
In a perfect world, all new e-business applications or site
revisions would be completely developed and then thoroughly tested in
an environment that was an exact duplicate of the production
environment. The real world is quite different.
Even companies committed to proper testing can seldom afford a
test environment that is identical to their production environment.
Time is also a factor, because an inflexible marketing schedule
dictates the release of most e-business applications and pushes them
out the door the second they are deemed fairly functional. Little or
no time is left for quality assurance (QA) testing at the end of the
development cycle.
In addition, you must consider the large number of potential
environments. Users access e-business applications from a vast range
of systems, and you can't possibly test them all. Even if only one
company or select group of suppliers will use the application, the
range of remote systems and connections can be daunting to test. In
nearly all cases, you can't create an exact duplicate of the
production environment, but instead, must build a subset of the
production environment for your test environment.
Testing on such an environment can be effective, as long as
you're working from the proper design and you correctly prioritize
and execute the various tests. For example, your company is rolling
out a major revision to a site. You have a current production
environment, but no testing environment in place. Production is
progressing well, but the new functionality is taking longer to
program than anticipated. But because marketing wants to make a big
splash, it has already geared up the PR machine, and the deadline
can't slip. Because you can't test the system all at once, you will
need to separate the different forms of review so you can perform
them in parallel with development.
First Priority
User testing and compatibility testing are good places to start.
In fact, some testing of this type should always occur before
development is too far along, so that developers can incorporate
necessary changes that the tests reveal. You need an environment for
both user and compatibility testing, but fortunately many of the
requirements for these tests overlap. In general, the environment
needs to be stable and isolated. Back-end connectivity is not
important for these tests, and functionality that relies on back-end
connectivity should be mocked up.
Although performance is an issue, scalability is not. The system
must perform at a speed similar to the production environment for
users to get a reasonable idea of system responsiveness, but the
limited number of users involved shouldn't create a significant load
on the environment.
Furthermore, because you can do a mock up of back-end systems, you
can't judge the production system's performance solely on the
performance of the user test environment. The environment must also
have PCs for users and compatibility testers. User testing should
generally be done on "ideal" PCs - that is, PCs configured to take
full advantage of all the site's features and that you know don't
have any compatibility issues with the site. Trained testers should
perform the compatibility testing and isolate its issues from
usability issues.
Feedback from user and compatibility testing will make some
changes to the application necessary. Although those changes add some
development time and require additional production resources,
performing the testing earlier rather than later can prevent some
nasty surprises. You should perform another compatibility test after
final integration to ensure that no additional issues have appeared.
That second compatibility test will run much more smoothly because
you will have identified and resolved most issues during the first
round of testing.
The Next Step
Functional testing is more demanding, and you must conduct such
tests on a logical duplicate of the production environment. The test
environment should have at least one of each of the production
architecture's functional elements, although the scale of these
functional elements is generally not important. For example, if the
production environment comprises 10 Pentium III 800MHz HTTP servers
coupled to two Pentium III 1GHz application servers and a Sun 6500
database server, the functional testing environment might have two
Pentium 600MHz HTTP servers, two Pentium 800MHz application servers,
and a Sun Enterprise 450 for the database server. This set up lets
you test all functionality, but does not represent the final system's
performance.
One important note about scaling down load-balanced systems:
Although a logical duplicate requires only one example of each
functional element, any elements that are load-balanced in the
production system should also be load-balanced in the testing
environment. You would need at least two examples of that component
in the testing environment, even if only two were in the production
environment, as in the previous example. If you fail to include the
load-balancing functionality in the test environment, you risk
encountering load-balancing issues in production. Even worse, you
risk session management issues, a common problem in load-balanced
environments, which you shouldn't test for the first time on a
production environment.
Making Connections
What the previous functional testing environment description does
not include is connectivity to outside systems. E-business systems
can have any number of connections to external systems, such as
legacy mainframe databases, ERP systems, inventory and fulfillment
systems, credit card authorization systems, and externally held user
data - and you must test the connectivity and interfaces to these
systems. However, testing these external connections along with
initial functional testing is often undesirable or impossible, for
two reasons:
Coordination and resources: For example, an e-commerce site may
connect to a routing system that holds inventory levels of multiple
distribution centers and routes orders for fulfillment based on the
customer's ZIP Code and available inventory at the distribution
centers. The system ships the order from the closest distribution
center that can complete the entire order in one shipment.
You don't want to waste time coordinating a test that may involve
the resources at three distribution centers, the Web site development
team, and the router development team only to find that the
functionality needed to put together an order in the first place is
not available. Although performing an end-to-end system test as early
as possible is desirable, you must be practical about the actual
state of the application and not waste the time of the various
parties involved.
A missing piece required for external connectivity: Although
distribution centers may be ready to receive orders, the details of
the router business logic may not be finalized. (Should an order to
New York that can be fulfilled in two orders by the Pennsylvania and
South Carolina centers be fulfilled in one order by the California
center?) In either case, functional testing must come before to
end-to-end testing. In these cases, the functional testing
environment will need to include a way of simulating the connection
to external systems. If the router is the missing component, you
could write a program that emulates the router's responses. Although
you won't know whether the system will work with the router attached,
you can continue with testing the checkout functionality of the
e-commerce system without waiting for the router component.
From One End to Another
When external systems are ready, you can attach them to the
functional testing environment for end-to-end system testing. At this
point, you still only need a logical duplicate of the production
environment, only adding the connections to the external systems.
During initial testing, this process usually involves attaching the
Web site testing environment to a testing environment or region at
the external end.
For example, credit card authorization services have testing
servers or can be configured to be in a test mode that emulates
credit card authorizations without actually charging a card. These
test modes differ from service to service in the way that they
emulate passes and rejects - some decide based on the value of the
order, some base it on the card number, and others pass everything in
test mode unless you request some reject samples. Again, before final
deployment, you'll want to switch the external systems out of test
mode and perform a few live tests, but the extra hassles (such as
credit card reimbursement and inventory adjustment) required to do
live tests should encourage you to do as much confirmation in test
mode as possible.
Find the Time
Performing security testing with the release version of the
application code - although ideal - is usually impractical. You must
somehow sandwich security testing in with all your other priorities.
One way of doing so is by placing a build that you know is stable
into the final production environment while functional and end-to-end
system testing is still occurring in the testing environment.
Security is as much a matter of infrastructure and configuration
as it is a matter of the application code, making the environmental
requirements much more stringent. You must perform security testing
in a final production environment configured exactly as it will be at
launch. You could perform extensive security testing in a production
environment with nine Web servers, and everything may be fine. But if
you add a tenth server prior to the launch date, you must retest
security.
Performing security testing on a non-production environment has
little value. At most it will tell you that the application and basic
security principles are sound. Just prior to launch, you should also
perform a final round of security tests to ensure that the migration
of final application code has not compromised the system's
security.
The Last Test
Performance testing has the most stringent environmental
requirements of all the QA tests. Any hardware, software, or
infrastructure deviation from the final production environment can
compromise the integrity of performance testing. Ideally, you would
conduct performance testing in the final production environment with
the final production code. However, you can seldom delay a launch for
performance testing once final production code is ready. The next
best situation, similar to the security testing approach, is to test
a late stable build in the production environment. Because these two
forms of testing require similar environments, overlapping them may
be tempting - but don't! Attempting to do security testing while
conducting performance testing can skew the results of the
performance tests, particularly if the security testing succeeds in
breaching the system.
Time, money, and the complexity of e-business all combine to make
QA testing a considerable challenge. But prioritizing the various
tests and understanding the environments they each require will let
you meet the impossible deadlines and budgets that are part of our
business, while still turning out a well-built application.
Alex Buccino (alex.buccino@marchfirst.com) is quality assurance
manager in MarchFirst's New York office.
