Many online businesses rely on Black Friday and Cyber Monday to drive their profit margins. During this four-day period, retailers will see traffic on their site skyrocket.
How can retailers make sure their sites are robust and won’t fail during this critical period? The answer lies in the application of intelligent testing.
Black Friday traditionally has been the day when retailers finally break even for the year. “Black” in this case refers to accounts finally going into the black. The rise of online commerce has driven Black Friday to new heights. Now the sales phenomenon lasts over the whole weekend and into Cyber Monday.
Over the five days from Thanksgiving to Cyber Monday 2018, 165 million shoppers
spent more than US$300 each, on average.
Most online retailers will see a massive surge in traffic over the Black Friday weekend. In fact they will see a double whammy. Not only do more people visit — they visit repeatedly in their search for the best deals. As a result, retailers’ backend services are placed under enormous strain.
A failure during this period would be devastating, bringing bad headlines and loss of revenue, and probably losing valuable future custom. So, how do you avoid these pitfalls? The answer is to ensure your site is completely bombproof and can handle the surge in load without a problem.
Stress Testing
Stress testing refers to the process of adding load to your website until it fails, or until the performance drops below an acceptable level.
Typically, there are two types of stress testing. In the first, you check that your site can handle the expected peak traffic load. In the second, you steadily increase the load to try and push your site to fail. This is important, as you need to check that it fails gracefully. Traditionally, this sort of testing has been done in a very static manner, but as we will see, this isn’t very realistic.
API-Based Stress Testing
The earliest form of stress testing involved creating a script to repeatedly call your API. The API, or application program interface, is how a user’s client (browser or app) connects with your backend server. You can simulate users by calling this direct using command line tools like
cURL or using special tools like
SoapUI or
Artillery.
The idea is to place so much load on your back end that it fails. This approach has the advantage of simplicity, although it can be challenging to write the script. Each session will need its own API key, so you will need a script with enough smarts to handle all the keys and sessions.
However, there are three big drawbacks to this approach:
- Modern Web applications rely on dozens of interlinked APIs. This approach can’t test all these interactions properly.
- All sessions are coming from the same physical (and logical) source. This means that your load balancers will not be doing their job properly.
- Real users don’t interact in a predictable manner. Modeling this randomness is extremely hard in a test script.
API testing is still useful, but typically only for verifying the behavior of the APIs.
The Importance of Realism
Once upon a time, a website was a simple beast. It typically used a LAMP stack with a Linux server, Apache webserver, MySQL database and PHP front end. The services all ran on a single server, possibly replicated for handling failures. The problem is, that model doesn’t scale. If you get a flash crowd, Apache is quickly overwhelmed, and users will see an error page.
Nowadays, sites are far more complex. Typically, they run from multiple locations (e.g. East Coast and West Coast). Sessions are shared between sites using a load balancer. This ensures all your sites get equal load using various heuristics to assign the load, such as source IP address.
Many sites are now containerized. Rather than a single server, the application is built up from a set of containers, each providing one of the services. These containers usually are able to scale up to respond to increased demand. If all your flows come from the same location, the load balancer will struggle to work properly.
Session-Based Testing
Tools like
LoadNinja and
WebLOAD are designed to provide more intelligent testing based on complete sessions. When they access a website, users create a user session. Modern websites are designed so that these sessions are agnostic to the actual connection. For example, a user who moves from a WiFi hotspot to cellular data won’t experience a dropped session. The geekier among you will know that “session” is layer 5 of the OSI model. Testing at this layer is far better than API testing, since it ensures all APIs are called in the correct order.
Generally, these tools need you to define a handful of standard user interactions or user journeys — for instance, a login flow, a product search, and a purchase. Often the tool records these user journeys. In other cases, you may have to define the test manually, much like you do when using Selenium for UI testing.
Having defined the user journeys, you then can use them to run stress tests. These tests are definitely better than the API testing approach. They are more realistic, especially if you define your scenarios well. However, they still have the major drawback that they are running from a single location. They also suffer from the issues that impact all script-based testing — namely, selector changes.
The Importance of Selectors
Ever since Selenium showed the way, script-based testing tools have used JavaScript selectors to identify UI elements in the system under test. Elements include buttons, images, fields in forms, and menu entries. For load testing tools, these elements are used to create simple scenarios that test the system in a predictable fashion. For instance, find and click the login button, enter valid login details, then submit.
The problem is, JavaScript selectors are not robust. Each time you change your CSS or UI layout, all the selectors will change. Even rendering the UI at a different resolution can trigger changes, as can using a different browser. This means that you need to update your scripts constantly. Tools like WebLoad attempt to help by ignoring most of the elements on the page, but you will still have issues if the layout changes.
Intelligent Testing
Recent advances in artificial intelligence (AI) have revolutionized testing. Tools such as
Mabl,
SmartBear and
Functionize have begun applying machine learning and other techniques to create intelligent testing tools.
The best of these tools employ intelligent test agents to replicate the behavior of skilled manual testers — for instance, providing virtually maintenance-free testing, and creating working tests direct from English test plans.
Typically, these tools use intelligence to identify the correct selectors, allowing them to be robust to most UI changes. It is even possible to create tests by analyzing real user interactions to spot new user flows. Rather than just test simple user journeys, intelligent test agents can create extremely rich and realistic user journeys that take into account how a real user interacts with your website.
Intelligent Selectors
AI allows you to build complex selectors for elements on a Web page. These selectors combine many attributes — such as the type of element, where it is in relation to other elements, what the element is called, CSS selectors, even complex XPaths.
Each time a test is run, the intelligent test agent learns more about every element on the page. This means that its tests are robust to CSS and layout changes — for instance, if the Buy Now button moves to the top of the page and is colored green to make it more prominent.
Complex User Journeys
Test scripts often use simplified user journeys. This is because each script takes days to create and days more to debug. Intelligent test tools support the creation of richer user journeys.
They typically fall into two types: intelligent test recorders and
natural language processing systems. The first records users as they interact with the website, using AI to cope with things like unnecessary clicks, clicks that miss the center of an element, etc. Intelligent test agents use NLP to take plain English test plans and use these as a set of instructions to the test system.
Cloud-Based Testing
AI requires significant computing resources, and thus most intelligent test tools run as cloud-based services. Each test is typically run from a single virtual server. These virtual servers may exist in multiple geographic locations — for instance,
AWS offers seven locations in the U.S. and a further 15 worldwide. This means each test looks like a unique user to your load balancer.
Intelligent Stress Testing
Intelligent test agents combine realistic user journeys with testing from multiple locations, giving you intelligent stress testing. Each cloud location starts a series of tests, ramping up the load steadily. As each test completes, a new test is started. This takes into account the different duration of each test (allowing for network delay, server delay, etc.)
This means you can generate tens of thousands of sessions that look and behave exactly like real users. Better still, you can record session-by-session exactly how your site responds. This allows you to see which pages are likely to cause problems, and gives you detailed insights into how your site performs under load.
This approach addresses all the problems with both API and session-based test tools. The test sessions look and behave exactly like real users, so they will generate the correct sequence of API calls. Your load balancers and infrastructure will behave as they should, because each session looks unique.
Finally, the system is intelligent, so it won’t try to call a new API before a page is properly loaded. This is a marked contrast to other approaches where you tend to use just a fixed delay before you start the next action.
Stress testing is essential for any e-commerce site, especially in the run-up to Black Friday and Cyber Monday. Traditional approaches, such as API and session-based testing help when you have a monolithic infrastructure, but modern websites are far more complex and deserve better testing.
Intelligent test agents offer stress testing that is far more accurate and effective, allowing you to be confident in how your site will behave under realistic conditions. It also can give you peace of mind that any failure will be handled gracefully.
Paul Clauson is chief evangelist at
Functionize.
