How to Load-Test 1 Million Users via JMeter and BlazeMeter?
Dunn & Bradstreet have reported that over 50% of Fortune 500 companies face downtime of their critical Business applications each week because of an unexpected and sudden increase in the load.
They have reportedly lost thousands of dollars each second during these outages.
Hence, any Business Software, be it Mobile or Web App or even the APIs – must be evaluated for its performance and its ability to handle the Load under anticipated user loads to cut these losses.
The primary objective of Load Testing is to identify
- How swiftly an application responds?
- What is the maximum number of Concurrent Users that the application can handle?
- How stable is the application under fluctuating load?
- How swiftly an application responds?
However, it is not as easy as it may sound to Load or Performance Test an application.
Here are some of the most common challenges that the Testers face while Load Testing an app:
- Uncertainty of How to Start the proceedings?
- What questions do they need to ask the client to connect the dots?
- How to prepare a Performance Test Plan?
- Uncertainty of How to Start the proceedings?
And these challenges are not just limited to the Testers.
There are instances where even the client is not sure about the parameters to be considered for evaluating the performance of their business application.
As startling as it may sound, but there have been incidents where the clients weren’t sure about the expected Response Time of the application or the amount of Load that they expected their application to handle, or even the volume of transactions that they expected their software to process.
They’re just not sure about a Concrete number, although they always have a generic figure for reference.
However, I don’t blame them.
They aren’t the expert Performance Testers in the house, you are! Let the Client take care of the Business part of their application while you look out for these answers for them! Be their Guiding Light, like I shall be for you here!
So, how do you begin? How do you start planning to Load-Test an application? I have a checklist for you below:
- Authorization Letter
There is no way you could ethically generate a Load or Traffic on a particular application without the legal consent of its Business Owner.
Basically, you would need an Authorisation Letter from the client that has them asking you (and providing you with all the necessary permissions) to Load Test their application.
While the letter should include the details of the owner of the software, it should also have them acknowledge the risks associated with Load-Testing the application.
Without this, as a Tester or a Project Manager, you could be in a Tricky situation.
Say, if the Application crashes while ethically Load Testing it and if the client refuses to acknowledge the deficiency in their system, you could be blamed for the crash.
- Comprehensive Coverage
- Identify a Benchmark for the Load Testing Scenarios How many users can the application easily sustain with the existing infrastructure? It is important to identify the Number of Concurrent Users the application could easily support.
The point where the application runs seamlessly with “n” number of users can be identified.
Remember, do not confuse this point with the Stress Point as here you are identifying the point where the application runs with a certain load, as otherwise with the Stress Point where the application breaks.
- Identify a Benchmark for the Load Testing Scenarios How many users can the application easily sustain with the existing infrastructure? It is important to identify the Number of Concurrent Users the application could easily support.
- Identify Bottlenecks
Bottlenecks are known to degrade the throughput of the application under a particular load.
While running a Load Test we determine the response time of each request that is made to the application.
And while doing so if most of your use cases are recording a similar response time and if one of the use cases is caught recording a higher than usual response time then this could be a potential bottleneck.
The reason for this could be anything, right from Unorganized Database Queries, CPU & Memory Utilization, Operating System Limitations to Unoptimized Code, or even Unstable Caching Mechanism. You’d have your task cut out here to identify the bottleneck.
- Identify Bottlenecks
- Ensure the app meets the Standard Performance Criteria
Having Standard Performance Criteria enables you with the right information for conducting the tests.
They particularly provide a vital starting point for Response Time objectives where there are no previous performance reviews, without having to predict or compare them with another application.
Standard Performance Criteria and measurements, such as Single-User Login time, the Request or the Response Time of the User’s screens, and so on, should be initiated with the system load as Zero.
- Ensure the app meets the Standard Performance Criteria
- Verify if the Application meets the SLA
If the Client has set their expectation of the User-Load to be 5000 Users for a particular instance in the Service Level Agreement (SLA) then it is important to verify if the application holds the ability to sustain that particular User-Load under the required conditions.
The application could either manage to sustain the traffic or it would fail and crash at the BreakPoint. Accordingly, you could take measures to fix the issue.
- Verify if the Application meets the SLA
- Knowledge about the System Infrastructure
It is important to stay updated and have the necessary knowledge about the software’s infrastructure.
Based on this knowledge, you’d be able to take the necessary calls to identify the Ideal Load, Throughput, that the application could process.
These days everything happens on the Cloud.
Popular Cloud Service Providers like AWS offer instances with predefined configurations for you to study before initiating tests.
- Knowledge about the System Infrastructure
- What is the type of instance?
- What is the capacity & Family of the CPU?
- What is the size of RAM and Hard Disk?
- Is there a Load Balancer in the instance?
- Is there a Storage Service employed to store assets and static files? If yes, is it scalable?
- Where is the Database Hosted? Is it a Relational Database or is it stored in the same instance where the application is hosted?
- What is the type of instance?
These are some of the basic yet vital system information that you need to be aware of, as a Performance / Load Tester.
Accordingly, based on your understanding of the infrastructure and the system’s performance, you’d be able to provide the necessary recommendations to the stakeholders at the end of the Test.
Next, Performance Testing and its subsets:
- Load Testing
Initiate the process by gradually increasing the User-Load.
Now the number might vary depending on a particular use case but in general, you might want to start with 100 Users and see the response.
You may then increase it to 500 Users and validate the response.
Gradually you could increase the Load Count to 1000 Users and check the response.
All this while, you would have received reports for each User-Load, which would have the parameters and performance metrics.
This could be shared with the Project Stakeholders. Metrics may include Average Response Time for each Use-case, Volume of Transaction processed in each case, and Concurrency. Concurrency is the simultaneous, but not necessarily similar activity that the users perform on the application.
Reports containing these metrics could be shared with the concerned stakeholders in a Single HTML report or a Spreadsheet or a CSV format.
This would enable them to have a Birds-Eye-View of the Performance of their application.
- Load Testing
- Stress Testing
This requires testing an application under maximum load to validate how it processes data and manages the high volumes of traffic.
Here, the goal is to identify the breaking point of an application. Say, you started with a Load of 1K Users and now you’re increasing the Load to 5K Users.
This gradual increase of Load would continue until the point where you start experiencing failures in the application or unexpected exceptions in the performance metrics of the application.
This point is referred to as the BreakPoint. This information is sometimes not asked by the stakeholders.
Through this test, you could determine the Stress Point of the Application and present this information to the stakeholders upfront.
- Stress Testing
- Endurance Test
Endurance is the duration for which the server could handle the Load.
For example, under continuous application of Load, the software could be good to handle a Load of 10000 Users but for an hour.
Since the server’s endurance is 1 hour, a continued application of 10K User-Load over an hour might result in a server crash or it may show signs of unexpected behavior.
So, it is imperative to determine the count of endurance for the application to function as expected.
- Endurance Test
Next, we shall discuss the Performance Test Plan and its components.
To have a good Test Plan, it is important to address the WHYs, WHATs, WHENs, and HOWs of the testing process. Consider having these in your Test Plan.
- Requirement
- Scope
- Entry criteria
- Workload
- Server Monitoring
- Test Scenarios
Besides performing all these activities, it is important to also keep a tab on the Server.
You could coordinate with the Server Manager to monitor the activities on the server Or you could also set up a Server Monitoring tool on the server to monitor its logs for information.
Some of the popular Server Monitoring Tools are Amazon’s CloudWatch, Microsoft’s perfmon, and Nagios.
Comparison between JMeter and BlazeMeter:
| JMeter | Blazameter |
|
|
|
|
|
|
|
|
|
|
Next, the Question of the million? How to Load Test a Million Users? What do you need to Test a Million Users’ Load?
To be honest, the practical question, here, should be – Do you really need to do a Million-Users’ Load Test? Consider a high traffic application like Instagram.
It probably has over a billion users at any given time. So, if Facebook had to Load-Test Instagram, do you think they’d do a Test of 1 Billion Users before launching the application? The answer is No.
Here’s why – Although the Load-Testing Tools allow you to test with as many virtual users as you would like, practically, this could be a very expensive affair as the tools are priced based on the number of concurrent virtual users.
To counter this, it’s always viable to go ahead with an Internal or Beta launch of the application.
This way you could get key insights about the actual behavior of the users and their traffic and usage patterns.
During load testing, it is often best to validate with a precise number of virtual users.
Experts are known to start with a reduced count of virtual users while still fetching accurate results, though it could be a risky move. A thorough understanding of the system architecture (as detailed above) enables you to take an appropriate call in such cases.
Also, having transparent communication with your dev and web-analytics teams works wonders in this regard.
To achieve our Load-Testing objective, we could use JMeter, which is one of the most popular open-source load simulation software, along with BlazeMeter, which happens to be one of the widely used cloud-based load simulation solutions, together.
Now, how to generate a Load of 1 Million users?
Practically, we don’t generate a Load of 1 Million users. So how do we test this scenario? Consider the following example:
An application is running on a t2.micro instance with a 4GB RAM and 50GB HDD and no Load-Balancer is employed.
The application is using S3 and RDS.
The application owner expects traffic of 1 Million users after 3 months of application launch.
In general, this setup may cause around $50 – $60 per month. Now, this setup could easily support a load of 100-200 users. So even if you were to target a count of 1000 users, it would cost you $60 x 5 = $300. Similarly, you may do the calculation for 10k users.
The cost would go up significantly. Just imagine what the cost would be if the objective is to Load Test 1 Million users? Hence, this is not a practical and feasible option for most of the clients. So how do we go about testing 1 Million User Load?
Initiate a test with 100 users, then gradually increase the count of the load in milestones to 500, 1K, 5K, and 10K respectively.
Keep monitoring the reports of each of these as you complete testing the milestones, for any variations in the system.
Now at 10K load, if you experience any wobble in the system then you have 5 Reports to fall back on to determine the root cause of the issue.
If there’s no wobble, then you have your system and its infrastructure ready to handle the load of 10K (maximum Load-Count) users.
This, of course, is a test of a small scale.
Now, since the requirement is to Test 1 million users, you could divide this requirement with the maximum Load-Count of the test of small scale i.e. Divide 1 million with 10K and you get the number 100.
This (100) is the percentage by which the infrastructure needs to be increased to support the load of 1 million users and the same needs to be recommended to the client and the stakeholders.
So, in this case, the application’s infrastructure needs to be upgraded by +100% to be able to support the load of 1 million users.
So, this is how by not actually doing a 1 million Load Test we managed to achieve the end goal by performing a small Test with a limited load.
This way the client can direct the project funds on scaling up the system infrastructure or the production server rather than spending thousands of dollars for Load Testing and its associated servers.