Performance Testing is a software testing practice to evaluate a system or applications performance in terms of its speed, stability, scalability and reliability. It’s a part of non-functional testing ensuring that application performs at its best in every situation. There are multiple factors in a distributed application which decides performance of an application. Below mentioned are few of them.
Server-side metrics – Server Response time
Server -side resource utilizations – CPU, Memory, Disk.
Network latencies
Client-side performance metrics – Transaction Response Time, Throughput, Failures and Error rates
Front end Metrics – Page load time
In this modern era, we expect everything to be operating in lightning speed. Hence it is a must rather than a luxury in having performance engineering and testing while developing a solution. Imagine use case of an ecommerce website. You plan to purchase a product and while you are searching for the product it is taking a long time to fetch your product search results, would you prefer to continue shopping using this website? . Business face a huge loss if they ignore the need of it.
Performance Testing Vs Performance Engineering
Performance engineering is a discipline of designing, developing, testing and tuning an application in creating a high performing and scalable application while performance testing is a subset of performance engineering wherein the practices of testing an application by mimicking the user load as expected in production is conducted and after evaluating execution results, performance optimizations are applied.
Now looking into performance engineering the efforts starts from application design phase itself. Appropriate measures are taken care in the beginning of a software development life cycle to deliver a high performance and scalable solution. Performance engineering phase also gets involved in tuning and optimizing the developed solutions based on the test execution results.
Performance engineering scope also includes in the process of resource capacity planning which fulfills the need of a delivering the need of a highly scalable application which withstands with growing business needs.
When to start?
Organizations follow a practice of staring performance effort during or after system integration test is completed. But there are many flaws in starting performance effort in later stages of a testing life cycle
Less time for the developers to make code changes in optimizing performance.
Technical architect or solution architects have to make cost and time intensive modifications in system design sometimes from the scratch
Less time for performance test engineer to understand the application architecture and build an effective Test strategy.
Poor capacity planning
As you can see there are many challenges while involving performance testing effort in a last-minute hurry. Hence to overcome these challenges its always ideal in involving performance engineering efforts in the design phase of a release cycle. By involving performance engineering in design phase, we gain a lot of benefits like
Developers get ample amount of time in optimizing code for a higher performance
System designing based on test execution results which makes the system architecture scalable, robust and stable with higher resilience.
Performance engineer can include more scenarios and test strategies in testing the application
We can effectively do capacity planning in terms of resource scalability, cost optimizations etc.
In this article about performance testing process, you can read about end-to-end life cycle of performance testing starting from requirement gathering to final sign off. Below mentioned are the stages of performance testing.
Requirement Gathering
Test Planning
Test Environment, Test Data & Monitoring Readiness
Scripting
Workload Modelling
Test Execution & Monitoring
Result Preparation & Analysis
Performance Bottleneck / Defect Fix, Retest
Performance Signoff
Requirement Gathering
All information required to start a performance testing is collected in requirement gathering phase.The key focus in requirement gathering phase will be in getting an understanding on system architecture, different system components available in the architecture and its integration with each other.
In requirement gathering phase information on scenarios, workflows ,components involved etc are collected. Volume used for performance testing is another important criteria.This discussion happens with developers,product owners,architects,infrastructure owners etc.
test planning
Test strategy and planning is the next phase after requirement gathering for an application. In this step, a test plan document prepared with below details by a performance test engineer.
Overall Test Strategy
Test Objective
Scenarios or User flows
Estimations – Resource and Timeline
Risk & Mitigations
Infrastructure and Capacity details
In scope & out of scope
Test Environment, Tools, Test Data and Monitoring details
Timelines
Internal and external stakeholders will review and approve the test plan document before next process in performance testing life cycle.
Test ENVIRONMENT, Test data readiness and monitoring setups
Performance test environment should be similar to production environment. This helps in maintaining accuracy and reliability of performance results. Data volume for performance testing should be based on infrastructure capacity specially incase if performance test environment is not similar to production environment in terms of its capacity.
Performance testing needs adequate volume of data to test a scenario more realistically. A full-fledged end – end monitoring setup has to created which helps a performance engineer triage a performance bottleneck. Multiple monitoring and log analysis tools are available in the market like Dynatrace, AppDynamics, New relics, Splunk etc . These tools will help to identify performance bottleneck quicker and easier.
scripting
In scripting phase, different tools like Loadrunner , JMeter, Neo load etc is used to develop performance scripts. These scripts might be an API based or a user interface based script. Scripts usually mimics the real user actions or load pattern. If you wish to read more about articles about JMeter, then refer JMeter Blogs
workload modelling
In workload modelling, number of users needed calculation done to simulate a load. It is calculated using little ‘s law. It helps performance engineer to design a realistic load while running a performance test.
test execution and monitoring
In execution phase different types of performance testing like Load test, endurance test, stress test etc are conducted along with a real time monitoring of client and server side metrics to evaluate applications performance. Based on the type of application and its business usage, performance engineer will decide about the types of execution needed for testing. To know more about types of performance test execution read this article on Types of Performance Testing
result preparations and analysis
In performance testing process, result preparation and its analysis is the crucial phase of entire performance testing life cycle. Performance engineer looks into different performance metrics like response time, throughput, error rates, resource usages, downtimes, outages and report them to different teams like development, infrastructure, devops and many more.
Performance Bottleneck / Defect FIX, Retest
In result analysis phase if a performance engineer is identifying a performance bottleneck, then performance test engineer will collect more data related to performance bottleneck and communicate with the relevant team with a defect. Developers will always be benefit by a detailed analysis and report on the identified bottleneck. Defects will help performance engineers to track the progress on fix for the identified performance bottleneck.
Performance Signoff
In the final stage of performance testing life cycle, an application is certified stating as “GOOD TO GO” to production followed by production monitoring.
Evaluating performance of an application is always a key factor particularly if the application is used by high number of users. In order to evaluate how good/bad application works, we need to conduct different kinds of performance test execution. If you like to read basics of performance testing and its importance, then go through this article about Performance Testing Introduction. In this blogpost Iam including different types of performance testing conducted to evaluate an application performance, stability, scalability, durability etc.
Below mentioned are the most common types of performance testing.
Load Testing
Endurance Testing/Soak Testing
Spike testing
Stress Testing
Volume Testing
Scalability Testing
Failover testing
load testing
Load testing is to evaluate how an application perform for a specific user load. Ideally in a load test, an identified number of virtual users performs a set of transactions (user flows) or Api’s requests and evaluates the application performance in terms of its response time, resource usages like CPU, memory and other performance criteria. User load decided based on the current production load. For a new application its calculated based on competitor business volume. A load test ideally runs for 1 or 2 hours.
endurance testing/soak testing
An applications performance for a longer duration evaluated in endurance testing. In a load test, an application performance evaluated for few hours. Here we are not getting an overview of its performance for a longer duration. Performance bottlenecks related to resource usage comes up while we do a testing for a longer duration. For example, out of memory errors due to high memory utilization mostly identified by running an endurance test. Ideally any execution running more than 5 hours can be an endurance test.
Spike testing
A spike testing is done by a sudden increase or spike in load and conducted to see the application performance due to sudden increase in the load. This type of testing is usually benefited if we need to keep our application ready for any kind of unexpected spike in load.
stress testing
In Stress testing , an application stressed by putting a higher load. It is conducted to identify application breakpoint. Applications reliability, stability and resilience are evaluated by running this test with higher load beyond the benchmarked limits of the application.
volume testing
In volume testing, a high volume of data pushed into system to evaluate its performance. High volume data generally pushed using a batch job for doing a volume testing.
scalability testing
An application scalability measured by conducting scalability testing. Its ability to scale up or down by an increase or decrease in load can be measured by conducting scalability testing. Scalability testing results used in capacity planning.
failover testing
Failover testing conducted to identify applications availability during any kind of region failovers or outages. The results used in taking steps in increasing application availability.
