95% faster testing through automation of Performance & Functional Tests for a Streaming Platform

Provide our customer with a robust test automation suite

Our customer, a Fortune 500 company consisting of more than 70,000 employees, aimed to develop a video streaming platform to broadcast their tech events to viewers all around the globe. In achieving this target, they faced two primary hurdles from a testing perspective:

• Extensive Testing Cycle: The testing cycle was quite intensive due to manual test coverage and regression for a large application. This would cause release cycles to be considerably delayed.

• Application Crashing Under Large Load: The application would start to falter and break when the number of users exceeded a threshold.

The opportunity in this scenario was to provide our customer with a robust test automation suite, which not only provides a solution to test their system under high load but also reduces the time of their test cycle and provides quick feedback.

Develop robust test automation suite for performance testing using WebdriverIO & Cucumber

• Automation Framework Selection: We selected WebdriverIO to be used along Cucumber for developing a test automation suite. These tools were the optimal choice for developing a test automation suite around behavior-driven development (BDD) principles.

• Report Generation: A report was generated at the end of the execution of both single as well as multiple feature files. The report consisted of screenshots captured at each step, along with the actual and expected values for the assertions. We utilized the cucumber js-json reporter in combination with the cucumberhtml reporter for this purpose.

• Data Generation At Runtime: We generated data at the time of the execution by sending API calls to the database associated with the application. Upon being updated, the generated data would reflect those changes on the front-end.

• Database Assertions and Verification of Functional Flows: Database assertions were performed by verifying that changes at the back-end are reflected on the front-end. Moreover, the functional flows (such as playing or bookmarking an episode, sharing the episode to social media sites) were also verified.

• Cross Browser Testing and CI/CD Pipeline: We utilized SauceLabs for cross browser testing and Jenkins for running the CI/CD pipeline. Sauce Labs was used for server-based cross-browser testing, while Jenkins was utilized for running the code on its server.

• Performance Testing: We also undertook performance testing on the platform, utilizing an in-house tool and Puppeteer scripts. The process started with the execution of the puppeteer scripts through the in-house tool, either through peak testing with multiple browsers opened simultaneously or step-based testing with a gradual increase in the number of open browsers. After the script execution, a result json file was generated, containing multiple performance metrics. Lastly, we parsed the json file using a python script to extract the relevant metrics in the form of the data-frames.

This effort provided us with several key lessons for the future:

• Communication between the client and vendor is extremely critical; as it ensures that both parties have visibility on the manner in which the solution is being implemented.

• Teams developing such testing solutions should have the flexibility and RnD skills to find workarounds for blocker issues.

• Outsourcing to overseas teams is a perfectly viable and feasible business model - provided both the companies develop a good working relationship.