One of the fallouts from the COVID-19 pandemic is the explosion of over-the-top (OTT) streaming media services. A market that represented $121.6 billion in 2019 is expected to reach $1,039 billion by 2027 at a CAGR of 29.4%. Competition among streaming services continues to heat up as established players such as Netflix, Amazon Prime, and Starz compete for viewer attention and newer players such as Disney+, HBO Max, and Peacock enter the market.
They say content is king but when it comes to OTT services, that is only partially the case. What customers demand is a superior and consistent viewing experience and that means better quality of service. Consistent testing against relevant parameters and data points is needed to ensure the quality of service and the quality of experience for OTT customers. Quality control (QC) and performance testing experts need to assess content storage management, transcoding frameworks, and metering and control frameworks to see that they meet performance requirements.
Delivering OTT services at peak quality with rapid delivery is a true competitive advantage for streaming services. Consumers have more choices than ever and are subscribing to dozens of different OTT services. With so much choice, inferior service inevitably leads to customer churn. Testing also is essential to avoiding higher operating costs and delays in issue resolution. That’s why QC is essential to guarantee an optimal customer experience across multiple viewing platforms.
Technologies that affect OTT performance
When it comes to OTT service, multiple factors can have a direct impact on performance. Each of these systems needs to be monitored and managed to ensure an optimal customer viewing experience.
- Edge computing and CDNs: Streaming services delivered using content delivery networks (CDNs) which are responsible for caching content. Edge computing is used to maximize CDN performance, moving the processing power close to the end device. This is the first layer of logic and links requests to the API within the service architecture and provides the abstraction layer to mid-tier services.
- Load balancing: To optimize data traffic, load balancing throttles data flow, rejecting inbound requests and redirecting them to maintain throughput.
- Microservices: The backend of any OTT service consists of hundreds of independent microservices, each implementing its own business logic
- Encoding and content delivery: Every media file is segmented into chunks sent at different bit rates.
- Data pipelines: The data pipeline enables cloud storage to handle millions of viewing and UI events each second.
Performance issues with any of these platforms can result in a chokepoint that affects the customer experience. QC must be maintained throughout, and that means ongoing testing. A frictionless customer experience depends on QA testing across geographics, networks, digital rights systems, and content management systems.
How to approach OTT quality control testing
Given the complexity of any OTT architecture, the only way to provide effective QA is through automation. Automating performance testing offers several advantages.
Continuous integration practice allows OTT providers to deliver ongoing performance enhancements and accelerate time to market. It also requires contiguous testing to maintain scalability and quickly identify performance problems.
Automation also prepares OTT providers to maintain service quality during a surge or times of peak demand. Using automated testing for analytics to generate timely reports makes it easier to track audience engagement as well as performance.
Every OTT provider adopts different QC practices, including testing at various data delivery points. Some of the most common tests include assessing the playback interruption in low buffer, maximum resolution buffer, average initialization time, and so on. Automated testing delivers sufficient data sampling to identify and rectify glitches, adjusting the resolution to accommodate available bandwidth.
In most OTT systems, performance validation occurs at three points:
- Testing after encoding and ingestion: Ingested files are tested for integrity and ensure they are properly encoded for consumption downstream. QA testing is done using VMAF (Video Multi-Method Assessment Fusion), an open-source algorithm developed by Netflix.
- Testing after transcoding: Input streams are converted into output profiles that need to be tested for Quality of Experience (QoE), including quality like rate, format, syntax, loudness, etc., and for Adaptive Bit Rate (ABR), which ensures frames are time-aligned across each profile for seamless switching.
- Testing at the CDN: The final test points are at the CDN, including QoS checks. QoE checks, ABR checks, entitlement checks, player control checks, and video playback analytics.
OTT services will continue to evolve, and so will testing requirements. Broadcasters and service providers continue to dedicate spending on more and better content, but they can’t monetize that content without a reliable, high-performance infrastructure that ensures content quality to attract and retain viewers.
Testing methodologies need to be versatile to accommodate various architectures, and they must be able to identify critical problem areas that need attention. The revolution in OTT testing will come in the form of QC automation and monitoring through the delivery infrastructure, from ingestion to delivery. The future of OTT depends on better testing to provide the best possible viewing experience.
