Airtel XStream Play uses QuestDB for real-time engagement and device insights

Introducing Airtel XStream Play

Airtel XStream Play is an "Over the Top" (OTT) media provider, offering high quality media streaming services. Millions of subscribers connect to XStream Play every month to watch their favourite movies, TV shows and - of course - Cricket and football matches. As a subsidiary of Bharti Airtel, they are part of a leading global telecommunications company with operations in 18 countries across Asia and Africa.

XStream Play is the fastest growing aggregator platform in India, with over 100 million users. Their library features a blend of Indian and Western content from partners such as Sony LIV, Lionsgate Play, Hoichoi, ManoramaMax, Ultra and much more.

Unpacking the XStream Play analytics opportunity

The Airtel XStream Play engineering team wanted to expand their user engagement analytics. As an "Over the Top" platform offering rich video media, everything happens by time. What time something was watched, how long it was watched, which time segments, and from which device, are a few of the questions which are of great interest to the team and their clients. These data points, which are pulled from a regionally distributed content delivery network, inform technical concerns like capacity planning and spending, and also creative concerns like which content is engaging, and where it might drop off.

Multiple clients, regional points-of-presence queued and batched into QuestDB

Why Airtel XStream Play chose QuestDB for monitoring user engagement

To try and capture their rapidly growing bursts of time series data, the Airtel XStream Play team began with Elasticsearch. For lower traffic, it worked. But once the team exceeded 100 million users, Elasticsearch failed to properly aggregate data. The team also found major latency issues as Elasticsearch struggled to manage the ingestion throughput.

XStream Play is growing fast. Current ingestion peak is ~500 million records per day, or 6,000 per second. For their analytics engine to fulfill both its current and future potential, the underlying database needed to be able to handle significantly higher throughput. That is what brought them to QuestDB.

Ajay Pilaniya, the Senior Engineer at XStream Play responsible for the implementation, shares how the team chose QuestDB:

"We initially ran benchmarks on TimescaleDB as a replacement, but there were major latency issues," says Ajay. "During that time we found a benchmark comparison between TimescaleDB and QuestDB during a google search and from there we started exploring QuestDB."

While TimescaleDB is a popular choice for time series given its close coupling to PostgreSQL, it was not able to meet the needs at the scale of Airtel XStream Play. "QuestDB was able to ingest our records much faster than any other database," adds Ajay.

TimescaleDB vs. QuestDB vs. InfluxDB via TSBS benchmark

Raw performance comparisons are vital for any engineering team deciding on a robust and reliable time series database. But until implementation begins, it is difficult to understand how a new technology will work for any unique, high volume workload. There are bound to be hurdles. Responsive support then becomes essential. Says Ajay: "We faced some challenges but developers on QuestDB slack channel were very helpful due to which we were able to finish our proof of concept."

How Airtel XStream Play uses QuestDB

Airtel XStream Play takes advantage of both the high-throughput ingest and high-performance querying of QuestDB. Now, searching from a very large corpus of information requires only a small selection of precise queries.

For example, consider Airtel XStream Play's diverse array of potential clients. Clients can look like any type of smartphone, smart TV or personal computer. Knowing what a device is capable of informs the team as to properties like available network bandwidth and screen resolution, so that content can be compressed or encoded and thus delivered more efficiently.

To determine this, the following QuestDB query returns clean data, and provides aggregate sums:

SELECT
  Upper(userdevice) AS UserDevice,
  Upper(userdevicebrand) AS UserDeviceBrand,
  Upper(useros) AS UserOS,
  Sum(totalrequestids) AS RequestId,
  Sum(totaluserids) AS UserId,
  Sum(segmentcount) AS SegmentCount,
  Sum(totalfilesize) AS TotalFileSize
FROM
  'device_analytics_'

Note that the example query contains no WHERE condition. It scans the entire table and returns all rows. This is a testament to QuestDB's performance capabilities within large workloads. The end result is a fast, flexible and functional internal dashboard.

Content analytics provides another strong example. The following SQL statement creates a table for content analytics data, which includes properties of the content that was accessed, the number of requests made for the content, and the number of unique users who accessed the content.

create table IF NOT EXISTS content_analytics_ (
  ts TIMESTAMP,
  ua TIMESTAMP,
  ca TIMESTAMP,
  ContentId SYMBOL CAPACITY 50000 CACHE,
  StreamingPartnerId SYMBOL CAPACITY 500 CACHE,
  TotalRequestIds long,
  TotalUserIds long,
  SegmentCount long,
  TotalFileSize long
) timestamp(ts) PARTITION BY DAY;

The query provides fast delivery of the rich data needed to visualize and better predict network volume. Consider the value of the following chart when planning for network capacity over specific windows, like cultural or seasonal events:

Demonstrating streaming users and hours

XStream Play is an at-scale media streaming service generating sustained and high throughput event data. To remain competitive, it is essential to act on data insights during growth. With QuestDB, Airtel and the XStream Play engineering team get everything they need without worrying that their storage engine will break as they keep growing. It's just rapid ingest, and simple, fast queries.

Ajay and the team at Airtel XStream Play have since been running QuestDB in production for over a year. Initially, QuestDB met most requirements, however the team had some specific needs that were not yet available.

"Most of the requirements were fulfilled by existing features and one missing requirement which I raised is also live now," says Ajay.

The missing piece? Data deduplication, which is live as of QuestDB 7.3.