The Fifth Elephant 2014

A conference on big data and analytics

Live analytical dashboards at scale - SQL style

Submitted by Shashwat Agarwal (@shashwatag) on Monday, 26 May 2014

videocam_off

Technical level

Intermediate

Section

Full talk

Status

Confirmed & Scheduled

View proposal in schedule

Vote on this proposal

Login to vote

Total votes:  +10

Objective

How to build a real-time, analytical dashboads that can enable business take decisions at scale? There are various technologies out there that fill one or the other use case - right from horizontally scalable queues such as kafka, stream processing systems such as storm, data stores such as openTSDB and druid that can provide dimensional lookup on large amount of data and visualisation libraries such as d3, cubism to view them. But there is a lot more that is never discussed if not for the details. In a service-oriented architecture, where dimensions and measures are coming from different sources, where some dimension is larger than the number of seconds in a year (for those who are wondering it is ~32 M), ensuring liveness and correctness at every minute is what Fireball is all about.

Description

Fireball is a stream processing engine at Flipkart. It powers real time analytical dashboards to enable business take time-sensitive decisions, at scale. Fireball can process millions of events (with flexible, json-like schema) per hour that require:
executing custom process (usually SQL-like) to derive business metrics from the incoming events over large number of dimensions (on an average 10 dimensions for each measure) * with very low latency and ensuring correctness all the time (enabling time-sensitive decision making)

So how do you build such a system? How do you store such a large amount of time-series data to ensure roll-ups, drill-downs on different dimensions? In this talk we'll go over the transformation of a standard stream processing platform and a CEP library into Fireball.

Speaker bio

I am Architect @ Flipkart and am part of the Data Platform effort.

Comments

  • 1
    Govind Kanshi (@govindsk) 4 years ago

    Dear Shashwat - This sounds interesting. I am assuming you would be explaining why/how you ended up developing something from scratch. Other issues which if you could cover could be - is stream(considering you mentioned per hour analytics) really required or columnar storage/appliance could have been useful to tackle data volume/scale. Do you have plans to share it with others so that they could either use it/develop it further ?

  • 1
    Shashwat Agarwal (@shashwatag) Proposer 4 years ago

    Hi Govind, I have updated the objective with some of the technologies that we explored/used. Please have a look.

  • 1
    Govind Kanshi (@govindsk) 4 years ago

    Thanks Shashwat - for updating the "objectives". We certainly have little bit more clarity. The talk is shortlisted, let us discuss it when we have the call. Thanks a gain for proposing this.

  • 1
    Regunath Balasubramanian (@regunathb) 4 years ago

    This is going to be a really interesting talk, given that Fireball is at the core of Flipkart's effort to build "Intelligence @ scale". For the technologies-inclined lot, here is an opportunity to see a system that uses Hadoop, Storm, Kafka and the like in a production setup - and where business decisions are influenced by data collected and analyzed on this system. A good from-the-trenches talk overall.

  • 1
    Hari Prasanna (@mostlycached) 4 years ago

    Hi Shashwat, could you put up the presentation for your talk online?

  • 1
    Shashwat Agarwal (@shashwatag) Proposer 4 years ago
  • 1
    Shivam Gupta (@shivamg) 4 years ago

    Hi Shashwat, As discussed in this talk that we can add bolts in running topology can you please help me in providing tutorial for it

Login with Twitter or Google to leave a comment