The Fifth Elephant 2014

A conference on big data and analytics

Using Cascalog and Clojure to make the elephant move!

Submitted by Harshad Saykhedkar (@harshss) on Sunday, 8 June 2014

videocam_off

Technical level

Intermediate

Section

Crisp talk

Status

Confirmed & Scheduled

View proposal in schedule

Vote on this proposal

Login to vote

Total votes:  +3

Objective

Intent is to highlight benefits gained by using Clojure, a functional language which works on JVM and Cascalog data processing library for Hadoop. The participants will be exposed to,

  • Rationale behind Cascalog

  • Basic abstractions of this library

  • Use cases where a high level functional language and a declarative programming library helped immensely

  • Short code snippets showing extraction of interesting customer behavioural patterns using Cascalog and Clojure

  • Highlight how functional abstractions might help bridge the gap between lab culture of data science and so called production environments

Description

Machine learning practitioners often spend 70%-80% of their time on cleaning, transforming and scrubbing data. Quick iterations and ability to create large number of features out of raw data are pre-requisites for any serious machine learning activity. There is also a large gap between a skill sets and objectives of data scientists and programmers. Data scientists need to slice and dice through real world big data. Implementing all the low level plumbing of multiple, dependent Hadoop jobs is a serious impediment to this goal. On the other hand, companies cannot afford to have lab tools running, as they have large impedance mismatch with production environments.
We have seen Cascalog library and functional abstractions of Clojure providing a sweet spot. It lets data scientists focus on their job of understanding data without worrying about complicated class hierarchies or un-intuitive domain specific languages. On the other hand, Cascalog and Clojure seamlessly integrate with JVM based stack. The result is a fun and productive way to process data at scale!

Speaker bio

Harshad Saykhedkar is senior data scientist at Sokrati, a digital advertising startup based out of Pune. He has experience of applying machine learning to problems in advertising, banking and telecom sectors for past four years. He has used multiple tools (R, Python, SAS) in this journey and lately fallen in love with the Clojure ecosystem, hacking his way to create data processing tools at Sokrati. Harshad holds a master's degree in Operations Research from Indian Institute of Technology, Mumbai.

Links

Comments

  • 1
    Viral B. Shah (@viralbshah) 4 years ago

    Always good to hear about more languages!

Login with Twitter or Google to leave a comment