In the age of big data, new ways to gather insights from data are proliferating. eBay, as a data-driven company, leverages many tools for analyzing big volume of data to uncover hidden patterns in both transactions and customer behavior. Now, an open source big data tool called Kylin, born at eBay, has graduated to become a top-level project stewarded by the Apache Software Foundation.
eBay has more than 800 million active listings, and a tremendous amount of activity data is generated everyday. Designing and developing state-of-the-art technology to support big data analytics is a top priority for us, and that is why Kylin was created.
In late 2013, eBay started investing in a technical initiative, code-named Kylin, targeted as a solution to provide multi-dimensional data analysis (OLAP) capability on the Hadoop platform. eBay then decided to open source it and share it with the community.
Kylin officially became an Apache Top Level project on Nov 18 of this year — a major milestone. It is the first open source project from eBay to earn this honor. Apache Kylin also received a Bossie Award from InfoWorld in September.
Kylin has succeeded as a big data analytics tool for several eBay teams, including the Data Services and Soltuions(DSS) team, eBay Classified Group, Gumtree, and Search Engine Optimization (SEO) teams. By leveraging Apache Kylin, eBay’s SEO dashboard refresh latency has been greatly reduced.
In addition, Apache Kylin is now used by many other companies, including Baidu, Expedia, JD.com, vip.com and China Mobile.
"Kylin is the best OLAP engine on big data so far," said Wilson Pang, Senior Director of Behavior Insights. "In eBay, we collect every user behavior on any eBay screen. While other OLAP engine struggles with the data volume, Kylin still enables milliseconds response. Moreover, we also start to leverage Kylin for near real time data streaming storage. All together, Kylin serves as a critical backend component for eBay’s product analytics platform."
The press continues to grow for Apache Kylin, with a recent feature in ADT Magazine.