There are two major obstacles to the big data industry. The first one is widely known, which is the privacy issue. The second is the complicated and lengthy preparation needed before data can be used, and the preparation process is still largely manual.
That reminds me of the long hours Taobao staff in the data department had to put up with. One pregnant team leader broke into tears and said she could not take it anymore
Worse, despite the long working hours, the business development department remained dissatisfied with the progress, complaining data update often failed to keep up. In the worst case, it took three months to get certain data updated.
I began to think of a better way to handle the situation, a system that is more consistent, efficient, disciplined and stable. I started to apply product management experience to data science.
Previously, demand for data development came from all directions, which resulted in overlaps. A standardized approach was lacking. From data collection to processing, different teams had different ideas. It was totally chaotic and could only get worse as the company expands further.
Finally, the chance to overhaul the system came along with the popularity of smartphones and their huge impact on data structure.
I decided to give up the original data system after thorough discussion and build a new one, which was called One Data. It took two years to complete the project.
One Data has three important considerations — data, algorithm and product — and these three elements are tightly-knit.
A versatile data system forms the basis for business development. Without one, it’s just a matter of time a company will run into trouble.
This article appeared in the Hong Kong Economic Journal on Nov. 29
Translation by Julie Zhu
[Chinese version 中文版]
– Contact us at [email protected]