Even in today's data-hungry analytics environment, collecting and keeping every tiny granule of retail data – down to the level of each individual transaction and Web click – might seem like overkill. But Sears Holdings has discovered that data storage costs have gotten low enough, and analytical tools have grown powerful enough, that this approach to Big Data has shortened the time needed for analytics projects by 60% to 70%, while also improving promotion conversions, lowering inventory levels and boosting sales.
"It's dramatically reduced the time needed for these projects," says Sears Holdings CTO Philip Shelley, as part of a panel discussion Sunday, January 13 at the NRF Big Show, "Big Benefits from Big Data." "Because we have large granular data sets without the delays of finding data, it's taken weeks out of campaign delivery times.
"It's involved a huge mindset change relevant to keeping data," he adds. "We used to only keep aggregates of data and throw away the detail, because it had been too big. Now we don't throw anything away, theoretically, ever.
"We 'de-normalize' the data and load it into very large flat tables, in its most granular detail and from many sources," he explains. "By loading the detail once, we then can transform and re-use it as many times as we want."
For example, if Sears wants to analyze the role of seasonality in the supply chain at an item level, "we have eight years of history at a detailed level. We can do the same things with pricing, because we have much more history than ever before, at the customer and the product level," says Shelley. "This allows us to make more meaningful offers and achieve better conversion with our campaigns, and also to lower inventory levels but achieve higher service levels. We saw lower inventories in 2012 but higher sales, and that kind of supply chain optimization is only possible with that level of detail."
Shelley, who is also CEO of MetaScale, credits Sears' use of a Hadoop system for providing a database "where you can affordably keep [data] in one place and apply tools to it, and consume it in an easy way. For us, Hadoop is the new mainframe – I preach that to my people all the time, and we treat it like one and govern it like one."
In fact, Sears had to go through a learning curve with both sourcing of data and its governance. Asked about key lessons learned in its Big Data journey, Shelley identifies "inadequate sourcing and governance. The only way I could solve it was to have a dedicated group of people that did nothing but data governance. It revolutionized the quality by having this group that 'owns' the data. Having that responsibility federated out didn't work."
The results have been worth the effort, he adds: "People want to use data in minutes, and to have it in one place. That's the paradigm shift – it's in one place that we can leverage."
For related content: JCPenney, Best Buy, Sears on 'Do or Die' List for 2013
Sears Expands Analytics Database
Sears Launches Cloud-Based Big Data Analytics Subsidiary