Get Chief Innovator
Delivered To Your Inbox.

big data and innovation

Big Data Analytics and Innovation

Data has been at the forefront as far back as one can remember, dominating every piece of human existence and transforming the way organizations innovate. Now, with the growth of information, Big Data analytics has taken center stage in most business circles. 

Big Data—basically a term for large, complex data sets that go beyond traditional processing approaches—is growing at a compound annual growth rate (CAGR) of 42 percent. When it comes to adding business value with Big Data, almost 75 percent of organizations are keen on including it into the scheme of things.

Growth of Big Data

With the uncertainty and complexity of today’s scenarios, an organization’s ability to make sense of the data will be a key differentiator. Product life cycles are diminishing rapidly, which calls for getting the new product not only right the first time but also faster to market. So how can Big Data help organizations innovate competitive and desirable products and services faster? The answer lies in looking at customer needs holistically.

Research shows that 80 percent of all new products fail due to lack of understanding of customer needs. Although lots of data is being collected, efforts to make sense of the data and build new products and solutions that respond to customer needs are limited. There’s a missed opportunity to overcome some of the emotions and frustrations that a customer might be repeatedly communicating. For example, as a customer, how often are you irritated with the airline for not delivering your priority checked bag? Or with the hotel for not placing your preferred choice of pillow in the room? 

Big Data Solutions

A range of Big Data solutions are available that are designed to help organizations analyze the mounds of information. Some of these tools are classified in the adjoining graphic where on one end you have Fast Data tools, then Big Analytics and finally Deep Insights. The quality of these tools in terms of moving beyond the obvious improves as we move from left to right. 

Big Data Solutions

Fast Data techniques, such as Hadoop and Teradata, provide the ability to see most of what you know in a short enough time in order to quickly take actions. These techniques have grown exponentially, barely keeping up with the growth of Big Data. Big Analytics tools like SPSS and SAS, on the other hand, help to turn information into knowledge using a combination of existing and new approaches. Finally, Deep Insights take us a step ahead where all the information at hand is considered, whether qualitative or quantitative; analytics processes are applied to it; and new knowledge and insights get generated for the business-specific situation.

Currently, organizations are collecting lots of qualitative data but are unable to uncover insights from it that will help them succeed at innovation. Hence the question arises: How can one effectively make sense of and use the information provided by the customers on their products and services? With all the noise around Big Data, how does one use it to extract information to drive innovation? The larger challenge will be to segregate the signals from the noise so that a better understanding of the customers future needs clearly emerge that will further enable organizations to direct their overall strategy and the resulting innovation efforts in a much more effective manner.

Examples of Big Data Driving Innovation

That may sound difficult, but it’s not impossible. Investments in Big Data analytics do result in returns. Companies can weed out obsolete techniques and bring in programs that actually work, such as real-time analytics gathering for regions, allowing companies to focus readily on profit and usability in the long run. Be it hidden patterns or unknown correlations, Big Data analytics can help identity data sets for digging deep into the market trends.  

American Express, for example, started looking for indicators that could predict loyalty and developed sophisticated predictive models to analyze historical transactions and 115 variables to forecast potential churn. The company believes it can now identify 24 percent of accounts that will close within the next four months. Wal-Mart relies on text analysis, machine learning and even synonym mining to produce relevant search results. They say that adding semantic search has improved online shoppers completing a purchase by 10 to 15 percent; that converts to billions of dollars. These and several other examples are making a strong case for Big Data analytics even on the ROI front.

Using PanSensic to read between the lines

Some of the leading companies in the world like the Tata Group, Dr. Reddy’s, Reckitt Benckiser, Unilever and Philips are trying to marry data analytics with innovation by providing solutions that give deep insights into what the customers actually need. PanSensic is a rare multi-dimensional sentiment analysis tool that uses smart lenses to sift through Big Data and help innovators pick up weak signals of changing consumer emotion. Using PanSensic, one can read between the lines of narrative for a brand new level of understanding. 

Making Sense of Big Data – Necessary for Growth

The collection and analysis of Big Data will not only inform and direct innovation efforts, but also serve as a critical input to the strategy creation process. Increasingly it will become a necessary competency for people within organizations as they identify their growth drivers and build new products and services. 

Just as computer literacy has become a necessary skill in the last century, making sense of Big Data will become a necessary skill set in the current century. And those companies and individuals that build this capability early will have the first mover advantage. 

Naresh Shahani is the managing director of India at BMGI. You can connect with him on LinkedIn here.

For more information on building your data analytics capability, explore BMGI’s DataMaster program

To receive Chief Innovator in your inbox, please sign up for our newsletter. We only send out a notice when a new blog is posted. You can also join us on LinkedIn at our Chief Innovator Group where you'll find the latest posts and can query the group or start a discussion.

Interested in writing a guest blog? Contact us at