At present, the corporate world is enamoured of “Big Data” or Analytics, which is regarded as the new “Management Revolution” Harvard Business Review has come out with a special issue on it. Is it just another hype which will fade out? How much of the “revolution” is hype and how much of it is real?

Deluge of Data

Harvard Business Review brought out a special issue on this theme on October 2012. The editorial in the issue states “Yes there’s a lot of hype. But companies that make a sophisticated analysis of the huge data streams now available can unlock deep insights and value – without having to make major new investments in technology.” This reminds me of a similar phenomenon in the 1970’s when the Quantitative Techniques entered into management. More or less similar claims are made, like for example intuition is no longer needed for decision-making and computers and mathematics will soon replace human judgement or intuition. But after a few years of flirting with “quants” management thinkers again started talking about the importance of intuition in decision-making.

Let us first listen carefully to experts on big data. First question is what it is all about. Andrew MacAfee and Eric Brynjolfson; data scientists from MIT and Sloan School of Management, in their article in Harvard Business Review “Big Data: The Management Revolution”, state that the main objective of big data movement is to “gleam intelligence from data and translate that into business advantage”. There is undoubtedly a need for such a movement because in recent years there is a tremendous increase in the volume, velocity and variety of data. This has its advantages as well as drawbacks. Too much of data becomes difficult to handle but if we can find a way to manage this big data which represents facts of the world in movement, it holds the key to a better understanding of the changing world. If we can find a way to manage this data creatively, which means to discern the hidden patterns, answer questions or indicate solutions, then it is a big help in decision-making. This is what Big data movement can do.

Data and Insight

But can it entirely replace intuition? MacAfee and Brynjoffson state: “Data-driven decisions are better decisions… it’s as simple as that. Big data enables managers to decide on the basis of evidence rather than intuition.” However, in the last part of their article in HBR they say: [“Companies succeed in big data era not simply because they have more or better data but because they have leadership teams that set clear goals, define what success looks like and ask the right question. Big data’s power does not erase the need for vision or human insight. On the contrary we still must have leaders who can spot a great opportunity, understand how a market is developing, think creatively and propose truly novel offering articulate a compelling vision persuade people to embrace it and deal effectively with customers, employees, stockholders and other stakeholders. The successful companies of the next decade will be the ones whose leaders can do all that while changing the way their organisations make many decisions”.]

So Big data can’t be a substitute for intuition and insight. Almost all the factors listed above involve decisions which require human insight. Big data can’t be a substitute for it though it can help in arriving at a better decision by providing well-structured and appropriate information based on facts. However when the decisions involve a deeper or wider understanding of human factors like motivation or well-being or a holistic comprehension of the totality of life, then data becomes less important and human insight acquires a much greater significance.

Right decisions depend on right understanding which can be at many levels. At the first level, there is the understanding of life as it is, made of the known, visible, manifest, and predictable and the mobile actualities of life. At a deeper level is the unknown, invisible unmanifest, dormant, unpredictable and the flux of future possibilities. At a wider level is the understanding of the ecological dimension, which means connectedness, interdependence, unity and wholeness of life, which is necessary for knowing the long-term consequences of our decisions. At another level is the realm of values, the ethical, aesthetic and spiritual dimension, and the factors which lead to human well-being and fulfilment, individual and collective, inner and outer, which may involve reconciling many dualities, dilemmas and conflicts thrown by life.

Big data, analytics, and whatever future development in IT which enhances the capacity to process information can only help in better understanding at the first level of visible facts and when combined with Artificial Intelligence provide a certain level of insight into the second level of the invisible like for example hidden patterns behind facts. But at all other deeper, wider or higher levels, analytics may not be of much help and we need a deeper and higher intuition. In the future, as we grow as a race in our inner consciousness, more and more of these higher non-material dimension will begin to manifest and get incorporated into human life. This is already happening. For example, factors like ethics, values, long term vision, social responsibility, sustainability, holistic perspective and human well-being are now increasingly recognised as key to success in the future world. All these factors and many more which may come in the future will throw up problems, situations and decision-contexts which cannot be solved entirely by analytics but requires a higher intuitive understanding.

Even in the first level, right decisions require a certain subjective freedom from excessive self-interest. Without this freedom, big data will be used to support personal decisions driven consciously or subconsciously by self-seeking motives. Secondly, quite often right questions are the spark which ignites creativity and innovation and analytics can’t ask questions. As McAfee and Brynjoffson quote Pablo Picasso as telling “Computers are useless they can only give you answers.”

M. S. Srinivasan