Data Boom

Data Boom

The widespread adoption of IIoT devices has translated into a surge in availability of high-quality actionable data, leading to the further evolution of predictive analytics. From being used primarily as a statistical modelling tool in the early 2000s by the finance and insurance industries, predictive analytics has evolved into a mainstream practice. The surge of data that became available to enterprises over the last decade with the boom in big data and machine learning meant that systems and tools had to get smarter to be able to leverage this data for critical insights. The coming together of multiple forces—cloud computing, IoT, distributed computing and the growth of the open-source ecosystem—all collectively contributed to creating an environment where the world had access to more data than it knew what to do with. Technology companies doubled down on creating newer, smarter algorithms and tools to harness this data.

The evolution of AI-based tools and the integration of AI into these analytics systems in the later part of the 2010 decade gave a huge boost to the efficacy and accuracy of these predictive systems. Developments in neural networks led to the evolution of tools like natural language processing (NLP) and image recognition software, opening up a whole new area for analytics. Companies could now parse through data across different formats and different platforms and still gain insights from it. Earlier, this data had to be in a specific format for companies to be able to segregate and analyse it. NLP made it possible to glean insights from unstructured data sources like newspaper articles and social media posts and even images, something that analytics systems could not use earlier. It allowed companies to tap into a vast source of information on what consumers were talking about and how they felt, not only about the company or product but other matters as well, enabling them to cater their brand messaging and communication to make it more appealing to their target audience.

Enterprises also started experimenting with chatbots since it was easier for consumers to interact with them using conversational language.

On the enterprise side, predictive and prescriptive analytics became a part of larger enterprise functions with better collaboration and insights across departments. Financial firms and banks started using analytics to analyse transaction patterns and used these to create fraud detection systems, while retailers used it to improve their demand forecasting and inventory management. Infrastructure firms could now use equipment running data to predict equipment failure, resulting in the emergence of predictive maintenance for power grids and other large infrastructure resources.

This scale of growth came with a whole new set of challenges. All around the world, consumer groups and governments woke up to the risks that unfiltered access to personal data could pose. There were also concerns around data manipulation and use of the generated data to influence outcomes, often with serious consequences. These were the early days of data privacy regulations, which were led by lawmakers in Europe. Talks around the need for a strict security framework governing the collection and use of personal data by enterprises gained strength. By 2016, Europe had put a clear structure in place that eventually became the General Data Protection Regulation (GDPR). Over the years, this has gone on to serve as a base model for several other countries that have wanted to adopt a data security framework.

Concerns around ensuring compliance and data security came to the forefront with tech firms and enterprises who were using these tools both doubling down on improving their cybersecurity frameworks and pledging to improve data security. Conversations around the role of bias and ethics in AI models also entered the mainstream.

The start of the 2020 decade saw the world grappling with the Covid-19 pandemic, which brought normal life to a virtual standstill for large parts of 2020 and 2021. This, however, translated into a steep uptick in demand for technologies of all kinds as everything, from workplaces to education to shopping and socialising, went online.

IT companies doubled down on finding newer and more efficient ways for people to live their lives online. There was a boom in collaboration tools like Microsoft Teams and Zoom, while companies also tried to find efficient ways to remotely manage their people as well as infrastructure resources. Investment in AI increased, and AI-based applications started finding use across an ever-increasing number of categories. For marketers, this meant the opportunity to provide hyper-personalised experiences and recommendations for each specific customer, often resulting in a bump up in sales and demand.

Alongside, advances in IoT and edge computing also led to the mainstreaming of platforms like OpenAI’s GPT or ChatGPT and Google’s Gemini. The high quality of data insights available found great use during the pandemic, speeding up the vaccine development process, mapping disease spread, modelling disease outbreak and predicting future disease hotspots. It also helped identify markers of high-risk individuals, helping prioritise their care and optimising the use of stretched medical resources. AI-based modelling has since been implemented in several other areas as well, like supply chain management and logistics, helping companies plan and track their supply chains more efficiently.