“Data has become an increasingly critical component to business success,” said “Modernizing Data with Strategic Purpose,” a recently published report by the technology consultancy Thoughtworks and MIT Technology Review Insights. “The ability to make timely and informed decisions, create tailored customer experiences, and identify new opportunities has enabled the rise of today’s leading digital companies. And with the recent explosion of interest in AI, the demand for data has only increased. No matter what an organization hopes to achieve, success is impossible without ready access to high-quality data. Despite advances in technologies, extracting and transforming enterprise data into a usable asset remains a tremendous challenge for most organizations.”
The report is based on a survey conducted in early 2024 of 350 senior data and technology executives from organizations in the US, the UK, Germany, Singapore, and Australia that earn $500 million or more in annual revenue, with almost half (47%) having revenue of $10 billion or more. Eight industries are represented in the survey: financial services, energy, health care and life sciences, manufacturing, public sector, publishing and media, retail, and travel and transport.
The survey sheds light on how enterprises are adjusting to the increasing importance of data in our emerging data-centric AI era. The data-centric AI era started about 25 years ago when the explosive growth of the internet led to what’s become known as big data, that is, the availability of huge amounts of digital data, including text, voice, and images. Digital data has continued to grow exponentially, and is now estimated to have reached around 150 zettabytes (1021 bytes). All these data has enabled AI to make great strides in the 2010s with multi-layered deep learning algorithms, and more recently with foundation models, including generative AI (GenAI), large language models, and chatbots.
“As AI pushes data modernization to the top of more organizations’ agendas, it will also become increasingly important to ensure that the data strategy is closely aligned to the broader business strategy and data leaders are able to clearly articulate how data and analytics can help business units achieve their objectives,” notes the report. “This will not only ensure critical buy-in, but also put the organization in the best position to achieve its goals and lay a solid foundation for continuous improvement, evolution, and value creation.”
Let me summarize the survey’s key findings.
Most organizations have a data strategy, but this is a recent development. The majority of organizations (78%) said that they have a fully developed data strategy. Not surprisingly, the results are different for organizations of different sizes. The percentage of larger organizations (annual revenue of $10 billion or more) was 85%, compared to 72% for smaller organizations (annual revenue of less than $10 billion). However, their data strategy initiatives are a recent development, with only 27% of executives saying that they’ve had a data strategy for more than two years, — 44% in larger organizations and 12% in smaller ones.
Data strategy alignment is incomplete. 39% of responding executives said that their data strategy is in complete alignment with their business strategy goals, (36% larger, 42% smaller); 38% said that their data and AI strategies are closely coordinated, (43% larger, 15% smaller); 40% said that different parts of the business have different strategies, (39% larger, 41% smaller); and 42% said that the data strategy was developed exclusive by their data and IT teams with minimal involvement from other parts of the business (48% larger, 36% smaller).
Which of the following best describes your organization’s current approach to its data capabilities? 23% of organizations have modernized multiple elements of their data capabilities within the past two years (43% larger, 6% smaller); 31% are currently modernizing their data capabilities (44% larger, 19% smaller); 23% are planning to modernize their data capabilities within two years; 19% are currently evaluating whether they need to modernize elements of their data capabilities; and 2% have no plans to do so.
What’s holding back data modernization? 44% of respondents said that regulatory compliance was their top impediment to modernization (49% larger, 39% smaller); 44% cited security concerns as their second top impediment (51% larger, 38% smaller); and 40% cited the high costs of modernization as a third impediment (33% larger, 46% smaller).
What are your organization’s main motivations for modernizing its data capabilities? 46% ranked improved decision-making across the business as the top reason for modernizing their data capabilities (36% larger, 55% smaller); 40% cited support for AI use cases as their second most important reason (58% larger, 23% smaller); and 38% cited reducing their environmental footprint as their third reason (54% larger, 23% smaller).
Which elements of your organization’s data capabilities are least ready today to support your data modernization objectives? 41% of responding executives cited data quality as their top concern (29% larger, 53% smaller); 33% cited data timeliness as their second major concern (34% larger, 31% smaller); 27% said that talent and skills to manage data was another major concern (36% larger, 19% smaller); 26% mentioned data strategy (13% larger, 38% smaller); 25% mentioned data architecture ( 16% larger, 33% smaller); and 25% mentioned data governance (29% larger, 21% smaller).
When will your organization take the following measures to modernize its data capabilities?
- review and update data governance: 45% have done so in the past two years, 45% will do in next twelve months, 8% plan to do in 1-2 years;
- adopt a new data architecture: 36% in the past two years, 52% in the next 12 months, and 8% plan to so in 1 to 2 years;
- widen the use of cloud data services: 39% in the past two years, 54% in the next 12 month, and 11% in 1 to 2 years;
- decentralize data storage and management: 23% in the past two years, 44% in the next 12 month, and 19% in 1 to 2 years;
- revise the data organization structure: 15% have done so in the past two years, 59% in the next 12 month, and 22% in 1 to 2 years;
- re-engineer data processes: 13% in the past two years, 62% in the next 12 months, and 22% plan to do so in 1 to 2 years;
- consolidate the number of data repositories: 12% have done so in the past two years, 66% in the next 12 month, and 16% in 1 to 2 years.
What are your organization’s top priorities for improving the quality and timeliness of the data used by different parts of the business? 48% said empower cross-functional data quality teams to enforce quality practices, 47% said implement DataOps, 35% cited improve data validation processes, 31% cited enterprise-wide training on data quality and standards, 29% said redefine data quality standards, and 25% said bring automation into data quality management.
“This research offers lessons for enterprise heads of data and technology that are embarking on, or considering, the modernization of their data estate, said the report in conclusion.” Foremost are these four lessons:
Keep AI goals in perspective. “AI offers substantial promise to add value to the business, but other modernization objectives shouldn’t get lost in the excitement. Delivering higher-quality data faster and more securely serves not only the needs of AI models but of many other business-critical systems, as well as other emerging technologies.”
Data leaders must demonstrate the business value of modernization. “A comprehensive data strategy is one that aligns fully with the business strategy, and data strategy and modernization approaches developed in isolation are sure to lead to wasted effort and resources. It is incumbent on senior data and technology leaders to understand how data can help business units achieve their objectives.”
Software engineering practices are coming to data. “The emergence of cross-functional data teams, DataOps practices, cloud-based solutions, and a focus on data as a product, for example, echo agile practices that have become standard in software engineering over the last two decades. As organizations seek to accelerate the delivery of value from data, they find that adopting modern data engineering practices can address challenges around data quality and usability.
Approach change prudently, but keep moving. “Data modernization may call for complex changes around a company’s organizational structure and data architecture. Many enterprises find it tempting to delay these until later stages of the project, when more is known. But modernization initiatives gain momentum by demonstrating early value, which means that leaders may benefit from an agile mindset, taking early bets and being prepared to pivot.”
Comments