Understanding the value

Riccardo Bertocco and Vishy Padmanabhan, Bain & Company, USA, consider Big Data analytics in oil and gas.

While companies in industries such as logistics, consumer products and utilities are discovering for the first time the power of advanced analytics, executives in the oil and gas industry have been analysing huge volumes of data to make technical and strategic decisions for decades. In their search to find where hydrocarbons lie deep beneath the surface, oil and gas companies have invested in seismic software, visualisation tools and other digital technologies for many years.

Now, the rise of pervasive computing devices – affordable sensors that collect and transmit data – as well as new analytic tools and advanced storage capabilities are opening more possibilities every year. Oil producers can capture more detailed data in real time at lower costs and from previously inaccessible areas, to improve oilfield and plant performance. For example, they can pair real-time downhole drilling data with production data of nearby wells to help adapt their drilling strategy, especially in unconventional fields.

These analytic advantages could help oil and gas companies improve production by 6 - 8% – percentages that are typical across industries according to Bain & Company’s 2014 survey of more than 400 executives across sectors. Analytical leaders, however, are still the exception. The survey showed that only around 4% of companies across industries have the capabilities to use advanced data analytics to deliver tangible business value.

While some oil and gas companies have invested in their analytics capabilities, many struggle to get their arms around this powerful new opportunity. Senior executives usually understand the concepts around Big Data and advanced analytics, but their teams have difficulty defining the path to value creation and the implications for technology strategy, operating model and organisation. Too often, companies delegate the task of capturing value from better analytics to the IT department, as a technology project.

Instead, the business should lead analytics work because it requires cross-functional ownership and participation. Some companies commit to ambitious technology transformations in search of analytic nirvana, but these transformations often fail to generate enough value along the way.

Analyst experience shows that developing the capability to produce value from advanced data analytics is a top-level agenda item, requiring a sustained focus by the senior management team, not just the CIO. Three critical questions should form the basis of an effective advanced-data analytics strategy:

  1. Which applications of advanced analytics will produce the most value for the company?
  2. Which organisational model will help see the value achieved?
  3. Does the company have the right capabilities and talent to make the most of the data?

Producing the most value

Not all data is ‘Big Data’, of course, and not all analytics require the horsepower and organisational model that Big Data applications typically require. Still, advanced analytics can play an important role in improving productivity in unconventionals, conventionals and midstream operations in oil and gas.


Because of the vast number of wells required in unconventional production and the speed with which producers construct them, data plays a critical role in the decisions that create value. Operators make decisions every day in the field, typically with limited involvement by central functions. Analytic capabilities help producers collect and analyse data on subsurface and geographic characteristics to improve their ability to characterise shale basins in detail, with less trial and error. There are three areas in particular where advanced analytics capabilities can help give producers an edge:

Geological interpretation

Analysing the geology below the surface and comparing it against well performance can help companies improve their ability to characterise shale basins with less trial and error.

New well delivery

Better analytics can improve the way companies manage the entire process of drilling and connecting a well, reducing lag time and minimising the number of wells in process at a time. For example, transmitting microseismic, 3D imaging over fibre-optic cables can improve new well delivery performance.

Well and field optimisation

Collecting and analysing massive volumes of geologic, operational and performance data, each with many variables constantly changing, can help companies improve and optimise drilling parameters, well spacing and completions techniques, especially as they drill more wells and bring them online.


Fewer decisions and wells are involved, but producers can still improve performance with access to more data than they had before. They can also move beyond measurement into predictive tools with a range of pattern-recognition techniques that help them spot trends, intervene early and create repeatable solutions with predictable outcomes. For example, sensors deep in the wells or on drilling equipment send a constant stream of information that can help producers understand if or when a piece of equipment might fail. As these sensors become less expensive, their numbers grow into the thousands and beyond, generating large volumes of information. Integrating this data into operations improves calibration and visualisation capabilities, reducing technical risks.


Data analytics can help monitor pipelines and equipment and allow a more predictable and precise approach to maintenance. For example, sensors can indicate when equipment comes under unusual stress, allowing operators to perform preventive shutdowns or interventions that may avoid accidents or spills. By way of illustration, a leading compressor manufacturer developed custom sensor models and used predictive analytic software to actively monitor the readings provided from these sensors, which has helped it schedule preventive maintenance of equipment for its midstream customers.

An organisational model to capture the value

Knowing the value inherent in better analytics is just the start. How should oil and gas executives define an organisational model – including the right structure, processes and decision rights – that will encourage timely, cross-functional collaboration and put the right data in the hands of decision makers?

Many oil and gas companies rely on operating models that focus on functional excellence, and they develop clear handoffs from one function to the next. This works well for predictable processes that follow moderate schedules. But the model breaks down when decisions must be made quickly – as in unconventional production. The handoffs for any given shale well can involve functions like geology, drilling, completions, construction, land, regulatory and production, and any of these may be involved at different points of the well construction process. Operators might have hundreds of wells active or in development, putting the functional model under significant strain.

Consider, for example, the new well delivery process, where performance metrics such as the time from spud to hook-up or the dead time between steps require visibility into activity data from each function involved. If the functions (including land, regulatory, pad construction, drilling, completions and operations) run on different systems and rely on differently constructed data models, it becomes very difficult to have a clear, integrated view of what is happening in the field.

Some companies address this challenge by deploying an asset-based organisational model, rather than a functional one. In this model, all the key functions are deployed in the field and report into one geographically based organisational structure. In Bain & Company’s experience, the model works well at the site or division level, but may not scale to the level of a large international organisation.

This kind of fast-paced, decision-driven environment requires better planning, which is also a precondition for, and a benefit of, better data and the advanced analytics capabilities that can make sense of it. Companies with more advanced analytics capabilities are more efficient in managing the range of data (including seismic, drilling logs, operational parameters such as drill bit rpms and weight-on-bit, frack performance data and production rates) that help optimise well design and production. Each function may have a lot of data, but unless the operating model can weave it together and place a ‘single version of the truth’ in the right hands at the right time, it is difficult to improve performance.

The right capabilities and talent make the most of data

Companies that build better analytics capabilities concentrate their efforts in three areas: technology architecture, interaction between IT and the business, and hiring and retaining strong analytic talent.

Technology architecture

As in other industries, many oil and gas companies have complex, legacy IT systems that have evolved over decades, and which now contain many different islands of disparate data sets. Adding real-time, unstructured, large volumes of data multiplies the problem – but that is where valuable insights arise.

In oil and gas, key understandings emerge from linking various types of data to a well or set of wells. The business side sometimes underestimates how difficult this can be. Many systems lack unique well identification numbers, which are essential for tracking the metadata on each well. It is not always clear who should assign the ID – the reservoir engineers, the regulatory department or the land department.

In addition, many systems prioritise financial tracking and reporting, and barriers may exist between these and the operational systems that collect and manage important geologic and performance data. Wells need clear designations that are consistent across systems, but often the data remains isolated in financial and operational silos. In other words, most companies design these systems for the financial needs of the centre, rather than the operational needs of the field.

But companies can become more nimble and effective by carefully focusing their IT investments on the most critical areas and saying no to requests that fall outside that focus. Some organisations with significant legacy investments take a two-track approach that includes a long-term plan to modernise the entire IT stack while allowing a well-funded (but well-bounded) separate program that moves quickly on the most important opportunities. For example, one leading international oil company implemented a sophisticated Hadoop analytic platform on Amazon Web Services’ cloud infrastructure and limited the touch points with its legacy technology infrastructure – a strategy that helped keep it cost-effective and agile.

Business-IT engagement

The IT functions in most oil and gas companies were designed to automate back-office support functions as cost-effectively as possible. To the extent that IT enabled the business, it did so in steady, well scoped and largely predictable projects that did not require quick and cross-functional collaboration.

As that is changing, IT should organise around decisions rather than functionally siloed processes. This relationship between IT and the business should evolve into a real-time collaboration with less friction. Clearly the solution is not to build a siloed unit focused on the Big Data opportunity. For analytics to have a significant effect on the company’s performance, it must be integrated into day-to-day operations.

Analytic talent

Analytic talent is a scarce resource and the talent profile in demand is not typically found within oil and gas IT functions. IT organisations historically hired talent that had product expertise in business intelligence (BI) tools, or they hired or outsourced talent to produce periodic and mostly static BI reports and to run back-office systems. Typically, the geologists and engineers employed by the functions do not have expertise in the latest analytic tools.

Additionally, the IT talent issue in most oil and gas companies extends to the architecture group. As cloud and open-source architectures become more popular, IT groups may find themselves less familiar with the latest technologies and, more importantly, may not feel empowered to make the changes that best-in-class companies are embracing.

But oil and gas executives should avoid the temptation of trying to solve this issue just by throwing high-priced analytic labour at the problem. The right talent model should balance industry knowledge with analytic skills in a model focused on solving specific problems and identifying new opportunities. This capabilities upgrade should include people who understand open-source models, cloud technologies, pervasive computing and iterative development methodologies. Executives should keep in mind their goal of a model that enables better collaboration among domain experts, IT architects, tool specialists and experienced analytic resources.


Oil and gas companies will need to improve their analytics capabilities in order to compete in an industry where decisions are moving faster and the stakes are growing ever higher. Creating a world-class analytics capability takes time and investment, and it can only happen with a sustained focus by top management. Senior executives should avoid delegating this challenge to their IT function; rather, they should work closely with IT leaders to transform their companies into data-driven organisations. Now is the time to develop a plan that maps an organisation’s ambitions against its capabilities – and describes the path toward a world-class, advanced analytics capability.

Adapted by David Bizley

Published on 15/04/2015

Get your FREE Oilfield Technology magazine »

Get your FREE trial of Hydrocarbon Engineering magazine »

Get your FREE trial of World Pipelines magazine »


Recommend magazines

  Oilfield Technology