Join us April 25 - 27 in The Woodlands at Enverus' 2022 EVOLVE Conference
Learn More

Big Data or Small Data – in Today’s Oilfield Everything is Important


If the title didn’t grab you, you either don’t work in the oil industry or your existence in the industry is amazingly unique and sheltered. For the rest of us, data is the lifeblood that allows us to find and produce oil and gas.

I’m not sure there is another industry that collects more data than the oil and gas industry – both in volume and in detail.

Like many companies that serve the oil and gas industry Drillinginfo does many different things in support of our customers. We do analytics and provide high end answers to our customers. We provide desktop solutions so customers can analyze data and come up with their own answers to tough questions. However, we sometimes take for granted the thing that allows us and our customers to do all this analysis is – the data. Data is the foundation of Drillinginfo, it is where we came from and it is the fuel that powers our decision engines.

In this post (part 1 of two) I’m going to give you a tiny glimpse into the world of oil and gas data at Drillinginfo – there are hundreds of talented and gifted people at Drillinginfo that work tirelessly collecting, analyzing, processing, error checking and loading data into our databases.

Data – where does it all come from?

It is amazing all the places we get oil and gas data from. You might instantly think of federal, state, and local/county governments. However, that is just the tip of the story. Drillinginfo is a worldwide corporation and we collect information on oil and gas operations in almost every part of the world.

In the early days of the industry oil scouts traded information on wells and field production. The early pioneers of the industry quickly realized that the more data they shared the more they learned. Today there are still oil scouts, but regulation and reporting at a state level here in the US makes a lot of that information better available.

Internationally, however, that is not the case. Drillinginfo has a team of amazing individuals who collect information from all over the world. Our team of international scouts are based around the globe and are bringing detailed regional coverage into the DI database every week [click here for a free example international scout report].

Here in the US, of course, we get a lot of information from the state and local governments whose job it is to regulate the industry. Some of this data comes to us digitally, but a lot of it comes to us on paper or the digital equivalent of paper – an image file. Most state and county offices still keep track of wells and leases – on paper. In most cases, we collect this data by scanning it and converting the resulting image to meaningful information that can be searched, indexed, posted on a map and analyzed.

What a lot of people don’t know is that when we find problems with this data we often correct it and share it back with the state agency.

We also work with vendors and have partnerships with other data collectors. Not everyone can be an expert in everything so when there is a vendor who has expertise and passion for quality data we will often partner with them (if it is in the best interest for both companies).

It is also amazing how much data is being given to us. As the oil and gas industry goes digital there is great pressure to get rid of paper records. This is happening at the same time many in the industry are thinking about retiring. We have a large processing facility just west of Austin where we process and scan logs and scout tickets so they can be saved from the paper shredder. The historical information about past wells sometimes can be more valuable than recent activity.

The number of documents we handle each week depends on activity and the specific information we are collecting; at times, however, Drillinginfo has processed over 2 million documents in a single week.

Data – what do we do with it?

In a nutshell – virtually all Drillinginfo data goes on the web for our customers. When possible or practical we try to make available any document that was used in the collection of the data. For example, copies of the actual drilling permits are often available as an image in case there is any question you can easily refer to the source document.

There are dozens of teams inside Drillinginfo whose job it is to handle a specific type of data such as lease polygons or drilling permits. When it comes to drilling permits, for example, we collect information from all states in the US that have oil and gas operations. Each state has its own permitting form and rules and regulations that surround the permitting of a well. There are about 35 states in the US that have active oil and gas operations. Add 200+ countries and joint country areas and this is a huge number of moving parts and pieces that are all different just to keep track of one thing – new wells across the globe.

Like a well-oiled machine Drillinginfo adds this information from all over the world to its repositories and makes it available almost the instant we add it. The web makes it easy and users have nothing to do to see the updated data – other than perhaps hitting refresh on their browser.

Permits and new wells are just a tiny example of the data we are constantly collecting – well events, directional surveys, completions, completion details, plugging, raster logs, LAS logs, scout cards, contracts and leases, bids, lease polygons, court house lease documents, production, field information and infrastructure information are just a part of what we collect.

RRCommScan oilfield data
Source: Texas Railroad Commission

We also provide what I call analytical data that is provided by people with special insight or technical expertise. Such data includes editorial analysis, log tops, graded acreage analysis, other types of maps in major plays and lots of other types of data and analysis, graphs, charts and map layers that would take pages to list.

Despite the fact we are processing data all the time, we are somewhat restricted by the delays and release schedule of various agencies. These vary greatly depending on what state and agency we are working with. Probably the most noticeable is production data which can vary from days to months.

The bulk of the employees that work for Drillinginfo are directly engaged in gathering, processing, analyzing and making the data available to customers.

Next week, in part two we will explore some of the teams that collect, ingest, process and QA the many types of data that we use to fuel our decision-making platforms.

Your Turn

What do you think? Leave a comment below.

The following two tabs change content below.

John Fierstien

is the Director of Data Inventory. He has worked as a geologist for several E&P companies and as someone who has been helping to create some of the best tools for geologists and geophysicists to help them find oil and gas. He received his Bachelor of Science in Biology and Geology from Central Michigan University and his Master of Science in Geology from the University of Pittsburgh.