Cloud Storage for E&P Data: Lessons Learned

It is increasingly clear that oil and gas companies are ramping up their efforts to put more and more E&P data into cloud storage.  The current industry dynamics are pressuring them to continue to drive down cost, declare a digitalization strategy and compete with companies in other industry verticals for capital.

So what are they learning as they make the journey?

Cloud Storage Cost

According to the eighth annual State of the Cloud survey by RightScale, controlling cloud costs is the top priority for companies for the third year in a row. Given the storage costs quoted by service providers and the expectations that those set, that’s not surprising.

Commercial cloud storage providers’ business model can be summed up as follows: it’s almost free to get your for E&P data there, and cheap to store it, but accessing, moving and downloading data can be expensive. The practical implications to this are:

• At the cheapest rates (the headline rates referred to above), your data is still effectively unavailable.
• Storage is not cheap at the layers that make the data easily accessible, for analytics or interpretation, for instance.
• Cloud costs are very difficult to predict and budget for.

Xaas

It’s true that a cloud solution enables SaaS, CaaS and PaaS, and that these can be a very powerful driver for both cost savings and productivity increases. There are a couple of contingencies that need to be considered, however. First, these services require ready and frequent access to the data, and as mentioned earlier, this has significant cost implications. Second, while SaaS is a rapidly growing segment, most iterations of this are on private clouds.

Moving data from one cloud to another is not straightforward, and also has cost implications.

Artificial Intelligence and Machine Learning

One of the most important things to consider when it comes to analytics and AI/ML workflows – and this is often overlooked at the planning stages – is that 90% of the work will be preparing the source data. You can’t properly train algorithms without highly regularized data. So while humans are adept at identifying and ignoring irregular data, AI/ML algorithms consider everything to be correct. As a result, enormous effort must be undertaken to identify and correct any irregularities before the algorithms can provide benefit.

As you can imagine, this is a big job when dealing with things like seismic and well logs, for example. This brings us to our last point…

E&P Data Management

Let’s start with two very important statements:

• Cloud storage is not E&P data management, and;
• Oil and gas data management on the cloud is not trivial. Don’t believe anyone who says that it is.

If all that you do is put your data on the cloud, without a data management layer to help you search, query and find it, in effect all you are doing is moving it from one filesystem (yours) to another (theirs). If you want to take advantage of all the other benefits of the cloud, you need a data management layer that sits between you and the data. Furthermore, it needs to be designed with cloud data management in mind, so that (for instance) you can query and understand your data without actually touching it, in order to keep your cloud costs down.

Public vs Private Cloud Storage

In less than two years, Katalyst has loaded over 20 petabytes of client data to the cloud. That rate is accelerating, perhaps not surprisingly, as more large oil and gas companies embark on digital transformation projects. It is important to note though that client strategies have evolved throughout this process, and in almost every case, the strategy has evolved towards a Katalyst cloud solution or hybrid cloud solution. The reason for this is simple; data that needs to be accessed should be on a storage architecture where that can happen without incurring extra cost.

For data that can sit in deep archive, the public cloud makes sense. So many companies are adopting a strategy where the ‘gold’ copy of the data sits in a low cost storage tier, and the working copies of the data are available for consumption on a Katalyst cloud solution. As long as you have a data management solution that can manage a hybrid solution, this becomes an optimal architecture.

According to a Flexera 2019 State of the Cloud Survey, 84% of companies going to the cloud had a multi-cloud strategy and almost 50% had plans to balance private and public clouds as a solution. The implication is clear; there is a strong trend towards hybrid cloud solutions.

Related Content

• Webinar – Considering the Cloud

Summary

The promise of the cloud is real, and it’s disruptive. It’s not something that one should be either ‘for’ or ‘against.’ It’s also not going away. While this is true, it is also true that headline cloud storage costs don’t represent the entire picture.

What is increasingly clear is that there is planning and thinking to do on the part of oil and gas companies as they embark on this journey. Several key considerations need to be considered, such as:

• What objectives you are trying to achieve? Depending on these, it’s possible that a commercial cloud strategy isn’t the right solution at this time.
• Your subsurface data storage and access requirements – what data do you need access to and what can you safely archive for long periods?
• Your present and future compute and infrastructure strategy – what will be required of your organization and how will it be able to support the initiatives going forward.

Once these are understood, it becomes possible to build an optimal solution. This then enables maximum benefit from the cloud.

It shouldn’t surprise anyone that there’s more to this than one-line storage quotes. This is an evolving space in the oil and gas industry with revolutionary promise. We are learning more about it every day, and as we continue to work with customers to transform their data we will continue to teach each other. The adoption of application and compute layers to the offering will no doubt cause further disruption and benefit.

It seems very clear that the next several years will re-write the way that oil and gas companies architect themselves. It may well be the biggest transformation we have ever seen in our industry.