Application of cluster computing for seismic imaging in the energy industry

Exploration for oil and gas has been increasing dramatically in the deepwater, ultra-deepwater and shelf region of the Gulf of Mexico and other regions around the world (for example North Sea in Europe). A key technology component that helps to identify possible hydrocarbon targets as well as reduce the risks associated with such deep water exploration is seismic imaging. Seismic imaging helps to accurately position, define and delineate prospective complex subsurface structures before the dollar intensive drilling operations start. The average cost of drilling a well in the Gulf of Mexico is in the range of $50 million. Thus there is a huge emphasis on using seismic imaging to accurately define the shape of these complex subsurface structures and define the velocity variations in the surrounding regime before drilling operations commence.

Seismic imaging for oil and gas exploration is among the most computer-intensive commercial applications that exist for the scientific high-performance computing sector. The accurate reconstruction of 3-D images of the subsurface from 3-D seismic data requires the handling of huge data volumes, usually in the order of 10-15 terabytes, as well as application of computationally intensive imaging algorithms. Only large scale parallel computers can apply these algorithms effectively to image modern seismic data and deliver the results within a useful turn-around time. Thus using and harnessing cluster resources effectively is a crucial and integral component for the energy industry.

This chapter gives some examples along with the various issues related to the use of cluster computing for solving some of today's 3-D seismic imaging problems.

Case Study 1

The first topic that is discussed addresses the I/O performance for seismic imaging application. For seismic imaging applications I/O issue as it turns out is a major bottleneck. Seismic datasets consisting of digitally recorded pressure waves can be very large (in the order of tens of terabytes). Thus it is not possible to read and store all the data and the required information in the memory at the same time. The data is therefore partially read and intermediate results are written out. This clearly indicates that I/O issues are extremely important in seismic imaging. The first case study that is described here discusses a strategy for improved I/O handling in a parallel architecture. The case study highlights the use of a parallel I/O system compared to the traditional standard I/O call and shows an improvement in performance of seismic imaging algorithms by almost an order of magnitude.

Input Output issues for seismic imaging in a parallel environment

Introduction

The processor and communication speeds of parallel computers have steadily increased. However the technology for increasing the I/O subsystems has not increased at the same rate. These are not well designed to efficiently handle massive I/O requirements like in seismic imaging. Most clusters generally use NFS file systems and MPI (message passing interface) as message passing libraries. The essential disadvantage of using NFS is that the I/O nodes are driven by standard UNIX read and write calls which are blocking requests. This is not a serious problem when the I/O volume is small, but when the volume increases as with seismic imaging applications it is important that the computations can be overlapped with I/O calls to maintain the efficiency. The main reason for poor application-level I/O performance is that most parallel I/O systems are unable to handle efficiently small requests, while parallel applications usually make many small I/O requests. Also another limitation of MPI-1 is that the I/O operation can be done only by the master processor, so while the huge amount of data is read by the master from the input file the workers remain idle. This also leads to an overhead of data distribution by the master to the workers.

This section will describe a strategy developed to handle such a specific I/O need in the seismic imaging industry known as ROMIO (high performance portable MPI-IO). It has been developed for a clustering environment where each machine has its own disk and processor so that each processor can do a local read and write. The process that is described here is a finite difference based 3-D post stack migration (imaging) algorithm with parallel I/O on a distributed memory parallel computer.

The Standard Imaging Algorithm and I/O Implementation

The imaging algorithm that will be used is a finite difference based depth migration algorithm for solving a parabolic partial differential equation in the frequency-space domain. This is a standard approximation for the scalar wave equation routinely used by the energy industry. The system is solved using a Crank Nicholson scheme with absorbing boundary conditions on the side. To solve the system the wavefield is decomposed into its monochromatic components by means of a temporal Fourier transform. The parabolic form of the scalar wave equation is then used to downward propagate each of the individual monochromatic waveform. An important advantage of this method is that it is naturally parallel in the frequency domain. Thus each frequency component can be individually extrapolated in depth, independently on each processor and there is no need of ant intertask communication. To set up the computation the input seismic data is first Fourier transformed w.r.t time and stored in a frequency sequential format. Only the required number of frequencies need to be stored, depending on the bandwidth requirement. This forms the input for the computational task along with the velocity model.

The standard parallel implementation is analogous to Master-Worker system. After reading all the required parameters, the Master determines the number of frequencies and frequency bandwidth to be assigned to each Worker. Then it reads and sends the frequency data to the designated Worker. Then the migration algorithm runs through the depth steps. The required velocity data for that depth step is sent to the Workers. Also the migrated data from all the Workers for that depth is collected by master, imaged and stored on the disk. The figure only shows one Master task and one Worker task, but in reality there are many Worker tasks. All the Worker tasks communicate with Master task in an identical fashion as shown in the figure.

Imaging Algorithm with Parallel I/O Implementation

However the above standard method of implementing this algorithm is not robust in terms of the I/O handling. Instead of the standard way of dealing with the I/O issues for this problem this section outlines a method of dealing with the I/O issues using a parallel implementation. The parallel implementation is analogous to SPMD (Single Program Multiple Data) system. Processors read all the required parameters and read the frequency data that is to be migrated by the individual processor, in parallel. Then the migration algorithm runs through the depth steps. All processors read the required velocity data for that depth step in parallel. The processor with rank zero, collects the migrated data from all the processors for that depth, images it and the stores it on the disk. A flow chart of this algorithm is shown in figure 1.

Figure1 Flowchart for Parallel I/O