Petrol topped $4 per gallon in the US recently. Crude oil is trading at all-time highs-above $125 a barrel. As a result oil and gas companies are delivering fat profits. In May, Exxon Mobil reported $10.9 billion in profits for its latest quarter, just short of its record-breaking $11.7 billion the quarter before.

It's tempting-and politically expedient-to explain such astounding numbers by saying that greedy oil companies are taking advantage of market fears, making money on the bent backs consumers. So many of us, after all, have no choice but to buy fuel. We fill our cars to drive to work, where buildings must be heated in winter, supplies must be shipped, products hauled in trucks and executives jetted hither and yon.

Yet economists will counter that taking advantage -spotting a revenue opportunity and moving on it- is exactly what companies should do: That's capitalism. Oil companies excel at identifying where their profit advantage lies. And they obtain that advantage through sophisticated business intelligence systems.

Without good business intelligence (BI), oil companies risk their livelihoods, says David Knapp, a senior editor at the Energy Intelligence Group, an information provider for the oil industry. "Those that have lagged in understanding have lagged in performance," Knapp says. BI is all about understanding what makes your company-and your industry-thrive. Mortgage lenders, for example, are currently going under in part because they didn't analyse enough of the right customer data and signed up risky borrowers.

Oil companies have always lived and died on BI, says Gary Lensing, VP and CIO for global exploration and production at Hess, an oil exploration specialist. "Data drives what we do, always quantifying where that value is."

Hess and its competitors harvest data from inside and outside their four walls, plus they factor in wild cards such as war, weather and global politics. BI in oil and gas isn't a simple matter of buying a set of analysis tools and feeding data into them. Oil companies pass information through multiple layers of software, with nearly every employee focused on collecting and storing some kind of data. Exxon, for example, wants its geophysicists to know Fortran, C and Java so they can code their own, quick analyses. When Hess drills a well, Lensing says, engineers collect status data every 15 seconds.

Typically, specialised applications for oil and gas-such as Geolog from Paradigm Geotechnology (to find patterns in seismic measures) or PDI FocalPoint from Professional DataSolutions (to track gas station store sales in a dashboard) - have their own analysis capabilities. But to get a global view of company performance, that data must be fed into off-the-shelf BI analysis and reporting packages familiar to most CIOs, such as those from Hyperian. Then the companies add supply-chain information. SAP for Oil & Gas modules manages the supply chains at companies like Hess and Valero. Those companies also use at least some of the SAP analysis and storage applications, including Business Warehouse. Oil companies store data in both common databases, such as Oracle, and specialised ones for the oil industry, such as OpenWorks or StratWorks from Halliburton.

When it comes to BI, Big Oil has a big view. "We're not as transactionally driven as other industries," Lensing says. "Are you trying to gain operational efficiencies by squeezing pennies out of transactions, or are you looking at core assets and trying to extract additional value?"

Examine how oil companies approach BI and you will uncover valuable lessons for improving your own BI efforts, whether you're trying to optimise profits or uncover untapped markets.

Factors in the Price of Gas

Old-timers called oil "Texas Tea," but the U.S. oil industry really started in Pennsylvania, with the 1859 discovery of light crude burbling between rocks in a farmer's creek. People at first used it to grease machinery and light lamps. Fifty years later, rigs pumped black gold from wells across the country and fortunes were made. Now, as then, oil men are cagey about their claims don't say much about what they know. But some will talk about how they know it.

In an industry where the top five oil companies last year booked $1.5 trillion in sales, thieves target that intelligence. In February, for example, Petrobras, the $112 billion state-owned oil giant in Brazil, had four laptops and two hard drives stolen. They contained "secret and important information," the company told Brazilian news outlets, about an ocean reservoir that in the next few years could produce up to eight billion barrels of oil. Brazilian police are said to be investigating. Geologic information like the sort believed to have been stolen from Petrobras is one piece of the "upstream" part of the business, where companies and countries explore and drill for oil deposits deep in the earth. Analysts combine geologic and seismic data with what-if engineering models showing how best to get the oil out and the projected costs of such a multiyear project, explains Louie Ehrlich, CIO and president of Chevron Information Technology.

Then there is the "downstream" work of refining crude oil into something usable, such as gasoline or diesel, and of getting those products sold and delivered. Those jobs generate information on refinery capacity and throughput, for example, and the cost of marketing and distribution.

Exxon and Chevron, the biggest oil companies in the United States, are known as "integrated," meaning they work both the upstream and downstream ends of the business. Petrobras does, too, though Ehrlich points out that no company is perfectly integrated, meaning that what it finds in the ground always ends up in its own refineries. Chevron might find crude that its refineries don't handle, he says. "Some types of oil require more complex refining capability to process." Chevron produces about two million barrels of oil per day and only refines about 15 per cent in its own refineries.

Others focus on just one end or the other. Valero, for example, is the biggest U.S. refiner, concentrating on the downstream work of turning oil into other things to sell.

Upstream usually costs more than downstream. Exxon, for example, spent $15.7 billion on upstream jobs in 2007. Chevron, $15.5 billion. But downstream costs stack up, too. Exxon's were $1.1 billion and Chevron's $3.4 billion.

Prices at the pump reflect these expenses. The cost of crude oil constitutes most of the price of gas, accounting for 73 per cent of today's $4-plus figure in the US, and the £5 a gallon that UK prices reached in March. Refining, meanwhile, is eight per cent; so is distribution and marketing. Each oil company analyses its costs and potential income, says David Smith, an IT consultant to the oil industry at Electronic Data Systems, trying to profit at each step (except for taxes, which are fixed).

Traditional economic principles of supply and demand alone fall short when you try to forecast prices, Smith says. "With political instability, fear about Iran and Iraq-those have ripple effects and an emotional response at the pump," he says.

"You have to blend that volatility with real-time market data and factors you can't predict."

Big Oil's Big Picture

After oil, the best kind of gusher to discover and manage these days is data, and therefore profits, in real time. Or close to it. That's what Hess is after.

For the past four years, the $32 billion integrated oil company has been building BI systems to trace and interpret data from start to finish along the exploration and production value chain in as close to real time as possible, says Lensing. The idea is to be able to see activity at all its assets in Norway, Denmark, the U.K., the U.S., Thailand and Africa. Are its four fields in Equatorial Guinea producing as expected today? Is the refinery in New Jersey running at capacity, or can it take in more barrels of oil before the end of the month? What have sales at its 1,370 gas stations been since last Saturday at noon?

No one business intelligence product can do it all, though. For financial analysis, Hess mainly uses tools from Hyperion, which Oracle bought last year. To estimate how much oil or natural gas its wells can produce, the company develops a model of the reservoir terrain based in part on readings from bouncing seismic waves in the area. For a look at patterns in well production, Hess runs a tool popular among pharmaceutical firms called Spotfire, from Tibco. Spotfire lets analysts visualise data by producing graphs, charts and other pictures, into which users can drill down with queries.

The company is also installing OSIsoft performance management software-in part to collect operations data-to measure, for example, how efficiently platforms and storage tanks are running. That project isn't finished yet. Meanwhile, Hess receives daily uploads about the performance of its joint ventures, such as one with Shell in the Gulf of Mexico, via secured FTP transfers.

One of the real-time parts of this BI chain is well data. An engineer in Houston can monitor drilling activity in West Africa, see an anomaly in how the drill bit sinks into the ocean floor and can send that data over satellite to a geoscientist in Houston, who can view the visualisation and e-mail a recommendation on how to adjust the machines, Lensing says.

"The ability for people on a platform to communicate with people in the home office and work on the same set of data means we can get more production done faster and more accurately," he says. "How you choose to analyse the data and the decisions you make-there's your competitive advantage."

More production faster means Hess could, in theory, sell more crude or refined products sooner while market prices are high, as they are now.

The Cost of New Business

For Petrobras, an oil field discovered off the coast of Brazil could become the world's third biggest, after one in Saudi Arabia and another in Kuwait. The potential bounty: 33 billion barrels.

That's an unofficial estimate attributed in April to Brazil's National Petroleum Agency. Petrobras officials decline to confirm it, insisting that more testing must be done. Olinto Gomes de Souza Jr., a senior geologist there, is helping analyse some of the test data.

After four years of exploration and computerised modeling, the company last November announced that it had hit oil 6,500 feet beneath the ocean surface and another 16,000 feet into the ocean floor. Now proof drilling continues, boring through rock and salt layers atop the oil. At each centimetre, Petrobras looks at 10 to 12 variables, including temperature, pressure, and weight of rock and sediment. Stored in an Oracle database, the information is queried with analytics software from SAS Institute.

After geologists assess the information, it's sliced and diced against financial realities. "The amount of money we spend is very high-$100 million for a well alone," de Souza says. "We want to get it right."

To reach its goal of becoming one of the five biggest oil companies in the world by 2020, Petrobras has to take some calculated risks. Recovering oil from this find will be expensive partly because it's so far down in the earth. "No company has tried to explore under it," he says. But promising data has triggered major staffing decisions: Petrobras has created a new group of senior managers to oversee exploration of this area and plans to hire 14,000 drillers, geologists and engineers. It takes years to go from initial exploration to crude oil production and sales of finished gasoline, so companies have to model markets five, 10, 15 years out. They use a mix of their own intelligence and public data, such as from the Energy Information Administration (EIA), says researcher Knapp.

For example, car makers continue to improve fuel efficiency, as well as to build electric-gas hybrids. A highly simplified analysis suggests that if people use less gasoline, gas prices should drop, which makes expensive drilling less profitable, Knapp explains.

Although demand for gas is growing in China and India, so far it's not enough to offset the expected drop in U.S. demand. New well and rig technologies could take some of the cost out of drilling, but no one knows exactly when or by how much. There is no shortage of data points; the value is in interpretation. "It's about filtering rather than finding a piece of information," he says. "Understanding what this whole pile of stuff can do for you is the key."

Adjusting to Change in Real Time

Every Wednesday morning, the shouts and hand gestures that make the Nymex trading floor in New York frantic begin to calm. Petroleum traders are waiting for the release of data from the U.S. Energy Information Administration (EIA) on countries' inventories of crude oil and gasoline, as well as world crude prices.

At 10:30 a.m., the EIA website sees a storm of activity: 1,000 page views per second for 15 seconds, says Charlie Riner, a lead analyst for the site. Oil companies, commodities traders, analyst firms, and government agencies in the United States and other countries have written bots to collect the data. Then traffic ebbs.

Inventories are the most closely watched data in the industry, says Joanne Shore, a senior petroleum analyst at the EIA, the statistics keeper for the U.S. Department of Energy. "This is what moves markets when it comes out," Shore says. If, for example, U.S. supplies fall sharply from the week before, that can mean demand is rising and prices likely will, too. It's not only traders who want this data. Corporations such as Valero fold it into its analysis of inventories and market activity so that they always know their standing compared to rivals.

In the oil and gas business, you are what you own. The amount of crude waiting to be refined, or the already-processed liquid in storage tanks ready to be sold and delivered, represents much of a company's value at a given moment. As a refiner, Valero buys barrels of oil to heat and pressurise into other products, such as diesel fuel, asphalt and lubricants. The $95 billion downstream company owns 17 refineries that together can produce 3.1 million barrels of product per day.

But Valero doesn't sell that much in a given day so it must store finished goods until they're ready to be shipped to customers. The company tracks its own inventory movements the way a first-time mother studies her infant. How much of which products did we sell this morning? How about now? And now?

Market analysts run inventory reports "a few hundred times a day," says Kirk Hewitt, vice president of accounting processing optimisation . As the cost of crude fluctuates during trading hours, Valero sales and marketing staff want frequent updates so they can sell products at the most profitable price and buy crude to feed their refineries at the best price.

"We're dealing with a commodity whose price changes every second," Hewitt explains. "So our margins change every minute. Our costs change every minute."

Companywide, Valero employees generate more than 20,000 reports per month. These range from petrol station profit-and-loss statements to the status of payable invoices to telecommunications charges.

Valero uses WebFocus tools from Information Builders for nearly all its BI reporting, but not for inventory reporting, which has to be quick and dirty. For that, users query the Valero SAP Business Warehouse system, which collects operations data from the SAP R/3 system at their refineries. Data includes items such as the volume of crude processed and the amount of products made from it. The information is presented in an Excel spreadsheet, he says, because "it's a fast way to get a snapshot."

Valero wants to make its BI faster overall. Now most reports use information from data warehouses populated each night with batch updates from SAP. But Hewitt says moving to a service-oriented architecture will enable more frequent updates, so the analysts can query more current data throughout the day. The company is working with SAP to implement SAP Exchange Infrastructure, or XI, to make that happen. Valero will still use WebFocus for what-if analysis and report presentation, he says.

Though the technology is changing, the purpose of the analysis isn't. Valero monitors the value of its inventory, along with sales and efficiency at its petrol stations, to make adjustments to the price of its products as it watches demand and supply shift. Just because gas prices are soaring doesn't mean Valero has an easy ride. Aside from gasoline, the company makes asphalt, sulphur and other secondary products. These prices haven't increased as much as gasoline has, or in proportion to the rise in the cost of crude needed to make them. Valero has to balance its dependencies.

Managing X-Factors

Financial markets often move on fear and uncertainty. The problem is, no one can predict which direction commodity prices will go in or how much they will gain or lose.

On April 21, news spread that unidentified attackers had punctured a Japanese oil tanker with rockets while the ship was sailing to Saudi Arabia. That same day, Royal Dutch Shell announced that African militia fighters, protesting about corporate oil activity on the Niger Delta, had damaged a pipeline in Nigeria. Worried about oil supplies, traders pushed oil to $117 per barrel, setting a new record.

Then there are less-violent but no less-volatile events like the hurricanes Rita and Katrina in 2005. Downtime at even one major refinery from a fire or explosion can drag down earnings at that company and affect the rest of the industry for years.

BP, has paid in financial terms and human lives. In 2005, explosions and fire at the BP refinery in Texas City, killed 15 people and hurt 170 others. The refinery, which by itself makes about 2.5 per cent of all the gasoline sold in the United States, had to be partially closed. Then it suffered damage from Rita and Katrina later that year and didn't re-open completely until February. BP profits in the U.S. have dropped, in part because of the Texas City disaster, from $12.6 billion the year the refinery blew up to $7.4 billion last year, according to BP's latest annual report.

Chevron, meanwhile, noted in its annual report that although product margins for the oil industry were generally higher for 2007, profit margins on its refined products "were negatively affected by planned and unplanned downtime at its three largest U.S. refineries." Because of the problems, Chevron U.S. refineries ran at 85 per cent capacity for crude oil distillation in 2007, down from 99 per cent in 2006. U.S. downstream profits at Chevron dipped to $966 million from $1.9 billion in 2006.

Smith, the EDS consultant, says competitors should have BI in place to assess an event like the Texas City disaster or partial shutdowns immediately. "If I'm Shell, I should understand what's the opportunity" to fill gas orders that BP and Chevron might not be able to, he says.

The Federal Reserve Bank of Dallas has developed a model to forecast the price of gasoline that accounts for surprise events. Stephen Brown, director of energy economics and microeconomic policy analysis there, uses a combination of Excel and EViews, a Microsoft Windows-based application designed to perform regression analysis. Unlike most BI tools, EViews was designed specifically for analysing time-series data. Advanced regression analysis capabilities aren't usually part of mainstream BI tools, although SAS and SPSS offer some.

First, Brown assumes that a certain number of unpredictable events will happen in a given year. For example, some refineries will shut down for some period because of fires or hurricane damage. Brown looks at refinery histories to calculate an average outage, then sets his model to account for it. "We have said that all these unusual events that have occurred in the past are going to occur on average in the future as they have in the past," he explains.

What Brown's model can't account for is politics. There's no way to calculate an average impact of country leaders acting erratically - something the $214 billion Chevron must deal with. About 26 per cent of its proven oil reserves are in Kazakhstan, the company says. Kazakhstan isn't the most stable of countries. It broke off from Russia in 1991 and is now ruled by a president granted lifetime powers and immunity from criminal prosecution.

Chevron does not comment on the security of company personnel or operations, according to a spokesman, Sean Comey. However, in its latest annual report, Chevron lists the Kazakhstan operation under the warning "Political instability could harm Chevron's business."

Oil isn't a lucrative industry; and all those profits don't come from good business intelligence practices alone. But it's a powerful notion to run a company with the mind-set that virtually every employee is a data analyst.

"Engineers and geoscientists and everyone have been taught BI from the start," says Lensing, the Hess CIO. Give people in any industry access to information along with tools to interpret the past, model the future and imagine different paths between the two, he says, and they can change the trajectory of companies.