Annual Energy Outlook Forecast Evaluation

by Susan H. Holte

The Office of Integrated Analysis and Forecasting has been providing an evaluation of the forecasts in the Annual Energy Outlook (AEO) annually since 1996. Each year, the forecast evaluation expands on that of the prior year by adding the most recent AEO and the most recent historical year of data. However, the underlying reasons for deviations between the projections and realized history tend to be the same from one evaluation to the next. The most significant conclusions are:

• Over the last two decades, there have been many significant changes in laws, policies, and regulations that could not have been anticipated or assumed in the projections. These include the National Appliance and Energy Conservation Act of 1987, the Natural Gas Wellhead Decontrol Act of 1989, the Clean Air Act Amendments of 1990, the ozone transport rule, the Energy Policy Act of 1992, the repeal of the Power Plant and Industrial Fuel Use Act of 1978, the North American Free Trade Agreement, the Omnibus Budget Reconciliation Act of 1993, the Outer Continental Shelf Deep Water Royalty Relief Act of 1995, the Tax Payer Relief Act of 1997, the Clinton Administration Climate Change Action Plan of 1993, orders issued by the Federal Energy Regulatory Commission on natural gas restructuring and open access to interstate electricity transmission lines, State initiatives for the restructuring of electricity markets, various equipment standards, and regulations issued by the Environmental Protection Agency. Many of these actions have had significant impacts on energy supply, demand, and prices; however, the impacts were not incorporated in the AEO projections until their enactment or effective dates in accordance with the practice that EIA remain policy neutral and that AEO projections include only current laws and regulations.

• Energy prices have been far more difficult to predict than consumption, production, and net imports, and prices have been more typically overestimated than underestimated. More rapid technological improvements, the erosion of the market power of the Organization of Petroleum Exporting Countries in the mid-1980s, excess productive capacity, and market competitiveness are all factors that led to lower energy prices than projected. In the 1980s and 1990s, productivity and technology improvements and the effects of gradual deregulation and changes in industry structure have more than offset the factors that have tended to raise energy prices, such as resource depletion and growing energy demand.

• Natural gas generally has been the fuel with the least accurate forecasts in consumption, production, and prices. Natural gas was the last fossil fuel to be affected by regulatory reforms following the strong regulation of energy markets in the 1970s and early 1980s. Especially as regulatory reforms were implemented, the behavior of natural gas in competitive markets was difficult to predict. Natural gas market trends in the past 25 years have not been stable, which raised the degree of difficulty in determining future levels for market prices and volumes. Furthermore, in past projections the natural gas market outlook was influenced strongly by the assumed world oil price, which was subject to its own error.

• External factors such as severe weather, economic cycles, and strikes have also had an impact on energy markets; however, these events cannot be anticipated in the mid- to long-term period and are not captured in the models underlying the AEO projections.

• Technological improvements in both the production and use of energy have had a significant impact on the price, supply, and consumption of energy. For the most part, earlier AEOs assumed much slower technology development than actually occurred, accounting for some of the deviation between the forecasts and history. This trend was recognized, in part, by this type of forecast evaluation exercise. Beginning with the Annual Energy Outlook 1994, the projections in the AEO were produced using the National Energy Modeling System (NEMS). Because NEMS was designed with methodologies to represent technology in a more detailed fashion, there has been an improvement in the capability to represent technological change throughout energy markets. Additional studies on technological improvement have led to more optimistic assumptions in the more recent projections, along with modeling innovations, such as learning-by-doing, in which experience gained with new generation technologies and advanced end-use technologies leads to cost reductions in the model. These enhancements have significantly improved the projection capability within NEMS.

The last column of Table 1 provides a summary of the average absolute forecast errors for each of the major variables included in this analysis, which are shown in more detail in Tables 2 through 18. The average absolute forecast error is computed as the mean, or average, of all the absolute values of the percentage errors, expressed as percentage differences of the Reference Case projection from actual, shown for each AEO, for each year in the forecast, for a given variable. Table 1 also shows the same summary of forecast errors from the previous evaluations. As indicated in Table 1, the forecasts of consumption, carbon dioxide emissions, production, and gross domestic product have generally been the most accurate, and the forecasts of prices the least accurate. The percent errors change from one year's evaluation to the next as an additional year of data and projections is added. The percent errors may also change due to data revisions from each year's Annual Energy Review.

For the most part, the percent errors remain similar or improve over time; however, the errors for net coal exports have increased significantly. Although the projections for net coal exports have improved in the last two AEOs, the percentage error is increasing because of the very high projections from the earlier AEOs. In this year's evaluation, the earlier projections for coal exports in 2000 were added which are more than double the actual level achieved in 2000 for most of the AEOs through AEO94. Relative to the evaluation last year, net coal exports, coal prices, and electricity prices have higher errors, and net petroleum imports and natural gas wellhead prices have lower errors.

[1] Energy Information Administration, Annual Energy Outlook 2001, DOE/EIA-0383(2000)(Washington, DC, December 2000), www.eia.gov/oiaf/aeo/index.html, is the most recent AEO.

E-Mail Subscription

EIA Home 
Contact Us