Analysis & Projections

‹ Analysis & Projections

AEO2010 Retrospective Review

Release Date: August 18, 2011   |  Next Release Date: August 2012   |  Report Number: DOE/EIA-0640(2010)

The Energy Information Administration (EIA) produces projections of energy supply and demand each year in the Annual Energy Outlook (AEO). The projections in the AEO are not statements of what will happen but of what might happen, given the assumptions and methodologies. The Reference case projection assumes trends that are consistent with historical and current market behavior, technological and demographic changes, and current laws and regulations. The potential impacts of pending or proposed legislation, regulations, and standards—or of sections of legislation that have been enacted but that require implementing regulations or appropriation of funds that are not provided or specified in the legislation itself—are not reflected in the projections[1]. Thus, the AEO Reference case provides an impartial baseline that can be used to analyze potential new policies or legislative initiatives. The analysis in the AEO primarily focuses on a Reference case, lower and higher economic growth cases, and lower and higher oil price cases. However, approximately 30 alternative cases are generally included in the AEO. Readers are encouraged to review the full range of cases, which address many of the uncertainties inherent in long-term projections.

Each year since 1996, EIA's Office of Energy Analysis has produced a comparison between realized energy outcomes and the Reference case projections included in previous editions of the AEO. Each year, the comparison adds the projections from the most recent AEO and updates the historical data to the most recently available. The comparison summarizes the variations of the AEO Reference case projections since 1982 to realized outcomes by calculating the average absolute percent differences for several of the major variables for AEO1982 through AEO2010 [2]. The average absolute percent difference is the simple mean of the absolute values of the percentage difference between the Reference case projection and the actual value. The historical data are typically taken from the Annual Energy Review (AER) [3]. The last column of Table 1 provides a summary of the most recent average absolute percent differences for 21 projection components. The detailed calculation of these differences is shown in Tables 3 through 23. These tables also provide the average absolute difference, which is the simple mean of the absolute value of the difference between the Reference case projection and the actual value. The calculated absolute average differences can change from one year's evaluation to the next due to prior year data revisions that often occur in the AER, the Monthly Energy Review (MER), and occasionally in gross domestic product (GDP).

The National Energy Modeling System (NEMS) has been used to prepare the Annual Energy Outlook since AEO1994. The annual projection process begins with the development of assumptions for two key drivers--the world oil price and the macroeconomic growth environment—that is determined outside of the NEMS. While the integrated nature of NEMS may result in some feedback that slightly modifies the initial assumptions about world oil price and the macroeconomic growth environment, these feedbacks tend to be relatively small, so that the initial assumptions for world oil price and the macroeconomic growth environment largely determine the overall projection environment. To the extent that this general environment deviates from the initial assumptions, the NEMS projection results will also deviate.

Table 2 provides a summary of the percentage of cases in which a particular data series is overestimated for all AEOs and for NEMS AEOs. These concepts are summarized for the entire series of AEO Reference cases with additional columns isolating the NEMS-based AEOs (i.e., those beginning with AEO1994). The percentage of projected overestimates for each variable is calculated as the number of overestimates of that variable relative to the total number of projections made (i.e., for each AEO and each year projected). In an “unbiased” projection, we would expect the percentage of overestimates to be close to 50 percent when the sample size is large enough. Many of the sample sizes are relatively small, statistically speaking. The percentage of overestimates in the NEMS AEOs has improved (i.e., moved closer to 50 percent) relative to pre-NEMS AEOs for 12 of the 21 concepts shown in Tables 3 through 23. The average absolute percent differences are smaller in magnitude for the NEMS AEOs for only 8 of the 21 comparisons. For AEO2010, actual data for the year 2009 is included in the table results. The year 2009 was profoundly affected by the current recession contributing to the worsening of several of the average absolute percent differences for NEMS AEOs in comparison with earlier projections – AEOs before AEO1990 did not offer projections for the year 2009.

As a general matter, energy consumption quantities tend to be less volatile and thus projected with greater accuracy than the relatively more volatile energy prices. Energy consumption has a certain amount of inertia inherent from the energy-consuming capital stock, lead times for capital purchase decisions, locked-in contract periods, and myopic decision making. Across all AEOs the average absolute percent difference between Reference case projections of energy consumption, energy production, and carbon dioxide emissions and realized outcomes have typically been under 6 percent, with the exception of natural gas consumption and production, which are just over 6 percent. As expected, the corresponding absolute percent differences are much greater for energy prices – in the 20 to 57 percent range. Both Table 2 and the individual tables which follow generally show reductions (i.e., improved performance) in the absolute average differences for energy prices and net energy imports over time. This general trend has been disrupted somewhat by the recent world oil price volatility of the last several years.

For this edition of the Retrospective, the projections for real GDP shown in Table 3 have been recast to emphasize average cumulative growth from the first year shown for each AEO instead of year-to-year changes. The rationale for this change is that when evaluating other differences in the projections, the cumulative projected growth relative to cumulative realized growth should have the stronger influence. Energy demand and supply projections depend more on the level of economic activity in the year to which the projections apply, which is reflected in the average economic growth rate from the projection date, and to a lesser extent rather on annual fluctuations in economic activity, that are reflected in growth rates over a single year. In the prior version of this table, the errors across all AEOs tended to be similar for a recession year (or an expansion year), since the AEO projects long term trends and doesn't include future business cycles. In this version of Table 3 recessions and expansions don't get the same emphasis since those years get cumulatively averaged from the first year for each AEO. Table 3 now extends to two pages to show the historical cumulative average real GDP growth from the AEO base year to facilitate comparisons.

The underlying reasons for deviations between the AEO Reference case projections and realized history have tended to be the same from one evaluation to the next. The most significant are:
  • Through AEO 2000, the cumulative average growth in real GDP was generally underestimated (Table 3). From 1985 through 1991 real GDP growth averaged 2.7%. From 1992 through 2000 real GDP averaged 3.8% and most AEOs under-projected during this period. Post-2001 real GDP growth has averaged 1.4% and AEOs produced during this period have over-projected during this period.
  • Overestimation of world oil prices, particularly in publications prior to AEO1997 (Table 4), resulted in underestimation of petroleum consumption. The crude oil price projections in the AEOs completed after 1997 tended to be underestimated, which led to overestimation of petroleum consumption. A prime example is the year 2008. All AEO projections for the year 2008, with the exception of AEO2009 (the 2008 figures in AEO2009 are part history and part projection), significantly underestimated the crude oil price (Table 4). These underestimated prices led to overestimates of petroleum consumption for all AEOs (again, other than AEO2009). (Table 5). Similar patterns in Tables 4 and 5 continue for the year 2009.
  • The fuel with the largest difference between the projections and actual consumption has generally been natural gas. Regulatory reforms that increased the role of competitive markets were implemented in the mid-1980s making it difficult to project future developments based on historical data. The technological improvement expectations embedded in early AEOs proved conservative and advances that made petroleum and natural gas less costly to produce were missed. After natural gas curtailments that artificially constrained natural gas use were eased in the mid-1980s, natural gas became an increasingly attractive fuel source, particularly for electricity generation and industrial uses. Historically, natural gas price instability was strongly influenced by the changes in the world oil price. More recently, the AEO Reference case has overestimated natural gas consumption (Table 9) due to the significant underestimate of the natural gas wellhead price (Table 8).
  • Coal prices to the electric power sector were almost always overestimated prior to AEO1999 and underestimated thereafter (Table 12). In general, the AEO coal projections produced prior to the use of NEMS (AEO1982 through AEO1993) did not explicitly model coal mining productivity. From 1985 through 2000, coal mining productivity improved by an average of 6.4 percent per year, reducing the cost of production, and resulting in lower coal prices. As a result, there was a tendency for pre-NEMS coal models to overestimate future coal prices. An additional factor, contributing to the overestimation of delivered coal prices in earlier AEOs was a sharp decline in coal transportation rates that began in the mid-1980s and continued through the 1990s. For the AEOs produced using NEMS (starting with AEO1994), coal mining labor productivity is explicitly modeled. However, the rather sudden switch from steadily increasing coal mining productivity during the 1980s and 1990s to a flat to declining productivity rate starting around 2000 and continuing though 2005 was not anticipated in most of the AEO Reference case projections generated using NEMS. As a result, there has been a recent tendency to underestimate coal prices especially post-2002.
  • For projection years 2001 through 2009 earlier AEOs (AEO1991 to AEO1997) tended to underestimate coal consumption while AEOs starting with AEO1998 tended to overestimate coal consumption (Table 13). This is generally consistent with the pattern of total electricity sales (Table 16), which reflects the electric generation sector's dominant role in U.S. coal consumption. Underestimation of natural gas prices to electricity supported the underestimation of coal consumption in the early AEOs.
  • From AEO1998 through AEO2006, U.S. coal production (Table 14) is overestimated in most years. For AEO1991 through AEO2002, there was also a tendency to overestimate coal exports and underestimate coal imports, both of which contributed to an overestimation of U.S. coal production. Virtually all AEOs overestimated coal production in 2009, a year where production dropped over 8 percent. A confounding factor regarding projections of coal production is the mostly unpredictable pattern of annual coal stock withdrawals and builds. For example, a 38 million ton build-up of coal stockpiles in 2001 resulted in a higher production number, contributing to an underestimation of coal production for 2001 when the large stockpile was consumed in several AEOs. This result follows from the general AEO assumption that the supply and demand for all fuels will balance for all projection years other than their initial year that is calibrated to EIA's Short Term Energy Outlook projection. Historically, other notable changes in coal stockpiles include stock drawdowns of 44 million tons and 41 million tons in 1993 (a strike year) and 2000, respectively.
  • Electricity prices were almost always overestimated in the AEO Reference case projections until the AEOs of the late 1990s, after which the pattern has reversed (Table 15). Electricity prices in the early AEOs assumed regulated, average cost pricing, where fuel costs make up roughly 40 to 50 percent of the total price. As discussed above, coal prices to electric generators were often overestimated in these AEOs, resulting in similar overestimation of electricity prices. In the more recent AEOs, electricity prices have been underestimated, again following the pattern in the coal and natural gas price projections and the partially deregulated generation market. In deregulated markets, natural gas tends to determine marginal electricity prices and follow the changes in the delivered natural gas prices. The recent underestimation of natural gas wellhead prices contributes to underestimates in more recent electricity price projections.
  • The level of future electricity sales was underestimated for nearly all projection years for the AEO1991 through AEO1997 Reference cases (Table 16). Since about 90 percent of the demand for coal results from electricity generation, the underestimation of electricity sales contributed further to the underestimation of coal consumption (in addition to coal prices being overestimated and natural gas prices being underestimated) in those years (Table 13). The underestimation of electricity sales was particularly large in AEO1994 through AEO1996.
  • Over the last two decades, there have been changes in laws, policies, and regulations that were not anticipated in the projections prior to their implementation. Even if some of these could have been predicted, the definition of the Reference case generally limits consideration to current laws, policies and regulations. Many of these actions have had significant impacts on energy supply, demand, and prices. For example, the Powerplant and Industrial Fuel Act (FUA) of 1978 restricted the use of natural gas in power plants and industrial boilers. After FUA was repealed in 1987, use of natural gas for electric generation and industrial processing increased sharply. Consequently, those AEOs completed prior to or immediately after repeal of the FUA, e.g., AEO1986, AEO1987, and AEO1989, underestimated natural gas consumption for year 2000 by considerably more than more recent AEOs.
  • Technological improvements in both the production and use of energy have had significant impacts on the price, supply, and consumption of energy. Earlier AEOs typically assumed much slower technology development than actually occurred. This tendency was identified, in part, by this type of retrospective review. Beginning with the AEO1994, the projections were produced using the NEMS, which was designed to represent technology in a more detailed fashion. This has lead to an improvement in the representation of technological change in the AEO. As NEMS has evolved, additional studies on technological improvement have led to more optimistic assumptions in the more recent projections. Further, the adoption of modeling innovations, such as learning-by-doing, have allowed the model to better reflect the impact on cost of experience with new technologies as they are adopted.
  • External factors such as severe weather, economic cycles, and other supply and/or demand disruptions like hurricanes that strike the Gulf of Mexico rigs have also had an impact on the relationship between projections and realized outcomes, particularly in the short term. These types of events are not anticipated in a mid- to long-term projection like the AEO.
  • Total energy consumption by sector was added to the comparison tables beginning with the retrospective for AEO2008 (Tables 18 to 21). Overall, the projection errors tend to be relatively small; however, some pattern of overestimation is evident. In the transportation sector, this tendency can generally be explained by the economic, fuel-specific, and external factors discussed above. A portion of the overestimation in the industrial sector results from underestimating the extent of structural shifts in the sector. The evolution of the U.S. economy away from energy intensive industries to less-energy intensive manufacturing and services has continued unabated for a few decades and has even accelerated in the many subcategories in the energy-intensive industries.  Turning to weather, actual residential and commercial energy use have also been affected in the last 15 years by weather patterns that have been generally warmer than the 30-year average “normal” used in developing AEO projections. Finally, all sectors have been affected by the length and depth of the current recession, which also causes a tendency for overestimating recent consumption.
  • Since AEO1994 (the NEMS-based AEOs) energy intensity has been the concept most often overestimated. Energy intensity for the retrospective comparisons is defined as the ratio of total energy consumption to nominal GDP. Nominal GDP is used for the comparisons because it has been revised less than real GDP and thus avoids additional comparison issues resulting from other changes to real GDP such as deflator revisions. As a ratio, the projection differences are dependent upon relative differences in energy consumption and nominal GDP. The post-AEO1994 overestimates of energy intensity are the result of two “reinforcing” features of past projections. During the period through AEO2000, GDP tended to be underestimated, which imparts a tendency to overestimate energy intensity. Post-AEO2000, energy consumption has tended to be overestimated, once again imparting a tendency to overestimate intensity. The combination of these two occurrences has resulted in energy intensity being overestimated for most years for AEO1994 forward.


Data Tables

Table 1. Comparison of Absolute Percent Difference Between AEO Reference Case Projections and Related Outcomes    
Table 2. Summary of Differences Between AEO Reference Cases and Realized Outcomes    

Projected vs. Actual

Table 3. Gross Domestic Product    
Table 4. World Oil Prices    
Table 5. Total Petroleum Consumption    
Table 6. Domestic Crude Oil Projection    
Table 7. Petroleum Net Imports    
Table 8. Natural Gas Wellhead Prices    
Table 9. Total Natural Gas Consumption    
Table 10. Natural Gas Production    
Table 11. Natural Gas Net Imports    
Table 12. Coal Prices to Electric Generating Plants    
Table 13. Total Coal Consumption    
Table 14. Coal Production    
Table 15. Average Electricity Prices    
Table 16. Total Electricity Sales    
Table 17. Total Energy Consumption    
Table 18. Total Residential Energy Consumption    
Table 19. Total Commercial Energy Consumption    
Table 20. Total Industrial Energy Consumption    
Table 21. Total Transportation Energy Consumption    
Table 22. Total Carbon Dioxide Emissions    
Table 23. Energy Intensity