U.S. Energy Information Administration - EIA - Independent Statistics and Analysis
Analysis & Projections
AEO Retrospective Review: Evaluation of 2012 and Prior Reference Case Projections
Release Date: March 15, 2013 | Next Release Date: March 2014 | Report Number: DOE/EIA-0640(2012)
The U.S. Energy Information Administration (EIA) produces projections of energy production, consumption and prices each year in the Annual Energy Outlook (AEO). Each year, EIA also produces an AEO Retrospective Review document, which presents a comparison between realized energy outcomes and the Reference case projections included in previous editions of the AEO. The purpose of the Retrospective is to show the relationship between past AEO projections and actual energy indicators, enable trend analysis, and inform discussions of potential improvements to the AEO.
The projections in the AEO are not statements of what will happen but of what might happen, given assumptions and methodologies. The AEO Reference case projection assumes trends that are consistent with historical and current market behavior, technological and demographic changes, and current laws and regulations. The potential impacts of pending or proposed legislation, regulations, and standards are not reflected in the Reference case projections1.
The discussion and analysis in each year’s AEO primarily focuses on a Reference case, lower and higher economic growth cases, and lower and higher oil price cases. In addition to these standard cases, numerous alternative cases have been included in recent AEOs. Readers are encouraged to review the full range of cases, which address many of the uncertainties inherent in long-term projections.
The AEO Retrospective comparison, by contrast, summarizes the differences between actual energy indicators and past AEO Reference case projections, by calculating the average absolute differences and the average absolute percent differences for several of the major variables for AEO1994 through AEO2012 2. The average absolute percent difference is the simple mean of the absolute values of the percentage difference between the Reference case projection and the actual value. The historical data are primarily from the Annual Energy Review (AER 3). This edition includes tables for prices in constant dollars in addition to nominal-dollar tables. Projections for consumption concepts depend on real prices, so constant-dollar price comparisons give a clearer picture of the linkage between consumption projection differences and price projection differences.
The annual projection process begins with the development of assumptions for two key drivers—the world oil price and the macroeconomic growth baseline—that are determined exogenously to the National Energy Modeling System (NEMS) used to prepare the AEO. While the integrated nature of NEMS results in some dynamic feedback that slightly modifies the initial assumptions about the world oil price and the macroeconomic growth baseline, these feedbacks tend to be relatively small, so the initial assumptions for these drivers largely determine the overall projection environment. To the extent that the realized environment deviates from the initial assumptions, the NEMS projection results will also generally deviate.
Table 1 provides a summary of 25 comparisons based on 21 primary concepts (four concepts are energy price projections presented in both real-dollar and nominal-dollar valuations). The detailed results that underlie Table 1 are presented in Tables 2 through 22. These tables show the Reference case projections, historical values and percent differences between projected and actual for all years available. They also provide the average absolute difference across all AEOs for each year of history. The tabulated results can change from one year's evaluation to the next due to data revisions for prior years that can occur in the Annual Energy Review (AER), the Monthly Energy Review (MER), and, occasionally, in gross domestic product (GDP).
The first column in Table 1 is the simple average of the absolute value of percent differences for each concept across all AEOs since the 1994 edition and across all available historical years. In general, energy consumption quantities tend to be less volatile and thus projected with greater accuracy than the relatively more volatile energy prices. This is because energy consumption trends reflect underlying sources of inertia in the energy-consuming capital stock, substantial lead times for capital purchase decisions, locked-in contract periods, and consumer perceptions of and responses to price changes. The average absolute percent differences for energy consumption, production, and carbon dioxide emissions range roughly between 4 and 11 percent. Net imports tend to exhibit somewhat larger average absolute differences, ranging between 13 and 29 percent. The absolute percent differences for energy prices range between 10 and 35 percent.
The second column of Table 1 provides a summary of the percentage of cases in which a particular concept is overestimated relative to its actual value out of the total number of annual projections made for which actual values are available across all AEOs since the 1994 edition. A zero would represent a concept that was always underestimated and 100 percent would represent a concept always overestimated. In an “unbiased” projection, one would expect the percentage of overestimates to be close to 50 percent when the sample size is large enough.
|Variable||Average Absolute Percent Differences||Percent of Projections Over- Estimated|
|Gross Domestic Product|
|Real Gross Domestic Product (Average Cumulative Growth)* (Table 2)||1.0||42.6|
|Imported Refiner Acquisition Cost of Crude Oil (Constant $) (Table 3a)||35.2||18.6|
|Imported Refiner Acquisition Cost of Crude Oil (Nominal $) (Table 3b)||34.7||19.7|
|Total Petroleum Consumption (Table 4)||6.2||66.5|
|Crude Oil Production (Table 5)||6.0||59.6|
|Petroleum Net Imports (Table 6)||13.3||67.0|
|Natural Gas Wellhead Prices (Constant $) (Table 7a)||30.7||26.1|
|Natural Gas Wellhead Prices (Nominal $) (Table 7b)||30.0||27.1|
|Total Natural Gas Consumption (Table 8)||7.8||70.2|
|Natural Gas Production (Table 9)||7.1||66.0|
|Natural Gas Net Imports (Table 10)||29.3||69.7|
|Coal Prices to Electric Generating Plants (Constant $)** (Table 11a)||18.6||39.9|
|Coal Prices to Electric Generating Plants (Nominal $)** (Table 11b)||20.8||43.6|
|Total Coal Consumption (Table 12)||7.4||60.1|
|Coal Production (Table 13)||5.5||63.3|
|Average Electricity Prices (Constant $) (Table 14a)||10.0||37.8|
|Average Electricity Prices (Nominal $) (Table 14b)||12.5||42.6|
|Total Electricity Sales (Table 15)||4.2||46.3|
|Total Energy, Carbon and Intensity|
|Total Energy Consumption (Table 16)||5.6||79.3|
|Delivered Residential Energy Consumption (Table 17)||4.3||69.7|
|Delivered Commercial Energy Consumption (Table 18)||6.0||44.1|
|Delivered Industrial Energy Consumption (Table 19)||11.2||88.3|
|Delivered Transportation Energy Consumption (Table 20)||6.3||78.2|
|Total Energy Related Carbon Dioxide Emissions (Table 21)||6.4||62.2|
|Energy Intensity (Energy Consumption / Real $ GDP) (Table 22)||10.2||94.7|
|AEO - Annual Energy Outlook
Source: These statistics summarize the calculations in Tables 2 through 22. The data in Tables 2 through 22 are based on the 1994 through 2012 Annual Energy Outlook Reference case projections. Historical data are taken from the Annual Energy Review 2011, DOE/EIA-0384(2011) (Washington, DC, September 2011), except for Tables 11a and 11b the source is the Monthly Energy Review September 2012, DOE/EIA-0035(2012/09) (Washington, DC, September 26, 2012).
* The basis for GDP comparison is the projection differences in the cumulative average growth rate of real GDP from the first year shown for each AEO. The summary information for projection differences given for GDP growth rates is absolute percentage point differences; for all other AEO concepts the comparison basis is absolute percent differences.
** Coal prices to electric generating plants are from the Monthly Energy Review. Beginning in AEO2003, EIA electric generating projections incorporated combined heat and power (CHP) electricity generation in electricity generating plants. Prior to AEO2003, coal price projections reflected data collected, estimated, and reported to electric utilities and excluded CHP power generation
The underlying reasons for deviations between the AEO Reference case projections and actual outcomes are numerous and varied. Changes in economic, market, and technological trends and in policies affecting energy choices are often important drivers of these deviations. EIA monitors these developments and uses the best available data to continuously update NEMS and the AEO projections. EIA makes extensive use of scenarios in the AEO to explore the implications of major breaks from historical trends. Even though the precise timing and magnitude of a change in trend cannot be predicted, the likelihood that there will be some changes in trend is virtually certain, and the impact of such changes is estimated using the AEO scenarios. Thus, the full range of potential outcomes cannot be evaluated with the AEO Reference case alone, but only in conjunction with the relevant AEO scenarios and other analysis as applicable to a specific situation.
Some notable factors contributing to deviations between actual outcomes and AEO projections are:
- Macroeconomic factors play a very important role in determining energy usage. The financial crisis and resulting economic weakness starting in 2007 was not anticipated in AEO modeling as NEMS does not project future business cycles. AEO2001 through AEO2009 over-projected macroeconomic growth (Table 2). Earlier AEOs systematically underestimated cumulative growth, even through 2007-2009, due in part to the robust 3.9% average growth in real GDP from 1993 through 2000 fueled by the "high tech bubble." Post-2000 real GDP growth has decreased to an average of 1.6%, and AEOs produced during this period have tended to over-project, even before the impacts of the latest recession. In general, the AEO tends to lag behind macroeconomic trends as a result of an analytic framework which returns economic factors to long-run trends, rather than attempting to predict business cycles or other economic phenomena.
- The imported refiner acquisition cost of crude oil in 2011 was over $100 per barrel, exceeding its previous annual high in 2008. Many of the AEOs prior to AEO2006 projected nominal prices less than $30 per barrel. Underestimation of crude oil prices since 2000 contributed to overestimation of petroleum consumption. This coupled with the effects of the 2007-2009 recession has generally resulted in overestimates of petroleum consumption for the years 2008 through 2011 (Table 4). Only since AEO2009 have the projection differences become more evenly distributed.
- Petroleum net imports peaked in 2005, falling successively in each year since. Before AEO2006, with few exceptions, AEOs had projected rising net imports for this period. From AEO2006 on, projections for net imports have included years of decline (Table 6). This effect was caused primarily by the economic situation, with a lesser contribution from increasing vehicle fuel efficiency. Recently finalized vehicle standards will increase the relative importance of fuel efficiency in future projections.
- Natural gas production has increased each year after 2005 after a long period of relatively flat production, and achieved a record high level in 2011 (Table 9). Natural gas net imports peaked in 2007, falling successively in each year since, with 2011 at the lowest level shown (Table 10). AEOs since AEO2006 have tended to underestimate production growth and consequently overestimate net imports.
- The fuel with the largest absolute percent difference between the projections and historical consumption has generally been natural gas. This can be partially traced to the performance of natural gas price projections. In earlier projections, the natural gas price was influenced by the changes in the world oil price, but this relationship has decoupled since 2000. Through AEO2005, the Reference case has generally overestimated natural gas consumption (Table 8), in part due to the tendency for significant underestimates of the natural gas wellhead price (Tables 7a, 7b). In addition, industrial natural gas usage is highly sensitive to business cycles, leading to greater volatility in overall natural gas consumption.
- Coal prices to electric generating plants set a new high in 2011 (Tables 11a, 11b). This coupled with the "new normal" of increased natural gas production and low and declining natural gas prices, has resulted in consumption for 2011 falling and being below its level in 1996 (Table 12). The previous recent low for coal consumption was set in 2009 (the only year since 1996 lower than 2011), and was influenced by the same price effects as 2011, but with the added pressure of falling real GDP for that year.
- Electricity prices were almost always underestimated in the Reference case projections from AEO2000 on (Tables 14a, 14b), following the pattern in the coal and natural gas price projections. In deregulated markets, natural gas tends to determine marginal electricity prices, and with part of the United States remaining regulated, this presents a complex pricing environment for a national projection; but overall, the underestimation of natural gas and coal prices noted above also results in underestimated electricity prices.
- Total energy consumption is lower in 2011 than it was in 2000 (although all but two intervening years were higher than 2011). Most AEOs projected rising consumption, with AEO2008 and AEO2009 being exceptions. This is the result of not anticipating the recent recession and slow recovery in the projections. Projection differences tend to be relatively small; however, some pattern of overestimation is evident (Tables 16 to 20).
- Projected energy intensity has been the concept most often overestimated (Table 22). Energy intensity for the retrospective comparisons is defined as the ratio of two other projected concepts: total energy consumption and real GDP. As a ratio, the projection differences are dependent upon relative differences in energy consumption and real GDP. The overestimates of energy intensity are the result of two "reinforcing" features of past projections. During the period through AEO2000, GDP tended to be underestimated, which imparts a tendency to overestimate energy intensity. Post-AEO2000, energy consumption has tended to be overestimated, once again imparting a tendency to overestimate intensity. The combination of these two occurrences has resulted in energy intensity being overestimated for nearly all years.