of the Spring Meeting of the
American Statistical Association (ASA)
Committee on Energy Statistics
April 22 and 23, 2004
Energy Information Administration
Natural Gas Prices and Industrial Sector Responses: An Experimental Module for the Short-Term Integrated Forecasting System (STIFS), Dave Costello, Office of Energy Markets and End Use (EMEU) and Frederick L. Joutz, Associate Professor, Department of Economics, The George Washington University. The Short-Term Integrated Forecasting System (STIFS) generates monthly forecasts of energy demand, supply and prices using some forecast information that is incorporated into STIFS that is generated by other models that do not run in an integrated framework with STIFS. This includes the macroeconomic forecasts and projections for certain energy supply variables. There is no direct feedback between the macroeconomic models projections and STIFS components. Members of the STIFS Team can attempt to coordinate iterations between the two models. However, this is not desirable for two main reasons. First, it suffers from specification problems in the richness and complexity of the dynamic interactions because the feedback is not directly estimated. Second, the iteration process requires staff time and resources that are limited.
This project tests an experimental model for the interaction between natural gas prices, natural gas consumption, and industrial sector activity. Two strategies are followed. The first involves a simple VAR framework capturing the time series dynamics testing for Granger causality and examining impulse response functions and forecast error variance decompositions. In the second approach, energy and economic variables are analyzed in terms of integration, co-integration for a long-run relationship between oil and natural gas prices. The general to specific modeling methodology is used to develop a data coherent parsimonious representation. Issues related to parameter constancy, encompassing, and forecasting are discussed. The forecasting performance of the two strategies is compared and the potential gain from using the experimental module is discussed.
The main comment from the Committee on this paper had to do with finding a way to deal with the impacts of regime change on the modeling results. One way was to limit the estimation of the models to the period constituting the new regime (2000 on, approximately). We suggested perhaps trying level shifts in the VAR equations and in addition trying to incorporate estimates of conditional volatility. The Committee suggested that we try incorporating artificial shocks into the data series to see how the model would handle those, and an attempt to understand these techniques and to try to implement them was decided upon.
Measuring the Quality of EIA Analysis: A Revised Approach,
This spring session revisited EIA's need to assess the quality of EIA's analytical products, called for in EIA's Strategic Plan. The methodology has been reviewed and revised, the survey has been redirected toward a new population, and the questionnaire revised.
Summary of ASA Comments and Suggestions:
EIA has developed an entirely new survey approach from (previously) surveying Independent Expert Reviewers of certain EIA analytical products, to surveying pre-conference registrant for the NEMS Conference by an email survey. The revised survey audience, survey method and survey response rate of 35% was considered admirable. Even though the survey results were quantifiable, they did not however point to actionable information to the fullest extent possible. However, the Committee did feel that direction of survey comments was indicative and could be taken seriously. The Committee thought the responses were coming from the right people, but suggested not to go to the NEMS Conference attendees right away, instead go to a larger set of people. In summary the revised approach represented substantial progress.
EIA's Intended Response
EIA will develop a web-based survey and will use a larger survey frame employing available List Serves. EIA will also ask respondents if they would be willing to be interviewed over the telephone as a follow-up to their survey responses. Such interviews could then be used as a means for in-depth probing resulting in the identification of a greater number of actionable items.
The Energy Information Administration is implementing a plan to expand the coverage of the current Short-Term Integrated Forecasting System (STIFS) from a single-region national model to a multi-region model. Variations in regional demand, supply, and prices provide opportunities to assess and estimate impacts of energy prices on energy demand, choices fuels, and choices of energy efficient technologies. A regional approach in projecting the short-term energy outlook will also allow users to get more detailed information on regional energy demand, supply, storage, movement of fuels, and potential transportation bottlenecks. There are, however, statistical issues as well as econometric issues that need to be resolved before a sound operational model can emerge.
This paper examined two specific issues facing the modeling team. The first one dealt with the specification of demand equations. An econometric model specification can be based on demand for each type of fuel such as demand for natural gas, electricity fuel, and oil. The approach allows modelers to specify demand and supply for a fuel. For example a model can estimate the demand for natural gas by end-use sectors and estimate gas demand and supply equations simultaneously. Econometrically, both demand and supply equations can be identified if adequate data are available. In addition, demand equations with emphasis on end-user sectors allow modelers to impose different restrictions on model parameters and could provide insights that may be useful to stakeholders. For example, modeling residential demand by fuel type allows analysts to study the effects of fuel prices on fuel substitution and efficiency improvement. We are exploring the pros and cons of these two different specifications and their performance in providing good predictors.
The second issue dealt with electricity demand and power generation. Monthly statistics on electricity demand do not reflect hourly variations in demand. In a hot summer day, peak load demand in the afternoon can be several folds higher than early morning demand. Power generation required to meeting peak demand could impose binding conditions to the electrical system. We propose to create pseudo load curves based on monthly data. For example, electricity demand in April and May are relatively flat. This information can be used to mimic the low-end of a summer month load curve. High-frequency weather and other information on the shape of load curves from Independent System Operators (ISO) may be used to infer peak demand during summer months. The cost of generation for each type of power plan will be calculated based on fuel costs, thermal conversion efficiency, and operating and maintenance costs. The cost of electricity and the shape of the load curve will then be used to determine dispatching of electricity from generators. Differences in regional power supply and demand will be used to assess interregional electricity flow. This approach could provide insights to the effects of forced outage and peak load demand on regional power flow and fuel shares in the power generation sector.Summary of Advice from
Re: "Issues in Short-Term Energy Modeling: Adding Regional Components to EIA's Short-Term Energy Model" from David Costello
The Committee basically commented that the recommendations on what to do with the proposed regionalization (specifically the regionalization of the electricity supply part of the STEO model) depended upon being clear about what EIA hoped to gain through the model change. If the purpose is to make a contribution toward understanding why load patterns are the way they are in specific regions, then the transmission (congestion) problem probably has to be addressed. We indicated that we were not going to be able to get at explicit consideration of the capacity bottleneck problem but would be able to get a feel for potential flows across regions. One clear goal is to get more accurate and explainable fuel use patterns for electricity supply. Our feeling was that the kind of regionalization we had in mind would help. The Committee seemed willing to accept that but was anxious to get more information from tests with actual models to see how the concept was actually going to be put to work. Some kind of demonstration for the Committee in future meetings would seem in order.And from Phillip Tseng (quoting Mark Bernstein from the meeting session summary transcript): Summary from Dr. Bernstein,
EIA’s Frames: How Do We Know if They are Sufficient? Grace Sutherland, Introduction, with participation from Renee Miller, Howard Bradsher-Fredrick, Shawna Waugh and Alethea Jennings, SMG, EIA
The quality of EIA’s data has been made a priority and has been made part of its Strategic Plan. Goal 1 of the EIA Strategic Plan states “…EIA’s information products will retain or improve their high quality…” One of the performance measures and targets for this goal involves evaluating the EIA frames. The measure is the percent of EIA survey frames with sufficient industry coverage to produce reliable supply, demand and price statistics.
We began by preparing a list of EIA survey frames and update procedures. We are now in the process of gathering existing information on the quality of our frames. Activities to assess frame quality include:
1. Checking the frame against alternative lists at the respondent level (suggestion taken from the Statistical Policy Working Paper 15).
2. Data comparisons at an aggregate level (certain EIA data compared with similar data outside of EIA).
3. Examining supply/disposition balances (supply should equal disposition, but because the data that comprise the supply/disposition balances are from different surveys, a balancing item is needed. If the balancing item becomes large, it could be an indication of a frame or other data quality problem).
This presentation provided examples of these activities and asked for the Committee’s guidance on other activities that could be used to assess our frames. We were also interested in the Committee’s thoughts on how we can develop criteria to define sufficient coverage.
ASA Energy Committee Suggestions
1. Try an adaptive sampling approach to develop lists for comparison. One way to do this is to ask known establishments, plants and so on, to identify competitors and customers or suppliers others within the same market.
2. Think of EIA frames as a sample from a larger population. The missing units would correspond to a non-response problem. Then calculate “propensity” scores by post-stratifying the sample using Census data.
3. Try dual system estimation to obtain estimates of coverage.
4. Balancing items and comparisons are not good measures of coverage because other problems such as reporting errors could also play a role in making these measures go out of sync. Ideally, one needs independent lists to assess coverage.
5. Census should provide as much information as possible to EIA, without disclosing sensitive information, so that EIA can identify the missing respondents.
EIA Response to Committee Suggestions
Respondents have been reluctant to give EIA the names of their customers; however, we have had success in obtaining names of suppliers or competitors. Some of our electric power surveys, for example, have used this technique. Therefore, EIA does not plan to pursue the adaptive sampling approach at the present time, but may consider it again in the future.
We are interested in pursuing propensity scores and dual system estimation. In fact, we have tried to apply the latter to the EIA and REPIS renewable frames.
The comment about balancing items and comparisons led us to rethink our approach to assessing coverage. Rather than try to develop a quantitative measure that could be applied to all surveys, we are going to be using a qualitative approach. We are compiling the information we have on coverage for all of EIA surveys and will send it to an inter-office team, representing both survey staff and data users to evaluate.
Electricity 2005, Robert Schnapp, Director, Electric Power Division, Office of Coal, Nuclear, Electric and Alternate Fuels, EIA
“Electricity 2005” is a project that EIA’s Electric Power Division began in 2003 to revise the electric power surveys to accurately capture the changing industry. This presentation focused on the four phases of the project; the issues and potential changes that are most important; and the relevance of EIA’s data confidentiality policy. Special attention was given to transmission data needs to tie this presentation to the following one on focus groups held to elicit input on the transmission data that EIA should collect.
Electricity Transmission Data Needs Focus Group Results, Howard Bradsher-Fredrick and Phillip Tseng, SMG, EIA
The Energy Information Administration (EIA) is charged with providing timely and relevant data to stakeholders. In response to changes in the electricity industry in recent years, EIA has used its normal review process to update the EIA survey forms, collect information consistent with the market, and publish reports that meet stakeholders’ needs. A normal procedure is holding stakeholders meetings to receive inputs and suggestions. This session was to report our findings at this ASA meeting.
Four focus groups were conducted from November 2003 through January 2004 on the topic of electricity transmission data needs. The participants in each of these groups were fairly homogenous with the following groupings: EIA, DOE policy offices, other Federal organizations, and non-Federal organizations. Each focus group had from 9 to 12 participants. These sessions were audio taped for the purposes of writing the report. The participants were asked questions from a structured protocol; these questions involved the following topics: characterizing EIA’s present transmission data collection, emerging issues in transmission, specific data needs, sources of data, and confidentiality issues.
The focus group participants stated that EIA should begin to provide more data regarding electricity transmission. Some of the data could be collected through existing forms. Other data could possibly be collected through alternative sources and by editing and processing data collected by other agencies and organizations.
Summary of ASA Comments and Advice (including comments on the previous introduction by Bob Schnapp and the following Transmission Draft Summary by Doug Hale) Comments by Howard Bradsher-Fredrick
I presented the paper, “Electricity Transmission Data Needs Focus Group Results,“ at the ASA Energy Committee Meeting on
The advice given by the committee in essence stated that EIA should integrate these approaches more effectively in the future. The committee did not believe that the stakeholder research was sufficiently reflected in the Hale paper nor was the entire research work sufficiently reflected in the Schnapp presentation and final plans and results.
EIA’s Response to the ASA Advice
EIA will use these observations and this advice to attempt to better integrate these components of the research and survey design process. For example, the focus groups should be conducted much earlier in the process so that stakeholder needs and concerns are addressed throughout the development. It should also be clearer how the research results were employed in the final plan for forms development and future work.
Transmission Data for Public Policy, Douglas R. Hale, EIA Senior Scientist on loan to Lawrence Berkeley Laboratory, DOE. At the fall, 2003 meeting Doug Hale briefed the committee on a project to document the Federal Government's needs for transmission data (
The report is now in interagency review, and is available to the committee on the ASA Meeting Home Page. Doug will return from the Lawrence Berkeley Laboratory to join the meeting, answer questions and discuss suggestions and complaints the committee may have with the report.
Suggestions from the
This briefing was to conclude the 3-part session on electricity, and to inform the Committee on the final draft product. The Committee had commented on an earlier draft, which was constructive and helpful. No further advice was sought. Please see previous “Advice” section for summary comments.Estimating Weekly Other Oils Stock, Ruey-Pyng Lu, SMG, EIA
EIA publishes “Stocks of Crude Oil and Petroleum Products” weekly and monthly at
http://www.eia.gov/pub/oil_gas/petroleum/data_publications/weekly_petroleum_status_report/current/pdf/table03.pdf . All components of “Monthly Other Oils” stocks are collected from all respondents in a census. In order to present more timely “Other Oils” stocks, the “Weekly Other Oils” stocks are estimated from the “Monthly Other Oils” stocks. Due to the seasonality and other unexpected disruptions, there are some discrepancies between these two series of stock number.
We want to determine the best way to estimate weekly stocks of other oils excluding propane (based on weekly data for major oil products, plus monthly data for major oil products and other oils products) and to evaluate alternative methods by comparing the estimate to monthly data.
Summary of ASA Advice
Committee Advice:The project is the breakdown of all the petroleum products. You have monthly and weekly crude, motor gasoline, distillate fuel oil, jet fuel, unfinished oils, propane, and other oils stocks, the problem is how to best estimate weekly other oils stock in terms of other petroleum products stocks. The exchange information among Petroleum Supply division staff, Short-term Energy Outlook modeler and SMG was a good outcome from this meeting, you may need to sit down together and come up with a combined approach.
Intended Responses to ASA AdviceSMG will continue work out some alternative methods to estimate weekly other oils stocks based on other petroleum products stocks. Then we will compare the errors of estimates from the short-term Energy outlook model, Petroleum Supply Division’s proposed new methods and other alternative methods.
EIA Survey Testing Methods,
The purpose of this breakout session is to describe, discuss, and seek
EIA has used several methods in the past six years to develop and redesign surveys to meet changing industry structures. These methods include:
Within these broad categories we have used combinations and permutations of these methods.
The Committee was asked to discuss:
Summary of Committee’s Advice
The committee thought that the current methods that EIA was using to test surveys were adequate. They recommended that we do as much testing as practical including both pre-survey design visits and site visits once a draft form was developed. They also thought that testing on form revisions from the first set of test would also be useful.
They also recommended that EIA try to control for the effect of managers and staff being in the same interview at the same time. They felt this might reflect on the candidness of the staff’s responses to EIA questions.
EIA can address the committee suggestions in several ways.
First, when time and resources permit, EIA would always prefer to do several iterations of testing on draft survey forms. If there is not time for new site visits, telephone calls, or emails to participants could be used to follow-up on changes.
Second, to the extent practical we can split the interviews into a manager group and a group with the actual respondents. We will explore how feasible this is from the company’s point of view. A more workable method may be to conduct the interviews with a larger group, and then have a follow-up interview with just the actual respondents. This could be done as a post interview session for an additional ½ hour, or could be done with a telephone follow-up several days after the site interview.
Natural Gas Production Monthly Survey, Inderjit Kundra. The purpose of this survey is to provide monthly estimates of natural gas production at the National and regional levels. The regions are
Summary of Committee’s Advice and EIA’s Intended Response
We presented our proposal for a probability proportional to size sample, including a large certainty group. The committee supported our efforts in designing the sample.
Improving EIA’s Website: Creating a Vision for the Future Colleen Blessing, National Energy Information Center and Melinda Hobbs, Office of Information Technology, EIA. EIA is embarking on a new phase of web development. This session will cover the process EIA has begun to gather information about what works and doesn’t work on the site and how we will use the feedback to develop a web strategy for the future. We will highlight some recent improvements as a hint of things to come. ASA members are encouraged to comment on the specific improvements or on anything related to EIA’s website.
Summary of Committee’s Advice
1. Re the Natural Gas Navigator, Dr. Hengartner said it would be nice to have a little check box, saying I want this one and that one, then all the data would be downloaded into one file.
2. They didn't like our search engine and wanted us to get something else.
3. Some wanted to be able to click on a graph and have the option to either download the data or see the time series.
4. Pull together data from different places into a single place.
5. Key in all the data we have available
6. ASA members all wanted a way to cite a web page for an academic reference. Need some kind of archive version. How to cite a data table? How individual pages can be referenced? We have a moving target and they want frozen files.EIA’s Response
We will pass Dr. Hengartner's suggestion along to the navigator team. The Web committee definitely is looking into different search options and different ways to access our information. This topic was discussed at our recent web workshop and is an issue on the current web audit. About the "click on graph" option and the option to either download the data or see the time series, this is an option we already have and the members hadn't realized it. On the suggestion to pull together data from different places into a single place, we think this is a good idea and are working in that direction.
On the suggestion to key all the data in that we have available we have data in the Annual Energy Review (AER) available on the Web going back to 1960 and in some cases 1949. We will look into making users more aware of the AER as a source of historical data.
The topic of how to cite data from web pages is now being discussed within EIA. Jerry Peabody mentioned configuration management approaches for archiving, but noted that this could involve lots of resources. Susan Holte thought users should make a hardcopy of what they accessed. Following a suggestion from the committee, we looked at how other agencies, like Census, do citations. They cite the date the data were accessed. The IIG plan to work on a style for citing Web pages but decided not to get into the archiving issue now. Since we still present data in publication form, the group didn't think there were a lot of circumstances where we wouldn't be able to replicate the data.
Revising Data Together Across EIA: Issues and Opportunities, Renee Miller and Alethea Jennings, SMG, EIA
We have spoken to the Committee in the past about how EIA has expanded its use of data collected in electric power surveys across the organization, such as natural gas, petroleum, and the integrated statistics publications. As a result, revisions to electric power data affect more than just the electric power publications. In addition, it is easier to revise data on the Web than in hard copy format. These two events have raised questions about the need to coordinate revisions across EIA.
We presented background on EIA’s current revision standard (which the Committee helped us develop in the eighties) and recent events that led to its review. Representatives from each affected office are working together to determine the best way to meet user needs and interoffice goals in support of EIA’s mission. We will present conclusions from recent discussions/debates and will ask for the Committee’s guidance on coordinating revisions.
In addition to thinking about the coordination of revisions, EIA is examining the circumstances under which data are revised. In our attempt to reduce the frequency of publishing revised data, we are examining the situations and events that prompt data revisions. EIA may be revising data more often than is necessary to meet user needs. Various situations cause data to revise, such as resubmissions, benchmarking (to the Annual publication), data corrections (where problems were discovered with the data originally reported) and late submissions. We will present an example of data to illustrate revisions occurring over time. We hope to obtain suggestions and ideas for reducing the number of revisions that we publish.
To satisfy users who are interested in having the latest data available, it has been suggested that EIA provide the latest versions of the data via the Website and inform users of this option. We will discuss some of the advantages and disadvantages of this approach and ask for the Committee’s thoughts.
The committee wanted to see notes that indicate when an NA in a table is represented by zeros in the tabulated data and when the NA represents imputed data.
They did not view frequent revisions as a problem from a user standpoint because different users have different needs and are interested in the data for different reasons. In particular, the committee thought that a table displaying a revision trail (similar to Table C1) should be available to provide additional information about a revised data series.
The committee also expressed concerns about reproducibility. They thought that the user should have access to or be able to reproduce data that were previously posted and currently updated.
Regarding the latest vs. greatest, since this varies with the user, we were advised to balance the cost of revising the data with the value associated with the importance of providing the most up to date data to our users.
EIA’s Intended Response to ASA Advice
Regarding the use of “NA” as zero vs. imputed data, we recognize that this is a problem. Given the new systems that are being developed, we expect that there will be opportunities for exploring ways to address this. The issue of reproducibility was raised within the Interoffice Issues Group (IIG). Several possibilities were suggested. However, no consensus was reached on how or if we should do this.
Survey Quality Assessments at EIA, Tom Broene, SMG
Other statistical agencies and private firms have found it helpful to conduct assessments of on-going surveys on a regular basis. EIA formed an inter-office team in 2003 to develop the procedure for an assessment and determine what data should be collected. This talk will present the matrix used to collect the data and discuss its development. So far, only a handful of assessments have been completed, but we hope to have an assessment for all EIA surveys by this summer.
The committee members participated in a frank discussion of assessing the quality of survey data. The committee had several specific suggestions on conducting an assessment in terms of the data we are collecting, how we collect it, and how to disseminate and use the results. They suggested that we directly ask whether the survey is capturing the appropriate data for both now and the foreseeable future. The committee suggested that we interview staff and managers separately. It was also suggested that working on sharing lessons-learned would increase the positive tone of the process. The committee also suggested that all employees, and especially contractors and new employees, be regularly informed of the impact and importance of EIA’s data.
EIA’s Intended Response to the ASA Advice
EIA plans to implement these suggestions. During our discussions with survey staff, we will ask whether the survey is capturing the appropriate data, and discuss the impact and importance of EIA’s data. We will interview staff and managers separately, and work on sharing lessons-learned to set a positive tone.
Questions regarding EIA’s Spring 2004 meeting with the American Statistical Association Committee on Energy Statistics and this summary may be directed to
The meeting agenda, papers, slides, and information on the committee members and EIA may be found on EIA’s Home Page at http://www.eia.gov/smg/asa_meeting_2004/spring/.
An unedited transcript may be found on EIA’s Home Page under Energy Events, Summary of Advice from the American Statistical Association, at http://www.eia.gov/calendar/asa_overview.htm.