Summary of Comments from the American Statistical Association (ASA)
Committee on Energy Statistics
at a meeting with the
Energy Information Administration (EIA)
April 19 and 20, 2007
1. Modeling Regional Electricity Generation, Phillip Tseng, SMG, EIA
In recent years, natural gas use in
the electric power sector has been on the rise. As a result, impacts of power
generation from natural gas plants on the very tight
This paper has two objectives: 1. Present a model built on historical generation, capacity, and sales data to predict monthly dispatching of coal and natural gas power plants. 2. Demonstrate that the same modeling framework can be used to simulate the effects of capacity loss in a sub-region on regional generation patterns and trade flow. Preliminary model results show that the model can produce good projections. However, the model can be further improved to capture fuel switching for dual fired oil- gas generators.
ASA Committee Recommendations:
One of the key comments from the committee is the use of bin numbers. Several committee members suggested that EIA use the utilization rates directly to create cumulative distribution functions as proxy for coal, gas, diesel, and residual fuel supply curves.
EIA Intended Response(s):
EIA will try the approach and test the results. Previously, EIA had used the CDF approach but did not get good results. We will implement the suggested method and compare the results from both approaches.
2. Petroleum Imports Data, Howard Bradsher-Fredrick, SMG, EIA
EIA is presently investigating Customs/Census Bureau petroleum imports data as a data source or as a means for providing a mechanism for cross checking EIA petroleum imports data. EIA presently collects petroleum imports data on its survey form EIA-814, “Monthly Imports Report.” In order to do so, EIA directly surveys the actual importers of petroleum products, typically large oil companies. Significant resources are expended in conducting this survey.
The Customs Service collects duties on imports in accordance with current policies and laws. In so doing, the Customs Service collects detailed data on each importation transaction. These transactions are made public to subscribers through the Census Bureau on a monthly basis with company-level identifying information removed.
This preliminary work has involved determining to what extent EIA data are quantitatively comparable on a monthly basis with the data provided by the Customs Service. It has also involved determining the comparability of EIA and Customs Service petroleum product definitions and geographical coverage. While it has been discovered that quantitative, definitional and geographical coverage differences exist for some or all products, Customs data could still prove somewhat useful to EIA.
The presentation concerned itself with comparing EIA monthly petroleum imports data with similar data collected by the U.S. Customs Service. The presentation showed that the definitions used by the two sources were quite different in many cases, the data reported were also usually quite different and issues of geographical coverage, timing, etc. could also be significantly different. The primary question posed to the committee was how the Customs data should be best used by EIA, but EIA is also interested in any general advice the committee might have.
ASA Committee Recommendations:
The committee members recommended that EIA interview respondents (to the EIA-814 Form) to discover if, and why, they are reporting different data to EIA than they do to the Customs Service. They also recommended that EIA meet with the Census/Customs staff members and discuss petroleum product definitions to determine if some of these differences in definitions can be reconciled.
EIA Intended Response(s):
EIA (with participants from OOG and SMG) have already arranged a meeting with experts at the Census Bureau to discuss petroleum product classification issues. SMG will also discuss the issue of interviewing EIA-814 respondents with OOG staff members to determine if this is a useful project to undertake.
3. Modeling Ethanol and Other Fuels, Anthony Radich, OAIF, EIA
EIA’s Office of Integrated Analysis and Forecasting has changed the way that liquid fuels are accounted for and reported in the Annual Energy Outlook. Petroleum products are now more accurately called “liquid fuels”, reflecting the potential displacement of petroleum-derived liquids by liquids from coal, natural gas, corn, vegetable oil, and cellulosic biomass. EIA has also changed its accounting for energy use in ethanol and biodiesel production processes. Corn and vegetable oil are now part of total biomass use, since coal and natural gas used to produce liquid fuels are counted as coal and natural gas use.
ASA Committee Recommendations and EIA Intended Response(s):
Committee recommendations are being reviewed to determine and EIA’s intended response is forthcoming.
4. Electricity 2008:
(A) Clearance of Electricity Surveys, Robert Schnapp, Director, Electric Power Division, CNEAF, EIA
Electricity 2008 is a project to assess and revise the Energy Information Administration electric industry surveys to more accurately collect and disseminate information about the electric power industry. The American Statistical Association has been briefed at prior ASA meetings on the issues that Electricity 2008 would be addressing. This presentation will describe the two major efforts to merger six surveys into two. The first would merge the existing Form EIA-906 “Power Plant Report,” Form EIA-920, “Combined Heat and Power Plant Report,” and Form EIA-423, “Monthly Cost and Quality of Fuels for Electric Plants,” as well as transferring operational information from the Form EIA-767, “Steam-Electric Plant Operation and Design Report,” to the proposed new Form EIA-923 “Power Plant Operations Report.” It would also have companies currently reporting on the FERC Form-423, “Monthly Report of Cost and Quality of Fuel for Electric Plants,” report cost and quality of fuel information on Form EIA-923. The second effort would transfer the static information collected on Form EIA-767, “Steam-Electric Plant Operation and Design Report,” to the Form EIA-860, “Annual Electric Generator Report.”
ASA Committee Recommendations and EIA Intended Response(s):
This session was an overview to inform the committee of EIA’s progress in this area. No summary of committee advice or statement of EIA’s intentions is required.
(B) Forms Design and Data Collection, Bob Rutchik, SMG, EIA
This presentation had two purposes. Bob Rutchik first went into detail on how EIA combined the four electric power surveys – EIA-423, EIA-906, EIA-920, and the EIA-767 into the Form EIA-923. Bob then illustrated and asked the Committee’s advice on some of the problems that arose in testing and notifying electric power plants from combining the four surveys. Specifically,
The latter problem is particularly crucial. Currently, one plant receives up to three surveys. That means there are three different respondents or contacts in a plant. With the Form EIA-923, one survey will be sent to a plant, or more specifically one contact. This person will need to get the different schedules to the right people/departments to complete it.
ASA Committee Recommendations and EIA Intended Response(s):
The Committee provided EIA with advice in all these areas, advice which EIA is currently or will be implementing. The committee advised EIA to conduct a combination field and cognitive test. EIA will send the Form EIA-923 to respondents and ask them to complete it and then analyze the data. That is the field test part. After that, it will debrief respondents about their answers. That is the cognitive part.
The Committee added that part of the testing should be on how the plants distributed, or not distributed, the survey among various departments to complete it. The Committee also recommended that a stratified sampling scheme was a “sensible” way to conduct the test.
Further, since the survey will eventually be an Internet Data Collection the Committee said usability testing “is the issue.”
5. National Institute for Statistical Science: Project Descriptions, Ruey-Pyng Lu, SMG, EIA
The Energy Information Administration (EIA) has established an Applied Methodology Student Research Program to be conducted jointly with the National Institute of Statistical Sciences (NISS). This program is designed to engage students in graduate programs in collaborative, interdisciplinary research on topics of interest to EIA. Research projects undertaken by students and their technical mentors at their home universities and their research sponsors at EIA will advance methodology and address specific research questions that are important to EIA programs. Students may apply either for full-time summer support or for research support during the academic year as either a full or partial stipend or as a cooperative arrangement that would provide research access to sensitive microdata. Ordinarily academic year research will be conducted at the student’s university; summer research may be conducted either on-site at EIA or at the student’s university with visits to EIA as necessary to consult with the EIA project sponsor and to access and utilize EIA data, as required by the particular project. Graduate students in statistical science disciplines (statistics, operations research, economics, etc) together with their faculty mentors are encouraged to identify a research area proposed by EIA and to define research objectives in conjunction with EIA staff preparatory to submitting a full proposal. Awards are made based on the strength of the applications. These are renewable based on accomplishments and approved future plans. Citizenship is not required; women and members of minority groups are especially encouraged to apply.
We are in the process of developing specific project statements, in the hope of encouraging students to work on specific projects of use to EIA. We know it is important to provide enough detail so the researcher will have an idea as to the methodology and to data and its availability. So far we have developed draft descriptions of four research projects.
ASA Committee Recommendations:
EIA could develop a couple of different types of grants that people could apply for. One which would be a shorter term, smaller-scale project versus a year-long fellowship in which they could apply. Be more explicit in terms of your project description about the dates and the terms of the contract, EIA could provide more information at your website; Look at the EPA Star fellowship website, it’s a good template to use. Also, EIA may want relationship with the instructor, not with the student.
The Mathematics of Information Technology and Complex systems (MITACS), and the National Program on Complex Data Structures (NPCDS) training initiative has a joint project with Statistics Canada to grant internship to students working on the identified specific projects, and this could be another reference for EIA to develop Applied Methodology Research program.
EIA Intended Response(s):
6. Allocating Municipal Solid Waste to Renewable and Non-Renewable Energy, Marie LaRiviere, CNEAF, EIA
Municipal solid waste (MSW) has widely been viewed as composed principally of biomass. Therefore, EIA and many other sources have historically classified it as a renewable energy source. However, because MSW is composed of both biogenic (renewable) and non-biogenic (non-renewable) material, some members of the energy community have expressed concern that EIA has been incorrectly classifying all energy produced from MSW as renewable. To answer these concerns, EIA has estimated the shares of MSW, both by weight and heat content, derived from renewable and non-renewable materials by year from 1989-2005. The methodology used to calculate the split is documented in this report.
In general, combustible non-renewable materials (e.g., plastics) are characterized by higher heat contents per unit weight than combustible renewable materials (e.g., paper products). Thus, the ratio of renewable to non-renewable material volumes can have a considerable effect on the heat content of the waste stream. Analysis of the calculated split showed the heat content per unit of weight of the waste stream has been steadily increasing over time. This change can likely be attributed to the changing composition of the MSW stream, as increasingly more plastics are being discarded at the same time that decreasing amounts of paper and paper products are entering the waste stream.
EIA will split MSW into renewable and non-renewable components for its future data releases, and the renewable and nonrenewable components of MSW will be reflected in restatements of prior data back to 2001.
ASA Committee Recommendations:
EIA Intended Response(s):
7. Comparison of Different Methods of Computing Yearly Growth Rates For Petroleum Supply, 1995-2004, Carol Blumberg, PD, EIA
This paper is part of a bigger project to compare the congruence of the monthly petroleum product supply values (in thousands of barrels per day) for finished motor gasoline supplied, distillate fuel oil supplied, and total products supplied as reported in various publications of the Petroleum Division of the Energy Information Administration (EIA) for the years 1995 to 2004. The focus of this paper is to compare 15 methods of using the data from these publications when year-to-year growth rates are computed by dividing the estimated average quantity per day for a month in a particular year by an estimate for that same month in the previous year. The monthly-based estimates used in computing these growth rates come from Petroleum Supply Monthly (PSM) and Petroleum Supply Annual (PSA). Weekly-based estimates, from data reported in Weekly Petroleum Status Report (WPSR), are used three different ways to compute monthly estimates. These five different estimates of monthly product supply are then used in various combinations to form the ratios that estimate the year-to-year growth rates. The reason that comparing growth rates based on weekly data, monthly data as reported in PSM, and monthly data as reported in PSA is important is that weekly-based estimates for a month are available much sooner than the others. The weekly-based estimates are available within 11 days of the end of the month while the PSM numbers take up to 60 days to be released and the PSA data take about 6 months after the end of the calendar year to be released. Since the weekly-based estimates are based on samples and the PSM and PSA measurements are based on censuses, the PSM and PSA data are generally more accurate. The PSA numbers contain revisions of the data published in the PSM due to late submissions or resubmissions. So, the question of main interest here is whether growth rates using weekly-based estimates or PSM numbers can mirror closely the growth rates computed using the PSA numbers.
ASA Committee Recommendations and EIA Intended Response(s):
EIA Intended Response(s):
8. Expanding Researcher Access to EIA Microdata, Jacob Bournazian, SMG, EIA
This break-out session was on evaluating alternative methods
for expanding researcher access to energy microdata. One option discussed was to enter into an
arrangement with another federal agency to host EIA data at a
ASA Committee Recommendations:
The committee members recommended pursuing the researcher program through the National Institute for Statistical Science and also pursuing either of the other two options to expand researcher access to micro-level data. EIA would need to provide consultative services to researchers to explain how to use the files. EIA should consider developing synthetic data sets for researchers to access so they can become familiar with the file structure and variables. Researchers should replace the synthetic data with actual company data level after the researcher’s computer programs are working properly. Another consideration is that often, a researcher desires to merge the requested data with other data sets. The discussants’ preference was to host EIA microdata at research data centers where other data series would be available and there is technical support for working with the files and merging EIA data with other outside data series.
EIA Intended Response(s):
EIA’s Responses. EIA will pursue a research program that allows access to microlevel data with the National Institute for Statistical Science. It will consider agency resources for expanding access at a research data center or granting researcher access to microlevel data through some type of remote access arrangement.
9. Update: NEMS Forecast Evaluation Methodology, George Lady, Consultant to SMG, EIA
The NEMS forecast evaluation methodology is intended to measure the differences between NEMS’ forecasts of energy production and consumption; and given this, to partition the differences found among the important, contributory influences, e.g., weather, product prices, and the level of economic activity. There are basically three methods that may be used for this purpose:
Method #1: Archive the versions of NEMS used to make forecasts. When the time comes, re-run NEMS substituting actual with assumed values for important assumptions. Configure the runs to isolate the impact of important influences.
Method #2: When forecasts are made initially, design and implement a roster of NEMS solutions designed to isolate the influence of important assumptions. Pool the solution data and summarize the model output through regression analyses of the implicit, embodied market supply and demand for energy products. When the time comes, use the regression equations, or other derivations based on the solution data, to isolate the impact on the forecast value of differences between assumed and actual values for the important assumptions.
Method #3: The same as Method #2, except limiting the solution set used to the NEMS scenarios prepared for the Annual Energy Outlook.
Initial results of using Method #3 were presented to the ASA on October 6, 2006. At that time the committee approved of the goals of the evaluation method and proposed approach.
For the update, presented April 20, 2007, results based on both Method #2 and Method #3 were presented. Special runs were made by OIAF staff to isolate the weather effects in the AEO2007 version of NEMS. Over 70 NEMS solutions were made available to the project that isolated the effects on energy demand of energy product prices, based upon the AEO2006 version of NEMS. The results presented to the Committee included the weather and price sensitivities from these runs as well as regression results based upon data pooled with these solutions. Method #2 was proposed to the committee as a practical way to isolate the impact of explanatory variables when comparing forecasts with the eventual, historical data.
ASA Committee Recommendations:
Written comments provided by Committee members before the presentation identified Method #1 as the best way to undertake the impact analysis. Nevertheless, this recommendation recognized the substantial, practical difficulties of archiving, and then reestablishing, computer environments and associated software, potentially decades after they were current. Given this, the regression approach was accepted as an acceptable, practical alternative.
In the discussion following the presentation, Committee members agreed with the purposes of the evaluation method and accepted the regression approach, based upon specialized NEMS solutions, as the best practical method for conducting the forecast evaluation. Concern was expressed over the experimental design used to configure the NEMS solutions to be used. The runs presented to the Committee were based upon the variation of a single assumption compared to a reference case. Other methods, involving multivariate changes in assumptions were identified and recommended for consideration.
Intended EIA Response(s):
SMG has a project underway to continue to develop and implement the evaluation of NEMS forecasts. One of the project tasks is to assimilate selected NEMS modules, those projecting the demand for energy products by the residential, commercial, industrial, and transportation sectors, and be able to run the modules with respect to selected variations in important input assumptions. Following the advice of the ASA Committee, the literature on designing solution sets for the purpose of impact analysis will be reviewed and; subject to project resource limitations, a multi-variate design will be formulated and implemented. The solution data generated by this process will then be pooled and submitted to a regression analysis. The impact of assumptions on forecasts differences will then be assessed using the regression results.
This briefing was to conclude the 3-part session on electricity, and to inform the Committee on the final draft product. The Committee had commented on an earlier draft, which was constructive and helpful. No further advice was sought. Please see previous “Advice” section for summary comments.