Summary of Advice to the Energy Information Administration from the ASA Committee on Energy Statistics

October 25-26, 2001

1. General Session: Introductory remarks from the Acting Administrator, Mary J. Hutzler, Energy Information Administration, and Acting Director, Nancy J. Kirkendall, Statistics and Methods Group, EIA

These opening remarks to the Committee neither sought nor received recommendations.

2. General Session: Briefing on Convergence Experiments With the National Energy Modeling System, work of the first ASA Fellow, Dr. Steven Gabriel. Briefing by Andy S. Kydes, Senior Technical Advisor, Office of Integrated Analysis and Forecasting, EIA.

This briefing to the Committee neither sought nor received recommendations.

3. General Session: Briefing on EIA's two most recent multi pollutant studies for the U.S. Senate: Analysis of Strategies for Reducing Multiple Emissions from Power Plants. EIA analyzed various strategies for reducing power plant emissions of sulfur dioxide, nitrogen oxides, carbon dioxide, and mercury. Briefing by Scott Sitzer, Acting Director, Office of Integrated Analysis and Forecasting, EIA.

This briefing to the Committee neither sought nor received recommendations.

4. General and Break-Out Sessions: Estimation of Natural Gas Production Data by John H. Wood, Director, Reserves and Production Division, and Beth Campbell, Director, Natural Gas Division, Office of Oil and Gas, EIA.

There is a new imperative to get earlier and better natural gas production estimates in order to provide earlier and better information and analysis about price changes, responses to price changes, and the periodic large differences between measures of supply and demand. Traditional surveys based on States reporting their relatively final monthly data to EIA cannot meet this need for earlier and better natural gas production data. Therefore, EIA is proposing a new methodology for gas production estimates that use the characteristics of State reported monthly data as it evolves from earliest preliminary data to final data. Our fundamental performance metric for these estimates is that they are made within 90 days of the close of the production month and that they are within plus or minus 1 percent of final production estimates 9 out of 10 times. These production estimates will be based on State data, but made by EIA. The responsibility for their quality, accuracy, and reliability will rest with EIA. This is in sharp contrast with just collecting and publishing the data reported by a State as final or the closest thing to final that they have. Texas, which produces about 25% of U.S. natural gas, is our test case. This State generally releases its first preliminary data 45 to 60 days after the end of a production month; January 2001 first preliminary data was reported in March 2001. This January data will usually be revised each subsequent month as more data is corrected, received, and processed. Even though Texas publishes A final@ data for a year only a few months into the next year, the correction and revision process of monthly gas production data actually goes on for a number of years. The new methodology results for Texas monthly production from 1995 through 2000 will be presented along with recommendations for disseminating the monthly estimates, estimate revisions, and State reported data.

Summary of ASA Committee Recommendations

The Committee members participating in the general and later the break-out session generally agreed that (1) early and frequently revised EIA natural gas production estimates would be better than the currently used survey procedures; (2) earlier estimates are worth wider confidence limits, and generally had mostly pro=s for 3A, 3B and 3C, the questions put before the committee. However, without the full data set and/or a more detailed discussion of the procedure, they could not really evaluate the procedure. They asked that the data set be made available and that a more detailed description of the statistical methodology be presented at the Spring 2002 meeting.

Intended EIA Reactions

EIA has already provided Committee members with the data set. Staff from the Reserves and Production Division will work with staff from the Statistics and Methods Group to develop and present a paper of the estimation procedure that includes a step by step description, rational for choices of statistical methods, description of fitting methods, and a detailed presentation of all equations, their derivations and interpretations.

5. Break-Out Session: How to Implement Significant Survey Redesigns: Form EIA-176. Presentation by Beth Campbell, Office of Oil and Gas, and Stan Freedman, Statistics and Methods Group, EIA.

Form EIA-176 "Annual Report of Natural and Supplemental Gas Supply and Disposition" is a complex multi part annual form which EIA has used since 1980. The one form collects data about a variety of activities from several categories of respondents and is the core of the EIA natural gas data collection program, affecting monthly as well as annual data presentations through sample selection and estimation of supply and disposition categories not collected in monthly surveys. As a result of cognitive interviews with survey respondents, meetings with data collection staff, and assessments of changes in industry operations, EIA believes that the EIA-176 should be redesigned. Collection authority for the EIA-176 expires at the end of 2002 and EIA must invite comments and propose revisions during 2002. EIA sought the Committee's suggestions for managing significant survey redesigns, including suggestions for useful actions and for determining the tradeoffs between incremental change and substantial change.

Summary of ASA Committee Recommendations

The ASA committee members agreed that Form EIA-176 was overly complicated and simplification was probably useful but they emphasized that it was important to identify and prioritize objectives for change. Other objectives, besides simplification, were discussed. These included:

• Improving the relationship between the monthly and annual forms to improve clarity and possibly reduce duplicative collection effort.
• Reducing effort and content for the annual form to increase resources for more complete and accurate monthly forms.
• Improving the relationship of forms content to current market structures and activities.
• Improving the form clarity in all ways which would reduce nonsampling error.

To resolve issues about priorities, the Committee suggested that it would be useful to do interviews with both data respondents and data users.

The Committee noted that whenever any significant change is undertaken in a survey system, there will be errors and problems. They recommended taking the time to conduct changes carefully. Examples included collecting data in new and old ways or staging change with a pilot for the new format. Parallel operations or pilots would provide useful learning opportunities. The Committee noted that the time remaining for review of the EIA-176 is very short. A certain amount of skepticism was expressed that all the considered changes could be addressed by March 2002. Without time for careful consideration, the Committee recommended correcting the most glaring problems and making minimal change until new goals are set.

Intended EIA Reactions:

6. Status Report: EIA's System for Analyzing Global Energy (SAGE): Barry Kapilow-Cohen, Office of Integrated Analysis and Forecasting, EIA.

EIA is currently developing a new technology-based international energy forecasting and policy analysis model. This modeling system expands upon the MARKAL modeling system which has been used in many countries including the US for over two decades. This presentation will briefly summarize the modeling framework and the objectives and timeline of the project. The presentation emphasized the new tools that have been developed to enable the EIA to develop 15 consistent models for the different regions and countries included in the initial model design. Feedback from the ASA on our system for generating regional models was considered to be particularly helpful at this stage of development.

Summary of ASA Committee Recommendations

This briefing to the Committee, somewhat awkward due to difficulties with the hardware and software, was intended as a demonstration. The purpose was to facilitate discussion and questions. There was no formal discussant. The Committee members asked lots of clarifying questions, and expressed an interest in model run results once the components are assembled.

Intended EIA Response

EIA's new technology-based international energy forecasting and policy analysis model continues to be of interest to the Committee, and will be the subject of periodic briefings at subsequent ASA Committee meetings.

7. A General Session: Survey of Steam-Only Facilities Using Biomass Feedstocks by John Carlin. EIA currently conducts surveys of energy facilities that produce electricity or cogenerate but does not conduct surveys of energy facilities that only produce steam. The focus of this paper is to assess the feasibility and cost-effectiveness of expanding and refining our information systems to include surveys of energy facilities that use biomass feedstocks to produce only steam. This assessment will include a presentation of how biomass energy production is currently estimated for inclusion in our energy summary statistics published in the Annual Energy Review and the Renewable Energy Annual. The questions on the new survey would be directed at gaining annual energy production information, technical performance characteristics, and resource availability information instrumental in constructing economic supply curves for the biomass industry. The potential scope of the new surveys is the Forestry and Logging Industry, Manufacturing Sector, NAICS 22 (North American Industry Classification System), Municipal Waste Combustors, and Municipal Landfills. The potential survey frames may be known, unknown, or part of a confidential Bureau of Census frame. The surveys may cover the population or may be based on samples of the population. All of the energy production statistics must be additive so totals of national annual production for each biomass fuel category are available.

Intended EIA Response: EIA will continue this approach.

8. General Session: An Update on EIA Performance Measures by Tom Broene. This session was an update on performance measures for data quality, and included a recap of an earlier presentation and the advice received from the ASA Committee at the Spring 2001 meeting, old and new checklists, and a brief summary of the March 2001 Journal of Official Statistics (vol 17 no.1) which dealt with quality of data from National Statistical Agencies, and is available online at http://www.jos.nu/Contents/issue.asp?vol=17&no=1

Reactions from the ASA Committee

This presentation was quite brief and had no formal discussants. Although the committee had no direct comments for the author, a general discussion developed immediately after and covered three main topics:

The acting administrator asked the committee what data points they would like to see early, even if with less accuracy. Committee replies included:

• Could obtain this information from your customers, but need to phrase the question right.
• Could distribute to customers a list of data sources, which would include EIA. Customers would then rank all on list of terms of various attributes.
• Ask your customers what they would do if they did not have EIA data.
• Discussed whether or not a range would be suitable for customers, but did not, in the end, recommend using a range.

The acting administrator asked each committee member in turn what data quality measures they would want to see if they were the administrator. Replies included;

• Are really looking at two types: 1) input measures, such as the layered measures; and 2) output measures, where we should look for the users to indicate what they think is important.
• Look for something correlated to what you produce, so that you can compare.
• Would be concerned about the imputation: Is concerned that the method not assume that data is missing at random.
• If the respondent does on-line editing, then we will lose information about the editing performance.
• An EIA employee commented that item non-response is not straight-forward to calculate since some fields are optional depending on other fields.

Discussed the Journal article claim that customer surveys do not provide useful measures of data accuracy.

Planned EIA Response:

• There will be more discussions with EIA senior management on these topics.

• We do not anticipate changing our telephone customer surveys to add the suggested questions; but will consider them for other customer contacts
• The comments on editing, item response rates, and imputation procedures will be taken under advisement.

9. New Modeling and Forecasting Approaches: Are Markets Changing, and Should We Change Models to Reflect the New Paradigm? Mark Bernstein, ASA, to lead a brainstorming session with other ASA Committee members.The intention of this session was to have a freewheeling, out-of-the-box thinking discussion of how energy is changing and whether that has implications for how we model it.

While there has always been uncertainty in energy markets, the level of uncertainty has increased considerably. The business community, in California in particular, and in the US generally, has begun to change the way they look at energy. They are recognizing that they may need to take a more active role in energy decision-making. While in the past, consumers of energy were relatively passive; there are indications that this is changing. Businesses may begin to look at energy as a volatile commodity that they need to treat with hedging strategies.

In California companies are looking at a variety of alternatives as means of hedging energy prices, from reducing demand, to generating their own electricity, to making sure they have a portfolio of energy options. It also appears that consumers in California responded to both information campaigns and rising prices to reduce demand for electricity.

Events of September 11^th may also have an impact on how the general public and industry perceives energy. This might change decision making on the types and price of energy. It is not inconceivable that an energy tax for security purposes may be proposed (which 2 months ago we would have said is impossible). This leads to a number of questions for discussion in this session:

If the decision makers and the decision parameters are changing, and if there are fundamental shifts in the markets for energy, how (if at all) should this impact how we model and forecast energy demand, supply and prices?

10. General Session on Alternative Fueled Vehicles Fred Mayes: The Office of Coal, Nuclear, Electric and Alternate Fuels (CNEAF) within EIA is responsible for providing information on alternative fueled vehicles (AFVs) and on alternate transportation fuel (ATF) consumption. Currently, AFVs (vehicles operating on fuels other than gasoline or diesel) comprise about 0.25% to 0.5% of all Aon-road@ vehicles in the United States. CNEAF now conducts a survey (Form EIA-886) which gathers certain AFV and ATF information, but relies upon analytical procedures to estimate the total number of AFVs and total ATF consumption.

This paper provides a background on AFVs in the U.S., and solicits the Committee=s input on many challenges CNEAF faces in providing complete and accurate information on AFVs and ATFs. Major questions which the paper will address were:

1. Should CNEAF continue to use both survey and estimation procedures, or should it eliminate its estimation process?

2. If surveys should be conducted:

ASA Committee Recommendations:

1. Survey universe of users and eliminate procedure for "estimating" U.S.

AFV population

2. Look to State inspection records as public source for AFV user frame

3. Look to industry trade associations (not AFV trade assn's.) for assistance on frame

4. Importance of municipal government AFVs

5. Don't worry about "pre-EPACT" AFVs (mostly propane)

6. Stress OEM (original equip. manuf.) AFVs; don't worry about conversions.

EIA=s Intended response

(EIA response is in process and will be provided.)

11. General Session: Measuring Consumption by Energy-Use Sector: Options for Obtaining Data in the Future Renee Miller. EIA obtains monthly and annual consumption data through collections from energy suppliers. Natural gas and electric utilities often use their service categories to classify the data. But now that pipelines are delivering for the account of others they do not necessarily keep data by energy use sector. EIA needs to start thinking about ways to obtain the data, if respondents can no longer report it. This was a short overview at the general session to give the committee an idea of a problem we will be facing.

Reactions from the ASA Committee

The Committee pointed out that it is important to have sectoral level data that is consistent over time and that EIA should, if necessary, seek legislation to require that the data be provided. They also noted that the respondents we have been trying to collect the data from may not be the only sources of the data within the company.

Planned EIA Responses

EIA will work with one of the Committee members to find other contacts to pursue within companies.

12. General Session: Clarification of Small Area/Imputation Technique, and Study of Resulting Bias James R. Knaub, Jr. This discussion addresses questions raised by the Committee at the November 2000 meeting pertaining to the new methodology being used in electric power surveys. The questions were with regard to mechanics and results. The Committee suggested that standard errors could be calculated more directly, and the possible bias resulting from model failure was also a source of concern. In the case of the former concern, a better explanation of this method in terms of small area estimation was given to show the advantages of this simplified procedure. To address the latter concern, a study was conducted using electric generation data (which has much greater variance than the sales and revenue data). Standard errors and biases were examined through test results to determine reasonable parameters for application to data publication. A literature search was also conducted. References for this study are available, and this presentation provided more detailed illustrations.

ASA Committee Recommendations

One Committee member again expressed concern over the use of cutoff samples. EIA pointed our that more than 200 test data samples taken from census data were all simulated cutoff samples. Concern was also expressed about the approximate variance formulation used, saying that it might be what caused thicker tails in the z-value histogram for the more tailored model formulation. However, another Committee member pointed out that the recommended solution (providing a more robust estimate of total and guarding against underestimate of total error) was such that the z-value histogram in that case had tails that appeared quite thin, and that there was a need to clarify some other graphs. A final suggestion was to use Q-Q plots.

Intended EIA Responses

The paper on the EIA/ASA Meeting web site will be revised to reflect Committee comments on variance. The variance estimate will remain as it is since it is an important feature of this methodology to simplify operations. To process the data as simply as possible is a huge advantage. This method works well in tandem with simple, graphical editing using scatter plots to compare data sets. For these and perhaps a number of other reasons, it was a major breakthrough to find something this simple that works this well.

13. General Session Information Quality Guidelines and the Standards Project: Nancy J. Kirkendall and Jay Casselberry

The Office of Management and Budget has established new requirements concerning information quality to be followed by all Federal agencies. The new OMB guidelines are available at

http://www.whitehouse.gov/omb/fedreg/final_information_quality_guidelines.html

EIA has two coordinated efforts underway. Nancy Kirkendall is leading an interagency team from statistical agencies to plan a coordinated approach (perhaps a single federal register notice). Jay Casselberry is leading an EIA wide team to plan its detailed approach (including updating and coordinating standards).

ASA Committee Suggestions:

1. Suggest looking at EPA, WIPP, and Yucca Mountain to see how they handle data quality objectives.

2. For response rates, NCES uses a minimum response rate for surveys and any survey with a lower response rate needs to take additional steps to ensure the quality of the information.

3. Quality standards need to address agency reviews for compliance as required by OMB's guidelines.

4. Models should keep a log of changes to help users identify what changes occurred and when.

5. Public complaints tracking system: The Committee was concerned that if it is too easy to explain, EIA may get a large number of complaints.

EIA Intended Responses:

1. First priority has been to look at standards used by Federal statistical agencies. I plan to look at the web sites for EPA, WIPP, and Yucca Mountain to see what materials they present with regard to data/information quality and utilize it, as appropriate, when developing our proposal.

2. For EIA standard on response rates, added wording to indicate we will also address conditions for response rates before information is disseminated.

3. Created a proposed standard to address reviews to ensure adherence to quality standards. Category I Reviews have been addressed via EIA policies which will be considered along with standards in this project.

4. Model standards modified to include wording on documenting model changes.

5. EIA is proposing to formalize the public complaint process to accurately identify, track, and respond to complaints about compliance with our quality guidelines.

6. Category 1 Reviews have been addressed via EIA policies - Policies will be considered along with Standards in this project.