1. General Session: Introductory remarks from the Acting Administrator: L.A. Pettis; and The Status of the EIA-sponsored ASA Fellowship: Nancy Kirkendall, Statistics and Methods Group
These opening remarks to the Committee neither sought nor received recommendations.
2. General Session: Progress Briefing on the International (MARKAL) Model Development: Mary Hutzler, Office of Integrated Analysis and Forecasting.
This briefing to the Committee neither sought nor received recommendations.
3. General Session: Update and Results of Cognitive Testing of EIA Graphics: Herb Miller, Statistics and Methods Group; with coauthors Colleen Blessing, National Energy Information Center, and Howard Bradsher-Fredrick, Antoinette Martin, Renee Miller and Robert Rutchik, Statistics and Methods Group.
Abstract: Update on results since the November 2000 meeting; progress, next steps and questions to the Committee.
Committee Reactions: The Committee thought the cognitive approach to testing EIA graphs was good and agreed with all of the conclusions and recommendations that were presented. A suggestion was made to include an explanation of the process in the write-up of the results. A reference to "Cognitive Aspects of Designing Statistical Maps," conducted by the National Center for Health Statistics was also provided as background information. We received an excellent presentation from Nicolas Hengartner on "What are Good Graphs?" Deficiencies in some of EIA's graphs were also presented - i.e., titles can be too wordy. Advice was also given on dual y-axis graphs, i.e. try to ensure that the lines do not cross. The advantages of multiple bar graphs compared to stacked bar charts was also presented.
EIA Response: The EIA team will incorporate committee recommendations and will next discuss how to get the recommendations implemented throughout EIA.
4. General Session: Out One Door and In the Other - Knowledge Sharing In a Period of Transition : John Weiner, Colleen Blessing and Renee Miller, EIA's Quality Council.
Abstract: Anticipating the imminent retirement of a significant number of long-time EIA employees and the arrival of a new generation of EIA staff, the EIA Quality Council established a series of knowledge and experience sharing activities. This presentation was a report to the Committee on the status of these activities and a request for comments and suggestions on other activities that have worked well in their organizations.
Committee Reactions: Generally, the ASA Committee agreed with the need to address knowledge-sharing issues, and how EIA proposes to do so. Major comments: (1) EIA should pay special attention to good documentation and up-to-date files, which are any organization's insurance policy, (2) As the senior generation retires, EIA might use the opportunity to consider strategies to pass authority to younger employees, (3) "EIA 101" courses should be mandatory, and their scope and frequency should be expanded, (4) successful recruitment requires much closer contact with top candidates throughout the application process.
EIA Response: The team will take all the ASA recommendations back to the EIA Quality Council.
5. Break-out session: EIA Interagency Project: Disclosure Auditing System: Mark Schipper , Office of Energy Markets and End Use.
Abstract: EIA, jointly with six other federal statistical agencies and the interagency Confidentiality and Data Access Committee, has been developing disclosure auditing software. To protect the confidentiality of survey respondents, statistical agencies, suppress table cells that might reveal individual/company data. While agencies sometimes use sophisticated suppression software, most software does not provide an evaluation of its suppression pattern. Auditing - estimating the minimum and maximum values for each suppressed table cell - provides information for such an evaluation. This was the third discussion with the committee about the audit project and included a demonstration of the beta version of the Disclosure Auditing System (DAS).
Committee Reaction: Although time was limited due to the lengthy demonstration, one of the Committee members related that the Benford distribution was discovered by the American astronomer Simon Newcomb in 1881 who noticed that the first pages of books of logarithms were soiled much more than the remaining pages. In 1938, Frank Benford arrived at same formula (below) after a comprehensive investigation of listings of data covering a variety of natural phenomena. In general, "…if the numbers under investigation are not entirely random but somehow socially or naturally related, then the distribution of the first digit is not uniform. More accurately, digit D appears as the first digit with the frequency proportional to log10(1 + 1/D)."
EIA Response: This was a follow-on presentation near the end of the software development project. This was primarily an update. Concerning the interesting item noted above, the software does not currently make use of a distributional assumption for the rounded values.
6. Break-out session: How to Develop "Emergency" Surveys: Suggestions for "Cutting Corners". Beth Campbell, Office of Oil and Gas; Stan Freedman, Statistics and Methods Group.
Abstract: The focus of this session was on survey design methodology not particular surveys. The Committee was asked to suggest ways EIA should do things differently to move from "quick and dirty" to something better Examples were drawn from EIA experience with the EIA-911.
Committee Reaction: For short lead time surveys, the Committee emphasized the importance of early focus on the analysis objectives before designing the data collection approach. The Committee proposed that emergency surveys were probably not a miniature of other survey efforts - i.e., something that was done as business-as-usual but faster or with some cutting of corners. They advised thinking about the management effort in a different way. They particularly thought it might be useful to look for parallel work in different areas, to prioritize the work by complexity of problems, and to use formats that would be familiar to respondents. Finally, they suggested that EIA prepare an "after-action" summary of their efforts on emergency surveys and take it to specialists in short turnaround surveys in the private sector or other agencies to ask for advice. In this way it would be possible to develop a record and assessment of the strengths and weaknesses of emergency surveys which could be shared in the future with originators of similar requests. They felt EIA should be prepared to go back to originators to discuss and amend the request based on the assessments of these emergency surveys.
EIA Response: EIA will prepare an "after-action" summary, and will give serious consideration to asking for advice from other experienced parties.
7. Break-out session: Monte Carlo Analysis of Uncertainty in Greenhouse Gas Emission Estimates, Perry M. Lindstrom and Souad A. Benromdhane, Office of Integrated Analysis and Forecasting.
Abstract: In response to "best practices" guidelines under development by the Intergovernmental Panel on Climate Change (IPCC), a "Tier 2" procedure for uncertainty analysis is applied using Monte Carlo simulations. Monte Carlo techniques are used to combine probability distribution functions (PDFs) for various sources of energy-related greenhouse gas emission data. The advantage of the Monte Carlo approach over the "Tier 1" procedure (the square root of the sum of the squares) is that it allows for the combination of normal and skewed PDFs that reflect as realistically as possible the nature of the various parameters. This is the first phase of the work which will eventually encompass all the major greenhouse gases. (This is further follow-up from the previous 2 meetings.)
Committee Reactions: The committee pointed out that presenting the range observed in simulation (minimum and maximum values) is not an appropriate measure as these values will go to infinity as the number of iterations go to infinity. They suggested that an interval in terms of percentiles(e.g. from 2.5 to 97.5 percent) be presented instead. Also, the committee felt that a normal distribution was probably the best assumption to use for all the Probability Density Functions (PDFs) and that it was not necessary to recreate the exact distribution from the original data at the level of aggregation at which this analysis is being performed. Therefore, they suggested abandoning the use of the uniform distribution. Finally, the committee urged that the work consider dependence across sectors and across time.
EIA Response: The results will be presented using an interval in terms of percentiles in the final paper. Where possible the simulations will be run using normal distributions in the place of uniform distributions. If time and resources permit we will consider dependence among variables. This later point may be limited by the choice of software that is being used for the study.
8. General Session: Analysis of Strategies for Reducing Multiple Emissions from Power Plants: Scott Sitzer, Office of Integrated Analysis and Forecasting.
Abstract: At the request of the House Government Reform Committee, EIA analyzed various strategies for reducing power plant emissions of sulfur dioxide, nitrogen oxides, carbon dioxide, and mercury. The Committee also requested an analysis of the impact on these emissions of a Renewable Portfolio Standard (RPS). The first part of the study, released at the end of 2000, analyzed the costs to consumers and the industry of reductions in SO2, NOx, and CO2. The second part of the study will be released this spring.
The analysis examined a number of cases specified by the Committee, including cases that required reductions of each emission separately, plus integrated cases. The targets specified for SO2 and NOx were 75 percent below their 1997 levels, with a compliance date of either 2005 or 2008; for CO2, the target was a level of emissions equivalent to 1990 levels by either 2005 or 2008; and 7 percent below 1990 levels by the period 2008-2012, similar to the requirements of the Kyoto Protocol.
The general findings were that an integrated approach would result in lower costs than a "piecemeal" approach; that reduction requirements for SO2 and/or NOx would have little impact on electricity prices; that reductions in CO2 would have much more impact on prices, either alone or as part of an integrated strategy; and that the costs of CO2 reduction requirements would dominate in an integrated case assuming SO2 and NOx reductions as well. There is also considerable uncertainty about the ability of the electricity and natural gas industries to meet such requirements by the earlier 2005 target date.
This briefing to the Committee neither sought nor received recommendations.
9. Break-out Session: Verifying an Electricity Model; Douglas Hale, Statistics and Methods Group.
Abstract: This paper presents the preliminary results of a project to compare the prices calculated by an optimal power flow representation of the Pennsylvania, New Jersey and Maryland power area with market data. The paper relates the correspondence of actual and calculated results to the quality of publicly available data, especially cost data. The work is underway and there is real risk that the correspondence may be poor.
Committee Reaction: The committee suggested that EIA look at whether capacity withholding from 11AM until about 4PM would improve the fit of calculated to actual prices.
EIA Response: The committee's recommendation will be taken.
10. Break-out Session: Frequently Asked Questions About Survey Response Rates: Ruey-Pyng Lu and Nancy Kirkendall.
Abstract: In this session we will discuss survey response rates in light of a project currently undertaken by EIA staff in support of OMB's statistical policy branch. The purpose is to provide agencies and OMB personnel who approve surveys with information about response rates: how to compute them, how to decide whether response rates are acceptable, how to encourage response, and how and when to assess whether there is nonresponse bias. The resulting paper will be part of an OMB product entitled "Frequently Asked Statistical Questions".
Committee Reaction: The Committee thought that the survey response rate
project is a challenging task. They did not like the regression model that was
used in the original FASQ to estimate response rates because it can project a
response rate greater than 100%. They agreed that it is important to get a
sufficiently high response rate, and recommended that our guidance should
include ways to make sure that in low response cases data can be used to support
analysis. In particular, we should discuss:
EIA Response: EIA will incorporate these ideas in a revised draft of the FASQ on response rates and give the committee an opportunity to review itin the future.
11. Break-out Session: How Does EIA Measure the Impact of its Data? Howard Magnas.
Abstract: EIA's current mission statement says: "EIA is a leader in providing high quality, policy-independent energy information to meet the requirements of Government, industry, and the public in a manner that promotes sound policymaking, efficient markets, and public understanding." Currently EIA uses as a measure of "sound policymaking" the number of information products for Federal policymakers per year (including briefings, testimony, reports); as a measure of "efficient markets" it uses the total number of web site visits in a year; and as a measure of "public understanding" it uses the number of media citations of EIA data in a year.
But measuring our impact on markets is difficult. As a next step towards this, we asked economist Edward Leamer to consider the approaches for evaluating the impact our Short Term Energy Outlook (STEO) forecasts have on markets. His paper will provide a springboard for other approaches to this admittedly difficult problem.
Committee Reaction: Participants discussed EIA's ongoing collection of performance measures (web hits, media citations of EIA data and Congressional and Policy briefings) and utilizing user surveys to measure impact of EIA data. The committee suggested that we add citations in academic and research work to the two media measures which we currently track. They also asked if we could track data references, where researchers are using our data and performing their own forecasts and analysis. ASA also suggested developing a list of important customers including others that produce energy forecasts, the association of energy economists, etc. They offered to help in the development of such a list. Individuals on this list could be contacted to get input on the use of our products, and their importance to users. One of EIA's customers was present as an observer and also offered to help in this effort.
EIA discussed the high cost of user surveys and EIA's projected budget cut for 2002. ASA suggested using conjoint analysis to developing a ranking for products that are being considered for elimination -- this would provide user input to the development of budget decrements when budgets become tight. A discussion emerged on the relative value and importance of EIA data projects compared to analysis and modeling work. ASA participants felt that EIA analysis and modeling projects were of less importance than EIA data; private modeling firms use EIA data in their models and these private modeling firm's models were used more extensively than EIA's.
It was suggested that we collect data to see whether futures prices move when EIA comes out with surprising data. This may be a long-shot, but it is also low cost. A committee member pointed out that USDA produces crop forecasts, and they do see futures prices move.
EIA Response: EIA will work to obtain citations in academic and research papers and references to use of EIA data in forecasts and analysis. We plan to develop a list of important customers and review it with committee members. We will then plan on an approach to solicit information from these customers concerning the use and importance of EIA products. The suggestion to examining the relationship between EIA releases and futures prices will be shared with other EIA offices.
12. General Session:Measures of Data Quality for EIA Surveys: Tom Broene, Statistics and Methods Group
Abstract: EIA conducts surveys of many different populations, and our knowledge of the quality of our data has varied from one survey to another. We have not prepared a full quality profile for many years, and are looking at alternatives that are easier for us to produce and for our users to digest. In order to obtain better and more uniform indicators of the quality of our data, we are seeking to implement a minimum standard set of measures for data quality. This paper reviews recent literature, past work within the agency, similar work elsewhere, and our recommended set of measures. We are particularly interested in the Committee's opinions on whether we are missing key items.
Committee Reactions:
The two ASA Committee discussants pointed out several items needing clarification or refinement. Both discussants thought that our summary of work elsewhere was thorough, and that our recommended quantitative measures were a good start. But they suggested more direct measures for several areas including frames, imputation, edits, and measurement error and agreed that having and using a checklist for survey managers to record whether, when and how often specific activities are conducted would be useful.
The question of whether or not to release information to the public was formally presented with a list or pro's and con's. Many useful suggestions were received from the committee, including displaying item response rates as a histogram across items, and limiting our measures to a few key items from each survey.
One committee member voiced the opinion that a composite quality index would not be a good idea.
EIA Response: EIA will proceed to implement selected performance measures in their newly revised computer survey systems, and will pursue the development of a checklist to be used by survey managers. We will also consider alternative ways of presenting the performance information. The opinion about a composite quality index may help us in evaluating future requests for this type of index.
General Session: Invited Paper Session at the Joint Statistical Meetings. Carol Gotway Crawford, ASA Energy Committee.
Abstract: Every year ASA Committees are allowed to request an Invited Paper Session at the Joint Statistical Meetings This year requests must be received by ASA in the June, 2001. The requests are all reviewed by the Committee on Meetings (this year, in August, 2001), and the winners are selected. Winners then have an invited paper session at the next Joint Statistical Meetings (August 2002).
This is an opportunity for the Committee to contribute ideas and potentially to participate. We are looking for perhaps 4 issues of interest to EIA -- with 2 EIA people and 2 Committee people to make presentations at the August 2002 Meeting. EIA is inviting a Committee discussion to stimulate this dialogue and see that this happens.
This discussion was not advisory in nature.
14. Break-Out Session: Electricity 2002: New Data Forms and New Confidentiality Policy. Dean Fennell, Office of Coal, Nuclear, Electricity and Alternative Fuels.
Abstract: EIA has been conducting an analysis of its electricity data collection forms for the past two years to determine what information should be collected in light of the changing structure of the industry. To accomplish that, EIA established the Electricity 2002 Project, which the ASA was briefed on at its last meeting. The purpose of this presentation is to summarize the proposed changes to the data collection forms and the proposed new data confidentiality policy. Both of these proposals were released to the public for their comment the week of March 12, 2001 in the Federal Register. EIA plans to submit its final proposals to the Office of Management and Budget in the fall.
Committee Reaction: The committee thought that the approach used by the Electric Power Division (EPD) in developing data requirements and adopting new confidentiality policy was appropriate. The committee suggested that a memorandum of understanding between EIA and State Public Utility Commissions might be one way by which confidential data could be shared. They also proposed that the EPD use a systematic approach for soliciting comments about confidentiality in addition the voluntary comments received from the Federal Register process. The systematic approach would require a mandatory response to a separate survey, which would require the approval from the Office of Management and Budget OMB.
EIA Response: The EPD looked into developing memoranda of understanding with the Public Utility Commissions a few years ago and learned that Federal agencies can only have formal agreements concerning the sharing of confidential data with other national government entities. EIA needs to send the Electric Power forms to OMB for clearance soon if data are to be collected beginning in January 2001. The timing constraints make the separate systematic survey approach to soliciting input concerning confidentiality untenable for the current forms. However, the committee's advice will be considered in the future.
Questions may be referred to: