EIA’s Frames:  How Do We Know if They are Sufficient? Grace Sutherland, Introduction, with participation from Renee Miller, Howard Bradsher-Fredrick, Shawna Waugh and Alethea Jennings, SMG, EIA

The quality of EIA’s data has been made a priority and has been made part of its Strategic Plan.  Goal 1 of the EIA Strategic Plan states “…EIA’s information products will retain or improve their high quality…”  One of the performance measures and targets for this goal involves evaluating the EIA frames. The measure is the percent of EIA survey frames with sufficient industry coverage to produce reliable supply, demand and price statistics.

We began by preparing a list of EIA survey frames and update procedures.  We are now in the process of gathering existing information on the quality of our frames.  Activities to assess frame quality include:

1. Checking the frame against alternative lists at the respondent level (suggestion taken from the Statistical Policy Working Paper 15).

2.  Data comparisons at an aggregate level (certain EIA data compared with similar data outside of EIA).

3.  Examining supply/disposition balances (supply should equal disposition, but because the data that comprise the supply/disposition balances are from different surveys, a balancing item is needed.  If the balancing item becomes large, it could be an indication of a frame or other data quality problem).

This presentation provided examples of these activities and asked for the Committee’s guidance on other activities that could be used to assess our frames.  We were also interested in the Committee’s thoughts on how we can develop criteria to define sufficient coverage.

ASA Energy Committee Suggestions

1.  Try an adaptive sampling approach to develop lists for comparison.  One way to do this is to ask known establishments, plants and so on, to identify competitors and customers or suppliers others within the same market.

2.  Think of EIA frames as a sample from a larger population.  The missing units would correspond to a non-response problem.  Then calculate “propensity” scores by post-stratifying the sample using Census data.

3.  Try dual system estimation to obtain estimates of coverage.

4.  Balancing items and comparisons are not good measures of coverage because other problems such as reporting errors could also play a role in making these measures go out of sync.  Ideally, one needs independent lists to assess coverage.

5.  Census should provide as much information as possible to EIA, without disclosing sensitive information, so that EIA can identify the missing respondents.

EIA Response to Committee Suggestions

Respondents have been reluctant to give EIA the names of their customers; however, we have had success in obtaining names of suppliers or competitors.  Some of our electric power surveys, for example, have used this technique.    Therefore, EIA does not plan to pursue the adaptive sampling approach at the present time, but may consider it again in the future.

We are interested in pursuing propensity scores and dual system estimation.  In fact, we have tried to apply the latter to the EIA and REPIS renewable frames.

The comment about balancing items and comparisons led us to rethink our approach to assessing coverage.  Rather than try to develop a quantitative measure that could be applied to all surveys, we are going to be using a qualitative approach.  We are compiling the information we have on coverage for all of EIA surveys and will send it to an inter-office team, representing both survey staff and data users to evaluate.

Electricity 2005, Robert Schnapp, Director, Electric Power Division, Office of Coal, Nuclear, Electric and Alternate Fuels, EIA

“Electricity 2005” is a project that EIA’s Electric Power Division began in 2003 to revise the electric power surveys to accurately capture the changing industry.  This presentation focused on the four phases of the project; the issues and potential changes that are most important; and the relevance of EIA’s data confidentiality policy.  Special attention was given to transmission data needs to tie this presentation to the following one on focus groups held to elicit input on the transmission data that EIA should collect.

Electricity Transmission Data Needs Focus Group Results, Howard Bradsher-Fredrick and Phillip Tseng, SMG, EIA

The Energy Information Administration (EIA) is charged with providing timely and relevant data to stakeholders. In response to changes in the electricity industry in recent years, EIA has used its normal review process to update the EIA survey forms, collect information consistent with the market, and publish reports that meet stakeholders’ needs. A normal procedure is holding stakeholders meetings to receive inputs and suggestions.  This session was to report our findings at this ASA meeting.

Four focus groups were conducted from November 2003 through January 2004 on the topic of electricity transmission data needs.  The participants in each of these groups were fairly homogenous with the following groupings: EIA, DOE policy offices, other Federal organizations, and non-Federal organizations.  Each focus group had from 9 to 12 participants.  These sessions were audio taped for the purposes of writing the report.  The participants were asked questions from a structured protocol; these questions involved the following topics: characterizing EIA’s present transmission data collection, emerging issues in transmission, specific data needs, sources of data, and confidentiality issues.

The focus group participants stated that EIA should begin to provide more data regarding electricity transmission. Some of the data could be collected through existing forms.  Other data could possibly be collected through alternative sources and by editing and processing data collected by other agencies and organizations.

Summary of ASA Comments and Advice (including comments on the previous introduction by Bob Schnapp and the following Transmission Draft Summary by Doug Hale)   Comments by Howard Bradsher-Fredrick

I presented the paper, “Electricity Transmission Data Needs Focus Group Results,“ at the ASA Energy Committee Meeting on April 22, 2004.  Their advice really covered my presentation in addition to Robert Schnapp’s and Doug Hale’s presentations on related topics.  These three presentations covered the identification of the various facets of EIA’s development of a revised form to cover electricity transmission.  My paper covered the research conducted (focus groups and interviews) in identifying the data needs of EIA stakeholders, while Doug Hale’s paper was a technical discussion of all facets of electricity transmission.  Bob Schnapp’s paper essentially showed the development of new forms and revisions to old forms in the long and short-term.

ASA Advice

The advice given by the committee in essence stated that EIA should integrate these approaches more effectively in the future.  The committee did not believe that the stakeholder research was sufficiently reflected in the Hale paper nor was the entire research work sufficiently reflected in the Schnapp presentation and final plans and results.

EIA’s Response to the ASA Advice

EIA will use these observations and this advice to attempt to better integrate these components of the research and survey design process.  For example, the focus groups should be conducted much earlier in the process so that stakeholder needs and concerns are addressed throughout the development.  It should also be clearer how the research results were employed in the final plan for forms development and future work.

Transmission Data for Public Policy, Douglas R. Hale, EIA Senior Scientist on loan to Lawrence Berkeley Laboratory, DOE.  At the fall, 2003 meeting Doug Hale briefed the committee on a project to document the Federal Government's needs for transmission data (October 17, 2003). He has since written the report along the lines he discussed with the committee, and has also incorporated most of the committee's suggestions and those of EIA staff and Independent Expert Reviewers.  

The report is now in interagency review, and is available to the committee on the ASA Meeting Home Page. Doug will return from the Lawrence Berkeley Laboratory to join the meeting, answer questions and discuss suggestions and complaints the committee may have with the report.

Suggestions from the ASA Committee and Intended EIA Response

This briefing was to conclude the 3-part session on electricity, and to inform the Committee on the final draft product.  The Committee had commented on an earlier draft, which was constructive and helpful.  No further advice was sought. Please see previous “Advice” section for summary comments.

EIA publishes “Stocks of Crude Oil and Petroleum Products” weekly and monthly at

http://www.eia.gov/pub/oil_gas/petroleum/data_publications/weekly_petroleum_status_report/current/pdf/table03.pdf . All components of “Monthly Other Oils” stocks are collected from all respondents in a census.  In order to present more timely “Other Oils” stocks, the “Weekly Other Oils” stocks are estimated from the “Monthly Other Oils” stocks. Due to the seasonality and other unexpected disruptions, there are some discrepancies between these two series of stock number.

We want to determine the best way to estimate weekly stocks of other oils excluding propane (based on weekly data for major oil products, plus monthly data for major oil products and other oils products) and to evaluate alternative methods by comparing the estimate to monthly data.

Summary of ASA Advice

Committee Advice:

Intended Responses to ASA Advice

EIA Survey Testing Methods, Stanley Freedman & Robert Rutchik, SMG, EIA

The purpose of this breakout session is to describe, discuss, and seek ASA Committee Advice on EIA’s testing methods used in survey design.  EIA would like to get advice from the Committee on the suitability of these methods for energy establishment surveys.

EIA has used several methods in the past six years to develop and redesign surveys to meet changing industry structures.  These methods include:

• Pre-survey design visits
• Cognitive interviews
• Respondent debriefings, and
• Usability testing

Within these broad categories we have used combinations and permutations of these methods. 

The Committee was asked to discuss:

• The appropriateness of these methods including the pros and cons of each for specific design efforts; and
• What alternative methods are available to EIA that might to useful?

Summary of Committee’s Advice

The committee thought that the current methods that EIA was using to test surveys were adequate.  They recommended that we do as much testing as practical including both pre-survey design visits and site visits once a draft form was developed.  They also thought that testing on form revisions from the first set of test would also be useful.

They also recommended that EIA try to control for the effect of managers and staff being in the same interview at the same time.  They felt this might reflect on the candidness of the staff’s responses to EIA questions.

EIA’s Response

EIA can address the committee suggestions in several ways. 

First, when time and resources permit, EIA would always prefer to do several iterations of testing on draft survey forms.  If there is not time for new site visits, telephone calls, or emails to participants could be used to follow-up on changes.

Second, to the extent practical we can split the interviews into a manager group and a group with the actual respondents.  We will explore how feasible this is from the company’s point of view.  A more workable method may be to conduct the interviews with a larger group, and then have a follow-up interview with just the actual respondents.  This could be done as a post interview session for an additional ½ hour, or could be done with a telephone follow-up several days after the site interview. 

Natural Gas Production Monthly Survey, Inderjit Kundra.  The purpose of this survey is to provide monthly estimates of natural gas production at the National and regional levels.  The regions are Texas, Federal Gulf, Louisiana, New Mexico, Oklahoma, Wyoming and the others.  For selecting a sample, the 2002 EIA-23 Frame will be used.  The operators showing zero or blank production will be excluded from this frame.  Two separate small samples of the operators showing zero or blank production will be selected to validate their responses.  We are in the process of designing the sample.  By the time we meet we will be in a position to ask the Committee=s comments about what we have done to accomplish the survey.

Summary of Committee’s Advice and EIA’s Intended Response

We presented our proposal for a probability proportional to size sample, including a large certainty group.  The committee supported our efforts in designing the sample.

Friday, April 23, 2004

Improving EIA’s Website:  Creating a Vision for the Future  Colleen Blessing, National Energy Information             Center and Melinda Hobbs, Office of Information Technology, EIA. EIA is embarking on a new phase of web development.  This session will cover the process EIA has begun to gather information about what works and doesn’t work on the site and how we will use the feedback to develop a web strategy for the future.  We will highlight some recent improvements as a hint of things to come.  ASA members are encouraged to comment on the specific improvements or on anything related to EIA’s website.

Summary of Committee’s Advice

1.  Re the Natural Gas Navigator, Dr. Hengartner said it would be nice to have a little check box, saying I want this one and that one, then all the data would be downloaded into one file.

2.  They didn't like our search engine and wanted us to get something else.   

3.  Some wanted to be able to click on a graph and have the option to either download the data or see the time series.

4.  Pull together data from different places into a single place.

5.  Key in all the data we have available

6.  ASA members all wanted a way to cite a web page for an academic reference. Need some kind of archive version.  How to cite a data table?  How individual pages can be referenced?  We have a moving target and they want frozen files. 

We will pass Dr. Hengartner's suggestion along to the navigator team.  The Web committee definitely is looking into different search options and different ways to access our information.  This topic was discussed at our recent web workshop and is an issue on the current web audit.  About the "click on graph" option and the option to either download the data or see the time series, this is an option we already have and the members hadn't realized it.  On the suggestion to pull together data from different places into a single place, we think this is a good idea and are working in that direction.

On the suggestion to key all the data in that we have available we have data in the Annual Energy Review (AER) available on the Web going back to 1960 and in some cases 1949.  We will look into making users more aware of the AER as a source of historical data.

The topic of how to cite data from web pages is now being discussed within EIA.  Jerry Peabody mentioned configuration management approaches for archiving, but noted that this could involve lots of resources.  Susan Holte thought users should make a hardcopy of what they accessed.  Following a suggestion from the committee, we looked at how other agencies, like Census, do citations.  They cite the date the data were accessed.  The IIG plan to work on a style for citing Web pages but decided not to get into the archiving issue now.  Since we still present data in publication form, the group didn't think there were a lot of circumstances where we wouldn't be able to replicate the data.

Revising Data Together Across EIA:  Issues and Opportunities, Renee Miller and Alethea Jennings, SMG, EIA

We have spoken to the Committee in the past about how EIA has expanded its use of data collected in electric power surveys across the organization, such as natural gas, petroleum, and the integrated statistics publications.  As a result, revisions to electric power data affect more than just the electric power publications.  In addition, it is easier to revise data on the Web than in hard copy format.  These two events have raised questions about the need to coordinate revisions across EIA.

We presented background on EIA’s current revision standard (which the Committee helped us develop in the eighties) and recent events that led to its review.  Representatives from each affected office are working together to determine the best way to meet user needs and interoffice goals in support of EIA’s mission.  We will present conclusions from recent discussions/debates and will ask for the Committee’s guidance on coordinating revisions.

In addition to thinking about the coordination of revisions, EIA is examining the circumstances under which data are revised.  In our attempt to reduce the frequency of publishing revised data, we are examining the situations and events that prompt data revisions.  EIA may be revising data more often than is necessary to meet user needs.  Various situations cause data to revise, such as resubmissions, benchmarking (to the Annual publication), data corrections (where problems were discovered with the data originally reported) and late submissions.  We will present an example of data to illustrate revisions occurring over time.  We hope to obtain suggestions and ideas for reducing the number of revisions that we publish.

To satisfy users who are interested in having the latest data available, it has been suggested that EIA provide the latest versions of the data via the Website and inform users of this option.  We will discuss some of the advantages and disadvantages of this approach and ask for the Committee’s thoughts.

ASA Committee Advice

The committee wanted to see notes that indicate when an NA in a table is represented by zeros in the tabulated data and when the NA represents imputed data.

They did not view frequent revisions as a problem from a user standpoint because different users have different needs and are interested in the data for different reasons.   In particular, the committee thought that a table displaying a revision trail (similar to Table C1) should be available to provide additional information about a revised data series. 

The committee also expressed concerns about reproducibility.  They thought that the user should have access to or be able to reproduce data that were previously posted and currently updated. 

Regarding the latest vs. greatest, since this varies with the user, we were advised to balance the cost of revising the data with the value associated with the importance of providing the most up to date data to our users.

EIA’s Intended Response to ASA Advice

Regarding the use of “NA” as zero vs. imputed data, we recognize that this is a problem.  Given the new systems that are being developed, we expect that there will be opportunities for exploring ways to address this.  The issue of reproducibility was raised within the Interoffice Issues Group (IIG).  Several possibilities were suggested.  However, no consensus was reached on how or if we should do this.

Survey Quality Assessments at EIA, Tom Broene, SMG

Other statistical agencies and private firms have found it helpful to conduct assessments of on-going surveys on a regular basis.  EIA formed an inter-office team in 2003 to develop the procedure for an assessment and determine what data should be collected.  This talk will present the matrix used to collect the data and discuss its development. So far, only a handful of assessments have been completed, but we hope to have an assessment for all EIA surveys by this summer.

Committee Advice

The committee members participated in a frank discussion of assessing the quality of survey data.  The committee had several specific suggestions on conducting an assessment in terms of the data we are collecting, how we collect it, and how to disseminate and use the results.  They suggested that we directly ask whether the survey is capturing the appropriate data for both now and the foreseeable future. The committee suggested that we interview staff and managers separately.  It was also suggested that working on sharing lessons-learned would increase the positive tone of the process.  The committee also suggested that all employees, and especially contractors and new employees, be regularly informed of the impact and importance of EIA’s data.

EIA’s Intended Response to the ASA Advice

EIA plans to implement these suggestions. During our discussions with survey staff, we will ask whether the survey is capturing the appropriate data, and discuss the impact and importance of EIA’s data. We will interview staff and managers separately, and work on sharing lessons-learned to set a positive tone.

NOTE: 

Questions regarding EIA’s Spring 2004 meeting with the American Statistical Association Committee on Energy Statistics and this summary may be directed to Bill Weinig, EIA’s liaison with the ASA Committee by email at william.weinig@eia.doe.gov or by phone at the (202) 287-1709. 

The meeting agenda, papers, slides, and information on the committee members and EIA may be found on EIA’s Home Page at http://www.eia.gov/smg/asa_meeting_2004/spring/. 

An unedited transcript may be found on EIA’s Home Page under Energy Events, Summary of Advice from the American Statistical Association, at http://www.eia.gov/calendar/asa_overview.htm.