Summary of Advice from the

American Statistical Association (ASA)

Committee on Energy Statistics

Meeting with the

Energy Information Administration (EIA)

April 28 and 29, 2005

Washington, D.C.

Thursday, April 28, 2005

Regionalizing the Short-Term Energy Outlook (STEO) Forecast: Margot Anderson, Director, EMEU Plenary session discussed the key features of the Regional STEO model.  Topics included: (1) the rationale for regionalizing the model, (2) how EIA approached developing the regional model, (3) key components of the model; and (4) how the regional forecast will compare with the current STEO forecast.

Summary of Advice from the ASA Committee

The ASA Committee recommended that EIA be sure to fully articulate in the documentation the motivation and purpose for doing regional forecasts. 

Summary of EIA’s Intended Response to the ASA Committee Advice

EIA intends to follow the Committee advice in the documentation.

STEO Performance Indicators: Diagnostics and Forecast Errors, Margot Anderson, Director, EMEU  Plenary session presented and discussed: (1) diagnostic tools for gauging forecast accuracy, (2) which variables make sense and which tools are practical, (3) how important variables and tools are in gauging model and forecast accuracy, and (4) how EIA might share the results with customers.

Summary of Advice from the ASA Committee

The ASA Committee recommended that EIA be sure to fully articulate in the documentation the motivation and purpose for doing regional forecasts. 

Summary of EIA’s Intended Response to the ASA Committee Advice

EIA intends to follow the Committee advice in the documentation.

STEO Electricity Modeling: Data Analysis and Model Calibration, Phillip Tseng, SMG, Lead, Dave Costello, EMEU, EIA  

EIA completed the development of a thirteen-region electricity demand and supply model in early 2005. Subsequently, EIA modelers used the generic least-cost dispatching algorithm to simulate regional generation patterns and fuel choices in meeting a given set of regional electricity demand. Comparison of model results with 2002 and 2003 monthly generation data showed that cost captures only part of dispatching decisions facing electric utilities. And it became clear that modifications to the input assumptions are needed to improve the performance of the model.

This paper documents how EIA data are used to identify and estimate key model parameters that are important to the performance of the model.  In addition, this paper will also demonstrate that the calibrated model will be as robust and can provide insightful information on winter heating fuel market, natural gas market, and summer electricity market.

The data used for model calibration include:

Key model parameters EIA modelers have to estimate are:

Summary of Advice from the ASA Committee

The members of ASA Energy Committee commented on the modeling methodology and the trade offs between a simpler and more complicated approach.

One member preferred a simpler approach such as using historical trends and market shares to determine fossil fuel generation. Other members voiced support for a more informed, structured approach in modeling dispatching decisions. 

The approach selected depends on how EIA plans to use the model.  For fuel switching, a structured model can provide more insight and the results are more tractable.   The levels of effort required in model maintenance and data requirements were also mentioned as factors in the methodology selection.  Several members indicated that the data seemed massive but the concept of the model is very simple and is very easy to understand.    

Summary of EIA’s Intended Response to the ASA Committee Advice

    The EIA modeling team tried the econometric approach but could not find the appropriate functional form that can produce good projections of generations by fossil fuel type.   The current dispatching model presented at the ASA Energy Committee meeting performs very well for the three trade zones.  It tracks the peaks and valleys of the historical generation pattern very well for most of the regions.  We believe the model can be improved further as the modeling team gains more experience in using the data and making projections.

Bureau of Census Frames Comparisons, Rick Hough and Vicki Haitot, Bureau of Census

The purpose of the frames evaluation conducted by Census was to determine the coverage of the following five EIA frames: (1) EIA-3, Quarterly Coal Consumption and Quality Report, Manufacturing Plants, (2) EIA-5, Quarterly Coal Consumption and Quality Report, Coke Plants; (3) EIA-860, "Annual Electric Generator Report" (for combined heat and power plants in NAICS codes 31-33 only; (4) EIA-63a, "Annual Solar Thermal Collector Manufacturers Survey"; and (5) EIA-63b, "Annual Photovoltaic Module and Cell Manufacturers Survey.

Two representatives from the Bureau of Census presented the following topics: (1) Methodology for matching and analysis of coverage (both by count and by volume) for all five EIA frames; (2) Results of matching and analysis for EIA-63a (solar thermal collectors), EIA-63b (photovoltaic cells and modules), and EIA-5 (coke plants); (3) Identifying differences between EIA and Census frames which impact coverage analysis; (4) Identifying characteristics of establishments missing on EIA frames; and (5) Identifying next steps for assessing coverage of EIA-3(Manufacturers that consume coal) and EIA-860 (Combined Heat and Power Plants).

Summary of Advice from the ASA Committee

The Committee expressed interest in the results from the final two assessments being conducted.  Committee members further suggested that the frame assessments may not be necessary every five years for surveys with small sample sizes. The likelihood of a big dramatic change is probably small for these surveys. Finally, committee members suggested using the first stage of the matching process to assess weather the more detailed second stage is necessary.

Summary of EIA’s Intended Response to the ASA Committee Advice

Results from the two final assessments are expected in August, 2005, and will be made available to EIA.  Census will evaluate the need for the frame assessments for surveys every five years for surveys with small sample sizes when the 2007 Economic Census data are available.  Census did conduct a feasibility assessment where they looked at the number of cases that could be identified in their database. In the future, this may not be necessary, and Census and EIA may discuss investigating using the first stage as the ASA committee suggested.        

Regional STE Model, Propane and Heating Oil Modules, Tancred Lidderdale, EMEU, presented the new regional short-term forecasting modules for propane and residential heating oil. The propane and heating oil modules are structurally similar with reduced-form equations for the U.S. wholesale price, census district residential prices and demands, and PAD District total stocks.  Tancred reviewed the structure of the models, some of the key estimation results, and forecast results for selected variables.

The objective of the regional propane and heating oil modules is to generate short-term monthly forecasts of residential retail prices in the four Census regions.  The modules are structurally identical reduced-form models. Regression equations for wholesale prices, regional demands, and regional inventory change are included to support to estimation of residential retail prices.

Regional demands are used to weight regional prices in the calculation of average monthly U.S. prices and average quarterly and annual regional prices. The discussion in the ASA review focused on two problems with the residential demand data and forecasting equations: (1) the reliability of the regional demand data series, and (2) the reliability of housing data used to predict regional demands

The first problem in forecasting regional demands is that EIA publishes only annual residential consumption volumes and does not publish monthly residential consumption data. Regional demand numbers for the model are calculated from unpublished EIA-782C survey volumes.  However, these survey volumes are only used for creating State and regional average prices for the Petroleum Marketing Monthly and do not represent reliable measures of actual consumption volumes. Moreover, rebasing of the survey every few years produces shifts in the data series.

The second problem in forecasting regional demands is that the numbers of houses in each region that use each fuel as the primary space heating fuel reported in the EIA Residential Energy Consumption Survey (RECS) have large margins of sample error. For example, the 95 percent confidence interval for the number of homes that use propane for space heating in the northeast is four hundred thousand plus or minus 50 percent. Moreover, the RECS survey is conducted only every four years.

Summary of Advice from the ASA Committee

Discussion at the ASA session focused our attention that the objective of the model was to forecast prices, not demands. Dr. Hengartner suggested that normalizing the numbers such as with a ratio estimate may eliminate some of the artifacts.

Summary of EIA’s Intended Response to the ASA Committee Advice

Following the meeting the regional models were revised by changing the demand series to demand shares. The model documentation now discusses how the calculated regional demand shares are used to weight prices and that while they may produce unreliable regional demand numbers (when multiplied by annual consumption data) they nevertheless yield more reliable price averages.  For example, consider two States, one with a price of $1.10 and the other with a price of $1.00.  Assume the actual demands in the two States are identical but are each measured with 20 percent error in opposite directions: State A’s measured demand share is 0.6 and State B’s is 0.4, rather than 0.5 each.  While the error in the demands is large, the error in the volume-weighted price average is small: $1.04 when demands are measured with error and $1.05 when demands are correct.

Hands-On Usability Testing of EIA's New Website Design:   Can You Find What You're Looking For? Colleen Blessing, National Energy Information Center, EIA  The last EIA website was redesigned in 2000.  We knew from extensive customer feedback that finding information on our site in many cases was not easy or intuitive.  We wanted our customers to be able to find what they needed and understand what they found.

Working since last fall, and based on input from many sources, the Web Redesign Group created a new prototype design we wanted ASA Committee members to "test drive."  This session was designed for Committee members to sit at a computer, use the new site, comment as they clicked through, giving us feedback on what they saw, and finding answers to questions or finding what they usually use on the current site. 

EIA conducted a similar usability test with ASA members before our redesign in 2000 and received useful feedback.  There was no preparation necessary for this session, except if Committee members wanted to record some reports or data tables that they would like to be able to find on the new site.  Looking at the new design for the first time in this session allowed EIA to capture Committee members’ first impressions.

Summary of Advice from the ASA Committee

The ASA committee members were asked to evaluate the effectiveness and understandability of a proposed new web page design by EIA's Cognitive Evaluation and Usability team.  Eleven ASA Committee members were tested in two separate sessions.   They were asked to use the new web page to find answers to commonly asked questions and then give comments and feedback about their experiences working with the site. 

Some specific areas that tested well were the new link names on the home page, hints under the major bin names, information in the new right side bar and the second-level pages. 

Some specific areas that did not test well were finding information inside specific publications, finding the glossary, the need for integrated presentation of information (having both history and forecasts links available from the same second-level page) and the link name Energy Overview. 

EIA collected much useful information about the usability and functionality of the new design.  The overall conclusion from the ASA committee members is that they liked the new design and thought we were on the right track. 

Summary of EIA's Intended Response to the ASA Committee Advice

Following the ASA tests, EIA conducted usability tests with 25 additional users, both internal EIA staff and external users from various customer groups.  Feedback from all these tests is being used to modify and improve the proposed new design.  Watching users actually work on the site gives direct messages about what they see, what they understand, and what they use.  Items that tested well will be kept; items that don't test well will be dropped or changed. 

Follow-up on Frames Team Activities, Howard Bradsher-Fredrick, SMG, EIA This session presented a summary of the activities and conclusions of the various incarnations of the frames team from March 2004 through April 2005. The frames team was an inter-office EIA team composed of staff members from the EIA program offices, SMG and OIT.  In general, all interested and affected offices had representation on each incarnation of the team

The impetus for the effort on improving frames emanated from the EIA Strategic Plan and specifically from the Strategic Information Technology Subgroup.  These frames teams were charged with identifying efficiencies EIA can pursue regarding frames and the computer systems that support frames.

In particular, the first team was chartered with providing recommendations concerning activities in the frames area that will lead EIA toward saving resources, facilitating more commonality in data and software, improving processes, and resulting in frames and frame information that can easily be shared within EIA.  The first frames team met from March 2004 to early May 2004 and developed a set of six recommendations.

The second incarnation of the team met from August 2004 to January 2005.  The primary focus of its activities was to assess each unique EIA survey frame for “sufficiency.”  In order for the team to ascertain sufficiency or insufficiency for individual frames, a number of tasks had to be accomplished, some of which involved extensive survey frame information gathering. 

The third incarnation of the team began meeting in February 2005.  It is charged with following up on the recommendations put forward by the first incarnation of the team.

Summary of Advice from the ASA Committee

The ASA Committee was not enthusiastic about EIA evaluating all of its frames for sufficiency on a frequent periodic basis.  The committee expressed the opinion that it would be a better use of resources to scrutinize and offer remedial recommendations on survey frames identified as being more likely to be problematical.  EIA is in the best position to exercise its own judgment concerning which frames are most likely to be in need of evaluation.   

The committee also questioned whether or not it was a worthwhile exercise to evaluate frames as often as once every three years.  It may be a more useful utilization of EIA’s resources to evaluate frames on a less frequent basis. The committee expressed no opinion on the question posed by the presenter, that of whether each survey frame should be evaluated as opposed to each master frame.

Summary of EIA’s Intended Response to the ASA Committee Advice

EIA appears to be caught between the advice given by the committee and the performance measurement evaluations required by the Department.  The committee does not believe that it is useful for EIA to evaluate all frames for sufficiency nor does it believe that sufficiency evaluations should be taken on as often as every three years.  This seems to be in direct conflict with the Department’s request to evaluate all frames for sufficiency on a regular periodic basis.  EIA will consider the advice given by the committee prior to comprehensively evaluating frames in the future.  This consideration will include both the issue of frequency of evaluation and whether all frames should be evaluated or only those that are most likely to be problematical (e.g., those in the greatest state of flux, those that are relatively new).

EIA’s Proposed Strategy for Addressing Declining Response Rates in the Residential Energy Consumption Survey (RECS), Dwight French, EMEU, EIA   Over the 25-year history of EIA’s Residential Energy Consumption Survey (RECS), response at various stages of this survey system has slowly but steadily decreased.  This trend, which has been true for most government household surveys, is especially troublesome for RECS, which is actually a sequence of interconnected data collections:  a survey of the household to get its energy-related characteristics, a request during the household survey to get the household respondent to sign a waiver form allowing the household’s energy suppliers to submit billing information, and then a survey of the suppliers to collect the energy consumption and expenditures information.  All of these collections are necessary to get a complete set of survey data; therefore, non-response at each stage has a cumulative negative effect on data quality. 

This presentation described an aggressive, multi-faceted strategy that EIA planned to address the comprehensive non-response issue, including additional and updated preliminary contacts, an incentive experiment, enhanced non-monetary incentives, the use of scanners to collect utility bill information at the household, a Spanish translation of the questionnaire, publicity about data confidentiality based on EIA’s new CIPSEA legislation, and a potential non-response sub-sampling process.   The committee was asked to critique the approaches and suggest any others they might think would be useful.

Summary of Advice from and Intended Response to the ASA Committee

The ASA Committee members offered advice in the following five areas:

1.  Advice:  A question was asked about the justification for our proposed incentive levels.  What did we do to determine that these were appropriate levels, in that they would be sufficient to change householders' minds about responding?  Could we ask a question of non-respondents to the survey as to what level of incentive WOULD be sufficient to make them respond? 

Response:  EIA did not do any official research study to determine appropriate incentive levels.  We were constrained by our financial resources, the fact that OMB wanted an experiment to test different incentive levels, and the concern that incentives that were too large might turn people off as an example of inappropriate Federal largesse.  EIA did review the literature on other incentive experiments and payments.  Our proposed incentives fit into the broad range of what has been done by others, but we have nothing that indicates that our incentives can be expected to be optimum in some sense.  And even if one of the incentive levels were to turn out to be "optimum", that would mean that the other levels aren't.  EIA will, of course, study the effects of the incentives after the RECS interviewing is complete.  But EIA will not ask the question about how much it would take for a non-respondent to be persuaded to respond.  The government does not reimburse people to provide information - we will be very careful to tell respondents that the incentive is a token of thanks, not a payment for information (which is why the incentive is unconditional).  Asking a non-respondent how much it would take for them to respond is tantamount to suggesting that the incentive is a reimbursement.

2.  Advice:  The committee suggested that if EIA were doing an incentive experiment, the selection of categories should NOT be associated with household income, because one doesn't want to have an income effect confounded with the incentive category effect. 

Response:  EIA's response is that the incentive amounts will be randomized across segments, but each household in a segment will get the same incentive (avoiding the jealous neighbor effect).  We do not plan to have household income as any part of the design for assigning incentive categories to segments.

3.  Advice:  A comment was brought up about using priority mail or some other attention-catching mailing for advance information, to emphasize the importance of the survey.  Reaction to this was mixed - some thought that priority mail has been over utilized, and used for junk-type mailings in some cases, so that its effect has been diluted.  But there has been some evidence that attention-getting mail can improve response at least slightly.

Response:  EIA's response is that only advance documents will be dealt with by mail, so it isn't as if we're doing a mail survey.  However, we will use the best mailing that budgets allow.

4.  Advice:  A suggestion was made that if we try to get energy data from a household, we get it for only a month rather than the entire reference year.

Response:  EIA's response is that we intend to ask the households, in the advance letter, to have their most recent energy bills available for the interviewer.  We will scan those.  However, we may be able to get a year's worth of consumption data from some of these bills, because certain utilities are now posting the previous 12 months' consumption on the bill to give the household a more complete picture of its energy needs.  If we can get a year's worth of consumption data from many of these bills, it will be an enormous help in cases where the supplier cannot be reached, refuses to cooperate, or cannot provide the information.

5.  Advice:  In response to an EIA question about how to compute and quote the survey response rate if non-response sub-sampling were employed, there seemed to be strong feeling from the committee members that it was misleading and statistically improper to weight up the non-response cases to compute the survey response rate rather than simply taking the un-weighted final number of respondents/the number of in-scope sample cases to be the response rate.  It is okay to use the non-response sub-sampling weights to compute parameter and variance estimates.  In fact, you'd really need to do so to avoid bias, unless the initial non-respondents prior to sub-sampling are found to be no different from the original respondents when the eventual responses are compared to the responses from the original respondents.  But don't use it to compute a weighted response rate.

Response:  EIA's response is that we might compute both response rates -- weighted and un-weighted, but that we would not avoid citing the un-weighted response rate.  It should be noted that non-response sub-sampling is NOT a given for the 2005 RECS.  The process will be used ONLY if the initial response is sufficiently low that sub-sampling is necessary to improve estimation and keep control of non-response conversion costs.  EIA's and its fieldwork contractor (NORC's) plan is that the survey process will be successful enough (achieving a minimum of 80% response within the budget for fieldwork) that non-response sub-sampling will not be needed.

Friday, April 29, 2005

Assessments: Presentations and a Panel Discussion (Two Parts), Nancy Kirkendall, moderator

Part I.  External Review of Survey Programs: A Progress Report, Brenda Cox, Battelle, under contract to EIA, and Nancy Kirkendall, EIA  The Performance Assessment Rating Tool (PART) was instituted to boost the quality of Federal programs by encouraging rigorous performance assessment for government programs.  These performance assessments identify program strengths and weaknesses, resulting in recommendations for improvement and recognition of desirable design elements already in place.  For programs that involve a family of related surveys, the individual surveys that comprise the survey program need to be evaluated as well as the effectiveness of their integration to create the survey program.  The session and paper presented  the progress made in this investigation designed to develop templates for performing an external evaluation of a family of related surveys.  The end products of this study will be a Program Evaluation Template and a Survey Evaluation Template, which can serve as guides for performing expert assessments of survey programs and their component surveys.  At the Fall 2004 meeting of the ASA Committee on Energy Statistics, we described the project’s goals and presented the survey template that was to be concept tested later that year.  This presentation updates that research by presenting the Program Evaluation Template and the insights gained from our proof of concept testing of both templates on the Petroleum Marketing Survey Program.

Summary of Advice from the ASA Committee

The Committee was concerned that EIA’s program of activities should involve more than programs of surveys.  There are also extensive modeling and data analyses activities occurring that also need to be considered.  The Committee was interested in the total number of programs that EIA might have.  Finally, the Committee was concerned about the exclusive use of publicly available documentation for the survey and program evaluations, with no staff interactions. 

Summary of EIA’s Intended Response to the ASA Committee Advice

For this trial evaluation, we have meet with Petroleum Marketing Program staff and gotten their reactions to the draft evaluations.  In the Methods Report documenting the results of this investigation, we will recommend staff interaction in eliciting input for the evaluations and for reacting to the draft evaluations. 

Nancy, a note from Joseph:  That’s all that was pertinent to my paper.  Nancy might like to add comments on evaluating programs that do not fit under the umbrella of “a program of surveys” addressed by the templates I am preparing.

Part II.  External Review of EIA Program, John Paul Deley, NEIC, EIA representing the EIA External Study Team: Colleen Blessing, Kathleen Cavanaugh, Doug Hale, Janice Poling, Erik Rasmussen and Barry Yaffe. At the last meeting the ASA Committee on Energy Statistics recommended that EIA consider addressing the Office of Management and Budget requirement for a program-wide independent evaluation of EIA via a “High-Powered Review Team”, and to use a model akin to an academic evaluation.  In response, EIA has decided to charter an external study team, consisting of well-known energy experts, and to ask it to comment on whether EIA is “doing the right things.”  EIA senior managers will use the results to help determine how EIA should focus its resources in the next five years to provide relevant energy information to policymakers, the marketplace, and to the public.  The study is titled, “Challenges, Choices and Changes for the US Energy Information Program.”

The overarching question that the external study team will address is:

Given continuing tight EIA budgets and other external drivers, is EIA doing the “right things” now, and what are the “right things” it should do in the next five years?

The question intentionally focuses on “is EIA doing the right things” (i.e., what products and services are provided to customers) and not on “is EIA doing things right” (i.e., how it conducts its internal business processes).  At the ASA session, EIA reviewed its goals for this external study, outlined project’s tentative timeline (scheduled to be completed in November 2005), and solicited ideas and suggestions from ASA Energy Committee members who may wish to have shared their experience with evaluative processes that resulted in positive change. 

Summary of Advice from the ASA Committee  

Individual members of the Committee suggested that: (1) EIA perform a “self study,” answering the study questions before posing them to the external tests,  (2) that the current outline was a “core dump” and that it would not help the external team understand everything about EIA, (3) that it would be difficult to completely isolate the question, “Are we doing the right things?” from the issue of “how” we do things, (4) that the relationship between EIA staff and contractors was often very cozy, that the contractors were not always executing their work up to current industry standards in econometric and statistical methods and suggested the review include whether we are using the best methods for the work that we do , (5) that past recommendations by similar studies were not accepted or acted upon; (6) that it is not possible for a high-level review to work without delving into the details; (7) that the balance of the Study Team (mix of industry, energy sector, and data/forecasting/analysis interests) was not equitable, and (8) that the issue of tradeoffs be more explicit.

Summary of EIA’s Intended Response to the ASA Committee Advice

The EIA External Study Support Team (ESST) received, analyzed and responded to all recommendations received by the ASA committee by either: 1) incorporating the suggestion directly into its planning process, or 2) referring the matter to EIA Senior Staff. 

EIA’s response will include (but not be limited to): (1) an EIA support team discussion of options for conducting an internal EIA self-study; (2) EIA support team consideration of revamping the Overview document; (3) EIA support team recognition of the difficulty in totally separating the right thing from doing things right; (4) EIA support team will recommend balance of study team (mix of industry, energy sector, interests, geography, etc); (5) EIA support team review of statements from the OMB PART report that EIA did not do well on, in particular, a)   “Regular independent evaluations used to support program improvement, assess effectiveness and relevance”  and b)  “Independent evaluations indicate program effective and achieving results”; (6) EIA support team consideration of recommendations to support program improvement, and to assess relevance; (7) EIA support team inclusion of words about “effectiveness, and achieving results” in the Overview document; (8) EIA support team will better enunciate that along with “Outputs” the Study team will be asked to think about “Outcomes”.  Outputs alone are too narrow; (9) EIA support team consideration of the relationship between EIA staff and contractors; (10) EIA support team recommendation to Senior staff that a focus on HOW we do business be a focus of a future review; (11) EIA support team will ask the external study team to establish priorities for their recommendations; and, (12) EIA support team recognizes that comments on the inadequacies of the EIA website may be reflected in this study.

Discussion:  Where Next?, Howard Gruenspecht, EIA Deputy Administrator and Moderator

Progress on EIA’s 914: Response rate and kinds of challenges, John Wood, Reserves and Production Division (RPD), EIA  At the fall 2004 meeting with the ASA Energy Committee, EIA introduced the new and approved Natural Gas Production Survey, Form EIA-914. The spring 2005 meeting will provide the first opportunity to discuss EIA’s preliminary assessment of the survey data for January, 2005, the first reporting month for this new survey. EIA will discuss the challenges faced during the early months of survey operation, as well as our plans for review, analysis, and comparisons of final natural gas production estimates based on the early sample of final production provided by the EIA-914 survey data with our previous methods of estimating final natural gas production. This review, comparison, and analysis is a critical step in the process of  deciding when EIA should start publicly releasing the EIA based production estimates as our official EIA natural gas production estimates.

Summary of Advice from the ASA Committee and EIA’s Intended Response

The first major point made in the ASA discussant summary was that the EIA-914 survey was a better way to provide natural gas production estimates; and the response rates, through a lot of effort and contact and follow-ups with the producers themselves, seemed to be quite impressive and the survey results look very good at this time. In response, EIA is proceeding to move the EIA- 914 estimates through the clearance process, to make the estimates public, and then make them the official EIA natural gas production numbers.

The ASA thought that the high response rates would inspire public confidence in the EIA-914-based natural gas production estimates and that EIA should describe for the public the follow-up we performed with operators to achieve the high response rates. In response, EIA will strive to maintain high response rates in the future, publicize the high response rates when the data are publicly released and prominently emphasize to the public the quality assurance and follow up with operators that seemed reassuring to the ASA Committee.

Capitalizing on the operator level data now available, a formal imputation system that estimates monthly production for non-responding operators based on previously submitted data has been put in place. We expect it will get more reliable over time as a longer reported data series is collected.

Estimate EIA’s 826 Since the Last Time: Differences, Estimation Groups, Outliers and Test Results,  Joseph Sedransk, Jim Knaub, Lindolfo Pedraza and Nancy Kirkendall, EIA

EIA Form-826 collects information, monthly, from regulated and unregulated companies that sell or deliver electric power to end users. It collects State-level sales volumes, sales revenues, and number of customers by end-use sector (residential, commercial, industrial, and total). The existing sample and methodology to estimate population totals were described at the fall 2004 meeting of the Energy Committee.  This session will describe current efforts to form homogeneous subpopulations for estimation, and evaluations based on predictions using annual data (where truth is assumed known).  We assess the importance of outlier detection, under alternative scenarios.  We will be asking for advice on next steps for analysis, and on how to frame convincing recommendations for implementation.

Summary of Advice from the ASA Committee

The ASA committee recommended that two of the summary statistics that were presented were the more useful ones; i.e., (1) the coefficient of variation, and (2) one of the per cent error quantities (per cent error in estimating the non-sampled companies in the stratum or per cent error in estimating the stratum total).

A second recommendation was to evaluate the alternative choices (e.g., outlier detection or not, use of IOUs in estimation or not) using monthly data rather than the annual data used in constructing the tables shown at the meeting.

There were no recommendations concerning alternative methods of forming the post-strata for estimation.

Summary of EIA’s Intended Response to the ASA Committee Advice

EIA will emphasize the two summary statistics noted above. An evaluation using monthly data has been planned, but has not yet been implemented.