ABSTRACTS
for the Fall Meeting of the
American Statistical Association (ASA)
Committee on Energy Statistics
October 28 and 29, 2004
with the
Energy Information Administration
1000 Independence Ave., SW.
Washington, D.C. 20585
Thursday, October
28, 2004
Quantifying the Impact of
Changes in Forms Bob Rutchik, SMG, EIA
The Energy Information Administration has spent considerable time and resources in designing a form to capture generation, fuel use, and stocks information from Combined Heat and Power facilities. The Form, the EIA-920, had its first monthly collection in January 2004. The purpose of the project is to evaluate the effectiveness of this new form.
More specifically, EIA is assessing whether EIA’s testing and design have resulted in an improved survey (over its predecessor form, the EIA-906). EIA has proposed these four criteria to measure that effectiveness.
1. Timeliness of response
2. Ability to provide data
3. Accuracy of response
4. Reduction in processing time for EIA
In addition to assessing whether the design and testing has resulted in an improved survey, EIA proposes to use this assessment as a template for analysis of the effectiveness of other new and redesigned EIA surveys. In short, EIA wants to include survey effectiveness as a fourth method in its program of (1) pre-survey design visits, (2) cognitive interviews of draft instruments, and (3) respondent debriefings on draft survey instruments. EIA will then have a complete program of survey design, testing, and evaluation.
EIA wants the Committee’s advice to see if:
· The criteria that EIA have proposed are valid
· There are other criterion that EIA could use
· What measures can be used to operationalize the criteria
· Can the criteria, collectively, be used as a model for future similar analysis
A Customer
Evaluation of the Short-Term Energy Outlook (STEO) Howard
Bradsher-Fredrick, SMG, EIA
The authors developed and administered a short survey questionnaire to query customers of the Short Term Energy Outlook (STEO). The purpose of this survey was to determine how the STEO was perceived by its regular customers. The results of this survey are to be used for strategic planning purposes (performance evaluation) and to aid in the improvement of the STEO. The survey questions were based as closely as possible upon the instrument employed in the customer evaluation of the Annual Energy Outlook (AEO) and the International Energy Outlook (IEO) during the spring of 2004. A total of 500 customers who are not employed by EIA were randomly selected from the Listserv maintained for regular STEO mailings as potential respondents. In order to ensure a reasonably high response rate, three iterations of non-response follow-up were administered.
The Office of Information Technology supported this effort through the development of a web-based questionnaire devised from the original e-mail based survey employed in the Spring of 2004 for the AEO and IEO.
Involved SMG staff were Bill Weinig, Howard Bradsher-Fredrick, Stan Freedman, Renee Miller, Phillip Tseng, Tom Broene, Inderjit Kundra, Herb Miller and Joseph Sedransk.
Assessing EIA
Frames: An update on EIA-Wide Grace Sutherland, Howard Bradsher-Fredrick and Shawna
Waugh, SMG, and Tom Lorenz, EIA’s Office
of Energy Markets and End Use
In the Spring, 2004, EIA presented to the committee the activities to try to evaluate its frames. The activities included checking respondent lists, comparing data at an aggregate level, examining supply/disposition balances, and comparing price data volumes. The committee made the following recommendations in response to these activities: 1) Ask known establishments to identify others within the same market, 2) calculate “propensity” scores by post-stratifying the sample using Census data 3) Apply the principles of dual system estimation to available frame data, 4) Obtain as much information from Census, without disclosing sensitive data in order to identify missing respondents, and 5) do not use “balancing item” for measuring coverage. The consensus of the committee regarding balancing items was that this was the least favorable method of assessing frames.
EIA’s response to the committee’s suggestions was:
1) EIA has asked respondents to provide names of their customers, but respondents have been reluctant. However, we have had success in obtaining names of suppliers or competitors. Some of our electric power surveys, for example, have used this technique. EIA does not plan to pursue the adaptive sampling approach at the present time, but may consider it again in the future. 2) EIA is in favor of pursuing dual system estimation and has done this with EIA electricity renewable frame and the National Renewable Energy Laboratory frames. 3) Because of the difficulty in coming up with a quantitative assessment of frame sufficiency for all surveys, EIA will pursue a qualitative approach. EIA has formed a new inter-office team to evaluate frame sufficiency. The team is using the information gathered previously on its surveys and will look at frame stability over a longer period of time, than what we currently have (the past year and ongoing). The team will be researching whether or not surveys are using comparable lists, and if they aren’t being used, what are the reasons, i.e. budget constraints, legal issues, etc. You will hear the results of this effort at the Spring 2005 meeting of the ASA Energy Committee.
Howard Bradsher-Fredrick of EIA will summarize the dual system estimation study (EIA electricity renewable frame and NREL frames.) Shawna Waugh of EIA will describe the collaboration between the Census Bureau and EIA regarding the evaluating of several of EIA frames in the manufacturing sector. Tom Lorenz of EIA will describe what he has learned by using data from EIA’s petroleum surveys (Monthly and Annual) to edit the Manufacturing Energy Consumption Survey (MECS) data.
EIA is interested in the committee’s opinion as to the direction EIA is going with the frames activities.
The EIA Short-Term Regional Electricity Model:
Capabilities and Data Requirements Phillip Tseng, SMG, and Dave Costello, EMEU, EIA
The United States Energy Information Administration (EIA) is developing a thirteen-region electricity demand and supply model in response to questions on regional energy issues from high-level decision makers. One of the important features of the new modeling system is transparency; it must provide tractable results and insights that stakeholders can easily understand. The electricity module of the Regional Short Term Energy Model serves that function and provides a vital link to the new integrated regional energy system in answering several key region specific questions on:
· Winter heating fuel market
· Natural gas market
· Summer gasoline market
· Summer electricity market
The objective of this paper is to document the structure of the electricity model, identify data requirements, and demonstrate the model’s capability in providing users an understanding of issues facing the current electricity market. The model will be used for generating routine short-term forecasts. However, the model can also be used as an analytical tool to provide useful insights into the electric power market itself and into the principal interactions between electricity supply and fossil fuel markets. The interactions and linkages between the electricity market, the natural gas market, and heating fuel market will be described explicitly. In addition, this paper will illustrate potential modeling applications that can help policy makers in their decisions on deployment of technologies that may be cost effective socially but not privately.
Natural Gas Production, Frames, Samples and Estimation Session in two parts: (1) Preston McDowney, SMG, and (2) John Woods, OOG, EIA
At our Spring 2004 meeting, you may
recall that you heard about EIA’s new Natural Gas Production survey, the
EIA-914. At that time, Inderjit Kundra
described our proposal to select a probability proportional to size (pps) sample, using the EIA-23 frame as the frame for the
new EIA-914. The Committee supported our
efforts to design the sample. This
session provides an update to that effort.
The Statistics and Methods Group (SMG) prepared a matched
file of respondents to the EIA-23 in 2000 and in 2002, and used them as the
basis as an assessment of the sampling and estimation methodology. We conducted a simulation using these data
sets to select a pps sample according to the
methodology we described in the spring.
We prepared a variety of estimates including the traditional estimate
using sampling weights, and several regression-based estimators (weighted (
) and unweighted (
), with and without deleting outliers and influential
observations.) The results of this
assessment made us reconsider our sampling strategy. We are now planning to use a cut-off sample
with a goal of approximately 90% coverage.
The Reserves and Production Division (RPD) of the Office of
Oil and Gas has compiled a long history of operator-level data, they have
evaluated a variety of estimation methods, and evaluated the changes that occur
in the dynamic natural gas production industry.
This session will provide two presentations: first, a
summary of the results of SMG’s simulation study by Preston McDowney and Kara
Norman, and second, the information on the dynamic natural gas production
industry, a description of the sample, and evaluations of estimation methods by
John Wood and Gary Long of RPD, OOG.
Methods for Assessing NEMS
Solution Data for Interpretive and Diagnostic Purposes George M. Lady, Ph.D., EIA Contractor
Background: The National Energy Modeling System
(NEMS) produces forecasts of energy prices, quantities and related variables by
year through the year 2025. Each forecast generates thousands of numbers
arranged in 229 distinct tabular displays. The size of the output only hints at
the complexity of the underlying models that together produce an integrated
(internally consistent) forecast.
Model users and clients have long been interested in having
relatively simple ways of sorting through individual forecasts, comparing
individual forecasts and identifying assumptions that have particularly large
effects on the results. The basic idea of this project is to treat NEMS
forecasts as if they were “real” data and present the “data” graphically. That
allows fairly efficient visual analysis of a host of individual “data” series.
In addition, it is easy to make graphical comparisons of forecasts across
various assumptions and model versions. Finally, we modeled the data with
simple regression equations. The regressions link model outputs (forecasts) to
the assumed data (such as GDP and world oil prices) that the model uses to make
its forecasts.
Representative applications of this software technology
include:
Comparisons of NEMS Solutions. The NEMS solution series
used for the regression analyses were compared directly for the years
2010-2020. The issue at stake was the apparent stability of the forecasts. The
following measurements were made:
-
for each forecast year the ratio of the standard
deviation across the alternative forecasts to the mean forecast value;
-
for each forecast series, the number of times there were changes in rank across
NEMS versions, i.e., the number of times time series plots of the solutions
would cross; and,
-
for the forecast year 2010,
whether the difference between the 1998 and 1999 projections was larger (as
would be expected) or smaller than the difference between the 2003 and 2004
projections.
NEMS Approximation and Interpretation. NEMS
solution series for the production and consumption of major fuels, as well as
important influences upon energy supply and demand, have been extracted and
analyzed for the 1998-2004 AEO
versions of NEMS. A linear model of energy supply and demand was fit to these
data which provided generally very accurate approximations of the NEMS
projections. The model is based upon over fifty regression equations that can
be executed as a group automatically. The sensitivities among NEMS variables
are measured by elasticities. The elasticities
are compared across NEMS versions to asses the stability and robustness of
NEMS’ representation of the U.S. energy system as reflected by the changes in size, and sometimes sign, of the elasticities. Accurate approximations of NEMS solutions have
also been constructed using kernel regression.
Backcasts.
A comparison of the NEMS projections to actual data for the year 2003 is
scheduled. The regression results will be used to partition differences between
those due to forecasting uncertainty in general versus those due to errors in
the projection of exogenous variables.
Issues. The NEMS solution processing results
referred to above are available for inspection. The measures chosen are
intended to be illustrative, rather than definitive At
issue is the usefulness of such measures and the specification of other
measures for the purpose of revealing and auditing the features of NEMS’
representation of energy markets and enabling diagnostic assessments of NEMS
forecasting methods.
Questions for the
Committee:
- What
measures, based upon the approximations of stable NEMS solutions, would assist
in the assessment of
NEMS solutions under development to help uncover programming
errors or inappropriate assumptions?
- What
procedures would be appropriate for using NEMS approximations to facilitate
convergence of the NEMS solution algorithm?
- What
diagnostics based upon regression-based approximations would be useful for interpreting NEMS
and supporting priorities for further model development?
- What
would be good ways to utilize prior NEMS solutions to audit and test current
solutions?
- What
would be good ways to utilize regression-based NEMS approximations to establish
the partition of forecasting errors among (such as) errors in forecasting
exogenous variables versus general forecasting uncertainty?
-
In constructing the regression equations how should
“goodness of fit” be traded off with the plausibility of specification in terms
of representing energy supply and demand relationships?
Introduction to Program Assessment Rating Tool (PART)
Program Evaluation Nancy Kirkendall, SMG, EIA
Abstract outstanding. (Expect background paper from T. Broene)
External Evaluations of Survey Programs, Brenda Cox, Contractor to
SMG, EIA
Survey Evaluation
is a broad construct, and can have both internal and external aspects.
Over the past
several meetings EIA has presented the Committee with information about our internal
evaluation programs.
This survey
evaluation project involves External Evaluations of families of EIA
surveys. Brenda Cox will present her
initial work on a template for an external survey evaluation. Dr. Cox is working under contract to SMG to
help develop templates for an external evaluation of a family of related
surveys and its component surveys. The
intention is for this work to feed directly into the annual OMB Performance
Assessment Rating Tool program, which was instituted to encourage rigorous
performance assessment to boost the quality of Federal programs. The templates will be proof of concept tested
on the Petroleum Marketing Surveys – a relatively stable, relatively
well-documented family of surveys.
External Evaluations of Forecasting and Models Douglas
Hale, SMG, EIA
The Office of Management and Budget has long been interested
in quantifying Federal program results to balance against their dollar costs.
That information could then be used to focus Federal resources on those
programs and projects that produce results.
The result of forecasting and analysis programs and products
is to “inform” decision makers and the public.
Though dollar metrics are not possible EIA could employ external
evaluations to show its programs and products are meeting important public
policy needs with high quality analysis.
The purpose of this session is to discuss how external
program and product reviews could be structured to demonstrate (or not)
results.
Friday, October 29, 2004
Data Analysis on the EIA-826-906, Joe Sedransk, lead, and Nancy Kirkendall, SMG, EIA
EIA Form 826 collects information, monthly, from regulated and unregulated companies that sell or deliver electric power to end users. It collects state-level sales volumes, sales revenues, and number of customers by end-use sector (residential, commercial, industrial, and total). The existing sample and methodology to estimate population totals will be described together with the results of an ongoing evaluation of this methodology. Statistical issues that remain, including formation of homogeneous subpopulations for estimation, making greater use of historical time series data and ways to increase precision by pooling similar data, will be presented.
EIA Form 906 collects monthly data from a sample of plants (of regulated companies and unregulated independent power producers) on total fuel used (by type) for power generation, total electricity generated by prime mover type, total fuel (by type) used to generate electricity by prime mover type, and fuel stocks at the end of the month. EIA Form 920 collects analogous data for plants for combined heat and power producers. Plans to extend the analyses described above for the EIA 826 will be outlined.
Post-Stratification Methodology for the 2002 Manufacturing Energy Consumption Survey (MECS) Rick Hough and Stacey Cole, U.S. Census Bureau
The goal of the Manufacturing Energy Consumption Survey (MECS) is to provide a comprehensive set of detailed energy statistics for the Manufacturing sector for government and non-government policy-makers. The survey is sponsored by the Energy Information Administration and is conducted every four years by the U. S. Census Bureau.
The 2002 MECS is a probability sample of approximately 15,000 establishments selected from the manufacturing sector of 2002 Economic Census (EC) mail-file. For a large portion of the 2002 EC mail-file, the individual establishment-level industry classifications have not been updated since the 1997 Economic Census. Approximately 5% of the establishments are expected to change industry classifications based on their reported 2002 census data. Given that the MECS sample design incorporates both industry and geographic stratification, there are concerns regarding the representativeness of both the sample frame and the resulting MECS sample. Consequently, we developed a post-stratification procedure to address these concerns.
The 2002 MECS is the first MECS where the reference year
coincides with the EC. This provides us
with the opportunity to improve the survey estimates by using the results from
the 2002 Census as a benchmark. Using
the set of complete set of in-scope establishments in the EC and their
associated energy data, we will derive an “energy consumption” control total
for each sampled cell. The sample
weights of MECS establishments within each cell will then be ratio adjusted
such that their weighted “cost of energy” is equal to the corresponding control
total. The focus of the paper is to
describe this methodology and the presentation will summarize the impact of the
procedure on the survey results.
Time Series Edits for the Electric Power EIA-906, Tom Broene, SMG, EIA
Many believe that simple time series models may provide
useful edits for data from many on-going surveys, especially those conducted
frequently, such as monthly or weekly. The EIA-906 Power Plant Report collects
monthly data from regulated and unregulated generators (excluding combined heat
and power plants.) The data varies in
complexity among the 1600 responding plants, with some consuming only one type
of fuel, but others having several types of equipment utilizing a variety of
fuels. We will describe our efforts to develop simple exponential smoothing
models for generation, fuel consumption, stocks, and the ratio of generation to
consumption. Our initial efforts have
focused on only the regulated respondents with complete data. We will have examples of cases where we were
able to fit the simple non-seasonal model, and cases where we could not.
If You Were King? Howard
Gruenspecht, Deputy Administrator, EIA, Discussion Leader
Each year as the agency confronts new budgets, continuing
resolutions and funding issues related to EIA programs and products, management
is called upon to deal with program choices due to these restraints. If EIA’s budget really looked grim and if we
needed to spend at “continuing resolution” levels,
a. What
would go into your thinking on downsizing?
b.
How would you go about product and program reductions?
c. Consider who is EIA and what should we become?
d. What do we
think we are, and what products do we feel are important? and
e. There are sometimes apparent
differences between being “customer driven” and “we driven.” Discuss which it is?