Thursday Morning, April 3, 2003
EIA’s Survey Quality Effort: Where is EIA Going? Shawna Waugh, Statistics and Methods Group (SMG) Moderator, Jim Joosten and Tom Murphy, Coal, Nuclear, Electric and Alternate Fuels (CNEAF), and Nancy Kirkendall, Tom Broene, John Vetter and Howard Bradsher-Fredrick, Facilitator, SMG.
Overview of EIA’s Survey Quality Initiatives, Nancy Kirkendall, Director, Statistics and Methods Group, (SMG) EIA. EIA is undertaking several initiatives to plan, implement, measure, and evaluate the quality of survey data. Recent agency-wide quality initiatives include the: EIA Strategic Plan, which emphasizes data quality; EIA’s top ten priorities for 2003, which includes layered performance measures; EIA’s survey quality effort; and the Continuity of Operations (COOP), which demonstrates a need for survey documentation.
A new initiative will involve survey managers identifying specific indicators of the survey process, and is intended to improve survey process and results. This initiative will involve collaboration between Survey Managers, SMG, and an EIA contractor.
ASA Committee Advice
The Committee agreed that a survey quality initiative would be useful to EIA program and survey managers, and recommended:
EIA’s Intended Response to ASA Advice
The project is a joint effort between EIA staff and its contractor, and is in its early stages. ASA Committee recommendations are under consideration, and decisions will be announced once available.Session Overview
Survey Quality Efforts of the Office of Coal, Nuclear, Electric and Alternate Fuels (CNEAF), Jim Joosten, CNEAF, EIA
This presentation briefly described the type of data collection and processing activities in CNEAF and the various programmatic initiatives that have been undertaken over the last 12 months to improve quality. CNEAF's quality program focuses primarily in the area of developing new Quality Control (QC) measures. It is designed to integrate within the following six core elements of EIA's overall quality program:
1. Quality Policy: How do we
prioritize and allocate resources?
2. Quality Standards: What do we have, how do they apply to the staff and how do we assure compliance with them?
3. Quality Guidelines: How do we develop sufficiently detailed quality guidance for the staff, and what do we expect them to do with the guidelines?
4. Quality Controls: What checks do we have to verify that the staff is doing quality work while the work is being done?
5. Quality Assurance: How does management independently assure itself that the staff’s quality control system is working?
6. Quality Improvement: How do we learn from the system and create a self-adjusting process to make it better? What feedback mechanisms do we have from the staff, contractors, and stakeholders? How do we disposition the feedback?
The CNEAF Quality Review Board (QRB) plays an important new role in this process and could serve as a model for an EIA-wide quality system.ASA Committee Advice
The ASA first addressed a question of what level of quality control is appropriate for the EIA and
CNEAF. Rather than merely accept an across- the-board quality program, they suggested a cost-versus-benefit evaluation of the program. In other words, for specific data collection areas, EIA should weigh what it will gain from increased quality controls against the expected resource expenditures. They recommended that EIA design a flexible rather than static quality management system. The ASA supported the idea of developing QC checklists but cautioned that some areas (e.g., improving response rates) may require a different tool. They also cautioned against developing check lists which are too short or too vague so as not to be particularly useful as a quality record or feedback tool to management.
The ASA favored CNEAF’s creation of a Quality Review Board that meets regularly, and encouraged the EIA to better document and understand its processes as a means to improve quality. They encouraged CNEAF to develop the QC tools with the full involvement of the survey staff. On the issue of adopting an ISO-9000 quality management program, the ASA committee agreed that CNEAF plans to build quality concepts consistent with that system into its management systems had merit. But they cautioned against pursuing an ISO 9000 certification as being too costly for too little benefit.
The ASA also encouraged CNEAF to use an internal, independent, peer review process to manage quality, and to exercise final control over the release of its work products. The Committee also suggested that EIA consider the addition of other quality tools, such as: cognitive testing, respondent debriefings, usability testing, response analysis surveys, etc.
Finally, to help ensure the sustainability of the quality initiative, the ASA encouraged the EIA to track its quality successes and make them known to the staff as a means to further motivate the staff.EIA’s Intended Response to ASA Advice
In response to the advice provide by the ASA, the EIA Office of Coal, Nuclear, Electric, and Alternative Fuels (CNEAF) has decided to implement the following actions:
The Office Quality Management program will be developed with sufficient flexibility and feedback mechanisms so that it can be reviewed on a regular basis, evaluated in terms of cost-versus-benefit, and adjusted. For example, certain actions aimed at achieving quality excellence (rather than quality minimums) will be included in the Survey Manager’s checklists as a memory aid. However, the manager will have the flexibility to bypass those quality steps provided that they document their rationale for later management review. Additionally, a Quality Improvement program will be developed with periodic input to the QRB so that CNEAF’s initial QA/QC measures are not final but rather adjustable over time as experience is gained. Performance measures will also be developed in the survey process to help management assess the true cost and benefits of the QA/QC/QI program in terms of staff resource usage versus product timeliness and quality.
The operating procedures for each survey will be updated or additional guidance will be provided to survey managers in order to supplement the shorter checklist items. The office will consider the creation of a frames manager or committee to help improve response rates and frame coverage.
The QRB concept will be continued with regular meetings approximately each month, and avenues to expand its coverage of quality issues will be examined.
A formal process such as the ISO 9000 quality system will be used as a model for developing the CNEAF quality documentation systems and quality feedback processes. However, ISO-9000 certification will not be pursued
The quality tools suggested by the ASA (cognitive testing, respondent debriefings, etc.) will be included in the checklists as a reminder for consideration.
An internal web page will be developed to track quality successes, EIA quality issues and successes will be discussed weekly in the CNEAF senior staff meetings, and the possibility for staff quality awards will be consideredSession Overview
Survey Quality via Performance-Based Service Contracting, Tom Murphy, CNEAF
One vehicle for improving survey quality is performance-based service contracting (PBSC). This type of contract emphasizes outcomes, rather than process, and thereby encourages contractors to seek the most efficient path to achieve a stated goal. The five key elements of PBSC include:
1. A statement of performance requirements,
representing a clear and concise expression of the objectives the contract
is expected to achieve.
2. A description of the performance standards tied to the performance requirements the contractor will be expected to satisfy.
3. A description of the quality assurance performance measures for the performance standards that will be used to evaluate the contractor’s performance;
4. A program of positive and negative economic incentives based upon the performance measurements to ensure compliance with performance standards; and
5. A statement of the evaluation criteria upon which the agency will rely in evaluating response to a Request for Proposal
This session was informational and neither sought nor received suggestions for change or improvement.
Thursday Afternoon, April 3, 2003Session Overview
New Confidentiality Law and EIA’s Response, Jay Casselberry, SMG, EIA
The E-Government Act of 2002, Public Law 107-347, was signed into law on December 17, 2002. Title V of that Act is the Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA). CIPSEA’s Subtitle A, Confidential Information Protection, affords a possible new level of confidentiality protection to statistical data and information collected by the Energy Information Administration (EIA) and other Federal agencies.
EIA’s first step was to analyze CIPSEA and decide what actions should be taken. Based on that work, EIA had to resolve questions concerning the Acts’ coverage and requirements, particularly with respect to EIA’s operations. EIA then established a cross-organizational team to recommend confidentiality to be applied to surveys and to work on operational issues of implementing CIPSEA.ASA Committee Advice
The Committee and Dr. Calvin Kent (past Committee member, previous EIA Administrator and current guest) provided extremely insightful comments on the presentation, the Confidential Information Protection and Statistical Efficiency Act (CIPSEA), and how EIA should address the questions raised. With regard to sharing of EIA information, the recommendations were that EIA should take a position that it will not release any confidential information for use by others for regulatory or legal purposes. EIA should beforehand address what information will be available to other agencies, under what conditions, and be prepared for the consequences if EIA chooses not to provide information for nonstatistical uses. The Committee also recommended developing informed consent materials before needed and creating data sharing agreements and procedures that strictly control all aspects of shared confidential information.EIA’s Intended Response to ASA Advice
EIA is still addressing how best to change in response to CIPSEA. A limited number of surveys will be included under CIPSEA, which will preclude any nonstatistical uses for information from those surveys. For confidential surveys that are not collected under CIPSEA, EIA will continue to resist any nonstatistical uses, but must accept the fact that laws require sharing for official uses and do not provide strong support for resisting nonstatistical requests. EIA will need to use its political and persuasive skills to avoid the potential for inappropriate uses of non-CIPSEA confidential information. EIA will be modifying its confidential wording, data sharing agreements, and internal controls in light of CIPSEA. EIA will also need to develop appropriate procedures for agents handling CIPSEA data. OMB will be providing more guidance on that topic.
An Alternative Natural Gas Production Estimation Procedure, Crystal Linkletter, ASA Fellowship recipient, and Professor Randy Sitter, ASA Committee member
There exists a delay between the production of natural gas in Texas and the reporting of that production. As a result, the true volume of natural gas produced in a month is not known until a year or more after production. It is important to have earlier and more accurate production estimates, and some estimation procedures have already been considered. The focus of this presentation was to look at alternative procedures that give reasonably accurate production volumes no more than three months after production.
One very intuitive approach involves inflating the latest available reported totals by a ratio that takes into consideration the right-truncation of the data. This method performs reasonably well, and does not require the collection of much data. A second approach relies on the nonparametric estimation of the conditional reporting lag distribution. This is the methodology commonly used to adjust for reporting delays in AIDS research and in the construction of product warranty databases. This procedure is very easy to apply to the natural gas production data, has the greatest efficiency of all the procedures considered, and allows for the construction of prediction intervals.
ASA Committee Advice
Use of well level data or even regional data would probably be better than the aggregated data obtained from the Texas Railroad commission. But, given that at present we only have the aggregated data, the non-parametric method just described by Crystal Linkletter, described in this session is the best method to use. This method had the smallest mean squared errors of any of the alternative methods. There are three additional reasons for preferring this method: 1) It is known to work well in other areas, such as AIDS research and Product Warranty Research, 2) It is simple to use, requiring only simple sums in a spreadsheet, and 3) One can construct confidence intervals on the predicted values. For these reasons the ASA committee recommended the use of the non-parametric model described in this session.
EIA Intended Response to ASA Advice
SMG has implemented the non-parametric model and will provide predicted values to the EIA Office of Oil and Gas (OOG). The model results will be evaluated and a final determination will be made by OOG after the initial review.
EIA’s (Draft) Electricity Transmission Study: What the Data Show, Douglas R. Hale, Statistics and Methods Group, EIA.
EIA is responsible for collecting, maintaining and analyzing data on energy prices, infrastructure and trends in supply and demand. Historically, EIA’s data collection and analysis efforts in the power sector have focused on generation (production) and the demand for electricity. The Federal Energy Regulatory Commission (FERC) has generally collected information on capacity and investments in the electricity transmission sector. Although the qualitative effects of transmission on prices and interregional trade are understood, there is currently limited factual and analytic basis for appraising how well regional grids are evolving to support competition. This report will assess whether publicly available transmission data and analysis tools answer questions important to resolving public policy issues.
ASA Committee Advice
Dr. Calvin Kent, guest of the ASA Committee, was concerned about the cost and political fallout from EIA taking on transmission data and analysis. Ultimately, that is the EIA Administrator's and the Department of Energy's call. EIA’s goal in this study is to figure out how much would need to be done by someone to get the data to support policy analysis.
Dr. Kent had specific comments as well. He suggested EIA:
1. Focus on investment, defined broadly enough to include more than lines
2. Incorporate homeland security concerns explicitly in the report.
EIA’s Intended Response to ASA Advice
Dr. Hale currently intends to take both suggestions.
Friday Morning, April 4, 2003
Redesign of the EIA-906, Power Plant Report, Stan Freedman, Statistics and Methods Group, and Bob Schnapp, CNEAF, EIA.
Form EIA-906, Power Plant Report, collects information from all regulated and unregulated electric power plants in the United States. Data collected include electric power generation, energy source consumption, fossil fuel stocks, and useful thermal output (UTO) from combined heat and power plants (CHP). Examples of CHP plants are industrial cogenerators such as some paper mills and chemical plants. EIA publishes UTO and electricity generated by CHP plants. Useful thermal output is the thermal energy made available for processes and applications other than electrical generation. EIA uses a facility’s reported total fuel consumption and UTO to calculate, essentially as a difference, the amount of fuel used by CHP plants to produce electricity.
EIA collects UTO and uses it as a means for determining fuel for electricity because of a belief that CHP respondents do not typically have metered or estimated values for fuel used to produce electricity. However, we have found that respondents have difficulty in reporting UTO, and in some cases they do not measure useful thermal output.
EIA recently began the redesign of the EIA-906. At this point, EIA has completed a survey concepts analysis and has scheduled several pre-survey design visits for March to see if respondents have the data EIA wants and can provide it to the agency in the form and periodicity EIA requires.
Using the information collected from the pre-survey design visits, EIA will construct data collection options from which it will chose one to construct a prototype survey. EIA will then cognitively test the prototype with approximately a dozen CHP respondents. This is scheduled for May and June 2003.
EIA wants the Committee’s advice on how to collect the data for the redesigned EIA-906, forms design, and cognitive testing of the draft instrument.
If cleared by the Office of Management and Budget, the new survey will be sent to respondents in 2004.
These were the questions that Stan Freedman and Bob Rutchik asked the Committee:
1. Should EIA ask respondents for additional
information to calculate the estimates, and continue to perform the computations
for fuel use here?
2. Should we have the respondents perform their own fuel consumption calculations?
3. Are there alternatives?
The Committee stated that there was no easy answer to determine how much fuel combined heat and power (CHP) plants use to produce electricity. They were not convinced, however, that permitting CHP respondents to calculate their own fuel use was the way that EIA should precede. They did not think that the respondents would produce accurate data.
The sense of the Committee, after a great deal of discussion, was that EIA could construct a respondent model. This model would be a combination of stratifying the respondents= by plant characteristics and certain Abaseline@ variables common to all plants. Using this model, EIA could survey the most problematic respondents, the ones that most affect the precision of the CHP fuel use estimate, and do the fuel use calculations.EIA’s Intended Response to ASA Advice
EIA soon is going to test cognitively a redesigned EIA-906 that asks CHP respondents to calculate the fuel that they use to produce electricity. EIA believes it does not have the resources at this point to construct a respondent model.
Using Data from Combined Heat and Power Plants to Estimate Natural Gas Industrial Prices, Ruey-Pyng Lu, Statistics and Methods Group, EIA.
The Energy Information Administration (EIA) collects and publishes data on prices and volumes of natural gas delivered to customers in five sectors (residential, commercial, industrial, vehicle fuel, and electric utilities). EIA has been obtaining the data by surveying local utilities and pipeline companies. Since industrial customers bought more natural gas from marketers or suppliers than their local utilities, EIA’s natural gas prices for the industrial sector represent only about 18% of the gas consumed in that sector.
In 2000 SMG contracted with the Census Bureau to conduct a feasibility study for surveying natural gas customers in the manufacturing sector (based on a frame maintained by the Census Bureau). The feasibility study demonstrated that the required information could be collected. However, the estimated cost exceeded EIA’s budget. EIA is studying alternatives to estimate natural gas industrial price.
Many industrial facilities that consume natural gas use it to generate electricity and now report on the EIA-423 (Monthly Report of Cost and Quality of Fuels for Electric Plants). The EIA-423 surveys all facilities with a nameplate capacity of 50 megawatts or greater and collects information about cost and quality of all fuels used to generate electricity. It is hoped that by matching the natural gas consumers in the EIA-423 with the natural gas customers in the EIA-176 (Annual Report of Natural and Supplemental Gas Supply and Disposition), a model can be developed to estimate the price of natural gas used in the industrial sector.
We have six months, January 2002 to June 2002, of EIA-423 data available to investigate the potential relationship of natural gas price and volume. But the data sets need more editing to verify the responses, there are some questionable responses in each month, and the EIA-423 survey manager is investigating those responses. Through the exploratory data analysis, there seems no apparent relationship between natural gas price and volume at all at national or census region level. We will compare the EIA-857 natural gas monthly data with EIA-423 data to establish a correlation to adopt a proxy for the industrial price. This session will represent the work done to date and solicit the committee’s advice.
Summary of Advice to the EIAASA Committee on Energy Statistics advice:
ASA Committee on Energy Statistics stated that there are three available sources of data from surveys on natural gas prices of industrial sector, EIA-423, FERC 423 and EIA-857. These multiple series are useful, and perhaps it wasn’t so useful to work hard on finding a way to aggregate them into one number that could be called the average price paid by the industrial sector. EIA may put out two series, data from EIA-423 and EIA-857, and it would be useful to characterize the coverage of those different series and survey frames because there may be some overlap between these surveys. It also sort of forces people who would want to just get the number from EIA, to sort of recognize that’s maybe not what they might want to do, and there is no single number that answer this question very well, and they shouldn’t pretend that there is one.
EIA’s Intended Response to ASA Advice
We will report two price data series from EIA-423 and EIA-857 surveys, along with their coverage and frames From these surveys, hopefully we can work out a model which can be used to estimate natural gas prices for the industrial sector at the census region level. One other alternative is to use other natural gas price series, such as the natural gas wellhead price or Henry hub price to develop a model and use it to estimate the natural gas price for the industrial sector.
Questions and comments may be referred to Bill Weinig, EI-70. Bill is EIAs liaison with the ASA Committee on Energy Statistics, and can be reached at (202) 287-1709, or by email at firstname.lastname@example.org.