Chapter 4.  Supplier Surveys: Data Quality and Scope

EIA’s supplier surveys cover all stages of the energy supply picture.  For crude oil and petroleum products, the program covers reserves, production, transportation, refining, blending, imports, exports, and storage. Much of the data collected is reported at the national, regional, and State levels.  For natural gas, EIA collects and publishes data on the operation of natural gas markets at the national and State levels to provide information about natural gas supplies (from domestic and foreign sources), natural gas movement and storage, natural gas consumption by major end-use sectors, and prices along the path from wellhead to end user.  EIA surveys cover the electric power industry by collecting data at the individual facility level and made available at the national, State, sector, and facility level (subject to confidentiality requirements) and are designed to capture data from traditional utilities as well as emerging participants (e.g., power marketers, and other entities engaged in the production, sales, or distribution of electricity).  EIA also collects extensive data on the coal industry and on the nuclear power and the uranium mining and milling industries.  EIA collects and disseminates a limited amount of data from the manufacturers of solar thermal collectors, solar photovoltaic cells and modules, and geothermal heat pump equipment.  These data along with information collected by EIA on its electric power surveys and the MECS provide statistics on the growth in renewable energy in the United States, assess the effectiveness of incentive programs, and provide a basis for projections of renewable energy consumption.  Finally, EIA collects and disseminates data on the production of alternative-fueled vehicles and hybrid-electric vehicles by suppliers, and on the use and fuel consumption of alternative-fueled vehicles by Federal, State, and fuel- provider fleets, and local transit companies.

Many stakeholders and customers have said that EIA supplier survey data are important for State-level policymaking, as well as for responding to emergencies and understanding markets within and affecting their States.  Stakeholders and customers emphasize the importance of EIA’s ability to maintain quality in its data.  They have pointed out room for improvement in EIA’s data, mentioning issues such as occasional discrepancies between weekly and monthly data and gaps in what EIA is currently able to collect and publish.  Taking the feedback received from stakeholders over the years, as well as from workshop participants and others during our 2008 outreach efforts, EIA has developed several data quality options aimed at reducing statistical error in EIA surveys.  Statistical error is inherent in all survey data, regardless of their source and the care and competence of data collectors.  There are various potential sources of statistical error, such as the following:

• Sampling.  A data collection may be based on a sample of the population rather than on a complete enumeration.  Variation occurs by chance because a particular sample is surveyed.  This variation decreases with increased sample size.  A preferred sample size is determined in order to keep key statistics under a specified threshold of sampling error, but sometimes the actual sample size is solely determined by cost.

• Coverage.  The frame (the list of those surveyed) may not be complete or may contain companies that are out of scope or no longer in business.  Complete and up-to-date frames are important in reducing statistical error due to lack of coverage in EIA surveys.  Frame improvements are particularly important in the deregulated electricity sector, which has undergone a tremendous number of mergers, acquisitions, divestitures, and company name changes over the past several years.

• Target population.  Reporting thresholds can be used to determine a survey frame and thereby exclude smaller companies from reporting.  Exclusions result in a frame that does not completely cover the target population of companies of all sizes.  Also, a frame may be incomplete and not cover the parts of the target population that are more difficult or more expensive to reach.

• Respondent.  Respondents may commit errors in reporting their data.  Respondents frequently report statistics in units other than those requested by EIA.  For example, respondents will report data in gallons rather than in barrels.

• Nonresponse.  Not all of the units that are surveyed respond (unit non-response), and some respondents may not provide all of the information requested (item non-response).

• Processing.  Errors may occur from transcribing data incorrectly.

• Concept.  The data collection elements may not measure the items that they were intended to measure, or the estimation methodology may provide inaccurate results.

Most of the options discussed below focus on the first three sources of survey error identified above, either by increasing sample sizes to provide better statistics or by improving survey coverage by updating frames.  Other quality initiatives include improving the scope of several EIA surveys by increasing the sample size to obtain State-level estimates for all 50 States and better estimates for current statistics, or by adding new surveys.  Current statistical data edits, nonresponse follow-up methods, and other internal EIA quality checks are covered by strict adherence to EIA statistical standards and control the last four sources of errors; therefore, no initiatives are proposed for any of these sources of error at this time.

Upgrade Frames

A comprehensive frame is essential to conducting a statistically valid survey.  Frames-related initiatives include:

• Develop Frame for Weekly Retail On-Highway Diesel Price Survey (EIA-888).  Construct an outlet frame of truck stops and service stations to improve the quality of EIA’s weekly diesel price data. EIA is currently unable to provide average prices for more detailed geographic areas than national, Petroleum Administration for Defense (PAD) Districts, sub-PAD regions, and California.  Initiative 4.1. Start-up Cost: $250,000; Annual Operating Cost: $50,000 (increment to EIA’s 2009 budget request). 

• Update Frame for Weekly Retail Gasoline Price Survey (EIA-878).  The frame of motor gasoline outlets was last updated in 2001.  Currently, EIA provides select State and city prices in addition to national, PADD, and sub-PADD average prices.  The updated frame could be used to select a new, larger sample that could produce State-level data for all States. Initiative 4.2. Start-up Cost: $260,000; Annual Operating Cost: $50,000 (increment to EIA’s 2009 budget request).

Expand Scope

Additional initiatives would expand the number of data elements on current EIA survey forms or would increase the sample size in order to obtain more accurate State-level data⁷.  Others would increase the sample size in order to reach additional States.  Still others propose new surveys to collect information on new energy data elements, including renewable energy.  This group of initiatives includes the following:

• Expand Natural Gas Production Survey (EIA-914) to collect data from all natural gas-producing States and to add crude oil and lease condensate production from all oil-producing States.  Expanding the survey would promote improved and timelier data on production trends nationally and particularly for a number of States where oil and gas production is rapidly changing.  Initiative 4.3.  Start-up Cost: $400,000; Annual Operating Cost: $325,000 (increment to EIA’s 2009 budget request).

• Expand Scope of the Monthly Gas Marketer Survey (EIA-910) to obtain data on sales by marketers for more States.  The current scope of the survey includes only the 12 States that offer natural gas retail choice programs with the highest concentration of natural gas sold by marketers.  Currently, there are 22 States with residential sales by marketers and 49 States with commercial and industrial sales by marketers.  Initiative 4.4. Start-up Cost: $600,000; Annual Operating Cost: $500,000 (incremental to EIA’s 2009 budget request).

• Expand Power Plant Operations Survey (EIA-923) to collect fossil-fuel receipts from smaller electric plants (those with capacities between 1 megawatt and 50 megawatts).  An expanded survey would complete the picture of the electric power industry and match the reporting threshold by which fuel consumption and electricity generation data are collected.  Initiative 4.5. Start-up Cost: $50,000; Annual Operating Cost: $65,000 (incremental to EIA’s 2009 budget request).  

• Expand Winter Heating Fuels Telephone Survey (EIA-877) from 24 to all 50 States and from its current data collection period of October through mid-March to a year-round survey.  This survey collects weekly data on retail prices of No. 2 heating oil and propane.  These data are used to assess hardships experienced by heating oil and propane users during periods of critical short supplies.  The current survey is a cooperative data collection effort between EIA and 24 States. (New initiatives for joint data collection are discussed in Chapter 5.)  Initiative 4.6 Start-up Cost: $750,000; Annual Operating Cost: $750,000 (incremental to EIA’s 2009 budget request).  

• Develop and Launch Weekly State Inventory Survey for Selected Petroleum Products similar to EIA’s current monthly collection of inventory data for selected petroleum products such as distillate fuel oil, motor gasoline, kerosene, residual fuel oil, and propane.  EIA would use the weekly data to better monitor any pending supply issues.  Initiative 4.7. Start-up Cost: $1,000,000; Annual Operating Cost: $500,000 (incremental to EIA’s 2009 budget request).

• Develop and Launch New Survey to Collect Biomass Energy Consumption using a frame developed by EPA of all forest-product facilities.  This survey would permit EIA to publish State-level biomass data.  Initiative 4.8. Start-up Cost: $190,000; Annual Operating Cost: $105,000 (incremental to EIA’s 2009 budget request).  

• Add Methane Data Elements to Existing Survey (EIA-895A), “Annual Quantity and Value of Natural Gas Production Report,” which currently collects vented and flared natural gas as a single value, despite the fact that vented natural gas is a methane emission and flared natural gas results in carbon dioxide emissions.  The distinction is important because methane is a more potent greenhouse gas than carbon dioxide.  This initiative would require respondents to provide separate values for vented and flared gas and would lead to improved accuracy of State-level greenhouse gas emissions.  Initiative 4.9 Start-up Cost: $70,000; Annual Operating Cost: $70,000 (incremental to EIA’s 2009 budget request).

Notes