EIA Standards

 

Statistical Standards

EIA has adopted the Office of Management and Budget (OMB) Standards and Guidelines for Statistical Surveys. These standards will be supplemented with EIA specific standards which are currently under development.

 

COOP, Records Management, and IT

• Information Technology (IT) Systems
• Information System Documentation
• Business Process Documentation

Model Documentation

• Model Documentation
• Model Documentation Requirements
• Model Documentation Components Checklist
• Model Archiving
• Use of Proprietary Models

Communication

EIA will develop, as needed, standards related to external communication and electronic dissemination on topics including plain language, information accessibility, media relations, and other communication-related issues.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Energy Information Administration Standard 2002-2

Title: Information Technology (IT) Systems

Superseded Version: Standards 88-02-01, 88-03-02, 88-03-03, and 88-03-04

Purpose: To ensure that the EIA IT systems are developed, maintained, and secured using best IT practices and that they adhere to established policies, standards, and procedures.

Applicability: All IT systems at EIA.

Background: EIA has standards for developing and documenting computer systems. These standards ensure that systematic procedures are used when new IT systems are developed and existing systems are revised; that these systems meet business and functional requirements; and that the systems can be operated, modified, and backed up. With the advent of the Internet, the EIA IT standards have been expanded to address data access and security issues arising in the Web environment.

Required Actions: All EIA and contractor staff involved in developing and maintaining IT systems are required to follow applicable Federal Information Processing (FIPS), Department of Energy, and EIA-specific IT system standards, policies, guidelines, and procedures listed below.

Related Information:

1. Federal Information Processing Standards (FIPS)
2. Department of Energy IT Standards
3. 2002-30, Business Process Documentation

Approval Date: September 26, 2002

 

Energy Information Administration Standard 2002-3

Title: Information System Documentation

Superseded Version: Standards 88-03-02 and 88-03-03

Purpose: To ensure that EIA information systems are documented adequately to allow personnel unfamiliar with them to become knowledgeable about them and to operate them, if necessary.

Applicability: All EIA information systems.

Required Actions:

1. If an EIA information system is a modeling system, it must follow the requirements of EIA Model Documentation Standard 2002-26.

2. All other EIA information systems (e.g., survey systems, secondary data systems, and other systems supporting other key EIA business practices) must have documentation of all operations (both automated and manual) necessary to operate, maintain, and update the systems.

• All new information systems should have up-to-date Users and Developer's Reference Manuals (see Enterprise Architecture Guidance).
• All information systems created prior to the approval date of this standard should have up-to-date Operations and Programmer's Maintenance Manuals as required by the predecessor
  EIA Standards/Statistics Manual.
• When documenting an information system that is not a modeling system, the following topics should be covered, if applicable: 1) an overview of integrated manual and automated operations, workflow, interfaces, and personnel requirements; 2) frames development and updating; 3) sampling design and methodology; and 4) a detailed description of each step in collection/processing/dissemination, including distribution of survey materials, establishment of access to on-line reporting options, data collection methodologies, respondent contact handling, non-response procedures, data entry procedures, data editing procedures, error correction/adjustment procedures, estimation methodology, quality comparison procedures, withholding procedures, revision procedures, dissemination, and back-up and archiving procedures. This information may be incorporated into the existing documentation or written as a separate document.

Related Information:

1. 2002-30, Business Process Documentation

Approval Date: September 26, 2002

 

Energy Information Administration Standard 2015-1

(also available in pdf)

Title: Model Documentation and Archiving

Superseded Version: Standards 96-01-03 and 2002-26

Energy Information Administration Standard 2015-1
Purpose: To ensure that the procedures, equations, and assumptions that define EIA models are publicly available.

Applicability: All models developed and maintained by EIA or its contractors. (In a large model or system, separable components or modules may be considered as individual models for documentation purposes.)

Section 2015-1.1. Model documentation requirements

1. Model documentation should correspond to a specific archived version of the model.
2. The documentation should include the required components listed in the Model Documentation Checklist and additional information as the Office of Energy Analysis (OEA) deems appropriate.
3. To the extent possible under the existing resource constraints, the documentation should be prepared in accordance with the "Guidelines for Mathematical Specification in Model Documentation."
4. The documentation will be available in the standard format determined by OEA. A Word template is available from OEA or the Office of Communications (OC).
5. All documentation should undergo a quality assurance review by OEA prior to dissemination. In addition, new or significantly revised model documentation may undergo reviews by other EIA offices and/or independent expert reviewers from outside EIA when appropriate.
6. When a product using model outputs is sent to the Administrator for approval, the responsible Office Director should specify what documentation is available. If up-to-date documentation is not available, the responsible program office should provide a schedule whereby it will complete the documentation within 90 days of the product's release. If the documentation cannot be completed within this time frame, the program office may request an exception.
7. The most current model documentation should be available on EIA's Website. Documentation related to previous versions of a model may also be available as part of a model archive package (see Section 2015-1.3).

Guidelines for mathematical specifications in model documentation

1. Text of the mathematical specification
   1. The mathematical specification of the model should be an unambiguous formal statement of the modeler's concept of the methodology and structure to represent real world phenomena. Given the description of the model's methodology and structure along with knowledge of the model's inputs and transformations, the reader should be able to form an intuitive understanding of the model rationale, the specific interrelationships and assumptions represented, and the essential model outputs.
   2. Portions of the mathematical specifications can be included in appendices; the text and the appendices jointly should provide a complete specification of the model. In practice, an interested person may investigate the documentation and computer code as well as discuss the model with EIA staff to more fully understand the model's intricacies.
   3. When standard, commonly used algorithms are applied, references to published literature may be provided. Standard methods used in the code need not be described in complete detail, but the documentation should convey the intuitive rationale underlying these methods.
   4. Documentation should be written in the indicative mood (e.g., "the subroutine calculates the parameters"). Occasional lapses into the imperative mood (e.g., "calculate the parameters") give the reader the impression that computer programming specifications have been pasted into the documentation without proper adaptation.
   
   
2. Documenting optimization methods employing iterative techniques
   1. In cases where an optimization problem is stated, or an iterative process is used to find a solution of some mathematical problem, it is important to state precisely what optimization problem is being solved, or to state precisely, in mathematical terms, the solution to be obtained when the iterative process is complete.
   2. Where iterative processes are used, the basic solution methodology should be described. If there are key parameters, such as a damping parameter, that are involved in the iterative process, then these should be described and the relevant equations containing these parameters given.
   3. The mathematical specification in the documentation need not deal rigorously with the possible problems of non-convergence or multiple solutions. However, if an iterative procedure in the code has a default condition which allows the computation of the program to continue even if convergence is not attained, then this condition should be stated.
   4. If a linear program problem is of such a size and complexity that it is impractical to include the complete specification, the documentation should provide a description of the structure of the problem in as much detail as practical.
   5. The documentation should provide all information needed by an expert to access electronically any desired coefficient or constant in the problem, to determine the dimensions of the objective function vector and constraint matrix, to identify which constraints involve strict or non-strict inequality, and to obtain all information necessary to generate the results of the optimization problem.
   
   
3. Presenting regression diagnostics
   1. In cases where parameters are estimated by statistical regression (ordinary least squares or a similar method), the parameter estimates, R² statistic, t-statistics, and other key diagnostics may be presented in one of two ways:
      1. A discussion may be included in the documentation, or
      2. An output listing from a statistical or econometric package may be presented. The output should be clearly labeled and modified, if necessary, to identify the regression model and the particular parameters for which the diagnostics are provided. Explanatory lines within the package output should be added, as needed, for clarity.
   2. When the regression is based on time series data, a unit root test statistic (e.g., Dickey-Fuller) and/or residual autocorrelation test statistic (e.g., Durbin-Watson) should be reported, along with an associated p-value.
   
   
4. Mathematical equations
   1. To the extent possible, each equation should appear immediately before or after its own explanatory text.
   2. b) When a variable that has been defined is used again in an equation that appears more than two pages after the definition, the definition should be repeated or a page number or equation number for the definition should be given (e.g., "where x is as defined on page n" or "where x is as defined in equation n"). Alternatively, the documentation can include a single comprehensive, alphabetized list of variables with definitions.
   3. When numbers (e.g., estimated parameter values, unit conversion factors) are inserted into an equation, their meaning should be explained in the text.
   4. All the fonts, including subscript fonts, used in the equations should be large enough to be easily read, even if this means breaking some equations into multiple lines. (Font sizes below 6 pt. should be avoided.) Ideally, the font used for full-sized symbols in the equations should be at least as large as the font used in the text. Second and subsequent lines of one equation should be indented to indicate that the equations are continued from a previous line.
   
   
5. Subscripts and indices
   1. Subscripts or indices should be used to indicate all index variables on which each variable depends, e.g., month or year for time series variables, region for geographically defined variables.
   2. A subscript may be suppressed in an equation only if
      1. it indexes all variables in the equation and
      2. the text clearly states that the subscript has been suppressed to provide a cleaner notation.
      For example, when the documentation describes a long series of computations performed within a loop over certain variables (e.g., year, region, industry), suppressing the subscripts that define the loop is often convenient and improves readability. Before the first equation in which subscripts are to be suppressed, the text should state, "In equations x through y, the subscripts a,..., z have been suppressed for a cleaner notation," where x and y are the numbers of the first and last equations, respectively, in which the suppression occurs, and a,..,z are the suppressed subscripts. After equation y, the text should remind the reader that the subscripts have been suppressed in the previous equations and state that, unless otherwise indicated, the suppression will not occur in subsequent equations.
   3. If the expression on the right-hand side of an equation explicitly depends on certain indices, the indices must also appear on the left-hand side, unless they are being aggregated out. For example, we may have
      (1)
      but we cannot have
      (2)
      Rather, (2) must be written
      (3)
   4. The subscripts that appear on a variable in its definition should be consistent with those that appear on the same variable in the equation in which the variable is used. The variable definition should include the definitions of the subscripts (e.g., S_t,r,y = average stock of vehicles of type t in region r during year y).
   
   
6. Relating the mathematical specification to the computer code
   1. Often a single mathematical equation in the documentation is equivalent to several computer code statements. Certain "temporary variables" that are used in the code to temporarily store values need not be discussed in the documentation. In some cases, words rather than a mathematical expression can be used to specify simple computations, but only if it is clear how the stated ideas would be expressed mathematically.
   2. When parallel computations are performed on several variables, one equation in the documentation can describe several lines of computer code. In such cases, the documentation should clearly describe the several instances to which this general equation applies, and relevant subscripts or arguments for representing the various categories should be clearly defined.
   3. Variable names used in the computer code are often too long to be convenient for use in the documentation. When a variable name in the computer code differs from the name used in the documentation, the documentation should provide a cross reference list. (Computer code variables that have no exact counterparts in the documentation, such as temporary variables, need not be cross referenced.) Cross referencing can be accomplished in one of two ways:
      1. A cross-reference table may be included in the documentation, or
      2. When there is a declaration in the computer code of a variable which is the equivalent of a documentation variable, a comment in the code may indicate the name of the equivalent variable in the documentation, in the distinctive form {vn}, where vn represents the variable name in the documentation. The distinctive form makes it easy to locate the names electronically. (If another distinctive form is used, then the documentation should state the form.) Comments may also be placed next to the variable names (e.g., a FORTRAN COMMON block) or in another location in the code.
      For a given model, the responsible program office should choose either (i) or (ii) above. (When a program office decides to switch options, a combination of (i) and (ii) may be used for a limited time during the transition period. This situation should be explained in a note accompanying the variable cross reference list.)

   In some cases, one documentation variable name might correspond to several computer variable names. In some circumstances, for example, the computer code might use two different variables that have the same definition but are in different units. If one variable name is used in the documentation to represent multiple variables in the computer code, an explanation should be included in the variable cross reference list.

Section 2015-1.2. Model documentation components checklist

Elements are required unless "optional" is specified. Materials need not be presented in the order discussed here. Not all of the information included in a single item below need appear together as a unit; the information may appear in different sections, if this arrangement improves the flow of the text.

1. A reference to the appropriate archive package.
2. A high level description of changes from the previous archived version. (This may be omitted in cases of one-time reports or in the case of substantial changes from the previous archived version.)
3. Model overview: A concise description of the model, its purposes and uses, how it generates forecasts, critical assumptions, and a discussion of any significant departures from accepted theory or practice.
4. Process flow diagram (optional): A flowchart showing the sequencing of the data inputs, calculations (processes), and outputs of the model.
5. Mathematical specifications: The equations representing the computations performed in the model. The relevant equations include those used to transform input data and parameters into model data and parameters, as well as equations that characterize the solutions of algorithms. For linear programming models, for example, the relevant equations include the objective technology matrix and constraint matrix. For additional guidelines, see the "Guidelines for Mathematical Specifications in Model Documentation."
6. Variable and parameter definitions: The following should be included for all variables and parameters used in the documentation:
   1. Clear definitions, including data sources for the input variables and parameters.
   2. Units of measurement. The reader should be able to see (though it may take some investigation in some cases) that the units on the right-hand side of an equation are the same as those on the left-hand side.
   3. Whether each variable is an input, output, or only used in intermediate calculations. In many cases, this will be clear from the text of the documentation.
   4. Whether data elements are direct inputs to the model or only used in preliminary calculations, e.g., to estimate fixed parameters for input to the model. If data were transformed or manipulated prior to use, an explanation should be provided.
   Computer-code variables that have no counterparts in the model documentation, such as temporary computer-code variables or variables used only in debug statements, need not be defined in the documentation. Their definitions should be included as comments in the computer code.
7. Model estimation procedures: The methods and data sources used to estimate parameters and other quantities in the model should be identified. Enough information about the estimation techniques should be given to allow an expert to exactly reproduce the estimation results. This includes a precise citation of data sources, the data series used, and an exact description of which portions of the data series are used in each calculation.
8. Existence and uniqueness of solutions (optional): For iterative or optimization problems, in cases where the existence or uniqueness of a solution has been demonstrated by analytical means, a description may be presented. Tests from different initial value conditions should be conducted to provide evidence of the uniqueness of solutions.
9. Sensitivity analysis (optional): Tests should be performed to determine whether changes in key model inputs cause key model outputs to respond in a sensible fashion. If a sensitivity analysis is performed, the results of the most recent analysis should be included in the documentation or a reference should be given to an information product that provides these results.

Section 2015-1.3. Model archiving

Title: Model Archiving

Superseded Version: Standards 91-01-04 and 2002-27

Purpose: To ensure that EIA model calculations are reproducible.

Applicability: All models used by EIA.

Required actions:

1. A model archive package should be prepared by the program office when model outputs are used in an EIA product that is publicly disseminated.
2. A model archive package should contain the following:
   1. All source code and program control files needed to compile, link, and execute the reference or base case scenario of the model. If alternate source code versions were used to run the model for other scenarios cited in the same product, these versions should also be provided unless a scenario-specific archive has been created.
   2. Input data files used by the model for the reference or base case cited in the model, along with other file versions that are needed to run the model for other scenarios cited in the same product. The data values should be provided in an accompanying computer file or files (e.g., ASCII, spreadsheet, or database files may be used). Each row and column of the data file should be labeled.
   3. Primary output (as opposed to debugging or trace files) from the reference or base case scenario used in the product. It is not necessary to provide all output files for all scenarios cited in the product, as long as the outputs not provided can be regenerated from runs of the archived model and the primary outputs from the model can be verified against disseminated results.
   4. Instructions for compiling and running the model and comparing the results to disseminated results. A description of changes needed to run the alternative scenarios published in the report or to create scenario-specific archives should be included.
   5. The source of the proprietary data and software, along with instructions for obtaining these, should be included in the archive instructions. However, an archive package should exclude proprietary data and software used in the model. (In some cases, the program office may choose to offer an alternate version of the model, modified to exclude proprietary data and software. The alternate version will have more limited functionality than the full model used internally by EIA.)
3. The program office should create and verify an archive package within 60 days of disseminating an EIA information product utilizing model outputs.
4. The archive package must be retained until no longer needed for current business. The program offices should consult with the EIA Office of Resource and Technology Management (ORTM) regarding records retention requirements.
5. (Optional) The program office may develop a policy for public dissemination of the archive that addresses such topics as:
   1. The model's transportability.
   2. Additional software required (such as proprietary embedded models) that is not provided by EIA with the model.
   3. EIA's expectations for how a public user will identify the model if the model has been modified by a user from outside EIA.
   4. Limits on EIA support.
   5. Any other issues pertinent to outside use.

Section 2015-1.4. Use of proprietary models

Title: Use of Proprietary Models

Superseded Version: Standards 91-01-05 and 2002-28

Purpose: To permit the use of proprietary models in EIA modeling systems and in conjunction with EIA products.

Applicability: To all models available to EIA through license, purchase, or subscription.

Required actions:

1. Every agreement for the acquisition or use of a model should provide for the following:
   1. Publicly available documentation of the model's design, theoretical basis, empirical implementation, and objective capabilities.
   2. A means for EIA to replicate model calculations for a period of three years after each application in an EIA product.
2. For an active EIA model,
   1. The model documentation should be available to the public.
   2. The model version and archive of all model inputs and outputs associated with a disseminated information product must be identified so the results can be replicated.
   3. All changes EIA makes to the model should be documented and archived to EIA standards.
   4. A proprietary model should not be embedded in an EIA modeling system unless the model is commercially available.

Approval Date: September 15, 2015

 

Energy Information Administration Standard 2002-30

Title: Business Process Documentation

Superseded Version: None

Purpose: To ensure that EIA's essential business processes are adequately documented so they can be operated efficiently and effectively by anyone with the necessary skills, information input and technical resources.

Overview: Documentation requirements for specific portions of EIA software systems, surveys and models are delineated in several existing EIA Standards. This Standard covers the documentation requirements for the remaining portions of these processes and for EIA's other business processes not covered by existing standards (e.g., administrative, financial, customer service). The documentation will consist primarily of manuals that provide instructions and reference materials to individuals learning how to perform the function. These manuals would be used not only by staff newly assigned to the function, but also by veteran employees who must perform the function when the usual performers are unavailable. This might occur when the usual performer is absent from work, retires, or is unavailable during post-disaster recovery operations.

Applicability: All business processes that are essential to EIA's Continuity of Operations. There are no waivers to this Standard.

Required Actions:
This Standard requires the following:

• Designation of those business processes that are essential to EIA's Continuity of Operations.
  

• Creation and maintenance of Process Performance Manuals (may also be called User Manuals, User Guides, Operations Manuals, Process Manuals, Instruction Manuals, etc.) for all business processes designated as essential.
  
• A Process Performance Manual (or its equivalent) will contain the following information:
  1. A narrative description of the process, its purpose, operating environment, components, information inputs (and sources) and information outputs (and destinations);
  2. An overview chart showing the connectivities of the process components among themselves and with outside entities;
  3. A set of instructions suitable for an untrained but skilled individual that explains how each process component is performed;
  4. An optional troubleshooting guide and set of frequently-asked-questions that provide help to those encountering difficulty in performing the process; and
  5. Names of individuals to contact who can provide assistance in the performance of the process.

Related Information:
1. EIA Data Systems Standards, 2002-2, 2002-3
2. EIA Models Standards, 2002-26, 2002-27, and 2002-28

Approval Date: September 26, 2002