AMERICAN STATISTICAL ASSOCIATION COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY INFORMATION ADMINISTRATION Thursday, April 6, 2006 Washington, D.C. 2 1 AMERICAN STATISTICAL ASSOCIATION MEMBERS: 2 NICOLAS HENGARTNER, Ph.D., Chair 3 NAGARAJ K. NEERCHAL, Ph.D., Vice Chair 4 DEREK BINGHAM 5 JOHNNY BLAIR 6 MARK BURTON, Ph.D. 7 CUTLER CLEVELAND 8 JAE EDMONDS, Ph.D. 9 MOSHE FEDER, D.Sc. 10 BARBARA FORSYTH 11 WALTER HILL 12 NEHA KHANNA 13 THOMAS RUTHERFORD 14 SUSAN M. SEREIKA 15 DARIUS SINGPURWALLA 16 PARTICIPANTS: 17 BOB ADLER, Energy Information Administration 18 MARGOT ANDERSON, Energy Information Administration 19 JUSTINE BARDEN, Energy Information Administration 20 BETTY BARLOW, EI-13 21 JAKE BOURNAZIAN, Energy Information 22 Administration BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 3 1 PARTICIPANTS (Cont'd): 2 SUSAN BUCCI, Energy Information Administration 3 GUY CARUSO, Energy Information Administration 4 JOHN CONTI, Office of Integrated Analysis and Forecasting 5 AUDREY CORLEY, EI-45 6 JOHN PAUL DELEY, Energy Information 7 Administration 8 STAN FREEDMAN, Energy Information Administration 9 DWIGHT FRENCH, Energy Information Administration 10 WILL GIFFORD, Energy Information Administration 11 HOWARD GRUENSPECHT, Energy Information Administration 12 VICKI HAITOT, Energy Information Administration 13 BILL HELKIE, Energy Information Administration 14 SHANNON HOWELL, Energy Information Administration 15 ALETHIA JENNINGS, Energy Information 16 Administration 17 STAN KAPLAN 18 NANCY KIRKENDALL, Energy Information Administration 19 TOM LORENZ, Energy Information Administration 20 PAUL McARDLE, EI-01 21 TINA PEARSON, Energy Information Administration 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 4 1 PARTICIPANTS (Cont'd): 2 MARK SCHIPPER, Energy Information Administration 3 SCOTT SITZER, Energy Information Administration 4 BOB SCHNAPP 5 JOHN STAUB, Office of Integrated Analysis and Forecasting 6 PHILLIP TSENG, Ph.D., Energy Information 7 Administration 8 BILL WATSON, CNEAF 9 SHAWNA WAUGH, Energy Information Administration 10 PAULA WEIR, Energy Information Administration 11 WILLIAM WIENIG 12 JOHN WOOD, Energy Information Administration 13 14 * * * * * 15 16 17 18 19 20 21 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 5 1 C O N T E N T S 2 Open Meeting, Nicolas Hengartner Chair, 7 ASA Committee on Energy Statistics 3 Greeting and Remarks, Guy Caruso, 14 4 Administrator, Energy Information Administration, EIA 5 Updates for the Committee Since the Fall 29 6 2005 Meeting, Nancy Kirkendall, Director, Statistics and Methods Group (SMG), EIA 7 How Can Modeling Suggest Data Needs? Open 40 8 Discussion. Nancy Kirkendall, SMG Chair, Margot Anderson, and other invited EIA 9 senior management 10 2006 MECS: Looking at Past Performance 89 Statistics to motivate New Methods of 11 Collection. Bob Adler and Tom Lorenz, EMEU, EIA. 12 ASA discussant on 2006 MECS past 155 13 performance: M. Feder 14 ASA discussant on New Oil Production SAGE 162 Model: N. Khanna 15 Additional Committee Discussion 168 16 Invitation for Public Comments, Nicolas 184 17 Hengartner, Chair, ASA 18 Measuring Perceptions of Applying 185 Alternative Disclosure Limitation 19 Methods, Jake Bournazian, SMG, EIA. 20 Functional Requirements for EIA's Internet 235 Data Collection System, Stan Freedman, 21 SMG, EIA. 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 6 1 CONTENTS (Cont'd): 2 ASA Discussant on Functional Requirements 304 for EIA's Internet Data Collection System, 3 D. Singpurwalla 4 ASA Discussant on Improving the SAGE 314 Petroleum Refinery Model: T. Rutherford 5 Additional Committee Discussion 323 6 Making Adjustments to Survey Data 330 7 When the Collected Data Do Not Meet Expectations, Stan Kaplan, CNEAF, EIA 8 Invitation for Public Comments 388 9 Adjournment of Thursday meeting 388 10 11 12 * * * * * 13 14 15 16 17 18 19 20 21 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 7 1 P R O C E E D I N G S 2 (8:39 a.m.) 3 DR. HENGARTNER: Oyez, oyez, oyez. 4 Ladies and gentlemen, thank you very much for 5 your attention. Since we're already 10 6 minutes late, let's start. 7 I'd like to welcome you all to 8 another meeting of the American Statistical 9 Association Advisory Committee on Energy 10 Statistics, and again, I want to emphasize 11 this is an ASA meeting and not an EIA 12 committee, and we meet periodically for 13 providing input and advice to the EIA. 14 This is an open meeting, open to 15 the public, and public comments, of course, 16 are very welcome, either in form of oral 17 comments -- and time we'll set aside for this 18 at the end of today and at the end of 19 Friday's sessions -- as well as written 20 comments that can be sent either to the EIA 21 or to the ASA. 22 I'd like to remind that all BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 8 1 attendees should hopefully sign in at the 2 entrance -- and if you're here you've 3 probably done that -- and also provide e-mail 4 address for whatever reason. 5 Restrooms: Restrooms are most 6 likely on that side, way over there, and 7 probably on the other side, way over there. 8 There is an open line for phone. 9 If you want to have someone call in and leave 10 messages, let me give you the phone number. 11 The area code is (202) 586-0832; the other 12 phone number where they may call here and 13 leave messages during our stay at the meeting 14 is, again, (202) 586-0946. 15 So for the committee member, you 16 know Kathleen Wert and Donna Arrington, they 17 are from the ASA, they will help you with 18 travel reimbursement and other 19 technicalities, so please see them if you 20 have any questions. 21 I'd like to remind all the speakers 22 and all the people commenting to please speak BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 9 1 clearly and loudly into the microphone. This 2 will be highly appreciated by the gentleman 3 here who is making transcripts of our 4 proceedings. And finally, before we start, 5 I'd like to go around the table, and actually 6 the entire room, so that everybody can 7 present themselves, and we'll take it from 8 there afterwards. So let me start. My name 9 is Nick Hengartner; I'm from Los Alamos, and 10 I'm chair of the ASA Committee on Energy 11 Statistics. 12 MS. KIRKENDALL: I'm Nancy 13 Kirkendall, the director of the Statistics 14 and Methods Group in Energy Information 15 Administration. 16 MR. CARUSO: Guy Caruso, 17 administrator, Energy Information 18 Administration. 19 MR. RUTHERFORD: Tom Rutherford, 20 I'm a professional economist. 21 DR. BURTON: Mark Burton, economist 22 from the University of Tennessee. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 10 1 MR. SINGPURWALLA: Darius 2 Singpurwalla, Law, Economic, and Consulting 3 Group. 4 MS. MILLER: Renée Miller, EIA. 5 MR. BOURNAZIAN: Jake Bournazian, 6 SMG and EIA. 7 MR. RASMUSSEN: Erik Rasmussen for 8 the administrator, EIA. 9 MS. LEE: Audrey Lee at DOE. 10 DR. TSENG: Phillip Tseng, SMG, 11 EIA. 12 MS. JENNINGS: Alethea Jennings, 13 SMG, EIA. 14 MR. STAUB: John Staub, EIA. 15 MR. SWEETNAM: Glen Sweetnam, in 16 OIF in EIA. 17 MR. CONTI: John Conti, OIF. 18 MS. KHANNA: Neha Khanna, Economics 19 and Environmental Studies, Binghamton 20 University. 21 MR. CLEVELAND: Cutler Cleveland, 22 Center for Energy Environmental Studies, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 11 1 Boston University. 2 MR. BINGHAM: Derek Bingham, 3 Department of Statistics, Simon Fraser 4 University. 5 DR. NEERCHAL: Nagaraj Neerchal, 6 Department of Math and Statistics at UMBC. 7 Also the vice chair of it. 8 MR. HELKIE: Bill Helkie, EIA. 9 MR. WATSON: Bill Watson, EIA. 10 MR. LORENZ: Tom Lorenz, EMEU, EIA. 11 MS. HOWELL: Shannon Howell, EIA. 12 MR. FENNELL: Dean Fennell, CNEAF, 13 EIA. 14 MS. FRENCH: Carol French, Office 15 of Oil and Gas, EIA. 16 MR. DELEY: John Paul Deley, NEIC, 17 EIA. 18 MR. LU: Ruey-Pyng Lu, EIA. 19 MR. DISBROW: Jim Disbrow, OIT, 20 EIA. 21 MS. HOJJATI: Behjat Hojjati, EIA. 22 MR. FAWZI: Aloulou Fawzi, OIF, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 12 1 EIA. 2 MS. HOLT: Susan Holt, Office of 3 the Administrator, EIA. 4 MR. BRADSHER-FREDERICK: Howard 5 Bradsher-Frederick, EIA and SMG. 6 MR. STROUD: Ron Stroud with SMG, 7 EIA. 8 MR. WOOD: John Wood, NRPD, EIA. 9 MS. WAUGH: Shawna Waugh, 10 Statistics and Methods Group. 11 DR. HENGARTNER: Thank you very 12 much. Let me continue. First of all, I'd 13 like to remind you this is a DOE meeting, and 14 it's under the DOE terms; that means that 15 Nancy Kirkendall here is the designated 16 official, and she has the right to interrupt, 17 adjourn, or take over the meeting as she sees 18 fit. 19 MS. KIRKENDALL: Boy, talk about 20 power. 21 DR. HENGARTNER: The other thing 22 I'd like to tell you is I think this is going BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 13 1 to be a very interesting meeting. Before we 2 start the meeting, I'd like to welcome Derek 3 Bingham; he's from Simon Fraser University, 4 he's essentially their subs for Randy Sitter, 5 so that's big shoes to fill, but I know Derek 6 will do a great job at this. 7 The first session, we're going to 8 have the pleasure of hearing the EIA 9 director, Mr. Guy Caruso, followed by Nancy 10 Kirkendall's overview of the Statistics and 11 Methods Group, and there are going to be two 12 sessions in the morning, I believe, and then 13 lunch -- very important, never forget lunch. 14 And lunch is going to be in room -- the 15 first-floor room which we've been all -- if 16 you don't know where it is, follow the crowd, 17 that's where it is. And tonight, we're going 18 to have dinner at the Marrakesh, which is a 19 Moroccan restaurant; it's a seven-course 20 meal, I believe, and so I'd like a show of 21 hands to see who might be wanting to join us 22 there for tonight so that we can make BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 14 1 appropriate reservations. So -- 2 MS. KIRKENDALL: I have two. 3 DR. HENGARTNER: Two, so -- seven. 4 Are you coming for dinner, Barbara? 5 MS. FORSYTH: Not tonight. I can 6 barely make it for breakfast. 7 (Laughter) 8 DR. HENGARTNER: Do you want to 9 introduce yourself? 10 MS. FORSYTH: Sorry to be late. 11 I'm Barb Forsyth. 12 DR. HENGARTNER: So I think that's 13 all there is. So tomorrow, reminder, we're 14 going to resume at 8:30, but I remind you 15 about those technicalities at the end of the 16 day. So without further due process, it's a 17 pleasure to recognize Mr. Guy Caruso, the 18 administrator of the Energy Information 19 Agency. Thank you very much. 20 MR. CARUSO: Thank you, Nick, and 21 good morning everyone. I want to make, 22 again, a special thanks to all of the ASA BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 15 1 committee members, and we appreciate your 2 taking the time and having this meeting to 3 advise us twice a year and as I've said many 4 times in previous meetings how much we really 5 appreciate your advice and how useful it is. 6 And I know you see the results of it -- I 7 hope -- from meeting to meeting and on a very 8 practical way, whether it's frames of our 9 surveys, to the website improvements, to 10 methodological changes in modeling, all of 11 these have benefited by the counsel and 12 advice of this committee. So thank you very 13 much, and thank you, Nick, for chairing the 14 session, and I'd look forward to, again, 15 having a good two days of discussions. 16 As always, in that six months' 17 interval between meetings, the EIA has been 18 very busy with a large number of activities, 19 whether they be our outlooks or new surveys, 20 improvements in old surveys, et cetera, all 21 of which we'll talk about today. 22 The usual Annual Energy Outlook, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 16 1 going out to 2030 this year, was released in 2 its entirety in February, and it's available, 3 of course, on the website, and I think when 4 one looks at what's the main highlights of 5 the AEO '06, this year, clearly, the 6 reassessment of the oil price outlook was by 7 far the biggest change and has implications, 8 of course, for interfuel competition as well 9 as the oil market outlook, and -- 10 One of the things when we make 11 reassessments -- in this case, oil prices -- 12 and indeed, even natural gas prices are up 13 considerably in this outlook -- it stretches 14 the NEMS, the National Energy Modeling System 15 that many of you are familiar with. And 16 clearly, this is a case where, this year, 17 looking at much higher prices, both for oil 18 and natural gas, and the implications for not 19 only oil but the other sectors besides 20 transportation, such as electric power and 21 the implications for coal -- all brought to 22 bear some significant methodological BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 17 1 challenges, and I think some of them will be 2 discussed today, but I think, more 3 importantly, we'll be looking hard at those 4 of how the model responds to these new 5 assumptions, and, as I'll mention when I talk 6 about the budget, we're hoping to launch a 7 serious upgrade of our National Energy 8 Modeling System, beginning in the second half 9 of '06, but continuing, we think, if money is 10 appropriated, for the next several years. 11 So, again, that was the big change, 12 and you could see it here, about a 13 $20-per-barrel increase over the long-range 14 assumptions here through 2030, going from 15 roughly $30 a barrel to roughly $50 a barrel 16 on average. And part of it is the investment 17 side of things, particularly from OPEC 18 countries, but also the recognition that the 19 cost of doing business in the upstream in the 20 oil sector is increasing with the commodities 21 boom, and the cost of doing just about 22 everything from drilling through the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 18 1 downstream and refining and distribution. 2 All have contributed to that. We are still 3 not among the camp that says that world oil 4 is peaking and declining in the near term and 5 therefore that's what's driving prices. 6 That's not what we're saying; we're saying 7 that the issues driving prices are mainly 8 above the ground, not below the ground, and I 9 know John Wood is here, and he'll talk about 10 mainly the new survey for crude, but 11 certainly John and his people have been very 12 firm in their belief that there's still lots 13 more oil to be discovered and more productive 14 capacity to be added over the next 20 to 30 15 years. 16 Of course, short term tends to get 17 a lot more attention, and our latest 18 short-term issue, as we head to the driving 19 season this year, is the transition in 20 reformulated gasoline composition by the 21 decision being made by almost all refiners to 22 remove MTBE -- methyl tertiary butyl ether -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 19 1 from reformulated gasoline as the main 2 oxygenate and octane booster and replacing it 3 with ethanol for the most part. That 4 transition is underway, and our view is that 5 it could be a bit of a rocky road to make 6 this transition, and that could lead to some 7 supply dislocations in certain regions of the 8 U.S., and it's gotten some attention both in 9 Congress and in the administration. And of 10 course, we've all witnessed some run-up in 11 gasoline prices already this spring; only 12 part of that, we believe, is part of this 13 transition, but nevertheless, it's an issue 14 that EIA is deeply involved in and raised the 15 flag early on, in January, by a paper that 16 was done by Joanne Shore in our oil and 17 natural gas office. 18 Again, as I mentioned, we continue 19 to look for ways to improve the collection of 20 data, in this case, on the upstream crude oil 21 collection. As you know, we've started 22 collecting natural gas production data, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 20 1 heavily relying on the advice of this 2 committee in developing that form, and John 3 Wood's here to talk about the 914, the 4 expansion of the data there, but we are 5 considering expanding that in 914 form to 6 include the collection of crude oil 7 production and very much appreciate hearing 8 your advice on that. 9 One of the things that's really 10 improved the productivity of collecting data 11 -- and the quality, we believe -- is the 12 Internet data collection systems that you've 13 heard about previously from the various -- 14 particular from our coal and natural -- in 15 nuclear and electricity -- CNEAF office. 16 That's going along very well. Several IDCs 17 were launched in the last year or so, and 18 they're operating pretty well, we think, and 19 definitely have improved the speed, the 20 timeliness, and accuracy of the responses 21 because so much of the editing is done at the 22 respondent's side of things and therefore BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 21 1 there's less follow-up and it allows us to do 2 other things, to work on the quality and the 3 distribution of the data. So that's going 4 very well and appreciate very much all of 5 your help in giving us advice on that. 6 We've -- after a long process, the 7 voluntary reporting of greenhouse gas 8 programs -- guidelines have been revised, and 9 we're about to begin the redesign of that 10 program, and we expect that we'd start around 11 the first of June and would be fully 12 implemented during 2007. This has gotten a 13 lot of attention with respect to the issues 14 of climate change, and EIA's playing a very 15 important role in -- and will continue to 16 play an important role in the office that is 17 headed by John Conti, OIF. 18 As many of you knew from being 19 participants in the improvement of our 20 website, we did launch the new site in 21 November. We've gotten substantial positive 22 feedback on this redesign, and continue to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 22 1 work on improvements and certainly, again, 2 welcome any advice you would have on the new 3 website. It's a work continually in 4 progress, as you're aware, and we're 5 currently doing a customer survey on the 6 kids' page, which is one of the most popular 7 components of our website. And on the 8 personnel side, just this week, Monday, a new 9 Web champion, Gina Pearson, has come on 10 board. She'll be the director of our 11 National Energy Information Center, NEIC, 12 succeeding John Weiner, who many of you know 13 from his many years as heading up NEIC. 14 Among Gina's many talents is her experience 15 and success in redesigning websites. She's 16 most recently come from the Energy Research 17 Service, where she completely redesigned -- 18 MS. KIRKENDALL: Economic. 19 MR. CARUSO: Economic Service? 20 MR. KIRKENDALL: Economic Research 21 Service. 22 MR. CARUSO: Economic Research BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 23 1 Service. It's part of the Department of 2 Agriculture, right? Yeah. And she's done a 3 outstanding job there and at Department of 4 Education, and we're looking forward to even 5 greater things as our Web evolves. 6 I mentioned one component of our 7 budget request for 2007, which is currently 8 being discussed on Capitol Hill. That is the 9 request for an increase of several million 10 dollars, which partly will be used for this 11 upgrading of the National Energy Modeling 12 System, and we look forward to hopefully 13 getting that money appropriated and working 14 with you all when this gets more to where we 15 move from a planning to implementation phase 16 and getting your advice on that. 17 This year, we have a budget of a 18 bit more than $85 million; it's a small 19 increase from '05, and enables us to pay for 20 about 370 federal employees and a number of 21 contractors. We have, as I mentioned, asked 22 for about $4.5 million increase for '07, with BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 24 1 a small increase in FTEs to 375, but more 2 importantly, the increase would go to, as I 3 said, upgrading the NEMS as well as some 4 other areas. 5 DOE is now moving toward a 6 five-year budget cycle, and we're certainly 7 participating in that, and, as I mentioned, 8 we think the NEMS upgrade will be a multiyear 9 task. It will require at least three years 10 to go from the planning through actual 11 utilization of that new revised NEMS. And 12 when the current system was designed and put 13 into use, it took about three years. I think 14 it started in '91 and it was '93 before the 15 Annual Energy Outlook was fully utilizing the 16 current NEMS. So clearly we look forward to 17 this five-year cycle that the department is 18 now engaged in, and our targets for 19 timeliness and quality continue to be quite 20 stringent. 21 And the other part of the 22 longer-term -- five years, if you will -- is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 25 1 our continued need for the succession 2 planning and workforce planning that we've 3 talked about before this group previously, 4 and that is with an organization that's now 5 approaching 30 years. Many of our most 6 experienced people are reaching the 7 retirement age, and we're having a fairly 8 substantial turnover. Last year, we lost 16 9 senior EIA employees, and we've been at least 10 successful so far in the recruitment and 11 succession to those 16, but we're looking at 12 probably 20 to 25 senior people leaving in 13 '06 and again in '07. So, again, as I've 14 said in the past, any assistance you can be 15 in terms of recruitment of your students or 16 your associates that you come into contact 17 with, we very much appreciate that because 18 you know the kind of work we do here and the 19 type of people that we need to keep up the 20 tradition of high quality. 21 About one year ago, in response to 22 OMB evaluation of our budget, one of the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 26 1 things they said, and we talked to you about 2 and asked your advice, was no one's ever 3 really done an overview evaluation of EIA. 4 We've seen evaluations of specific surveys, 5 specific components of the program, but what 6 we'd like to see is an independent external 7 review of EIA in its entirety, and I was 8 fortunate to be able to convince Denny 9 Ellerman of MIT to chair a team to do exactly 10 that, look at what it is that EIA does and 11 are we doing the right thing, is the main 12 mission we asked of Denny and his team -- and 13 he went around and recruited four other 14 outstanding individuals who many of you know: 15 Paul Joskow of MIT; Kathy Cooper's a former 16 undersecretary of Commerce and former chief 17 economist of Exxon Mobil; Jay Hakes, my 18 immediate predecessor; and Phil Sharp, who 19 was then at the Kennedy School and now is the 20 director of Resources for the Future and 21 former congressman. They looked very hard at 22 what EIA's original legislation authorization BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 27 1 asked of us 30 years ago and how we've done 2 in meeting that mandate, and, even more 3 importantly, looking out in the future. Are 4 we preparing for what the future's going to 5 bring and the pressures and the requirements 6 on us? 7 They met twice last year, and we 8 expect that we'll receive their full report 9 -- we've gotten a preliminary briefing from 10 Denny last month -- and we expect to have the 11 full report, which we would make available, 12 of course, to you, and we'd hope to have a 13 full discussion of what they have found in 14 their review, and, more importantly, what 15 they are recommending for the future for EIA, 16 including areas where we may want to place 17 more emphasis and areas where we may want to 18 place less emphasis and indeed, even to maybe 19 even stop doing some things. So we found it 20 to be an outstanding opportunity to actually 21 almost do a self-evaluation in order to 22 prepare for the external evaluation. As you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 28 1 had so wisely recommended, we approached this 2 kind of like a visiting scholar's approach 3 that many of your universities use, and I 4 think that was a very wise advice on your 5 part because we found it to be very useful. 6 As I mentioned, we are approaching 7 our 30th year, having come into being as the 8 Energy Information Administration in 1977, 9 and so we're looking forward to having a 10 celebration some time. Probably it'll be a 11 rolling celebration throughout '07 but we'll 12 focus it mainly on the educational aspects 13 that we think we can play an important role 14 in the energy education of not only the 15 decision makers, but the public mainly 16 through our website. 17 So we'll be asking for your ideas 18 and input as well as we plan on this, so, 19 again, let me close by once again thanking 20 you, Nick, and all the members of the 21 committee for your past advice and especially 22 for the advice we'll be getting today, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 29 1 tomorrow, and in the future. So if there are 2 any questions, or anybody like to go into any 3 more detail on any things I've mentioned, I'd 4 be happy to do that now or any time during 5 the next couple days. 6 Okay, thanks. 7 DR. HENGARTNER: Thank you very 8 much. 9 (Applause) 10 DR. HENGARTNER: I'd like now to 11 welcome Nancy Kirkendall to give us a review 12 of what the Statistics and Methods Group has 13 been up to in the last six months. 14 MS. KIRKENDALL: What I'm actually 15 going to review is what we did in response to 16 some of the committee's suggestions from the 17 sessions we had last time. Sometimes that's 18 the same thing and sometimes it's not. 19 I have picked out a number of the 20 sessions that we had last time, and I thought 21 I'd give you a little update on what we've 22 done or what we haven't done in some cases. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 30 1 So let me start out -- this is the Short-Term 2 Energy Outlook. Last time, Margot Anderson 3 talked to you about her plans to produce a 4 report that would analyze the forecast errors 5 produced from the STEO, the Short-Term Energy 6 Outlook. You made a number of 7 recommendations. One of the things a lot of 8 people said -- and this was more than one of 9 the people in the room -- said that one of 10 the reasons for developing a model is to 11 identify new data needs. And when I went 12 through the suggestions, I thought, okay, 13 well, now what do we do. So we've put that 14 on the agenda to talk about in more detail, 15 and we'd like to get more of your ideas on 16 what we can do with that idea. 17 The other suggestion that I thought 18 was interesting was to separate the model 19 into two parts, or the modeling errors into 20 two parts: The error in the modeling and the 21 error in the input variables. And that's 22 something that I still think is really BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 31 1 interesting and useful. We haven't had a 2 chance to do that yet. However, Margot's 3 group has produced a draft of a forecast 4 error report. They've done analysis of 5 forecast errors for one-, six-, and 6 twelve-month forecast horizons, and that is 7 currently out for review, so hopefully we'll 8 have something to share with you next time to 9 get more input on it. 10 Vehicle energy use. Mark Schipper, 11 also from the Office of Energy Markets and 12 End Use, gave a talk about the work they've 13 been doing with the Department of 14 Transportation. This is basically to fill a 15 data gap caused by losing the residential 16 transportation energy consumption survey. 17 And so the committee suggested that they 18 liked the modeled data and you also 19 recommended that we continue collaborating 20 with the Department of Transportation, and of 21 course we're planning on doing that. And 22 since the last meeting we have offered a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 32 1 public-use version of a report from the 2 modeled data, so there's been further work on 3 that. 4 There were two sessions by Renée 5 Miller of the Statistics and Methods Group 6 last time. One of them talked about some 7 data quality initiatives, how to come up with 8 a nice little product that would illustrate, 9 perhaps, a different data series to try to 10 measure the same thing and use it as an 11 educational tool. The other one talked about 12 discrepant estimates. Any time you have a 13 survey to measure something, if you have two 14 surveys, you get two different numbers. 15 Sometimes this is useful and sometimes it's 16 mystifying. So we talked to you about that. 17 So this is a summary of your advice. One of 18 them -- and Mark Bernstein told us this all 19 regularly through his time on the committee 20 -- and that is that in all of our 21 presentations, we're supposed to state our 22 objectives clearly. Hopefully, this time, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 33 1 you'll see the objectives more clearly in the 2 presentations, but we'll see. But in terms 3 of advice on this particular issue, you 4 suggested that we need to focus on the 5 biggest data quality problems first, which 6 makes a lot of sense -- sometimes it's not 7 easy to figure out which those are -- to 8 measure data quality, do an internal report, 9 and then develop a Web product after that. 10 So the purpose is to evaluate data quality, 11 so the advice was to have a more systematic 12 approach that would lead to possibly a Web 13 publication, but maybe not in some cases. 14 The other thing that you have 15 strongly recommended that we do is to inform 16 users about data quality issues, and on that 17 one, we know we need to do that, but we're 18 not clear quite how to yet. The issue has 19 been raised with the interoffice issue group 20 that Renée Miller chairs and Howard 21 Gruenspecht is a member of that. So we'll 22 still be debating how to do that. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 34 1 We talked about -- we've looked a 2 little bit at looking at performance measures 3 to try to find some measures that would be an 4 indicator of data quality problems we should 5 look at. This is just ongoing work. Renée's 6 group is going to take on, in -- sometime in 7 the '80s or '90s there was an evaluation -- a 8 comparison of data from the Residential 9 Energy Consumption Survey and data that we 10 get on our supply surveys. And naturally 11 enough, they didn't match, and we tried to 12 explain why. So that report was done -- it 13 was a good report, but it was done a long 14 time ago, and they're going to update it. So 15 it's a good model to use. We need to look at 16 different data -- at the data sources and see 17 what we can learn from that. So that -- 18 we'll work on that beginning sometime this 19 spring or summer. 20 And then the last one I already 21 mentioned. The survey self-assessment 22 process -- this is Tom Broene's work -- and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 35 1 it was a big undertaking. Tom interviewed 2 all of the survey managers in EIA to try to 3 help fill out a little evaluation form, and 4 unfortunately it turns out to be a huge 5 burden to do that. And you obviously 6 recognized that and have suggested that we 7 need to be more flexible in how we develop it 8 and how we implement it because we need to 9 have a process we can actually do on a 10 regular basis. So that's sort of a summary 11 of what your advice was. 12 We are reconsidering the process, 13 and the only thing that we think we'll keep 14 from the old process is the -- we ask for 15 survey-specific targets for improvement, and 16 I think we still would like to do that. At 17 least it tries to focus each survey on what 18 they need to do -- or it could be survey 19 system -- flexibility idea. 20 This was an analysis between -- I 21 think they were trying -- it was Bill Trapman 22 and Leila from the Office of Oil and Gas -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 36 1 it was a modeling exercise trying to look for 2 predictive relationships between futures 3 prices and Henry Hub prices. And I think the 4 committee was supportive of the analysis and 5 didn't have particular recommendations for 6 what to do different, but they thought with 7 that particular analysis -- you did advise 8 trying to incorporate NYMEX prices into STEO 9 for predictive purposes. And that's sort of 10 interesting because I think that's one of the 11 things the external review team has also 12 talked about, is that EIA should make better 13 use of some of the public data, like the 14 NYMEX prices. So that's kind of interesting 15 when advice kind of comes together. Probably 16 means that we need to be serious about taking 17 it. 18 And then finally, last time we 19 reported to you on the frames evaluation that 20 was done between -- it was a cooperative 21 relationship between the Census Bureau and 22 EIA. It was a good effort. You commented BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 37 1 that we had a pretty high coverage on these 2 surveys -- 75 to 100 percent by volume -- and 3 that was especially true, given there were a 4 lot of problems in matching. The name and 5 address -- all we had to match was name and 6 address lists, and they were not exactly the 7 same. They weren't formatted the same and so 8 there was a level of detail that was missing. 9 You suggested collecting EIN numbers to help 10 in matching; I think we are considering that, 11 and we'll see how that plays out. And you 12 suggested that we should use a cost-benefit 13 approach in deciding whether to try to fix 14 some of these frames. I think that means 15 work on the ones that are important and don't 16 worry so much about the others. And so of 17 course we don't have as many resources as we 18 might want for this sort of thing, so we're 19 definitely doing that. Right now, CNEAF has 20 identified a different data gap related to 21 their frame, and that's what they're working 22 on. They've discovered that we're missing BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 38 1 some production of waste coal, and they're 2 trying to add that on the production side, 3 which is not the surveys that we're doing 4 here. 5 But another comment on this 6 evaluation, tomorrow after the break we're 7 going to be giving a little certificate of 8 appreciation to the people from the Census 9 Bureau who helped us on this project and who 10 gave presentations to you. So that was 11 really good. 12 Well, that's my summary from last 13 fall. If you have any questions, please ask. 14 Okay, well, I think we have an 15 interesting set of topics to talk to you 16 about today, and we look forward to your 17 advice. 18 DR. HENGARTNER: Thank you very 19 much, Nancy. 20 (Applause) 21 DR. HENGARTNER: Before we 22 continue, I'd like to invite the people who BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 39 1 trickled in after the first set of 2 introductions to introduce themselves to the 3 microphone so that you can be on the record 4 as being present. Moshe, do you want to 5 start? 6 DR. FEDER: My name is Moshe Feder, 7 I'm a statistician from the Research Triangle 8 Institute. 9 MR. HILL: Walter Hill, St. Mary's 10 College of Maryland, Department of Political 11 Science. 12 MR. FREEDMAN: Stan Freedman, EIA. 13 MR. ZHANG: Bin Zhang, EIA. 14 MS. BLUMBERG: Carol Blumberg of 15 Winona State University, and I'm actually 16 interviewing for a job here today. 17 (Laughter) 18 MR. BROENE: Tom Broene, EIA. 19 MS. ANDERSON: Margot Anderson, 20 EIA. 21 MS. BARDEN: Justine Barden, EIA. 22 MS. WEIR: Paula Weir, EIA. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 40 1 MR. WEINIG: Would you step forward 2 and use one of the microphones, please? 3 MR. BENNECHE: Joe Benneche, EIA. 4 DR. HENGARTNER: Okay, thank you 5 very much. I'd like to continue -- we're on 6 schedule, fortunately, with a -- I think it's 7 an open discussion, is that correct, by -- 8 how can we model suggested data needs, and 9 it's going to be essentially led by Margot 10 and Nancy and other invited EIA senior 11 management. 12 MS. KIRKENDALL: There is a summary 13 of the comments you made. I think it's under 14 item one in your notebooks, and all I did was 15 pull out some of the quotes that I thought 16 were appropriate for this discussion. EIA is 17 really fortunate that we are both users -- 18 producers of data and we use our own data, 19 and you would hope that we could do a good 20 job with a feedback relationship between the 21 users and the producers of the data. That 22 would help improve both sides of the issue. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 41 1 However, we haven't ever managed to figure 2 out how to do that, and so we would be 3 interested in your advice on how the modeling 4 and data and have a good feedback -- how do 5 you learn from your modeling what your data 6 needs are? Any ideas on what kind of a 7 feedback mechanism might work, if you've seen 8 anything like that in the other places. 9 MS. KHANNA: I have a very 10 preliminary question -- this could be because 11 I wasn't here the last time -- which without 12 language and models, how do you figure out 13 what your -----? How does the EIA even start 14 that process? 15 MS. ANDERSON: Well, we already -- 16 in most cases, we're not starting up with any 17 new models. We basically have two or three 18 models that we use for -- do I need -- 19 MR. WEINIG: I couldn't hear. I'm 20 not sure anyone here heard the opening 21 remarks. 22 MS. KHANNA: Sorry. Okay, I hope BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 42 1 you can hear this now. My basic question 2 was, without knowing a model, how do you even 3 begin to assess your data needs? And so for 4 me the starting point always is here is what 5 I'm trying to model, and that almost always 6 drives my data needs. So I'm a little 7 surprised by the question in the first place. 8 MS. ANDERSON: I think we were 9 looking at it from the aspect of having 10 existing models and how do you enhance 11 existing models and identify the data gap 12 that you might fill by collecting new data or 13 doing new surveys or collecting already 14 commercially available data. We don't have a 15 lot of models in EIA; we tend to have a 16 couple of forecasting models, a short-term 17 model, some long-term forecasting models, but 18 for the most cases, we're not building new 19 modeling frameworks. So sort of the question 20 that first comes out of is, we have these 21 data programs first and the modeling came 22 somewhat after having the data programs BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 43 1 intact. So the data programs were -- what 2 kind of data do we have that can support the 3 modeling that we would like to do, the 4 short-term and the longer-term forecasting. 5 And it's opposed to the other way around of 6 starting a model from fresh and then 7 determining what data. John might have some 8 ideas on if we go through this process, when 9 we go through this process, of redoing the 10 national energy model, and it certainly might 11 engender some new questions that we might 12 want to either collect some data or rejigger 13 some data that we're already collecting or 14 use a portion of data more frequently. But 15 right now we're not building any new models 16 that would require us to reevaluate the data 17 programs, we're trying to get a sense of how 18 do we use the models that we have to ask 19 ourselves what data might improve the 20 predictive capability of those models. I 21 think. That's the question I'm asking -- or 22 at least -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 44 1 MR. CONTI: Let me maybe expand on 2 that a little bit in getting back to the 3 question. First of all, I think once upon a 4 time, when people were trying to understand 5 the energy industry, it wasn't what data do 6 we need for the models, it was what analytic 7 information did we need to have in order to 8 understand the energy questions that were 9 being asked of us. And so EIA created data 10 programs to provide those answers. As the 11 questions became more complex, they developed 12 models to help them do the analytics to 13 answer the questions. 14 Now we ask the question, well, what 15 does the model tell you in terms of data 16 requirements. Well, the model often tells us 17 that we are missing key data that we'd like 18 to have. And we've identified areas in the 19 past where this data is missing. For 20 example, in the ethanol area, we sat down 21 with people, both in DOE and other parts of 22 EIA, and identified -- we'd like to have this BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 45 1 data, and they said, "Well, thank you very 2 much, we'd like to have it, too. We can't 3 afford to give you that data." As a matter 4 of fact, just a couple of weeks ago, EIA 5 announced it was eliminating two data forms, 6 one of which helps us understand crude 7 quality and crude prices, something that's 8 probably of analytic interest in the future. 9 It's not the fact that you identify a need 10 with the model and then, okay, we just run 11 out and get the data, you still have to have 12 a budget to be able to get that data. So -- 13 that's where I'll stop. 14 I'm John Conti, director of the 15 Office of Integrated Analysis and 16 Forecasting. 17 MS. KIRKENDALL: The other comment 18 was a lot of our data series -- EIA was 19 organized and established in '78, and there 20 was a predecessor organization, too. So our 21 data collections have been around for a long 22 time, and we tinker with them. You add BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 46 1 products, you subtract products, you find a 2 few new surveys you need to do. But a lot of 3 the data programs have been around for a long 4 time. 5 MS. ANDERSON: And never intended 6 to be necessarily used in a modeling context. 7 They were almost ends of themselves because 8 people either gave a series or gave 9 information on series that weren't really 10 asked to do -- look at these relations -- 11 MS. KIRKENDALL: Well, except that 12 you were providing the data for people who 13 want to do that kind of analysis. That's 14 what our data program's for; it's supposed to 15 be -- doesn't have to be for us. And so it's 16 supposed to support analysis that's outside 17 the EIA, too. 18 MS. ANDERSON: Correct. 19 MS. KIRKENDALL: It's just that we 20 do have an opportunity to learn more about 21 what data are needed from our own internal 22 uses and modeling. So that's sort of -- how BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 47 1 do we do that a little better. And John 2 talked about the ethanol usage -- is there 3 another way of -- and that's going to be 4 coming around and we'll probably end up doing 5 something about that. 6 MR. CLEVELAND: Well, I'm glad to 7 hear somebody say that, but as far as I 8 understand it, I think key staff people have 9 identified data needs in the past then it 10 really doesn't get past that, identifying the 11 need. I don't think the problem is 12 identifying the need, I think the problem is 13 figuring out how could we meet that need 14 within existing budgets. 15 MR. KIRKENDALL: Or get that need 16 moved up to a higher-level discussion when 17 we're talking about allocation of scarce 18 resources. At least at that discussion point 19 we know what's on the table for data needs. 20 You might not be able to do anything, but at 21 least it gets into the discussion. 22 MR. CLEVELAND: I guess one of my BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 48 1 concerns is to what extent the questions that 2 you're even asking have been constrained by 3 the historical data collection and modeling 4 process, and I'm wondering -- and I'm sure 5 that your external review panel is asking if 6 we had to blow it up and start all over -- 7 which I'm not proposing to do -- what 8 questions would we be asking and what models 9 would we be trying to build to answer those 10 new important questions and, in turn, what 11 data -- what do we need to do with that, 12 rather than approaching it from the other 13 way. 14 And a secondary question is, to 15 what extent is the demand for data by people 16 outside of the EIA, like me, in terms of 17 determining what you do and don't collect, 18 and how do you balance that against your own 19 internal needs? 20 MS. KIRKENDALL: So a question is 21 what's a good way for getting input from you, 22 for example, on what you think the data needs BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 49 1 are -- and other external users. 2 MR. CLEVELAND: Yeah, except when I 3 hear the budget situation, it's hard to me to 4 even conceive that our needs would be that 5 high on the list. But it seems like 6 something you should be doing. 7 MS. ANDERSON: It kind of goes back 8 to this issue of defining your objectives. 9 If you don't know why you're doing something, 10 whether it's to create a structural model or 11 a forecasting model, you're not clear what 12 the questions are that you're trying to 13 answer, you're going to fail on trying to 14 determine what data must you collect and who 15 do you need to go ask about your data. 16 So it goes back to the advice that 17 we've been given and we try to take, but 18 you've got to define your objectives because 19 we can't do everything. And it's not such an 20 easy exercise to define your objectives in an 21 organization that has such a diverse client 22 base and feels that it is required to answer BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 50 1 or cover all aspects of the energy sector. 2 And we really do try to stay balanced; we 3 really do try to have universal coverage, and 4 regret always when something has to go away 5 and there are multiple objectives that 6 sometimes work at cross-purposes. 7 But I think it is really 8 identifying the key objectives, and it is 9 asking ourselves, if we could start all over 10 again, would we do what we have today -- even 11 to ask ourselves that in little microcosms. 12 If we didn't have a short term for our 13 forecasting program, would we create one or 14 would we do something else? And those are -- 15 we really created that form -- I think the 16 external study team has caused all of us to 17 take a long breath and kind of ask some of 18 those fundamental questions. But even if we 19 go through that exercise internally, you run 20 up with the same kind of constraints that 21 John's identifying. 22 How can you communicate that to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 51 1 folks that tend not to live and breathe this 2 and are much more -- adhere to the status 3 quo? We've been getting it all these years, 4 we've been getting this product all these 5 years, change is very jarring. 6 So I don't want to say it's 7 impossible, but you kind of have to tackle 8 these things on the margins a bit. But it 9 does go back to define your objectives: What 10 are we trying to do here at EIA? 11 MR. CONTI: Maybe getting back to 12 the first part of that question, Guy 13 mentioned earlier how we will be developing a 14 National Energy Modeling System, and as we 15 start down this path, one of the first things 16 we'll do is we'll do a requirements analysis. 17 And within this requirements analysis, we'll 18 be speaking to a bunch of our stakeholders to 19 try and understand the questions that they 20 want us to be able to answer in the mid to 21 long term. And then, assessing all of those 22 needs, we'll try and determine which ones are BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 52 1 feasible to address -- and there'll probably 2 be some things we won't be able to address -- 3 and then the ones that we think that are 4 relevant to address and possible to address, 5 we'll do an assessment of, does the data 6 exist, and if the data doesn't exist, we'll 7 do an assessment of, well, how expensive is 8 it to get that data. So in the end, we might 9 not be doing it, but we will have gone 10 through a couple of steps to, does it need to 11 get done, how expensive is it to get done, 12 where can we get the data, and how do you get 13 that into a model. 14 MR. RUTHERFORD: One other comment 15 I'd just add to that is the -- Tom Rutherford 16 -- I think the discussion has been quite -- 17 my comment at the last thing was perhaps a 18 bit of a naïve question because clearly it's 19 kind of obvious, well, duh, that's what we 20 do, right, is we collect data, we do models, 21 we answer questions. So obviously this is 22 part of the process. The question is, to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 53 1 what extent can you formalize that process, 2 right. So it's obvious that you guys are 3 involved in -- I wasn't suggesting by my 4 comment that you weren't going about things 5 this way. 6 MR. CONTI: I think that's 7 important. How do we formalize the need for 8 specific data elements, since I'm the modeler 9 and not the data collector, and where does 10 that fit into the priority list on the data 11 collector's side? 12 MR. RUTHERFORD: Yeah. 13 MR. CONTI: Is it their number one 14 priority? They have a bunch of priority 15 things they want to do, and being able to 16 answer some question from someone they don't 17 know is not necessarily very high on their 18 priority list. 19 MR. RUTHERFORD: The second comment 20 I'd make is that I think the process -- if we 21 think about the cost-benefit -- I think your 22 portraying of the issue as a cost-benefit BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 54 1 tradeoff is right on the money -- but I'd say 2 it's more difficult than solving a linear 3 program to determine this cost-benefit 4 process because clearly the collection of 5 data involves discrete decisions about 6 setting up the apparatus for collecting data, 7 and then the process of just turning the 8 crank to get the data out is probably much 9 less costly. So it's really an integer 10 program you have to solve to determine what 11 to do. In other words, once you have a 12 series going, then the cost of keeping it 13 going is probably much less than the process 14 of deciding what to do. So I'd say it's sort 15 of a difficult issue, but I think that -- 16 just my question really harks back to this 17 idea: We have policy issues which can be 18 solved on the basis of back-of-the-envelope 19 calculations that are informed by data, and 20 there are other more subtle questions that 21 models help tease out which may identify data 22 that aren't obvious at the outset. And so BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 55 1 just being aware of that as part of the 2 process I think is helpful. 3 MS. KHANNA: There's another way of 4 framing the discussion, which is to say are 5 specific elements of the energy market 6 changing in any way fundamentally, and how 7 does that drive our modeling and therefore 8 data -- or data and therefore modeling -- 9 needs. Because I look at the EU and the huge 10 shift in oil prices, which makes a lot of 11 sense to me, but that's a whole different 12 band than we've seen historically. I know 13 you're talking about 30 year band and the $50 14 to $60 price range; what's driving that? 15 That's a fundamental change in the oil market 16 as far as I can tell. 17 MS. ANDERSON: The external study 18 team focused a lot on this, particularly in 19 the area of electricity and electric sector 20 restructuring and whether we really were 21 collecting the data that were consistent with 22 these huge changes in that part of the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 56 1 market, and I think we'll have some 2 recommendations on what we might do to be 3 more up to date in that area. So they 4 certainly were cognizant of these issues, and 5 they also talked a lot about the 6 international factors driving markets, which 7 would certainly come to your point at looking 8 at oil markets in a different way. 9 MS. KHANNA: But it also goes back 10 to John Conti's point about taking the 11 discussion up to a different level. If you 12 can frame it in the sense, look, here is how 13 the market is changing, and we have a whole 14 different set of questions we want answered, 15 it might help you overcome the hurdle which 16 you are perceiving. 17 MS. ANDERSON: I keep harking back 18 to if there are different clients for data 19 series, they may be people like you who are 20 using the raw data into a model you're 21 creating that looks a whole lot different 22 than any models we're doing, and then the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 57 1 client base that is for model output, which 2 is a different sets of clients. And so our 3 data are not only for input for our models, 4 but serve their own purpose, their own end 5 result, and trying to find areas where what 6 we're collecting for that audience is the 7 same as what we're using internally, and 8 that's where you have a really great 9 situation. But there's an awful lot of data 10 we collect that doesn't go directly into a 11 short-term or a long-term model, but is used 12 purely as output for answering perhaps more 13 basic questions but not integrated questions 14 are maybe not used in a modern framework. 15 And is that client any less valuable to us or 16 valuable to EIA than the clients that use the 17 model base that ----- 18 MS. KHANNA: I'd be willing to 19 hazard that the same users that use your raw 20 data for their modeling would be upset if you 21 actually stopped providing some of the series 22 that you are because you're both users of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 58 1 your raw data as well as your model output 2 for the most part. 3 MS. ANDERSON: That's right. 4 DR. BURTON: I'm going to suggest 5 something that probably is close to heresy. 6 The discussion and the decision processes 7 that you're talking about take place in the 8 private sector all the time, but they're 9 guided by pricing. If you're constrained in 10 what you're able to do, you provide the 11 products that people are willing to pay the 12 most for and you observe that. You don't 13 have -- at least given the current structure 14 -- the ability to do that. You don't have 15 the price signals from the users, both 16 internal and external, that a private firm 17 would have, but that's what you're trying to 18 do, is essentially develop a decision process 19 to substitute for what a market would do in a 20 private setting. 21 DR. NAGARA: Could I ask a specific 22 question? Are there some examples where you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 59 1 felt that if you can run this survey instead 2 of once a year, twice a year, that would make 3 a tremendous impact on your model accuracy, 4 for example? I'm wondering whether there's, 5 based on all the models more or less here 6 today, I just wish I had more data on this 7 particular one? I think that in order to 8 answer Nancy's question, to really bring it 9 up to a higher level of discussion, if you 10 have a example of something ---- say, hey, 11 listen to me, you are doing pretty good here, 12 but you can to four times as well by spending 13 just a little bit more. 14 MS. ANDERSON: Well, maybe the best 15 example is on the consumption survey side, 16 that if the Short-Term Energy Outlook had 17 annual consumption data from the Residential 18 Energy Consumption Survey, we would be more 19 accurate in our ability to predict heating 20 expenditures or fuel usage across the year by 21 region. Yes, that's probably an area where 22 we would enhance the accuracy of our BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 60 1 estimates. Again, we get back to -- it's 2 unlikely that we're -- I mean, we're going 3 the opposite direction. We're collecting it 4 in less-frequent intervals ----- some years. 5 Most of the other data series that drive the 6 Short-Term Energy Outlook are collected on a 7 monthly or weekly basis, and we're 8 constrained by the monthly basis for the 9 Short-Term Energy Outlook, and I don't think 10 we want a weekly Short-Term Energy Outlook. 11 Maybe we do, but I keep hoping we don't. 12 (Laughter) 13 DR. NAGARA: Please don't say that. 14 MS. ANDERSON: But certainly there 15 are some data that we've cut back on, 16 definitely on the consumption side, that I 17 think would help both of our modeling 18 efforts, the long-term and the short-term. 19 MS. KIRKENDALL: The only problem 20 with that is that the consumption surveys are 21 among the most expensive ones we run, and 22 that's why the frequency has been reduced. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 61 1 Not that they weren't useful. 2 MR. CLEVELAND: Responding to 3 Mark's comment, which I found interesting, 4 thank God our decisions aren't driven by 5 market considerations, and if they ever start 6 becoming that, you're in deep trouble. Look 7 at the kids' page, the hits on the kids' 8 page, how important -- you're in a really 9 difficult place because energy now, more so 10 than ever, is a huge business, and energy 11 data, billion-dollar decisions, daily, hinge 12 on your information. But energy also is 13 becoming more important in a broader social 14 context. The role of energy in education -- 15 look at the number of hits on the kids' page 16 -- is amazing. Information about climate 17 change and CO2, the increasing role of energy 18 in international markets, which aren't these 19 short-term financial issues, are also 20 increasingly important. And so you really -- 21 I would argue those issues need serious 22 attention as well as the stuff that people BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 62 1 are banging on your door for every day, and 2 that puts you in a -- in one sense, a good 3 position because energy is more important 4 throughout all aspects of society, but in a 5 budget-constrained environment it's a 6 difficult row to hoe. 7 MR. WOOD: I've certainly said in 8 front of this group -- and I was here before 9 the beginning, I think, so that's more than 10 30 years -- that the analytical needs of EIA, 11 as mandated -- those areas where you have a 12 responsibility for -- and anything we can 13 justify a reason for including -- ought to 14 drive the data collection. In practice, over 15 time, sometimes that doesn't -- since a 16 data-collection enterprise may be quite 17 expensive, to change them -- and they always 18 have constituencies, if only internal -- is 19 sometimes difficult. But nevertheless, the 20 analytical needs of the organization and the 21 public for energy data and analysis are what 22 ought to drive the data collection process, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 63 1 and that means periodically something new 2 might have to be added and something old 3 might have to be dropped, although I would 4 caution against dropping a lot of things just 5 because they went down in Web hits during one 6 quarter -- the proverbial drive my company so 7 that the market numbers are hit for this 8 quarter because long-term data series that 9 extend back 50 and 100 and 150 years are 10 really important if you think you're going to 11 be giving insight into questions which may 12 unfold over 25 or 50 or 100 years. 13 Secondly, I'm going to ask a 14 question tomorrow, so I might as well start 15 asking it today, on how in detail you 16 actually make a decision whether to add 17 something or cut something. And in 18 particular, in the 914 survey last year, in 19 large part, potentially to get through the 20 OMB process, we decided that we needed to get 21 natural gas production data for a couple of 22 major producing areas, and so did we want the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 64 1 number one area or did we want the top five 2 or the top ten, and top ten sounded like a 3 big number, so we ended up with six -- and 4 everybody else. Six is a nice number; it's 5 divisible by two and one and three, but the 6 compelling logic is slightly not totally 7 obvious. On the other hand, there are also 8 budget questions: The more states we collect 9 in detail, the larger perceived burden it is 10 on the respondents, and two, the larger 11 budget requirements for EIA. And so what we 12 did, and what drove it was, these are the top 13 six producing areas, so we're going to do 14 them individually because we ought to pay 15 attention to them. I think I'm going to 16 suggest we do more like the top ten when we 17 modify the 914, and there's something that 18 goes with some of the opening remarks. We 19 think there are going to be some major 20 increases in production from unconventional 21 oil resources. Those states right now do not 22 produce very high amounts of oil, but we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 65 1 assume one or two of them will, and we assume 2 -- or I assume -- that we ought to add them 3 to the list of the states we are formally 4 going to collect data from. And so that is 5 how to -- where do we cut it off? There are 6 37 states that produce oil and gas, it seems 7 like the top ten would be a useful group to 8 have detailed state-level information on, and 9 we're going to do that in the time period of 10 a month. The next question is, if we assume 11 that the future of oil production is going to 12 ride on unconventional resources, should we 13 add a couple of states right now that we 14 assume are going to change their production 15 levels in the immediate future or in the next 16 decade. 17 MR. BINGHAM: Can I ask a quick 18 question? I'm Derek Bingham, by the way. So 19 I'm just looking at the remarks from last 20 time so I -- this is my first time here, so 21 bear with me -- but I'm just kind of curious 22 -- so two people commented about prediction BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 66 1 errors, I guess, and I'm curious how much 2 time is spent kind of explaining the 3 prediction errors in some of these models. 4 And the reason I would say that, for instance 5 -- this may correlate with what Lee's saying 6 -- is that if you didn't include these areas 7 that may be quite important, then at least 8 you could go back after the fact say, well, 9 it turns out that these areas are now 10 becoming an important part of the oil supply 11 or whatever it is that they're producing. 12 They're producing whatever it is. And I 13 guess my question is for looking at and 14 explaining why you missed in a model and then 15 either building components into the model for 16 that reason because they're now important 17 where before they weren't. Are there large 18 efforts in that respect? 19 MS. ANDERSON: Well, I can talk 20 about what we're doing in the short-term 21 context, and John can talk about for the 22 longer term. This is the program that Nancy BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 67 1 alluded to in the beginning in her bullets 2 that we are -- her office is now and her team 3 is reviewing as a proposal we have on the 4 table for looking at forecast areas for about 5 40 key variables that come out of the 6 Short-Term Energy Outlook, precisely to 7 answer the questions you're asking, try to 8 understand why we're off, and if that's a 9 missing data problem, could we fix it by more 10 data, could we fix it by changing the model, 11 as well as just informing the public of how 12 accurate are we and explaining that in some 13 cases you can't get any more accurate than 14 what we're doing because these are often due 15 to weather variations and things that we're 16 not going to really enhance our predictive 17 capability of. 18 So, in fact, we're trying to do a 19 better job of explaining to our readership 20 what our errors are, as well as to ease that 21 information back to our modelers to try and 22 either fix the models or improve data, data BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 68 1 accuracy -- whether that's a problem with the 2 accuracy of the actual data, the timeliness 3 of the data, or, in fact, missing data, we 4 don't know yet, but we would certainly want 5 to move the program in that direction. John 6 already does some of that in the long-term 7 model. 8 MR. CONTI: We've had a forecast 9 evaluation report, I believe, that EIA's 10 published for about a decade and a half now, 11 and it doesn't go into as many variables as 12 Margot does or will; we have more aggregate 13 variables. But we've done it for every AEO 14 since 1982, and so some of those earlier 15 AEOs, we actually have, where our 15- or 16 20-year predictions are actually data values 17 now, and we go back and we assess why we 18 didn't predict -- well, if we notice big 19 discrepancies, we try and go back and assess 20 why we didn't predict it correctly, and we 21 see if we still have that same faulty logic 22 and models we use today. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 69 1 We do tend to look at the most 2 recent years to see if we're trending 3 differently from the actual numbers and to 4 try and understand that. But we don't sort 5 of have -- it's just a matter of resources 6 and how much resources you allocate to each 7 of these efforts, but we do do that. 8 Actually, one question that I was going to 9 ask the committee somewhere in the next 10 couple of days was in this forecast 11 evaluation report we look at the sum of the 12 absolute errors to give an indication of how 13 far we're off. Well, that's a good idea of 14 how far you're off, but I'm always interested 15 also in bias. I'm not so worried if we're 16 equally dispersed high and low, but I'm more 17 worried if we have a bias to our forecast. 18 So it was a question I was going to -- what's 19 a good measure of bias? I was thinking of 20 maybe putting out graphs where people could 21 visualize it. To me, that's the most 22 instructive rather than some other BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 70 1 statistical single number measurement. 2 MS. ANDERSON: And we'll be using a 3 bias measure in our estimates. It's one that 4 we're proposing. 5 DR. HENGARTNER: One of the nice 6 things about the model is that it's not too 7 picky of what kind of data it needs. I find 8 that the many applications that -- Rolling 9 Stones are right, you don't always get what 10 you want, but you get what you need. 11 In a way the question I'm asking 12 is, we are very focused on continuing 13 historical series, which is a good thing, but 14 in terms of improving our models and 15 predictions, there are other sources of data 16 that may be available out there essentially 17 for free -- maybe not exactly free, but close 18 to free -- which are not exactly what we 19 want, but they're close enough that they 20 could be incorporated into the modeling 21 effort. That's what models are for, right? 22 To borrow strength of things that kind of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 71 1 inform us -- and yes, it's not a perfect 2 match, but it has value. 3 And so while I don't have any 4 specific examples -- I see John nodding his 5 head -- I'd like you to tell me, yes, 6 actually, that's something I've been thinking 7 about. 8 MR. CONTI: I'm sure there are a 9 lot of analysts in the audience here, but I 10 know there are variables or data we use that 11 EIA does not collect, that we purchase from 12 outside sources, and only because it's just a 13 lot cheaper to do. It's used commercially, 14 so somebody picks it up and sells it to a lot 15 of people, and we just buy it and procure it. 16 So I don't know if that was the direction 17 you're going in, but for example, in 18 commercial floor space, we tend to buy those 19 projections. We buy a lot of our 20 macroeconomic information. And we use other 21 sources from other agencies. So we purchase 22 data where it's cost effective, but if it's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 72 1 just raw energy data that no one else 2 collects, and we sort of have responsibility 3 for more or less -- if it were to be 4 collected, we would be the ones to collect it 5 -- and it gets back to prioritizing -- how do 6 you prioritize which data you collect? 7 For example, John mentioned it 8 became very relevant for us to collect this 9 weekly natural gas storage information. I 10 don't need that in my long-term models, 11 Margot doesn't probably need weekly in her 12 monthly models, but I --- most of us would be 13 lynched if we said we shouldn't collect it 14 weekly because, no, we don't use it. No, 15 there are a lot of people outside of EIA that 16 would kill us if we stopped collecting it on 17 a weekly basis. 18 DR. HENGARTNER: But I'm trying to 19 answer more the question, what data do we 20 need or what does our model inform us about 21 data needs. And as I said, there is this 22 dichotomy of two things, and you again go BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 73 1 back, prioritize, what are the questions you 2 want to answer -- 3 MR. CONTI: Right. 4 DR. HENGARTNER: -- what are our 5 clients' need in terms of raw series, but if 6 you're going to go and really answer the 7 question, what does the model tell us about 8 our data needs -- 9 MR. CONTI: Right. 10 DR. HENGARTNER: -- then you are 11 entitled, and I would highly encourage you, 12 to think broadly and in new ways because 13 that's what models are for. And raw data is 14 something else, and I don't want to discount 15 your need of raw datas -- and I'm not 16 advocating that either, so -- 17 MR. CONTI: Right. 18 DR. HENGARTNER: -- don't get me 19 wrong here -- it's just -- I'm trying to 20 answer very pointedly the question that I 21 think was set out in the beginning. 22 MR. CONTI: Okay. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 74 1 DR. HENGARTNER: Yes? 2 MR. WATSON: Bill Watson with 3 CNEAF. I'd like to come back to the issue of 4 use of more data by external parties, 5 especially people that are trying to do their 6 own modeling. We do have at EIA -- at least 7 from my perspective -- a fairly systematic 8 way to understand what the data needs are of 9 people that like our AEO modeling effort, 10 come to our AEO outlook meeting, and we get a 11 fair amount of feedback from them. John has 12 already talked about an assessment that would 13 look at the data and the data needs for any 14 enhancements or changes in the modeling 15 efforts in that area. I'm not aware -- and 16 this is partly a question, partly a statement 17 -- that we have anything like that on the 18 side of the users of our survey data. We 19 don't have an annual meeting, or we don't 20 have a systematic way to communicate with 21 people that just use our data series. Many 22 of them would be modelers, and they would BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 75 1 have some good opinions probably and good 2 recommendations about whether we're missing 3 some data gaps there from their perspective. 4 So I guess maybe the question I would ask to 5 the committee is, do we need to have some 6 sort of balance here between the internal use 7 of the data for modeling and the use of the 8 data by external modelers and some way to 9 assess on the external side what the data 10 gaps may or may not be. 11 Just to give an example, in the 12 coal area, especially in the last 12 to 24 13 months, we get a continuing series of calls 14 from people asking about coal prices: What 15 data do you have on coal prices; what data do 16 you have on coal transportation costs; what 17 data do you have, other than your annual data 18 that's with a two-year lag, on coal stocks. 19 And so we're hearing this from the outside 20 community, but it's not good, I think, to 21 base recommendations on ad hoc information 22 such as telephone calls; you need to do it BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 76 1 systematically. 2 MR. CLEVELAND: Well, that raises 3 the question of how well do you know who your 4 client base is. I mean, there's Web hits and 5 downloads, but that's pretty limited kind of 6 information. And so I agree the Annual 7 Energy Outlook meeting, which I have 8 participated in, is a good way to get 9 feedback from at least some slice of your 10 users for that, but it seems to me that some 11 other type of form -- we don't necessarily 12 have to do it every year -- whereby you try 13 to identify who these other users are of the 14 various types of data you collect, bringing 15 them together and getting some of that kind 16 of feedback. But the question is how do you 17 think you know who your users are? We know 18 the squeaky wheels are the ones that -- we 19 know who the big squeaky wheels are, but 20 that's clearly not who we want to be making 21 all our decisions around. 22 DR. BURTON: The question that you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 77 1 asked, I think is where I thought this 2 discussion was probably going to go to begin 3 with. And one thing that I find is fortunate 4 -- and may be a way to sort of sell modeling 5 efforts -- is by doing modeling internally, 6 you're likely to run into a lot of the same 7 problems that we as external users encounter, 8 and the fact that you do do modeling 9 internally is a way to inform the process 10 about what data is most useful. 11 MR. CLEVELAND: So you do do 12 customer surveys? What kind -- what are 13 those like? 14 MS. KIRKENDALL: We've done -- 15 let's see, when are we done? Where's 16 Colleen? I want to -- we've participated in 17 the American Customer Satisfaction Index, and 18 that's a survey that's associated with the 19 Web so it's a random selection and people are 20 invited to participate in the American 21 Customer Satisfaction Index. That's one way 22 we've done surveys of people who are on our BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 78 1 Listservs, thinking that people who sign up 2 to be on a Listserv are keenly interested in 3 those products they've signed up for. And of 4 course, it's not complete, but -- 5 MR. CARUSO: And we do the ones 6 where -- the pop-up surveys for users. 7 MS. KIRKENDALL: We do pop-up 8 surveys for users. 9 MR. FREEDMAN: We've done surveys 10 of the NEMS conference -----. 11 MS. KIRKENDALL: Yeah, we do 12 surveys of the NEMS conference. They do that 13 regularly to monitor how well the conference 14 runs. We've also done surveys -- it's a 15 sample from the Listserv for the AEO and for 16 the STEO -- to get some input from them on -- 17 DR. BURTON: Do you get a good 18 response rate from them? 19 MS. KIRKENDALL: Well, no. 20 MS. ANDERSON: No. 21 MS. KIRKENDALL: About 25 percent. 22 MR. ANDERSON: Nor is there often BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 79 1 space to get really at some of the detailed 2 questions you want to get at that maybe if 3 you brought people together in a focus-type 4 setting, you might get -- and you -- 5 MS. KIRKENDALL: Yeah. 6 MR. CONTI: I was going to say, we 7 do that with -- we have a follow-up on the 8 NEMS conference where we get sort of really 9 heavy users of our models and our forecasts, 10 and we do in-depth surveys with them and try 11 and understand their needs and faults they 12 have or problems they've identified. And so 13 we do what we do at the conference for 14 everybody, and we get a fair response rate, 15 and then we go into detail with some people 16 from a broad array of areas that do energy 17 modeling. 18 MR. SINGPURWALLA: Do the people 19 outside the Energy Department, are their 20 issues and needs different than the ones 21 inside? 22 MR. CONTI: Sometimes they're more BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 80 1 specific. Yeah, they have a more specific 2 interest and they want to go into greater 3 detail than we do. But a lot of times, we 4 tend to focus on people that are trying to do 5 similar things. Sort of broad national-level 6 forecasting. 7 MR. SINGPURWALLA: Right. 8 MS. ANDERSON: But have we ever 9 done anything that might get the users of the 10 consumption surveys together in a room or in 11 a conference and ask specifically about the 12 coverage and the use of the data that we are 13 putting out, whether it's -- how we put it 14 out or how frequently we put it out or the 15 depth of coverage, and I don't think we've 16 done that on all of our surveys, we've 17 probably done it on some and asked more 18 Web-based questions. But we probably haven't 19 gone into as full detail on some of our 20 surveys; it might be beneficial to do that. 21 MR. CLEVELAND: Yeah. I don't 22 think the issue is NEMS so much -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 81 1 MS. ANDERSON: Yeah. 2 MR. CLEVELAND: -- because I think 3 you could do a pretty good job of getting 4 feedback on that, it's the other broader 5 array of data products whose constituencies 6 are much flatter and broader and less 7 well-identified than I think is the issue 8 here. 9 MR. CONTI: Some of those issues do 10 come up. We typically have workshops -- and 11 we meet a couple times a year -- in the 12 modeling area, and some of those go back to 13 -- like the industrial workshop tends to go 14 back to the Manufacturing Energy Consumption 15 Survey, and we get people from EPA and 16 sometimes outside the Alliance to Save 17 Energy, and there'll be discussions about the 18 MECS at that point in time -- and Tom's been 19 there and people from your office have been 20 there to talk about the MECS. So it's not 21 focused on the MECS, but it's focused on 22 industrial modeling and some of the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 82 1 discussions deal with the MECS. 2 MS. KIRKENDALL: We do -- when 3 we're considering redesigning surveys -- it 4 used to be at least -- but when electric 5 power was going to redesign their surveys, 6 and I think they did this in natural gas, too 7 -- they had a series of focus groups with 8 users, or different kinds of users. So you 9 might have our own users and then other 10 federal users and other groups of users that 11 you could identify to come in and talk about 12 data needs. So those kinds of meetings would 13 be held at the time you want to redesign the 14 forms, so it's not an ongoing effort. 15 MS. FORSYTH: Those same people 16 might also be used, not just to talk about 17 data needs, but to talk about other data 18 users that might be beyond the scope of the 19 current data collection so that they'd be a 20 source of perhaps exploratory data in that 21 kind of question that you're asking. There 22 are some that will listen. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 83 1 DR. HENGARTNER: We're 10 minutes 2 early. Any more comments? 3 MS. KIRKENDALL: Well, if you think 4 of anything, let us know. 5 MR. CLEVELAND: I do have one other 6 one, actually. And I'm wondering if the 7 external review panel -- this seemed to be 8 one of the fundamental questions that they 9 must have talked about or -- did they collect 10 any other information in their review about 11 whether or not -- how we go about collecting 12 data and identifying who the users are and so 13 on and so forth? 14 MS. KIRKENDALL: They looked at all 15 of our -- we put together a lot of 16 information from our customer feedback, all 17 the vehicles that we have, so they got 18 summaries of all of that. 19 MS. ANDERSON: They certainly made 20 some suggestions on things we ought to take a 21 second look at and really ask ourselves why 22 are you collecting this, is it still relevant BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 84 1 to collect this information, and for us to 2 really ask ourselves that question we're 3 going to have to do precisely what you asked: 4 Make sure we fully understand who's using it 5 and what the costs are of scaling back or 6 eliminating entirely. They did feel that due 7 to change in mission or change in the sector, 8 there are some things that we're collecting 9 that are really not relevant to the evolution 10 of the energy sector, or could be augmented 11 by private sector data, and it may, in fact, 12 be a net savings to EIA. So I think that's 13 an interesting discussion. 14 MS. KIRKENDALL: Yeah, but they 15 really didn't go into detail on our core 16 program; they sort of assumed that the core 17 program and the data that we need to inform 18 the energy users was, if not wonderful, at 19 least it was necessary, and they didn't get 20 into the details because it wasn't a long 21 enough -- 22 MR. CLEVELAND: Sure. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 85 1 MS. KIRKENDALL: -- time period to 2 do that, really. So that there were a few 3 things they identified as low-hanging fruit 4 that they weren't sure were relevant. 5 MS. KHANNA: I just had one 6 cautionary plea as you consider using data 7 collected by other sources, especially 8 private sector, is to focus a lot of data on 9 the accuracy and robustness of the data 10 collection process as well as the final data 11 that you get. And a great advantage of EIA 12 collecting its own data is you know exactly 13 how it is collected and what the pitfalls and 14 strengths of it are. If you buy -- as a data 15 user, I mean, I'm always worried about that. 16 MR. RUTHERFORD: We have five 17 minutes to go until the planned time, so I 18 thought I'd just add a comment. I thought 19 Bill's comment about thinking creatively 20 about ways of soliciting information about 21 how data is used is an intriguing one. I'm 22 thinking there are two ways that you could BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 86 1 leverage this: One would be to sort of 2 organize a special issue in the Energy 3 Journal or something about a specific topic 4 that uses, say, natural gas data from DOE 5 would be the theme to try to leverage input 6 from academics who have worked in the area. 7 Also, if you're thinking about sort of model 8 revisions at DOE, having a chance to sort of 9 get a survey up close of what's been done 10 with your data would be useful for thinking 11 about how to move forward. And the other 12 thing is perhaps prizes. I don't know, you 13 could have some sort of -- anyway, there's -- 14 I just -- I think it was just -- this is sort 15 of off the top of my head; it just seemed 16 like an interesting idea. 17 MR. CLEVELAND: Yeah, I like the 18 idea of this special issue. An analogous 19 thing on the modeling front might be to get 20 Hill Huntington and the Stanford Energy 21 Modeling Forum folks together and maybe have 22 one year -- one cycle of the modeling forum BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 87 1 to essentially give you guys feedback on what 2 you guys are doing. I think that would be a 3 very interesting use of some pretty 4 high-powered talent. 5 MR. CARUSO: Yeah, we've worked 6 very closely with Hill and the EMF, so that 7 would be good. I like both of those 8 suggestions. 9 DR. HENGARTNER: Okay, seems this 10 discussion has ran its course. We're 11 fortunate to be ahead of schedule, and this 12 is a good thing. I'd like to take a break 13 and remind you that we're going to continue 14 afterwards in break-out sessions in Room 7-E 15 -- that's here; Derek Bingham, Johnny Blair, 16 Moshe Feder, Barbara Forsyth, myself, Walter 17 Hill, and Darius Singpurwalla should remain 18 here. And Mark Burton, Mr. Cleveland, Jae 19 Edmonds, Neha Khanna, Nagaraj Neerchal, and 20 Tom Rutherford will go downstairs to Room 21 5-E. And we will reconvene in this room 22 afterwards for discussion from these BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 88 1 break-out sessions at 11:35. So we have 2 about 15 minutes break, and before we all 3 leave this room, those who trickled in, one 4 last time, please identify yourselves so we 5 can have you on the record. Thank you very 6 much. 7 MR. GRUENSPECHT: I'm Howard 8 Gruenspecht with EIA. 9 (Recess) 10 DR. HENGARTNER: This is the next 11 break-out session. It's going to be the 2006 12 MECS. By the way, I was told that those 13 microphones are picking up just fine if you 14 project towards them. I'm just relaying the 15 message I heard from the master. So this is 16 the 2006 MECS looking at past performance 17 statistics to motivate new method of 18 collection, and it's Bob Adler and Tom Lorenz 19 who are going to be leading our discussion -- 20 at least giving us an introduction -- and 21 hopefully we're going to all chime in and 22 argue and discuss some more after this BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 89 1 presentation. So, welcome Tom. Thank you. 2 MR. ADLER: Okay, I'm Bob Adler, 3 and that's Tom Lorenz. First of all, can 4 everybody hear me? We will be talking about 5 how the 2002 MECS performance statistics kind 6 of justify and motivate the 2006 MECS and our 7 big redesign. We want to thank Rick Hough, 8 Vicki Haitot, and Stacey Cole, and since 9 Susan's here, I'll thank her too, Susan 10 Bucci -- 11 MS. BUCCI: Thanks. 12 (Laughter) 13 MR. ADLER: I know which side of 14 the bread the butter's on. They helped us 15 prepare some of the data and also read some 16 of the drafts we had of the paper. 17 I'm having trouble with this. 18 Okay, as I said, we'll be looking 19 at two kinds of performance statistics. The 20 first is data status flags, and the second 21 will be what we've been calling differential 22 response rates. Hopefully that will lead to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 90 1 a discussion on these topics, how we boost 2 the Internet participation, performance 3 statistics that would be useful to track. 4 When we talk about that, the risks in 5 targeting non-response follow-up to larger 6 establishments, and there'll be a few others, 7 and I hope it'll sort of generate from there 8 -- or degenerate. 9 The organization of this talk, 10 we'll first -- hopefully briefly -- we'll 11 talk about the background on the MECS. Then 12 I'll talk about the examination of the data 13 status flags that are resident on the StEPS 14 database, which keeps the MECS data. Then 15 we'll go into a discussion. Tom will talk on 16 the Excel version of the electronic forum 17 that we introduced in 2002 to a select group 18 of respondents, and we'll look at the 19 performance statistics between those two 20 groups. And then I'll come back and talk 21 about the differential response rates. 22 Okay, the MECS. I think most of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 91 1 you know what it is. It is the government's 2 look at manufacturing establishments for 3 their energy consumption. We hope it's 4 unique, otherwise OMB wouldn't let us do it. 5 It's based on the NAICS codes. We'll try not 6 to talk a lot of NAICS during this 7 discussion. 8 What is important for this is that 9 we have three survey forms that go out; we 10 have the "A," "B," and "C." The "A" is 11 mailed to most of the establishments; it's 12 sort of the default form. We have the "B" 13 form, which goes to only petroleum 14 refineries. It's actually a shorter form 15 because it would otherwise duplicate some of 16 the other EIA data systems that are out 17 there. And finally, the "C" form, which is 18 sent to our energy-intensive and industries 19 that have complicated flows, so we ask a 20 little bit more; they're a little bit more 21 customized. 22 It has been a self-administered BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 92 1 written questionnaire. As I said, we are 2 moving away from that, hopefully, and we did 3 have this 2002 Excel version that was our 4 first step in doing that. It's designed and 5 sponsored by EIA, although Census will be 6 helping with the design this year, too. And 7 it is totally confidential, according to 8 Title 13. It's mandatory by federal law. 9 People have a tendency to not get too excited 10 by that. And it's a statistical sample from 11 a list frame of 15,000 establishments. 12 Okay, these are the data status 13 flags. It makes a nice acronym of RAISE, and 14 the "R" flags and the "A" flags are the two 15 that really are the most common, and I'll 16 really be contrasting those. 17 The reported data is an "R." And 18 this means that one way or another, a 19 respondent has come back and said this is 20 what the data is. Now, it could have been 21 through the report form itself, or it could 22 have been a follow-up using telephone or an BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 93 1 e-mail. 2 The "A" flag is an analyst 3 correction of a reporting error. That means 4 just that. There are a number -- if they can 5 see that a wrong unit was used or a wrong 6 price was used, then they're going ahead and 7 based on their knowledge and what's on the 8 form, they have enough to bring together 9 something to that and then change the data 10 appropriately -- we hope. But it is an 11 analyst correction, and I'll get back to the 12 reported data -- also could have analyst 13 corrections. 14 "I" is imputed data. It's really 15 -- and you guys can help me at a later time 16 -- I don't think -- it's very rare, it just 17 means that they -- it's usually imputing an 18 expenditure or price -- expenditure or a 19 quantity from one or the other based on -- if 20 that's all we have available. 21 "S" is very important; it's the 22 source data. If we don't have -- this BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 94 1 particular year we ran it the same time as 2 the Census of Manufacturers -- or now it's 3 called the Economic Census/Manufacturing -- 4 and that's actually drawing data from another 5 source. 6 And "E" is a system edit -- it's 7 also rare -- and it means that we've seen 8 enough of it so that we can put a program 9 edit. 10 Let me get back to the "R" flag for 11 just a second. The "R" means that we can 12 look at -- that generally, you're talking 13 about an actual response data. However, we 14 need to somehow differentiate, in some cases, 15 whether an analyst was involved or not. Now, 16 StEPS allows us to do that by comparing two 17 versions of the data: The edited version of 18 the data and the "reported" version of the 19 data, and I know they sound the same. By 20 doing that, you can then determine whether an 21 analyst was involved in actually making that 22 correction. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 95 1 Now, some of these data, we're 2 going to talk about "R" as if it's good and 3 "A" as if it's bad; obviously, there are some 4 problems in that. Now, the "S" data means 5 it's a good piece of data, but there's some 6 trouble with our questionnaire because we had 7 to grab that piece from another data source. 8 So one -- I guess that's -- let's just get 9 into the data then -- and I've handed this 10 out, and I guess it's dark so you can't read 11 it there either -- but what this is, it's a 12 nice graph, although it probably is against 13 all of our SMG-EIA standards, but it's a good 14 way of looking at something very quickly and 15 glancing down and seeing that, okay, where 16 are the concentration of "A"s and "S"s and 17 "R"s that point out something amiss with the 18 data. Now, the blue flags -- blue is the 19 reported data -- and I've labeled one, two, 20 and three. The LPG nonfuel is number one, 21 and you can see there are relatively few that 22 are actually reported data. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 96 1 I talked a little bit before about 2 the "C" form; usually, if you're not on the 3 "C" form, you should not have LPG as a 4 feedstock or a nonfuel, so what that is is 5 essentially analysts coming in and saying, 6 I'm sorry, this is not right, they shouldn't 7 have that. And again, you guys will tell us 8 if that's true later. And also, within that 9 blue, we've had the most difference between 10 the reported analyst type and the reported 11 actual data, so that's something that we need 12 to look at, and I think it's something we can 13 clear up a bit more with the electronic form. 14 The second, number two, is actually 15 much more important. The natural gas as a 16 fuel, as you might guess, is very common; 17 9,000 cases reported natural gas as a fuel, 18 and that's sample cases. So what you're 19 looking at, the blue, that looks pretty good, 20 and there was less than 10 percent of the 21 blue that actually represent analyst 22 corrections. However, that green is about 20 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 97 1 percent -- a little less -- and if we add the 2 others -- and that really means that analysts 3 have to spend their time correcting that 4 data. 5 Now, what is that data? A lot of 6 times, it's a missed unit, a missed price, a 7 lot of things that we can collect better if 8 we had an electronic version of the form. So 9 a lot of that data -- now, if we were to take 10 20 percent of 9,000 cases, you can sort of 11 get at what exactly the analysts have to do 12 and spend their time. Imagine shifting that 13 to the respondent and have them do it; we can 14 probably get the MECS out a lot faster -- at 15 least that's out goal. 16 Finally, the third one is the other 17 LPGs, it's abbreviated 'enes as nonfuel 18 propylene, ethylene, butylene. What that 19 means -- it's sort of a rare type of LPG, and 20 I really just want to point out the blue -- 21 purple -- whatever that color is at the other 22 end that's not blue, not green, and I -- it BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 98 1 looks black to me. But what that is is the 2 source data. We had to load up from the 3 Census of Manufacturers their nonfuel use, 4 and as it turned out, what they do is they 5 call it material use -- is that right -- and 6 that term -- uses a material -- and that term 7 seemed to have keyed them off to respond 8 correctly more than our standard MECS terms 9 of nonfuel and feedstock. And yes, we did 10 explain what nonfuel and feedstock, and yes, 11 they were supposed to respond. Nevertheless, 12 somehow they got it in their heads that they 13 weren't supposed to. 14 Okay, I've already talked about all 15 of this, so I'm going to pass it to Tom, who 16 will talk about the differences between Excel 17 versions. 18 MR. LORENZ: Hi, everyone. You 19 guys have any questions in between, just 20 holler out, just let me know. As Bob 21 mentioned, in 2002 we created an Excel 22 version of one of our forms, and it happened BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 99 1 to be the "C" form. The reason why we chose 2 the "C" form was, one, because it was our 3 longest form, and number two, because, as Bob 4 mentioned, the respondents of that type of 5 form are the higher-intensity energy users, 6 such as the chemical, paper, and primary 7 metal industries, so they're more used to 8 filling out energy-type data, and we felt 9 that they would probably best be able to fill 10 out the Excel version. We made the Excel 11 version look like the written questionnaire 12 so that the ease of -- and anxiety that 13 people would have of filling out an 14 electronic questionnaire would be somewhat 15 reduced. The added capabilities that we had 16 to the Excel version, we had price and value 17 range checks of most of our major energy 18 sources. We also had calculations -- 19 additions and things like that -- for our 20 total consumption of energy sources. And we 21 copied those totals to where we needed them 22 at other points in time in the form -- with BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 100 1 the paper form, we have the respondent just 2 copy the number themselves into the other 3 sections such as end use and fuel switching. 4 So we did that automatically for them. 5 There were some limitations that we 6 had with this Excel form, mainly time was a 7 factor. Because we had to have the 8 respondents save the information to a CD 9 and/or diskette, we couldn't include all the 10 online or edits that we wanted -- screen 11 edits that we really truly wanted for our 12 Excel version. We also -- because it was an 13 Excel version, and we were sending it in -- 14 we were having the respondents send in either 15 a CD or diskette. It wasn't a true Internet 16 data collection system in the sense that they 17 were not reporting it via the Internet. So 18 we didn't have the capability of having the 19 Census Bureau incorporate some of the 20 programmatic and integrated systems for their 21 data capture. And one of those faults was 22 that we really had to -- because they sent us BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 101 1 the information, they had to key it in the 2 same manner as they would a paper form, which 3 was one of the downfalls, and that's why 4 we're actually really targeting going more 5 towards an IDC which would eliminate a good 6 portion of that keying aspect, which 7 increases our error. 8 For the Excel version we had 323 9 responses, which was about 8.9 percent of the 10 total number of "C"-form responders. 11 However, the Excel responders accounted for 12 about 12.4 percent of the total consumption 13 that was reported using the "C" form. Now, 14 we had about 23 respondents who sent us a 15 diskette or a CD -- mainly a diskette -- that 16 failed for some reason or another. About 80 17 percent of those 23 respondents, it was 18 incorrectly formatted was the message that 19 was received. The other was a bad diskette. 20 So whether it was -- I don't know what 21 exactly happened in the mailing process or 22 they didn't format it or what the -- but BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 102 1 that's just another reason for us to really 2 target the Internet data collection process. 3 DR. HENGARTNER: Have you tried to 4 run those diskettes on a Mac? 5 MR. LORENZ: Yeah. And it just 6 wasn't -- there was either nothing on it or 7 it wasn't reading or just something was -- I 8 know now a lot of times that the diskettes 9 are already formatted, so I don't know really 10 what happened in the process. 11 I'm going to just talk a little bit 12 about a couple of the major energy sources, 13 particularly first and foremost, electricity, 14 the electronic versus the paper, and I'm 15 going to target, as Bob mentioned before, the 16 reported data, which is the "R"s and then the 17 "A" flag, which is the analyst corrections. 18 With the first -- actually the third and 19 fourth columns, the one edit that we had was 20 for the price and range checks. As you can 21 see, there's not much of a difference between 22 the electronic and paper forms. However, the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 103 1 last column, the 96.3 percent and the 85.7 2 percent, there's about a 10 percent 3 difference there, and we attribute that to 4 the fact that there was an edit there where 5 we actually calculated the information for 6 them instead of having them do it. So that 7 increased our electronic response rate to a 8 much higher percentage. 9 Now, with natural gas, there are a 10 few things actually with natural gas. I'll 11 go through the third column, where the total 12 natural gas purchased. As you can see, 13 there's quite a variance in the "A"s between 14 the electronic and paper, about 11 percent. 15 And if we go back here -- I'm going to go 16 back real quick -- you notice that there's 17 quite a bit higher than what's on the 18 electricity. We attribute that to the fact 19 that a lot of times we ask for the respondent 20 to give us their information in thousands of 21 cubic feet, which is Mcf, and with natural 22 gas, they have about five or six different BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 104 1 various type of units that they could report 2 it in. So one of the things we're looking at 3 for 2006 is to incorporate some type of a 4 drop-down box to have them choose their units 5 so that we can actually, I guess, correct it 6 for them and process it for them in the 7 processing system. 8 Also, in the total natural gas fuel 9 consumption column, which is the last column, 10 again, you can see the variance where it 11 worked in the same manner as in electricity, 12 where there was about a 12 percent difference 13 in their total consumption between electronic 14 and paper. And again, we attribute that to 15 the fact that they didn't have to calculate 16 and add up their numbers; we did it for them. 17 And the one last thing that I want to mention 18 is down in the bottom right-hand corner, the 19 8.3 percent is attributed to an "E" flag, and 20 that was systematic error that we found. We 21 basically just made a systematic correction, 22 let the system do the correction for us. So BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 105 1 we're going to incorporate some of these 2 things into the system that we're building 3 where the Census Bureau is held, and we're 4 using their system, which is Census Taker, so 5 we can get better responses much faster and 6 improved. 7 DR. FEDER: I may have missed it -- 8 you said in '02 it was up to the respondent 9 to choose which mode or which instrument to 10 use, right? 11 MR. LORENZ: Right. 12 DR. FEDER: For '06, it's going to 13 be all electronic, or -- 14 MR. LORENZ: That's our goal, yes. 15 We want to -- yeah, but we can't really 16 expect it, though, because that's our goal is 17 to hopefully have as many people as possible 18 because it'll save on mailing the forms out 19 and all that, but we also do understand that 20 there are some manufacturers we have -- 21 because it is a survey sample -- we have some 22 that are large establishments and some that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 106 1 are small establishments, and sometimes they 2 may not have the capabilities or their 3 establishment or company may not allow them 4 access to the Internet. So we do have a 5 paper form. But we're going to really market 6 the electronic form or the Excel or Internet 7 data collection systems very heavily. And 8 that's our goal is to really reduce the 9 mailing that we have to do with the survey 10 and instead have them complete the survey 11 electronically through the Internet. Thus, 12 we can do a lot of the programmatic things 13 behind the scenes in the data processing 14 system and collect the data through the 15 Internet. 16 So, with that, I'm going to hand 17 this back over to Bob to talk about response 18 rates. 19 SPEAKER: Well, if it gets to be a 20 higher response rate if they respond and 21 decide which instrument to use, ------------- 22 I'm trying to imagine what you're thinking. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 107 1 You think -----. If the respondent chose the 2 instrument, you'd get a higher response rate. 3 If the respondent can't use the electronic 4 form or doesn't have Excel or has trouble 5 with the floppy disks or ---- being used, the 6 response rate would be higher. 7 MR. LORENZ: Well, the economics of 8 running the MECS, it's going to be more 9 economical if we don't have to mail out all 10 the forms. So we're hoping to market that as 11 much as we can, but we can't limit that to -- 12 because, depending upon whether they have the 13 Internet access or not, we can't say, you 14 can't fill out this survey because you don't 15 have Internet access, so we do have to have 16 some type of a backup. And that paper form 17 is our backup. Typically, you will -- 18 hopefully we will see -- what we saw is that 19 there was a little bit better response rate; 20 there were fewer corrections that needed to 21 be made with the Excel form in 2002, so we're 22 hoping that that holds true for the 2006 when BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 108 1 we go true Internet. 2 DR. FEDER: If you send them a free 3 laptop, they'll be happy to -- 4 (Laughter) 5 MS. FORSYTH: I had a question. 6 When you're talking about response rates 7 there, that you had less edits, are you 8 talking about item non-response or unit 9 non-response or both? 10 MR. LORENZ: I think it's item. 11 MS. FORSYTH: Item level. And you 12 were talking about unit non-response, right? 13 Okay. 14 MR. SINGPURWALLA: Is it cheaper 15 overall to do paper or Excel? 16 MR. LORENZ: Well, we had to pay 17 for the printing costs -- it's -- hopefully, 18 it's going to be -- it'll probably be about 19 the same, but what we're hoping is that in 20 the back end -- yeah, it'll reduce our 21 editing time. 22 MR. SINGPURWALLA: Yeah. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 109 1 MR. LORENZ: And so in that sense 2 it'll save us some times. 3 MS. FORSYTH: Getting rid of those 4 diskettes will help too. 5 SPEAKER: The Internet thing. 6 MR. LORENZ: That was really our 7 only method of collecting it at that point in 8 time, but -- yeah. 9 DR. FEDER: I wonder if the '02 10 mode that was user selected did not affect 11 even some of the responses. Clearly there 12 could be some association between the type of 13 industry, the type of respondent, and their 14 choice of modes. So there could be potential 15 bias in letting them -- 16 MR. ADLER: That's one of the 17 questions we had. 18 DR. FEDER: That's the answer. 19 That's it. I'll go with Bob. 20 MR. ADLER: Yeah. The problem with 21 using that is that it is a self-selecting 22 group. On the other hand, we're not doing an BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 110 1 experiment there, we were actually collecting 2 data, so -- we thought that it would work out 3 better that they did, in fact -- I hope those 4 statistics show that they did do better. 5 Now, did they do better because 6 it's just a better group of responders or 7 that our edits on site actually helped? I 8 hope it's the latter. I don't really have a 9 reason to think that this was a better group 10 of responders, but I don't know. I don't 11 think we have enough to really discern that. 12 MS. BUCCI: Tom, on the paper 13 version, when you were making those 14 comparisons, were you looking at all three 15 form types, the "A," "B," and "C," or just 16 the "C"? 17 MR. ADLER: Just the "C." 18 MR. LORENZ: Just the "C," yeah. 19 MR. ADLER: So we did try to at 20 least have that much -- it was the same 21 industries overall. They were 22 energy-intensive industries. We did not BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 111 1 count the "A" and the "B," which would 2 probably show a lot more errors because they 3 -- the "A" especially has a tendency not to 4 know what we're talking about. 5 I'll go ahead and talk about this 6 real quickly because I know we're running 7 long. Another performance statistic that we 8 looked at was really response rates, and 9 we've been unfortunately suffering from 10 declining unit response for, I think, a 11 couple of survey cycles. I think maybe our 12 highest was in '94. In '98 it's gone down, 13 and 2002 also has gone down. We think that 14 partly that is downsizing, competing with the 15 overseas market more than they have been, 16 could be an attitude toward government 17 surveys -- I'm not allowed to say that, but I 18 did. 19 (Laughter) 20 MR. ADLER: Yeah, right there. But 21 there's a general sense that's probably 22 happened within the last couple of decades BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 112 1 that surveys are not the first priority of a 2 manufacturing company and establishment. On 3 the other hand, our budgets are being 4 squeezed, as well as the manufacturers, so we 5 have to do things to shift our money and 6 shift our resources to best get at a better 7 quality of the data, even if we can't keep up 8 the non-response rate or the response rates 9 that we had a decade ago. 10 MR. SINGPURWALLA: Is there a 11 reason that the budgets are being squeezed 12 for the surveys? Do you have any hypothesis 13 on that? 14 MR. ADLER: I think it's -- that's 15 a tough question for me to answer. I think 16 it's just that the budgets were -- 17 SPEAKER: ------- 18 SPEAKER: Really? It's everything 19 across the board. It's not like money's 20 being shifted. 21 MR. ADLER: I don't think money is 22 being shifted away from our type of survey BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 113 1 into others, but I think, yeah, I think we 2 are just seeing a decline -- 3 MR. SINGPUERWALLA: Well, even, 4 like, surveys in general across -- even 5 outside DOE -- 6 MS. KIRKENDALL: Actually, EIA's 7 probably suffered more than other agencies in 8 terms of declining budgets. We have a little 9 graph that we put in our budget material that 10 shows other agencies --------- a little bit 11 and ours were -- 12 MR. ADLER: Do you guys do more 13 surveys than most other -- 14 MS. KIRKENDALL: No, but that's a 15 comparison to other statistical agencies, so 16 I don't know whether I've ever compared 17 counts of surveys, but we all do surveys. 18 MR. FRENCH: I don't know if there 19 is time for comment from the audience, but to 20 be fair, effective with fiscal year '06, the 21 consumption area of EIA did get a several 22 hundred thousand dollar a year boost in our BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 114 1 funding, which essentially brings us back to 2 be able to conduct our consumption surveys as 3 we have them right now, and as we've had them 4 the last couple of cycles -- once again, at 5 those same levels, with higher cost levels, 6 of course. So right now we're okay at the 7 levels that we're doing things. 8 MR. ADLER: Our survey is selected 9 on the basis of an energy measure of size, 10 which is the cost of fuels in electric energy 11 from various sources, the annual survey of 12 manufacturers. It's on the frame and we then 13 use that as a size measure. So the achieved 14 coverage rate is in terms of this measure of 15 size. It's a quantity over a quantity. The 16 response rate that you see there is mailed 17 forms and acceptable returns. We got more 18 mail than we're showing there, but some of 19 these returns brought our actual response 20 rate down. 21 What we did is we targeted the 22 cells that have the most influence on MECS BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 115 1 data, and so that's -- on the left-hand side 2 you see a cell size that's generally, in 3 terms of that cell's measure of size, we 4 called -- we rated the influence of that cell 5 on the total MECS data that we come out. 6 Really, I guess it would be the total fuels 7 correspond most to that. These are the 8 target coverage rates and, just to get over 9 that, I think if you see that -- we also 10 targeted certainty and non-certainty cases 11 differently -- we felt that not only are they 12 quantitatively different, they are 13 qualitatively different -- so that we have a 14 different non-response adjustment factor 15 based on certainty and non-certainty. 16 As you can see, we did achieve 17 actually a better coverage rate than the 18 target in all cases, regardless of our 19 response rate, and so we're going to just 20 publish that. No, we do have to show both, 21 but I think that the achieved coverage rate 22 gives us another view of the quality of the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 116 1 reported data, and I think something to take 2 into account when we look at the MECS data -- 3 when we achieve the coverage rate that we 4 were after, we stopped following up on those 5 cells, and that's where the money -- that's 6 where we can target the budgets a little bit 7 better by doing that. 8 Okay, I think we've already talked 9 a lot about this. We'll shift to an Internet 10 collection. We are going to do something to 11 get back in contact with the actual people 12 who fill out the forms earlier than we have 13 to try to boost our response rate and also 14 sell the Internet version of the form. 15 The status flags will have a little 16 bit more information because they'll be two 17 character, so I think we'll be able to 18 discern a little bit more from the status 19 flags than we have before. And then the 20 questions for the committee -- I think you've 21 already started talking about them. We 22 talked about the possibility of bias, and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 117 1 we'd like to know, boosting participation in 2 the electronic form and other performance 3 statistics that may be useful for us to 4 track, both now and in the future, and this 5 last part, what are the risks in targeting 6 non-response, really, to larger 7 establishments within a cell. That's it. 8 Thank you. 9 (Applause) 10 DR. FEDER: Can you put back the 11 questions to the committee? The slide with 12 the questions? 13 MR. ADLER: Oh, sorry. 14 DR. FEDER: I couldn't -------- 15 MS. FORSYTH: On the last page. 16 DR. FEDER: Sorry. 17 MS. FORSYTH: You want 1 and 2 or 3 18 and 4? 19 DR. FEDER: Barbara commented that 20 it's in the handout, so -- 21 MS. FORSYTH: Oh, it's in the 22 handout, okay. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 118 1 DR. HENGARTNER: Yes, Moshe? 2 DR. FEDER: My question actually is 3 why do you do it every four years? Why not 4 have sort of a rolling survey? Looks to me 5 like this activity is very intense when you 6 do it, doesn't provide you any information in 7 the three years when you don't have the 8 survey. Why not have survey part of the 9 population every year and make it easier? I 10 mean, is there a reason why it's done this 11 way? 12 MR. LORENZ: Cost. 13 MR. ADLER: Well, there is that 14 aspect of cost. I think that the starting up 15 each time has been a problem, and I think 16 even though we would go to a smaller 17 population, the actual getting them geared up 18 is -- there's a base-level cost that is 19 associated with it. I really should defer 20 that question to Dwight. That's what you get 21 paid the big bucks for. 22 DR. FEDER: But would it be cheaper BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 119 1 to not do it every four years but do it on a 2 continuing basis and -- 3 MR. LORENZ: Well, one of the 4 things -- can I add something to that -- one 5 of things I think -- maybe I'm hoping -- and 6 I think Paul agrees with me -- is that once 7 we do get this Internet data collection 8 system up and running, that we can then 9 incorporate it maybe once every other year or 10 something of that nature where it's more 11 frequent because we can -- more timely and 12 edit the data and get it out. 13 DR. FEDER: In surveys of human 14 populations, usually what one is trying to do 15 is try to balance good level estimates with 16 trend estimates, and to do that, you cannot 17 do it like that. I know presidential 18 elections cannot be held every year, but this 19 is not the case. I would try for a smarter 20 "design" that can help measure annual changes 21 with better estimates of the level of the 22 absolute number. I know from my experience BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 120 1 in Canada, where the census is held every 2 five years, that it's a huge thing to every 3 five years gear up for that. So I know some 4 things are easier that you've been 5 continuously learned from the experience than 6 to keep modifying. Here, if you you're your 7 recommendation, yeah, you should do this and 8 that, and you actually implement it, and it 9 doesn't work, you have to wait another four 10 years to change it. 11 MR. LORENZ: That's right. 12 MR. BINGHAM: Yeah. 13 DR. FEDER: With an Internet 14 instrument, it's easier to make changes on a 15 flow basis and good experience, so just a 16 question. 17 MR. FRENCH: Dwight French again. 18 You're not the first person to have mentioned 19 continuous measurement; it was mentioned in 20 terms, I believe in many past years, of both 21 our residential and our commercial building 22 surveys as well. There are some logistics BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 121 1 problems that would prevent continuous 2 measurement in those surveys -- maybe a 3 little easier to do in the MECS environment 4 than in the personal interview survey 5 environment. However, this issue of, well, 6 you can change the form and then go on -- if 7 we were to do one quarter of the sample every 8 year and try and do more continuous 9 measurement, and you kept changing the 10 questionnaire, then when you tried to do 11 rolling four-year average statistics, you 12 would essentially be rolling methodology as 13 well. And so one of the temptations to do 14 this in batch mode is you do a survey, you 15 see what doesn't work, you make your changes, 16 you do it again four years later. 17 DR. FEDER: But, Dwight, I'm sure 18 you will agree that comparing the '06 with 19 the '02 will be very questionable in this 20 case because of the weak changes that you are 21 doing now. 22 MR. FRENCH: There are changes all BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 122 1 the time that cause this issue. For example, 2 when we went from SIC to NAICS, we had a big 3 dislocation, and we did do some comparability 4 statistics, but we still had issues to deal 5 with in regard to that. Yes, we come up with 6 those issues all the time. 7 MR. SINGPURWALLA: Is the most 8 expensive part of performing the survey the 9 follow-ups to it? 10 MR. ADLER: I'd say the editing 11 portion, which is, yeah, receiving the data, 12 processing the data, and cleaning it up, 13 whether that's actually calling back or not, 14 is probably where you get most of your money. 15 I think -----, yeah. The mail-out costs are 16 substantial, but I think they're not the 17 lion's share. 18 MR. BINGHAM: Were there 19 substantially fewer errors with the 20 electronic forms, and the errors that did 21 occur, were they different than what had 22 occurred before? It's two questions. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 123 1 MR. LORENZ: I did a good portion 2 of the editing, and Vicki also did some of 3 the editing as well. And, yeah, I did notice 4 quite a few less errors. Most of it was good 5 reported data. Sometimes when there was an 6 error, or there was something that was 7 questionable, I would sometimes call the 8 respondent myself, and a lot of times, they 9 were hammering about this Internet -- this is 10 a good first step, let's go Internet, but 11 they really liked the Excel form as it was; 12 they thought it was a good first step, and a 13 lot of the errors I saw were mainly in -- a 14 good portion of them were keying, but that's 15 going to happen. They're not going to -- 16 DR. HENGARTNER: So just in terms 17 of cost, looking at this nice little graph we 18 have here, in fact, you actually look at all 19 the data you've collected, an analyst looks 20 at it and then makes a determination if it's 21 good or bad, right? And so getting better 22 data collection may not reduce the cost of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 124 1 having to go through the whole data set 2 anyway. 3 MR. ADLER: Well, I think it will 4 reduce the cost. One, redesigning the 5 questions as we look at the one thing about 6 the 'enes that needed a better word. It's 7 like having a cognitive interview as we're 8 going live. So I think that should help us 9 with the errors. And I think the big thing 10 is shifting the burden of data cleanup from 11 the analysts to the respondents, and I 12 wouldn't put that to them like that, but they 13 would be saving us money and actually helping 14 us clean up the quality of the data. 15 We would still have to run the 16 edits through, but the feeling is that the 17 data itself would be cleaner coming out from 18 the electronic versions. 19 MS. FORSYTH: Can I just 20 cosmetically make a small change to say that 21 you're not really putting a burden on the 22 respondent, you're putting it on the form. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 125 1 MR. ADLER: Okay. 2 MS. FORSYTH: I mean, it's the form 3 -- so that something that's not the 4 respondent is doing the work -- I mean, the 5 form is the technical consistency in -- 6 MR. ADLER: All right, but the 7 respondent -- 8 MS. FORSTYH: -- gives you better 9 data. 10 MR. ADLER: Yeah, and it may take 11 them five seconds longer than they did 12 before, but yes. 13 MS. KIRKENDALL: Well, but that may 14 lead to getting a phone call from EIA -- 15 MR. ADLER: Absolutely, yeah, yeah. 16 MS. KIRKENDALL: ------- 17 DR. FEDER: It's the drop-down unit 18 selection. That by itself is a huge -- 19 MR. ADLER: Right. 20 MR. SINGPURWALLA: Is the goal to 21 run it through some automatic edit checks? 22 MR. ADLER: We would be doing that. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 126 1 MR. SINGPURWALLA: And then you'd 2 spit back something like an e-mail to the 3 user. It'd be, like, hey, this looks funky 4 or -- 5 SPEAKER: No, that's not -- 6 MR. FREEDMAN: We're going to talk 7 about this a little bit this afternoon in a 8 broader context, but the general objective is 9 to provide that feedback to the user in real 10 time so they can actually correct it on the 11 screen and so the incremental amount of time 12 is not noticeable to the respondent. 13 DR. FEDER: What are you suggested 14 could be added in addition to this. It would 15 be nice because if I were a respondent, I 16 would like to write down what I actually 17 responded to -- I would pull my records -- so 18 it would be nice to get -- 19 MR. LORENZ: One of the things that 20 I think we're going to -- and I learned this 21 from the Fed CASIC -- is that we're going to 22 use the paper form as kind of a worksheet for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 127 1 them so that they can kind of work -- almost 2 like doing your taxes, they can use the paper 3 form as a worksheet and keep it themselves or 4 even save a copy on their computer. 5 MS. FORSYTH: Well, traditionally, 6 it's pretty easy to hit a "print" button as 7 well, so that you just print what they -- 8 MR. LORENZ: Yeah, create -- 9 MR. SINGPURWALLA: Or you could -- 10 MS. FORSYTH: It depends on -- 11 MR. SINGPURWALLA: ---- file of 12 some sort like a significant report that 13 could look like your old paper form even -- 14 it just has the values right in there. 15 MS. HAITOT: Can I go back to 16 Nicolas's -- my name's Vicki Haitot, by the 17 way -- we didn't look at every single case. 18 We ran searches and it would pull up things 19 that were awfully wrong. And sometimes we'd 20 have to call back. The electronic form is 21 going to keep us from having to call those 22 people because it will not have as many at BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 128 1 the other end, and if we do the drop-down box 2 for natural gas, we'll have them pick their 3 unit because a lot of the natural gas issues 4 was we had to figure out what units they were 5 reported in because it obviously wasn't cubic 6 feet. And we have to call them, and then 7 they would not necessarily be there, and 8 they'd take weeks to get back to you. 9 MR. LORENZ: Or the person has 10 moved on and changed jobs or whatever, so -- 11 MS. HAITOT: It will save a little 12 bit of time because we're not going to have 13 as many that pop out of the edits or the 14 searches we do that are wrong. 15 DR. FEDER: The issues of response 16 rates -- have you considered an incentive, 17 like fridge magnets or something? 18 SPEAKER: It doesn't work. 19 (Laughter) 20 DR. FEDER: Well, we had a survey 21 of industries where we gave some things for a 22 desktop memo holder, very nice one -- I wish BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 129 1 I could get one -- I joined that project too 2 late to get it -- things like that. So when 3 I -- maybe not a magnet. But incentives do 4 work, it's strange how -- one survey, by the 5 way, sent respondents a $2 bill, which is a 6 collector's item. That was for survey of 7 physicians, I believe. Because if you send 8 them 10 bucks, they're not impressed by it, 9 but a $2 bill -- 10 MR. ADLER: I don't think we've 11 really looked at incentives, and I guess we 12 were looking at just how much it would cost 13 to really get them to do it. And you're 14 right, eventually you get to a real person, 15 but these real people have to respond to 16 whatever pressures they're under to do other 17 work, and yeah, we just thought that what 18 could we give them that would go against that 19 that would compete with that? And we haven't 20 come up with any -- I think they're -- a 21 vacation day, that was maybe one. I don't 22 think we had the ability to do that. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 130 1 MS. FORSYTH: I think Moshe does 2 have a good point that it doesn't have to be 3 a good thing in order to create goodwill, and 4 really, especially with the mandatory survey, 5 that's what something arriving with your 6 invitation to participate does, it 7 establishes goodwill and kind of a positive 8 feeling in the respondent. 9 MS. KIRKENDALL: Do you know who 10 your respondent is beforehand? 11 MR. ADLER: Well, that's -- then 12 again, I think we have an address, a person 13 we are addressing it to based on the frame 14 information, but that's not normally -- or 15 often, I should say -- the person who will 16 eventually be completing the MECS. So, 17 again, we will be trying to identify that 18 person earlier than we have before, and that 19 means that maybe we'll be drawing the sample 20 earlier and have a pre-mail-out. So, yeah, I 21 mean, I think if we could nail down that, 22 then the idea of doing some sort of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 131 1 incentives would work better. 2 MS. FORSYTH: We're using an 3 untested model. Like, in a year, I can give 4 you results, but we have a facility study 5 that we're conducting where there's a point 6 of contact at the facility, and then, of 7 course, the person who's going to key in the 8 data. And we don't know the person who's 9 going to key in the data, but we have to 10 recruit through the point of contact, and 11 we're giving incentives at both levels: A 12 somewhat personalized incentive for the point 13 of contact, and then a food incentive that 14 can be distributed across multiple staff who 15 may be involved in actually keying the data. 16 And again, I don't know if it works 17 or not, but that was one way of getting 18 around the, we don't know who it is, and it's 19 probably someone who's not going to receive 20 an incentive package if we send just one. 21 And I'd add that the incentive that goes to 22 the point of contact is relatively BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 132 1 inexpensive. I'm kind of surprised at how 2 inexpensive it is. 3 MS. KIRKENDALL: What is it? 4 MS. FORSYTH: For us, it's a 5 portfolio -- it's a letter-sized -- it's not 6 leather-bound, but it has a pad and a pencil, 7 and it has the study logo on the cover, so 8 it's kind of nice looking, and it's something 9 people can carry to meetings and make notes 10 in. And a further advantage is on the 11 inside, we have -- I shouldn't be giving away 12 all our secrets, but -- 13 (Laughter) 14 MS. FORSYTH: But on the inside, 15 you have point of contact information on kind 16 of a yellow sticky-type thing that they can 17 pull off, so it actually serves multiple 18 purposes, the incentive. 19 DR. FEDER: Can respondents, if 20 they are too busy -- you mentioned that time 21 was -- can they start filling the form and 22 then save it and go back to it -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 133 1 MR. ADLER: Yeah, definitely. 2 MR. LORENZ: Yes. 3 DR. FEDER: Okay, good. 4 MR. LORENZ: Part of the system -- 5 DR. FEDER: Good, good. 6 MS. FORSYTH: Interesting to 7 measure when and how often that happens. 8 Just -- 9 DR. FEDER: I once quit responding 10 to a survey because I had to go, and the 11 system didn't let me save it, and I just -- I 12 don't want to use any bad word, but I just 13 said, I'm not going to do it because -- 14 MR. LORENZ: Yeah, I mean, we -- 15 DR. FEDER: I hope you are not 16 writing that. 17 (Laughter) 18 MR. LORENZ: Yeah, the system we 19 have to use because we are working through 20 the Census Bureau, because we do collect the 21 data under Title 13, we could do one of two 22 things. We could either create it ourselves BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 134 1 and program it ourselves, which could take 2 years and months, or we could use their 3 system, which is Census Taker, and that seems 4 like a more viable option to me. Is it 5 perfect as far as collecting the data 6 perfectly all the time? No. But then again, 7 no system is. So it is what it is, and 8 that's the best we have available. 9 MR. SCHIPPER: With Census Taker, 10 is it a programming language that's applied? 11 Because I know that other surveys that -- us 12 and Dwight's group -- the system that's used 13 is VLAZ, and we have a VLAZ programmer. Do 14 you have a Census Taker application that you 15 can then program your own -- 16 MR. LORENZ: No, there's no 17 programming language. It's going to be used 18 throughout the whole Internet and all that. 19 And Stan will get further into that this 20 afternoon because we looked with the IDC on 21 part of that team as well, and we looked at 22 Census Taker as part of the IDC team, and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 135 1 that was one of the systems that we looked 2 at. 3 DR. HENGARTNER: In terms of 4 statistics, is everybody satisfied? I mean, 5 I'm looking at the achieved average coverage 6 rate of maybe 85 to 90 percent. Should we be 7 concerned with non-response? That's a pretty 8 nice number, just on face value. 9 MR. ADLER: I guess, again, that's 10 the coverage rate. Our non-response is, in 11 terms of actual forms, going down. I'd be 12 happy not to, but I think there is a concern, 13 and I think there's an OMB standard that we 14 are in danger of going below the threshold, 15 and that has me worried. And I would like to 16 convince OMB that even if that happens by one 17 percent, we are trying to maintain this 18 coverage rate that is still maintaining the 19 quality of the data. 20 MS. KIRKENDALL: I think we're ---- 21 coverage rate is probably more important 22 indicator than actual response rate. You BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 136 1 might have to make a case, but I think you 2 can make a case. 3 MS. HAITOT: We called a lot of the 4 large guys that didn't respond to it to get 5 the coverage rate up there. 6 DR. FEDER: That partially depends 7 on the level of the estimate that you're 8 interested in. For national estimates and so 9 on, the coverage rate is what you should 10 worry about. If your data contain some 11 specifics on units that are not contributable 12 to the overall coverage rate important for 13 some ----- of the market, then you should be 14 interested in those particular response 15 rates. Because if I wanted to use the data 16 to do some design on small companies, their 17 contribution to the response rate might be -- 18 MS. KIRKENDALL: Well, if you want 19 to do a study on small companies, then a 20 response rate may not be quite what you -- 21 DR. FEDER: Small companies or some 22 more esoteric types of energy use. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 137 1 DR. HENGARTNER: If I have the 2 choice of improving coverage rate between the 3 certainty and the non-certainty samples, 4 which one would you focus on, Moshe? 5 DR. FEDER: I don't think I have an 6 easy answer to this question, frankly. It 7 again depends on the use of the data. 8 Certain units tend to be bigger and easier to 9 approach; they had a person who's easy to be 10 tasked with responding and so on. But I 11 think often uses data on smaller companies on 12 the non-certainty units. I would say you 13 need to have good response rates from both. 14 I wouldn't target one in particular. 15 MS. KIRKENDALL: See, but that's 16 always a tradeoff, of course, because if you 17 take the same resources and you don't focus 18 on the big ones, then your overall coverage 19 rate is not going to be very -- 20 DR. FEDER: That's what I think -- 21 well, Mark, maybe when he gets back, we 22 should ask him this question because that's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 138 1 one question for a specialist and a data user 2 rather than a statistician. 3 MS. FORSYTH: I want to go back to 4 your point about the small niches that you 5 might be missing and whether you know about 6 those. That's not really a question of bias, 7 right, because if they're small, they 8 wouldn't necessarily contribute much to -- 9 DR. FEDER: That's not -- 10 MS. FORSTYH: -- so it's -- 11 SPEAKER: It's a different -- 12 DR. FEDER: The bias is not on a 13 national level, the bias is at the small area 14 level. 15 MS. FORSYTH: Right, right, 16 within -- 17 DR. FEDER: Yeah. 18 MS. FORSYTH: -- so you wouldn't 19 see it. Okay. 20 MR. HILL: But are you going to 21 mention you're doing a stratified sample and 22 you really want -- there are two questions: BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 139 1 One is the overall number -- 2 DR. FEDER: Right. 3 MR. HILL: -- and one -- 4 DR. FEDER: Exactly, exactly. 5 MR. HILL: --------. 6 DR. FEDER: Exactly. 7 MS. KIRKENDALL: Well, that's 8 another difficulty. --- is a very large 9 survey that tries to be everything to 10 everybody, so targeting that one 11 particular -- 12 DR. FEDER: One size fits all. 13 MS. KIRKENDALL: -- user's needs is 14 impossible. So I think that's why the 15 targeted response rate was going to try to 16 get at least the national level of estimates 17 or a pretty good quality, and yeah, you get 18 down to regional or however you cut it down 19 -- it might not be quite so -------. 20 MR. ADLER: Yeah, I worry when 21 people do microdata work, and they start 22 using ---- maybe by industry, by region, and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 140 1 really getting down into trying to identify 2 differences there. We really don't guarantee 3 that the sample works at that level. 4 Certainly, if you're really cutting it down 5 and if people have the option, those that 6 have census agent status to do those things, 7 and it's also possible that the data aren't 8 as clean for those establishments, for those 9 items at that point because they don't impact 10 on the tables that we present, but they may, 11 in fact, impact on the researchers' data at 12 that point. 13 DR. FEDER: But my point is -- and 14 I remember when I joined this committee three 15 years ago that I was surprised that certain 16 establishments were not included in some data 17 collection efforts just because they were 18 small. And that is not the current 19 statistical practice; the current practice is 20 to downweigh them, but not to exclude them 21 altogether. And then when I look at these 22 coverage rates, which are good, and we say BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 141 1 that's okay, even though the response rate 2 may be going down, I just want to say, you're 3 right for the national estimates, and that 4 might be okay. 5 I think we need to ask a data user 6 for their view on that, because they might 7 tell you that some of the interest might be 8 in new technologies that might not be covered 9 because their contribution to the coverage 10 rates might be low, but the interest might be 11 going high because of the trend, because we 12 know that in the future, maybe their niche is 13 going to be more important, and we might miss 14 a new development in the market that analysts 15 might want -- also my concern was the 16 four-year frequency because did we know four 17 years ago what's going to be with the energy 18 market, what we are seeing now? No. So this 19 is why I'm asking. 20 But again, my only knowledge about 21 energy is how much I pay for the gas I buy, 22 that's why I think sometimes small-area BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 142 1 estimation -- I'm sorry, I didn't hear the 2 technique, but the interest in small-area 3 statistics might be worth it. That's all. 4 MR. SCHIPPER: To answer your 5 question, the Census Bureau conducts the 6 census every five years, but also conducts an 7 annual survey of manufacturers, which does 8 not embody the scope or level of detail of 9 energy-related information that the MECS 10 would cover. But there are trend data 11 available related to energy costs and 12 electricity usage on an annual basis. So 13 essentially MECS comes around every four 14 years to kind of benchmark the entire energy 15 portfolio for manufacturing. 16 DR. FEDER: And I presume that some 17 of the modeling is taking all that data into 18 account, right, so we can -- 19 MR. SCHIPPER: In other words, the 20 energy modelers at EIA enjoy it when MECS 21 comes around, but they have alternative data 22 for those intervals --- MECS ----. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 143 1 MR. GIFFORD: Just another comment 2 on the small establishments. The ones that 3 are excluded from the MECS frame -- and Bob, 4 you may remember this better than I -- I 5 think are only account -- in terms of 6 coverage, about two and a half percent. So 7 if -- and you really need to think about the 8 motivation for the survey. For -- want to 9 concentrate on publishing total consumption 10 numbers, then those guys may need to go just 11 because it's not -- 12 MR. ADLER: They represent about 13 150,000 -- these are rough numbers -- 150,000 14 out of about 350 -- 15 MR. GIFFORD: Right. 16 MR. ADLER: So by cutting them out, 17 we've lowered the population number by 18 150,000, we're down to 200,000, but we also 19 still maintain 98 percent of the payroll in 20 manufacturing as a measure of the frame 21 coverage. So I think it's a good tradeoff, 22 but obviously when you talk about BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 144 1 establishment counts -- and some of our data 2 do show that -- we have to be clear that 3 that's what we're talking about, that this 4 particular population -- 5 MS. KIRKENDALL: This survey is 6 more like other agencies' probability 7 surveys, it's not a cutoff survey. 8 MR. ADLER: No. 9 MS. KIRKENDALL: Every survey has 10 some cutoff. A lot of the Census Bureau's 11 cut off at five establishments or something 12 because you just can't get any information 13 for the itty-bitty tiny guys. And that's 14 where our frame comes from. We do have other 15 cutoff samples, but that's not what this one 16 is. And we do have a probability sample of 17 the smaller guys, and they, too, try to get 18 the response in and just can't concentrate on 19 it. They concentrate instead on getting the 20 response rates up on the big ones, the most 21 energy-intensive ones. 22 DR. HENGARTNER: The one thing I'd BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 145 1 like to see, which you haven't shown us, what 2 is the non-response rate on the follow-up? 3 Because I'm sure not every follow-up 4 survey -- 5 MS. KIRKENDALL: Interesting 6 question. 7 MS. FORSYTH: Are you talking about 8 unit follow-up or item follow-up? 9 DR. HENGARTNER: I would say unit 10 for sake of argument. Yeah, not -- item, 11 it's too much work; unit -- I'm going back 12 then ask the company, please give us the 13 data. Now, I'm not convinced that they're 14 going to respond 100 percent to your 15 follow-up. 16 MR. ADLER: Oh, they don't. 17 DR. HENGARTNER: They don't. So, I 18 mean, one of the questions to ask, tell me 19 what statistics would be interesting to know 20 -- well, follow-up non-response is in a way 21 indicative of actually diminishing returns -- 22 MR. BINGHAM: You might want to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 146 1 measure whatever your -- these are the large 2 establishments here and the small ones as 3 well -- to see what information on -- 4 DR. HENGARTNER: And send it to the 5 IRS. 6 (Laughter) 7 MS. KIRKENDALL: You may have an 8 indication of that. Did you follow up with 9 all of the non-respondents in the certainty 10 groups? No. 11 SPEAKER: Is that a question? I 12 mean -- 13 MS. KIRKENDALL: Well, yeah, I was 14 asking the Census Bureau who knows these 15 things. 16 MS. BUCCI: No. We start with the 17 larger ones. 18 MS. KIRKENDALL: Okay, so even 19 within a certainty group, you may not do 20 follow-up with everyone of the -- 21 MS. FORSYTH: But once you get your 22 coverage rate, you stop, right? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 147 1 SPEAKER: Right. 2 MS. KIRKENDALL: Okay, right. 3 DR. FEDER: But you need not only 4 the total numbers of non-response, you need 5 the characteristics so one could even build a 6 propensity model for making future 7 adjustments. Because if you know more about 8 the type of establishments that do not 9 respond, you can do something with it, such 10 as use non-response adjustment that's taking 11 into account the industry, the size, the 12 location, and whatever. 13 MR. BINGHAM: This plot might 14 actually help you in some sense and tell you 15 ----. So I guess my question would be in 16 terms of coverage on certain mixtures of 17 butylene, propylene fuels, you might have an 18 idea of who the users -------. 19 MR. LORENZ: That's what we do, 20 yeah, yeah. 21 MR. BINGHAM: And if you somehow -- 22 you know you got really ----- some of these BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 148 1 particular ----. 2 MR. ADLER: I should say that when 3 I -- this was a special final non-response 4 follow-up, which is the calling. We do send 5 out letters to everybody who doesn't respond. 6 And it lays out the mandatory requirement, 7 and they get as many as two of that -- 8 MS. BUCCI: Yeah, it's two mail 9 follow-ups. They actually get the form and a 10 letter saying that they did not respond, and 11 then the third follow-up would be phone 12 calls. 13 MR. ADLER: Right. And so I didn't 14 make that clear, but we're not letting people 15 off the hook right away. We do have a 16 general follow-up -- a couple of ones -- and 17 it doesn't succeed in scaring them enough, so 18 we then go to this other one. I'm sorry, but 19 that's what a mandatory requirement is 20 supposed to do, and it's not working. That 21 piece of it doesn't work as well. 22 MS. BUCCI: One of the things you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 149 1 may want to consider for Census -- I mean, 2 obviously, Census is also a mandatory survey 3 -- when we are having problems with response 4 rates, we actually have real high-up letters 5 coming from counsel or whatever going 6 directly to the presidents of the companies 7 saying that this is mandatory, here are the 8 penalties for not filing -- 9 MS. KIRKENDALL: Right, send it to 10 somebody other than the contact -- 11 MS. BUCCI: -- we send it to 12 somebody pretty quickly, those type of 13 things, and we don't send it to everyone, but 14 we do target certain establishments or 15 companies to get those letters. So that 16 might be a possibility. 17 MR. ADLER: Tom called -- 18 MS. BUCCI: Yes, Tom -- 19 (Laughter) 20 MR. LORENZ: I'm not ashamed. 21 SPEAKER: Is that fair? 22 MS. BUCCI: I'm sorry -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 150 1 MR. FREEDMAN: Do you have any 2 measure of the effectiveness of that? 3 MS. BUCCI: I'm not sure. 4 MR. LORENZ: And I talked to the 5 CFO. 6 MS. KIRKENDALL: He called the 7 president and talked to the CFO, he says it 8 works. And so other ways of contacting them 9 may be good, too. Yeah, the other thing -- 10 just ------ talk may be redesigning our 11 second and third contact letters to try to 12 make it sound like -- I don't know if 13 threatening would help, but to try to make it 14 sound like they really ought to respond for 15 some reason, and then -- it's an area of 16 research, I'm sure. I mean, everybody's 17 worried about response rates, and I was 18 thinking about how to encourage people to 19 respond to your surveys. 20 MS. FORSYTH: Right. Not many 21 people that I work with get the advantage of 22 it being mandatory -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 151 1 MS. KIRKENDALL: Right. 2 MS. FORSYTH: -- and so -- 3 actually, an experiment in a mandatory 4 context -- I mean, the Census has probably 5 done them, experiments of the tone you take 6 in these follow-ups and the impact of the 7 tone, but I don't think it would have be a 8 big experiment to -- 9 MS. KIRKENDALL: Yeah, to get -- 10 right, to -- actually, that wouldn't be so 11 bad if they could build into the -- just -- 12 DR. FEDER: Yeah, but non-response 13 could be for failure to comply could go to 14 the wrong contact person. 15 MS. FORSYTH: That's right. 16 DR. FEDER: Point of contact. 17 MS. FORSYTH: Point of contact. 18 Yeah, point of contact is a huge issue. 19 DR. FEDER: -- address or sometimes 20 company going out of business or changing 21 ownership, but I presume, then, those letters 22 hopefully come back to you. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 152 1 MR. LORENZ: Not always. 2 MS. FORSYTH: Not if it's the wrong 3 person. 4 MS. BUCCI: Part of what happens is 5 our point of contact comes from the annual 6 survey of manufacturers, and that's updated 7 in our systems. When we get a contact for 8 the MECS survey, we can keep it in StEPS, but 9 we don't pull our frame from StEPS, so we 10 don't have a place to keep that contact, so 11 analysts are going back and looking at it -- 12 may not be printed on the form. So they are 13 typically going to an accountant, and we want 14 to hit the plan engineer for the survey. 15 MS. HAITOT: But a lot of times, 16 the MECS respondent isn't there after four 17 years either. 18 MS. BUCCI: That's true. 19 SPEAKER: Right. 20 SPEAKER: That's a long time. 21 MS. BUCCI: One of the things that 22 I think we've tossed around, too, for the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 153 1 electronic reporting is possibly doing 2 company visits up front, offering the service 3 of the Census Bureau or EIA staff to go and 4 help them -- yes, Tom -- 5 MS. HAITOT: And Vicki. 6 MS. BUCCI: -- and Vicki to go out 7 and help them fill out their forms and -- 8 through the census-taker and try to encourage 9 a response in that way. But that costs, too. 10 MR. GIFFORD: One more comment on 11 the MECS is -- when we met with the 12 census-taker folks, this is quite a different 13 survey than anything they've used it for 14 before in terms of size. And it's every four 15 years -- and EIA -- the Internet data 16 collection surveys, for the most part, are in 17 smaller surveys that have responses every, 18 say, month or week or year, where, with this, 19 you kind of lose the learning by the 20 respondent, you have contact problems, so we 21 don't really know how it's going to go down, 22 but hopefully it should be fairy successful, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 154 1 but you don't know at this point because we 2 asked for some statistics on a comparable 3 survey, and they just didn't have it at this 4 point. 5 MR. ADLER: Well, I think, though, 6 that this does set us up better for the 7 future than sticking with the electronic 8 questionnaire. I think, as Tom was saying, 9 by doing that we may have the ability to go 10 out more frequently, and then that learning 11 curve would be improved. But yeah, the real 12 reason, I think, is that even if no one has 13 seen it, if it's brand new to every 14 respondent that comes out, we really think 15 that we will just have a cleaner data set 16 faster than we would if continuing with the 17 written form. 18 MR. SINGPURWALLA: Well, it's 19 automatically consolidated, too, which is 20 nice. So analysis will be a lot cleaner 21 and -- 22 MR. ADLER: Yeah. So eventually we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 155 1 cut out keying entirely, and that's another 2 five -- certainly a block of money that could 3 be shifted to other areas. 4 DR. HENGARTNER: Nagaraj, don't go 5 out. 6 (Laughter) 7 DR. HENGARTNER: I think we're 8 wrapping up. We have the other break-out 9 session coming up from downstairs, so I'd 10 like to thank everybody for their comments, 11 and we'll be summarizing this break-out 12 session shortly. Thank you very much. 13 (Applause) 14 (Recess) 15 DR. HENGARTNER: I'd like to have a 16 summary of both break-out sessions. Our 17 first session was about the 2006 MECS survey 18 about past performance and predictions of 19 what's going to happen, and I'd like to 20 invite Mr. Moshe Feder to summarize our 21 discussion we had earlier today. 22 DR. FEDER: Thanks, Nick. First of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 156 1 all, MECS stands for Manufacturing Energy 2 Consumption Survey, and what's interesting 3 about this survey, it's done every four 4 years. And that's for cost consideration and 5 I may say a little bit about this. There's 6 some information that was provided to me 7 after the break-out session. 8 The last time MECS was conducted 9 was in, I guess, '03 on the '02 data, and 10 four years later, in '07, MECS is going to be 11 conducted on the '06 consumption, so that's 12 why it's called 2006 MECS. In 2002, 13 respondents had the option of responding 14 either using a paper instrument or filling 15 out an Excel equivalent of that that was made 16 very much similar to the "C" form. 17 There are three forms, "A," "B," 18 and "C," "C" being the most complex, and 19 that's the instrument that energy-intensive 20 establishments receive or ones that have 21 complicated energy flows. The experience had 22 been generally that with the Excel BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 157 1 instrument, there was less need for 2 correction of information. 3 In addition, for the '06 survey, 4 the intent is to have some further 5 enhancements; for example, one that I really 6 liked is having a pull-down menu for the 7 choice of units because it turns out that 8 different establishments use different units, 9 and that caused some problem in the past. 10 Obviously, if you use a unit that's 10,000 11 times another unit, your data will be quite 12 different. So that's another enhancement. 13 They do have attached to every measurement 14 flags that indicate whether it's the original 15 raw data as provided by the respondent, an 16 analyst correction, imputed data, information 17 coming from another survey, or a system edit. 18 I won't go into the specifics -- 19 that would take too much time -- but there 20 was an interesting -- as much as possible, 21 having the original data; obviously, edited 22 data and system-edited data and so on are BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 158 1 less desirable. There was less of that with 2 the electronic format, and hopefully in the 3 past, all respondents, with a very few 4 exceptions, will use the electronic form. 5 We discussed also the issues of 6 response rates and coverage rates and which 7 measure to use and what should be done to 8 improve response rates. 9 Now, what is a coverage rate? 10 Turns out that a few establishments dominate 11 the estimate for any particular kind of 12 activity, and those contribute more, 13 obviously, to the coverage rates. Small 14 establishments don't have big impact. And 15 the target, if I got this correctly, for the 16 coverage rate is about 90 percent. There has 17 been a slight decline in response rates, but 18 for most purposes the big interest is in the 19 coverage rate because as long as you cover -- 20 it's 90 percent of the target measure, then 21 you're okay. However, there are some OMB 22 requirements for response rates as well. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 159 1 One of the issues that I asked is 2 why have this every four years. Obviously, 3 there's a cost involved here, but why not 4 divide it over the four years, and why -- 5 explain to me something which I think is a 6 very good point. There are two reasons, 7 actually, to not have it every year. One is 8 if you divide it over four years, then you 9 get lots of fluctuations in your estimates 10 because of the skewed nature of the 11 distribution. If in one year you survey the 12 bigger establishments, you get different 13 estimates than in other years. Another is 14 the disclosure analysis, which is required 15 and actually mandated by law. The smaller 16 the sample you have, the more likely you are 17 to disclose information, and also it will 18 require more resources. 19 So it is a cost-effective thing to 20 do it every four years, and someone else 21 provided another point of information: There 22 is annual collection by the Census of some BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 160 1 relevant information, so it's not like every 2 four years we collect data, and then we have 3 three years of total -- 4 MS. KIRKENDALL: Preparing. 5 DR. FEDER: Yeah. And so we do 6 have the ability to use for modeling some 7 measurement of change across years, not just 8 every four years. 9 Now, I come into the issues of 10 non-response and how to improve response 11 rates and what should the measure be. And I 12 think people generally accepted that coverage 13 rates are more important than response rates, 14 although response rates might be of interest 15 as well, especially since you can have 16 biasing in the smaller domains. But overall, 17 the coverage rates are a good measure of the 18 effectiveness of the survey. 19 How do you improve response rates? 20 Well, follow-up is one tool, and we know that 21 even with follow-up, not everybody responds. 22 We also need to have not just information on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 161 1 response rates, but information on the 2 non-respondents. Who are the 3 non-respondents, how to they differ from the 4 ones that responded, especially after the 5 follow-up, so you can do some non-response 6 adjustments and know who you're missing. 7 We also talked about, briefly, the 8 issue of incentives. You can improve 9 responses by handing out some incentives. 10 Those should be given both to the 11 point-of-contact person and also to people 12 who actually do the entry of the data, and we 13 give some examples from other surveys. In 14 addition, we know that some non-response is 15 caused by bad contact information; especially 16 since this is done every four years. Maybe 17 in the intervening time some companies 18 changed officers or structure, addresses, and 19 so on, so attention must be given to that as 20 well. 21 We did not adhere to the list of 22 questions very specifically, so I'm not going BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 162 1 to go through each one individually, but I 2 think that pretty much covers what we talked 3 about. 4 DR. HENGARTNER: Thank you very 5 much, Moshe. Anybody else who was in the 6 break-out session who wants to add something 7 to Moshe's résumé? Okay, thank you very 8 much. I'd like now to invite Neha Khanna to 9 summarize the discussion on a new oil 10 production representation for the SAGE model. 11 MS. KHANNA: Thank you. This was a 12 very lively discussion -----. Just to give 13 the rest of the committee a brief overview of 14 what the SAGE model is, it's, as we 15 understood eventually, a 16-region, 16 42-end-use sector, linear programming, 17 cost-minimization model that takes demand as 18 given, and also it's a model for non-OPEC 19 production, it takes OPEC production or 20 price, depending on which area you want to 21 look at it, as given. And it took us about a 22 few minutes to actually reach that level of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 163 1 understanding because we had so many 2 questions, I think we didn't let the 3 presenters even finish what they had to say. 4 The questions were, I think, like 5 the other group; we didn't exactly answer the 6 questions that we were asked. We didn't even 7 allow the presenter to reach their questions 8 because we kept butting in. But I think part 9 of the reason for that is that the model is 10 still under development, and so we wanted 11 more details than were, in fact, brought to 12 the tables by the presenters partly because 13 they don't as yet have those details 14 themselves. 15 So I think the one punch line -- 16 I'll come to the detailed questions -- but 17 our one punch line was, we want more 18 information, we want details, details, and 19 more details, and we want to see more 20 presentations on the SAGE model and how it's 21 being developed. 22 So that's the summary of the entire BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 164 1 session. The specific questions that we were 2 particularly concerned about were, for 3 example -- I'll take some examples. One is, 4 how do we -- how will the model -- there's a 5 cost model and oil production model which 6 sort of gives us resource codes for different 7 non-OPEC regions, and that goes into a SAGE 8 model that I understand was originally 9 developed for actually greenhouse gas 10 emissions and is now being expanded for oil 11 production. 12 And so one of the questions that we 13 spent a lot of time on was, how is reserve 14 expansion and technology incorporated into 15 the oil production model. Because the main 16 question -- the main source of concern there 17 was the USGS uses -- its assumption that the 18 technologies that are used and are available 19 in the U.S. and North Sea are what are going 20 to be applicable and available in the rest of 21 the world, and that's a very powerful 22 assumption, and if that's an assumption that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 165 1 EIA is happy to buy into, that must be 2 clearly and explicitly stated, because you 3 may have users of the model who may not buy 4 into that assumption. 5 So this is sort of the question of 6 what are you assuming, make it explicit, and 7 how sensitive are your modeling results to 8 these assumptions; it's something that we 9 would definitely like to have a lot of detail 10 on. 11 A question that came up again was 12 to sort of -- may be more difficult for the 13 EIA to explicitly answer, but something to at 14 least keep in mind is that the USGS 15 estimates, if we look at the historic world 16 assessment model, the probability curves for 17 undiscovered resources have shifted 18 rightwards, especially in the last 19 assessment, not so much in the previous 20 assessment. And the question is, if they're 21 going to use the SAGE model to project 30 22 years onwards, as it's being done today -- or BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 166 1 maybe up to 2100, another figure that was 2 thrown up -- what are we going to assume 3 about those curves? Are we going to assume 4 they're static, are we going to try and 5 project what the USGS is going to do? That's 6 a question -- the longer the time frame gets, 7 the more important that question becomes. 8 Another question that we wanted to 9 really get more detail on was, based on 10 non-oil prices -- because that's what we 11 understood that the model is taking as an 12 input -- how does the model convert -- or how 13 do we model the conversion of resources, 14 undiscovered resources, into reserves, proven 15 reserves? How does that happen? That detail 16 was not available as yet, and it's, I think, 17 one that is going to require a lot of 18 discussion and the committee will have a lot 19 of interest in that detail. 20 Likewise, we talk about developing 21 Hubbard curves, which are just sort of oil 22 production curves for each reason, rise or BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 167 1 each peak or decline, and the location of 2 those curves is sensitive to price ------ 3 options. 4 And so the question was, in the 5 future, as prices and technologies change, we 6 may have expected prices and expected 7 technology curves. How are these curves 8 going to shift, what is the mechanism built 9 into the model that shifts these curves, 10 because that's an important detail, an 11 important assumption that people will want to 12 know about. And then in the end, we went 13 into another discussion which maybe should 14 have actually come in the beginning, which 15 is, how -- the basic question was how complex 16 does the model need to be. Because as we 17 understand this, there is a huge number of 18 equations and a large number of variables, 19 but is that driven more because this is a 20 linear programming model? 21 We could actually collapse the 22 whole thing into a much smaller model by BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 168 1 removing nonlinear curves and maybe growth 2 rates in those curves, which is just one 3 parameter that can carry the entire curve. 4 A more fundamental question is what 5 is driving this model. Is it because we had 6 it and we want to use it? Is it because 7 we're trying to answer a certain kind of 8 question? That's something which we really 9 got into in the end, but I think it's an 10 important question to ask. I think I'm going 11 to leave it there. As I said, it was a very 12 lively discussion. I've just given you a 13 flavor of the questions that we asked. I 14 hope I've picked up the most important ones, 15 but if other people who were in the room 16 found something else more important or 17 equally important, please add to my -- 18 DR. FEDER: I have a question. 19 What do actually the consumers of this model 20 get? Do they get the actual model or outputs 21 from this model? 22 MS. KHANNA: We didn't ask that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 169 1 question, so we don't know the answer. I'm 2 assuming the outputs from the model, but -- 3 MR. STAUB: Most of the output of 4 the model is used for producing the 5 International Energy Outlook publication. 6 And in theory -- we provide, like, for 7 example, capacity data on electricity 8 generation for the world to various consumers 9 and things like that. 10 DR. FEDER: What I meant is, if a 11 user wanted to do sensitivity analysis, do 12 parameter sweeps, or anything to use the 13 actual model, could they do it, or -- 14 MR. STAUB: If they have two years, 15 they could probably learn it and manipulate 16 it and things like that. It's -- 17 DR. FEDER: Okay. 18 MS. BARDEN: Well, and also, a lot 19 of what we were talking about in the meeting 20 was a new supply representation for just 21 world oil, which would -- 22 MS. KHANNA: That's right -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 170 1 MS. BARDEN: -- still be separate 2 from SAGE. So in a sense, we within EIA 3 would actually be consumers for those two 4 models, and we would be able to change 5 assumptions and get different results out 6 that we could then feed into SAGE, and the 7 general public would be the consumers of the 8 SAGE results of those different cases. 9 MS. KHANNA: Yeah, that's a really 10 good point. Thank you for mentioning that. 11 We didn't talk about SAGE, really, we talked 12 about the oil production model because we got 13 all so excited about that. 14 DR. HENGARTNER: Any more questions 15 and comments since it was such a lively 16 discussion? Thank you very much. We're 17 getting along. 18 At this stage, I'd like to invite 19 additional comments from the committee if 20 anyone else wants to add to those 21 discussions. And if not, I'd like to invite 22 the public for general comments. Okay -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 171 1 MR. HILL: Will you take minor 2 comments on the -- oil, you're looking at 3 country-level data or do you actually have 4 company level? What are the units of 5 analysis of the larger oil model? 6 MS. BARDEN: It's based on USGS 7 assessment to a large extent, so their 8 granularity would be the lowest we could go 9 down. The results of the model of this oil 10 supply model would be at the country level; 11 SAGE is at a 16-region world level. 12 MR. HILL: I think some of the 13 political discussions might be at the level 14 of the company. I guess you won't pick that 15 up. You won't pick that up. 16 MS. BARDEN: No. I mean, 17 especially once we roll things up into the 16 18 regions within SAGE -- 19 MR. HILL: It's all aggregated. 20 MS. BARDEN: Yeah. 21 MR. CLEVELAND: It's built from the 22 ground up with the USGS data, which is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 172 1 geologic province-level data, so you probably 2 put political boundaries over that and 3 ascribe their numbers to particular nations, 4 I'm assuming. So we're stuck with the 5 geologists' data. 6 DR. HENGARTNER: I just want to 7 remind the committee -- I will inform you 8 that many years ago the statisticians said, 9 bring us the problems early so we can have an 10 input. Right? And we keep repeating that 11 mantra, we want to see the data early on. 12 Well, this new oil production model 13 is exactly such an attempt, and our input to 14 them is probably going to have a big impact 15 on how things are going to evolve in the 16 future. At least that's the thought. So, 17 again, want to leave it open that if you have 18 ideas and things that come to mind about this 19 particular model and endeavor, do not 20 hesitate -- now is the time to speak up 21 because now is the time that we actually can 22 bring input and expertise that, as a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 173 1 committee, EIA will value. 2 Yes, Neha? 3 (Laughter) 4 DR. HENGARTNER: Oh, oh, you opened 5 Pandora's box. 6 MS. KHANNA: Yeah, you put me -- I 7 want to take a yard, of course. Well, I 8 think my personal reaction to that is that's 9 fantastic, but we need a lot more detail, 10 which I understood the constraints today, it 11 was not possible to provide us those details. 12 But we would love to have more details so we 13 can chew on them and then spit them out in 14 different directions and reach some kind of 15 consensus about where the model should go. 16 DR. HENGARTNER: So in that case, 17 how do you want those details? Do you want 18 them to e-mail them to you? 19 MS. KHANNA: I'm happy to look at 20 them. I will not speak for other people. 21 DR. HENGARTNER: Okay, you've 22 written that down, right? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 174 1 (Laughter) 2 MR. SINGPURWALLA: Like more 3 collaborative, instead of just seeing a quick 4 pop twice a year. Would people be interested 5 -- and I don't even know if this is possible 6 -- in getting back and forth with people who 7 would be focusing on the model throughout the 8 year? Or is that putting more on your plate 9 than you want? 10 MS. KHANNA: Well, yes and no. Of 11 course, it's putting more on my plate, or on 12 anyone's plate on the committee. I think 13 what was great about the break-out session 14 was that different people in the room had 15 different strengths and different expertise, 16 and, as a group, it worked really well, I 17 think. And I think that was exciting about 18 the session. 19 So I think it may be that if we 20 know, for example, that we're going to 21 develop this model, first of all, we should 22 probably have it more frequently on the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 175 1 agenda. And secondly, provide a lot more 2 detail well in advance, not just a week 3 before -- and I know we've talked about this 4 and there are reasons why this happens and 5 good reasons. But I think something like 6 this, which is going to be so important, 7 maybe if we got three weeks before or several 8 days before so we could really have a time to 9 go into it and think about it -- maybe even 10 e-mail each other about it or something -- 11 that would be great. 12 DR. NEERCHAL: I think today, the 13 break-out session, it seems to me the details 14 needed were on the SAGE model, to some 15 extent. I think we needed to know what was 16 in the SAGE model, and so some of it was put 17 into the rhetoric that is kind of a 18 continuing work based on SAGE model, and it's 19 kind of a -- they need to know who the 20 audience is going to be, whether they need 21 the discussion on SAGE model or not. I don't 22 know how you're going to coordinate that back BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 176 1 and forth, just to figure out -- I think it 2 is fair to say we need the information more 3 than one week ahead of time, but at the same 4 time, you send it to me two months before the 5 meeting starts, I'm not even going to open 6 that attachment sometimes. So I think we are 7 equally guilty of that kind of -- 8 DR. HENGARTNER: Yes? 9 MR. CLEVELAND: I agree 10 wholeheartedly, and I think this is a little 11 bit different issue. You're talking about 12 modeling world oil production, which is the 13 hottest energy issue on the planet right now, 14 and so clearly you're going to have to be 15 extra careful about how you document it and 16 so on. And it would be useful -- the devil 17 is in the details, and it also -- our 18 interaction with you is -- you have a 19 contractor here who's doing a lot of the 20 work, the EEA, and so if you really want to 21 get into some of the important technical 22 details, you may want to bring in someone BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 177 1 from that shop to be involved in these 2 discussions because -- I mean, Phillip and 3 Justine did a good job with what they had, 4 but some of the shapes of these functions -- 5 a lot of the technical details are only known 6 by the people who are really working on them. 7 So it does speak to the issue of -- 8 on this particular model, I think you want to 9 bend over backwards to make sure that you're 10 getting all the advice that you possibly can 11 be brought to bear on it because there are 12 going to be a lot of people looking at what 13 the EIA has to say about the peak in world 14 oil production. 15 DR. HENGARTNER: Moshe? 16 DR. FEDER: Just on the advance 17 receiving of information, Nagaraj, I think if 18 I -- if Nick or -- when you become our 19 chairperson, send me the information. I'm 20 going to have to discuss it; I can assure you 21 I will open it. It's like when you get the 22 reminder from the general, you actually -- so BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 178 1 I think -- I actually -- in the past, I 2 really requested it. It's good to have the 3 information in advance, especially if I'm 4 supposed to be discussing it, or even be in 5 the break-out session for that. So a 6 selective provision of this material will be 7 useful to those who actually have to say 8 something meaningful about it. 9 But if this is maybe feasible, 10 maybe it could be posted on the Web page and 11 all you need to do is let her know when it's 12 available because I know how people are 13 working right up to the last minute, revising 14 the material. If you put it on the Web page, 15 I would look at it, not if I'm not in that 16 break-out session, frankly, but if I am, I 17 will, just because maybe I'm slow and I need 18 the time to think about it. 19 DR. HENGARTNER: Barbara? 20 MS. FORSYTH: I wanted to ask a 21 question about the process for the world 22 model development. Is there a technical BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 179 1 working group that's working with you so 2 that, in essence, we'd be working somewhat in 3 a parallel technical advisory capacity? 4 Because sometimes a technical working group 5 can serve a more focused function and then we 6 meet probably less frequently. 7 SPEAKER: That's a good idea. 8 DR. FEDER: That's a good idea. 9 MS. KHANNA: You have a smaller 10 group of people. 11 MS. FORSYTH: Right. 12 DR. HENGARTNER: Mark? 13 DR. BURTON: That actually goes to 14 the point that I wanted to make in that we 15 certainly encourage being involved early in 16 the process. This is one where if we've not 17 been, it probably would have served well to 18 have been involved even earlier. I think 19 that Tom made some good points in the 20 break-out regarding sort of the conceptual 21 framework of the model and what some of the 22 alternatives might have been at an earlier BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 180 1 point. So, again, I'm not familiar with the 2 history in particular, but this may have been 3 a place where the committee could have added 4 some value from sort of day one. 5 MR. RUTHERFORD: Being a model 6 builder myself, though, I just -- I'm always 7 concerned about model-building by 8 committee -- 9 (Laughter) 10 MR. RUTHERFORD: -- so I tend to be 11 a little bit skeptical about -- I think today 12 was good. We got to complain about this and 13 that. I would also add, in terms of getting 14 your two bits in early, most of what I said 15 today, I said in my review of NEMS in 1993, 16 so I think my views are known. And the 17 general, I just think it's kind of good. But 18 I agree wholeheartedly with the idea of 19 bringing in the people who are doing the 20 modeling so we can basically have an up-front 21 interchange. 22 So I guess I sort of disagree with BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 181 1 the rest of the committee. I think that 2 either I'm involved in making a model, or I'm 3 just giving my best reaction. I think that 4 studying in detail what the modeler is doing 5 and then writing out -- I mean, really, to be 6 honest, to be completely frank, taking the 7 time to write out the English words that 8 describe what the problem are in the model is 9 much harder for me than just writing out the 10 GAMS code for how the model should be 11 structured. That's my own take on it; it's 12 just generally -- I'd rather just write the 13 math down. But if I'm not going to be 14 involved in doing that, I think that the 15 format was fine, it's just I think that the 16 idea of bringing the people actually doing 17 the work so we could ask specific questions 18 would be helpful. So I tend to be a little 19 bit less -- it's not that I don't want to be 20 involved in helping things, but I'm very kind 21 of -- having been on the receiving end of 22 advice from committees about models, I tend BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 182 1 to be reluctant to be on the -- 2 DR. FEDER: Giving end. 3 MR. RUTHERFORD: -- delivering end, 4 right. 5 MR. HELKIE: I think the operable 6 thing what Tom said was 1993, okay, and this 7 is 2006. People change. I think we're about 8 ready to get to a more comprehensive re-look. 9 And I think the English language reaction, 10 especially of people who have experience 11 building models, would be very useful early. 12 And I think if we got to pay you to do it, 13 that's fine, but I think it's still 14 worthwhile to get a broad view of people who 15 have experience building models who care 16 about the output of the models and can 17 provide their perspective the policy 18 community. 19 I just think that's useful, and as 20 you say, it's more useful than going into the 21 guts and saying, well, gee, that primer ought 22 to be point five instead of point six or BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 183 1 something like that. Because that's not as 2 useful as getting in in the beginning and 3 saying, this is where it ought to go, right, 4 and these are the reasons why. Okay, so I 5 guess I disagree with your conclusion. 6 DR. HENGARTNER: To wrap it up, I 7 think it's fair -- oh, oh, oh, no. 8 MR. CLEVELAND: No, I said my 9 piece, thanks. Didn't I? 10 DR. HENGARTNER: Yes, you did. 11 MR. CLEVELAND: I did. 12 DR. HENGARTNER: You did. I would 13 suggest that part of the compromise is to 14 definitely talk about this modeling some more 15 in six months. If people are interested in 16 being involved in giving feedback, they 17 should be. I mean, Neha offered some of her 18 time, and may -- I'm putting her on the spot, 19 which is unfair -- 20 DR. FEDER: But you'll do it 21 anyway. 22 (Laughter) BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 184 1 DR. HENGARTNER: But I think 2 there's room for polite debate. The 3 formation of a committee is something that is 4 outside, I think, the charter of the ASA 5 Committee of Energy Statistics, although it 6 is an idea that Nancy definitely can 7 consider. 8 So I'd like to invite, again, the 9 general public to make comments. And before 10 we all leave for lunch, I actually realize 11 that we are privileged to have among us Bill 12 Helkie, who is now technical advisor to the 13 EIA and actually comes from the office of 14 Alan Greenspan. So welcome. 15 MR. HELKIE: Thank you. 16 (Applause) 17 DR. HENGARTNER: Okay, so lunch is 18 downstairs, and we will reconvene here after 19 lunch at 1:15. Thank you very much. 20 (Whereupon, at 12:15 p.m., a luncheon 21 recess was taken.) 22 * * * * * BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 185 1 A F T E R N O O N S E S S I O N 2 (1:15 p.m.) 3 DR. HENGARTNER: Okay, welcome back 4 to the afternoon session. It's my great 5 pleasure to introduce Jake Bournazian, who's 6 going to talk about measuring perception of 7 applying alternative disclosure limitation 8 methods. 9 MR. BOURNAZIAN: Well, good 10 afternoon. Today's presentation is about a 11 new research proposal that's going to involve 12 studying the perceptions of our users and 13 survey respondents on some unique issues 14 relating to data confidentiality. 15 There have been prior studies done 16 on user perceptions in the past, but what's 17 special about this research proposal is 18 there's never been a study on the perceptions 19 of applying alternative disclosure 20 limitations -- methods for protecting 21 confidential data. 22 When we're talking about studying BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 186 1 perceptions on this topic, we really need to 2 study two separate groups. Not only are we 3 going to be studying the perceptions of two 4 distinct groups, but we also want to get 5 different information from these two groups. 6 Specifically, here, we're data users. We 7 want to know what they're thinking about and 8 how they feel on the data utility of viewing 9 tables that are filled with either all 10 adjusted values or some cell values that are 11 adjusted in a table. What's the data utility 12 of that kind of product to them? 13 In the second group, we want to be 14 looking at our survey respondents, and we 15 want to know what they're thinking about when 16 they're informed that an agency is 17 considering applying a different disclosure 18 limitation method, such as data swapping. 19 Will they become less diligent and accurate 20 in filling out their survey forms if they 21 know or are aware of these alternative 22 methods that are going to be applied? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 187 1 In this morning's session, I heard 2 a comment made that it's not good to base 3 decisions on ad hoc information. And other 4 speakers mentioned that we should be knowing 5 a little bit on how our data users are 6 perceiving and using this data. Well, 7 everyone in this room might have some 8 subjective opinion about these issues here up 9 on the screen, but for the agency to act in 10 its best interest in fulfilling its mission, 11 we need objective, accurate information which 12 relate to these perceptions. Next slide. 13 So let's put the study in the 14 proper context of what we're talking about, 15 because this is very technical area. New 16 disclosure methods are available. I grouped 17 them into two categories for today's 18 discussion: The first category is going to 19 adjust the data prior to doing any kind of 20 tabulation. You may have two branches of 21 these -- one is data swapping and the second 22 noise addiction. Data swapping's been around BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 188 1 for decades, very common, prevalent use to 2 protect household data, frequency data, but 3 recently it's being used to protect tabular 4 data. In the Census 2000 information 5 products, they used data swapping to protect 6 that data for some of their tables. Also the 7 Census Bureau pioneered noise addition in 8 1998 on a research and development survey. 9 National Agricultural Statistics Service 10 implemented noise addition on the 11 Agricultural Resource Management Survey last 12 year. They unveiled an online query system 13 where users go to their website and customize 14 and design their own queries, get their 15 tables, and the confidentiality of the data 16 is protected through noise addition. 17 There's another category here, 18 solves the problem from a different approach. 19 We're going to adjust the data after the 20 aggregate estimates are calculated. So we're 21 going to adjust after tabulation. We're 22 referencing here controlled tabular BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 189 1 adjustment; this is a broad category which 2 references a family of methodologies, 3 relating all to the same kind of approach. 4 Next slide. 5 So let me explain very briefly what 6 these methodologies do by operating on a 7 single example. We're going to take the same 8 problem, we're going to solve it three 9 different ways using these methodologies, and 10 you can see how they work. All right, so the 11 problem here is we have a table with a 12 sensitive cell needing protection. That's 13 what triggers some kind of action from the 14 agency. Next slide. 15 Now, the most common method across 16 all federal agencies is cell suppression, and 17 it's certainly the method used here at EIA. 18 When you're talking about applying cell 19 suppression for protecting confidential data, 20 cell suppression, in my opinion, is a one-way 21 street. Cell suppression says, from the 22 agency's standpoint, we're going to meet our BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 190 1 needs -- which is to protect the 2 confidentiality of those company-level 3 responses -- but how about the user that's 4 interested in one of those suppressed cells? 5 Well, you're out of luck because we have to 6 take care of our need as an agency. And 7 there's a total loss to the user. Well, 8 that's the way life has been for the last two 9 decades. What happens, though, is that 10 through time, industry changes, these tables 11 can accumulate pockmarks. Some tables might 12 have 20 to 40 percent suppression, but that 13 suppression could grow, sometimes frustrating 14 the utility of the table altogether or almost 15 destroying a lot of the value. 16 So you may be thinking, wow, if 17 you've got that much suppression, redesign 18 the table. Well, it's not that simple when 19 you have limited resources. Limited 20 resources for a program office is going to 21 put this on the lowest of low priorities 22 because if you have other production BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 191 1 activities going on, redesigning a table is 2 not right up on there to spend your 3 resources. So what happens, like a house 4 that has paint flaking off of it and the need 5 of repair, these EIA publications can carry 6 four of these tables over the years, which 7 carry a lot of pockmarks from their cell 8 suppression. But we've got some solutions. 9 Mathematics and IT science to the rescue! 10 Next slide. 11 Okay, let's look at math and 12 science, part one. I told you we can try and 13 protect that sensitive cell by using a method 14 called a controlled tabular adjustment. The 15 approach here is we're going to go after that 16 sensitive cell and adjust its value a 17 sufficient distance away from its actual 18 value. And you can specify a percentage 19 protection range you want to give that cell. 20 Now, once you adjust that sensitive cell, you 21 have to go back and adjust some of the 22 non-sensitive cells in that table. And in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 192 1 this example, it's three rows, three columns, 2 I have marginals -- totals down at the bottom 3 of each column and end of each row. So it's 4 a very simple table. Now, I adjusted actual 5 values here in my example. I could have 6 taken the reciprocal value of those cell 7 values or I could have taken the logged 8 value, and I could have said, well, I don't 9 want any of my totals to change, just adjust 10 the interior cells. Or in this case, in my 11 example, I allowed the marginals to be 12 adjusted, thereby saving some interior cells 13 from needing to be adjusted at all. Once 14 again, there's a lot of variations, but it's 15 the same approach when you're using one of 16 these variations on controlled tabular 17 adjustment. Next slide. 18 All right, so math and science, 19 part two. We're going to solve this problem 20 from a completely different approach. We 21 want to now modify the company-level 22 responses prior to doing any tabulation. And BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 193 1 we're going to try and protect that sensitive 2 cell this way: With noise addition, you can 3 specify, once again, the percentage 4 protection you want to give those sensitive 5 cells. You can also target the noise and add 6 more noise to the sensitive cells and avoid 7 adding any noise to the non-sensitive cells. 8 You have to take a real specific approach, 9 though, in using this methodology. You're 10 going to first want to randomly assign the 11 direction of the noise, then come back in and 12 randomly assign the noise factors. You can 13 pick a number of types of distribution, but 14 whatever distribution you start off with in 15 adjusting upward, you must take the exact 16 opposite distribution in adjusting downward. 17 When you assign that noise, what 18 you are searching and going to obtain is a 19 distribution of the noise factors that's 20 symmetrical around number one because 21 ultimately you do not want to impact your 22 aggregate estimates to the lowest bar, you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 194 1 want to save any adjustment on that. So in 2 this example, we've drilled down to the 3 company-level values, and now we're applying 4 our noise factors. Now, another point here 5 for this methodology is once a noise factor's 6 assigned to a company, that company gets that 7 noise factor no matter where its data appears 8 in a table, in a sensitive cell or 9 non-sensitive cell. So here we adjusted the 10 values and came up with a new cell total to 11 protect the actual true value, which is 12 deemed sensitive from whatever primary rule 13 we're using. Next slide. 14 Okay, and then also, like I pointed 15 out, every company in that table's getting a 16 noise factor, and so that means wherever that 17 company's data appears in the table, it's 18 getting adjusted the same way. Here, 19 non-sensitive cells are also adjusted, but to 20 a lesser extent; less noise is added to those 21 cell values. Next slide. 22 Okay, let's go back and solve the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 195 1 same exact problem a third way. We're going 2 to use data swapping. Now, the way you apply 3 data swapping for protecting tabular data is 4 pretty much the same way you did it when you 5 were protecting frequency tables in 6 public-use files; that is, you go after your 7 sensitive cells that your primary rule has 8 identified, try to identify what's going on 9 in those cells, what's unique, and you're 10 going to design a data key structure to 11 target data swapping on your sensitive cells 12 to the best of your ability. Now, I used a 13 very simple key structure, three variables. 14 You could go on for a lot longer because I 15 see data keys five variables or more. But 16 the point here is we're going to go after a 17 sensitive cell and do some swapping. You 18 only need to find a proper swapping 19 candidate, and for me, in this case, sales 20 type, product type, seller type. You could 21 have variations on seller type -- could be a 22 certain type of power producer or it could be BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 196 1 a certain size of supplier. You might have a 2 lot of medium-sized suppliers; we do the swap 3 and recalculate and I do adjusted value. You 4 want to keep your swap rates real low when 5 you apply this methodology and don't want to 6 go over five percent because this methodology 7 impacts your cell values a lot more than the 8 other two methodologies do. And of course, 9 you need some kind of swapping variable when 10 you apply this methodology. 11 So let me just summarize this area 12 that we're going to want to study perceptions 13 on. Yes, cell suppression is the most common 14 method; these new methodologies have been 15 developed specifically to meet a growing user 16 demand for more data. One point to note is 17 noise addition and controlled tabular 18 adjustment can adjust your non-sensitive 19 cells, and it may not cause a statistically 20 significant change from the actual estimates. 21 But in more simplified terms, data swapping 22 is more like using a rubber mallet to impact BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 197 1 a cell, and controlled tabular adjustment and 2 noise addition is like using a tack hammer to 3 impact a sensitive cell. But the benefits 4 are clear. You're going to get a fully 5 populated table, more usefulness. And I have 6 no problem footnoting a table; if you 7 overshot your mark in cell value changes by 8 more than 10 percent, you could put a flag on 9 those cells in the table and tell the user 10 these cells are flagged because they're not 11 as accurate as other cells in a table. An 12 intruder can't take that information and 13 unlock your table, and a data user would 14 really find that kind of performance measure 15 useful in working with the data. Of course, 16 the cost -- got to watch it. These 17 methodologies need to be applied in an art 18 form as much as a science. And of course, 19 it's subject to criticism that you can really 20 screw up a table in applying these 21 methodologies. To that criticism, I always 22 reply, well, the delete key on your personal BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 198 1 computer can really devastate your hard drive 2 if used inappropriately, but when that key is 3 used appropriately, it's a very important 4 tool on your keyboard. Same with these 5 methodologies; you have to be very careful in 6 applying them. 7 Okay, so let's look at the survey 8 literature in this area because the Census 9 Bureau has been a leader in doing studies on 10 user perceptions, and they did a study six 11 years ago; it was the survey of business 12 perceptions of confidentiality. Here they 13 were studying what kinds of information do 14 businesses deem confidential. They also 15 wanted to explore how they felt about data 16 sharing -- this was the closest study I could 17 find that parallels what we're talking about 18 this afternoon. Well, they built it as a 19 nice frame file from purchasing Dun & 20 Bradstreet, 11.3 million businesses, drew a 21 sample of 5,000 companies, set up four 22 strata, based on employment -- 0 to 50 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 199 1 employees, 50 to 250, 250 to 500 and 2 companies with over 500 employees -- but each 3 stratum had 1,750 companies in it. They also 4 looked at companies that were multi-location 5 and single location, split there. But 6 anyway, they ran the survey, did it by mail, 7 came up with a pretty low response rate, 8 under 11 percent. Three reasons they cited 9 in their report: One, it was a voluntary 10 survey; two, it was done by a contractor in a 11 contractor's name, not in the name of the 12 Census Bureau; three, they conducted the 13 survey over the holidays. So here, probably 14 a good sample design, probably a good survey 15 questionnaire they used, too. But timing 16 really just wiped out a lot of hard effort 17 there. Timing can be everything sometimes in 18 life, and this showed it. 19 But anyway, there was some useful 20 information coming from this survey. This 21 was present in each one of the strata, so we 22 can rely on that something is going on in the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 200 1 minds of the businesses. They were split on 2 whether the government could protect 3 confidential data; the greater the trust and 4 the belief in competency of the statistical 5 agency, the more willingness to provide data. 6 And I pulled these out because I felt it was 7 noteworthy. 8 I think EIA has good credibility 9 across all its energy markets, and we could 10 probably work from this if we were to conduct 11 such a study. Also relating to data sharing, 12 they repeated this portion of the study and 13 were finding that the respondents were 14 filtering. Businesses, when they're asked 15 about how they felt about data sharing were 16 thinking, well, does this reduce my reporting 17 burden? Everything was filtered in that 18 process and in those terms. And we may want 19 to consider that also, because we're talking 20 about alternative methods. So if a company 21 hears, well, EIA's considering data swapping, 22 does that mean if I don't report in April BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 201 1 2006, just swap my data from January, and 2 we'll call it even. Then data swapping may 3 sound pretty good to that respondent. So 4 we'll have to address maybe reporting burden. 5 Next. 6 Okay, data user studies: A lot 7 done here by the Census Bureau. I didn't 8 spend a lot of time on this, because data 9 users -- or households -- are a lot different 10 than our EIA data users, but we might be able 11 to learn something here. One thing that came 12 through was identical to the business 13 perceptions: Households equally had a 14 distrust, and they were split on the ability 15 of the agency to protect their data. They 16 were questioning that confidentiality pledge. 17 Now, what also was interesting, the Census 18 Bureau relied on two modes of data collection 19 throughout 20 years: Focus groups and mail. 20 They did some cognitive research, but that 21 was mainly to design a good questionnaire, 22 although they tried to draw some conclusions BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 202 1 from that cognitive research, but focus 2 groups and mail are their primary modes. 3 Next slide. 4 Okay, so specifically, to the study 5 we're talking about here at EIA, we have some 6 additional considerations. One, we're 7 talking about disclosure limitation 8 methodologies, and these different methods 9 impact the data differently -- I tried to 10 show you that with a generic example -- and 11 also these methods are more appropriate, 12 depending on the type of data. You don't 13 want to pick one method and say, well, 14 that'll work on all kinds of data, you want 15 to taper it -- what's appropriate. 16 Very important point here: EIA has 17 a very diverse user community. We've got 18 government agencies, a wide range of federal 19 agencies, state agencies, local, municipal 20 governments, we've got analysts -- and I can 21 carve that category up between consulting 22 firms, financial analysts, policy analysts, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 203 1 academia, researchers -- media -- I can break 2 media up into TV, radio, newspapers, 3 magazines. So a lot of variety across our 4 user community. Level of statistical 5 knowledge is limited to the folks we want to 6 reach and talk to, and I'm presenting here to 7 a roomful of sophisticated statisticians on 8 some methods, but that's not going to be the 9 same when you're talking to companies and 10 data users. Also, some negative attitudes 11 exist. I think for that reason, the Census 12 Bureau on a longitudinal basis is always 13 surveying user perceptions. The pendulum 14 between privacy and willingness to accept 15 some intrusion from the government and 16 reporting information is always swinging back 17 and forth since they passed the Privacy Act 18 in '74. It goes back and forth -- I mean, 19 it's where you want to do these studies -- I 20 heard somebody say there's no systematic way 21 of communicating with our users and then 22 measuring. And this is one way that the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 204 1 study, it gets conducted, we want to repeat 2 it or at least carry it on longitudinally to 3 pick up that swing in the pendulums. 4 We saw it swinging the other 5 direction, away from privacy, after 9/11, 6 then people hear about the President wanting 7 to allow wiretaps without a court order, and 8 you see that pendulum moving back. So we've 9 got to recognize doing this study, what's 10 your atmosphere and climate in which you're 11 conducting it. Next slide. 12 All right, I'd like to wrap up with 13 this slide. I put down on this slide some of 14 the big issues that were coming to my mind 15 when I was writing this paper. This first 16 issue, though, when I was thinking about it, 17 reminded me of what my mother used to tell me 18 when I was growing up. She'd say, Jake, 19 sometimes you can't have your cake and eat it 20 too. That used to really annoy me, but I 21 thought of the same thing when, on this first 22 issue, who should be the target respondent. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 205 1 The reason why is because that contact person 2 opens their mail. When they get mail from 3 EIA, they open it -- oh, survey -- they fill 4 it out -- I mean, that's why they get paid, 5 that's their job. They even know the data -- 6 wouldn't that be nice. Well, the Census 7 Bureau always picked the authority figure 8 with the company, and that's a very good 9 point, too, because why should I have the 10 comptroller of a company filling out a survey 11 form when I really want the president's 12 perception on this issue? Well, can EIA have 13 their cake and eat it too? Well, I thought, 14 if it's a mail survey, maybe I'll mail it to 15 the contact person and tell him to give it to 16 his boss. Or mail to the authority figure 17 and say ------ next year, so-and-so's your 18 contact person, best contact person in 30 19 years, why don't you get some feedback from 20 them in filling this out? Somehow maybe we 21 can share ideas, or we may have to pick one. 22 I don't know. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 206 1 Next issue, should all survey 2 respondents be included in the sample? If 3 Southern Minnesota Power and Electric is in 4 that survey on perceptions, do I need 5 Northern Minnesota Power and Electric? 6 Maybe. Maybe people in Minnesota think 7 differently in the north than they do from 8 the south, but we have to address that in 9 picking off our survey respondents to prove 10 we're going to participate. 11 How about selecting those data 12 users? I heard our administrator throw out a 13 statistic this morning; he said in February 14 2006, we had 300,000 users on our website -- 15 and as I mentioned before, quite a diverse 16 user community. Do we want just Internet 17 users? Well, we have a lot of other users, 18 although the Internet's our primary mode of 19 disseminating data. How about that mode of 20 data collection? Okay, Census Bureau relies 21 on focus groups and mail, but we even have a 22 session on Internet data collection and we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 207 1 have other modes available today. Maybe they 2 could be used, or a combination of these 3 modes, but maybe we can talk about that. 4 Also, like I said, this is a technical area. 5 I used a generic example to explain these 6 methods pretty quickly for you -- I used text 7 in my paper with brief examples, but 8 customized examples always work better for 9 me. When I'm trying to understand something, 10 what works better? But if you get too far 11 down the line on a customized example, aren't 12 you giving a road map to a data intruder? I 13 believe most of our survey respondents, at 14 one point in their life or another, have 15 picked at the table they report data on and 16 tried to look at what their competitors were 17 doing. But you might have to mask it if it's 18 customized. Actually, on this issue, since 19 you're the first audience I've tried these 20 generic examples on, this will be our own 21 focus group on this point. As long as more 22 than nine people don't talk on that. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 208 1 Okay, other issues. I ran out of 2 font space here on this slide, so -- but 3 there's a lot of other issues. Anything 4 you're thinking about that we need to be 5 paying attention to or looking at in 6 designing this study, we'd love to hear. 7 Now, we just summarize that this is 8 a pretty daunting project. I've talked to 9 some other colleagues and other federal 10 agencies about doing this kind of a study, 11 and they've shied away from it because it's 12 very challenging, and it's also a topic that 13 agencies really don't like to go too public 14 on when they talk about their data 15 confidentiality practices. But behold the 16 turtle: It only makes progress when it 17 sticks its neck out, and I think we, working 18 together, sharing these ideas today, not only 19 can we design an effective study that will 20 beat a response rate of 11 percent, I think 21 we can design a study on perceptions which is 22 going to give us real useful information so BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 209 1 that we can understand, gain some insights, 2 on what our users and survey respondents are 3 thinking about when EIA considers applying 4 alternative disclosure limitation methods. 5 DR. HENGARTNER: Thank you very 6 much, Jake. 7 (Applause) 8 DR. HENGARTNER: I'd like to invite 9 Nagaraj to start the comment session if you 10 have some prepared comments for us. 11 DR. NEEERCHAL: First of all, I 12 would like to really thank Jake for a really, 13 really clear presentation, and the write-up 14 is also very clear, and I really enjoyed it. 15 It's particularly important to me because I'm 16 brand new to this subject, so I think that 17 helped me a lot. And I think that's why I 18 think that my comments probably are going to 19 be more from the user point of view. I think 20 -- it's -- I'm not an expert in this area. 21 This is a situation where there is a real 22 cop-and-robber type of thing going on. I BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 210 1 think as Jake kind of alluded to at the end, 2 for Bill Winkler, who is an expert in this 3 record linkage stuff from Census Bureau, he 4 always says that if there is an algorithm 5 that masks the data, I have an algorithm that 6 will unearth it. I think I heard him say in 7 one of his presentations. So I think that is 8 definitely the case. Having said that, I 9 don't think I need to go over the methodology 10 here. I think Jake has done a very good job 11 of presenting it. 12 I do want to say couple of points. 13 If I'm a user, I really do not -- perhaps I'm 14 not even able to understand the different 15 methods. So all I really want to know is how 16 does the data look before you apply the 17 method and after you apply the method? I 18 want you to just give me a few tables to look 19 at. And that is one of the things I'll keep 20 in mind when I'm designing a survey like 21 that. Instead of hammering the user with a 22 lot of algorithms and asking their opinion BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 211 1 about which is going to be better, I would 2 probably focus on giving them a few examples 3 -- hey, this is what it's going to look like 4 under method one, this is how it's going to 5 look under method two, and before and after. 6 So I think that's one approach. Now, for 7 example, I think clearly in all the examples 8 Jake presented, the lowest data point in the 9 table always stayed as the lowest data point 10 in the table. Lowest and highest. If you 11 have something like electric power, I think 12 you always know who is the "highest" -- 13 everybody knows that. Income data, everybody 14 knows Bill Gates is on the top of that list. 15 So you can mask it, you can add noise to it, 16 if it is the highest -- if you know it is the 17 population, who that point is. 18 So I think that the user can 19 determine, based on the before and after that 20 there are such features, the ones that matter 21 to that user the most, whether those features 22 are still recognizable or not. This is one BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 212 1 way to really get the feedback from the user 2 in terms of what are these important features 3 that they really want to be masked or not 4 masked, depending on their point of view. 5 Having said that, I think that is why, if 6 you're looking at people who are responding, 7 I think that it's always going to look a 8 little different. They probably don't want 9 their data to be that organized. On the 10 other hand, users of the data, they want full 11 accuracy. And the more you mask, the 12 correlation between the presented data and 13 the real data is going to go down, and if it 14 is very low, then what good is that data. So 15 I think those are the issues to be concerned, 16 and I'm very pleased at the way it is set 17 out. I think you have done a lot of review 18 of the literature; I think you're working 19 with the people from that area, other parts 20 of the government, and I think that it seems 21 to me that some of these issues will be -- 22 I'm not the person to advise you on that ---- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 213 1 you will get good advice from those areas. 2 One other thing I wanted to mention 3 -- I think that it probably is not associated 4 with this area -- is Larry Cox from NCHS. 5 Recently he has been working on this same 6 topic, changing -- changing is a layman's way 7 of saying it -- changing data, but respecting 8 some column totals and row totals, and 9 applying some linear programming type of 10 algorithms to tables and presenting that. 11 And the only other comment I have 12 is that when you present this data, you have 13 to say if it is masked or not. So already 14 you're giving away some of your information. 15 The moment I know that the table is masked, 16 and I'm trying to use it, I think -- I'm one 17 of those people -- I just say, let me see how 18 they mask it. I think that's the first thing 19 that's going to come to my mind. Maybe 20 everybody is like that, I don't know. I say, 21 let me see what might be the algorithm. If I 22 know you're going to use one of the three, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 214 1 then I'll say, let me see, let me guess which 2 one -- and then of course I cannot discover 3 the real data because the noise level is 4 different and you had to give me a whole lot 5 of parameters, but I still would play with 6 it. So other than that, I think it's a very 7 nice presentation, laid out nicely, and ready 8 for the committee to jump in. I think these 9 are very clear questions. I think the 10 committee should jump in and give some 11 solutions. 12 DR. HENGARTNER: Thank you very 13 much. I'd like now to move over to Moshe, 14 who has kindly agreed to step into Johnny 15 Blair's shoes. 16 DR. FEDER: Disclosure limitation 17 is very important because I find -- I work in 18 this area, too, although my interest is in 19 microdata files and not as much in tabulated 20 data. And first of all, I want to say that 21 was an excellent presentation. I really 22 enjoyed hearing the issues that you guys BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 215 1 have. I find every day new ways of potential 2 breach of confidentiality. Recently, I've 3 seen some ideas that people came up with to 4 breach confidentiality using even the server 5 weights in macrodata files, and things like 6 that, so I think it's very important. And 7 one thing that I think is very important to 8 keep in mind, all it takes is one breach of 9 confidentiality to do serious damage not just 10 to the EIA's data collection efforts, but to 11 any agency's data collection, because once it 12 appears in the press and people will know 13 that there's potential for confidential 14 information to leak out, then that would 15 cause a big damage to response rates, 16 accuracy of responses, and so on. So I think 17 it's very important. And perception, like 18 you said, is very important here. There's 19 always a tradeoff between data quality and 20 protection of confidentiality. The more you 21 protect, the less information is there, so 22 striking a good balance is very important. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 216 1 I have a couple of questions here 2 for Jake and to others who work in this. You 3 presented two ways of approaching it. One is 4 pre-tabulation, by adding either noise or 5 doing data swapping to the data before the 6 tables are being extracted -- and actually, 7 modeling is a similar issue. If you ------ 8 the aggression model, you can sometimes will 9 information by looking at the regression 10 coefficients, especially if some of them are 11 categorical, and you can use that to infer on 12 some individuals. And so in particular, when 13 people provide multiple products using the 14 same data, how do you make sure that by using 15 two tables or two regression models or in 16 some other way combine information from these 17 and make -- I'll give you an example that I 18 know the Australians are working on. They 19 have an online system when you submit a 20 request for a table, and you submit one table 21 and say, give me this information for 22 individuals age, let's say, 12 to 17, then BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 217 1 you do another one, let's say, 16 to 18, and 2 you cross the two tables and get an age range 3 which is too narrow by the limitations of any 4 single table and combining it. So there are 5 many challenging issues here, and the first 6 approach is to treat the data prior to 7 tabulation avoids that because if the 8 microdata file are protected, then the tables 9 in theory should be protected. 10 The three-way table is another 11 challenge. You mentioned data-swapping, 12 which I think is a common technique, but 13 wouldn't the two swapping partners know 14 enough -- if I can recognize my data, 15 wouldn't I then know something about the 16 other person or entity? So I think there are 17 two, and that's almost unlike any other 18 statistical activity that we do. Usually, a 19 statistician's job is to provide as much 20 information as you can, period. Here, not 21 only are we supposed to suppress information, 22 but do it in a way that someone will not be BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 218 1 able to -- I think Nick or Nagaraj mentioned 2 that they quoted Bill Winkler saying that he 3 can always come up with a way to break the 4 code, so I think if you can maybe shed some 5 light on that. How do you constantly avoid 6 new ways to break the code, and what do you 7 do with data swapping if a person who's 8 responsible for the data that's part of the 9 swapping will be able to undo the process by 10 identifying his establishment's response? 11 I'll stop here. 12 DR. HENGARTNER: Thank you very 13 much. I'd like to open the discussion to the 14 general committee. We have about 10 minutes 15 to do so. 16 I have a question. What I find 17 with those data confidentiality issues, 18 especially from the perception point of view, 19 is that I'm not convinced that we all 20 understand how data swapping or how adding a 21 little bit of noise will ensure the 22 confidentiality. How do I know that adding BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 219 1 noise will make me safe, right? And maybe it 2 doesn't. And if you are going to make a 3 survey and explain which method you like 4 best, you ought to do yourself the favor -- 5 and the reader as well, or the person you're 6 serving -- to explain why it works. I'm 7 always tempted to say that, well, you mention 8 that some of the readers are not that 9 statistically literate; in fact, you don't 10 need to be statistically literate or not -- I 11 mean, I'm not even convinced, in light of 12 Nagaraj and Moshe's discussion that adding 13 noise sometimes truly does ensure 14 confidentiality. If you have an outlier, he 15 might remain an outlier. And so being able 16 to make that discussion would be interesting 17 -- or that distinction. Yeah? 18 DR. FEDER: I can tell you that at 19 least one agency -- and we won't go into -- 20 not one that I'm actually very closely 21 related to -- all they do is do treatment of 22 some records and say, since you never know BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 220 1 which one was treated -- or you don't know 2 which data were contaminated by noise, you 3 never know enough information. That would 4 not apply the case of the EIA simply because 5 you have big establishments whose domination 6 of some cell estimates is big and so on. 7 But that's a good question. What 8 are we trying to do here? We want to create 9 enough distortion to the data that you never 10 know for sure enough about the protected 11 information, but what is enough? I know that 12 Chris Skinner -- and actually, it goes back 13 to the Bethlehem -- and I forget who the 14 other author of that paper is -- they came up 15 with some disclosure risk measures which 16 involve uncertainty measures and what you 17 need to decide is what -- and that's tied to 18 perception -- what is the perception of the 19 public about enough uncertainty that can be 20 viewed as protection of confidentiality. But 21 the least you can do is say, I can tell you 22 that the probability of identification and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 221 1 subsequent -- putting some confidence limits 2 on protected information is such-and-such. 3 And if that uncertainty is enough, according 4 to perception of the public, then you're 5 protected. 6 DR. NEERCHAL: Could I ask a 7 question? What are the specific types of 8 information that your respondents would be 9 most sensitive towards in terms of being 10 published? I think you're already publishing 11 some data online or large ---- reports, and 12 obviously -- have you had anybody complain 13 about why did you put that number up there? 14 That is one of the things that would be 15 important to identify: The kind of 16 information that they're most sensitive 17 about. And I think on that side of it, it's 18 important to get your stakeholders at the 19 table. I think here -- I think whether you 20 are doing an adequate job or not will depend 21 on who has something to "lose" -- and I'm not 22 -- may not be financially, but something to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 222 1 worry about, they don't like to be published. 2 I think that would be something I would like 3 to know. 4 MS. KIRKENDALL: Let me respond. 5 In the electric power area the Office of Coal 6 has done a lot of work with respondents 7 because they go back and forth. They'd like 8 to be able to make data public. And so every 9 time, they'll go make a proposal on the 10 Federal Register notice, saying that we're 11 going to make all of these data items public. 12 And so they get a ---- price and the 13 company's saying, well, it's all right for 14 these variables, but we need to protect 15 those. So I think they're unique in the 16 federal government that some of their surveys 17 -- they have a lot of surveys -- some data 18 items are confidential and some are not. The 19 ones that are not confidential are actually 20 put up on the Web, they're 21 company-identified, and the ones that are 22 confidential have to be treated in some way BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 223 1 in the tables to make sure they're protected. 2 So you're right, and you do have to work with 3 your respondents. And it may be that that's 4 a way to be able to make more available is 5 just to find out which data items on the form 6 can be made public. 7 DR. NEERCHAL: And you may want to 8 let them know that, hey, this particular 9 column is highly correlated with this one, 10 so -- 11 MS. KIRKENDALL: They're typically 12 worried about cost information, about price 13 information. I guess in one survey, they're 14 worried about stock information, which I was 15 surprised about, but the money -- 16 SPEAKER: Anything that has a 17 dollar sign. 18 DR. HENGARTNER: Just to make sure, 19 Neha, Mark, and then Stan. So that's the 20 order. 21 MS. KHANNA: As a user of data, I 22 have a question. It's fine to swap data or BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 224 1 to contaminate the data in some way for 2 confidentiality. But at the same time, we 3 don't want to destroy inherent relationships 4 within the data, which, of course, you're not 5 going to even know what relationship a user 6 is looking for. But that raises an issue of 7 how much noise are we going to add. On the 8 one hand, we may want to change the order and 9 ranking of the data, but that may be 10 destroying some relationship. I don't know 11 if there's anything in the statistical 12 literature out there that addresses this 13 question. 14 DR. HENGARTNER: Mark? 15 DR. BURTON: The one thing I love 16 about these meetings is they give me repeated 17 opportunities to show how little I know. I 18 work with some transportation data, where the 19 firms reporting the data have the option of 20 having values masked. It turns out that most 21 of them elect not to. They just, for 22 whatever reason, are not sensitive. Some, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 225 1 however, will go through the masking process. 2 If you have individual respondents that do 3 want confidentiality and other respondents 4 that don't, what does that do to the 5 statistical characteristic of the data? 6 DR. HENGARTNER: Stan? 7 MR. FREEDMAN: Nancy said something 8 that reminded me of a discussion that we had 9 with a respondent one time when we were out 10 doing a site visit -- and Jake, I'm sorry I 11 didn't think of this sooner because I would 12 have mentioned it to you. The issue of 13 sensitivity also relates to timeliness. Not 14 disclosing stock data or cost-of-fuel data, 15 for example, might be important to the 16 respondent three months or six months after 17 it's reported, but a year or two years 18 afterwards, it's no longer important. It 19 loses its significance because it's no longer 20 timely. So you may need to figure that into 21 the study as well. And like I said, I'm 22 sorry I didn't think about -- but we talked BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 226 1 about it earlier, but I think it depends on 2 the data and it loses its sensitivity over 3 time. 4 DR. HENGARTNER: Moshe? 5 DR. FEDER: Yeah, the issue that 6 Neha talked about -- destroying multivaried 7 relationships between cells -- is actually -- 8 Jake did allude to that, and I was going to 9 ask about it, too, because in my activity and 10 disclosure limitation, we do a 11 post-processing assessment of the "damage" 12 that we have done to the data, and 13 traditionally, what we have done is we looked 14 at univariate statistics, overall estimates 15 of whatever we are estimating or think people 16 might want to estimate, and we are talking 17 about here tons of data, so what you do is 18 you do some distribution on that and say how 19 many estimates stayed within, say, two 20 percent of the original ones, how many went 21 up, how many went down. And then we look at 22 some multivariate relationships, which BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 227 1 originally had been just limited to estimates 2 by some domains. So if I were interested, 3 and I survey human populations, how many 4 estimates for females have change, how many 5 for males, how many by some region, and so 6 on. But then I started being worried more 7 about what Neha is worried about because 8 analysts want to know how do my regression 9 models remain after we've done the analysis. 10 And this is precisely what I'm going to do 11 when I get back to my office, hopefully on 12 Monday, and propose some modeling. And 13 someone actually was on team, but did it for 14 another project -- but I was pleased to see 15 that in his case -- and we are talking that 16 case on a logistic regression model -- the 17 regression coefficients have changed, but not 18 the direction. So, you say, this contributes 19 to the outcome, but not exactly the same way. 20 And I recommend to do the same thing. I 21 don't think you can worry about in the design 22 stage exactly how much you can afford to do, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 228 1 but post-processing analysis, which is also, 2 within limitation, disclosed to the public, 3 is a good idea. So you can say we have done 4 some multivaried modeling and we know that, 5 yes, there are some changes to regression 6 coefficients or whatever, but they generally 7 keep the same direction, so inference is not 8 too bad. But, yeah, that's something that 9 I'm worried about, too. 10 DR. HENGARTNER: Yes? Can you come 11 in closer and state your name, please? 12 MS. HOWELL: I'm Shannon Howell, 13 and I have a question. I was just thinking 14 about a case that I saw where suppression of 15 confidential information came up, and with 16 any of these alternative methods, if anybody 17 could see a problem with very thin cells, 18 wherein you might have just one or two 19 respondents, if you have one respondent, you 20 publish something that's not the number they 21 gave you, they might become slightly upset or 22 worry about how you're handling the data. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 229 1 And two, if you have just a couple of 2 respondents, they might very easily be able 3 to figure out what you're doing. 4 (Laughter) 5 DR. HENGARTNER: First Moshe, and 6 then the gentleman in the back. 7 DR. FEDER: I actually happen to 8 know a little bit about what's done on this 9 cell, which is not necessarily one where the 10 sample size is small. Could be a cell 11 dominated by one major market player and also 12 by some kind of cooperation between 13 individuals dominating the cell to share the 14 information and thereby inferring about the 15 remainder. And I know that in Canada, I 16 believe the number was fifteen or just one of 17 two, and in our case, we have a more complex 18 cell suppression formula, but it's got to be 19 sufficiently large so you won't be able to 20 find out what's in it by using some 21 information that you know from private 22 sources. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 230 1 MS. HOWELL: Obviously in these 2 cases, you could still suppress the 3 information entirely, withhold it. But other 4 than going to greater extent of alternative 5 data, could you get around that? 6 DR. FEDER: Mathematically, these 7 questions are very challenging. That's why 8 the first approach that Jack (sic) was 9 talking about -- treating the microdata 10 before production of the table -- you avoid 11 that. And in fact, there's an issue with the 12 three-way table, there's an issue with 13 someone combining one table with one 14 regression model and so on. Where does it 15 stop, how do you know? Or, as I said, using 16 information on yourself or on your 17 competitors that are sharing it with you -- 18 DR. NEERCHAL: I think one of the 19 other options I think you should keep in mind 20 is, can you get away by publishing only some 21 of the ----- information and if there is some 22 particular way to be sensitive. And then ask BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 231 1 yourself, do some regional summaries only, 2 and is that enough for the kind of users you 3 have in mind. I think that's -- or if it is 4 a regression ---- population of correlation 5 -- I think you have to ask that question. 6 I'm sure the person who worked in this 7 problem is the one who invented the phrase 8 "damned if you do and damned if you don't." 9 (Laughter) 10 DR. HENGARTNER: There's one 11 question there, and then one more after that, 12 and I am unfortunately going to have to break 13 up the session, but I invite people to 14 continue discussion; maybe tomorrow we'll 15 have some time left at the end of the 16 meeting. So two more comments. 17 MR. WATSON: Bill Watson, CNEAF. 18 Here I'll talk only about cull, and it may 19 not necessarily generalize to other data sets 20 that we have at EIA. Almost all of our 21 microdata is confidential, so nobody except 22 staff that are at EIA or at other federal BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 232 1 agencies that will sign a data disclosure 2 agreement will actually be able to get at the 3 microdata. And we do not ever mask the data, 4 the microdata, but when we go to tables with 5 it, we have some very well thought out 6 suppression rules, and so we'll -- you see a 7 lot of "W"s in our tables -- withheld data. 8 An objective of the withholding rules is to 9 make sure that nobody could back out or use 10 mathematics to calculate somebody else's 11 data. So this would be in a situation where 12 there were only one or two people reporting 13 in a state, and you had a region total or you 14 had other states in a national total, and you 15 could back out some of the data that was 16 associated at the firm level with somebody. 17 And that works to maintain the 18 confidentiality. It does have the obvious 19 problem, and every once in a while -- 20 especially in the areas that have already 21 been mentioned -- anything associated with 22 dollars, such as price or cost, and also with BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 233 1 stocks -- people are frequently calling us 2 and saying, can't you release that data, 3 can't we see that microdata there? Because 4 that's really what they're after, but I was 5 kind of surprised to see this topic here 6 because I thought we had sort of cottoned on 7 to the withholding thing and that was it. So 8 I'm very happy, actually, to see that the 9 conversation is going this way now. 10 And I also support the idea that 11 Stan brought up, that we ought to have a 12 sunshine rule here to perhaps see if we could 13 convince our respondents and after one year 14 or two years or whatever it is, if the 15 microdata no longer would be withheld or not 16 be confidential after a certain period of 17 time. 18 DR. HENGARTNER: Last person. 19 MS. WAUGH: I'm Shawna Waugh from 20 Statistics and Methods. One of the 21 non-statistical solutions not discussed, and 22 I think ---- included in the survey is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 234 1 whether or not companies are willing to sign 2 waivers. 3 SPEAKER: We can't hear you, 4 Shawna. 5 MS. WAUGH: Another non-statistical 6 solution is asking companies to sign waivers. 7 And EIA has done that in the past, for 8 example, with your case where one company is 9 reporting a large quantity and therefore 10 might be able to discover the amount. I 11 really do think that working with our 12 respondents is another way by requesting 13 waivers, and that that might be an important 14 aspect to research when you're looking at 15 disclosure. 16 DR. HENGARTNER: Thank you very 17 much. This seems to be a very interesting 18 topic, and I do not want to cut down the 19 discussion, so I made a note to myself to 20 reopen this discussion if anybody else wants 21 to add to it at a later time during our 22 meeting. But I'd like to proceed ahead to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 235 1 two break-out sessions and I'd like Mark 2 Burton, Cutler Cleveland, Jae Edmonds, Neha 3 Khanna, Nagaraj Neerchal, and Tom Rutherford 4 to go downstairs to Room 5E-069 for a 5 break-out session on improving the SAGE 6 petroleum refinery model. Yes, that seems to 7 be a follow-up. 8 (Recess) 9 DR. HENGARTNER: Okay, ladies and 10 gentlemen, let's continue. I'd like to 11 welcome Stan Freedman to discuss with us the 12 functional requirements for EIA Internet data 13 collection systems for our break-out session. 14 So welcome, Stan. 15 MR. FREEDMAN: Thank you very much. 16 Can we turn the lights out up front, please? 17 Okay, what I would like to do is put this 18 project in some historical and organizational 19 context and then talk about the actual 20 functional requirements we have come up with 21 for the IDC, the Internet system, and then 22 get comments from the committee, primarily BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 236 1 related to did we leave any functionality out 2 of the system, were there other systems we 3 should have looked at and considered, and, 4 from a respondent's point of view, do you 5 think that this will be a successful Internet 6 system to get information into EIA. 7 About a year ago, maybe a little 8 bit more, a subgroup of the Strategic 9 Planning Group at EIA charged a number of us 10 with the responsibility of coming up with a 11 functional requirements document for an 12 EIA-wide system to collect data over the 13 Internet. To be very clear about the 14 boundaries of the project, this was 15 functional requirements, it was not 16 specification for code, it was not to 17 actually paint the screens or decide the size 18 of the buttons or whether the "submit" button 19 should be in the upper left-hand corner or 20 the lower right-hand corner, or the color of 21 the font or the background. Those things 22 need to be decided and determined through BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 237 1 usability testing and literature review, and 2 one size, we felt, did not necessarily fit 3 all, so it was not appropriate to dictate a 4 specific set of screens and that kind of 5 thing. What we wanted to do was talk about 6 functionality, what did the system need to be 7 able to do, how would it satisfy respondents' 8 needs, how would it satisfy the needs of the 9 people at EIA who would be using the system 10 to collect the information. So that's what 11 we were charged with. 12 Our research methods: We first 13 formed an interoffice team. The reason for 14 that is, as you might expect, if you want an 15 agency-wide system, you need agency-wide 16 buy-in. So there was a representative from 17 each of the offices at EIA, except for the 18 Office of Resource Management that handles 19 our personnel budget. All of the major 20 surveys at EIA were represented, as was the 21 Office of Information Technology. Those of 22 us from the Statistics and Methods Group who BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 238 1 served on the team were involved in usability 2 and interface with users and that kind of 3 thing. So it really was an interagency 4 group. 5 I would like to quickly introduce 6 the members of the group; they are the ones 7 who really did all the heavy lifting and the 8 ones who worked very hard for a year in 9 coming up with these requirements. It wasn't 10 always easy for us to get on the same page or 11 to agree what some of these functional 12 requirements ought to be, but I think what 13 you'll see is that we have a wide variety of 14 things. We have Betty Barlow from OIT. 15 Betty, you could just raise your hand? 16 Audrey Corley from the Office of Oil and Gas; 17 John Paul Deley, who's back there from NEIC, 18 who was our records management and retention 19 expert on the project -- 20 (Laughter) 21 MR. DELEY: That's for sure. 22 MR. FREEDMAN: -- and never let us BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 239 1 forget about the importance of documentation 2 and maintaining our information. My arm was 3 twisted to be the chair of the group, or the 4 cat-herder, as the case may be. Alethea 5 Jennings from SMG more than ably assisted me 6 with that, and bailed me out whenever there 7 was a problem, and told me that I was really 8 off track and wasn't paying any attention. 9 We had Tom Lorenz, who you heard from this 10 morning, from the Energy Markets and End Use, 11 who was representing the consumption surveys. 12 Paul McArdle, whom I saw in the back here, 13 who has an Internet survey that is 14 operational now. So the people we had on the 15 committee were not simply theoretical, they 16 are people who have implemented IDCs at EIA. 17 Rick Salkov -- I didn't see Rick, is he here? 18 He's from the Office of Coal, Nuclear, 19 Electric, and Alternative Fuels. Just about 20 all of their surveys are being collected over 21 the Internet now, so his experience with that 22 was invaluable. And Paula Weir, also from BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 240 1 the Oil and Gas, who has a survey in her area 2 that is collected over the Internet and also 3 has done quite a bit of research in the area 4 of IDCs and has, I believe, even written some 5 papers in that area. 6 So we had a good mix of 7 individuals, both in terms of application and 8 theoretical knowledge. Where we are right 9 now: In February, we briefed the Goal Four 10 Subcommittee of the Strategic Planning Group. 11 Nancy, who I think is at another session, is 12 actually the champion of that group. We 13 basically told them where we are. We had 14 come up with the handout that you got a week 15 or two ago of the specific functional 16 requirements of the system. We're now in the 17 process of actually drafting the report, and 18 that doesn't mean that we are past the point 19 of input from the ASA committee; in fact, 20 quite the opposite. Had we come to you a 21 year ago with a bunch of very unformed ideas, 22 I'm not sure that your input could have been BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 241 1 as thoroughly understood and digested as I 2 hope it will be today. 3 The first thing we did in trying 4 not to reinvent the wheel is we visited other 5 statistical agencies and organizations that 6 collected information over the Internet. We 7 visited Census -- you heard Tom this morning 8 talk about a software system there, Census 9 Taker, that is used -- I guess, Tom -- 10 primarily on their economic surveys, and we 11 talked to them about that, what 12 functionalities they had; we had a demo of 13 that. We also had a demo of the system that 14 is going to be used to collect the 2010 15 Census, the household survey. We met with 16 people from Bureau of Labor Statistics, 17 Agriculture, we met with Westat, and we 18 looked at the Blaise system, and how that 19 worked. We also took advantage of the 20 systems that were already in place at EAI; we 21 had extensive input and briefings from 22 electricity and coal in terms of what worked BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 242 1 for them. Paul spent a lot of time talking 2 about his survey, Greenhouse Gases, and the 3 shop survey. We also did fairly extensive 4 literature search, looking at papers, 5 conference proceedings, and seminars that had 6 been attended by the committee members. 7 Now, you may be asking yourself the 8 question which we asked ourselves when we 9 started out: If there already are a few 10 offices at EIA, maybe between a third and a 11 half of our surveys that are collecting some 12 or all of their data over the Internet, why 13 do we want an EIA-wide IDC? What's the 14 point? There are some things out there that 15 already exist. Obviously, it's going to cost 16 money to develop an agency-wide system. 17 There was a belief by the strategic planning 18 committee -- and I think that we subscribe to 19 this, or our group does -- that in the long 20 run, it is more efficient and cost-effective 21 to have a system and to have a modular system 22 to collect information than to have three or BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 243 1 four different systems that have to be 2 supported with different sets of programmers, 3 that have to be supported with different 4 computational platforms, and things like 5 that. 6 Part of the political environment 7 that we were operating in was to design a 8 system that would allow existing systems to 9 be migrated into it when there were 10 significant changes to the forms, the data 11 that were collected, or the software systems 12 simply needed to be updated. It is not and 13 will not be our intention to make a proposal 14 and then impose it upon everybody at once; 15 these are for new systems or systems that 16 will be revised. So that was one of our 17 operating criteria. 18 We basically had four guiding 19 principles: The first was that it be easy to 20 implement, modify or -- as I said before -- 21 migrate to the IDC. The second is that it 22 provide tools that enable the survey group -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 244 1 and by that, we mean the people who are 2 actually running the survey -- to have 3 ownership of their survey application. One 4 of the problems when you try to develop an 5 agency-wide tool is to get people to buy into 6 it and accept it and to take ownership of it. 7 Is it theirs, do they have a stake in its 8 outcome, and will it satisfy their needs? 9 The third point, which is probably the most 10 important from outside the agency looking in 11 -- and Barbara and others mentioned that this 12 morning when we were talking about the MECS 13 -- is the IDC, the Internet collection 14 system, has to provide value-added from the 15 respondent's point of view. Why do they want 16 to bother to use it? If they're used to 17 faxing in a form or filling in an electronic 18 spreadsheet, what's the value, what's the 19 benefit to them in filling in the data over 20 the Internet? And the third is that anything 21 we did needed to promote high quality 22 information coming into EIA, and it had to do BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 245 1 that through editing that was available to 2 the respondent. One of the goals was to 3 reduce the errors in the data before they 4 came into EIA -- that users be clearly 5 notified about when their data was submitted 6 to EIA -- in other words, they could 7 understand where they stood in the system. 8 Moshe mentioned this morning about 9 filling in a survey form, getting halfway 10 through, the session isn't saved, not wanting 11 to be bothered, so that needed to be handled 12 -- and then sort of the commonsensical things 13 about the navigation being clear and 14 straightforward and the system be usable. We 15 wanted to make sure that everything that was 16 ever developed there would be a partnership 17 between the coders, the developers, our 18 Office of Information Technology, and the 19 program offices. The responsibilities would 20 include OIT developing and maintaining the 21 overall infrastructure and providing tools 22 for the program offices to implement their BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 246 1 particular surveys. And one of the things 2 that we found when we looked at previous 3 experiences at EIA is that there needed to be 4 assistance provided to our respondents 5 through help centers and things like that 6 when they tried to use the IDC. The idea 7 here is that we would have a modular system 8 that survey managers and subject matter 9 experts could pick and choose the things that 10 applied to their survey. One size, we 11 realized, did not fit all. If you have a 12 survey with four questions on it, the look or 13 the screen that would be painted for that 14 survey would be very different, for example, 15 than for the MECS survey that has a lot of 16 complicated questions and might have skip 17 patterns and things like that. Our intention 18 was to have functionality that would include 19 all of those things and then have the survey 20 managers pick and choose those pieces that 21 best fit their system, and then that be 22 integrated into their particular online BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 247 1 collection. 2 We also are recommending to the 3 Strategic Planning Committee that we initiate 4 a pilot to put one survey into this IDC. And 5 although we are still in the process of 6 talking about how that implementation ought 7 to happen -- and I would be very interested 8 to hear the views of the committee in terms 9 of how we make that successful -- the idea 10 would be to start out simply and build the 11 system modularly and not try to put 12 everything into the system at one time. 13 We divided functional requirements 14 into several areas, and these are discussed 15 in detail in the handout that you got, and I 16 can pull up any of the slides to talk about 17 what the specifics are on that. 18 The first area of functionality is 19 actually the collection of the data. It is 20 what the respondent sees when they log into 21 the system. What functionalities are 22 required, what should that common look and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 248 1 feel be, and that kind of thing. That is the 2 heart of the system and took us the longest 3 to come up with specific functional 4 requirements. The system needed to have 5 interoperability; the pieces needed to link 6 together. And the IDC needed to link as 7 seamlessly as possible with the production 8 systems that actually aggregate the data and 9 do editing after the information is in and 10 produce the publications. Again, the IDC is 11 simply the pipeline that gets the data past 12 EIA's firewall. Then the data would be 13 passed to the existing processing systems. 14 This was not a system to be developed to go 15 from the respondent's submission to the 16 Internet table on the Web. So it had to link 17 seamlessly with that. 18 The other thing we felt was 19 important was to have some sort of 20 performance measurement in the system, which 21 would include response rates. There is 22 functionality recommended to talk about BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 249 1 errors that are made on specific cells in 2 submissions, which would give you some idea 3 of what questions respondents were having 4 particular problems with and things like 5 that. And again, we have someone here today 6 who can talk more about what those 7 performance measures would be. 8 Retention and disposition were 9 important. How long is the data maintained 10 in the IDC? What might the respondent have 11 access to, in terms of what they had 12 submitted previously? How long should we 13 keep that information in our system, both to 14 have an audit trail during the production 15 cycle and an audit trail later on? If it's 16 an annual survey, does it need to be for the 17 whole year, does it need to be for two years, 18 does it need to be for three years? That 19 functionality needs to be built into the 20 system; the needs for different offices 21 obviously will be different, but there needs 22 to be a retention and disposition schedule of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 250 1 that information. 2 Probably the most important lessons 3 learned from our site visits, and certainly 4 from the input that we got from the 5 Electricity Division, was that of essential 6 support services. The one thing that you 7 don't want to happen is to roll out an 8 Internet system and nobody's able to log into 9 it because the survey will only handle six 10 people at a time instead of 600 people at a 11 time, when they're all trying to submit the 12 data at the same time. Can people access 13 their passwords? Do their passwords work? 14 Whom do they call when they don't work? They 15 want somebody on the other end of the phone 16 who will be able to answer their questions. 17 And it's not always the same person; 18 sometimes it's an IT person, sometimes it's a 19 subject-matter person. Users tend not to 20 discriminate -- if they don't understand the 21 question on a form, they're going to call the 22 first phone number that they see in front of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 251 1 them or click the link to the first e-mail, 2 and they don't want to be referred to six 3 places for an answer to the question. That 4 was something that EIA may not have planned 5 as well for in the beginning with some of 6 their systems as they could have; we didn't 7 want to make that mistake again. And that's 8 a pretty resource-intensive activity. 9 And the other area of importance 10 was security. Right now, EIA uses a single 11 sign-on system; it seems to work well, and 12 our recommendation is that we continue to use 13 that, the idea being there is one secure 14 place where a respondent would go to be able 15 to access any of their Internet forms. They 16 wouldn't have to go to four or five different 17 URLs -- they may need different passwords or 18 authentications for different surveys that 19 they're responding to, but there would be one 20 place where they go. So those were the areas 21 that we were concerned about. Again, from 22 the respondent's point of view, it was the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 252 1 common look and feel. 2 The other lesson that we learned 3 and heard over and over again -- in fact, I 4 can remember Bob Schnapp, who I think is in 5 the back of the room, saying to us at a 6 meeting, the value of the IDC to us is to 7 force the respondents to provide us with the 8 correct data the first time. And we talked 9 about some of those things this morning. 10 MECS is interested in eliminating 11 keying errors and unit errors -- people 12 reporting in volume instead of heat units and 13 vice versa -- there's a lot that you can do 14 on the front end of an Internet system to 15 ensure that those mistakes are not made. We 16 classified our editing into two categories, 17 and you'll see if you go through the specific 18 requirements, one of them -- Paul, help me 19 out here if I'm not using the term we're 20 currently using in the report -- hard edits 21 versus soft -- okay, thank you. I was out 22 for two weeks sick, and the words might have BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 253 1 changed -- but the idea was still the same, 2 and that is that there were some edits that, 3 when they happened, the respondent would be 4 notified and notified in a place on the form 5 that made some sense that they could see, and 6 they would not be able to proceed until they 7 corrected those edits -- things like 8 incorrect state codes, incorrect units, and 9 things like that, those were the hard edits, 10 you don't get to submit your data until 11 you've satisfied those. Then there were the 12 soft edits, things that might be statistical 13 edits, things that would have range checks, 14 and our decision was to leave whether they'd 15 be hard or soft to the program people, but 16 those edits could be overridden with 17 explanations or not overridden, depending on 18 what the edit would be, but the respondent 19 would still be notified. Again, over and 20 over again, we heard that many of the errors 21 are clerical or keying errors that are made 22 when there's a submission, and the computer BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 254 1 is real good at picking those things up, and 2 every effort should be made to capture as 3 many of those as possible and eliminate them 4 before the submission actually came into EIA. 5 So with that, I would like to first 6 open it up to any other member of the 7 committee who would like to add anything 8 specifically about their areas, common look 9 and feel, interoperability, performance 10 measurement, to help inform the committee and 11 then to get comments back from the committee 12 in terms of, did we leave anything out, are 13 there any systems out there that we should 14 have looked at that I haven't mentioned. 15 If you're interested, we passed 16 around a flow chart of how the data would 17 come into the system. If you're interested 18 in that level of detail, we could certainly 19 go over that to explain how that would work. 20 So if we could have the lights -- and then is 21 there anybody from the committee who'd like 22 to add anything either about common look and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 255 1 feel, performance measurement -- 2 DR. HENGARTNER: Thank you very 3 much, Stan. So as Stan suggested, this is 4 now an open-floor discussion, so I invite 5 everybody sitting on the sideline to move in 6 closer, especially those who are expert about 7 Internet and Web design. Please feel free to 8 join us, the committee members as well that 9 Stan was referring to. There are plenty of 10 spaces here. It makes our discussion easier 11 and more lively. Thank you. You, too, Stan. 12 MR. SINGPURWALLA: Where is all the 13 data stored when you push it through the 14 Internet thing? Is it on, like, Oracle 15 tables or -- 16 MR. FREEDMAN: I'm going to ask 17 Betty to address that question. 18 MS. BARLOW: Yeah, it would be 19 Oracle. 20 MR. SINGPURWALLA: Okay. 21 MR. FREEDMAN: Do you want to 22 explain a little bit about how the data would BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 256 1 come in and the storage of passing the files, 2 if that would be helpful -- at least in terms 3 of how things work now and how we potentially 4 envision it. 5 MS. BARLOW: Well, they'll be 6 stored -- if I say the word "screen subnet," 7 do you know what that would refer to? It's 8 an area that's walled off from our internal 9 LAN and also walled off from the Internet. 10 So that's where we could have the databases 11 and the applications. So the data come into 12 that Oracle database in the screen subnet. 13 And there are actually two possibilities that 14 Stan sort of alluded to. We understand that 15 some of the survey managers will want to have 16 that data transferred to another database 17 inside because they already have a processing 18 system which has its database. So they'll 19 want to use a different database for their 20 internal processing. We also envision that 21 some of the survey managers might want to use 22 that database in the screen subnet directly, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 257 1 use that very database, to do whatever they 2 do inside their internal processing. So both 3 options would be available. 4 MR. FREEDMAN: One of the things 5 that happens now on some of the IDCs in the 6 electricity are -- and I think, Scott, maybe 7 in the coal area too -- is that if a 8 respondent does not submit their data over 9 the Internet -- if they fax it in, for 10 example -- EIA people can use the system to 11 key the data into the database. The other 12 thing is there will be, I'm sure, 13 requirements from some of the survey systems 14 to allow respondents to look at their 15 historical information when correcting an 16 edit flag. So there will be functionality 17 built into the system for those surveys that 18 require it that would allow respondents to go 19 back and look at their historical 20 information, and that gets into the issue of 21 how long and what kind of data get retained 22 in the IDC. The IDC is not intended to be a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 258 1 permanent repository for data for 20 years -- 2 it doesn't make sense and it's not practical 3 -- but there will be a period of time when 4 there are data that have to be stored in 5 order to satisfy edit criteria and that kind 6 of thing. I don't know if that's more of an 7 answer to your question than you wanted -- 8 MR. SINGPURWALLA: No, no, no, 9 that's a great. Wouldn't how much data you 10 can save depend more on the size of the work 11 or database behind the IDC? 12 MR. FREEDMAN: I would have to 13 refer that to Betty. 14 MS. BARLOW: Which is variable. We 15 can always increase it. 16 MR. SINGPURWALLA: Yeah, yeah. 17 MS. BARLOW: And there's another 18 point, too, that I would make. Some people 19 are concerned about having data in the screen 20 subnet. We have the intention -- I don't 21 know when it's going to happen -- of putting 22 a proxy server in the screen subnet, and that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 259 1 proxy server is simply there to route 2 incoming information. And so in that case, 3 you could have both the application and the 4 app server -- containing the applications -- 5 and the database server inside, on the LAN, 6 and you have less exposure in that case. So 7 that's the plan down the road somewhere. 8 DR. HENGARTNER: I have one concern 9 with the ability of recalling data from past 10 survey. That concern has to do with 11 confidentiality. Would you be able to spit 12 back data that you are deeming confidential 13 to a user which may be different from the one 14 who had actually entered the data? 15 MR. FREEDMAN: The only one who 16 would have access to the system is the person 17 who had the password in the company. Now, if 18 a company chose to give their password out to 19 all sorts of people in different departments, 20 I'm not sure what we could do about that. I 21 think I understand what your point is -- 22 DR. HENGARTNER: It's not even a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 260 1 point, it's a question. It's a red flag in 2 my mind that if you are providing data back, 3 then that is not quite publishing it, but it 4 comes very close to it, and it's a danger. 5 MS. BARLOW: But it's only 6 accessible to the company people, or people, 7 as Stan said, to whom they choose to give the 8 password, or somebody else has identity -- 9 MR. SINGPURWALLA: So only I could 10 see the data that I put in. 11 MS. BARLOW: Well, not you 12 literally, but your company. 13 MR. SINGPURWALLA: Well, your 14 company. 15 MS. BARLOW: Yeah, your company. 16 MR. SINGPURWALLA: And whoever ---- 17 to the committee -- 18 DR. FEDER: Someone who's trusted 19 by the company. 20 MR. SINGPURWALLA: Mm-hmm. 21 DR. HENGARTNER: Yeah, but this is 22 one more password you need to crack to get BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 261 1 all this information. Let's face it, if I'm 2 going to make industrial espionage, that 3 would be a weak link, and I would be 4 concerned about that. 5 MS. BARLOW: Well, you have to have 6 a user ID and password to get access to that 7 particular data. 8 DR. FEDER: It's like I have my 9 password for my banking online -- 10 MS. BARLOW: Right, same thing. 11 DR. FEDER: -- if I give it to 12 someone, it's my -- 13 SPEAKER: It's your problem, not 14 the bank's. 15 DR. FEDER: Can I just -- I 16 actually liked the flow chart that was -- 17 MR. FREEDMAN: Would you like us to 18 put up on the screen -- 19 DR. FEDER: No, no, no, no. I --- 20 the help files that I liked. I would just 21 add one more thing, a user feedback form 22 that's separate from the questionnaire, where BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 262 1 you go by e-mail to the developers, not part 2 of the questionnaire because you know and I 3 know how long it takes until survey data 4 processed. I just would like you to go and 5 say I used your system, and I really wanted 6 to get to this, but the skip pattern prevents 7 me from the -- would you please enable me -- 8 MR. FREEDMAN: I understand, yes, 9 yes. 10 DR. FEDER: -- or something like 11 that, so a user feedback. 12 MR. FREEDMAN: Okay. Bob, I think, 13 was going to mention something about -- 14 MR. SCHNAPP: Yeah, well, I think 15 about going back and looking back at previous 16 years' data, an issue that for -- raised the 17 other day -- 18 MR. FREEDMAN: Bob, they want you 19 to use the mike. 20 MR. SCHNAPP: I thought I talked 21 loud enough. I might not talk good, but I 22 can talk loud, they say. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 263 1 There's an issue about going back 2 in time to look at previous years' data, and 3 in a different respect, it's important 4 because if the company is sold, is the new 5 company allowed to see the previous several 6 years' worth of data? Do they buy the data 7 with the company or not? And I'm sure that 8 there are contracts that cover it -- they 9 don't come to us for us to see -- and there 10 are a lot of companies that sell off part of 11 their assets, keep assets, and they don't 12 want the new company to know some of the 13 things that they were doing before or costs 14 that they had. So it's something that was 15 raised, at least in our group recently, that 16 we were going to begin to address, and so 17 maybe if people here have an idea, people 18 have to listen to it. 19 MR. FREEDMAN: That's something we 20 actually hadn't thought about in our group. 21 I don't think it's -- Paula, you wanted to 22 say something. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 264 1 MS. WEIR: Well, I was going to 2 say, I think it varies by survey, and that's 3 one of the areas that we thought each survey 4 would decide what their edit criteria going 5 in -- there's a part of this called the 6 administrative module that we would be using 7 to put in our parameters and our rules and 8 all that, and within that, as well as what's 9 hard, what's soft. It would be within that 10 framework that the survey manager would 11 decide. They could just choose not to do 12 historical edits, period, or if there's a 13 sale, merger, that you don't go back in time. 14 But, Stan, the part I was going to 15 comment on is that -- in case people didn't 16 get a chance to see this -- Stan had talked 17 about two points, one of which was that we 18 really view this as a way to save money 19 because of data entry. There is no money to 20 be saved in data entry; I mean, that's it. 21 There is some money in systems design, but 22 the other point is in the data editing. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 265 1 Forty percent of our survey costs are spent 2 on data editing. The more we can get the 3 respondent to do, that's money back in our 4 pocket that we could use on systems as well 5 as overall quality. So it's the overall 6 quality that we also expect to go up. 7 And the second point was about the 8 buy-in; from the respondent's point of view, 9 why should I do this? I'm doing just fine, 10 thank you very much -- is the two things that 11 we're considering -- or what we've put in the 12 requirements is the electronic file transfer 13 that Betty referred to a little bit. Because 14 a large company doesn't want to be entering 15 data in through an IDC, so we will have a 16 format for electronic file transfers. Once 17 the data's in to us, they can have the same 18 screens, it's treated the same, but they 19 don't have to go through that data entry 20 process. 21 And the second thing is unlike 22 household surveys, where the incentive for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 266 1 response -- you're always trying to balance 2 the quality of the edits with the high 3 response rates -- in household surveys, there 4 are incentives to be gotten; you give them 25 5 bucks for filling out the survey or something 6 like that. In business surveys, such 7 incentive haven't existed until Internet data 8 collection, and the thoughts that are being 9 considered is they're information that you 10 can give back to the company that they 11 wouldn't have otherwise. But you've got a 12 perspective, different, because you've got a 13 lot of data. What can you give back to that 14 company to encourage them to use this tool? 15 And that's the other part we're encouraging, 16 that the use of that, the development of that 17 type of approach, you can still do it within 18 a confidentiality framework, but we see 19 things that nobody else sees, and we have all 20 the data. And that's how you try to get 21 their buy-in. 22 MR. FREEDMAN: And if the committee BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 267 1 has any idea of what some of those things 2 might be, that would be helpful for us. 3 DR. HENGARTNER: First Derek and 4 then -- 5 MR. BINGHAM: I think this morning, 6 it was mentioned -- so you were talking about 7 keying errors and things like that and that 8 there's potential for lack of -- maybe not a 9 lot of potential for reduction in that, but 10 the analyst goes in and changes it. That was 11 the one thing I learned from Tom this 12 morning. He can go in and say, that's not 13 right, and he'll change things. Now, here's 14 a question: If you have historic data up 15 there, do Tom's changes get flushed back into 16 the original database where the historic data 17 is? 18 MS. WEIR: Yes. 19 MR. BINGHAM: Are you going to 20 notify a company beforehand that -- 21 MS. BARLOW: That's a survey 22 decision. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 268 1 MR. BINGHAM: Because if you're 2 going in, right, I could imagine -- so if 3 there's the original database, which the Web 4 application sends data to, it gets 5 transferred over -- there's probably 6 functionality that transfers this data to 7 some other database, which the survey folks 8 use -- and an analyst like Tom goes in and 9 says, that's wrong, I'm changing that. There 10 has to be a procedure to send it back to the 11 original database, and the next year the guy 12 logs in and says, how -- a piece of historic 13 information comes up, and it's not what he or 14 she entered -- they may be puzzled as to why 15 this wasn't the same, and they would think of 16 it as an error. So I'm just curious if there 17 are any reporting mechanism to say, oh, by 18 the way, these are the things that we 19 changed. 20 MR. McARDLE: I can address that, 21 at least on a survey-specific -- for the 22 greenhouse gas survey. If we find an error, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 269 1 and we change it, we will contact the company 2 then and there, via telephone, and get the 3 contact representative from the entity 4 submitting the data to discuss that and make 5 sure they're on board with the change. And 6 that's how we'll close it out. We'll note 7 that in what we call a -- we keep a 8 transcript of these -- not necessarily a 9 transcript, but a chronicle of these 10 communications, and we'll note it in that 11 document that we spoke to so-and-so on this 12 date, and we agreed to change this number and 13 we'll put that into our auxiliary database of 14 contacts. 15 MR. SINGPURWALLA: You're 16 suggesting, though, that the functionality 17 exists for notifying the respondent if those 18 changes are made. 19 MR. BINGHAM: I don't know, so 20 something like that, like a log that would -- 21 if they look at the -- accessing the historic 22 data, then it would either be flagged and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 270 1 say, this was changed and -- 2 MR. McARDLE: In some cases -- say, 3 if there was extensive changes -- we may ask 4 them to submit another certification letter. 5 MR. BINGHAM: Sure. 6 MR. McARDLE: But if it's something 7 small, we'd probably just note it in what we 8 call our RTD, which is a record tracking 9 document, which tracks those transactions, so 10 to speak. But if it's extensive, we'll say, 11 okay, we got a lot of changes, go look at 12 this again, and then send us another 13 certification letter saying that you agree 14 that these are the data that are represented 15 in the company. 16 MR. WATSON: Bill Watson, CNEAF. 17 I'd like to come back to the point that was 18 raised earlier about the committee's desire 19 to design a common or one-fits-all-size IDC 20 system for EIA. We're pretty much down the 21 road at this point with IDC, and I think the 22 model that we've been using up to this point, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 271 1 I would characterize it as being a 2 distributed computing model. And I think it 3 has a lot of strengths compared to having 4 something that is an attempt, actually, to 5 define enough stuff so that everybody could 6 use a common model. And the reason I say 7 that is because in almost every other area 8 where we do IT, it has already been 9 demonstrated many, many times -- for the most 10 part, there are some exceptions -- that 11 distributed computing tends to be a better 12 model than centralized computing. 13 Now, in the distributed computing 14 system that we now have with our IDC, I 15 really see four essential things that make 16 the system work and make it work well. One 17 is the use of user's guides. It's absolutely 18 essential that our respondents understand 19 what they're expected to do and have 20 something that they can see electronically 21 that's very simple, very visual, that helps 22 them get through the single sign-on system, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 272 1 that helps them get to their survey form. We 2 had very good luck with providing those to 3 our respondents in the case of certain call 4 surveys. 5 A second thing is a notification 6 system. People need to be reminded that 7 there's a certain due date, and they might 8 have to be reminded several times. And the 9 IDC systems really facilitate that. It's 10 very easy to send mass e-mail notices out to 11 people and remind them. So you can get very 12 high response rates by due dates. We 13 typically get 80 to 85 percent of people 14 responding on coal surveys by the due date. 15 So that would be another feature. 16 And then, increasingly, we have 17 developed tools that allow us to look at 18 response rates, to look at how many forms are 19 clean, how many forms come in by IDC versus 20 fax or e-mail, and all of these things are 21 now electronic. All of the work that we did 22 previously, a lot of it would be a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 273 1 combination of manual methods and electronic 2 methods. That has fallen to the side; our 3 methods are just so much better in that area. 4 And then I think the most important 5 thing here is best practices. I think we 6 have, for the most part, not explored that 7 very well because what's happened, at least 8 from my perspective, is that coal does its 9 IDC, electric does its IDC, renewables, they 10 do their IDC, oil does their IDC, natural gas 11 does their IDC -- we're not talking with each 12 other, we're not sharing our experiences. I 13 have no idea about what other IDC systems 14 look like. I'm just kind of curious about 15 why you didn't think about sort of the way we 16 were going and decided maybe that's not so 17 good and we need to go in a different 18 direction here. Because I think the 19 direction we're going -- if we really exploit 20 best practices and have some sense of what 21 works and works well -- because people have 22 demonstrated it works well -- it's going to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 274 1 be more powerful than almost anything else we 2 can do here. 3 MR. FREEDMAN: Bill, maybe I wasn't 4 as clear as I could be in the presentation. 5 I think that we did exactly what you're 6 suggesting that we do, that we did look at 7 the best practices of each of the groups 8 within EIA and organizations outside of EIA 9 to draw upon that to come up with the 10 functional requirements document. 11 In terms of whether the computing 12 ought to be centralized or distributed in 13 terms of costs and efficiency, I'm not 14 necessarily the best one to give you 15 statistics on that, but from what I have seen 16 and from what we have seen looking at other 17 systems, there are some efficiencies to be 18 gained not to have to support multiple 19 platforms and multiple systems, so that when 20 there is a change that needs to be made or 21 software modifications, it doesn't need to be 22 made once in coal and once in electricity. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 275 1 But in terms of looking at best 2 practices and integrating those things and 3 taking the work that's already been done and 4 integrating that into the functional 5 requirements of the IDC, I think that's 6 exactly what the team did, as opposed to 7 saying we're throwing everything out and 8 we're starting over again. In fact, we 9 purposely did not do that. We built on 10 what's been done in electricity, what's been 11 done in coal, greenhouse gases, and the other 12 statistical agencies. So I'm sorry if that 13 wasn't as clear as it could be. 14 Please, Audrey. 15 MS. CORLEY: I'm Audrey Corley. 16 Almost everything that you named are 17 requirements that we have in the functional 18 requirements document: The easily accessible 19 user's guide, electronic reports for 20 performance statistics, non-respondent early 21 reminders through mass e-mail notification, 22 looking at best practices from BLS, from BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 276 1 internally -- we went to Agriculture, we went 2 to the Census Bureau, saw two different types 3 of Internet data collection tools at the 4 Census Bureau, and we came up and saw 5 CNEAF's. Everything that you name, with the 6 exception of the computer system, we have, 7 and it's going to be in the document as a 8 requirement. So we've been doing this for 9 about a year and a half. And we come from 10 all different areas in EAI, so we definitely 11 took into account everything that you're 12 saying. 13 MR. FREEDMAN: And in fact, when 14 you see -- if you go through the detailed 15 slide, and then once the requirements 16 document is actually written, I suspect that 17 you will recognize many of the things in 18 those requirements that exist in the coal 19 system right now, because the coal and 20 electric systems are pretty well developed 21 and have a pretty long track record now -- 22 what, two or three years -- with BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 277 1 electricity -- 2 MR. McARDLE: Four. 3 MR. FREEDMAN: -- four years for 4 what worked and what didn't work. And we're 5 not trying to redo those. It's pretty clear 6 that some of the things that are working 7 there work real well. And some of the best 8 practices work well, and some of the things 9 where CNEAF got off to a little bit of a 10 rough start served to inform the discussion 11 about how the requirements ought to be 12 different. Betty -- 13 MS. BARLOW: Stan didn't give you 14 any details about the common look and feel, 15 but we really think that this is an important 16 point for future surveys in an IDC in EIA for 17 this reason. We have a fair number of 18 respondents who report on more than one 19 survey in more than one area, too, more than 20 one energy area. And we feel it's important 21 that the respondent, first of all, always 22 knows he's at EIA and that he doesn't have to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 278 1 learn over again when he goes to his second 2 survey what he learned on his first survey. 3 So we want some commonality. We're not 4 trying to say they're all going to be exactly 5 alike -- and Stan did make that point -- 6 that's very important for us. There's a lot 7 of room for the survey manager to make 8 choices. 9 However, we do think that there 10 should be common structure and that we have, 11 for example, a common look to the buttons, a 12 common set of buttons so the respondent 13 always knows what it means from one survey to 14 another about the word "submit," say, or the 15 word "cancel." He knows what that means, he 16 doesn't have to relearn it for the second 17 survey. We felt that these ideas were 18 extremely important, and that's one of the 19 good reasons why we think that this IDC is a 20 good idea, in addition to saving development 21 costs, which is what Paula mentioned. 22 MR. SINGPURWALLA: How many times BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 279 1 could a responder be hit by a survey within a 2 year? Like, how many times am I going to 3 have to log in, typically? 4 MS. BARLOW: Well, weekly is 52. 5 MR. SINGPURWALLA: Weekly. 6 MS. BARLOW: We have some weekly 7 surveys. 8 SPEAKER: Some surveys are weekly, 9 some are monthly, some are annually. The one 10 I work on is quadrennially. 11 MR. SINGPURWALLA: Yeah, that's the 12 one. 13 SPEAKER: So it's kind of a 14 significant job if you're being hit with a 15 weekly survey. 16 MR. FREEDMAN: Also you need to 17 appreciate the fact that an electric power 18 generator might get two or three electricity 19 surveys, and some of those might be monthly, 20 and some of them might be annual, so there 21 could be 15 or 20 times a company -- not 22 necessarily the same respondent filling in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 280 1 each form -- but the company have to submit 2 some survey instrument to EIA. 3 MR. SINGPURWALLA: Can the IDC, 4 based on my common sign-in name and password 5 -- and I don't even know if this is the right 6 term for it -- push data the other way so I 7 don't have to type in my address all the 8 time? 9 MS. BARLOW: Yes. 10 MR. McARDLE: Yes, we can do that. 11 MR. FREEDMAN: That functionality 12 is a requirement if the survey wants to do 13 that. Many of EIA's surveys populate the 14 form -- back in the old days, when they were 15 printed and sent out -- with historical 16 information that the respondent would simply 17 update or change, especially when it comes to 18 contact information, street address, phone 19 numbers, and that kind of thing. So yes, the 20 functionality will be there to populate the 21 form when the respondent pulls it up. 22 MR. WATSON: A part of that is when BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 281 1 the respondent gets to that page and they see 2 that information, if that's a new person 3 who's been hired by the firm, and they don't 4 see their name there, and their name needs to 5 be there in the existing systems and in this 6 system that's being designed, that would be a 7 functionality for somebody to enter a screen 8 and provide a change to that information, 9 which would then be put into the system, so 10 that the next time they logged on, that 11 information would be updated to them. 12 MR. SINGPURWALLA: I don't know the 13 surveys all that well, but I'm thinking that 14 they probably have different sections in 15 them, in one survey. 16 MR. WATSON: They do. 17 MR. SINGPURWALLA: Have you guys 18 gotten to the point where you figure out when 19 you log in to something, is there, like, a 20 home screen for the user? 21 MR. WATSON: Yes. 22 MS. BARLOW: The first screen that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 282 1 we envision that they would see -- after the 2 login, of course -- is a screen that gives 3 them all the access and allows them to update 4 their account profile, like contact 5 information, and then gives them access to 6 whatever surveys they should have access to. 7 MR. SINGPURWALLA: Oh, and, like, 8 the name and address form, whatever, if they 9 want to make changes, like I said, 10 basically -- 11 MS. BARLOW: Right, yeah. 12 MR. WATSON: But then once you're 13 into the form -- and I guess this would be 14 carried over to whatever the committee has or 15 team has designed here -- many of the IDC 16 systems have tabs at the top, like you see in 17 a lot of Windows applications, and so you're 18 tabbing from one part of the form to the 19 other. And people have talked about the 20 payoff here in terms of self-editing, there 21 is a -- call it the process tab, which you 22 sort of hit at the end, and it will give you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 283 1 a tally on how many things have thrown up 2 edit flags. 3 So there would be a place there 4 that you would click to see all the edits, 5 and you could either say, oh, I made a 6 mistake and go back and correct the data; or 7 the data are correct, and you enter an edit 8 override comment. Now, that edit override 9 comment is captured electronically and then 10 somebody on the federal or contract staff on 11 the receiving side sees that in an electronic 12 report and most of the time, these days, 13 people put in reasonable answers, and so you 14 just say, okay, that's the end of it. The 15 previous paper system, you had to call that 16 person and get them on the phone and clear 17 that edit with them. 18 And just to give you an idea about 19 what's happening, on one of the surveys that 20 I run, on the old paper system, in an annual 21 cycle, we would have roughly 800 telephone 22 calls of the type I'm talking about to clear BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 284 1 an edit because it came up on a paper form. 2 We got the paper form, we couldn't talk to 3 the person except by calling them on the 4 phone. Now, they clear that -- and there are 5 a few that we have to follow up on; we follow 6 up on roughly 80 of them. So it's reduced 90 7 percent of the effort, and that's a 8 tremendous payoff to the assistants. 9 MR. SINGPURWALLA: Have you guys 10 quantify it like how much this is going to 11 save at all? 12 MR. WATSON: I have attempted to, 13 but I don't know ---- share with anybody. 14 MR. SINGPURWALLA: Then it's 15 probably pretty good. 16 MR. WATSON: It should be 17 substantial at some point. I think the 18 important thing to keep in mind, here, of 19 course, is these systems are sort of building 20 to the point where they're fully implemented, 21 and we have a good database of e-mail 22 addresses and all of that stuff that you need BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 285 1 for good electronic systems. So it takes a 2 little while to get up the curve. But I 3 think when you get up the curve -- my 4 expectation is that we ought to be able to do 5 what we were doing previously with a quarter 6 of the money and staff. That's sort of what 7 I see happening here. 8 MR. SINGPURWALLA: My last question 9 would be, did you speak to end users as well 10 when getting it ------? 11 MR. WATSON: You mean respondents? 12 MR. SINGPURWALLA: Yeah, 13 respondents. 14 MR. WATSON: No, we haven't. There 15 is a requirement during the development 16 effort, however, to go out and do user 17 testing with respondents, and EIA has 18 actually done that with the shop program with 19 Paula and greenhouse gases, where we've 20 actually gone out with a laptop and gone 21 through -- well, and we did it with the 22 electricity markets as well, where we've BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 286 1 actually gone out and done user testing and 2 gotten feedback in terms of navigation, look 3 and feel, understanding what the buttons are. 4 And as we begin to develop these pieces, 5 we'll want to test that out to make a 6 determination at that point where the layout 7 ought to be. 8 The other thing that we have found 9 from previous testing that -- again, I didn't 10 go over any of the details of the 11 requirements because it would take a couple 12 days to go over all of the specifics, but one 13 of the big things that we always found when 14 we talked to respondents from the user 15 testing point of view was the issue of 16 notification, submission, and then being able 17 to save the information in some way that they 18 could share it with others in the company, 19 either for verification before the submission 20 takes place, or as a historical record. So 21 those functional requirements are also in the 22 system. People are very unsatisfied with BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 287 1 closing a browser and not getting a 2 notification back that the data actually 3 arrived at EIA. Even though it may have 4 arrived 20 minutes before when they moved 5 from one screen to another, unless they see 6 something in front of them -- I mean, I'm the 7 same way. Unless something comes back and 8 says, thank you, I have it -- 9 MS. FORSYTH: With a confirmation 10 number. 11 MR. FREEDMAN: Yeah, exactly. So 12 that kind of functionality -- we already know 13 from testing and talking to users, but I 14 think there are many things that we will find 15 out about the forms. I also think that that 16 testing informs design process. There is not 17 a requirement -- and I have a personal 18 opinion about this that I'm not sure everyone 19 on the committee shares -- but I don't think 20 that there ought to be a requirement that the 21 Internet form replicate the paper form. 22 Now, there are people who do not BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 288 1 agree with that, and it's a legitimate 2 debate. But I think the testing will help 3 inform that process, and certainly the survey 4 manager will be able to choose -- some of the 5 CNEAF forms work very well with tabs because 6 there are different schedules, but there are 7 other forms that may work better in a more 8 linear top-to-bottom, depending on whether 9 there were skip patterns and things like 10 that. We didn't want to mandate everything 11 had to be in tabs or schedules, and there are 12 things you can do on the Internet or 13 electronically that you can't do on a piece 14 of paper, and again, that may change the 15 design of the form. 16 MS. PEARSON: Tina Pearson, NEIC. 17 To follow up on that, I have a question -- 18 this is my first week here, so bear with me 19 if it's somewhat naïve. The electronic forms 20 do not replace the paper forms -- do they 21 coexist or can you go completely electronic? 22 MR. FREEDMAN: This is kind of an BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 289 1 interesting paradox. John and Paul, help me 2 out here if I get the laws wrong. 3 (Laughter) 4 MR. FREEDMAN: There is an 5 e-government act that encourages federal 6 agencies to allow people -- 7 MR. LORENZ: Paperwork Reduction 8 Act. 9 MR. FREEDMAN: Paperwork Reduction 10 Act, thank you -- to respond to them 11 electronically, and that's a good thing, and 12 that's the direction we want to move in. On 13 the other hand, when you clear a data 14 collection through the Office of Management 15 and Budget, you have to allow a company to 16 fill it out on a piece of paper, put it in an 17 envelope, and mail it back to you. You 18 cannot force them to do it electronically. 19 So Audrey's raising her -- is there an 20 exception, Audrey? Correct me if I'm wrong. 21 MS. CORLEY: No, I don't have an 22 exception, I have a comment. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 290 1 MS. PEARSON: So you do both. You 2 do both. 3 MR. FREEDMAN: Well, we have to 4 have the capability to do both, to have 5 something faxed, to e-mail it in. There have 6 been offices here at EIA that have strongly 7 encouraged and put procedures in place to 8 make sure that their respondents will fill 9 data in over the Internet, and it has met 10 with a pretty high degree of success. But if 11 somebody were to write to their Congressman 12 and say, I want to mail this form back to 13 EIA, four months later when we get it through 14 the X-ray machine with the anthrax, we'll get 15 the form. Audrey, go ahead, please. 16 MS. CORLEY: I just wanted to give 17 an example. We have one survey we conduct 18 annually -- I'm in Oil and Gas, we conduct 32 19 surveys. We have respondents that will write 20 to you who -- especially on a crude oil and 21 gas production survey -- and they say, we 22 don't have a fax machine, we don't have a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 291 1 computer, that's the way we like it, send us 2 the paper. And you don't want to tell them, 3 no, you -- we don't want to end up not giving 4 them any way to report, so we send them the 5 paper. But what we first try to do is e-mail 6 them -- but, of course, they don't have 7 e-mail. So there's always at least -- in the 8 short run, there are going to be a number of 9 companies that you still have to mail the 10 forms out to. 11 MS. BARLOW: What is it, 12 percentage-wise? Do you have a notion? 13 MS. CORLEY: Okay, Paula, I -----. 14 (Laughter) 15 MS. WEIR: In oil and gas, I 16 believe we've been using some other 17 electronic methods that -- the Excel 18 spreadsheets have been -- 19 MS. BARLOW: Yeah, it's all 20 electronic. 21 MS. WEIR: -- the most successful, 22 yeah, and our predecessor, -----. But it BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 292 1 does vary. CNEAF has certainly the highest 2 Internet of all -- 3 MS. BARLOW: Well, I didn't mean 4 Internet, I just meant electronic versus 5 paper. 6 MS. WEIR: Any electronic. I think 7 they're still ahead -- 8 MS. CORLEY: I think the surveys -- 9 it's 50-50. 10 MS. BARLOW: Really? 11 MS. CORLEY: A couple, like the 23, 12 which is an annual survey, many, many more 13 people still mail in. But for the monthly 14 surveys and for the weekly -- shoot, for the 15 weekly, it might be 70, 80 percent. 16 MS. WEIR: It varies by the 17 frequency that the survey's conducted, as 18 well as the population. We have the gasoline 19 and diesel -- there's no electronic other 20 than telephone. I mean, it's a CATI system, 21 but it's never going to be an Internet system 22 because when you're sitting at the gas pump, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 293 1 you can't send your data through Internet 2 right now. Maybe in the future, they'll all 3 have something like that, but right now, it's 4 not so -- that's why it varies so much by 5 survey. 6 But we believe -- with all this 7 multimode, is what we usually call it -- 8 there's multimodes of going on -- we see the 9 Internet as probably the most important in 10 development, and we also see it as the point 11 at which it can all be integrated no matter 12 how it comes from to start getting all this 13 stuff together in one place for better 14 performance measures to design our survey 15 instruments, in whatever form it's in, better 16 because we will have that information out in 17 one place to do it regardless of multimode. 18 But we're trying to think way out there in 19 developing this. 20 MS. PEARSON: Could you talk a 21 little bit about what to extent, if at all, 22 the feeling is in the short term or the long BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 294 1 term how much the IDC would feed the public 2 information, dissemination mechanism that 3 you've got, be it Web, print, e-mail, those 4 kinds of things? 5 MS. BARLOW: This is not 6 dissemination at all. It's just a 7 collection -- 8 MS. PEARSON: Right, I understand. 9 This is just collection -- 10 MR. McARDLE: Actually, I can 11 address that. There's actually another EIA 12 committee called the Database Management 13 System Commonality Team that's also reporting 14 to the EIA's Strategic Plan Goal 14, so we're 15 addressing that issue, and I think Nancy 16 Kirkendall is trying to set up a briefing for 17 you on -- I'm a member of that team too, as 18 is ------, but he's not here. But we're 19 addressing that issue as well, but this is 20 just designed for the data collection aspect 21 of it. The back end and the dissemination is 22 where the database management systems come BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 295 1 in, and that's a separate step along the way. 2 MS. WEIR: Stan? 3 MR. FREEDMAN: Go ahead, Paula, I 4 just -- 5 MS. WEIR: Separate, but related. 6 We also recognize that as the data come in 7 the door, there are opportunities that only 8 exist as the data come in the door. The 9 whole idea of metadata as it passes through 10 to dissemination, we think is very important, 11 so we're trying to hold that concept at the 12 same time, even though it's not a 13 dissemination system, it is the beginning of 14 the throughput of information. 15 MS. FORSYTH: Stan? 16 MR. FREEDMAN: Please. 17 MS. FORSYTH: You asked about 18 potential incentives to respondents, and so 19 while you were talking, I was making a list. 20 And I'm a naïve observer, so you guys 21 probably already know these -- well, you can 22 handle this -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 296 1 MR. FREEDMAN: Oh, okay, I was 2 going to put my -- okay. 3 MS. FORSYTH: So I'm naïve, and you 4 probably already know this because some of 5 the advantages or some of the incentives are 6 already implicit to some of your paper users, 7 but you're suggesting offering technical 8 assistance, and so the ability to get 9 tailored technical assistance like, I'm 10 having a problem with Question X. Either 11 through electronic communication or through a 12 telephone conversation -- that's a benefit 13 that's not easy to get when you're doing a 14 paper form, it's just not as automatic or not 15 as interactive. You were talking about 16 feedback on status, and I think that's not 17 something that you get right away from a 18 paper form, certainly not from a mailed paper 19 form. So we've received your submission, 20 well, that's a benefit to the respondent, 21 compared to a mail-back. 22 Maybe with a fax, "fax received" is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 297 1 sufficient, but I think that feedback on 2 their status is an incentive. We talked this 3 morning in the talk -- I think it was Bob 4 Adler's talk -- about the ability that an 5 Internet data collection system gives you to 6 be flexible with reporting units. 7 I understand that some of your 8 respondents now have their forms ready to go, 9 but you will always have new respondents who 10 will have to convert themselves to your fixed 11 units or they could use your IDC and select 12 their own units for reporting and have the 13 form do the computations required, so I think 14 that's a potential benefit, the flexibility 15 it gives you in reporting. Queries to their 16 own data, we've already talked about. I 17 think that's useful not only for editing 18 purposes, but maybe they want a historical 19 record or maybe they want to go back and get 20 trends over time, and so -- 21 MR. FREEDMAN: One of the things -- 22 just to interrupt you quickly, because it is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 298 1 my favorite line form the cognitive 2 interviews that we do, respondents will 3 always say to us, I take great store in what 4 I reported the last time. So very often, 5 they fill out the forms by looking at what 6 they reported last reporting period -- 7 MS. FORSYTH: Well, this makes 8 their submitting burden so easy. They just 9 pull up the last form and -- 10 MR. FREEDMAN: Right. There are 11 some people who look at that as a 12 disadvantage -- yeah, I understand that 13 --------- it is a two-edged sword, without a 14 doubt. 15 MS. FORSYTH: A related benefit is 16 reports of their own data. You could 17 formulate reports that they can't necessarily 18 query. 19 MR. FREEDMAN: That's what Paul was 20 talking about. 21 MS. FORSYTH: Right, right. The 22 potential for reduced burden -- you already BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 299 1 mentioned this -- from follow-up calls 2 because of errors or inconsistencies, the 3 ability to link to a dissemination system 4 kind of seamlessly -- and then someone 5 already mentioned the -- again, our 6 relatively seamless opportunity for 7 electronic data to transfer, which I think is 8 probably really appealing to some of your 9 larger suppliers. I mean, they're probably 10 there and ready. 11 MR. FREEDMAN: Many large 12 respondents have said to us, what we'd really 13 like to do is push a button on the 5th of the 14 month and transmit the data. 15 MS. FORSYTH: Right, right. 16 MR. FREEDMAN: That functionality 17 will be built into the system. We'll have to 18 specify the format, of course, so they won't 19 have to use the IDC to key in numbers -- 20 MS. FORSYTH: Right, right. 21 MR. FREEDMAN: -- they'll just 22 simply submit the data from their database. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 300 1 MS. FORSYTH: Right, right. So 2 those are the main things I thought of, and 3 maybe other people have ideas. 4 MR. FREEDMAN: You actually raised 5 something about assisting respondents. Stan 6 Kaplan is going to talk in a few minutes 7 about the EIA 920, which is a combined heat 8 and power survey, and some respondents on 9 that survey are very problematic because of 10 the complexities of figuring out the number 11 they have to report to EIA; it's not 12 something that's generally recorded. Stan 13 said to me in a offline conversation one day, 14 wouldn't it be nice if we had an IDC where 15 people would put in some input parameters and 16 the IDC would compute the number for them 17 that they want to submit to EIA, and then 18 they could look at it and verify it and -- 19 again, there are dangers there as well, but 20 that's a potential use of an IDC technology 21 to improve response, so that's a point well 22 taken, although we don't have that in the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 301 1 functional requirements right now, it's -- 2 MS. FORSYTH: Right. And that's 3 survey-specific. 4 MR. FREEDMAN: Yeah. Oh, 5 absolutely. 6 MS. JENNINGS: I just have one 7 quick thing I'd like to say. Alethia 8 Jennings. I know we didn't want to get into 9 details, and we were trying to keep this 10 brief, but the one thing I do want to 11 mention, because of the fact that Tom Lorenz 12 and Bob Adler presented the MECS to you 13 today, the MECS is a very long survey. How 14 many pages is it physically? 15 MR. LORENZ: On average, between 16 the three forms, it's 60 pages. 17 MS. JENNINGS: It's 60 pages. 18 That's a very long survey, and a survey that 19 long -- it's a value to the respondent to 20 have one of the feature that we have built in 21 as a requirement here, which is a progress of 22 measure. And we've talked about the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 302 1 respondent being able to not only save the 2 data, but to see how far along they are in 3 the respondent process, the submission 4 process, before they save the data, and 5 that's very useful, more useful than on a 6 short form. But something like the MECS, 7 that's very valuable, and that saves the 8 respondent some of the burden as well. So I 9 think that's an important point we need to 10 bring out. 11 DR. HENGARTNER: Last comment. 12 MR. WATSON: A question. Stan, 13 could you give us some idea about sort of 14 what the process is now with the team going 15 forward? Will there be a rollout, are you 16 going to come around and sort of present 17 results to different groups? 18 MR. FREEDMAN: What I hope will 19 happen is that we will finish up the report 20 in a week or two, we will give it to our 21 champion, Nancy Kirkendall, she will think 22 it's wonderful, give it to Howard BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 303 1 Gruenspecht, who will give us a half a 2 million dollars to begin to build the system. 3 But in reality, what will happen is 4 we'll present the report to the Goal Four 5 Subcommittee, functional requirements, then I 6 think they will vet that through the 7 strategic planning process. They will 8 probably review it, and then if they want us 9 to, we will roll it out into the rest of the 10 organization, give them the opportunity to 11 comment on -- I think Nancy maybe can talk 12 about that a little bit more once we give the 13 report to the Goal Four Subcommittee for the 14 IDC. 15 MS. KIRKENDALL: Well, we have to 16 decide what to do with it. 17 MR. FREEDMAN: Okay. I think, 18 Bill, the intention is to get the buy-in from 19 the Strategic Planning Committee and then 20 look towards an application for 21 implementation to see if, in fact, we can 22 develop a modular system that will work in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 304 1 concert with what we have now. But I think 2 the key issue there is funding. 3 DR. HENGARTNER: Thank you very 4 much. We have a 10, 15-minute coffee break. 5 (Recess) 6 DR. HENGARTNER: Okay, ladies and 7 gentlemen, let's resume the last leg of 8 today's meeting. I'd like to recognize 9 Darius Singpurwalla, who will give us the -- 10 is he here? Yes, he is. Okay, just checking 11 -- who is going to give us a rundown of what 12 we've learned during the functional 13 requirement for EIA Internet data collection 14 systems. Darius. 15 MR. SINGPURWALLA: Sure. Thanks, 16 Nick. I want to give a big round of applause 17 to Stan, who came up and gave a great 18 presentation -- 19 (Applause) 20 MR. FREEDMAN: Thank you, but that 21 round of applause goes to the committee who 22 did all the work. I guess I want to give you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 305 1 a couple PowerPoint clients. 2 MR. SINGPURWALLA: There was the 3 rumor that he's sick for two weeks, but I 4 just heard he was actually at the beach, 5 so -- 6 (Laughter) 7 MR. FREEDMAN: I was sick at the 8 beach. 9 MR. SINGPURWALLA: Yeah. So that's 10 good. One thing that we've been talking 11 about the past day -- and you know we got a 12 lot more -- another presentation on it this 13 afternoon, which was just as good as the 14 first one -- is the whole Internet data 15 collection system that we're moving to. The 16 EIA's putting out a bunch of surveys, 17 obviously, and for lots of really good 18 reasons they want to move away from either 19 the paper-pen method of responders filling 20 out the surveys and then mailing them back, 21 or, as we heard about today, a technological 22 step up from that where they do it in Excel BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 306 1 and then send it back on either, like, a CD 2 or a diskette, and then the third evolution 3 that what they're going for here is they want 4 to do something on the Internet where the 5 responders will turn on their computers and 6 then log into the Internet data collection 7 system, a screen's going to pop up, and they 8 can fill out various information about 9 themselves or various surveys that they apply 10 to that they want to fill out. Then that 11 information is pushed back there to the EIA 12 or wherever they're going to be storing their 13 Oracle database, and that will be stored in 14 the database, and the database back here can 15 be used for reporting or whatever. So that's 16 kind of the direction they're taking it right 17 now, which is good. 18 It's a good subject for me to hear, 19 because this is kind of something that I do a 20 lot in my own job, and I think one of the 21 things that's really impressing me about the 22 way that this committee's doing it is that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 307 1 they're getting the requirements and 2 everything that they want to do and the 3 functionalities that they want to do, they're 4 laying those out up front. 5 The focus of this committee, as 6 Stan said in the beginning, is not to make 7 the screens look pretty or -- it didn't seem 8 to be like working with the actual database. 9 They want the functionalities in there as to 10 how the IDC is going to work, and that's a 11 really big thing that you need to do, and 12 it's often a step that I know from experience 13 gets skipped, and then at the end of the day, 14 there are going to be a lot of people 15 scrambling around, being, like, why didn't we 16 have this functionality, why didn't we have 17 that it's the project that I'm working on 18 currently, and I've already got about 10 19 voice mails of people demanding why something 20 hasn't been put into the data input part of 21 the database that I'm working on. And it's 22 not my fault -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 308 1 (Laughter) 2 MR. SINGPURWALLA: -- but it's more 3 related to the planning, that people didn't 4 see it up front, and so I think it's great, 5 and I think it's a super-important first step 6 that a lot of people ignore. I thought one 7 of the most useful handouts that we got that 8 I don't think came on the website with the 9 agenda on it is this data collection flow. 10 So, I mean, if we could get copies 11 distributed to the rest or put this up on the 12 thing, I think it's great because it really 13 gives -- and skimming through it really 14 quick, just the first page, you can really 15 see what the IDC is planning on doing and 16 what's its functionality is actually going to 17 be. We talked about it a little bit more in 18 depth, and I kind of mentioned some -- it's 19 in the beginning of the recap -- of a user 20 being able to log on and that one specific 21 user's going to have access to all the 22 surveys that he's going to do. He'll pop up BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 309 1 right up to a home page for his own self, and 2 then he can just start clicking and going 3 nuts and doing whatever he or she needs to do 4 to get their job done. And in a situation 5 where you're filling out a weekly survey -- 6 which I didn't know existed -- like, that's 7 going to be really helpful to do, like, you 8 don't have to type on your name and address 9 the whole time and even when we had some 10 paper-and-pencil surveys going around, they 11 kind of rectified that problem. It's good to 12 know that this IDC is going to have some of 13 the functionality to do that. 14 There were two presentations given 15 to us. One is the one that Stan went over, 16 the functional requirements for EIA's data 17 collection system, which is kind of a good 18 overall view of how it works. And then the 19 second one, which is kind of the one that I 20 burned through last night, goes in a lot more 21 detail about some of the functionality that 22 the IDC is going to have, and as someone who BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 310 1 works in this industry a lot, everything that 2 I would expect and want to be there is there: 3 The whole graphical user interface, the 4 single sign-on that we talked about. There's 5 going to be a common look and feel between 6 each of the different surveys across the 7 board, so if you're filling out one survey, 8 you kind of get used to that feel, and then 9 maybe your job responsibilities grow, and 10 they give you the other survey to fill, you 11 click on that, it's not going to look like 12 apples and oranges, I think -- two really 13 different things. That's what I'm going for 14 here. So I'd also recommend popping through 15 the second presentation as well to get a 16 better idea on what the specific 17 functionalities are. I think that was 18 useful, too. 19 Some of the other things that the 20 IDC is going to be able to do is, it's going 21 to be fully centralized, so I believe that 22 that means at the end of the day all those BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 311 1 surveys are going to be in one Oracle table. 2 Centralized core structure -- these are from 3 my notes yesterday, the single sign-on, and 4 all the other functionalities. 5 The last thing that I wanted to say 6 that I thought was really good is that 7 Barbara came up with a really nice list of 8 incentives that businesses can actually use 9 to be going ahead in responding to this. 10 It's, like, we -- this is my third meeting -- 11 we talk about surveys a lot, and incentives 12 always seem to come up. 13 And this is the first time that I 14 really thought about it on a business level, 15 instead of something like a household level, 16 because you're not going to send $25 to a 17 business; that's like a meal reimbursement 18 for them when they worked late on Saturday, 19 that's a cakewalk. But Barbara gave -- I 20 managed to write down five different 21 incentives, the whole technical assistance 22 thing. It's information. Businesses use BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 312 1 this information, and they're going to use 2 this information to make decisions, and those 3 decisions are either going to make them or 4 save them a lot of money, which is better 5 than the $25 gift certificate to Pizza Hut. 6 Flexibility on reporting is another 7 incentive, and then queries to their own 8 datasets if they've been able to input this 9 information, and now they can slice it and 10 dice it in any way they want. And, again, 11 reporting. So those are some of the 12 incentives that Barbara touched up on. 13 Again, I think it's so important to mention 14 that they're doing all the stuff up front as 15 opposed to doing it on the fly. And sending 16 some programmer in there, and he builds one 17 interface, and then you keep with the 18 back-and-forth, the back-and-forth. I think 19 when you build tools in general, any type of 20 software tool, it's the up-front planning 21 that's really going to save you time and 22 money down the road, and that's the biggest BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 313 1 thing that I got from this, is that they're 2 really on the ball with that. So it's 3 impressive. I'm done unless -- 4 DR. HENGARTNER: Thank you, thank 5 you. Does anyone want to add anything to 6 Darius' very extensive summary of our 7 discussions? Yes? 8 MR. HILL: An offhand comment. 9 Some of the surveys are weekly, some of them 10 are every four years, so that's a huge 11 difference -- 12 DR. HENGARTNER: Walter, Walter -- 13 MR. HILL: During the morning 14 session, there was a comment that those 15 surveys are done every four years, and there 16 was a comment -- it was almost an offhand 17 comment this afternoon that some of the 18 surveys are weekly, so it seems to me that's 19 a huge burden that's placed on the group to 20 try to come up with surveys over that length 21 of -- I mean, ------- for about 200 -- from 22 one week to 200 weeks that they're working BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 314 1 with. 2 MR. SINGPURWALLA: Yeah. 3 MR. HILL: It's a depressing task 4 that they have. 5 MR. SINGPURWALLA: Oh, yeah. 6 DR. HENGARTNER: Anybody else wants 7 to add to this discussion? Okay, thank you. 8 I'd like to move on to the feedback from the 9 other break-out session on improving the SAGE 10 petroleum refinery model, and I invite Tom 11 Rutherford to give us that summary. 12 MR. RUTHERFORD: Yes. John Staub 13 gave us a very nice introduction and overview 14 of the factors and the considerations that 15 are sort of informing their attempts to 16 improve upon the SAGE representation of 17 refinery sector in the SAGE model. This was 18 discussed this morning. SAGE is this 19 recursive model that characterizes a 20 16-region global model of demand and supply 21 factors influencing a wide range of energy 22 sectors, 42 energy products in total. So BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 315 1 it's a very large model for which the 2 existing refinery sector is extremely 3 simplistic and unacceptable for purposes of 4 -- it violates simple physical laws, they're 5 basically -- it just doesn't pass the laugh 6 test is what Alan Mann would say. 7 So what we're currently engaged in 8 is a process of improving upon this model, 9 which gives rise to considerations of 10 different ways in which this could be proved. 11 And the outline -- the paper talk this 12 morning was very interesting in terms of its 13 discussion of what the sort of the background 14 policy dimensions were and what the range of 15 factors are that influences the market for 16 refinery capacity. And particularly we 17 looked at what's the utilization of funneling 18 capacity has moved from 60 or 70 percent in 19 the mid '80s up to over 90 percent globally 20 in the last couple years. So there's been 21 big change in terms of structure the 22 industry, and there's this awareness that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 316 1 expansions have to take place. 2 Now, one of the things that's a 3 challenge for working in this area is that we 4 have this long history of chemical engineers 5 using precisely the tools we're using for 6 calculating economic equilibria have been 7 used for years to do refinery engineering. 8 So this process of -- the problem is that 9 different crude types hold different 10 products, and the way you combine them and 11 recombine them and what types of processes 12 you invest in will determine what comes out. 13 What comes out includes things like heavy 14 distillates, jet fuel, light gasoline, 15 heating oil -- there's a whole range of 16 products -- and how you combine the crude 17 oils will determine what comes out and the 18 process of designing a refinery involves 19 choosing what products you want to make from 20 what the crude grades are that come in. 21 In terms of thinking about the 22 model, it's a forward-looking -- or it's a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 317 1 model that's thinking about the future, and 2 we have a backdrop where -- and this was very 3 nicely laid out in the talk, how the 4 differences in what the range of products are 5 that are consumed differ dramatically across 6 countries. In particular, the U.S. tends to 7 be much more reliant on much lighter products 8 -- diesel fuel and gasoline -- whereas China, 9 other developing countries, have 10 traditionally been much heavier users of 11 heavy products. But those are converging, 12 and so we're going to see expanding demands 13 for light products, and as a consequence, 14 there'll be a need for large-scale 15 investments globally in refinery capacity. 16 And so those are the types of processes that 17 they're hoping to be able to extend SAGE to 18 deal with. Their approach is relying on a 19 long history of refining models that have 20 been built at DOE, and it was going to focus 21 on a fairly simplistic -- I would call it -- 22 refer to it as a reductionist model which BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 318 1 basically comes up with a series of simple 2 representations of the processes involved in 3 doing refining, and then having the model 4 choose how those processes are combined to 5 produce given products. So the investment 6 decision in each period will be based on 7 minimizing the cost of producing of these 8 various sets of outputs. 9 But what makes the whole process 10 complicated is you have a set of outputs, you 11 have a set of inputs, and the refining model 12 provides an intermediary between those 13 decisions, and so there's quite a bit of 14 complexity there. The challenge is to 15 introduce sufficient complexity to 16 characterize the decision process in how the 17 market operates, but not too much because you 18 don't want to make this unmanageable, and 19 that's where there's a bit of a -- I think 20 there was quite a bit of discussion about 21 what the appropriate level of complexity is 22 in the model. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 319 1 In economics, there's always this 2 tension between top-down versus bottom-up. 3 If you build a bottom-up model, then you 4 basically assert that you characterize all 5 these various -- this menu of choices that 6 the model has, and the model will then choose 7 among this large menu and make decisions, and 8 then at the end of the day, if somebody says, 9 well, why does your model produce this number 10 you can write down, you say, well, we 11 installed the catalytic cracker in China in 12 this year, and that's led to this result, 13 right, so you can point to very specific 14 details. 15 The problem with that approach is 16 if you try to keep it simple then you may not 17 be happy with what the model spits out. And 18 so then you have a challenge having to deal 19 with, how do we fix problems that arise in 20 terms of the model not being plausible. In 21 other words, in the questions for the 22 committee, one of the questions was, how do BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 320 1 we -- several preliminary runs of the model 2 demonstrated a problem in which it used too 3 much butane to produce gasoline. One 4 solution is to include octane constraints, 5 but the drawback is the model becomes more 6 complicated. So the problem with the 7 bottom-up model is invariably, you keep 8 returning to the model having to add 9 constraints to make it pass the current set 10 of requirements for plausibility, and so that 11 leads at the end of the day to a model which 12 is quite complicated. You have a lot of 13 things to explain about what you're doing. 14 And so there's a bit of a -- we discussed -- 15 I thought the -- again, the discussion, as 16 was this morning's discussion, was quite 17 lively, and I think quite informative, and we 18 didn't, I think, reach any agreement -- we 19 agreed that -- together, I think the one 20 thing we agreed on is that this is a darn 21 tough market to model, and there are lots of 22 challenges. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 321 1 There were several suggestions for 2 alternative approaches that might be simpler, 3 sort of more top-down type approaches, but 4 there was also discussions about what the 5 virtues were of the approach that was being 6 used. I thought Phillip laid out quite good 7 defense of the fact that they have a long 8 history of doing this, and they're quite 9 confident that they can make sense of this. 10 But I think the jury is still out; you can't 11 really judge the cake until you eat it. 12 So I think that this is an 13 interesting line of research, and I think 14 there's quite a bit of interest in the 15 project and how the model will be applied and 16 what we can learn from it. So I think that 17 that -- I don't know if I've touched on 18 everything. I'll leave it to the other 19 people who were there. 20 Oh, the final thing I forgot to 21 mention was that one of the interesting sort 22 of challenges in the current round of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 322 1 thinking, it depends on how you reflect on 2 factors that have led to the lack of 3 investment and refining capacity in the U.S. 4 in the last decade, and if -- are there are 5 different schools of thought about this, but 6 one of the schools of thought we'd like to 7 entertain is the idea that there might be 8 continued restrictions or reluctance to 9 invest in refining capacity in the U.S., 10 which would lead us to a very 11 different-looking world and the world market 12 maybe trade in refined products rather than 13 trade in crude oil. And then the question 14 is, where do the refined products get 15 produced, how do we characterize the factors 16 that give rise to where refineries tend to 17 get installed, and there was discussion about 18 transport costs, and there's a way of penny 19 switching. 20 One of the challenges is basically 21 how we model this whole process, and I'd say, 22 again, I think the committee's feeling was, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 323 1 this is a tough one to characterize, to 2 explain concretely. Clearly if the model 3 produces forecasts that all the refinery 4 capacity will be installed in Venezuela in 5 the next 20 years, that this will be a 6 difficult one to sell politically, but the 7 model may choose that because that may be the 8 cheapest way to do things. 9 So there are some challenges, I 10 think, in terms of producing a model which 11 will provide forecasts which can be published 12 in the AEO and will be referred to by large 13 groups of people. There are political 14 ramifications, which are not part of the 15 model, but I think that that's one of the 16 challenges with this type of formulation. 17 DR. HENGARTNER: Thank you very 18 much, Tom. Does anyone want to add to this 19 discussion? 20 DR. FEDER: I have a just a 21 question or a comment. Does the model 22 produce just one outcome or does ----- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 324 1 probability with the outcome that the 2 refineries are going to be in Venezuela or 3 something like that? 4 MR. RUTHERFORD: It's 5 deterministic, but I guess we'd let Phillip 6 answer it. 7 DR. TSENG: I think when we have 8 refinery representation in a SAGE model, the 9 choices will be -- we have 16 regions. And 10 in the next 25, 30 years, there will be 11 additional refining capacity equivalent to 12 about 45 million barrels a day processing 13 capability. And it has to be located 14 somewhere. And it could be Venezuela, or it 15 could be Middle East, it could be in Asia or 16 South America. So the issue here is if we 17 start the model with basic economic 18 principle, then it could be, just like Tom 19 mentioned, Venezuela. And politically, it's 20 impossible. And then the issue will be, if 21 we don't model it, how do we address the 22 tradeoff crude oil imports and petroleum BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 325 1 product imports. So we like to get really 2 comments from the committee as exactly what's 3 the best way to handle those kind of issues. 4 DR. FEDER: If I can draw on my 5 experience, it's a project I've been involved 6 with for about a year on modeling of 7 epidemics. What one is trying to do is get 8 both some scenario or scenarios, but also 9 some sensitivity of the results to 10 stochasticity in the model and assumptions 11 that might affect the outcome, and like any 12 statistical analysis involved, not just point 13 estimates, but also standard errors and some 14 kind of forecasting error. So it would be 15 good to allow the model to run multiple times 16 with likely different parameters and see how 17 sensitive the outcome is to that. So if, for 18 instance, a relative advantage of Venezuela 19 over, let's say, the Middle East in terms of 20 certain -- I'm sure I'm not saying it right 21 -- but some ways, how would the outcome 22 change, if it matters? If it does not BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 326 1 matter, then fine, but as a statistician, I'm 2 a little bit uncomfortable with having a 3 point estimate without a standard error 4 attached to it. So -- I mean, how much can 5 we trust the output if you don't know how 6 uncertain -- 7 DR. TSENG: I think we can run 8 scenarios, but another important issue is 9 still -- the bottom line is if one region -- 10 like in North America -- there's investment 11 in refining capacity, then at the end of the 12 day, more refined products will have to come 13 in to meet the end-use demand. And that's 14 kind of question that we believe will come to 15 EIA maybe a few years down the road. Because 16 the product import will become more 17 significant, and in general, the value of 18 refined product is also higher than crude 19 oil. 20 MR. RUTHERFORD: I should say that 21 the discussion about Venezuela as the 22 supplier, that was my interjection, just to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 327 1 make sharper what the issues were. This was 2 not something that was discussed in any 3 document that I've seen at DOE. 4 (Laughter) 5 DR. FEDER: But it's a good hook to 6 ask my question. My point is only, suppose 7 that we had some forecast on the future gas 8 consumption of the average family sedan 9 and/or SUV, and suppose that I put 39 in my 10 model, and that gave me a certain prediction. 11 I would like to run the model again with 37 12 instead of 39 and see does that really change 13 qualitatively, not quantitatively, the 14 output, things like that. So, sensitivity to 15 input parameters is always an issue. And 16 that gives you a way of assessing variability 17 in the outcome. 18 DR. TSENG: That's one way to run 19 the model once it's set up. It can be done 20 easily. We can do parametric analysis. 21 DR. FEDER: I mean, it's like 22 weather forecasting models. And in fact, if BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 328 1 you remember, was it Ophelia or one of those 2 hurricanes, they had four models -- and I 3 think we talked of it last time -- that three 4 models gave a projected course that went 5 eastward and missing even Cuba altogether, 6 and one projected that it would go to Maine, 7 which was all the way north. So that's the 8 same thing as doing multiple runs of the 9 model, you run slightly different models, and 10 you look at the output, and then you say, 11 yeah, I think this might be the future course 12 of pricing and needs and so on, but I need to 13 attach some probability to that outcome and 14 say that will happen with us 70 percent 15 probability, but 30 percent probability will 16 have another outcome. I may be asking for 17 too much, I don't know, just a statistician's 18 thoughts about model outcome. 19 MR. CLEVELAND: I think you're 20 right, and think the types of information 21 that they're trying to incorporate in the 22 model would allow you to -- well, not BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 329 1 necessarily assign error bars to data points, 2 run simulations that could answer those types 3 of questions, but I think one of the tangents 4 we discussed was that a lot of the decisions 5 that actually get made about where refineries 6 are plunked down are not going to be based on 7 the type of economic costs you're going to be 8 able to model. 9 There are different types of risk 10 that oil companies, national or private, are 11 going to face in different regions of the 12 world, which are very difficult to quantify 13 economically, in economic terms. So the 14 issue for EIA is going to be assume we can 15 get a handle on the uncertainty of economic 16 data -- how much do we want to get into the 17 business of talking about risk associated 18 with doing business at different parts of the 19 world, which is really going to be one of the 20 driving forces behind investments in these 21 big capital-intensive projects. 22 DR. HENGARTNER: Any more BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 330 1 questions, comments? Okay, I'd like to move 2 on to the next item on our agenda. I'd like 3 to welcome Stan Kaplan, who is going to talk 4 about making adjustments to survey data when 5 the collected data do not meet expectations. 6 MR. KAPLAN: This presentation is 7 entitled "Making Adjustments to Survey Data 8 When the Collected Data Do Not Meet 9 Expectations," but an alternative title could 10 have been "Good Intentions Gone Awry" because 11 what you're going to learn is that we had a 12 data problem, took steps to try and fix it -- 13 actually fairly sophisticated steps -- and 14 ended up with a new problem probably as big 15 as the original problem. And in fact, I'm 16 not going to speak a whole lot about the 17 adjustment process itself, which you'll see 18 is really fairly mechanical and I don't think 19 all that interesting. The interesting thing, 20 I think -- you all can judge -- is what led 21 to the problem and what steps we're 22 considering to definitively fix it. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 331 1 Now, just as a way of getting 2 started, break in with questions any time. 3 And also, there's a lot of detail in here, 4 and I'm actually going to skip over a lot of 5 these slides. A lot of the detail here is 6 for you to read in your leisure hours. I'm 7 happy to go through anything, but I'm not 8 going to try and bog us down in all the 9 minutiae which is discussed in the 10 presentation. 11 Where am I aiming at? Doesn't 12 matter. Okay. This is about electric power 13 data, and one of the most key and fundamental 14 pieces of data we collect is fuel 15 consumption, fuel consumed to make 16 electricity. That's a very basic piece of 17 information, how much gas, coal, oil, and so 18 on is being consumed each month, each year, 19 to produce electricity in the United States. 20 And we collect that data from basically two 21 types of respondents, two types of power 22 plants: One is traditional power plants, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 332 1 that is, it's a plant which was built for one 2 purpose, which is to make electricity, and it 3 consumes fuel and makes electricity and 4 that's what it does. But there's a second 5 kind of respondent, sometimes referred to as 6 co-generators, more properly I guess as 7 industrial-commercial combined heat and power 8 plants, CHP plants. And that's going to be 9 the focus of the discussion today. CHP 10 plants are most typically industrial 11 facilities, and their primary business is not 12 making electricity, their primary business is 13 making paper, making chemicals, making 14 refined products. There are commercial CHP 15 plants -- these are most typically hospitals 16 or other institutions which are producing 17 electricity as well as space heat. But 18 they're primarily industrials, and one of the 19 key things to keep in mind is that they 20 produce electricity as a secondary business; 21 their primary interest is consuming fuel to 22 make what's referred to sometimes as process BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 333 1 steam, other times as useful thermal output, 2 or UTO. And they use that steam for space 3 heating or as part of manufacturing process. 4 They use some of the steam to make 5 electricity, they consume a lot of the power 6 themselves typically, they may sell some of 7 it on the grid. But again, their major 8 business is making paper, making chemicals, 9 making widgets, whatever. The electricity is 10 a side light business for them. 11 Now, these combined heat and power 12 plants, one reason they exist is that they 13 can be much more efficient and therefore much 14 more economical than building a stand-alone 15 power plant and a stand alone boiler to make 16 steam for a process. So there are any number 17 of configurations for these plants, but 18 here's a typical one: Now, for a moment, 19 just ignore this piece, okay? So if you have 20 a ordinary stand-alone power plant, the fuel 21 goes in -- in this case, into a combustion 22 turbine -- electricity comes out, but there's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 334 1 also waste heat, these hot exhaust gases, 2 which, in a typical power plant, are wasted, 3 they're vented to the atmosphere or otherwise 4 just lost. 5 If you could put back into your 6 mental picture this piece up here, this is 7 what makes it a combined heat and power 8 plant. Rather than these hot exhaust gases 9 being wasted, the energy is captured -- in 10 this case in what's called a heat recovery 11 boiler -- so the otherwise lost heat is now 12 being used to make steam, and the steam goes 13 off to the process, which again, you're 14 making paper, making chemicals, whatever. So 15 more of the fuel coming in at this end is 16 ultimately recovered in something useful at 17 the back end, either electricity or steam to 18 accomplish the manufacturing process. So 19 these plants can be far more efficient than 20 building two separate plants. 21 However, it leads to a question, 22 and this is an important question for EIA, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 335 1 which is, how do we allocate the fuel? 2 You've got fuel coming in, but you have two 3 things coming out: Electricity and process 4 steam. So the respondents to our surveys 5 have to allocate some of the fuel as being 6 used to generate steam and heat and some of 7 the fuel being used to generate electricity. 8 And this is the issue which has proven to be 9 very difficult, one we've struggled with now, 10 really, for years. 11 Why is this difficult? Okay, there 12 are two reasons I would focus on. The first 13 is that when we ask people how do you 14 allocate fuel to electricity versus fuel to 15 process steam, we're asking them for a data 16 element which normally they don't collect for 17 their own purposes. 18 When we ask someone how much total 19 fuel do you consume or how much electricity 20 do you produce, we're asking for a piece of 21 information which is of interest to them 22 because it's vital to their own cost BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 336 1 accounting, to knowing how much revenue 2 they're going to bring in. 3 When we ask them how are you 4 allocating fuel to electricity versus fuel to 5 UTO, it's a question which, for the most 6 part, the respondents simply aren't 7 interested in. They don't compute it for 8 their own purposes. So we're asking them to 9 do something which is outside their normal 10 course of business. 11 Then to further compound the 12 problem, we're asking them to compute 13 something for which there is no standard 14 methodology. There's no theoretically 15 correct single way to make the computation, 16 and beyond that, there's no single way which 17 it is typically done in the business. 18 All right, so what has EIA done? 19 For a number of years, we took the allocate 20 fuel -- compute the fuel allocation by first 21 determining useful thermal output. Without 22 going through all the calculation BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 337 1 permutations, basically, we would figure out 2 this piece -- fuel use to make steam -- we do 3 collect this piece, the total fuel use -- 4 they certainly are interested in that and 5 they give us that information -- then you do 6 a subtraction; what's left over is fuel use 7 generated electricity. And we went to that 8 approach only after careful consultations 9 with industry where we were assured that this 10 information was available. Only it wasn't. 11 And so we ended up getting lots of numbers 12 coming in which just didn't make sense. They 13 produced plan efficiencies or planned heat 14 rates which were just way out of bounds. 15 So that just was not working; it 16 was taking a tremendous amount of effort on 17 the part of our staff to try to clean up the 18 data through innumerable phone calls and 19 e-mails. By way of context, combined heat 20 and power plants account for -- I think it's 21 about five percent of total U.S. generation, 22 but they have been consuming, and still are BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 338 1 consuming, about a third of all the resources 2 we put in to this area of data collection. 3 All right. So that didn't work so well. 4 Now, the other alternative -- 5 right, eight other alternatives -- again, we 6 collect this -- this is no problem, total 7 fuel -- get from the respondents fuel use to 8 make electricity. In other words, get it 9 directly, and then by subtraction, we can get 10 fuel used to generate heat and steam, which 11 is of interest to people. So we went through 12 an even more elaborate process in which we 13 had a lot of internal discussions, had an 14 outside consultant assisting us, went through 15 about 40 altogether interviews with actual 16 power plants in which we discussed concepts, 17 went through test forms, had them fill out 18 test forms. I think there's also discussions 19 even with this committee about this at least 20 twice. 21 And so we launched our new form, 22 it's called the EIA-920. And here I am today BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 339 1 because the change in methodology produced 2 both radically different and radically 3 unexpected results. Basically, it was the 4 wrong answer. And for reasons I'll discuss 5 -- not just because it doesn't make us feel 6 good, but its numbers which in many cases 7 just don't make sense. 8 Now, before I go further into this 9 as to why it's a problem, we need to at least 10 briefly just go over just a few measures of 11 combined heat and power plant efficiency. 12 There are three things to talk about here. 13 One is the heat rate of a plant, which is the 14 BTUs of fuel input required to produce one 15 kilowatt hour of electricity. The more 16 efficient a plant is, the fewer BTUs of input 17 required to product one output unit of kWh 18 output, all right? 19 The second one is the 20 power-to-steam ratio, which in essence is the 21 ratio of the heat value of the electricity a 22 plant is producing to the heat value of the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 340 1 steam it is producing for manufacturing 2 process, all right? So the more fuel being 3 devoted to power production, the higher the 4 ratio will be, okay? 5 And then finally, we have thermal 6 efficiency, which is the overall efficiency 7 of the plant. And what you have here is the 8 ratio of the sum of the heat content of the 9 power and the steam produced to total fuel 10 input. So, for example, if the thermal 11 efficiency of a plant is 50 percent, that 12 means that half of all the fuel input is 13 being put to a productive use, whether it's 14 making electricity or making steam, and half 15 of it is wasted. By contrast, if you look at 16 a typical stand-alone power plant, the 17 thermal efficiency would be, at best, about 18 33 to 35, 36 percent. So a combined heat and 19 power plant should be way above that. 20 All right, now, the approach we 21 took -- go back for a second -- if you 22 recall, was we went to this approach: Ask BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 341 1 them how much fuel's being used to generate 2 electricity, subtract it from total fuel, you 3 get fuel used to generate steam. Well, what 4 we found was -- in retrospect, this maybe is 5 not so surprising -- is that when we asked 6 people to focus on how much fuel they are 7 using to make electricity, that completely 8 changed the orientation of their thinking, 9 and they began to focus -- in some cases 10 exclusively -- on what do I need to compute 11 fuel for electricity -- I'm not even going to 12 worry about really what it takes to compute 13 fuel for useful thermal output. As a result 14 of this change in focus and the way they were 15 looking at the problem, the responses we got 16 in 2004, the first year we did this, showed 17 vastly more fuel being used to make 18 electricity than in the past. 19 Basically, we asked people how much 20 fuel are you using to make electricity, and 21 they answered us. But they didn't answer us 22 in the expected way, they didn't go through BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 342 1 some calculation to figure out, well, what 2 should it really be as versus fuel for steam, 3 they just took kind of a stand-alone approach 4 and came back to us with how much fuel is 5 being used to make electricity. And you can 6 see this in the measures of efficiency. In 7 the 2003 data collection, the average heat 8 rate for a combined heat and power plant was 9 about 9,800, 9,900 BTUs per kWh. In 2004, it 10 ballooned to almost 14,000, and that was 11 because the respondents, when we asked them 12 how much fuel they used to make electricity, 13 well, they answered us by focusing on that 14 question. Here is the power-to-steam ratio, 15 which goes from .33 to .41. What this, 16 again, reflects is more fuel being used to 17 make electricity than being used to make 18 steam. And then finally, this is the average 19 thermal efficiency of the plant, and it 20 plummets from 60 percent to 49 percent. And 21 again, actually, it reflects two things: One 22 is plants allocating much more of their fuel BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 343 1 to electricity production, which is 2 relatively inefficient, than to making steam, 3 which is more efficient, so you see the 4 decline there. What we also had -- that's 5 why I'm saying these numbers in some cases 6 just don't make sense. We had quite a few 7 respondents who allocated zero fuel to making 8 steam, to making useful thermal output. 9 MR. CLEVELAND: Stan? Can I just 10 interrupt? 11 MR. KAPLAN: Yes. Sure. 12 MR. CLEVELAND: I'm sorry to 13 interrupt. How were they making that 14 assessment? They know the fuel is going in a 15 combustion turbine, and what comes out the 16 two ways is a function of the efficiency of 17 the combustion turbine. So when you ask them 18 how much fuel is being used to make 19 electricity, what are they basing that on? 20 MR. KAPLAN: Well, you see, there 21 are different ways you could calculate it. 22 You could calculate a stand-alone BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 344 1 hypothetical heat rate for the combustion 2 turbine okay? 3 MR. CLEVELAND: Right. 4 MR. KAPLAN: You can measure the 5 temperature of the steam at different points 6 in this process. You can measure the 7 temperature of the gases coming out. 8 Actually, I've got to admit, some of this 9 goes beyond certainly my technical expertise, 10 but we did enough work to verify that there 11 are doable ways of measuring how this fuel's 12 being allocated. There's also a more 13 hypothetical way, which I skipped over here, 14 but it's in the presentation, which is called 15 "Thermal Displacement," in which basically, 16 you calculate what your fuel consumption 17 would look like for a hypothetical 18 stand-alone point and use that as the basis 19 for making your calculations. So there's a 20 variety of ways to do it. 21 What we found, though, was that 22 people -- again, after all these interviews BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 345 1 and such -- people, although they told us 2 they could do it and would do it, just 3 weren't doing it, and what I'm about to get 4 to, in fact, is a most extreme example, which 5 is the plants which allocated zero fuel -- 6 did I answer your question, by the way -- 7 MR. CLEVELAND: Yeah, mm-hmm. 8 MR. KAPLAN: -- to making steam. 9 You come back to our case here, we 10 had 177 respondents out of 610 tell us that 11 although they were producing steam, they 12 didn't use any fuel to make it. And the 13 reason they said that -- and this is a 14 typical answer -- is that, well, the fuel 15 goes in to my combustion turbine, and I'm 16 going to run this thing no matter what 17 because I need electricity to make my plant 18 go. So all the hot exhaust gases that come 19 out of it, it's a freebie. It's going to be 20 there anyway. So therefore, I consume zero 21 fuel to make the hot gases which then go here 22 to the heat recovery boiler to make the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 346 1 steam. So therefore I've consumed no fuel to 2 make the steam which I do produce, all right? 3 So this didn't work out real well. 4 And in many cases, we ended up with reported 5 efficiencies for these plants which were 6 comparable to or even worse than if they had 7 built two stand-alone power plants with none 8 of the efficiencies of capturing the waste 9 heat. Those numbers just don't make sense. 10 I mean, again, this is just not a question of 11 this not making us feel good or not meeting 12 expectations from a reasonable -- call it 13 theoretical -- standpoint, it just doesn't 14 add up. And we have the plants saying they 15 produce -- yeah, sure. 16 DR. NEERCHAL: How long when they 17 designed the plant should they have had some 18 calculation of the ratio of the fuel what 19 this part was and that part should -- 20 MR. KAPLAN: You would think that, 21 or you might think that. I certainly thought 22 of going into this, for example, that at some BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 347 1 of these plants, they'd be very carefully 2 measuring their steam flows. Some of these 3 plants, like paper mills, have very 4 complicated flows of steam; it goes all over 5 the plant to different processes, the 6 temperatures change, the pressures change. I 7 know that I had thought they were all 8 measuring this carefully for their cost 9 accounting. Their answer is, they don't. 10 They don't. 11 They're interested in the total 12 amount of fuel going in, what it's costing 13 them, they're interested in how much 14 electricity they produce, what the total cost 15 of their final products is, but as far as 16 doing detailed measuring and detailed 17 metering, surely there are exceptions, but 18 for the most part, they just don't do it. 19 Yeah? 20 MR. RUTHERFORD: It's basically 21 impossible to evaluate the state of that of 22 that efficiency measure. If you don't know BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 348 1 what the heat content of the steam that's 2 being used in the process is, then -- 3 MR. KAPLAN: It's very hard. It's 4 very hard. 5 MR. RUTHERFORD: You basically 6 can't do it. 7 MR. KAPLAN: And that's why one 8 recommended method is this thermal 9 displacement method which you do a series of 10 hypothetical calculations to determine what 11 the fuel allocation should be. And I'll get 12 back to that. All right. 13 So because of these problems, we 14 had to correct the data in the aggregate, 15 which was just wasn't acceptable, but we 16 didn't have the time or the resources to go 17 back and call hundreds of respondents and 18 help them figure out the right answer. So we 19 needed to go through a process to adjust the 20 data. 21 I'm going to go through this fairly 22 quickly because -- you all can judge, I don't BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 349 1 think it's all that interesting; it was a 2 fairly mechanical process. Our object was to 3 bring the aggregate measures of efficiency 4 for this sector, the combined heat and power 5 sector, into a reasonable range compared to 6 historical trends. We wanted to change as 7 little reported data as possible. 8 The total fuel reported was never 9 adjusted -- we felt that was a good number -- 10 just the allocation of fuel. Out of 610 11 reporting plants, we ended up adjusting 184. 12 And the basic process was that each CHP plant 13 was evaluated against a maximum acceptable 14 value for the power-to-steam ratio based on 15 their 2003 data. If a plant exceeded the 16 maximum, the allocation of fuel for the plant 17 was adjusted either to match the 2003 18 power-to-steam ratio or, if that number, also 19 in retrospect, didn't look so good, to match 20 a 2003 industry-wide average for the plant 21 type. And then the process had various 22 checks to make sure we didn't end up BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 350 1 ultimately with numbers which, in some other 2 respect, didn't make sense. Yes? 3 MS. KHANNA: But both of those 4 numbers ---- 2003 number, for example, is 5 based on the earlier survey which asked them 6 to look at the other side of the equation -- 7 MR. KAPLAN: That's right. 8 MS. KHANNA: -- so that's also a 9 suspect number to start off with. 10 MR. KAPLAN: Yes, it is. It is. 11 We think it's a much better number, it makes 12 a whole lot more sense, but there was no 13 question that there were problems with that 14 earlier data. Now, for that earlier data, 15 though, I will say this: We did go through 16 the process of making literally hundreds of 17 phone calls. We had I don't know how many 18 people over a period of months making 19 hundreds of phone calls to speak with the 20 respondents directly, try to understand why 21 the numbers looked, in many cases, odd, and 22 get them more in line. So while that data is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 351 1 certainly not perfect, we went through a very 2 long and arduous process to make it at least 3 reasonable. And so in trying to bring the 4 2004 data in line with the 2003 data, I think 5 we were taking a reasonable approach. 6 There's a lot more detail here on 7 the adjustment process ratio I'd be happy to 8 discuss later. It goes on at some length. 9 We ended up adjusting, I guess, about 180 10 plants -- 184 plants, the ------ again being 11 basically to bring the plants which seem most 12 out of line into line with their own 13 historical patterns, and so we ended up with 14 aggregate data which not surprisingly matches 15 the historical average. 16 (Laughter) 17 SPEAKER: Duh. 18 MR. KAPLAN: Yeah. Pretty amazing, 19 huh? So the heat rate for 2003 in the 20 aggregate had been about 9,900, it had 21 ballooned in the unadjusted 2004 data to 22 almost 14,000. In the adjusted data, it's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 352 1 back down towards around 10,000. 2 DR. NEERCHAL: Could I ask a 3 question? 4 MR. KAPLAN: Yeah. 5 DR. NEERCHAL: --- mention it -- 6 maybe I missed it -- but how did you choose 7 the 184 plants to adjust? 8 MR. KAPLAN: Well, what we did was, 9 first we wanted to adjust the reported data 10 as little as we could, all right? Secondly, 11 there are plants -- some of these plants are 12 very complicated configurations, and from our 13 information, we could identify frequently 14 which of those plants had the most 15 complicated configurations. We want to stay 16 away from those because the more complicated 17 the configuration, the harder to tell if the 18 data was really bad or if there was just 19 something going on, given that it was a 20 complicated plant. So that was an important 21 criteria for narrowing it down. 22 We ultimately ended up focusing on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 353 1 plants which are called steam-electric 2 plants, very straightforward boiler plants, 3 and also certain kind of combined cycle 4 plants, which have what are called heat 5 recovery steam generators. So that narrowed 6 it down to a fairly small subset of plants 7 which were potential candidates for 8 adjustment. 9 Then of these potential candidates, 10 we checked to see which ones actually had 11 out-of-range indicators and that narrowed it 12 down to the 184, which I mentioned, which 13 were ultimately adjusted. Does that help? 14 DR. NEERCHAL: Yes. Are there 15 plants that you'd say are doing a pretty good 16 job of doing the allocation? Do you know? 17 MR. KAPLAN: Yeah, some plants 18 certainly do -- 19 DR. NEERCHAL: How many do a good 20 job out of the 610? 21 MR. KAPLAN: That's a good 22 question. Fewer than half. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 354 1 MR. CLEVELAND: Well, the point is 2 you don't know what good is because it's a 3 totally arbitrary method. 4 DR. NEERCHAL: No, based on their 5 explaining their method clearly in a 6 particular ------. 7 MR. CLEVELAND: This is exactly 8 what we went through in the 1960s with 9 regulating gas prices. The precursor of FERC 10 had to regulate gas prices based on cost of 11 production; the problem is you have oil that 12 you drill a well -- which costs so much -- up 13 out of the ground comes oil and gas, so how 14 much of that cost do you assign to oil and 15 gas, and they had hearings for a gazillion 16 hours and it's totally arbitrary. 17 MR. KAPLAN: Yes, it's definitely 18 analogous. But one thing we do know, though, 19 is that while it may be largely arbitrary, 20 what we do know is that it's got to be, when 21 you look at it, more efficient than if you 22 had built two separate plants -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 355 1 MR. CLEVELAND: Well, but that's 2 not the question. Everyone knows it's more 3 efficient, the question you started with was 4 the assumption that you have a perceived need 5 to allocate a certain amount of energy to a 6 certain amount of electricity so it fits in 7 with the rest of your stuff. So -- 8 MR. KAPLAN: All right, so here are 9 the adjusted heat rates, here's the adjusted 10 power-to-steam ratios that came back in line 11 from '03 to -- reported '04 to adjusted '04, 12 and then finally the thermal efficiency of 13 the overall plants. All right, now this is 14 all well and good. We are currently 15 collecting 2005 data, which is going to have 16 the same problems as the 2004 data and 2006 17 data. And we're going to need to adjust that 18 data. We clearly need some definitive 19 solution for this, or else we're going to be 20 in the data adjustment business for CHP 21 plants for the indefinite future, which all 22 good things to the side, is from a resource BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 356 1 standpoint really not acceptable. It just 2 occupies too much time, too much in the way 3 of staff resources, to be constantly trying 4 to solve these problems. 5 Now, one thing we're currently 6 doing in conjunction with our SMG team is 7 we're testing with respondents additional 8 instructions intended to help them complete 9 the survey. However we'd be happy to be 10 surprised, but it's highly unlikely that 11 simply sending out new instructions is going 12 to solve the problem. Yeah? 13 MR. FREEDMAN: The fact of the 14 matter is that the companies we called, it is 15 just as problematic with these additional 16 instructions. Some of them don't understand 17 the concept. We went to respondents who were 18 reporting free steam, and they'd go through 19 and they'd say, oh, yeah, this is a nice 20 instruction booklet, it makes a lot of sense, 21 and we'd ask why they were reporting free 22 steam, and they said, oh, well, you didn't BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 357 1 want us to allocate it, so I -- it was a good 2 idea, it was well intentioned, in some 3 instances a light bulb went off where the 4 respondent said, oh, okay, I see what you 5 want to do, but it was not the silver bullet 6 that we had hoped it would be. 7 MR. KAPLAN: Right. 8 MR. FREEDMAN: You haven't seen 9 those results yet, but we have. 10 MR. KAPLAN: Okay, okay. All 11 right. 12 MR. FREEDMAN: That means we don't 13 have to write a report? 14 MR. KAPLAN: Yeah, sure. The basic 15 problem -- and again, it's worth just briefly 16 recapping -- why we don't expect or have now 17 seen that a simple fix like better 18 instructions is not going to work is because 19 we have two fundamental issues. The first is 20 that we're trying to collect something which 21 is basically not of interest to many or most 22 of these respondents and not a piece of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 358 1 information that they compute or otherwise 2 gather as part of their normal course of 3 business, which certainly makes it unusual, 4 if not maybe even unique -- I haven't thought 5 through all the data we collect -- but it's 6 certainly unusual compared to the vast 7 majority of the data we collect on the 8 electric side, that we're asking for 9 something which is normally not of interest 10 to them, okay? 11 And the second problem is that, 12 again, there is no hypothetically correct or 13 otherwise standardized way to do the 14 allocation. So what do we do -- and we're 15 just in the earliest stages of discussing how 16 to go about, again, finding our definitive 17 solution. But the basic direction we're 18 heading is that we're going to have to be -- 19 meaning EIA -- much more -- you can call it 20 directive or have to narrow the options for 21 respondents far more than in the past. This 22 can mean specifying the estimation method BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 359 1 that has to be used -- just telling them you 2 have to use this approach to calculate it, 3 and also specifying typical or acceptable 4 values for key input into the method, like 5 the efficiency of a boiler if the respondent 6 does not have reasonable data. 7 Now, as far as implementing this, 8 there are sort of different levels of 9 control. One is, we can build into our -- 10 most of our data is collected online using 11 what's called the Internet data collection 12 system. One thing we can do is build into 13 this what's called the IDC system a combined 14 heat and power calculator that the respondent 15 would have right their on his or her screen 16 that would say enter these numbers here, push 17 the button, out pops the allocation. 18 Another approach is for EIA to make 19 all the estimates itself. That is, given the 20 information we do know from other surveys 21 about plant configurations and historical 22 data we've gathered, we'll just collect the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 360 1 data we know that they have -- like total 2 fuel -- take typical values and just do the 3 estimates ourselves. That's another possible 4 approach. Again, we're in -- it's the 5 earliest stages of talking about this, but 6 the basic thrust of our internal discussions 7 is that we have to be more directive, we have 8 to narrow the options for how this can be 9 done, and we have to be ready to really just 10 provide key inputs rather than letting 11 respondents go off and put in numbers which 12 just produce results which are just not 13 defendable. 14 So considerations as far as what 15 approach to implement -- is more research 16 needed into the typical values or typical 17 ranges for key parameters such as what the 18 power-to-steam ratio should be for a certain 19 kind of plants or what the boiler efficiency 20 should be? Will we need to collect 21 additional information on plant 22 configurations? That would be a difficult BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 361 1 thing to do, but certainly worth considering. 2 Very important is what is the development 3 time and cost for implementing different 4 approaches? Right now, we're heading towards 5 what we call our Electricity 2008 process, 6 which is based on our three-year cycle for 7 reauthorizing our forms with OMB. And our 8 intent is that if we're going to have a new 9 process, to have it ready to go by certainly 10 no later than January 2008, and hopefully 11 this time have the definitive process. 12 After I had finished the 13 presentation and e-mailed it off, I realized 14 I should have added one more slide. There's 15 a famous cartoon ran in The New Yorker years 16 ago by a man named Arno, and it shows an 17 airplane that's just crashed, and there's a 18 guy parachuting out and a bunch of generals 19 going, oh, my God. In the foreground is this 20 professor rubbing his hands together with a 21 big smile on his face saying back to the 22 drawing board -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 362 1 (Laughter) 2 MR. KAPLAN: And we are somewhat in 3 that mode. Any suggestions you all have 4 would be very welcome. We really do need to 5 kind of put this one to bed; it's been a 6 persistent problem for a long time now. Any 7 questions, comments, suggestions? Any 8 solutions we can implement tomorrow would all 9 be welcome. Yes? 10 MR. CLEVELAND: One, I think you're 11 right, I think you need to impose your own 12 system of doing this because there are way 13 too many ways in which engineers and the 14 financial folks at different institutions 15 would do this allocation. 16 What about -- this could be 17 completely off the wall -- you're focused on 18 the engineering side of things to do this 19 allocation, but these are a lot of economic 20 decisions that these people are making. And 21 so why not allocate, think about, or 22 investigate allocating the fuel import based BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 363 1 on the relative values of the outputs of the 2 steam and the electricity -- 3 MR. KAPLAN: Okay. 4 MR. CLEVELAND: -- because one of 5 the problems with the engineering approach is 6 that you're taking a heat unit of electricity 7 and a heat unit of steam and assuming they're 8 equal, but from an economic perspective, 9 they're certainly not. The marginal product 10 of a heat unit of electricity is much higher; 11 that's why you're willing to pay a lot more 12 money for a heat unit of electricity. So 13 basing things on heat units when you're 14 looking at steam and electricity is 15 economically bogus. 16 So you might think about looking at 17 -- and you can -- even though they may not be 18 this -- electricity is easier value, the 19 steam is harder, but clearly there's an 20 avoided cost here -- there's an opportunity 21 cost to this waste heat, and if they didn't 22 capture the waste heat to generate the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 364 1 process steam, they would have to generate 2 the process steam in a different way, and 3 that would have a cost that you could assess. 4 So you might look at the allocation based on 5 relative value of the product rather than 6 some kind of thermodynamic efficiency. 7 DR. NEERCHAL: Except that in one 8 -- and I think this is a very nice start, but 9 I don't know that they are consciously making 10 that allocation or letting their system do 11 that allocation for them. That part is not 12 clear to me, that sometimes they have 13 inherited a system that happens to be doing 14 the allocation for them or are trying to 15 allocate. 16 MR. CLEVELAND: I think they think 17 about the economics very closely and are 18 making those decisions -- 19 MR. KAPLAN: Well, they're 20 certainly interested in the total value of 21 what's coming out, the electricity and the 22 manufacturing product which the steam is used BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 365 1 to produce. But what continues to be 2 surprising is that they don't meter 3 internally more carefully given the 4 importance of the inputs to what you get at 5 the other end. They just don't do it. 6 DR. HENGARTNER: Well, isn't this a 7 case of beauty is in the eye of the beholder, 8 that if you ask the question why are you 9 running that turbine, one will say it's 10 electricity and they'll be holding it free 11 steam, something free, that's wonderful. 12 Someone else may say he needs the steam to 13 run something else, and by the way, he also 14 gets electricity. And so, depending on what 15 they value, they will give different answers, 16 at least -- this is very naïve ------. 17 MR. KAPLAN: It's not naïve from 18 the standpoint of the problem we have at all 19 because -- I gave as an example the case 20 where they're allocating zero fuel to useful 21 thermal output. We do have other people 22 whose view is I'm producing steam to make my BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 366 1 widgets, and I'm going to make my widgets no 2 matter what. And consequently, they end up 3 allocating very, very -- never zero, but -- 4 well, rarely it's zero -- but very, very 5 little fuel to the waste steam which is 6 captured in this case to make the 7 electricity. So you end up with heat rates 8 which, in some -- I don't know if it's 9 thermodynamics, but the heat value of a unit 10 of electricity kilowatt hour is 3,412 BTUs 11 per kilowatt hour, and we've had people come 12 in with heat rates below that, which is 13 physically impossible. 14 MR. CLEVELAND: But why would you 15 even entertain a system that would produce 16 that result? I don't understand what -- 17 MR. KAPLAN: Well, you see, from 18 their standpoint -- it's the reverse of the 19 example I gave, actually. Their standpoint 20 is, I'm a paper mill. I'm going to make 21 paper, and from my standpoint, all this fuel 22 is being used to make paper. I make BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 367 1 electricity, too, so I allocate a little bit 2 to making the electricity, but the number's 3 so small because they're focused on the paper 4 part of it that you end up with the results 5 which just don't add up. 6 MR. CLEVELAND: You can just stop 7 listening to them and do something on your 8 own that makes more sense. 9 DR. HENGARTNER: But that does 10 exactly what you were saying, is that it's 11 the economics, and their view of the 12 economics is that I need to produce steel, or 13 I need to produce electricity. And 14 therefore, the question you should ask first, 15 what is your primary line of business, what's 16 your primary focus. And depending what it 17 is, it's going to be one calculation or the 18 other one that's going to be relevant in 19 their eyes, and maybe in yours as well, 20 because the one who gets free steam or free 21 electricity, what there is, that's how it's 22 perceived, and maybe that's how we should BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 368 1 calculate it. I don't know. 2 DR. BURTON: Nick, I don't think so 3 because I think the goal in this is to figure 4 out how much of a very real fuel you're using 5 to produce a very real amount of electricity. 6 So to make it arbitrary I don't think is -- 7 you're not going to be adding together what 8 you want to be adding together. 9 MR. KAPLAN: Yes. 10 MR. RUTHERFORD: I'll just say 11 there are all these deep issues about how 12 these guys operate their business, so I'm not 13 going to dwell on them. I'm just focusing on 14 how you fix the data. So, yeah -- the only 15 comment I'd suggest -- and there undoubtedly 16 are statisticians here that'd be better able 17 to give you specific suggestions but -- it 18 sounds similar to problems we've run into 19 with household surveys where we don't have 20 information about where people earn their 21 money, and we have to try to figure out 22 things, and we -- there's this technique BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 369 1 called small sample estimation, which, 2 basically, when you have numbers that are 3 missing for some set of the data, you can use 4 a statistical model to impute rather than 5 simply projecting them all onto the extreme 6 value. I'm just concerned -- and again, this 7 is a -- in other words, if you had 8 information about what their line of business 9 was, what the split -- how much electricity 10 made -- you could basically run a regression 11 and produce an estimation from your full 12 sample on what the factors were that gave 13 rise to what you were trying to estimate. 14 Then for those outliers that you 15 don't trust their response, you could fill in 16 using the statistical model rather than 17 simply just saying we'll just give them the 18 minimum value. It's a very mechanical 19 suggestion, but I'm just -- 20 DR. NEERCHAL: That is what I had 21 in mind when I asked him. Are there at least 22 some reliable data. -------- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 370 1 MR. RUTHERFORD: The thing is, to 2 do that you have to have more information, 3 you need to know something about what field 4 -- what they're producing and what -- you're 5 going to be able to construct a model, a 6 statistical model. 7 DR. HENGARTNER: Neha? 8 MS. KHANNA: So I have a naïve 9 question, which is, it seems like there's a 10 lot of institutional effort being spent on 11 collecting one piece of information. So the 12 question is, why do you do this? I mean, how 13 important is this piece of information to 14 your overall exercise? Would it matter if 15 you didn't have this information at all? 16 MR. KAPLAN: That's a very good 17 question. Compared to the total resources 18 which go into the surveys I collect, 19 generation of fuel consumption, we're putting 20 enormously disproportionate amount of those 21 resources into collecting from those combined 22 heat and power plants, trying to get the data BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 371 1 straight. We do have to collect the data, 2 I'll say for at least two reasons. One is 3 that people do want to have complete data. 4 Well, it may just be five percent of looking 5 at -- in terms of generation and also small 6 percentages of fuel, it's still part of the 7 puzzle. And we know that customers do want 8 to have that piece. 9 The other part, though, which is 10 kind of a subset of what I just mentioned, 11 has to do with natural gas. There is a fair 12 amount of natural gas which is consumed in 13 various kinds of combined heat and power 14 plants. And we know from customer comments 15 -- again, speaking from personal experience 16 -- that in the past, when the EIA data did 17 not make it easy to figure out how much gas 18 was being used by industrial consumers to 19 make steam versus making electricity, for 20 people interested in studying the natural gas 21 business, that was a real problem. And -- 22 sorry. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 372 1 MR. SITZER: I was going to ask -- 2 had a comment. 3 MR. KAPLAN: All right, and so that 4 is a particularly important piece which we 5 need to fill in. 6 MR. SITZER: I'm Scott Sitzer, I'm 7 at the Office of Standards -- 8 SPEAKER: Scott, would you step up 9 to a microphone, please, or have a seat at 10 the microphone? Thank you. 11 MR. SITZER: I want to answer your 12 question also. I think your question maybe 13 is not so much why do we collect the data, 14 but why do we spend so much time trying to 15 allocate it between electricity and UTO, and 16 I think there are two groups of users that 17 are interested in it. 18 One is our internal modelers, and 19 even though none of them is here right now, 20 they're really the ones who noticed when we 21 first collected the information for 2005 or 22 2004 how much the heat rate had increased for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 373 1 CHP, so they brought it to our attention and 2 they thought that we should possibly do 3 something about it. 4 And the other is people who are 5 interested in promoting such things as CHP 6 because they believe that it's a more 7 efficient way of producing electricity and 8 also producing the steam that goes into other 9 industrial processes. And they have made an 10 argument about how much more efficient it is 11 and how much fewer BTUs of fuel you need to 12 produce electricity. So we have attempted to 13 fill that data gap. 14 But I think your question's very 15 good: Why should we do this. If we collect 16 the information on consumption and we collect 17 the information on electricity, maybe we need 18 to leave it to others to figure out how to do 19 that allocation. 20 MR. CLEVELAND: I agree. This is 21 not the place to make the argument for 22 combined heat and power, I don't think. You BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 374 1 know what, another easy way to do it is to 2 say if that steam turbine that they're 3 generating electricity with is -- let's say 4 it has a standard efficiency and you're 5 burning 10,000 BTUs of fuel to get one 6 kilowatt hour of electricity, and the 7 question is, how much fuel was used to 8 generate the kilowatt hour of electricity, 9 the literal answer is 10,000 BTUs of fuel. 10 MR. KAPLAN: And that is a 11 perfectly legitimate approach, and that is 12 the kind of direction which we probably need 13 to move, something which is straightforward 14 -- I was mentioning before, and you'll see it 15 in here -- this thermal displacement approach 16 which is, in so many words, is the kind of 17 thing you just said. That is, make a 18 hypothetical calculation of, say, what the 19 stand-alone efficiency should be and then use 20 that as your basis. 21 DR. HENGARTNER: Walter? 22 MR. HILL: Yes, I've got a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 375 1 variation on what a couple of comments have 2 probably been said. Maybe the question is 3 along the lines of can you impute the number 4 from things that are unknown by the other 5 companies, given a boiler size of X with this 6 configuration and maybe something else, can 7 you estimate how much is -- 8 MR. KAPLAN: Well, in a pretty 9 simple way, that's really what we did for 10 many plants for the 2004 data; that is, if 11 the measures of efficiency for a plant for 12 2004 were out of line, and then if we looked 13 at their own measures for 2003, and those 14 were also out of line, then we used typical 15 industry averages. 16 MR. HILL: I've seen the 17 electricity data on there, the Web page, and 18 it is noticeable, so it would be nice -- it's 19 nice to have the number, even though you're 20 talking five percent or so of the use. 21 MR. KAPLAN: Well, you see, you've 22 just invalidated one of my key assumptions. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 376 1 You're the second person today to say they 2 actually look at those numbers. 3 (Laughter) 4 MR. KAPLAN: And prior to today, 5 I've had nobody actually give me a phone call 6 or say anything to indicate they actually 7 look at those allocations, but evidently, 8 someone's doing it. 9 DR. BURTON: Let me be the third, 10 but I also need to add that when I looked at 11 them, I thought, how'd they do that. 12 (Laughter) 13 SPEAKER: Now you know. 14 DR. FEDER: You mentioned for the 15 2004, you used the adjustment whenever the 16 steam-to-power ratio passed a certain 17 threshold, and then Scott mentioned that the 18 modelers were interested in the data. And I 19 was just thinking, now, if you're free to 20 model to that, you'll get -- and indeed, you 21 showed in that graph that we did see an 22 increase in that ratio compared to '02. So BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 377 1 clearly using this cutoff does introduce some 2 bias into this simple model of power-to-steam 3 ratio. 4 MR. KAPLAN: Well, it's certainly 5 not a particularly sophisticated technique 6 for adjusting the data, and -- 7 DR. FEDER: I was thinking along 8 what Neha was asking: Why collect it and not 9 leave it there and let the modelers worry 10 about it, even internally here, use whatever 11 methods they use, use historical data and not 12 -- sorry to say this among statisticians -- 13 massage the data. 14 MR. KAPLAN: Well, that's one 15 approach. Another approach is to find a way 16 to do the allocation which is both simple, 17 economical, and reasonable. If we don't do 18 the allocation at all -- I'm willing to think 19 about this -- but particularly for people 20 interested in natural gas data, that may pose 21 a problem. 22 MR. CLEVELAND: Why? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 378 1 MR. KAPLAN: Because when you're 2 trying to figure out what's going on with 3 natural gas consumption, a significant chunk 4 of it -- I don't want to call it a 5 percentage, but it's material, it's not tiny 6 -- is in the industrial sector. And when 7 people have done analyses of what is driving 8 gas demand, when you look in the industrial 9 sector and you can't tell how much of it is 10 going to make electricity and how much of it 11 is going to make steam, it makes the analysis 12 difficult. Now, one could say, well, fine, 13 do your own allocation -- and maybe that's 14 the thing we should do -- but that is a 15 particular issue which we're going to have to 16 think through and address. 17 MR. CLEVELAND: I think that's the 18 approach, though, because everyone -- there 19 are many different allocation schemes and the 20 type of analysis that you're going to be 21 using the allocation scheme in is going to 22 drive your choice of allocation scheme. And BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 379 1 so I don't think you picking one is going to 2 be, by definition, suboptimal for a large 3 percentage of the people wanting that data. 4 So why not just let the people who are 5 interested in that choose their own 6 allocation scheme? And they have to defend 7 it. 8 MR. KAPLAN: Another argument is 9 that if everyone rolls their own, then when 10 people start trying to compare data they're 11 all going to have different numbers, and if 12 EIA has a methodology and people think it's 13 reasonable, then it establishes a common 14 starting point for studies of the data. 15 DR. FEDER: Whenever respondents 16 have to make adjustments, you get some 17 strange things. Again, from the Canadian 18 experience in this area, census in Quebec, 19 they found out that, for some reason, many 20 more women tend to describe their 21 relationship as common law marriage where the 22 men described it as just a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 380 1 boyfriend-girlfriend relationship. So if you 2 did the estimates based -- no, that's a true 3 fact. If you use -- 4 MR. KAPLAN: I can believe it. 5 DR. FEDER: -- oh, and Nick is from 6 Quebec, maybe he can -- so if you did the 7 estimates using females or males, you got 8 different responses. So there is a 9 perception issue here, or a user definition. 10 That's why I thought what Neha said, why 11 collect it, why not let users decide that 12 it's more -- no, maybe it's -- 13 MR. KAPLAN: You actually just 14 raised an interesting and a really good 15 point, which is that we're talking about the 16 respondents. Well, who literally is the 17 respondent? Sometimes it's the plan 18 engineer, sometimes it's the data clerk in 19 some central office. So, again, men versus 20 women, plan engineers versus data clerks. I 21 think it's another argument for imposing more 22 uniformity on how this is being done. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 381 1 DR. HENGARTNER: Barbara, ---- have 2 stolen your thunder. 3 MS. FORSYTH: Hey, that was my 4 thunder. 5 (Laughter) 6 MS. FORSYTH: I think -- I was 7 supposed to be the discussant, and I think -- 8 my main point was that when you collect data 9 two different ways and they should add up and 10 they don't, and you spend way too much money 11 figuring out why they don't, you probably 12 have at least as much a cognitive problem as 13 you have a statistical problem, and I think 14 that's basically what we're saying here, that 15 the models that respondents are using to 16 report their data don't match the model that 17 EIA has of the process, and if you have a 18 different model, you should estimate it and I 19 think your new approaches, computing them 20 yourself or developing a Web tool that 21 computes it are both far better than 22 collecting data from two perspectives, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 382 1 neither of which is right, or both of which 2 are right. 3 MR. FREEDMAN: And in fact, 4 Barbara, I think given our discussions with 5 respondents -- and I take some responsibility 6 for problems that Stan was describing -- 7 MR. KAPLAN: We'll be talking about 8 that later. 9 MR. FREEDMAN: He's very generous 10 not to mention our partnership in this. It's 11 too generous to even say that there are 12 perceptual or cognitive models that are going 13 on here. A lot of it is, this doesn't make 14 any sense to us. I don't collect this, I 15 don't want to really be bothered -- 16 MS. FORSYTH: Exactly. 17 MR. FREEDMAN: -- so I'm going to 18 give EIA a number that will result in not 19 being called back. Well, in 2003, it didn't 20 work out -- 21 MS. FORSYTH: It didn't work. 22 MR. FREEDMAN: -- because they got BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 383 1 called back, and in 2004, I guess maybe it 2 did because -- 3 DR. BURTON: They didn't get called 4 back. 5 MR. FREEDMAN: They didn't get 6 called back, but it is not an easy conceptual 7 problem or a measurement problem to ask 8 somebody to report something that's not in 9 their perceptual framework. 10 MS. FORSYTH: Right, exactly. So 11 it is a cognitive model. 12 MR. FREEDMAN: Yeah. 13 DR. NEERCHAL: Wait -- I would be 14 curious ----- out of the 610 data points, 15 they have chosen to adjust 184 of them and 16 not adjust the others, so give me those 184 17 at the ------ and use one of the imputation 18 type of idea that Tom was talking about and 19 see how that compares to the adjustment that 20 you have actually implemented. That would be 21 a curious exercise, I think. 22 DR. BURTON: Truthfully, though, I BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 384 1 would have some concern that even the ones 2 that didn't trigger the adjustment in the 3 past, that they simply managed to make up 4 better numbers. 5 DR. NEERCHAL: I'm getting to the 6 actual exercise. 7 MR. KAPLAN: We know one 8 respondent, for example, who we had to 9 contact for something, but their numbers look 10 good every single month, and it's because -- 11 we know this from talking with the 12 respondent -- because the respondent has 13 picked out what he thinks is the right 14 power-to-steam ratio, he uses it every month, 15 and he makes sure the numbers add up right. 16 (Laughter) 17 DR. NEERCHAL: They're inliers. 18 MS. KHANNA: I just -- I had a 19 comment aside to your comment, which is 20 environmental economists do this all the 21 time. We elicit willingness to pay numbers 22 for scenic views, things that people don't BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 385 1 even think about. So there are techniques, 2 flawed no doubt, but they are there to try 3 and elicit data values that are not always 4 correct. 5 MR. FREEDMAN: I'd be interested in 6 hearing more about that from you because this 7 is a problem in other areas, too, not just 8 with ----. 9 DR. HENGARTNER: But there's one 10 big caveat: To be able to make that -----, 11 you have all the data collected beforehand, 12 because it's a regression-type technique. 13 MS. KHANNA: No, no, you actually 14 can ------ to pay estimates. I mean, --- 15 collect. You got out and ask Nick, what 16 would you be willing to pay for the view 17 outside here. 18 MS. FORSYTH: They don't behave 19 coherently either, right? 20 MS. KHANNA: No, that's what I 21 said, it's not a method without its problems, 22 but it's a well-established field. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 386 1 DR. BURTON: But there's a material 2 difference between doing some sort of 3 willingness to pay and saying how much of 4 this fuel was used to generate electricity. 5 There's the willingness to pay is their own 6 perception, and what you do is you try and 7 help them refine that perception. Here their 8 perception has nothing to do with what is or 9 is not the correct answer. 10 MS. KHANNA: I don't think so. I 11 actually think there is a perception because 12 if you've -- you got to -- 13 DR. BURTON: Their perception 14 doesn't matter, though, in generating the 15 right answer. There is a certain amount of 16 that fuel that's going into generating 17 electricity, not more, not less, that much. 18 MS. KHANNA: It depends on how you 19 think of it. 20 DR. BURTON: You just think of 21 another answer -- 22 MS. KHANNA: If you think of it BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 387 1 like an economist. I've got -- 2 DR. BURTON: Gee, I hope you do. 3 MS. KHANNA: I've got -- apropos, 4 I'm producing electricity, everything else is 5 my byproduct, which I didn't really care 6 about five years ago, but today, because we 7 have this technology that I can capture this 8 heat and use process heat, I'll do it, and 9 that's why the person says zero fuel to 10 process heat because it's not why he did it. 11 So it is a question of perceptions, 12 and I agree that the willingness-to-pay 13 technique is looking for something different, 14 but your point was perception and cognition, 15 and that's what the technique is aimed at 16 trying to get at and how do they understand 17 cognitive processes. And there may be 18 something there. 19 MS. FORSYTH: You could 20 certainly -- 21 MR. CLEVELAND: I'd be interested 22 in hearing more about it. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 388 1 MS. FORSYTH: And you certainly 2 could use it to model the differences between 3 the estimates you get from these two 4 different approaches, that willingness to pay 5 could shed light on the discrepancies. 6 MR. KAPLAN: Is that it? 7 DR. HENGARTNER: I think so. Thank 8 you very much, Stan. 9 MR. KAPLAN: Okay, thank you. 10 Sure. 11 (Applause) 12 DR. HENGARTNER: This brings us to 13 almost the end of today's session. There is 14 an invitation to the public to comment. 15 Okay, then it's my pleasure to 16 adjourn this meeting until tomorrow. I 17 remind you we are starting promptly at 8:30. 18 (Whereupon, at 5:01 p.m., the 19 PROCEEDINGS were adjourned.) 20 * * * * * 21 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382