AMERICAN STATISTICAL ASSOCIATION (ASA) MEETING OF THE COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY INFORMATION ADMINISTRATION (EIA) Washington, D.C. Thursday, April 28, 2005 2 1 COMMITTEE MEMBERS: 2 NICOLAS HENGARTNER, Chair Los Alamos National Laboratory 3 MARK BERNSTEIN 4 RAND Corporation 5 CUTLER CLEVELAND Center for Energy and Environmental Studies 6 JAE EDMONDS 7 Pacific Northwest National Laboratory 8 MOSHE FEDER Research Triangle Institute 9 BARBARA FORSYTH 10 Westat 11 NEHA KHANNA Binghamton University 12 NAGARAJ K. NEERCHAL 13 University of Maryland Baltimore County 14 SUSAN M. SEREIKA University of Pittsburgh 15 DARIUS SINGPURWALLA 16 LECG 17 RANDY R. SITTER Simon Fraser University 18 ALSO PRESENT: 19 MARGOT ANDERSON 20 Energy Information Administration 21 LORI ANITI Energy Information Administration 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 3 1 ALSO PRESENT (CONT'D): 2 STEPHANIE BATTLES Energy Information Administration 3 COLLEEN BLESSING 4 Energy Information Administration 5 BILL BOSTIC United States Census Bureau 6 ERIN BROEDECKER 7 Energy Information Administration 8 CHARLES BROENE Energy Information Administration 9 SUSAN BUCCI 10 United States Census Bureau 11 GUY CARUSO Energy Information Administration 12 DAVE COSTELLO 13 Energy Information Administration 14 HOWARD BRADSHER-FREDRICK 15 Energy Information Administration 16 FRED FREME Energy Information Administration 17 CAROL FRENCH 18 Energy Information Administration 19 DWIGHT FRENCH Energy Information Administration 20 KAREN FRIEDMAN 21 Energy Information Administration 22 NICKI HAITOT United States Census Bureau BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 4 1 ALSO PRESENT (CONT'D): 2 SUSAN HOLTE Energy Information Administration 3 CRAWFORD HONEYCUTT 4 Energy Information Administration 5 RICK HOUGH United States Census Bureau 6 PAUL HSEN 7 United States Census Bureau 8 ALETHEA JENNINGS Energy Information Administration 9 JIM JOOSTON 10 Energy Information Administration 11 NANCY KIRKENDALL Energy Information Administration 12 TANCRED LIDDERDALE 13 Energy Information Administration 14 RUEY PYNG LU Energy Information Administration 15 PRESTON McDOWNEY 16 Energy Information Administration 17 RENEE MILLER Energy Information Administration 18 MARK RODEKOHR 19 Energy Information Administration 20 REGINALD SANDERS OnLocation, Inc. 21 LAWRENCE STRAUSS 22 Energy Information Administration BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 5 1 ALSO PRESENT (CONT'D): 2 AMY SWEENEY Energy Information Administration 3 PHILLIP TSENG 4 Energy Information Administration 5 BILL WEINIG Energy Information Administration 6 JOHN WOOD 7 Energy Information Administration 8 9 10 11 12 13 * * * * * 14 15 16 17 18 19 20 21 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 6 1 C O N T E N T S 2 AGENDA SESSION: PAGE 3 Updates Since Fall 2004 Meeting 14 4 Greetings and Remarks 24 5 Regionalizing the Short-Term 41 Energy Outlook (STEO) Forecast 6 STEO Performance Indicators: 78 7 Diagnostics and Forecast Errors 8 STEO Electricity Modeling 97 9 ASA Committee Discussants 161 10 Regional STEO Propane and Heating Oil Modules 174 11 ASA Committee Discussant on 233 12 Regional STEO Propane and Heating Oil Modules 13 Follow-Up on Frames Team 235 14 Activities 15 EIA's Proposed Strategy for 265 Addressing Declining Response 16 Rates in the Residential Energy Consumption Survey (RECS) 17 ASA Discussant on EIA's Proposed 325 18 Strategy for Declining RECS Response Rates 19 Hands-On Usability Testing of 341 20 EIA's New Website Design 21 22 * * * * * BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 7 1 P R O C E E D I N G S 2 DR. HENGARTNER: Ladies and 3 gentlemen, members of the committee, members 4 of the audience, this is the first meeting of 5 our committee since it was determined that we 6 are not covered by the provisions of the 7 Federal Advisory Committee Act. The DOE's 8 Office of General Counsel has reexamined the 9 statute of the Committee and advised that 10 only committee subjected to the department's 11 actual management or control are covered by 12 the act. And in fact it does absolutely 13 change nothing for us in the way our 14 committee is working. So this is simply how 15 we're perceived by the DOE. It's not 16 internal on how we're working and how we're 17 appointed and so on. So this should be 18 transparent to all of us except for 19 management. 20 Moving to the real agenda, let me 21 remind you that this is an ASA Committee, as 22 is obvious and not an EIA committee, which BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 8 1 periodically provides input and advice to the 2 EIA. This meeting is open to the public and 3 public comments are welcome. Time will be 4 set aside at the end of each day and morning 5 and afternoon for public comments. All 6 attendees, EIA employees and committee 7 members and members of the public, should 8 sign the sign-in sheet at the entrance and 9 please write down your phone number and 10 e-mail. 11 The other thing I would like to let 12 you know, there are restrooms. If you go out 13 of the door in to the hallway you turn to the 14 left and it's at the end of it. That's 15 important for the new committee members. 16 There is also a telephone here. We will come 17 to that in a minute. There is also a 18 telephone feed here if anybody needs to have 19 someone call you. The telephone number, I am 20 giving it you, it's 202-586-3071 or 586-6273 21 or 586-6202. Now, you don't need to remember 22 these numbers. I have them written down BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 9 1 here. If you need them, just come and ask 2 me. 3 Kathleen Wert is in charge of the 4 ASA and is helping the committee members with 5 reimbursement and other logistical questions 6 and she has a new help, Margarita Navos, who 7 will be helping her in that respect. On the 8 technical side each committee member is asked 9 to speak loudly and clearly, to articulate, 10 and please speak in the microphone such that 11 the recorder here can actually help in his 12 transcriptions. Now, finally before we start 13 I would like to welcome our two new committee 14 members, Cutler Cleveland from Boston. 15 Welcome. 16 MR. CLEVELAND: Thank you. 17 DR. HENGARTNER: And Mr. Darius 18 Singpurwalla. He recently moved to DC so 19 welcome. 20 MR. SINGPURWALLA: Thanks. 21 DR. HENGARTNER: Now that I have 22 introduced the new members I think it's just BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 10 1 befitting that we all introduce ourselves 2 starting with myself. This includes members 3 of the public here present. My name is Nick 4 Hengartner, I am from Los Alamos National 5 Laboratory, and I am serving here as the 6 chairman of this committee. 7 MS. KIRKENDALL: Nancy Kirkendall, 8 I am Director of Statistics and Methods Group 9 in EIA. 10 MR. CARUSO: Guy Caruso, 11 administrator of EIA. 12 MR. WEINIG: Bill Weinig, 13 Statistics and Methods Group, EIA. 14 MR. BERNSTEIN: Mark Bernstein on 15 the ASA Committee. 16 MR. CLEVELAND: Cutler Cleveland, 17 ASA Committee. 18 MR. EDMONDS: Jae Edmonds, ASA 19 Committee. 20 DR. FEDER: Moshe Feder, ASA 21 Committee. 22 MS. FORSYTH: Barbara Forsyth, ASA BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 11 1 Committee. 2 MS. KHANNA: Neha Khanna, ASA 3 Committee. 4 MS. SEREIKA: Susan Sereika, ASA 5 Committee. 6 MR. SINGPURWALLA: Darius 7 Singpurwalla, ASA Committee. 8 DR. SITTER: Randy Sitter, ASA 9 Committee. 10 MR. STRAUSS: Lawrence Strauss, 11 Statistics and Methods Group, EIA. 12 MR. WOOD: John Wood, Director of 13 Reserves and Production Division, EIA. 14 MS. ANDERSON: Margot Anderson, 15 EIA. 16 MS. FRENCH: Carol French, EIA. 17 MS. MILLER: Renee Miller, EIA. 18 MS. BLESSING: Colleen Blessing, 19 EIA. 20 MR. PSENG: Phillip Tseng, EIA. 21 MR. BROENE: Charles Broene, EIA. 22 MS. JENNINGS: Alethea Jennings, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 12 1 EIA. 2 MS. FRIEDMAN: Karen Friedman, EIA. 3 MS. SWEENEY: Amy Sweeney, EIA. 4 MR. WADE: Steve Wade, EIA. 5 MS. HOLTE: Susan Holte, EIA. 6 MR. HONEYCUTT: Crawford Honeycutt, 7 EIA. 8 MR. JOOSTON: Jim Jooston, EIA. 9 MS. BOEDECKER: Erin Boedecker, 10 EIA. 11 MR. MCDOWNEY: Preston McDowney, 12 EIA. 13 MR. FREME: Fred Freme, EIA. 14 MR. LU: Ruey Pyng Lu, EIA. 15 DR. HENGARTNER: Thank you very 16 much for introducing yourself. Nancy here is 17 very important to this meeting because she is 18 the designated federal officer. That means 19 that she must attend the meetings and she 20 may, if she wants, chair the meeting or call 21 a stop to the meeting if she deems it 22 necessary, and she must approve all the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 13 1 agendas and call the meetings of the ASA 2 Advisory Committee. So Nancy here is very 3 important. 4 MS. KIRKENDALL: The only thing is 5 that's probably not true any more because I 6 think that's a FACA thing. 7 DR. HENGARTNER: Now that I've 8 painted you in such a good light, Nancy. 9 I think this is going to be a good 10 meeting. I looked at the agenda that Nancy 11 provided and this looks like a really 12 interesting meeting. I think the last few 13 meetings generally have been superb. I hope 14 that this will continue and I hope that we 15 are all going to come away with some good 16 recommendations and some advice that will be 17 useful for the EIA. 18 So I will be the taskmaster. I 19 will keep you on schedule. That means we are 20 10 minutes late as we are. Guy Caruso will 21 give a talk about the state of EIA. 22 Unfortunately he is looking for his slides BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 14 1 and therefore Nancy Kirkendall will start as 2 presenter. 3 I think we are going to switch 4 Nancy and Guy and Nancy will start if that's 5 all right with you. And the other feature 6 for this morning is short-term range in 7 forecasting systems and most importantly I 8 want to remind the committee members there is 9 lunch provided, so the economists are wrong, 10 there is a free lunch, and it is going to be 11 located in the usual room, which is EA 226 on 12 the first floor. So you want to start, 13 Nancy? 14 MS. KIRKENDALL: As Nick said, we 15 are no longer under FACA. There are new 16 rules by the government ethics office. That 17 is why we got out from under FACA and they 18 were that the ASA president appoints members 19 and it's true we make recommendations but he 20 still sneaks a member sometimes but that is 21 more little challenges in communication than 22 anything else. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 15 1 And then the other main reason was 2 that we do not ask the committee to provide 3 consensus recommendations. We want your 4 technical recommendations and when of course 5 you all chime in and agree that carries more 6 weight than if it's one voice but we really 7 do like to hear a variety of opinions and, as 8 Nick said, we don't expect to operate any 9 differently than we have in the past. We 10 thought it has worked pretty well so we do 11 not see any reason to. 12 So my purpose today is to give you 13 updates, a preview of coming attractions 14 including this meeting and the next meeting. 15 So these are topics that you are going to 16 hear more about today. You have heard about 17 them in the past. And particularly this 18 morning we have a lot of discussion about the 19 short-term energy outlook in the past. 20 Phil Tseng and Dave Costello have 21 talked about their efforts in the modeling of 22 electricity. This is a big effort in EIA in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 16 1 the past. Our short-term energy forecasts 2 have all been done at the national level and 3 now we are trying to move to the regional 4 models so we have got a lot of issues to talk 5 about with you. 6 Margot Anderson is going to give 7 you background information and talk about big 8 picture items. Phil Tseng and Dave Costello 9 will tell you more about where they are on 10 the electricity regional modeling and Tanc 11 Lidderdale will talk about all the rest of 12 the regional modeling. So I think that will 13 be a very interesting session. 14 We have also talked a lot about 15 assessing EIA frames. This is an item in our 16 strategic plan is to take a look at our 17 frames and see how good they are. Howard 18 Bradsher-Fredrick will be talking about the 19 results from an EIA look at our frames. It's 20 really hard to have a good quantitative 21 assessment so it had to use a little 22 qualitative judgment in it but he will tell BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 17 1 you the results and what we've learned from 2 that. 3 We also have our colleagues at the 4 Census Bureau doing some comparisons of some 5 frames and you will have an update of that. 6 We will also have more information about that 7 project at the fall meeting because they'll 8 look at some of our bigger surveys. I think 9 today they're talking about two of our 10 smaller surveys where they've completed the 11 work. 12 As a number of you have heard, we 13 are doing usability testing of a new website 14 design. I think all of the committee members 15 should be there, two separate break-out 16 sessions so all of the committee members will 17 get to participate in the testing, and that 18 will be valuable information. 19 You will hear a new talk. This is 20 a new talk. Did we skip one? I did skip 21 one. This is continued. We are going to 22 hear about the natural gas production survey. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 18 1 There's a new survey. It collects monthly 2 natural gas production. The committee has 3 heard a lot about how we have estimated 4 natural gas production in the past 5 particularly in certain states, in Texas. 6 Randy Sitter's student did some work that we 7 actually used. 8 You heard about our efforts to 9 estimate in the Gulf of Mexico where there is 10 very little data and now what we have is a 11 new monthly survey. John was going to be 12 talking about the results. It started 13 operation in January. The first data are 14 just being processed and are close to being 15 ready to release. So he is going to ask some 16 advice on when do we decide that they should 17 be our official data, how do we decide that 18 they are accurate enough to release, and that 19 sort of thing. 20 826 is a monthly survey that 21 collects information on electricity sales and 22 resale and we are trying to work with BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 19 1 estimation methods. Do we use outlier 2 detection, you've heard about that. We will 3 talk about it on Friday. We will talk about 4 it some more next time, I'm sure. 5 So this is an ongoing project and 6 then another major effort that's going on, we 7 talked last time about external evaluations. 8 As we told you then, we had a poor report in 9 the OMB's PART report. That's a performance 10 assessment rating tool. They go through and 11 ask a bunch of questions and we flunked in 12 terms of being able to say that we have 13 regular external reviews that look at our 14 relevance programs, are we doing the right 15 things, and we had two discussions last time. 16 One was Brenda Cox talking about 17 doing an evaluation of a survey program, and 18 she will give us an update on that work, and 19 the other one was even more exciting. The 20 committee came back and told us we needed to 21 have a high-powered review team modeled on 22 academic evaluations and so we actually have BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 20 1 a plan. We're going to go forward with it 2 and we will hear about that on Friday too. 3 So that one is particularly exciting. 4 I think I covered the first one. 5 The second bullet is declining response 6 rates. Our Residential Energy Consumption 7 Survey, Dwight French, they're looking at a 8 lot of different initiatives that might be 9 used for improving response rates for RECS. 10 I think one of the most interesting things is 11 that they are going to be doing an incentive 12 experiment with it. They're also looking at 13 other things that I think will be more 14 applicable to some of our other surveys so 15 interesting things going on there. 16 So these are topics that you will 17 probably hear about at the next meeting but 18 we did not have time to squeeze them in. 19 Either one of two things, we either did not 20 have time to talk about them this time or we 21 weren't ready to talk about them again. 22 There's post-stratification for the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 21 1 manufacturing energy consumption survey. 2 This is collaborative work with the Census 3 Bureau and so they'll come back and talk to 4 us again, at least we hope. 5 We had a group from my office did 6 some cognitive testing on the Form EIA 920. 7 That's a survey that goes to basically 8 manufacturing plants that asks about their 9 electricity generation and consumption. They 10 used cognitive interviews to redesign the 11 form and so this test is to try to figure out 12 what was the impact of that work. We have 13 done a lot of work since then and hope to 14 have a nice presentation next time. 15 Then the last item, customer 16 evaluations of our forecasting and analytical 17 products, these are surveys we have talked 18 about a couple of times. Last year we did a 19 survey of participants in our NEMS Conference 20 to get an assessment of our annual and 21 international energy outlook. 22 In the fall we did a similar survey BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 22 1 but we used a sample of subscribers to the 2 list serve for the short-term energy outlook 3 and we have decided that that methodology 4 works better so we are going to use the same 5 methodology in the next month or two to 6 assess the AEO and the IEO. One of the 7 recommendations of the committee last time 8 was that we should not do an annual 9 assessment of each product and so maybe in 10 the fall we can talk about how to schedule a 11 rotating assessment so that we can capture 12 things in a less burdensome manner for 13 everybody. 14 These are just other things that we 15 talked about. Work is going on, George Latey 16 (?) is continuing his work on diagnostic 17 modeling on NEMS solutions to try to 18 illuminate how good the forecast are to get 19 some sensitivity analysis for our 20 forecasting. Howard Bradsher-Fredrick is 21 going to present a paper on dual system 22 estimation at the Federal Committee on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 23 1 Statistical Methodologies Research 2 Conference, which will be in the fall, so 3 that should be interesting. 4 And then we have a session at the 5 joint statistical meetings in August on the 6 new natural gas production survey. It's 7 going to be an introduction to the problem, 8 description of the survey methods. We will 9 have done an extensive simulation including 10 the estimators that you recommended and then 11 John will also talk about what we're doing 12 now and how it's all going. And that is on 13 Wednesday at 8:30 in the morning so if you're 14 at the joint statistical meetings you should 15 come and hear the talk. 16 And that is all that I have so if 17 we have Guy's slides then I can turn this 18 over to Guy. 19 DR. HENGARTNER: I would like to 20 introduce Guy Caruso, Administrator of EIA, 21 who will give us an overview of what the EIA 22 is currently doing as well as tell us what we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 24 1 are going to pay for gas. 2 MR. CARUSO: There is a direct 3 correlation between the price of gasoline and 4 the amount of requests the EIA gets in hits 5 on the website. First let me congratulate 6 you, Nick, for promotion to chairman and 7 welcome, Cutler and Darius, to the committee. 8 I appreciate all of your time that you put 9 into this committee as colleagues that had 10 been involved for a number of years. 11 We really take your advice 12 seriously and it clearly makes a difference 13 in many of the things we do, as you'll see 14 from the agenda, in some of the products that 15 we've changed and the methodological changes 16 we've made so a big thanks for the time you 17 take to do this and I know the pay isn't that 18 great. 19 I want to talk a little bit about 20 some of our activities since the last meeting 21 and it's on the first slide and I think, as 22 Nick's comment implied, when energy prices BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 25 1 are high there is a real effect on our work, 2 not only the day to day products like our 3 weekly petroleum numbers, weekly natural gas 4 numbers, but currently once again an energy 5 bill is being debated on the Hill and we get 6 asked to do a lot of service reports related 7 to the bill or elements that are connected to 8 the bill and two weeks ago we released the 9 analysis of the National Commission on Energy 10 Policy that was a report put out in December 11 of '04 which makes a lot of policy 12 recommendations, many of which go well beyond 13 anything the Congress has actually passed or 14 even contemplates passing like substantial 15 increase in CAFE standards and things like 16 that. 17 But in any case it's an example of 18 things we're being asked to do. We're being 19 asked right now to do some work for Senators 20 Dorgan and Jeffords and Imhofe as well. So 21 there has been a lot of activities directly 22 related to debates on the Hill. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 26 1 To answer the next question 2 directly, the summer fuels outlook which we 3 released two weeks ago, we say that the 4 gasoline price is going to average $2.35 in 5 May which will be the peak. By Labor Day it 6 should be down to 2.20. 7 One of the things that occurred 8 since our last meeting was a substantial 9 revision in one of the weekly natural gas 10 storage numbers, actually Thanksgiving week, 11 and we've taken a hard look at our revisions 12 policy on the weekly data and just yesterday 13 on the web the Federal Register notice was 14 released with our new revisions policy and if 15 anyone is interested in getting into more 16 detail I'll be happy to do that but it's 17 another issue that I know we've discussed in 18 this committee at that specific report about 19 revisions policy in general, and then you'll 20 have on the agenda the natural gas production 21 survey which John Wood has talked to you 22 about in previous meetings and we'll get some BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 27 1 more reviews on that. 2 Nancy mentioned the new development 3 with an external study team being formed and 4 we'll have that on the agenda some new 5 developments with respect to the strategic 6 plan and I'll give you a few updates on where 7 we are with our '06 budget. In fact this 8 week we're meeting with all the officers to 9 talk about '07 so the budget's clearly an 10 important aspect of getting some of these 11 things down that we've talked about in this 12 committee. 13 The next slide just talks about 14 some of the things we've done for Congress. 15 I mentioned some of the service reports but 16 then there is the regular products such as 17 the winter fuels outlook which we do every 18 October and then the annual energy outlook in 19 both its reference case and the full report 20 which was released in late February. 21 We are working on the international 22 energy outlook now and will have that out by BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 28 1 early July. This next slide shows a couple 2 of these service reports, one on the mercury 3 controls that Senators Imhofe and Voinovich 4 requested. I mentioned the National 5 Commission on Energy Policy already. That 6 was requested by Senator Bingaman and there 7 are a couple of others in the pipeline now. 8 So I think the main point here is 9 that EIA is being asked to do even more on 10 what-if scenarios with respect to the current 11 energy bill and other energy policy changes 12 that are being contemplated and the modeling 13 and other techniques that have come before 14 this committee are being used extensively. 15 I already talked a little bit about 16 this, the average price for the summer, 2.28 17 with a peak in May. Of course, it seems to 18 have gotten more political attention. The 19 President's speech yesterday clearly was an 20 example of that and, of course, the bill was 21 passed by the House. The House version, HR 22 6, was passed last week. The Senate will BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 29 1 debate their version in the third week of May 2 and they're hoping to have something to go to 3 conference so that the President can sign it 4 by August, that's the goal. 5 The next one again talks about how 6 often we are being asked to provide data and 7 an analysis not only to Congress but the 8 administration and certainly lots of press 9 attention and our products continue to be 10 heavily used by the financial community and I 11 think we talked about last time that there 12 are actually now derivatives based on the 13 actual numbers that the EIA releases on a 14 weekly basis. 15 You can bet on what our natural gas 16 number's going to be at 10:30 this morning. 17 An actual financial instrument has been 18 created and is now up and running and we were 19 concerned about what impact that might have 20 and I think the main impact we have seen so 21 far is this very close scrutiny to the 22 numbers and any kind of deviation. For BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 30 1 example, the large revision I mentioned in 2 the week of Thanksgiving last year created a 3 huge stir and any time there is any kind of a 4 technical problem where the numbers are even 5 two minutes late from 10:30 we get a huge 6 amount of criticism from the media. 7 So I think for us as an 8 organization managing the information system 9 what it's done is put on more pressure for 10 both not only accuracy but timeliness and 11 things like revisions policy, which one 12 wouldn't think would get that much attention, 13 was a subject of a lot of trade journal 14 attention which I think in the long run is 15 good as long as the resources are there to 16 deal with it. 17 The real problem is increased 18 detention and requirements for greater 19 timeliness, even more attention to data 20 quality, which, as you know, in this 21 committee we pay a lot of attention to under 22 declining real resources so it's a real BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 31 1 challenge —————— storage I've already really 2 referred to that. That's really what we are 3 talking about here in not only the financial 4 instruments but the attention to the 5 revisions policy. 6 Next one is the production survey. 7 John Wood will go into more detail on the 8 later agenda item here but we are very 9 hopeful that that will provide more timely 10 and accurate monthly production data which, 11 as you know from past discussions in this 12 committee, we have relied exclusively on 13 state data. So this will be the first time 14 EIA collects that on a monthly basis directly 15 from the operators. The sample, the frames, 16 and the methodology we've discussed in this 17 committee. 18 We're very excited about the 19 initial results but I won't take anything 20 away from John Slater's comments so I just 21 want to mention that. The external review 22 team, I'm not sure if you had mentioned, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 32 1 Nancy, this was the outcome of the PART 2 process. Our OMB review downgraded us 3 because we have not had an independent review 4 of the full EIA program by outside experts. 5 We have had a number of independent 6 expert reviews on specific issues, 7 methodologies, and other more narrowly 8 focused reviews. There was a National 9 Academy Study in '91, but that was really 10 focused on NEMS so even that NAS wasn't a 11 full program review. So what we have done is 12 we have established a concept for an external 13 study team of five very senior people. Up to 14 now and I think I'd be announcing to the 15 public for the first time that Denny Ellerman 16 of MIT will chair that study team and he will 17 be the one selecting the other four team 18 members so there is indeed truly an 19 independent set of experts, and that will be 20 on the agenda later on so I won't go into any 21 more detail on that. 22 On the strategic plan you've heard BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 33 1 about that, a number of other meetings. We 2 have added a fourth goal which started out as 3 an IT, information technology, goal but as it 4 evolved it certainly has broadened to include 5 a number of other cross-cutting issues and we 6 will be discussing with you some of those 7 issues in more detail. The next slide shows 8 some of the things. 9 Four groups have been established 10 within EIA to discuss survey frames, IDC, 11 Internet data collection, things like 12 disclosure limitations and database 13 commonality and I think what's particularly 14 interesting about this is that there is a 15 great deal of interoffice cooperation on this 16 goal 4 and we see lots of opportunity for 17 improvements of efficiencies and the number 18 of things that we've been hoping to use the 19 IT goal to lead to so I think from the 20 perspective of the managers we're looking 21 forward to implementing that goal 4. 22 On the budget we have roughly an BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 34 1 $86 million request before Congress for FY 2 '06 and the House Energy and Water Committee, 3 which as of this budget our budget was moved 4 from Interior to Energy and Water so now we 5 are in the same budget as all of DOE except 6 for the weapons and NSA part of DOE and so 7 this will be a new process for us, a new set 8 of Congressional members to deal with, and I 9 have been spending a lot of time up there 10 meeting with many of the members of the House 11 Energy and Water Committee and they are going 12 to mark up our budget, I believe, May 12 is 13 the latest date I heard. So we are hopeful 14 that roughly 86 million, 85.9, is about a 15 $2-million increase over FY '05. 16 And a number of specific things 17 around the board, they were about what's in 18 this request for '06. We want to fund the 19 consumption surveys. We have been trying to 20 get by with a minimal amount of funding and 21 we've decided that it is affecting the 22 quality or could potentially affect the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 35 1 quality so we put some more money in '06 to 2 assure continued quality of the quadrennial 3 consumption surveys and the issue of cyber 4 security has clearly become even more 5 important and in particular with the 6 increased sensitivity of the weekly petroleum 7 and natural gas data. We hope to spend some 8 more money on that in '06. 9 There was a new voluntary reporting 10 of greenhouse gas, the 1605B, that Jae 11 Edmonds knows better than anyone else in the 12 room here. The EIA will be asked to collect 13 that data and it will add to our budget 14 requirements in '06. 15 So let me close by saying again how 16 much we appreciate your comments and advice 17 on the items we're going to be talking about 18 over the next day and a half. The next item 19 will be short-term energy outlook. Margot 20 will talk to you about some of the things we 21 are doing there and I mentioned the external 22 review team. Production survey, John Wood is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 36 1 with us. 2 We also are working hard to make 3 our website even more usable and 4 maneuverable. We've done a number of 5 customer surveys and we are trying to 6 implement those and John Weiner is heading up 7 an effort within EIA to do that and we look 8 forward to hearing from you about that as 9 well. And then there will be a number of 10 other statistical and modeling issues. 11 Welcome back to Washington for 12 those of you coming from out of town and 13 thanks once again and, Nick, I appreciate 14 your leadership on this. 15 DR. HENGARTNER: Don't go away. 16 MR. CARUSO: Any questions besides 17 the price of gasoline? 18 DR. HENGARTNER: I mean, the whole 19 audience is silent. That's a bad sign. 20 DR. FEDER: Guy, I noticed that you 21 had a lower bound on the price of barrel of 22 oil that was 50. Is there any upper limit? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 37 1 MR. CARUSO: Well, there are some 2 economists in here. In the short run it's 3 probably not much of an upper limit but in 4 our long-term outlook we have raised the 5 expectation where oil prices are going and 6 it's partly because of the increased cost of 7 finding and developing crude but in the short 8 run the real issue is that there is hardly 9 any unused productive capacity in the world 10 so there is no cushion to deal with shocks to 11 the system other than strategic petroleum 12 reserve and that, of course, is limited. So 13 in the upper bound there has been a report in 14 recent weeks from Goldman Sachs using $105 15 per barrel price. 16 If you put any price there in the 17 short run I think the real answer is at some 18 point there would have to be some reaction to 19 use the SPR to invoke the IEA short-term 20 response system so we have a statistical 21 boundary on our short term and the STEO does 22 that every month but I think that's a 95 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 38 1 percent confidence interval so it doesn't 2 cover the more extreme cases even. 3 DR. HENGARTNER: Neha. 4 MS. KHANNA: Since we're talking 5 oil I have to ask a question. What do you 6 think are possibilities of the effectiveness 7 of the open price band? I mean, it's been 8 violated ———————— the last two years but they 9 stuck to it for 15 years. What is the EIA's 10 expectation about where the price band is 11 going and how effective it will be? 12 MR. CARUSO: They suspended the 13 price band I guess it was in the fall of '04. 14 OPEC has a committee that supposedly is to be 15 reviewing this. We have some contact with 16 the members of that committee and they don't 17 want to set a new band partly because no one 18 knows where the price is going to settle out 19 so I'm not sure when or if OPEC will 20 reestablish that band. 21 I mean, when you talk to the Saudis 22 privately and even some of the public BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 39 1 comments seem to be they're thinking in the 2 $40-50 range but they are formally saying 3 that but that seems to be what they're 4 thinking. 5 We did in the AEO this year 6 something we haven't done before partly 7 because some people thought it might be too 8 politically sensitive, look at what the OPEC 9 revenues would be under various price bands 10 and what it showed was three high price cases 11 above the reference case. In two of those 12 cases the OPEC revenues were higher at lower 13 production which has always been ———————————— 14 revenues —————————————— 15 DR. HENGARTNER: Thank you very 16 much. Before we continue I'd like the people 17 who just walked into the room or who came 18 after we introduced ourselves to come up to 19 the micro and tell who you are and what your 20 affiliation is. 21 DR. NEERCHAL: I apologize for 22 walking in late. I think I am the committee BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 40 1 member who lives closest to Washington, DC. 2 I have the right to be walking in the latest, 3 I think. I'm Nagaraj Neerchal from the 4 University of Maryland Baltimore County. 5 MR. HOUGH: My name is Rick Hough. 6 I'm with the US Census Bureau. 7 MS. HAITOT: Nicki Haitot, US 8 Census Bureau. 9 MR. BOSTIC: Bill Bostic, US Census 10 Bureau. 11 MR. HSEN: Paul Hsen, US Census 12 Bureau. 13 MS. BUCCI: Susan Bucci, US Census 14 Bureau. 15 MR. BRADSHER-FREDRICK: Howard 16 Bradsher-Fredrick, EIA. 17 MR. FRENCH: Dwight French, I run 18 those pesky quadrennial consumption surveys 19 in the EIA. 20 DR. HENGARTNER: Thank you very 21 much. It's my pleasure to invite Margot 22 Anderson to come and talk to us about BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 41 1 regional and short-term energy outlook. 2 That's the STEO. 3 MS. ANDERSON: Good morning, 4 everyone. Thank you for having me. It's a 5 pleasure to speak with you all day about two 6 topics very close to my heart and the 7 business that I am required to do within EIA. 8 I am relatively new to EIA and I get to 9 oversee many of the activities that you are 10 going to hear about today which deal with the 11 short-term outlook and the quadrennial 12 surveys as well. I want to talk today a bit 13 about the project we have underway to 14 regionalize our short-term energy outlook. 15 I think in ASA meetings in the past 16 you have had a presentation on various 17 components. I want to put the rewind button 18 on and go back and do an overview of what 19 we're trying to accomplish and why we're 20 regionalizing and what some of the challenges 21 are. 22 Later today you are going to hear BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 42 1 from some of the folks that are very deep in 2 the actual modeling and you'll hear about the 3 electricity component of the regional steel 4 and you'll hear, I believe, about the propane 5 component and other components as well. 6 With us in the room are several of 7 the folks intimately involved with regional 8 steel, Mark Rodekohr over here who is leading 9 the charge, I see Tanc here and Phil and Dave 10 Costello and I am sure there are others but I 11 haven't put my glasses on. So when we get to 12 any question that I can't answer, I know that 13 they will help me out. 14 I want to talk about the purpose 15 and objectives, I want to talk about our 16 schedule, and I want to talk about the 17 overview of the actual model and the key 18 components of the model so you can see how it 19 all fits together, and then I want to briefly 20 address some of the challenges that we're 21 facing as we move from a national model to a 22 regional model, might I say, at the same BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 43 1 time. A lot of the work that we're doing to 2 regionalize this model is occurring while 3 we're putting out the monthly short term 4 energy outlook, all done with about the same 5 staff, and it has presented quite a challenge 6 to us to be going back and forth from the 7 building of the regional to putting out the 8 monthly actual outlook. 9 Why are we doing this? What is our 10 purpose and what are our objectives? We got 11 into this about mid-2003 and the objective is 12 to maintain a national perspective while 13 providing some regional richness to our 14 short-term outlook. For years we've been 15 presenting just the national numbers but we 16 know that the national number isn't fully 17 representative of what's going on in each of 18 the regions and so we would like to be able 19 to tell folks in the northeast or the 20 southwest or in California why they might be 21 facing a different set of energy prices, how 22 their demands are different vis-à-vis the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 44 1 entire US demand level or against each other 2 in the regions. 3 So it provides the kind of richness 4 that we often get asked about when people say 5 what's going on in my area, what's going on 6 in my State, how can you tell me something 7 more about how it will affect me as opposed 8 to a national average, so it provides this 9 richness and relevance to what we think key 10 customers want. In addition it adds this 11 depth. We gain a better understanding of 12 local regional supply problems. By zeroing 13 in and looking at what's happening in these 14 regions it helps inform the national 15 perspective as well. 16 So it provides a whole new set of 17 options for us to look at where there may be 18 problems that we want to zero in on whether 19 on the demand side or on the supply side. It 20 may add, and the caveat here is may, accuracy 21 to the national level estimates. 22 It's not clear to us at this time BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 45 1 whether the national forecast will be 2 significantly different than it would be from 3 using the national model that we do now but 4 we do think that by adding this regional 5 richness we may add some improved accuracy to 6 the national level forecast and I'm going to 7 be talking about accuracy in forecasts in my 8 second presentation today. 9 So this is some of the motivation 10 for why we got into this. Let me review the 11 schedule with you. Again, we began in 12 mid-2003 and we are scheduled for completion 13 in late summer of 2005. What are we doing 14 now? We're continuing to do the model 15 development. The very large part of doing 16 the regional model is dealing with the 17 electricity side of the model and that's why 18 we're spending so much time on it and why 19 you're going to hear about it in much more 20 detail today. 21 So we're undergoing model 22 development. Certainly testing goes on all BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 46 1 the time as we build modules and we preparing 2 the documentation. So in essence we're going 3 through a testing period, we're going to have 4 expert review of the documentation, and we 5 want to go live in late summer 2005. 6 We've got a lot of issues about how 7 we communicate what we're doing to the 8 public. We want to continue, of course, the 9 monthly release of a short-term energy 10 outlook. What will that look like? Will it 11 be more extensive than what we're doing now? 12 Well, sure, we're going to have a lot more 13 tables associated with the regional 14 components of the model. How much that 15 translates into a textual description of 16 what's going on is still undecided yet, how 17 much do we expand or contract or reformulate 18 the actual short-term outlook that you see. 19 It clearly is a lot more information but 20 trying to organize it in a way that will be 21 useful to the readership is an open question. 22 We currently have a downloadable PC BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 47 1 version of the short-term energy outlook. 2 Are we going to continue that when we have a 3 much more complicated model? This is an open 4 question for ourselves. We have a relatively 5 small but a hardcore set of users that like 6 to download the model and whether we're going 7 to be able to offer this under current budget 8 and resource constraints we don't know. We 9 would like to able to offer it to customers 10 in some version but this is again an open 11 question of who will be able to get access to 12 a downloadable version. 13 We'd like to expand the web 14 presentation and the query system to include 15 the regional detail. We may not be able to 16 publish all the information every month but 17 we're investigating whether there may be 18 additional information in an improved format, 19 i.e., a query system, that allows the 20 readership to get at the questions that they 21 want to get at without us having to package 22 it for them in output that we would release BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 48 1 every month. 2 So by adding this great deal of 3 complexity of the model we suspect that we 4 will have an increased number of customers 5 who are interested in the regional component 6 and how we present that information for 7 consumption is a big issue for us. Clearly 8 we will provide all the documentation as we 9 get it expert reviewed and available to those 10 folks that are interested in looking at the 11 specifics of what we did. 12 I suspect that while we will go 13 live we will continue to upgrade this model 14 over time. I don't think that once we're 15 done we're done. It will be an evolutionary 16 process of refining the model as we get 17 comment back and as we understand how people 18 are using it and as we can improve our access 19 and analysis of the data. So I'd like to 20 look at it as evolutionary with big points at 21 which we release things to the public. 22 Some of these things I've already BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 49 1 covered but I think most of you know that the 2 system that we use now, the STIF system, is 3 moving to the RSTEM system, which expands on 4 the STIF systems to include these regional 5 demand and price determinants for most energy 6 products. It's an econometric model that 7 captures demand, inventories, prices, and 8 it's used to forecast the things that Guy was 9 talking about earlier. It's used to support 10 the monthly outlook and it's used to support 11 the summer fuels and the winter fuels outlook 12 as well which will continue now with regional 13 richness. 14 RSTEM will allow more detail 15 treatment of the trends in the regions far 16 more than was possible under STIFS. The 17 frequency of the outlook will remain the 18 same. I should have mentioned that, I think, 19 in the previous slide. We are not going to a 20 longer or a shorter forecast period. We're 21 still going with the 12- to 24-month look 22 ahead. So that will stay the same under the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 50 1 new system. As an aside we often get asked 2 about forecast for the three to five-year 3 period. This model is not it; it is a 4 short-term energy outlook model. 5 What are the components of the 6 model? The components are similar to what 7 they are now only they're regionalized and it 8 makes it a whole lot more complicated and I 9 have a slide in a minute to demonstrate just 10 how more complicated it is. We have the 11 electricity side that looks at demand prices, 12 low generation, and dispatch model. Phil's 13 going to be talking about this later today. 14 A natural gas component that's looking at 15 prices in storage, supply, and demand, a 16 petroleum component looking at heating oil, 17 gasoline, propane in a macro bridge. 18 This is a gross simplification of 19 what's going on. There's a tremendous amount 20 of detail for each one of these components. 21 Some are more regional than others, some are 22 more detailed than others, some go from a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 51 1 national level and allocate out to a regional 2 or state level, and some build back up. It's 3 not the case that every component in this 4 model is a bottom-up approach. There's a 5 mixture and a back and forth between national 6 and regional components. Not all the regions 7 are the same for each sector. 8 The degree of regionalization 9 varies considerably. We have significant 10 regional detail and data for some components 11 of this model but not for others. And so in 12 some cases we can get down to more regions 13 and in other cases we are constrained by our 14 ability to have the data necessary to support 15 the kind of forecasting we think is 16 appropriate in this case. So you will see 17 significant detail in the natural gas and 18 electricity side of the model, less so for 19 some petroleum products and for some end 20 uses. 21 So there is a varying degree of 22 regionalization. I think that's critical to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 52 1 remember that not all regions will get the 2 same level of detail that they may be 3 expecting by going to a regional model. It 4 will vary tremendously based on the data 5 requirements and the data needs and the data 6 quality that we have for the different end 7 uses and for the different markets. 8 This is a little graphic that shows 9 you that in the STIFS model we're dealing 10 with about 1300 variables and we add up the 11 variables of what we got to keep track of. 12 This is not just the variables that are being 13 forecast. These are the variables and the 14 data that we got to keep track of within the 15 model to a model that's looking at about 16 15,000, almost 16,000, variables. The level 17 of complexity has increased many, many times 18 over. 19 So it's a much more complicated 20 model and the data requirements are huge and 21 a lot of the issues that we deal with are 22 really on the data side as opposed to on the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 53 1 model side making sure that the data that we 2 have go through the appropriate diagnostics 3 that are clean, that are in the shape that we 4 want them at the time we need them, and these 5 are all issues that we have to grapple with 6 not only within our office but we are 7 dependent upon others within EIA to get the 8 data that we need at the time that we need it 9 to make sure that they make it into the 10 monthly model. So there's a lot of 11 logistical problems that show up in moving 12 from a more simplified national model to this 13 much more complicated regional model much 14 less some of the complexity of the modeling 15 particularly on the electricity side. 16 What do we do when we go through 17 the usual procedures when we're building the 18 separate components of the model to judge the 19 accuracy and the capability of the model to 20 predict? We use generally accepted criteria, 21 we evaluate data obviously for measurement 22 error, we look at overall fits, we look for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 54 1 independence of residuals, all the classic 2 things, the good practices that you do when 3 building econometric or time series models, 4 to check to make sure that they're tracking 5 in sample forecasts and that they are 6 accurately predicting the future. Once you 7 put it all together you need to run the kind 8 of tests for the national and regional levels 9 as well as to look at the forecast as the 10 model interacts with other components and I 11 want to talk a bit more about some of those 12 complexities in the second half of my 13 discussion today. 14 Clearly there are lots of 15 challenges with upgrading the model and 16 moving to a much more complicated system. As 17 I've said three or four times already, the 18 electricity side is really the most 19 complicated, the most detailed information 20 that is available, and the most complicated 21 type of modeling that we do is in the 22 electricity sector of trying to determine low BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 55 1 generation and dispatch and regional demands 2 and regional supplies and thankfully we have 3 Phil and several others that are leading us 4 through this. 5 Another problem is avoiding 6 aggregation errors. When we're adding up 7 from a lot of different pieces there are 8 problems associated with whether what you get 9 at the top level doesn't contain all the 10 biases that you might be embedding in each of 11 the individual systems or the individual data 12 and equations. 13 Maintaining model and data quality 14 is another big issue. When we're talking 15 about as many variables as we are it takes a 16 considerable amount of resources to check for 17 measurement errors and data quality, on a 18 monthly basis and to try and build into this 19 system diagnostics to make sure that you're 20 not making gross errors in data is a key part 21 of this. A single person or a couple of 22 people can't eyeball all of this data to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 56 1 determine where there may be outliers or 2 missing data and trying to come up with 3 systems that can do this electronically is 4 certainly something we do on individual bases 5 but to put it together in a systematic way so 6 that we have the confidence that we're not 7 making data errors is absolutely critical. 8 I talked earlier about augmenting 9 the model through time and I think this is a 10 challenge that we will have as we move 11 forward and interact with our customers and 12 with analysts about how to improve the model 13 as we get better information on how people 14 are using it and getting the data in a more 15 timely fashion I think will change the way we 16 do the model over time. 17 Developing publicly accessible 18 outputs and products, this is a big deal for 19 us at EIA because we pride ourselves on our 20 transparency and our ability to get 21 information out to folks about how we do 22 stuff. We want people to open us up and take BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 57 1 a look inside and we need to develop an 2 appropriate approach to how we're going to 3 make what is now going to be a whole lot more 4 information available to the public and to do 5 it in a way that is digestible. And so 6 building on the website that we have to make 7 the documentation available to clearly allow 8 people to chart through what will be a lot 9 more data is going to be an important part of 10 communicating with our customers and getting 11 feedback from our customers about how well 12 this new system is working for them will be 13 an important part of this feature too. 14 So I hope to able to come back at 15 some time in the future and talk about how 16 we're going to ask people in a more formal 17 basis how are we doing, how's it doing, how's 18 the model doing, and is it doing what you 19 want it to do, and is it as good as what we 20 were doing before when we just had the 21 national model so assessment will be a big 22 part of our going forward program. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 58 1 Obviously we need to develop 2 appropriate diagnostic and model evaluation 3 tools. I alluded to this in talking about 4 model and data quality but model evaluation 5 tools are also critical and it's really the 6 issue that I want to get to in my next talk 7 about best practices for evaluating forecasts 8 in general and in essence to solicit some 9 advice on what we ought to be doing to 10 develop a program in EIA certainly within the 11 regional model that helps us gauge forecast 12 accuracy and do a report card on ourselves as 13 well as be able to compare ourselves with 14 other forecasters to get a sense of how we 15 might improve the model based on some 16 diagnostics and some forecast accuracy tools 17 that really ought to be a part of our core 18 program. 19 So I would like to end this 20 presentation and maybe answer a few questions 21 or go into the next one. What do you want me 22 to do? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 59 1 MR. HENGARTER: That's the nice 2 thing of having to talk separate in two 3 pieces. It allows the public and the members 4 of the committee to ask questions in the 5 middle without interrupting your flow. 6 MS. ANDERSON: Great. 7 MR. HENGARTER: So if there are any 8 questions it's a good time. Yes? 9 DR. NEERCHAL: I'm sure the data 10 quality and the availability vary from region 11 to region quite a bit. Was that also part of 12 the consideration when you decided the degree 13 of regionalization? 14 MS. ANDERSON: Sure, particularly 15 availability, yes, it was, and I don't know 16 what more to say about that but certainly the 17 data quality and the data availability have a 18 lot to do with whether you're breaking it 19 down by PADD regions or whether you can get 20 down to some state level data for some 21 components of the model. And we hope that 22 supply might create its own demand or that by BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 60 1 having the model and by making the model 2 available and by people saying that it could 3 be a useful tool for predicting at a regional 4 level might help upgrade the quality of some 5 of the data or expand some of the data that 6 some people are collecting whether it's us or 7 whether it's others. 8 DR. NEERCHAL: So how much worse 9 does it get when you go from this national to 10 regional? 11 MS. ANDERSON: Well, depending on 12 the sector you're looking at it can get real 13 bad. I mean, in many ways we're aggregating 14 up from not very good data, aggregation 15 problems. In other cases we're just using a 16 much more aggregated number and parsing out 17 by what we know about demand or supply in 18 individual regions. We don't really have any 19 regional richness but we may have to infer in 20 some areas based on historical data. 21 MR. HENGARTER: Jae? 22 MR. EDMONDS: Margot, can you just BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 61 1 give us a sense of the big numbers that are 2 going to be coming out and that are going to 3 be the focus of attention that could come out 4 of the short-term model and what the key 5 numbers going in to the model are just to get 6 a sense of the kinds of problems you have to 7 deal with? 8 MS. ANDERSON: Well, in terms of 9 output I think we're going to focus on the 10 kinds of things we do now in the national 11 model, prices, demands, supply. I mean, 12 right now what we report out in monthly are a 13 long conversation about what we think the 14 prices are going to be for natural gas, 15 petroleum products, electricity, coal, and we 16 also have a discussion on the demands for 17 each, somewhat less discussion on supply for 18 each of those fuels although we usually have 19 in a paragraph or two about world oil 20 balances because that seems to be a primary 21 focus. 22 In some ways your question raises a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 62 1 question for me about whether the shift in 2 the emphasis will change when we go from 3 national to regional. Right now our focus is 4 really on oil markets and has been for some 5 time lately because of all the increases but 6 it may be that customers are going to want to 7 rebalance our emphasis in a sense of looking 8 at spending more time looking at regional 9 breakouts of prices as opposed to a discourse 10 on the national level. 11 So the outputs will be generally 12 the same. They'll just be more detail on 13 them and so we would be able to look at 14 gasoline or propane prices by our regions and 15 be able to let customers know that if you're 16 living on the west coast your gas price path 17 is likely to look different than if you're 18 living in the southeast. 19 In terms of the input to the model 20 the numbers are similar. The information 21 that you need to predict it at a regional 22 level is similar to what you need at the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 63 1 national level. There's just a whole lot 2 more of them and you may have to fill in 3 where there are some gaps of missing data at 4 the more regional or state level. Does that 5 get at what you're asking? 6 MR. EDMONDS: Actually I was just 7 thinking if you could enumerate the key 8 drivers to the short term forecast. You've 9 got some big benchmarking drivers like you're 10 macro-economic variables coming in but then 11 what are the other things that you're going 12 to be looking for on the outside to drive the 13 development of price forecast? 14 MS. ANDERSON: Mark, you want to 15 answer? 16 MR. RODEKOHR: Yes, there are 17 several factors. One is weather. Clearly if 18 we have a national weather forecast that it's 19 going to be a normal winter -- 20 MR. HENGARTER: Mark, come to the 21 table. 22 MS. ANDERSON: If you volunteer you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 64 1 got to use the mic. 2 MR. RODEKOHR: There are several 3 factors and the weather is one of them. For 4 example, you can have a national weather 5 forecast that says it will be normal for the 6 winter; however, that masks regional 7 differences, colder in the northeast, 8 whatever the case may be. So that can and 9 has made an impact on average prices for the 10 country and certainly prices in regions and 11 we've known this for years now. That's one 12 factor. 13 Another factor is regional 14 differences in prices with related things 15 like gasoline. Refinery outages here and 16 there can make a difference. We're not 17 tracking every refinery outage for every week 18 but nevertheless you can have some 19 differences. 20 In terms of electricity it's 21 transmission constraints, it's distribution 22 constraints. For example, you have first run BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 65 1 plants. We've disaggregated our electricity 2 dispatching into four categories roughly. 3 First of all you're going to use all your 4 hydro. You're going to use all your 5 renewables, which is next to nothing. 6 First the real issue on electricity 7 dispatching is what you do with coal, oil, 8 and natural gas and those depend on relative 9 prices and a lot of other factors and we're 10 trying to get a handle on that. And that's 11 probably the biggest cost part of the 12 regional STEO is trying to get handle on 13 that. We're also focusing on four states, 14 Texas, California, New York, and -- what's 15 the other one? 16 MR. TSENG: Florida. 17 MR. RODEKOHR: Florida, very 18 helpful, thank you, and Phil's going to tell 19 you all about this this afternoon. 20 MS. ANDERSON: Mark, talk a little 21 bit about how we disaggregate the macro 22 component that we're using at the national BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 66 1 level to look at the regional level so we can 2 perhaps try to break down the -- 3 MR. RODEKOHR: Well, I don't know a 4 whole lot about that but we are trying to 5 look at regional macro components as well. 6 Clearly some of the industries are obviously 7 affected differently by higher energy prices 8 in other industries and those are very 9 regional in nature and we are trying to get a 10 better handle on that as well. As Margot 11 said, whether this leads to a better national 12 forecast is one issue. 13 The other issue is can this lead to 14 a better forecast in terms of what policy 15 makers are worried about and that's one of 16 the key considerations that we're after 17 policy. Many policy people are very worried 18 about what happens in their particular region 19 or part of the country and that's what we're 20 trying to address with this effort. That's 21 part of what we're trying to address. 22 I personally have no view if we're BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 67 1 going to have a better national forecast but 2 clearly when you look at the press, when you 3 look at the policy issues, they're not all 4 just national in order. Margot, do you want 5 to -- 6 MS. ANDERSON: Are there other 7 questions? John? 8 MR. WOOD: Do I have to come to the 9 mic? 10 MR. RODEKOHR: The mic's over 11 there, by the way. 12 MR. WOOD: I would just mention 13 that a component that we've looked out a lot 14 in the last five years, natural gas, detail 15 of the quality, how well you have to know 16 some of the inputs to have a good chance at 17 really capturing natural gas prices, for 18 example, is amazingly what, tight. It looked 19 like just a couple of percent difference 20 between natural gas supply and demand moved 21 the natural gas price from $9 an MCF to 2 in 22 the 2001-'2 period and you can get a feel for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 68 1 what moves it and then where the natural gas 2 supply might be on a regional basis would 3 have given you a lot of insight. During that 4 period of time there was plenty of natural 5 gas supply in the Rocky Mountains but there 6 were shortfalls in the Gulf of Mexico with 7 the main distribution system is so I think 8 that idea of the regionalization gives you a 9 lot better insight into how ———————————— 10 MS. ANDERSON: Well, I hope so. 11 It's a be careful what you ask for kind of a 12 thing. We're going to have a lot more to say 13 about what's going on in the country and the 14 pressure to be as accurate and timely as 15 possible has increased considerably so 16 there's just a lot more to look at and I 17 think there are lot more sensitivities when 18 you are looking at it on a regional or state 19 basis than there would at a national level. 20 So it's going to be a very interesting year. 21 MR. HENGARTER: Last question, 22 Mark? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 69 1 MR. BERNSTEIN: More concerns and 2 observations and I know you realize I'm 3 really concerned about that 16,000 number 4 down there. I'm sure we'll get into it with 5 Phil because the bulk of that is the 12,000 6 dispatch number but that's a real concern on 7 a monthly basis to be able to keep track of 8 things and make sure they're consistent. I 9 know you realize that but I just needed to 10 say it. 11 I do believe you're in the right 12 direction on disaggregating regionally. 13 We're doing some work now on regional 14 disaggregation of price elasticities and we 15 see significant differences between regions. 16 So it will clearly affect forecast but the 17 other observation I want to make is more of a 18 broader policy issue in EIA is the problem of 19 forecasting versus assessing the risks of 20 problems. 21 What I would want to get out of a 22 regional short-term energy outlook are things BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 70 1 like what's the risk in southern California 2 of having blackouts this summer, assessing 3 the risks next winter in the northeast of 4 having significantly higher heating oil 5 prices, things like that as opposed to a 6 forecast which no matter how good we are at 7 forecasting we know there are problems but 8 more in a sense of what are the conditions 9 likely to lead to problems so that we can 10 have red flag signals. 11 It's the same issue I have with the 12 annual energy outlook too but I think it's 13 more relevant for the short term energy 14 outlook and so I'd like you to be thinking 15 about and considering a different approach to 16 presenting the information and evaluating 17 options and I think there are hot point 18 things that are problems and there are 19 regions where there aren't any problems and 20 you can change your efforts to emphasize 21 things where you notice there are going to be 22 problems versus places that are not. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 71 1 MS. ANDERSON: Can I comment on 2 that? 3 MR. HENGARTER: Yes. 4 MS. ANDERSON: I think that's a 5 great point. We certainly tend to do more 6 risk assessment on the international oil 7 side. We do choke point analysis, et cetera. 8 We do at DOE we have an office of energy 9 assurance, energy transmission and assurance 10 I think it just changed it's name to, that 11 was an office that came out of the first term 12 of the Bush administration to assess 13 vulnerabilities to the energy system and 14 certainly they do some of that work. Whether 15 EIA can be of assistance in using a model 16 like this to identify where there might be 17 these kinds of constraints certainly would be 18 worth looking into. 19 MR. BERNSTEIN: They don't put out 20 stuff that -- 21 MS. ANDERSON: They won't put it 22 out and they may not put it out for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 72 1 sensitivity reasons or for policy reasons 2 because they may be assessing other kinds of 3 threats as well but you're right. I don't 4 think you'd find a published place where you 5 could put a probability although the policy 6 office used to do some of that as well of 7 trying to assess where there may be working 8 with NERC to look at the summer electricity 9 outlook but you're right. There's no 10 systematic way of doing that and it's not 11 model based. So the points well taken that 12 models may be able to be used for that and 13 then what is the obligation of an EIA of 14 putting that kind of information, making it 15 that available. It's a good point. 16 MR. BERNSTEIN: The key is that 17 just there's nobody out there responsible for 18 noting that there may be problems out in the 19 future. The only place that actually has 20 that information is EIA. 21 MS. ANDERSON: What about NERC on 22 the electricity side? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 73 1 MR. BERNSTEIN: NERC's stuff is 2 pretty poor. My one anecdote example is in 3 2000-2001 time frame in California when NERC 4 said that there are going to be no problems 5 in California in the next six months and then 6 all hell broke loose. 7 MS. ANDERSON: Well, that leads us 8 to our next discussion about how do you 9 forecast well. 10 MR. CLEVELAND: Just a brief 11 comment to follow up on Mark's, I think that 12 once you demonstrate this capability to do 13 this regional type of work you will be given 14 this kind of work because in essence the 15 special requests you get are often regionally 16 driven, many of them, and so now all of a 17 sudden if you are going to be able to say 18 things specifically about the California 19 electricity market or the New England heat 20 oil market you're going to get requests from 21 people from those regions to do this type of 22 work. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 74 1 I think Mark's point is a good one 2 that should you be doing it pro-actively I 3 think is an important issue to discuss but 4 you'll be doing a lot more of that type of 5 work and you're going to be under even more 6 scrutiny because you're going to be saying 7 things that are with a lot more specificity 8 and as soon as you start saying things with a 9 lot more specificity your models or your 10 numbers are going to be scrutinized even more 11 closely so be ready. 12 MS. ANDERSON: Dave, Tanc, anything 13 you want to add to that? Mark? 14 MR. RODEKOHR: I'd like to add a 15 little bit to that. Two things I think 16 you've said, I think Mark has made a very 17 good point in the sense that we could add to 18 our STEO I won't say a model-based assessment 19 of risk but where we think things could 20 potentially be problems and I see no problem 21 with that at all because we know what it is. 22 Is it based on a model? Maybe yes, maybe no. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 75 1 Setting expectations for the 2 regional model, I think, is very important 3 and the things you just said are the things 4 that we're going to need most. We do not 5 have transmission link models embedded in 6 this. We have a regional forecast but we 7 don't have all the transmission links. 8 And we've already, trust me, been 9 getting many questions about if pipeline goes 10 down for X number of days what's going to 11 happen to the world and we keep trying to 12 answer these questions but we don't have a 13 modeling technique to do it. So I think we 14 need to set our expectations better but I 15 agree with you. A risk assessment is saying 16 in a summer or winter outlook what are the 17 problems we think could happen not based 18 necessarily on a modeling exercise because we 19 do not have the detail of a NEMS or another 20 huge model that looks at all the 21 transportation links. We never pretended 22 that we had this is not a bad idea at all. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 76 1 But it is more notional than it is 2 modeling based and the regional model has 3 never been built as something that will 4 capture all the transportation links and all 5 the things that certainly policy makers at 6 times are worried about. Let me give you a 7 few examples here. 8 Our projections are monthly. Often 9 times events happen that last a few days or a 10 week. We have no way of knowing those few 11 days or a week in a modeling exercise are 12 going to happen so we need to tailor our 13 product to what they need but nevertheless 14 never over-promise at the same time. But the 15 notion of a risk assessment, I think, at 16 certain times a year is not a bad idea at all 17 but we have not simply buried in certain 18 sentences or paragraphs some of that stuff. 19 MR. EDMONDS: A quick point of 20 clarification here, one of your slides when 21 you did the variable count you distinguished 22 between econometric and dispatch variables. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 77 1 I know what an econometric variable is but 2 what's a dispatch variable? 3 MS. ANDERSON: Phil, help me. 4 MR. TSENG: I'll spend ———————— 5 minutes with you. 6 MS. ANDERSON: And on the numbers 7 that 12,000 number may go down considerably 8 if we can find a way to streamline on this 9 dispatch model. So we throw that up to 10 demonstrate where we are now, having to track 11 all that. 12 MR. EDMONDS: So what it's saying 13 is that that's an electricity dispatch? 14 MS. ANDERSON: Yes, it is, 15 electricity -- 16 MR. EDMONDS: And basically it's 17 all coming in from outside? 18 MS. ANDERSON: Well, all coming in 19 on that one component of the model that's 20 dealing with trying to find the optimal way 21 that the plant just sends out the power at 22 what time. And we have all this plant data BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 78 1 and that's generating all these different 2 variables that can be used to assess who's 3 getting electricity when based on demands and 4 on prices. And so because we have so much 5 detail you've going to have plant by plant by 6 plant all these variables that are 7 contributing to that number but it is 8 electricity dispatch. 9 DR. HENGARTNER: I have the feeling 10 we'll be hearing a lot more of this. The 11 next topic -- 12 MS. ANDERSON: I'll talk really 13 fast. 14 DR. HENGARTNER: Yes, it's a 15 pleasure to continue Margot's exposition. 16 MS. ANDERSON: Thank you. I want 17 to talk about performance indicators, 18 diagnostics, and forecast errors. This is an 19 issue that I've been thinking about is, I 20 have come to EIA about what can we do to 21 improve our ability to assess the accuracy 22 and the quality of our forecasts. The caveat BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 79 1 here isn't to beat ourselves up if we're not 2 perfect every month. 3 The idea about doing this is just 4 to get a better sense of how we can improve 5 our own models by coming up with a program 6 that continually allows us to assess what 7 we're doing and what's the appropriate report 8 card that we ought to use. So I want to talk 9 a bit about past and current practice, 10 principles and tools for forecast evaluation, 11 and some things we're thinking about as we 12 consider designing a good practices program 13 for assessing our forecast. 14 Well, I've hinted at why evaluate. 15 I mean, to many this is an obvious thing to 16 do. We want to assess the accuracy of what 17 we're doing, we want to improve the data in 18 the models, we want to enhance accuracy and 19 credibility, we want to increase 20 transparency, we want to help demonstrate the 21 complexity of forecasting. 22 Unless you're constantly reviewing BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 80 1 what you're doing it's difficult to 2 communicate why you might not be doing it as 3 well as some people think you ought to be 4 doing it who are not in the game. Having 5 said all of this, it's not so easy just to 6 come up and say here are a couple of tools 7 and if you just do this every month you'll be 8 better off and you can tell people you're 20 9 percent, 30 percent, 2 percent off. 10 It's using the information to 11 evaluate what you're doing and it's not 12 looking at all 12,000, 1400, whatever it is, 13 number of variables. It's really designing a 14 program that looks at some key things that 15 can really help you better understand what 16 you're doing within the models and why or why 17 not you may be missing what the actual 18 outcome is and there may be obviously some 19 really good reasons that you're not going to 20 ever capture within a model. 21 But I think there is an impetus to 22 do this at varying levels. You can have a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 81 1 $20 program and you can have a $100 program 2 and it's deciding which program would be best 3 under this context. During the '80s and the 4 early '90s we routinely published an annual 5 supplement to what was then a quarterly 6 short-term outlook model and we put out a 7 series of forecast errors and looked at some 8 key variables and said here's how we're doing 9 and here's why it's different than what it 10 actually was and explained what might have 11 been going on in those markets and why we 12 missed it and talked a bit about what we 13 might do to improve models. And so there was 14 a good general discussion that came out every 15 year for people to take a look at. 16 That was discontinued in the early 17 '90s due to a variety of reasons, the biggest 18 of which was budget. Again, STEO was a 19 quarterly model. Accuracy, it was typically 20 absolute average errors and percent errors. 21 We didn't go any deeper to look at any 22 regression-based tests to determine what's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 82 1 going on and what's missing and what you 2 might get from additional information. It 3 was a fairly simple way of looking at what 4 did you predict, what was it actually, how 5 much were you off. 6 Now, currently several of the 7 programs, NEMS does publish forecast errors 8 with their annual report but we in STEO don't 9 publish our forecast errors. Each analyst 10 certainly keeps track of forecast errors for 11 their components and we keep track internally 12 of forecast errors for some key variables. 13 We don't have an externally 14 published document nor do we typically share 15 our forecast errors or our discussions about 16 forecast errors. We haven't done that some 17 time and that's one of the open questions 18 that we'd like some guidance on, on what 19 might be an ideal program to get back into 20 this game and what should we be doing. 21 I have some examples of what we 22 keep track of internally and, again, these BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 83 1 are just the percent differences between a 2 quarterly forecast, a quarterly average, 3 because the STEO is, of course, now monthly 4 and what the actual was. And I'm putting 5 these up here not so you could say they were 6 really off. I'm just putting these up here 7 to show you what we keep track of and we have 8 a series of about 12 or 14 of these graphs 9 that we keep track of and take a look at to 10 just tell ourselves are there any obvious 11 things that we're overlooking, are there any 12 trends going on here, but we don't do more 13 analysis on this. Again, we don't look at 14 correlation among the errors. We eyeball 15 this, we say where have we been under- 16 looking or overlooking, no surprise that in 17 areas like international oil prices were less 18 good than were on some of the demand 19 equations or on some of the demand forecasts 20 which tend to be more stable over time. 21 So I got a couple of them here. 22 Here it is for gasoline price. This goes BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 84 1 from '95 to 2004. You look at it and worry 2 whether there are any more systematic things 3 or gee, what happened back there in 2001 when 4 we were really off. The scales are different 5 for each graph. The percent is different for 6 each one. They're not comparable by eyeball. 7 Some are moving in a much tighter band, some 8 are in a much smaller band. Here is for 9 petroleum band where the band is really much 10 tighter. This is 3 percent error on either 11 side. So this is just an example of what we 12 have internally but you don't see this. 13 Should a forecasting program show 14 this to you every month? Should we show it 15 to you once a year? What should we show to 16 you? What should be the number of variables 17 that we look at? How deeply should we look 18 at them in what context are questions that we 19 have. 20 Obviously there are lots of 21 candidate metrics, some more appropriate for 22 time series, some more appropriate for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 85 1 econometric modeling, and there are any 2 numbers that you can pick from moving from 3 something fairly simple to something much 4 more complicated doing modeling on the errors 5 themselves to try and figure out what's going 6 on. 7 So there's a variety to choose 8 from, none of this new to you, and I want to 9 talk a little bit about basic principles, 10 about if you were going to design best 11 practices for forecast performance what would 12 you be worrying about. You'd be worrying 13 about unbiased forecast, you'd be worrying 14 about efficient forecast, and you'd be 15 worrying about uncorrelated errors. 16 What's interesting is we often talk 17 to other modelers who have a whole lot more 18 data than we do, access to maybe a whole lot 19 more data, particularly in the commercial 20 sector, and the question always comes up in 21 my mind if we had access to that data too, 22 which are often very expensive, and maybe BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 86 1 highly detailed data about production in 2 every field and every country would we be 3 able to have a better forecast if we had that 4 data. What's the value of getting that 5 additional information and is it worth EIA to 6 go out and get that information, purchase 7 that information from commercial sources, 8 because it will improve our forecast? 9 Certainly at first blush it sounds as if we 10 need more data, we need better data. Without 11 really doing an assessment of what that 12 better data might buy you it's difficult to 13 make that tradeoff. 14 So these are some of the things 15 that you'd want to be able to accomplish in a 16 forecast evaluation program. And if you were 17 designing a forecast evaluation program from 18 scratch what would you worry about? You'd 19 worry about which tests to apply. You can't 20 apply 20 tests. What's the smallest number 21 of tests that you could apply to give 22 yourself the confidence that you need to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 87 1 answer your core questions? How often should 2 you do this, every month with all those 3 numbers that are in the regional STEO? Is 4 once a year okay? Is every quarter okay? 5 Which forecast, just a one step ahead, or you 6 do an average over your year or over your 7 quarter or over two years? Which data do you 8 use, the data that you are using when you 9 make the forecast or your real data when you 10 go back and you adjust for all the numbers 11 that were preliminary when you made your 12 forecast? 13 There are two schools of thought on 14 which are the best data that you judge your 15 actual from your forecast number. Which 16 variables? We have hundreds of variables 17 that we will be predicting the outcome. 18 Which variables are we going to look at? 19 What are the most important? When we move 20 from the national to the regional model that 21 changes dramatically and there are going to 22 be a lot of interest for some regions and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 88 1 less interest for others. How do we pick in 2 an objective way which variables we're going 3 to track or that we're going to test? 4 How do you use the results? Again, 5 the results aren't necessarily just for a 6 report card to the public to say we're better 7 than somebody else. The results are to help 8 us do diagnostics and improve the model. My 9 first goal is really for ourselves to do 10 better and to have fairly simplified 11 electronic ways to improve what we're doing 12 particularly as we move from a national to a 13 regional model. 14 It's not to flagellate ourselves 15 and the public. It's a diagnostic tool. 16 Certainly my gut feeling is that our forecast 17 can stand up to anybody's forecast and that 18 we're typically in very good shape on a lot 19 of variables and I think that we've 20 demonstrated that over the past. It's not a 21 competition. It's about improving what we 22 do. So that's how to use the results. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 89 1 Should we compare ourselves with 2 commercial forecasters? We like to do that a 3 lot, that we were more accurate than X, Y, or 4 Z. Should we do that in a more formal basis? 5 Should that just be an internal thing? Why 6 would we want to do that? How will we use 7 their information to improve our forecasts? 8 And how do we communicate with 9 customers? Again, it goes back to the issue 10 I mentioned earlier. Is this something we 11 publish every year, is it something we put on 12 the web, do we have a discussion with our 13 customers, how do we communicate what this 14 program might look like to those that might 15 be interested in assessing what we're doing? 16 I'm a manager in EIA. I always got 17 to worry about the bottom line and who do we 18 have to do this and how we're going to get 19 this done under, as Guy said, declining real 20 budgets? My manager in me tells me that 21 these are the kinds of things that we need to 22 build into our programs when we build them BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 90 1 from scratch and we need to budget for them 2 from the very beginning. When we don't do 3 that that always is an add-on to what we're 4 currently doing. But I think that there's 5 tremendous value in having some kind of a 6 program even if it's rudimentary that is a 7 systematic way of looking at our forecast 8 accuracy and being able at least as a 9 management and a model building group to use 10 that information every so often, monthly, 11 quarterly, annually, to really steer us in a 12 way of making some changes in the right 13 direction on what we're doing on our models. 14 So I'll leave this up while we have 15 questions here but these are words to live by 16 about forecasting and the last one is the one 17 that I think is most valuable for us. It is 18 far better to foresee even without certainty 19 than not to foresee at all. It's really 20 critical that EIA has a solid forecasting 21 program. We know we're going to be off 22 periodically, as is everybody else, and if BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 91 1 you look back through the academic papers and 2 compare macroeconomic forecasts or energy 3 forecasting there have been a couple of 4 papers in the last several years on this 5 issue. There aren't a lot of people that get 6 it right very often particularly when you're 7 looking at the long-term models but we all 8 could improve by looking at what we're doing 9 and making sure that we are doing the best 10 job possible with the resources that we have 11 and I'm happy to open it up. We're almost on 12 time, too. 13 DR. HENGARTNER: We are on time and 14 so every question you ask is going to eat 15 into your break time. 16 MS. ANDERSON: Forwarned. 17 DR. HENGARTNER: So Darius? 18 MR. SINGPURWALLA: Nothing, I just 19 have a general question actually. Who are 20 the main customers for EIA? 21 MS. ANDERSON: For the short-term 22 energy outlook or for EIA in general? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 92 1 MR. SINGPURWALLA: For EIA in 2 general. 3 MS. ANDERSON: Oh, gosh. I mean, 4 it really is very broad, certainly industry, 5 other government officials, academics, the 6 public schools -- 7 MR. SINGPURWALLA: And they just 8 come directly to you with like, specific data 9 requests or reports and stuff or -- 10 MS. ANDERSON: Sure, some of them 11 are on list serves to get products every 12 month. Some of them are searchers and they 13 come into the main web page and search around 14 efficiently to find what they need. 15 Sometimes you get specific requests from a 16 staff person over at the White House who 17 wants to know something. It is highly varied 18 about how people are using our stuff. 19 MR. RODEKOHR: In addition to that 20 it's about 15 or 20 percent international on 21 the website but a tremendous amount of our 22 stuff is things that are difficult to count BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 93 1 and those include the press. Especially with 2 the short term energy outlook at certain 3 times in the year we will get a lot of 4 national press. That's not counted but 5 there's still millions of viewers. 6 So it's all over the map and so 7 it's a topic that —————— and other people in 8 the EIA spend a lot of time trying to work on 9 and identify. It's not easy to come up with 10 a very clear answer. Web statistics are 11 easy. That's a walk in the park. I can give 12 you all those, more than you ever care to 13 see, but the other stuff is less easy. 14 MS. ANDERSON: And clearly it 15 always makes it difficult to hone your 16 product line because it's not like your 17 Procter & Gamble. You have so many different 18 kinds of customers with varying weights and 19 some products that are costly may have a very 20 limited but very vocal audience. And because 21 our bottom line isn't how much profit we're 22 making of off each product it means a lot of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 94 1 products you produce regardless of their 2 cost-effectiveness. That's not the way we 3 would determine whether a product remains in 4 the family of outputs. I probably over 5 answered that but -- 6 MR. EDMONDS: Just quickly, I would 7 think one of your main customers that didn't 8 get mentioned would be the Hill. 9 MS. ANDERSON: Oh, Congress, of 10 course, yes, of course. 11 MR. EDMONDS: I would think they 12 would be extremely high up on the list. 13 MS. ANDERSON: My apologies, yes, 14 of course. 15 MR. CLEVELAND: Especially when 16 you're wrong about their home district. 17 MS. ANDERSON: Well, yes, there's 18 that. 19 DR. HENGARTNER: Neha? 20 MS. KHANNA: You threw up a lot of 21 questions when you talked about designing 22 best practices but it seems that you could BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 95 1 very easily filter down, let's say, from your 2 3,000 variables or 3500 different kinds of 3 tests to maybe a basketful. That seems to be 4 at least the first step. I mean, what are 5 the key variables that have multiple error 6 effects in your forecasts? What are the key 7 variables that people typically ask you? It 8 seems that that filtering should be done 9 first before we even start addressing some of 10 these questions. Is that something you've 11 already done internally? 12 MS. ANDERSON: Well, to the extent 13 of the way we keep track of the variables. 14 Now, we have made an initial triage from the 15 national model and it's about 14 or 15 key 16 variables, gasoline price, petroleum prices, 17 demands, coal demand, coal price, natural 18 gas, so we have done that for the program 19 that we're using now which is looking at this 20 absolute percent of changes. But you're 21 right and when you go to a regional model you 22 may come up with 25 or 50 things that you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 96 1 want to track. 2 MS. KHANNA: Or it could be 3 regional specific? 4 MS. ANDERSON: Right, and you 5 could, I think, set up some automated system 6 that would peel off those variables and put 7 them in a file as is done now and run the 8 tests that you need to. But yes, triage on a 9 regional model is a bit more complicated but 10 probably not that much more complicated. 11 DR. HENGARTNER: I will ask just 12 one question, don't expect an answer, but 13 just bring it out to the committee so that we 14 can think about it. One of the issues that I 15 see with this modeling you're doing this we 16 have 16,000 variables. There's this risk of 17 over-fitting and when you talk about looking 18 at model or prediction errors I'm always 19 asking myself well, which part is prediction 20 error and which one is model error and which 21 one comes from the over-fitting. And so it 22 seems that that's a concern for a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 97 1 statistician like myself. 2 I'm not expecting an answer right 3 now but I'm thinking that that's something 4 that we should keep in the back of our mind 5 as we continue discussing this topic because 6 we already alluded to it previously and I 7 think this is going to come back and I just 8 want to be opened. That's I think one of the 9 real challenges as in this kind of work and I 10 don't have an answer yet. So thank you very 11 much. 12 MS. ANDERSON: Thank you very much. 13 DR. HENGARTNER: We have a 15- 14 minute break. There's going to be break-out 15 session after the break so the Census Bureau 16 is going to be on the fifth floor, and Shawna 17 and Bill will be your guides to the break-out 18 session room on the 5th floor. The names are 19 on the programs for those who need to know. 20 (Recess) 21 MR. TSENG: Today I'll talk about 22 this electricity model. Margot gave a very BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 98 1 good overview of the regional short-term 2 model and I'll get into a little more detail 3 about the electricity model. I think some of 4 the issues you raised earlier I'll try to 5 address. And before I start I'd like to 6 thank the Office of the Energy Markets and 7 End Use to keep this presentation. This 8 project is actually EMEU's project and I 9 worked very closely with Steve and a few 10 other people at EMEU and so I'm just standing 11 here reporting and I don't want to take all 12 the credit. 13 MR. COSTELLO: Or blame. 14 MS. ANDERSON: And/or blame. 15 MR. TSENG: Well, the purpose of 16 the model and model structure, model 17 calibration, data sources, performance and 18 modeling questions ———————— other topics I'll 19 be talking about but my focus will be about 20 model calibration and data sources and data 21 issues and as a modeler we all know if we 22 don't have good data then model results are BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 99 1 meaningless. Earlier in the morning I heard 2 some questions about what's the purpose of 3 the model. Can we use the model to do some 4 useful risk analysis? And I think one major 5 objective of this regional model is to 6 provide a link to the new integrated regional 7 energy system in answering region-specific 8 questions such as the summer electricity 9 market, natural gas market, and the winter 10 heating fuel market. 11 Mark raised the question in 12 California what kind of questions or answers 13 or warnings can this kind of model provide. 14 For example, when we look at summer 15 electricity market because we have some 16 regions we can actually analyze some of the 17 impact in a region and see what happens to 18 the fuel market and even generation 19 potentials. A few weeks ago I heard the 20 governor actually declared Washington State a 21 drought state. And what happened is in the 22 western region if Washington doesn't have BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 100 1 enough hydropower California in the summer 2 may have some different issues in terms of 3 dispatching and importing of electricity. 4 And we can actually use the model to simulate 5 if there's less hydropower from Washington 6 State what happens to the dispatching 7 generation in California and what happens to 8 the fuel use. 9 For example, California has many, 10 many natural gas power plants and so how much 11 more natural gas power plants will be run and 12 how much additional natural gas will be 13 consumed? And that can have an impact on the 14 regional natural gas market as well as 15 national natural gas consumption and 16 eventually natural gas prices. So that's the 17 kind of thing we would like to see the model 18 actually provide some linkage so we can track 19 what happens. 20 And in terms of the winter heating 21 fuel market I think in the northeast the 22 model also can provide some information on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 101 1 when we have above normal heating degree-days 2 or cooling degree-days what happens to the 3 natural gas consumption month by month and 4 what happens to natural gas storage. We can 5 do some accounting there and as a result what 6 happens to natural gas prices, and so you can 7 actually provide some insight. Of course, 8 the heating fuel market also relates to the 9 gasoline market and diesel fuel production 10 and some other issues but natural gas plays a 11 role there. So that's the purpose of the 12 model, what can a model do and how people 13 want to use it really depends on the modeler 14 as well as to potential users or 15 stakeholders. 16 The next few slides I will talk 17 about model structure. I think earlier 18 people talked about this model is very 19 complicated and there are so many exogenous 20 variables but in reality the concept of the 21 model is quite straightforward. It's 22 important to understand the concept. The BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 102 1 implementation is a different issue. 2 Especially I'm not the one doing it but Dave 3 is more concerned about a lot of the detail 4 because he has to do it every month. 5 The regional electricity model has 6 13 regions and we group them into three trade 7 zones like eastern interconnection, western 8 interconnection, and Texas. Each region has 9 its own demand and supply representations. 10 We have a 14th region, Hawaii and Alaska. We 11 don't really model it. We treat it 12 exogenously because it doesn't play a role in 13 the lower 48 states but for accounting 14 purposes we need to know how much electricity 15 generated, what kind of fuel those regions 16 consume. 17 So this is a map to show you the 13 18 regions. The western region is the one I 19 will spend more time. Basically western 20 region is mountain region. We have Pacific 21 region, Washington, Oregon State, and then 22 California so we will break up the Pacific BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 103 1 region into two parts and I will use the 2 western interconnection as one example to 3 illustrate some of the modeling issues and 4 data issues and some of the preliminary 5 result I just got yesterday. But we have 6 basically 13 regions from the map and then 7 three separate prisms and the eastern 8 interconnection has nine regions. We have 9 Texas as separate. 10 For each region the way we solve it 11 I'm talking about the concept. I'm not 12 talking about the number of variables we need 13 to solve it, not yet. For each region an 14 average daily 24-hour load curve is derived 15 from monthly demand. And so we have monthly 16 demand and then we convert it to load based 17 on historical load data and temperature data 18 and that's on the demand side. 19 And a supply curve for each region 20 is created from the EIA survey form, the EIA 21 860, and we have the capacity and heat rates 22 and a number of other plant characteristics. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 104 1 So the model basically searches for a 2 minimum-cost supply which meets hourly 3 demand. So we have 24-hour load, and for 4 each hour we have the supply curve for each 5 region and then the demand we trace to the 6 supply curve and find the point where supply 7 equals demand and that's the supply. And 8 then from there we can estimate the 9 generation by fuel type and fuel consumption. 10 So this is the general concept of 11 the model. It's pretty straightforward. We 12 have regions and within each regions we have 13 demand and supply and I will talk about the 14 creation or the data we use to create the 15 demand and supply. 16 But the first step when we do model 17 calibration is really the concept again is 18 for a single region we need to make sure for 19 model calibration the historical regional or 20 for the trade zone the total demand plus 21 transmission and distribution losses minus 22 net import must be equal to the historical BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 105 1 generation because the objective of this 2 model is really seeing given the demand and 3 supply, assuming there is no trade, there are 4 T&D losses, then demand must be equal to 5 supply in general. That's the basic concept. 6 For a single region that's the situation so 7 there is a lot of data problems we have to 8 deal with but we have found ways to handle 9 the data issues and so basically the general 10 concept is for a single region demand must be 11 equal to generation and for the trade zone 12 such as the western interconnection simply 13 the sum of the demand plus T&D losses minus 14 imports must be equal to the regional 15 generation. 16 DR. HENGARTNER: Why don't they see 17 exports? I mean, if you have imports why 18 don't -- 19 MS. KHANNA: Net imports. 20 MR. TSENG: The import can be from 21 outside the region. 22 DR. HENGARTNER: Oh, so imported BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 106 1 includes exports? 2 MR. TSENG: Yes, it's net imports. 3 This part looks very simple but if we don't 4 get it right the model will not replicate the 5 historical generation pattern. So this is 6 one example I look at. For a single region 7 13 is the Texas region and we have the fuel 8 mix. For example, we have residual fuel, we 9 have diesel fuel, we have natural gas, coal, 10 nuclear, hydro, and combined heat and power. 11 But here the focus is really the fossil fuel. 12 The dispatching model is really modeling the 13 dispatching of residual fuel, diesel fuel, 14 natural gas, and coal power plants and for 15 nuclear, hydro, and combined heat and power. 16 For nuclear and hydro they are base 17 loads so most likely they run all the time 18 and they have the priority. And for the 19 combined heat and power the economics is such 20 that they are very different from the 21 electric utilities so we don't really model 22 them. For the purpose of model calibration BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 107 1 we simply subtract them from the demand and 2 then we try to match the generation and focus 3 on the generation pattern of the fossil 4 plant. Yes? 5 MR. WOOD: You might note the 6 largest number I see on there is for August, 7 I assume, month eight? 8 MR. TSENG: Right. 9 MR. WOOD: If you were in Boston or 10 the New England area do you have a 11 comparison? 12 MR. TSENG: No, I have the numbers 13 but I don't have it with me. 14 MR. WOOD: One would presume it 15 might be in January or February, not August? 16 MR. TSENG: Well, actually I look 17 at New York. In the summer they say July is 18 the highest for the -- 19 MR. CLEVELAND: It's still the 20 summer peak but the winter peak is catching 21 up, though? 22 MR. TSENG: Yes, right, but there BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 108 1 are winter peaks. I think New England states 2 my recollection is they are probably double 3 peaks. They have early in the morning, like 4 4:00-5:00 o'clock you have a peak and then 5 you have afternoon peak. And so the shape of 6 the load is like two separate curves. You go 7 up, down, and then go up again in the 8 afternoon for the New England regions in the 9 winter. Otherwise most likely it's more like 10 a bell-shaped curve for most regions. 11 DR. HENGARTNER: Can you explain to 12 me ——————— you said was a baseline and they 13 always work. Why is there so much variation 14 in it? I mean, there is a drop of over half 15 in month 11? 16 MR. TSENG: Well, these are 17 historical data. Why it's high or low, I 18 think it's based on regional demand patterns 19 because in Texas really the purpose of 20 generation is just to meet the demand. 21 MR. BERNSTEIN: Phil, the question 22 has specifically to do with the nuclear. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 109 1 Months 10-11, must be a plant went offline. 2 MR. TSENG: Oh, yes. 3 DR. HENGARTNER: That means that 4 something must have happened because the -- 5 MR. TSENG: See, a plant outage, I 6 think some nuclear power plants may be shut 7 down for maintenance. 8 MR. WOOD: And they schedule them 9 in a lighter electrical generation period. 10 September is a great month in Texas. It's 11 not hot and it's not cold. 12 MR. COSTELLO: We have forecasts by 13 region which take account of outages on a 14 monthly basis. In other words we don't do 15 the nuclear forecast. Actually the Coal, 16 Nuclear, and Electric Group does that for us 17 but they do it plant by plant and they have 18 those scheduled things. This might not have 19 been a scheduled one there. We know on a 20 given month basis at least EIA's best guess 21 as to when there might tend to be operating 22 less so that is taken into account. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 110 1 DR. HENGARTNER: In other words the 2 nuke just makes power, period —————— the same 3 power they produce so there shouldn't be much 4 fluctuation. 5 MR. COSTELLO: And there isn't 6 normally. 7 DR. HENGARTNER: There isn't 8 normally except that I see it there so that 9 brings an interesting question about knowing 10 maintenance schedules and things like that 11 which just complicates your model but if that 12 is something you are interested we need to 13 take that into account. 14 MR. TSENG: For nuclear that's how 15 we treat it, as exogenous, and we talk to 16 experts. And for short-term forecasting the 17 plant outage information is available. 18 MS. ANITI: I just have one 19 question. I was just wondering why you don't 20 include wind. Is that not -- 21 MR. BERNSTEIN: Wind? 22 MR. TSENG: We actually include -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 111 1 MS. ANITI: We knew this was in 2 general but I thought wind is pretty high in 3 Texas. 4 MR. TSENG: Wind is pretty high in 5 California. Actually originally when we 6 model the power generation dispatching we had 7 everything and we had even solar. But when 8 you look at the numbers and in terms of the 9 level of effort to capture renewable it gets 10 to be very difficult. And also for renewable 11 such as wind and solar the dispatching is not 12 economics. Dispatching is basically weather- 13 related and the purpose of dispatching 14 algorithm is really trying to capture the 15 economics of dispatching of fossil fuel. And 16 if it's not economics we lump them into a 17 group and we then use time series analysis 18 and treat it as exogenous in the dispatching 19 model. 20 MS. ANITI: So you don't find the 21 air that's very large for ———————— 22 MR. TSENG: Well, actually wind BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 112 1 dispatching varies. In California the 2 utilization of the wind capacity is about 10 3 to 15 percent in December and January and it 4 can go up to about more than 50 percent in 5 May and June so it varies. So time series 6 approach is better than economics approach 7 because the economic model doesn't predict 8 the weather pattern and so the wind 9 generation cannot be captured using economic 10 model. 11 MR. BERNSTEIN: So if I get it 12 right what you are saying is that the other 13 generation forms are being brought 14 exogenously some place else outside the 15 dispatch? 16 MR. TSENG: Right, outside the 17 dispatch model. That's why we have so many 18 exogenous variables listed because we do need 19 a different approach for the dispatching 20 model. Now, for a region like the western 21 interconnection now we are talking about 22 trade zone so then we are looking at BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 113 1 estimated demand and then generation. So you 2 can see for each region the demand level is 3 different from the generation level. Like 4 Region 10 is California. The consumption 5 level or the estimated demand is a lot higher 6 than its own generation. The generation 7 number is historical. So you can see 8 California imports electricity from Regions 8 9 and 9. 10 So just to give you the flavor what 11 the model is trying to do, it's basically 12 trying to replicate the historical pattern, 13 not only the fuel share but also the 14 generation share. 15 DR. HENGARTNER: Like when you say 16 California imports from Regions 8 and 9 but 17 there is only one region bordering California 18 which is the Mountain Region. 19 MR. TSENG: The Mountain Region and 20 the Pacific with Oregon and Washington State. 21 DR. HENGARTNER: But that's the 22 same region. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 114 1 MR. TSENG: Oh, no, we separate 2 California out. 3 DR. HENGARTNER: I'm sorry. I'm 4 looking at the maps and trying to just 5 understand what you're saying. 6 MR. TSENG: So what we do, once we 7 have the general dispatching algorithm is 8 basically is cost based and then we actually 9 started running the model and we have got 10 some results. For some cases it looks okay 11 but some cases it doesn't so we need to add 12 some model parameters to control the behavior 13 of the model. And actually some of the 14 questions we like to ask to the committee is 15 are we doing the write-up right away or we on 16 the right track. 17 So the steps in model calibration 18 basically the first step I had mentioned is 19 develop and verify that regional demand and 20 generation data are comparable because that's 21 the most important thing. If the demand is 22 too high or too low we can never replicate BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 115 1 the dispatching pattern. 2 In the second step we analyze 3 historical generation data at a generator 4 level to identify outliers. Then we look at 5 some historical data and see what happens in 6 the real world and right now we have focused 7 on year 2002 data and we have something like 8 8,000 records of generators and power plants 9 and we look at the generation, we look at 10 fuel consumption, and we try to understand 11 the generation pattern. 12 And the third step is we estimate 13 some of the key model parameters. I will 14 talk about that. And then we go through the 15 fourth step is basically check the model 16 result, check for data consistency, and do 17 another iteration to see if we get better 18 results. And so basically understand what 19 the model is doing even though we created the 20 algorithm but the behavior of the model is 21 not all controlled by us. It has its own 22 personality. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 116 1 The historical demand and 2 generation data, we have several sources. 3 Regional demand is arrived from the EIA 826, 4 which is monthly sales data, and the 5 difficulty is the monthly sales are based on 6 billing cycles and in most instances they are 7 not calendar month data. So if the billing 8 cycle could be from the 26th of one month, 9 ends in the 29th of the following month, and 10 apparently even within the same states the 11 billing cycles could be different for 12 different customers. So that creates some of 13 the data issue in terms of how we can get a 14 consistent demand data. Regional generation 15 is from the EIA 906, which is calendar month, 16 so that's the kind of data issues we face and 17 we have to deal with. 18 So in terms of analysis we have to 19 convert the demand from billing cycle to 20 calendar month and adjust demand supply in 21 imports and create identity. So just the 22 first steps in order to validate a model and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 117 1 then to examine the behavior of the model. 2 The second thing we do is we 3 analyze generation data. It's very tedious 4 but it's getting interesting to understand 5 what the data is telling us. So we used net 6 generation and fuel use data to estimate heat 7 rate. So we have the fuel consumption, we 8 have the generation, and we use the heat rate 9 and the fuel prices to compute variable cost. 10 That's what we can do using the data and we 11 use data on generation capacity and the net 12 generation to compute utilization rates. And 13 then we do some regression analysis to see if 14 we can identify any patterns. 15 So this chart basically gives you a 16 flavor of how does it look and we basically 17 did a double log and we'll run regression 18 between the utilization rates in July and the 19 variable cost in July and see well, what's 20 the pattern. And -- 21 MR. CLEVELAND: Phil, can I 22 interrupt for a second? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 118 1 MR. TSENG: Yes. 2 MR. CLEVELAND: So the vertical 3 access those observations are individual 4 generating units? 5 MR. TSENG: Individual generation 6 units, yes. We are looking at power plant 7 data. Sometimes it's generator data. If a 8 power plant has, say, two different 9 generators like a coal steam and a gas 10 combined cycle then we have two data points. 11 MR. CLEVELAND: So the utilization 12 rate is the observed utilization rate? 13 MR. TSENG: Yes. 14 MR. CLEVELAND: That's observed? 15 MR. TSENG: Observed. 16 MR. CLEVELAND: And the variable 17 cost is the one you have calculated based on 18 this formula? 19 MR. TSENG: No, utilization rate is 20 also calculated based on the net generation 21 and the capacity. 22 MR. BERNSTEIN: So when you look at BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 119 1 that I would say to myself, well, you are 2 probably looking at two different utilities. 3 I've got one utility that's vertically around 4 4 and one utility that's vertically around 6. 5 MR. CLEVELAND: A small one at 8. 6 MR. BERNSTEIN: I mean, did you 7 look to see which plants they were and are 8 those on the left different utilities than 9 those on the right? 10 MR. TSENG: Well, all those 11 utilities are electric utilities. We 12 excluded CHPs because if we don't CHPs -- 13 MR. BERNSTEIN: No, no, but there 14 is you know, a clear -- 15 MR. TSENG: You mean like 16 independent power producers? 17 MR. BERNSTEIN: No. I mean, there 18 is a clear differentiation between plants 19 that are costing around 4 and plants that are 20 costing around 6 and I'd say there is 21 probably a statistical difference in that 22 same -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 120 1 MR. CLEVELAND: Whatever that curve 2 is it's misleading. 3 MR. TSENG: Right, what I want to 4 show is the challenge we are facing. 5 MR. BERNSTEIN: Right, but what I 6 am asking is if you have gone back and looked 7 in there and see are the plants in the one 8 column coming from one utility and the plants 9 in the other column coming from another 10 utility? And is it more of a -- 11 MR. TSENG: They are from different 12 power plants. 13 MR. BERNSTEIN: No, I -- 14 MR. TSENG: In this sample I think 15 we have something like 80 probably, 70 to 80. 16 MR. BERNSTEIN: I know, but are 17 those on the left in Southern California 18 Edison territory and those in the middle PG&E 19 territory? 20 MR. TSENG: Oh, I didn't look at it 21 that way. 22 MR. BERNSTEIN: I mean, because my BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 121 1 guess is what you are seeing there within 2 California is the fact of the matter is that 3 Southern California Edison or Los Angeles 4 Department of Water and Power has a different 5 cost structure for their fuels and for their 6 plants than does PG&E as well as San Diego 7 Gas and Electric and you're trying to 8 economically dispatch across California which 9 you may not be able to do. 10 MR. TSENG: Right, that's -- 11 MR. BERNSTEIN: ———————— other 12 states. 13 MR. TSENG: Exactly, that's why I 14 would describe to you the steps we use to 15 actually structure the dispatching and now I 16 have one example to show you the results. 17 MR. SANDERS: Is it part of a 18 distribution because you got different types 19 of power plants, from combined cycles to 20 steam to turbines to I wouldn't say coal, not 21 in California, but, I mean, I think that's a 22 large part of it, the variation shown. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 122 1 MR. TSENG: And also the reality is 2 some of the similarly high cost power plants 3 are running at high utilization rate could be 4 in the low pocket so you could have 5 infrastructure constraints like transmission 6 lines capacity is limited just like the ———— 7 Dispatching is based on the location and 8 margin and pricing. So if it's in your 9 pocket you have no choice but dispatch the 10 high cost power plant. And so in our 11 modeling efforts we need to capture those 12 kinds of things using statistical methods 13 instead of the economic approach. 14 DR. HENGARTNER: Essentially, I 15 mean, if you know the infrastructure on how 16 the gird works you could identify which 17 regions are constrained with the input and 18 that's where you are going to have those 19 higher cost power plants being used. 20 MR. TSENG: Exactly, and that's 21 what we want to capture in our model. 22 MR. CLEVELAND: Those are all the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 123 1 fossil fuel power plants in California? 2 MR. TSENG: All the fossil in 3 California, yes. 4 MR. CLEVELAND: There are only 28 5 of them. 6 MR. TSENG: I thought there were 7 like, 70 or 80 of them. 8 DR. HENGARTNER: Not on this plot. 9 MS. KIRKENDALL: Unless they get 10 plotted one on top of other. 11 MR. TSENG: So this just to give 12 you a flavor of the pattern and the next 13 thing is once we understand the data we look 14 at the data and we look at a pattern and we 15 say well, there are some high cost high 16 utilization rate power plants. And if we put 17 these power plants in the dispatching 18 solution algorithm based on the cost those 19 power plants would not be dispatched. 20 So it doesn't mimic the historical 21 generation pattern and we need to capture 22 that in order to replicate the history. So BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 124 1 for regions such as California for input 2 regions it's pretty easy to target the total 3 generation. So all we have to do is run the 4 model and identify import constraint and we 5 say well, this is the inflow limits 6 California can take. It could be seasonal so 7 we allow some like 5percent up and down 8 variation but in general that's the limit. 9 When we run a model we see the total 10 generation in California matches the 11 historical generation. That's the way we 12 identify the import constraints. 13 And then the next step is based on 14 the scattergram and regression analysis. We 15 assign generators. Actually we have five 16 load groups but for fossil plants we assign 17 each to one of the four groups. We assign it 18 as a plant type code, equal to one if the 19 utilization rate is greater than 75 percent. 20 So we are basically structured based on 75 21 percent, 50 to 75, 25 to 50, and 25 and 22 below. So 75 and above we will be putting BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 125 1 the plant type code 1. So they will be 2 dispatched earlier. So that's the way we 3 structure the power plants. 4 So the model solution is still the 5 same solution algorithm except dispatching 6 order we have two levels of sorting criteria. 7 One is if the plant has a dispatching order 1 8 they would be dispatched first and within the 9 group of 1 everything is based on economics. 10 MS. KHANNA: Question? 11 MR. TSENG: Yes. 12 MS. KHANNA: Does the variable cost 13 vary across dispatching? Could it have a 14 plot in group 4 that actually has a lower 15 marginal cost than one in group 1 but your 16 model is forcing you ———————————————— 17 MR. TSENG: Yeah, it could. 18 Actually we found some observations with very 19 low cost but with very small utilization 20 rate. And we have no way to understand why 21 that's the case so we put those power plants 22 in group 4 anyway because the utilization BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 126 1 rate is very low. 2 So this is the statistical 3 approach. We look at the statistics and we 4 say well, if this is the historical pattern 5 we want to be able to capture the historical 6 pattern. And if we have more time maybe we 7 can visit the plants and ask them exactly why 8 they are doing the way they are doing, why 9 they don't generate more power. Is it 10 because of the transmission or is it because 11 something other than economics? So this 12 becomes actually a very important element in 13 replicating the historical generation pattern 14 and fuel share. 15 Another area we have to do is the 16 numerical criteria for fuel switching. In 17 2002 the price of natural gas and oil was 18 pretty stable so we didn't have to worry 19 about fuel switching. That's why for model 20 validation purposes the first step is we just 21 look at year 2002 data and then we say well, 22 can we replicate the historical pattern. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 127 1 And 2003 is another world because 2 natural gas price went up a lot and I know 3 for a fact New York actually switched a lot 4 of the dual fuel power plants from natural 5 gas to oil so the oil consumption in New York 6 State went up. And so once we finish this 7 step we would then run the model to the next 8 12 months. 9 Another parameter is we need to 10 understand what other factors like gas power 11 plants switched to oil then storage becomes 12 an issue. Supply of diesel fuel becomes an 13 issue. Environmental issues all can affect 14 the switching and we need to understand a 15 little bit more and then establish some kind 16 of criteria for fuel switching. 17 Another thing is coal power plants 18 burn a little bit oil and little bit natural 19 gas. It could be one-tenth, probably two- 20 tenths or five-tenths of a percent but since 21 co-generation number is so big, five-tenths 22 of a percent, it makes up a big number and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 128 1 that can actually create a bias in terms of 2 how much natural gas is consumed in power 3 generation so we need to look at that as 4 well. Those are all included in the EIA 906 5 data survey and we have the information to do 6 the analysis. 7 And the last one is hydropower. 8 Actually it's not always used to meet base 9 load. It's not running all the time as base 10 load and we have some evidence to indicate 11 that some of the hydropower especially like 12 in New York are used to meet peak load 13 demand. And so we sometimes for model 14 calibration purposes adjust the hydropower 15 dispatching and so hydropower can be used to 16 meet peak load demand as well. So those are 17 the parameters we have to calibrate. 18 And the last step is we check for 19 data inconsistency. When we run a model we 20 would look at results and then compare with 21 the historical data and make some adjustment 22 to the power plant's dispatching order. Like BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 129 1 the plant code in some regions sometimes we 2 call it category 2 dispatching plants but 3 after we run the simulation we see actually 4 that some of the power plants should be in 5 category 1 because the of cost structure. 6 Just look at the size of the 7 region. The Mountain Region covers so many 8 different states so the fuel prices can be 9 very different and some of the dispatching 10 pattern may have something to do with the 11 fuel prices as well but we have only one 12 representative regional price for each 13 region. So when we calculate the variable 14 cost there could be some misleading effects 15 as well. So that's the process we have to go 16 through. Yes? 17 MS. KHANNA: So that tells me that 18 what category a plant belongs to is really 19 endogenous or should be endogenous rather 20 than being coded exogenously or there should 21 be some part —————————— at least. 22 MR. TSENG: Well, most of the power BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 130 1 plants right now the way that we run the 2 model is determined. Like, we have five 3 categories. Nuclear and hydro we assign a 4 dispatching order of 0, which means it will 5 be dispatched first. For fossil plants we 6 assign a dispatching order of 1 through 4. 7 It's based on historical pattern. So if 8 historical data indicates a power plant is 9 running 80 percent most of the time through 10 the 12-month period we have historical data 11 then we put those power plants in category 1. 12 MS. KHANNA: That made a lot of 13 sense right up to your last statement where 14 you said that sometimes because let's say if 15 natural gas prices go crazy suddenly you'll 16 find a plant is going to be operated more or 17 less than it has historically been. So can't 18 your model capture maybe some limited amount 19 of jumping around in reaction to changes of 20 cost ———— different from the historical 21 pattern? 22 MR. TSENG: Well, I think that's a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 131 1 model behavior issue because sometimes 2 certain power plants if we have a high cost 3 power plant even though say, for example, the 4 utilization rate is 49 or 51 and if you are 5 putting a different category it may be bumped 6 down all the way to the bottom of the 7 category and the utilization rate of the 8 power plant can never reflect a historical 9 pattern which we don't know. And so we do 10 have to make some adjustment based on the 11 simulation results and then based on the 12 observed data. 13 DR. HENGARTNER: I think what I 14 understand here is if you have power plants 15 that can either use gas or oil in your 16 analysis they are treated as two separate 17 power plants? 18 MR. TSENG: No, they are the same 19 power plant. 20 DR. HENGARTNER: Oh, they are the 21 same? 22 MR. TSENG: Yes, they are the same. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 132 1 MS. KHANNA: That's because of the 2 fuel switching. 3 DR. EDMONDS: Right, but you're 4 actually asking about changing the dispatch 5 order; that is, you might have a natural gas 6 turbine that switches, depending on the price 7 of gas, switches from either being a peaking 8 unit to being a base load or -- 9 MS. KHANNA: Right, whereas their 10 model has a cost in one mode and it can't 11 jump from there. 12 DR. EDMONDS: Yes, and actually one 13 other thing that might help us understand is 14 you've got a 0 bin which always gets 15 dispatched, a bin 1, a bin 2, a bin 3, and a 16 bin 4, and those bins are historical 17 utilization rates? 18 MR. TSENG: Yes. 19 DR. EDMONDS: Now, when you run the 20 model how do you determine the utilization 21 rate of bins 1 through 4? In other words you 22 have got these historical utilization rates BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 133 1 and you have got these power plants in bins 2 and so you know in a sense how much they have 3 been operating historically but you still 4 have a demand that you have calculated in 5 another part of the model and now you have 6 your base, the 0, that's there. Then how do 7 you determine how much from bin 1, bin 2, bin 8 3 and bin 4 go into meeting that total 9 demand? 10 MR. CLEVELAND: Cost, they're 11 stacked by cost, right? 12 MR. TSENG: Yes. 13 DR. EDMONDS: Do all of bin 1, all 14 of bin 2 -- 15 MR. CLEVELAND: You just stack them 16 up. 17 MR. TSENG: Exactly, yes. 18 DR. EDMONDS: And then if you need 19 some more, you just go into bin 3? 20 MR. TSENG: Yes. 21 MR. CLEVELAND: Do them in rank 22 order of cost. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 134 1 DR. EDMONDS: So in effect the 2 model will never operate in bin 4 because it 3 doesn't have to? 4 MS. KHANNA: Unless there is a peak 5 load or -- 6 MR. TSENG: Oh, it will because the 7 reasons power plants are in bin 4 because 8 there is demand for those power plants. In 9 the summer, for example, the shape of the 10 load curve is very different. The peak 11 demand level in California, I think, could be 12 as high as like 45,000 megawatts but in -- 13 MR. CLEVELAND: So in the 14 summertime you'll get into bin 4; in the 15 wintertime, you will never run those plants? 16 MR. TSENG: Right, that's why we 17 see utilization rate for some power plants is 18 like 10 percent or 5 percent. Part of the 19 reason is they are not used unless the demand 20 is really high. 21 DR. HENGARTNER: But now I'm 22 confused because if cost is what drives it BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 135 1 and if I have a regional grid structure that 2 forces high cost plants to work then I am 3 actually going to misplace those in the wrong 4 bins. 5 MR. CLEVELAND: There something in 6 the model that deals with those bottlenecks. 7 DR. HENGARTNER: No, he says he is 8 forgetting about that. 9 MR. TSENG: Actually the way we 10 structured four bins is if there is an 11 infrastructure bottleneck the statistics will 12 reflect you have a high cost and high 13 utilization rate power plants and we put them 14 in bin 1 so they will be used. So we are 15 capturing some of the historical pattern 16 here. 17 DR. HENGARTNER: So then we don't 18 do it according to cost? 19 MR. TSENG: Yes, we do -- 20 DR. EDMONDS: Once you're in the 21 bin it's by cost but the bins go 22 sequentially. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 136 1 MS. KHANNA: So that's okay. 2 MR. COSTELLO: We knew that if we 3 did just the whole region or the whole 4 trading region by cost that we were going to 5 get some plants that wouldn't get used that 6 really in historical reality do get used and 7 without going to the plant and finding out 8 exactly why that was we just assume that they 9 really needed to be in a prior order for some 10 other reason like one of those that Phil 11 mentioned. I don't think we're quite sure 12 how well it works for 2003 and 2004 yet but 13 that's where we are going, right? 14 MR. TSENG: Right, yes. I think we 15 are very close for 2002. Actually for the 16 Western region I have a few examples. I will 17 go into that. So that's the way we do it. 18 It's an iterative process and we have to 19 understand the data and we have to understand 20 the behavior of the model and then we have to 21 structure this model in a way which can 22 capture some of the institutional factors, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 137 1 which are not all economics. 2 MR. BERNSTEIN: I hate to ask this 3 but, given that you have historical data, why 4 bother doing a dispatch model at all? Why 5 not use historical trends because you are 6 talking about short-term forecast? It seems 7 like a whole lot of work when you're going to 8 have to check it against the historical data 9 anyway. Why don't we use whatever is there? 10 MR. TSENG: Actually at the 11 beginning I did a lot of regression analysis 12 and I thought maybe a simple fuel share 13 approach would do. But happened is the fuel 14 share changes. It can go down to 4-5 percent 15 for natural gas and go up to like 20 percent 16 and depend on the demand. And now the 17 question would be how do we know the shape of 18 the load curve and how that shape of the load 19 curve relates to the generation by fuel type. 20 And so if you look at statistical methods and 21 if you could come up with something I think 22 that will save us a lot of trouble. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 138 1 MR. BERNSTEIN: But you're not 2 actually interested in the dispatch order. 3 What you are really interested in is marginal 4 price. You're interested in marginal 5 capacity constraints and I just -- 6 DR. EDMONDS: But I think he also 7 needs to get the demands for the fuels for 8 the power sector. 9 MR. BERNSTEIN: Right, but you 10 could do that using historical information on 11 historical generation. There are enough 12 years of data to be able to do that without 13 having to create your own dispatch model 14 because what I worry about in do your own 15 dispatch model is how many utilities are 16 actually still using economic dispatch really 17 versus some other strange thing that some 18 utilities are doing. 19 Secondly, as new plants come online 20 it's going to be really hard to capture that. 21 If renewable portfolio standards in a number 22 of states actually happen it's going to get a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 139 1 lot harder to do a dispatch model in a 2 situation where I think it's going to be less 3 and less useful. 4 MR. CLEVELAND: But if new 5 standards are coming along then your 6 historical data approach goes out the window 7 because then you won't be able to forecast 8 that. 9 MR. BERNSTEIN: Well, but you can. 10 You know what chunk it's taking away and -- 11 MR. CLEVELAND: But you also have 12 no guarantee that supply is going to equal 13 demand which is really important to my 14 understanding power markets to make sure that 15 you're going to capture the dynamics right 16 which you would totally lose in a -- 17 MR. BERNSTEIN: But you can do that 18 without running an optimization dispatch 19 model. Because it's going to be so hard to 20 maintain this thing, these 12,000 variables, 21 I just -- 22 MR. CLEVELAND: You said the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 140 1 dispatching stuff isn't important but I think 2 it will be. That kind of stuff is important. 3 There are people who will be interested in 4 that plant level data dispatching and in New 5 England at least -- 6 MR. BERNSTEIN: But they're not 7 going to show that. 8 MR. CLEVELAND: Well, the data will 9 be -- 10 MR. TSENG: We actually can track 11 the utilization by plants because the 12 information is there. It can be retrieved. 13 But we don't because -- 14 MR. BERNSTEIN: But you're not 15 going to show that in your forecast. You're 16 not going to show in the short-term forecast 17 your dispatch order of power plants. I think 18 that would be a big mistake. So the 19 historical data is there. That's available 20 for anybody to use. The purpose of this is 21 to lay out a short-term forecast and I just 22 wonder. I mean, you may be right. This may BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 141 1 be the way to go. I'm just fundamentally 2 wondering if it's worth that effort. 3 MS. KHANNA: You raise a good 4 point. I mean, it's all about cost benefit 5 in the end. But I worry in moving away from 6 let's say the economics-based model or the 7 dispatch model because if your forecasts 8 start going off if you just have let's say an 9 historical statistical model you have no idea 10 of why you're going off. 11 The advantage of this is you have 12 some intuition, some logic, built into the 13 model and you can check the logic of the 14 model against what you're observing and you 15 can try and figure out okay, if I don't 16 predict what is actually going on then I have 17 to look at what I am doing whereas if you 18 just use pure statistics you don't know what 19 you are doing with due apologies to the 20 statisticians here. 21 MR. BERNSTEIN: But you have enough 22 richness in the data, though, to be able to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 142 1 uncover that. You have enough richness in 2 the data and enough disaggregation to be able 3 to analyze what factors have caused changes 4 in the way the dispatching works. And so you 5 use that to move forward. I mean, I'd be 6 interested to see, I think you are not there 7 yet, as to how close and the other problem I 8 have is 2002. I am not sure that's a good 9 representative. 10 How much fiddling are you going to 11 have to do to your model to get it to match 12 the historical perspective and maybe you are 13 going to get to that? 14 MR. TSENG: Actually I have some 15 numbers I can show. 16 DR. HENGARTNER: Can I ask one 17 question to Mark? If I understand well 18 short-term predictions the only thing you 19 really need to know is how small changes to 20 the actual system matter and therefore we are 21 not trying to extrapolate way out? 22 MR. BERNSTEIN: Right. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 143 1 DR. HENGARTNER: And therefore this 2 whole predictive thing that Neha doesn't 3 believe statisticians understand actually, I 4 mean, we may not be needed. I mean, it's 5 essentially it's just local shifts to the 6 current system that we need to understand. 7 MS. KHANNA: We're only talking 12 8 to 24 months, right? 9 MR. BERNSTEIN: Right. 10 MS. KHANNA: That's fairly long in 11 my opinion. 12 DR. EDMONDS: I have to say 12,000 13 variables doesn't sound like that many to me. 14 And if they are available I actually have to 15 say I like a structural approach to 16 addressing the problems so I find the way 17 you're going at this to be somewhat 18 satisfying to me because if you have those 19 data available you have a much more 20 fundamental understanding of the phenomena 21 that you are trying to model and why not take 22 advantage of that information if it's at your BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 144 1 disposal as long as there isn't a method that 2 gives you a fundamentally better estimate but 3 my sense is that you're on to a methodology 4 that's going to give you a fundamentally good 5 estimate. 6 DR. FEDER: Jae, I think you were 7 making assumptions that you will really 8 understand it if you have the model. Think 9 of how many years it took for the weather 10 forecast to get it right and they have daily 11 correction to the models because you can see 12 what the weather is, how well you are 13 predicting it, so I think 12,000 and even 14 1,000 is a lot. 15 MR. CLEVELAND: But a lot of the 16 relationships here are based on pretty simple 17 fundamental economic theory and so you feel a 18 little more comfortable at saying Y is a 19 function of X because, unlike the weather, 20 where there is a good objective literature 21 out there ———————————— 22 MR. EDMONDS: Well, if you look at BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 145 1 weather models they only have —————————— the 2 equations. 3 DR. HENGARTNER: Gentlemen, there 4 is a deadline of 15 minutes for Phil to 5 finish. We can discuss the weather at lunch. 6 MR. TSENG: Does John have a quick 7 question? 8 DR. HENGARTNER: John. 9 MR. WOOD: At some point you really 10 have to understand the physical thing that is 11 going on and in Texas the electric bidding 12 process of how you get paid for your electric 13 power now is on an hourly market basis and 14 nuke plants and coal plants are willing to 15 pay to put their power into the grid at 3:00 16 o'clock in the morning on a nice day and so 17 there is some obvious logic of why coal 18 plants and nuclear plants are not going to be 19 very sensitive to natural gas prices or oil 20 prices or anything else at a period of time. 21 And then there is a tremendous 22 amount of new swing generation power that's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 146 1 natural gas-driven only in Texas and they 2 really are going to have a different response 3 behavior and whatever modeling approach you 4 take it ought to allow you to have some 5 insight into the physical nature of that 6 market whichever it turns out to be. 7 MR. TSENG: Yes. I mean, basically 8 the dispatching is demand driven and the load 9 shape changes by the hour and so if demand is 10 low some of the power plants will not be 11 used. And in terms of pricing by the hour we 12 actually have prices by the hour based on the 13 margin of cost. And how do we use it in 14 terms of the example you used in terms of 15 nuclear power plants? Two a.m. it wants to 16 generate because it cannot shut down and we 17 actually observed sometimes you can see 18 negative price because they simply are saying 19 well, we will pay you to use our power. We 20 cannot shut down our nuclear power plants. 21 And we have a long way to go to capture that 22 kind of sophistication. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 147 1 I'm going to move on to the data 2 sources so I will quickly list the kind of 3 information we have and actually almost all 4 the information we use to build a model and 5 make model dispatching decisions based on EIA 6 and the FERC information. The EIA 906, 7 monthly data on generation, I mentioned that 8 the EIA 826 monthly electricity utility 9 sales, and EIA 423 and the FERC 423 with 10 monthly costs and quality of fuel for 11 electricity plants and EIA 860, capacity 12 information, and for imports from Canada and 13 Mexico we have from Fossil Fuel Office the 14 information there. 15 Now, before I show you any result I 16 want to just say a few words about data. I 17 think data is one major issue for almost all 18 modelers. The sales and generation data are 19 subject to reporting errors and estimation 20 errors because sometimes we have non- 21 respondents and then an EIA data analyst will 22 have to come up with some methods to fill the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 148 1 gaps and so when we run a model sometimes we 2 see some pattern which is not exactly 3 consistent with what we observed from the 4 data. 5 And also regional aggregation may 6 introduce biases to the cost-based 7 dispatching. Even I heard northern 8 California and southern California may have 9 different natural gas prices but in a model 10 since we have only one region we have one 11 regional price for California. And that may 12 also create some dispatching difficulties 13 when we use cost alone. 14 And the oil and gas consumed in 15 coal power plants are hard to predict so we 16 apply a simple factor based on historical 17 share. And the timing of hydropower 18 dispatching we would basically play with it 19 and then see makes the model behave the way 20 we think it's reasonable and we have 21 variables we can publish so people can 22 provide comments on the validity of those BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 149 1 assumptions. 2 And lastly I think when Nick 3 mentioned about nuclear power plants, why 4 it's half of the normal, this model we don't 5 do ex ante because we don't know what may 6 happen. Like if we have forced outage there 7 is no way we can capture that. For the ex 8 post we can always make adjustment to the 9 plant availability and we can adjust it. And 10 so the model result can replicate a history 11 but in most cases we just try to understand 12 what it is instead of trying to change the 13 model structure to replicate one or two or 14 three months. 15 So the difficulty in terms of 16 modeling the dispatching behavior is we are 17 looking at month by month instead of most 18 models look at a baseline year and then say 19 well, we calibrate to one year. It's much 20 easier. But when we have to go through maybe 21 12 or 24 data points and then make 22 projections that's a challenge. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 150 1 The next few slides I will show you 2 some of the historical data and then I have 3 some model results. I think this data will 4 illustrate how difficult it is to use 5 econometrics or statistic method. 6 Like this one is all fossil plants 7 in the western interconnection and I exclude 8 the variable cost greater than 20 cents per 9 kilowatt hour. So from the scattergram I 10 couldn't see any pattern. So when we run 11 regression analysis it's very hard to capture 12 what it is. 13 DR. HENGARTNER: What do you mean 14 by a rate larger than 1? 15 MR. TSENG: This is another thing. 16 When we looked at the 906 data we found some 17 plants reported utilization, the net 18 generation number, and we know the capacity 19 and we calculate the utilization rate based 20 on the generation and capacity and we come up 21 with big numbers. 22 In most cases we can resolve the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 151 1 differences. What happened is the EIA 906 2 and 860 are two different survey forms. And 3 so 906 collects the generation and 860 4 collects the capacity. And so there's a data 5 processing issue in matching the record. And 6 so when we look at the data the first 7 reaction was you have bad data point. But 8 when we looked at detail this is how tedious 9 it is and we found out one plant, for 10 example, you have this high point here like 11 there is a top point here. Basically it's 12 like almost 300 percent. And we look at 860 13 data and we found out that the capacity 14 number in the 906 missed two big power 15 plants. And so if we include those two big 16 power plants then the capacity utilization 17 rate becomes like 92 percent instead of 18 almost 300 percent. So those are the data 19 issues we found and we have to face. 20 So what I am saying is when we do 21 analysis a lot of times people want to use 22 automated approach and we found looking at BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 152 1 the data can be tedious but it's also very 2 productive. 3 This is another scattergram, just 4 give you a flavor. If we eliminate some of 5 the outliers and this is the pattern and, 6 again, regression analysis will not help us a 7 whole lot. 8 MR. BERNSTEIN: Well, but if you go 9 backward, I mean, what that tells me is that 10 maybe they're not doing economic dispatch 11 because you are seeing plants, you are seeing 12 a rectangle, and to me that says maybe either 13 at the region level it's not being 14 economically dispatched which we know because 15 it's done at the utility level but even at 16 the utility level maybe they're not really 17 economically dispatching any more. 18 That's what I would question when I 19 look at that data, not necessarily whether 20 the approach is right but I would say to 21 myself well, I would expect to see if they 22 were economically dispatching some BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 153 1 relationship between utilization rate and 2 variable cost. And I guess this is July? 3 MR. TSENG: Right. 4 MR. BERNSTEIN: And so 2002 in the 5 western region it wasn't particularly hot or 6 anything but maybe they're just having to use 7 everything because of peak demands but I just 8 don't see a pattern and so I would say to 9 myself is there any relationship between 10 utilization rate and cost. And I would ask 11 that first before I develop a dispatch model. 12 DR. HENGARTNER: So to confirmed 13 your hypothesis, we would look at the month 14 of August, say, decide if there was more or 15 less utilization, and see if there is more 16 utilization if the load costs are going up or 17 something like that who has the most change? 18 Is that what you are suggesting? 19 MR. BERNSTEIN: I think that's one 20 of the things, I mean, but I would certainly 21 try to figure out from the data if there is 22 anything that helps us determine what is the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 154 1 relationship between cost and use. 2 MR. WOOD: You get to pass through 3 most of the marginal generation cost for fuel 4 and so if it doesn't take customers off-line 5 and there is a little delay before you get to 6 reclaim the rate but you do get to pass it 7 through theoretically so what you are saying 8 there may be some very good reasons that 9 since you get to pass through the cost that 10 you have another reason for a dispatch. 11 MR. TSENG: Well, actually looking 12 at the data, the interesting part is when we 13 mix a large trade zone like the Mountain 14 Region with the Pacific region the fuel 15 prices are very different. Natural gas in 16 Oregon, Washington is almost a buck less than 17 California and Mountain Region and coal 18 prices in the Mountain Region is probably a 19 buck less than the other two regions. 20 And so when we plot it it actually 21 can be very misleading. That's the thing is 22 if you look at a pattern you would say well, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 155 1 there is no pattern. The reality is there 2 may be some pattern and, using a structured 3 approach, we can probably capture some of 4 those relationships. 5 MR. BERNSTEIN: Right, but have to 6 do I think a bunch more analysis before 7 getting to the point of creating this 8 dispatch model and that's what I -- 9 MR. TSENG: Okay, here I will show 10 you some of the results. Again, I am using 11 the year 2002 monthly data and have model 12 results. I think I can show what it is. 13 Basically what I did was we have 4 bins where 14 we put the fossil fuel plants. And for 15 California we have the import limits and 16 that's about it in terms of the dispatching 17 order. I made some adjustment, move power 18 plants from bin 3 to bin 2 or bin 2 to bin 1, 19 some minor adjustment, and actually it looks 20 pretty good if you look at out of top month, 21 eighth month, in terms of total regional 22 generation they are within reasonable level. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 156 1 And part of the reason April, May, 2 June and July the numbers looked screwy was 3 for whatever reason Oregon and Washington 4 State had significant drop in gas and coal 5 generation. So those four months suddenly 6 the generation in the region dropped almost 7 like, by 50 percent. 8 I have no understanding of what 9 happened there but I think if I have time I 10 can talk to people why the numbers are such. 11 Other than that if you believe historical 12 generation pattern can replicate a history 13 that's the kind of numbers we actually 14 generated from the model result. 15 And another criterion I can look at 16 is the differences between simulated value 17 and the 2002 annual total. If I sum the gas 18 and coal so basically the model likes coal a 19 little bit more; however, if I make an 20 adjustment making assumptions, say like 0.5 21 percent of natural gas is consumed in coal 22 power plants then I can add like 1,600 to gas BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 157 1 generation and subtract 1,600 to coal. So 2 cumulatively the total impact in terms of 3 performance or to generate the model is not 4 all that bad. The only thing is I have four 5 months of peculiar data pattern and using 6 historical pattern you will not capture that 7 because I don't know why in one month, for 8 example, in July or June the coal generation 9 in Washington or Oregon, that region, dropped 10 from average 1200 to less than 200. 11 MR. BERNSTEIN: I will tell you 12 why, because one aluminum plant decided they 13 were making more money. I mean, this is the 14 profit. Kaiser could make more money 15 reselling their electricity that they had on 16 long-term contract with ———————— power than 17 to actually consume it and so they shut down. 18 MR. TSENG: That's 2002. 19 MR. BERNSTEIN: That's just one 20 particular example but this is the inherent 21 problem that you're going to have trying to 22 do a dispatching approach —————————————— BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 158 1 MR. EDMONDS: Well, you run into 2 that problem anyway, right? That's a unique 3 event problem. 4 MR. BERNSTEIN: But the fact of the 5 matter is you need that power -- well, keep 6 going. 7 MR. TSENG: I think it's time for 8 me to ask questions. Well, the question is 9 are the regression techniques we used 10 acceptable for changing a dispatching order. 11 I mean, if there is a better way to capture 12 the dispatching and understand the behavior 13 of the power generators we would definitely 14 like to learn about it. And a second 15 question, as Margot earlier mentioned about 16 evaluation, given the data quality what is an 17 acceptable level of margin for error? 18 Thanks. 19 MR. BEINSTEIN: I mean, I still 20 think the issue is what are you trying to use 21 it for. And I'm not clear on that and I 22 think there has to be some clarification on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 159 1 what you really want to use this for and one 2 way of dealing with this is to create 3 marginal cost curves for electricity in 4 regions based on historical data as opposed 5 to dispatching it so you get a production 6 cost curve or a marginal cost curve that 7 gives you both by fuels and by total 8 generation and use that and you adjust that 9 as new historical data comes in. That's one 10 way to do it. 11 You're not looking to produce a 12 forecast of dispatch. You're using it to 13 produce a forecast of electricity demand at 14 the regional level, supply constraints at the 15 regional level, and prices at the regional 16 level. 17 MR. TSENG: Well, actually we did 18 look at creating a marginal cost curve; 19 however, it's a monthly effort so the 20 dispatching hours basically is creating 21 supply curve month by month using the monthly 22 fuel cost information we get from the FERC BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 160 1 223 for the historical validation process and 2 for forecasting the monthly fuel prices are 3 different. And so if we want to create a 4 marginal cost curve based on those power 5 plants then the concept is almost identical 6 to the dispatching algorithm that we are 7 using. 8 MR. BEINSTEIN: But I would do it 9 more on historical. See, what you're doing 10 is you're mixing historical information with 11 doing an optimization routine for dispatching 12 the plants and I'm not sure you need that 13 last step. 14 DR. HENGARTNER: Thank you very 15 much Phillip. I'm sorry that we ran out of 16 time, but it's surely not the last time we'll 17 talk to you about this. 18 MR. TSENG: I'll be happy to answer 19 any questions. 20 DR. HENGARTNER: At this time we'd 21 like to make the summary of what we had in 22 the break-out session. I'd like to invite BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 161 1 first the statisticians that went downstairs 2 to report on what their discussion was about. 3 MR. CLEVELAND: What happened to 4 me? 5 MS. FORSYTH: You come next. 6 MR. CLEVELAND: Oh, okay. I 7 thought you were going to part C and we were 8 in actually B. That's good. 9 DR. HENGARTNER: I was going to 10 skip you but it didn't work. 11 MS. FORSYTH: So we talked about 12 frame comparisons downstairs and it's an 13 update from work that we talked about last 14 fall so the Census is been working with EIA 15 to help EIA assess the kinds of frame errors 16 if any they have in five of their data 17 collection efforts. 18 And the census has completed work 19 evaluating three of those frames and 20 presented some preliminary results and a 21 question. The preliminary results are that 22 in general the EIA frames look very good. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 162 1 All three of the frames that they've examined 2 so far are small frames so that when you're 3 missing a unit or two it looks like you're 4 missing a lot and in fact you are missing a 5 lot. You're missing a large portion of the 6 frame. 7 And so in one of the evaluations 8 there were several units missing producing 9 like, 70 percent match rate when you're 10 looking at the frame units but in terms of 11 the volume of products, which was one of the 12 key indicators, the volume of product 13 captured is still relatively high so those 14 relatively few nonmatching frame units are 15 also producing relatively low amounts of 16 product and therefore including them in the 17 frame doesn't alter the expected estimates 18 much. So it looks like there are some units 19 missing but there have relatively low impact 20 and therefore the frames are pretty good for 21 EIA's purposes. 22 The evaluations so far were with BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 163 1 small frames. There are two more waiting 2 that will be with larger frames and there's a 3 larger potential for error there. So we are 4 interested to see what those results will be. 5 The question put to the committee was whether 6 this frame evaluation should be an ongoing 7 effort, just part of the economic census, 8 and, of course, the committee didn't answer 9 the question but at least part of the reason 10 is that we can't really because we can see 11 the benefits of doing the evaluation but we 12 don't really know the cost and so we can't 13 really help do the real decision-making that 14 needs to be made. 15 As part of the ongoing EIA 16 evaluation looks like the frames are pretty 17 good to start with and so the likelihood of a 18 big dramatic change at least in the frames 19 that we looked at so far is probably small, 20 and so the five-year evaluation may not be a 21 really important thing for at least those 22 three small frames we've looked at. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 164 1 And one thing that did come up in 2 the course of the conversation is that 3 currently the way the census is done, the 4 matching, it's like a two-stage matching 5 process. First the EIA establishments are 6 matched to the business registered at the 7 Census Bureau and then once that matching is 8 done you can get a general assessment of how 9 much change there is likely to be and then 10 from there you can go on to look in more 11 detail at the comparison between the EIA 12 frame and the economic census results. 13 And if you retain that two-stage 14 process then you could look at a more regular 15 evaluation of the EIA frames and do that 16 first stage to assess whether the more 17 detailed second stage is necessary and that 18 might be a way, if it is costly, of assessing 19 whether the second more costly stage is 20 necessary. I mean, if the first stage 21 indicates there isn't much reason to believe 22 there is error then you may not need to go BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 165 1 into the more extensive second stage. So I 2 think that's what we discussed. 3 DR. HENGARTNER: Does anybody else 4 want to add to that summary? 5 MS. FORSYTH: I feel like an old 6 hand at frames. I've done frames before. 7 DR. HENGARTNER: Thank you very 8 much, Barbara. Well, Dr. Cleveland. 9 MR. CLEVELAND: When I was asked to 10 do this as a discussion I didn't think I was 11 going to have to summarize. I had generated 12 some issues or questions pertaining to Phil's 13 presentation. 14 DR. HENGARTNER: That's actually 15 quite appropriate. 16 MR. CLEVELAND: Well, I think one 17 of the fundamental issues was raised which I 18 think seems they have already been answered 19 by EIA is whether a dispatch model is the way 20 in which you want to try and approach 21 regional electricity forecasting. My opinion 22 is that it probably is and certainly will BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 166 1 enable you to compare your results to the 2 other structural models that are out there, 3 particularly in the private sector —————— 4 —————————— a number of important issues and, 5 of course, with these structural models the 6 parameters and how much you have to monkey 7 with them to get the model to balance is, of 8 course, where the rubber meets the road, and 9 I think with what I had to read it's a 10 difficult model to assess given the 11 documentation but I see a number of, I think, 12 areas where more information could be 13 provided in order for us to assess what's 14 happening. 15 There are two interrelated issues 16 of data aggregation and then the estimation 17 techniques. It would be useful to have a 18 table showing all the different types of key 19 data that you have and what their spatial 20 resolution is if it wasn't cleared. There 21 are lots of different data in here. It's 22 often times difficult to tell exactly what BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 167 1 level of spatial aggregation is because you 2 have generation data and cost data and then 3 you have utility data so you have all these 4 different layers of spatial resolution. It's 5 important to be able to judge the 6 appropriateness of your techniques without 7 having that information. 8 It also wasn't clear when you asked 9 the question, Phil, about your regression 10 techniques. I don't know what you did. Are 11 these panel data that you are estimating the 12 cost functions? Is that panel data? 13 MR. TSENG: It's just 14 cross-sections, one year. 15 MR. CLEVELAND: One year of 16 cross-section, and when you do have panel 17 data there, I mean, why not use panel data, 18 for example, using multiple years? Knowing 19 what data are available to even see whether 20 the techniques you are using are appropriate 21 is as important as knowing what the data are. 22 Are you using ordinary least squares, fixed BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 168 1 random effects, estimated panel data? There 2 are lots of choices and, again, it depends on 3 what questions you're really trying to ask. 4 The issue of calibration is also important 5 with these structural models. It's always 6 hard to know the —————————— issue is always a 7 difficult one. 8 Another possibility is to use some 9 kind of statistical technique to take an out 10 of sample forecast using one or two years 11 worth of data to see how your model will 12 actually predict some of these key parameters 13 in a statistical sense which would give you 14 some kind of statistical sense of how well 15 the model is doing. 16 I think another important area is 17 this whole fuel switching issue and there's a 18 fairly rich literature out there on how 19 people respond to prices in fuel choices and 20 some of the more recent work shows that what 21 we thought used to be the case is in fact 22 much more complicated. There are these so BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 169 1 called asymmetric responses to price changes. 2 Dermot Gately and some other people have 3 looked at that. So I think you want to look 4 at this fuel switching behavior to make sure 5 that the techniques and data used are 6 consistent with what economists are telling 7 us and what we've observed in the real world. 8 But I think this idea of the 9 optimizing behavior, the write-up that I read 10 and the graphs I saw and your verbiage seemed 11 very inconsistent. They may not be but on 12 one hand in the paper, the write-up, you're 13 saying we use regression techniques to 14 estimate this relationship between cost and 15 utilization but we see the relationships are 16 all over the map so how exactly did you 17 estimate those relationships and those 18 parameters and were those parameters from the 19 regression model what you used to calibrate 20 the model and how much did you have to change 21 those parameters that were used from the 22 regression? I mean, these are all nitty- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 170 1 gritty details but -- 2 MR. TSENG: It's a work in 3 progress. 4 MR. CLEVELAND: A work in progress, 5 I understand that but this is the kind of 6 information that we would need to be able to 7 give you some feedback into whether these 8 techniques are appropriate. When you look at 9 the data, as Mark observed, the data say non- 10 optimizing economic behavior but in fact 11 there probably is a fair amount of economic 12 optimization. In some ISOs I think they're 13 required by law. I think in New England, if 14 I'm correct, you're required by law to 15 dispatch by cost except under certain extreme 16 circumstances. So that's the kind of 17 information that we need to answer those 18 questions. 19 Well, I think it's a good start and 20 I think that the structural approach I'm not 21 daunted by the amount of data. I'm daunted 22 by perhaps the resources you might have BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 171 1 available to try and maintain this model, 2 which is a formidable one, and that's a 3 separate issue. 4 DR. HENGARTNER: Thank you very 5 much. Anybody else who was in this session 6 who wants to add to this very complete and 7 nice analysis? Everything has been said and 8 done? 9 MR. CLEVELAND: Everybody's hungry. 10 DR. HENGARTNER: Yes, well, thank 11 you very much for the summary of those 12 sessions. Before I move on and close this 13 morning session I want to do two things. 14 First I would like to invite the general 15 public to come up and speak out if they have 16 any comments, understanding that it's between 17 them and lunch, and the other thing, talking 18 about food, I have a note here saying that 19 the committee will have dinner at the 20 Lebanese Taverna, which is close to Woodley 21 Park. I have directions here. For now I 22 just need to have show of hands, who would BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 172 1 like to come, who wouldn't. 2 MS. KHANNA: Which one do you want 3 to see, to come or not come? 4 DR. HENGARTNER: The right hand 5 comes, the left doesn't. Hands are who 6 comes. 7 MS. KIRKENDALL: Plus one. 8 DR. HENGARTNER: Plus one? 9 MS. KIRKENDALL: My husband. 10 DR. HENGARTNER: Oh, very good. So 11 I'll let Bill explain you how to get there 12 since I'm not a native of Washington. 13 MR. WEINIG: Should I do that now? 14 MS. KIRKENDALL: Yes, please. 15 MR. WEINIG: I would recommend that 16 you go the Smithsonian Metro stop, which is 17 just a block out and a block over, and take 18 either of the two trains up to Metro center. 19 Transfer to the red line, Shady Grove, and 20 take the Shady Grove train up to Woodley 21 Park. There's only one exit at the Woodley 22 Park Metro stop and that should be taken up a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 173 1 couple of escalators to the street level. 2 Lebanese Taverna is across the 3 street and you can see its sign on the front 4 door directly across Connecticut Avenue. The 5 other way if any of you are driving is to 6 drive up to Connecticut Avenue and it'll be 7 on the right. Parking is at the first street 8 thereafter, one right turn and then another 9 right turn. 10 MR. CLEVELAND: What time are we 11 expecting? 12 MR. WEINIG: Well, I'm going to try 13 and establish a beachhead outside if the 14 weather permits at 5:30 because the 15 restaurant opens at 5:30 and we'll be able to 16 have, I think, a pretty pleasant evening as 17 we did the last time or the time before. 18 Otherwise the reservations are at 6:00. 19 Any questions? 20 (Whereupon, at 12:03 p.m., a 21 luncheon Recess was taken.) 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 174 1 A F T E R N O O N S E S S I O N 2 DR. HENGARTNER: Ladies and gents, 3 welcome back. 4 You go by Tancred? 5 MR. LIDDERDALE: Tanc, Tancred, 6 yes. 7 DR. HENGARTNER: Tanc is going to 8 give us some more STEO discussion about oil 9 and gas prices so welcome. 10 MR. LIDDERDALE: Thank you. What 11 I'm responsible for is putting together a 12 short-term regional model for propane and 13 heating oil residential prices. I'm covering 14 the propane and just heating oil residential 15 prices models now and I'm covering them 16 together because structurally they are 17 identical models. And the problems we have 18 in one model we have the same problems in the 19 other model. And I want to go quickly over 20 the structure of the models but more 21 particularly point out where I'm having 22 problems. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 175 1 Questions you brought up earlier 2 when Margot was up here, what are the data 3 limitations, and I'll show you some problems 4 I have in the data that I'm dealing with and 5 what implications and what problems that 6 creates on the modeling side. This is not as 7 ambitious a model as Phillip's electricity 8 model because propane and heating oil make up 9 each less than 10 percent, more like 5 10 percent, of the heating markets. We're 11 taking a much narrower view to what we are 12 modeling here. 13 For example, residential propane 14 makes up about 25 percent of total propane 15 demand and around 6 or 7 percent of the 16 heating market. Heating oil makes up around 17 10 percent of the total distillate market 18 and, of course, the big one in the distillate 19 side is on-highway and off-highway diesel 20 fuel. So we're looking residential prices 21 and they each are now a part of those two 22 markets. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 176 1 For an overview of propane 2 residential propane consumption is biggest in 3 the midwest followed by the south, less so in 4 the west and the northeast. One reason for 5 modeling residential propane prices is there 6 are large differences in propane prices 7 across regions ranging from an average $1.54 8 in the northeast to a $1.05 in the midwest. 9 So whereas we maybe reporting a $1.20 average 10 propane price for those in the northeast 11 you're wondering what the heck you're doing. 12 So hopefully going to the regional 13 gives us a bit better view of regional 14 differences but we're only doing the four 15 major census regions. We're not trying to go 16 down to the district levels so we're at a 17 much more aggregate although regionalized 18 level in these models. 19 On the heating oil side heating oil 20 is far and away dominated by the northeast. 21 Heating oil demand in the other sectors of 22 the country is much less. Not as much of a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 177 1 difference in heating oil prices across the 2 regions from a $1.38 in the northeast to a 3 $1.24 in the Midwest. Heating oil is going 4 to be much more closely tied to diesel fuel 5 prices so you don't have the big regional 6 price differences. Again, we're modeling 7 residential prices in the four major census 8 regions. We're not trying to bring it down 9 any further than that at least at this point. 10 So what I want to do is cover the 11 four major areas of the model, residential 12 consumption, inventories, wholesale price, 13 and retail price. Our target or objective on 14 these two models is really residential prices 15 and the estimation of consumption and 16 inventories is really in support of 17 estimating retail prices. 18 And the approach I'm taking is very 19 simple. It's a reduced ———— model. Since 20 these residential markets only make up 10 or 21 25 percent of the total demand we're not 22 trying to put together a comprehensive BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 178 1 regional supply and demand model. More 2 importantly, we don't have a lot of 3 information on breaking out the regional 4 flows and the demands in the other sectors. 5 Residential consumption, the 6 primary purpose is to weight residential 7 prices to come up with an annual average. We 8 have to weight them by the monthly volumes. 9 And also to create a US aggregate price where 10 we have to weight the regional prices by 11 demand. 12 We're using residential consumption 13 to weight regional prices, retail prices, by 14 region to come up with the US aggregate and 15 over time to come up with annual or quarterly 16 averages. As an example for the heating oil 17 model we're starting out with the bottom row 18 are exogenous variables for this particular 19 model. On the right we're talking about 20 crude oil prices as they feed into the 21 wholesale and retail prices, total US 22 distillate demand, which is estimated BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 179 1 separately, heating oil being only 10 percent 2 of total distillate demand. 3 Census region-state tax rates, 4 heating degree-days, and the last one is the 5 number of homes that use heating oil as their 6 primary source of heat, and we're taking 7 these exogenous variables and feeding them 8 into district inventories, wholesale price up 9 to retail price, regional demand, and from 10 those aggregating up to the US level. The 11 propane model is structured very similarly. 12 There may be one additional variable in 13 there, industrial production index for the 14 petrochemical market. 15 On residential consumptions, as I 16 said, we're using these residential 17 consumptions to weight the regional prices 18 and the three primary explanatory variables 19 in our demand equation are first the number 20 of homes that are using propane or heating 21 oil as their primary source of heat, number 22 two, the real residential price, number BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 180 1 three, heating degree-days where it's colder 2 or warmer you are going to consume more or 3 less and, four, we've got monthly dummies. 4 This is strictly a reduced-form linear 5 regression equation. 6 DR. HENGARTNER: Real estate price 7 being a proxy for the size of the houses? 8 MR. CLEVELAND: No, the real 9 residential price? 10 DR. HENGARTNER: Yes, why is that 11 important? 12 MR. LIDDERDALE: Trying to get a 13 demand elasticity where I would expect, and 14 this is where I run into one of my problems, 15 that if prices jump expected demand 16 elasticity will be very small. But this is 17 one of the problems I'm going to talk about. 18 MS. KHANNA: Can I ask a really 19 quick questions? 20 MR. LIDDERDALE: Sure. 21 MS. KHANNA: Have you in looking at 22 your demand model ever considered the age BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 181 1 distribution of the households? 2 MR. LIDDERDALE: Well, we're going 3 to point out a real problem that if you even 4 go into that detail on households there's no 5 way. When I approached this model one of the 6 problems is that demand is very seasonal. 7 During the winter propane demand or heating 8 oil demand is going to be up towards the 9 order of a million barrels a day. During the 10 summer it's on the order of a hundred 11 thousand barrels a day. So one of the 12 problems I faced is that when we're talking 13 about an increase in price the winter impact 14 is going to be a lot larger than the summer 15 impact. Same thing with the number of homes. 16 So my first stab at this was simply 17 splitting out the price series into a winter 18 series and a summer series and estimating 19 separate coefficients and, of course, this is 20 not a very good way of doing it and I'll show 21 you in a bit more detail what I mean here. 22 But I've got these slides for each of these BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 182 1 areas and we've got the estimated 2 co-efficient and a standardized beta co- 3 efficient. For the standardized beta 4 co-efficient it's a one standard deviation 5 and the explanatory variable times the 6 co-efficient yields, how many standard 7 deviations in the dependant variable. So the 8 standardized beta co-efficient is going to be 9 an order of magnitude of importance. 10 The winter price, very important as 11 an explanatory variable, a higher price, 12 lower demand. The number of homes is an 13 important explanatory variable. These are 14 all consistent with what we expect. The 15 summer variable's not very important because 16 you've got a small demand, you're going to 17 have a small change in a small number, and 18 it's going to be hard to show up. 19 First issue I ran into for 20 residential consumption is we've got to 21 change. Residential consumption is a sample 22 survey and not a universe and every several BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 183 1 years they change the survey frame where they 2 do a frame survey and from that frame survey 3 they take their sample. So consequently 4 there are changes in the data series every 5 few years or so. 6 The housing data, the problem is 7 we're only talking less than 10 percent of 8 the total market and again for the 9 residential energy consumption survey it's a 10 sample so we're taking the sample of a small 11 market, not a big problem in natural gas, a 12 big problem in propane and heating oil, and 13 I've got some numbers to show exactly what's 14 going on there. And, again, I mentioned the 15 problem with price elasticities. 16 Here's the slide that's I was 17 looking for. This shows you the seasonality 18 in demand for the middle Atlantic and the New 19 England census regions. And you can see up 20 through 2003, from 2002 to 2003, there was a 21 big jump in demand in both regions and this 22 can even be pinpointed to a number of states BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 184 1 whether it's New York, Connecticut, et 2 cetera. 3 All I can do is attribute this to a 4 change in the survey frame and the only way I 5 can deal with this is really I put in a dummy 6 variable for all months before 2003 and then 7 all months after 2003. So you run into this 8 kind of thing as you really have to look 9 closely at the numbers you're dealing with 10 and you can have a change in the series that 11 can't be explained by any change in market, 12 weather, or anything else. 13 The way I tried to handle this 14 winter-summer business, of course, it's 15 obvious that this is not going to work, is I 16 created a winter real price series and a 17 summer real price series but, of course, it's 18 not going to work when you're going from zero 19 to 67 cents a gallon. 20 And doing it this way, of course, 21 gives me a very high price elasticity. One 22 solution I'm going to be dealing with is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 185 1 estimating demand as a function of 2 expenditures where you're multiplying the 3 price times the demand so that in the summer 4 expenditures are going to be small, in the 5 winter expenditures are going to be large. I 6 did a quick test on that and it's giving me 7 much more reasonable price elasticities. I 8 would have expected -0.2 or even less than 9 that. 10 So this is a problem of bad 11 specification that's easy enough to get 12 around. But now we have the same question 13 coming up in the number of homes and the 14 number of homes presents several problems. 15 First the number of homes that use propane 16 and heating oil comes from a survey that 17 occurs only every four years, the residential 18 energy consumption survey. 19 So you can see we've got the 20 residential energy consumption survey data 21 points going 1989, 1993, 1997, 2001, I think 22 again 2004. We're coming up. So we've got a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 186 1 monthly model that's one of the key variables 2 only it carries maybe once in every four 3 years and the best we could do is interpolate 4 whether it is a linear. We could be fancy 5 and do a spline interpolation or something 6 else. 7 But nevertheless from this it looks 8 like okay, you can come up with some kind of 9 smooth series that is not going to give you 10 too much indigestion but if you look at it, 11 say, a different way between 1993 and 1997 12 the heating oil and propane market shares in 13 the Northeast census district were pretty 14 stable. All of a sudden into the 2001 survey 15 you've got a big shift and from 2001 on out 16 we are back to a stable market. 17 This doesn't give you a whole lot 18 of confidence so if we look further at the 19 data we can see that if we break out the 20 number of homes by propane and heating oil in 21 the northeast we've got four hundred thousand 22 homes. From the survey the 95 percent BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 187 1 confidence interval is four hundred thousand 2 homes plus or minus 50 percent, for heating 3 oil a little bit better, northeast 6.2 4 million homes plus or minus 24 percent, so 5 obviously we've got problems in using the 6 data. 7 Now, this brings up my question of 8 how do you relate demand to the number of 9 houses. What we would have done to get 10 around the problem of the winter-summer is 11 for the dependent variable have demand per 12 household. So it would have taken total 13 demand and divide it by the total number of 14 households. 15 Given the quality or the accuracy 16 of the data that's not something I could do. 17 I couldn't take total demand, which we'd 18 expect some reasonably high accuracy, and 19 divide it by the number of homes because a 20 sample survey has a low level of accuracy. 21 So I've got total demand regressed against 22 the number of homes and then I run into the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 188 1 problem where if I've got one additional home 2 the effect in the winter is going to be much 3 greater than the effect in the summer. 4 So I try to break out residential 5 consumption winter demand by house and 6 compare the reported number from the 7 residential energy consumption survey demand 8 per house to my regression estimates and the 9 reliability of these regression estimates 10 follows the reliability of the number of 11 households. 12 These are just the winter numbers 13 and if we add in the summer numbers the 14 summer numbers are just as bad or worse 15 because we've got low demands. So this is a 16 problem I'm not sure how to approach other 17 than ignoring the number of houses. Maybe 18 you have some other suggestion. 19 And I'd like to keep the number of 20 houses but still I don't know how to factor 21 out the winter from the summer effect without 22 having a winter series versus a summer series BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 189 1 which acts obviously is not the solution. 2 But, given that bad model, that mis- 3 specified model, I did a dynamic simulation 4 over the last four years and it's not a 5 particularly bad fit if looking at the upper 6 graph, comparing the actual to the forecast 7 demand, it's hard to see any real problem. 8 If you look at it in the lower graph as far 9 as the actual error you're going to have 10 higher percentage errors during the summer 11 months but even then we're running under 5 12 percent, a higher percentage error, because 13 we've got a small error in a small number. 14 And the winter errors are falling 15 much lower to 1 or 2 percent error so it 16 comes down to housing is a problem but it's 17 not a significant determinant of the model. 18 Prices are a problem but again not a 19 significant determination of the model. This 20 really comes down to it's a time series 21 seasonal model that is mildly affected by 22 prices and number of houses over time. So BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 190 1 that's it on the residential consumption 2 side. 3 Inventories, I don't have 4 inventories by census district so what I have 5 to do is model inventories by the Petroleum 6 Administration for Defense district, PADD 7 district, which have five districts in the 8 country. So there's not a perfect match-up 9 for inventories to census regions so I had to 10 make an approximation that in the northeast 11 that's going to be represented by PADD 12 District 1, which is the east coast, the 13 midwest is PADD District 2, which is the 14 midwest, so I got a rough match-up between 15 PADDs for inventories and census regions. 16 For inventories I'm estimating 17 inventories as first differences not 18 necessarily for statistical reasons but it's 19 convenient. Whenever I modeled inventories 20 as levels rather than first differences I 21 always had some problems jumping of the first 22 historical month. If the first historical BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 191 1 month was something of an aberration the 2 first forecast month came up with completely 3 unreasonable import numbers. So modeling 4 inventories ———————— first differences got me 5 out over that problem that I have much less 6 of a problem between the transition from 7 historical to forecast data. 8 So modeling inventories as a 9 function of heating degree-days, the 10 wholesale product price markup over crude 11 oil, your profit margin, beginning inventory 12 deviation from the four-year average, if 13 you're starting out with high or low 14 inventories that affects the inventory change 15 and again monthly dummies because it's very 16 seasonal. 17 For PADD 1, the east coast, heating 18 degree-days, the most significant variable, 19 as heating degree-days goes up it gets 20 colder, inventory change is smaller, or even 21 inventory draws are larger. The wholesale 22 product price to the WTI crude oil price, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 192 1 positive as expected but not statistically 2 significant. As wholesale prices rise a lot 3 more than crude oil price you get a bigger 4 inventory build or smaller draw, not 5 statistically significant at the 90 percent 6 confidence interval but according to the 7 standardized beta co-efficient it does have 8 some impact and consequently I'm leaving it 9 in there at this point just for that reason 10 and as the model changes make an assessment. 11 At some point you want to say well, if it's 12 not statistically significant take it out but 13 there could be something else affecting the 14 significance. To me it should be an 15 important variable just theoretically so I'm 16 leaving it in for that reason at this point. 17 DR. FEDER: It's not just that. I 18 mean, it could be that each one individually 19 is insignificant but together they are. 20 MR. LIDDERDALE: Sure. Beginning 21 inventory deviation, if stocks start high 22 then the build is smaller or the draw is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 193 1 larger. So the estimated coefficients are of 2 the right sign and the standardized beta 3 coefficients are not substantively 4 insignificant. 5 Wholesale price is where we start 6 getting into the important area since 7 consumption was used more for weighting, 8 inventories are more used as a right hand 9 side variable in prices, so now we're really 10 getting into the hearts of the propane and 11 heating oil models. We're starting out with 12 a wholesale price and a wholesale price is 13 estimated as differentials to crude oil 14 price. 15 Let me give a bit more explanation 16 of what I'm doing here. I originally modeled 17 wholesale price as a function of the current 18 crude price and a one-month lag of the crude 19 price because there is some delayed pass- 20 through. Even if it's only a week or two 21 that still has a month to month effect. The 22 problem I was running into was I would have BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 194 1 assumed that alpha 1 plus alpha 2 should have 2 added up to 1.0, that the total crude price 3 should be a one to one pass-through even if 4 delayed. 5 The problem I was running into is 6 that my estimated alpha 1 and alpha 2 were a 7 little bit larger than 1.0 and the result of 8 that was as crude oil prices have shot up 9 over the last year my forecast price premium 10 or the price differential for wholesale was 11 going up more than I would have liked. So 12 consequently my solution is either I can 13 constrain alpha 1 plus alpha 2 to equal 1.0 14 which is easy enough to do but an easier 15 solution as far as doing it in EViews takes 16 me a couple of less steps is I estimate 17 wholesale price to crude oil price difference 18 as a function of the current crude oil price 19 minus crude oil price lagged one month. 20 And what that does is the 21 specification constrain alpha 1 and alpha 2 22 to equal 1.0 and alpha 1 in the second BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 195 1 equation, of course, represents the impact of 2 the one-month lag price so that the current 3 price if alpha 1 is 0.15 the current month 4 crude oil price represents 85 percent of the 5 wholesale price and the one-month lag price 6 15 percent. So the distillate price premium 7 is estimated as a function of the WTI crude 8 oil price difference, total US distillate 9 demand, and beginning of month US distillate 10 inventories. 11 And for my standardized beta 12 coefficients, the WTI, the most important 13 variable turns out to be inventories and this 14 is total US inventories. I've tried 15 different PADDs but it turns out total US is 16 most important. When inventories are low 17 prices are higher or the price premium to 18 crude oil is higher and vice versa. 19 The WTI price difference, that's 20 where I'm saying the minus 0.175 represents 21 the lagged month impact on current wholesale 22 prices so about 82 percent of the distillate BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 196 1 wholesale price is impacted by current crude 2 oil price and the other 17 and 1/2 percent by 3 the one month lag. 4 US distillate demand, I could have 5 used heating degree-days and I probably ought 6 to go in and instead of using demand number 7 using heating degree-day deviations from 8 normal. I recall I tested that and there 9 wasn't much difference but nevertheless when 10 demand is higher whether it's from heating 11 degree-days or the demand number itself 12 prices or the price premium is going to be 13 higher. 14 The problem I have in this model is 15 what wholesale prices do I use. And I have 16 several options. There are lots of measures 17 of wholesale prices. For example, just 18 looking at the propane sector, EIA reports 19 from a survey a petrochemical sector price, 20 which is the largest demand for propane in 21 the country, almost 50 percent of the propane 22 goes to the petrochemical market. There are BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 197 1 many measures of spot prices out there 2 available. Mont Belvieu, Texas, and Conway, 3 Kansas, are the two major propane hubs in the 4 country. That's where most of the propane is 5 stored. 6 These are natural gas gathering 7 plants where you've got the natural gas 8 liquids plants. They're recovering propane 9 from the natural gas and you've got 10 underground storage in Mont Belvieu, Texas. 11 You've got single underground storage domes, 12 salt dome storage wells, that hold up to a 13 million barrels of propane each. So the spot 14 prices in Mont Belvieu, and Conway, Kansas 15 are important. 16 What I did was I tested wholesale 17 prices against each of these and found out 18 that the EIA petrochemical sector gave the 19 best regression results, the most stable 20 results, even feeding through to the retail 21 sector and it makes some sense in that as 22 spot price doesn't necessarily reflect the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 198 1 average price to the market. You can have a 2 high spot price at which you have few or no 3 sellers and you can have a low spot price 4 where you have a lot of sellers. Anyway the 5 petrochemical sector price actually 6 represents a consumption- weighted price and 7 using the petrochemical sector price to 8 explain regional retail prices gave much 9 better results than a spot price. 10 So now we get to the retail 11 residential prices and again I estimate these 12 as first differences, the difference between 13 the retail price and the wholesale market 14 price, so I'm estimating all the prices as 15 markup starting from wholesale over crude and 16 retail over wholesale. So the difference 17 between residential retail price and 18 wholesale price is a function of the 19 difference between the current and previous 20 month wholesale price, inventories, and 21 heating degree-days. 22 The most important and about the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 199 1 only variable you need to explain the retail 2 price is the wholesale price in just about 3 every region. Inventories have little effect 4 on the difference between retail and 5 wholesale prices. Weather or demand has very 6 little effect on the difference between 7 retail and wholesale prices in the propane 8 and heating oil residential markets. It was 9 a little bit of a surprise result to me, 10 surprised me a little bit, but it's a very 11 stable relationship between retail prices and 12 wholesale price. So once you've explained 13 the wholesale price you've pretty much 14 explained the retail price. 15 The problem that we have in the 16 residential retail price my big issue again 17 is data, that if we look at this first 18 difference or the difference between the 19 residential price and the wholesale price you 20 can see up through January '01 or at January 21 '01, around there, there was a big shift in 22 the series. For the US there was a 12-cent BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 200 1 jump in the retail-wholesale price 2 differential and that's reflected in each of 3 the regions. 4 You can't really explain it 5 theoretically or anything happening in the 6 market. Again it's a problem with we're 7 dealing with a sample that's reframed every 8 two, three, or four years so I've got to deal 9 with this through simply dummy variables. 10 And, that's it. Yeah? 11 DR. FEDER: I just want to refer to 12 the issue with the changing frames. 13 MR. LIDDERDALE: Yes. 14 DR. FEDER: Tanc, you probably know 15 the data much better than I so I don't know 16 if its feasible but I wonder if one could not 17 calculate the same estimates based on the 18 common part of the two frames just to 19 estimate the change of frame bias? 20 MR. LIDDERDALE: I would say you 21 could. The question is is it worth it and 22 can we catch the numbers that easily. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 201 1 DR. FEDER: I have a biased 2 preference here but I hold the view that 3 sometimes measure of change is much more 4 important than measure of level and 5 especially when you deal with data that are 6 so volatile and affecting the markets. So I 7 think yes, I think it's very important to 8 know if the change is a true change or is it 9 an artifact of the same. 10 MR. LIDDERDALE: That's an 11 excellent point. 12 MS. KIRKENDALL: A question, 13 though, is this from the 782 survey? 14 MR. LIDDERDALE: The prime 15 supplier? No, 782, yes, it is. 16 MS. KIRKENDALL: Because I thought 17 that they did a redesign of the sampling 18 methodology sometime too. It may be it was 19 in January of 2001. 20 MR. LIDDERDALE: Yes, I think 21 that's what it is. 22 MS. KIRKENDALL: And they do a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 202 1 frame every four years and that's one thing 2 but they also did a totally new sample 3 design. So that's probably a bigger thing 4 than the frame update. 5 MR. LIDDERDALE: Yes, obviously, 6 because we only have the one shift, the one 7 intervention. I attributed it to a change in 8 the frame but obviously I'm not that familiar 9 with the data and your point's well taken but 10 if it's a change in the sampling method -- 11 MS. KIRKENDALL: I only know that 12 they did have a big change in the sampling 13 method. They'd used the same method for 14 years and years and years and sometime in 15 that early 2000s they switched. It will be 16 interesting to find out if it was in January 17 '01 because I don't remember the date. 18 MR. LIDDERDALE: Most of the time 19 when I see something like this I go and say I 20 want the original data so I want to see 21 what's happening. Obviously I haven't done 22 this for propane and heating oil. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 203 1 DR. FEDER: I think it's too costly 2 and impossible to go back in time and do a 3 parallel run so you can measure the redesign 4 effect which sometimes people do but I -- 5 MR. CLEVELAND: I assume you have a 6 time machine. 7 DR. FEDER: We need a time machine 8 here but one could still take the old design 9 and just calculate the inclusion 10 probabilities for those units that were 11 included and this way estimate the effect of 12 the redesign. That may be too much, I don't 13 know, and maybe the historical value in '01 14 is not that important any more but in general 15 even if it might be prohibitively costly to 16 actually take an independent sample using the 17 two designs it's still not, I think, too much 18 to at least calculate the inclusion 19 probabilities and calculate estimates because 20 that's only math, its not actual collection, 21 and at least then you know what's happening 22 because I deal sometimes with measuring BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 204 1 change over time and I'm frustrated where you 2 can do it because they changed and we are 3 dealing with the same issue now right now. 4 MR. LIDDERDALE: Another issue too 5 is that the difference between pre- and post- 6 January '01 is more than simply a level 7 shift, that the whole seasonality could have 8 changed so it's almost that you want to start 9 the estimation period at January '01 or 10 compare regressions of the two periods 11 separately to see if in fact it's simply a 12 level shift or you've got changes in other 13 underlining other underlining character- 14 istics. That's something I haven't thought 15 of or pursued. 16 DR. FEDER: Just to repeat an 17 argument or to explain more, when I look at 18 the data that I have here and I see that the 19 US uses 1.07 billion tons of coal to me it's 20 almost like an incomprehensible number. It's 21 the mass of the sun but if you tell me that 22 that increased by 2 percent over the last BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 205 1 year that I can understand and that's why I 2 claim that level doesn't mean much to me. 3 Change is very easy to understand and relate 4 to. 5 MR. LIDDERDALE: Thanks. 6 DR. HENGARTNER: In some way this 7 might suggest that if you could normalize 8 these numbers by something, I mean, like a 9 ratio estimate, going just back to what Moshe 10 knows about, then it might actually be able 11 to disappear these artifacts because, I mean, 12 you think well, I'm measuring something 13 slightly different because I have a different 14 sampling strategy but by measuring more than 15 one thing I'll make sample units. 16 And so, the hope is that maybe 17 there is some characteristic with which these 18 numbers are associated that could explain 19 this dramatic shift. I don't know what it 20 is. It's a wild thought that if you have 21 time to look into sometimes because it seems 22 to almost —————————— they just blow it up a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 206 1 little bit because ———————————— mean shift 2 like Moshe said but there is also a shift in 3 variance, it looks like. 4 MR. LIDDERDALE: It's definite. 5 MR. CLEVELAND: The whole 6 statistical issue is a huge one here and the 7 issue of levels. A lot of these data that 8 you're estimating in levels have time trends 9 and there is no talk here about whether there 10 is a stochastic ———————————— time trends of 11 these data. If the variables co-integrate it 12 seems like you are using ordinary least 13 squares which may be, frankly, completely 14 inappropriate if these things have stochastic 15 time trends. You need to test these data for 16 the presence of stochastic time trends. If 17 they do you need to go on to do co- 18 integration techniques if you're estimating 19 the levels. 20 MR. LIDDERDALE: But the 21 differences don't have that time trend so we 22 are back to the problem with the consumption BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 207 1 and if I was able to do consumption per 2 household that would take care of my problem, 3 most likely, but I can't do that so there are 4 multiple problems. 5 MR. CLEVELAND: Any dependent 6 variable that are estimated in levels has 7 this issue and so I think it's something that 8 you need to look at. You have a lot of 9 insignificant coefficients. Something's off 10 somewhere. 11 MR. LIDDERDALE: Well, running into 12 the insignificant coefficients, I had the 13 problem in consumption, obviously. That's a 14 level with a time trend. The first 15 difference is are there wholesale over crude 16 oil and retail that don't have a time trend 17 problem there. But I had insignificance on 18 the retail over wholesale which was a 19 surprising result but actually makes sense 20 that those variables affect the wholesale 21 market and not the retail. So I've got 22 problems that I need to find a solution for. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 208 1 Otherwise we can spend $100 million 2 on the residential energy consumption survey. 3 That would take care of my problem. 4 MR. BERNSTEIN: Well, I suppose 5 this is my frustration on the time with this 6 and this is back to Nancy and Bill at some 7 level because I ask this every single time, I 8 think, in the five years I've been here. I 9 don't know what your objective is. I'm not 10 sure what you are trying to do here so I 11 don't know what to tell you to do on this 12 because I really don't know what you're 13 trying to estimate, what your end goal is, 14 and maybe that would help us decide what your 15 best approach to the problem is. 16 MR. LIDDERDALE: Our goal was to 17 forecast regional residential prices. Number 18 one, there are large differences in 19 residential pricing across regions, propane 20 going from a $1.05 up to a $1.50-something 21 per gallon. We want to be able to target 22 because these residential prices are used in BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 209 1 a number of groups, for example, the low 2 income housing energy assistance program. 3 MR. BERNSTEIN: So your goal is to 4 estimate regional prices for propane and 5 heating oil? 6 MR. LIDDERDALE: Exactly. 7 MR. BERNSTEIN: So the first step, 8 I think, is what you've been trying to do, 9 though I'm not sure why this way, is to 10 understand on propane why there are regional 11 differences, number one, and then on heating 12 oil what drives wintertime heating oil price. 13 First of all it seems to me you can ignore 14 all the summertime stuff for heating oil. 15 I'm not an econometrician so I'll be 16 corrected but I'm not sure why you would even 17 bother running those months at all in the 18 model. I would just run a winter heating oil 19 model. 20 But I'm also wondering if the only 21 way for you to get that data is interpolating 22 between RECS years maybe one needs to step BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 210 1 back and get a more fundamental understanding 2 of what drives the distillate price and then 3 thinking of a different way to differentiate 4 the residential portion of that. 5 I mean, that's what you try to do 6 but I'm not sure that the RECS data gets you 7 anywhere there and I'm not sure the RECS data 8 is going to get any better. We'll hear about 9 that later, I mean, and so the question is 10 distillate price, what's driving that price 11 in the wintertime and what portion of that is 12 residential and is there a different set of 13 data. 14 We all read about heating oil 15 prices and if it gets cold the heating oil 16 prices go up but is it being driven by demand 17 which you're indicating maybe not or is it 18 driven by expectations in the market, and I'm 19 not sure how you model that, where people are 20 seeing winter prices going up and so the 21 winter cold and so they're driving the base 22 price up. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 211 1 So I don't know what to tell you 2 here but I think that if we go back to my 3 goal is price and my goal for heating oil is 4 Northeast winter heating oil let's then focus 5 on that and then figure out with the best 6 data you can what's the best way to 7 understand what drives those prices and I'm 8 not sure you are there yet. 9 And then on the propane all you're 10 really interested in, it appears at this 11 point, is regional differences and what 12 drives those regional differences. And maybe 13 there is a different more aggregate approach 14 to drive that. My guess is not just 15 residential propane prices that differ by 16 region but also the petrochemical prices and 17 every thing else and so do you need to 18 differentiate the residential out of that to 19 understand what drives differences in prices 20 between regions. I don't know the answer to 21 those questions. 22 MR. CLEVELAND: In his defense, I BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 212 1 mean, as a home heating oil consumer, 2 probably the only one in the room, you did 3 find, I think, that inventories were 4 significant and important? 5 MR. LIDDERDALE: At the wholesale 6 price. 7 MR. CLEVELAND: It's important to 8 know what that elasticity is. 9 MR. BERNSTEIN: But heating 10 degree-days was not. 11 MR. LIDDERDALE: That's because in 12 the wholesale price I used demand rather than 13 heating degree-days. 14 MR. BERNSTEIN: But the residential 15 heating oil price to wholesale price per 16 heating degree-days didn't make a difference. 17 MR. LIDDERDALE: Right, and that's 18 the surprising result. 19 MR. BERNSTEIN: Well, that's 20 probably because the wholesale price is going 21 up the same way the heating oil price is 22 going up too so you're probably not getting a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 213 1 difference. 2 MR. LIDDERDALE: Well, the 3 wholesale price is a US average wholesale 4 price and so that's why I used aggregate 5 demand instead of US heating degree-days or 6 something but what we're saying is that 7 wholesale price responds to weather and just 8 because you're cold in the northeast and warm 9 in the midwest doesn't mean the residential 10 price over the wholesale price, that premium, 11 goes up any more in the northeast. To me was 12 a surprising result. I didn't expect that. 13 And the whole effect on demand 14 shocks weather shocks manifests itself in the 15 wholesale price which to me is an interesting 16 result in itself. 17 MR. BERNSTEIN: But if I take a 18 look at that and if I assume in this one that 19 it's just a frame or whatever issue there 20 that if I look at those patterns and I just 21 shift them it seems to me over time the 22 difference between residential price and BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 214 1 wholesale price really hasn't changed much at 2 all in that '95 to 2005 time period. 3 So what it tells me is that the 4 fact of the matter is that what's going to 5 drive the price is not the fact that the 6 northeast is cold is going to affect the 7 residential price in the northeast. It'll 8 affect the wholesale price because traders 9 are looking at that cold weather and they're 10 going to bring up their wholesale price 11 regardless of demand in anticipation of 12 potential higher demand. I don't know. 13 MR. LIDDERDALE: In the wholesale 14 price —————— I use demand as maybe a proxy 15 for weather or weather as a proxy for demand, 16 either way. 17 MR. BERNSTEIN: And, now, the 18 residential price is also calculated, right? 19 It's not data? 20 MR. LIDDERDALE: No, it's a EIA 21 survey, the residential price? It's from an 22 EIA survey. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 215 1 MR. BERNSTEIN: So when I look at 2 this regression on the retail residential 3 price of standardized beta coefficients so 4 you've got the wholesale price first 5 difference that the residential heating oil 6 price minus wholesale price spread is data. 7 Everything there is data? 8 MR. LIDDERDALE: Yes. 9 MR. BERNSTEIN: Including the 10 census region? 11 MR. LIDDERDALE: Yes. 12 MR. BERNSTEIN: So you don't really 13 need demand in there at all? 14 MR. LIDDERDALE: No, I don't. 15 MR. BERNSTEIN: But the only thing 16 that was significant was -- 17 MR. LIDDERDALE: The wholesale 18 price. 19 MR. BERNSTEIN: The wholesale 20 price. 21 MR. LIDDERDALE: It's something 22 that I hadn't been prepared for, that on the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 216 1 retail level inventories don't matter and 2 weather doesn't matter because that's all 3 taken in at the wholesale level. 4 MR. BERNSTEIN: Is there a 5 correlation between your calculated demand 6 and the census region heating degree-days? 7 MR. LIDDERDALE: Yes, there was. I 8 don't have the demand —————————— results in 9 there. 10 MR. BERNSTEIN: I'm looking at the 11 demand to the heating degree-days. Did you 12 do that? 13 MR. LIDDERDALE: I think I did. 14 MR. BERNSTEIN: That will tell you 15 somewhat how close you're -- 16 MR. LIDDERDALE: We've got the 17 demand which, of course, this is the 18 problematic regression here but consumption 19 heating degree-days statistically significant 20 current month and prior month, the prior 21 month because while you've got your 22 residential tanks you may not get them filled BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 217 1 up till the next month so there could be a 2 lag shock. 3 But statistically significance and 4 apart from the price, which was a problematic 5 variable, so given a better specification 6 it's going to change a little. But heating 7 degree-days does affect demand in a direction 8 expected and that's about all I can say at 9 this point. 10 DR. HENGARTNER: Winter number of 11 homes, that you said was actually a very 12 unreliable number? 13 MR. LIDDERDALE: Well, it's worse 14 than that. Instead of winter real price we 15 call it winter number of homes and that's 16 what the series looks like. So the series is 17 mis-specified and I don't have a better 18 specification because I can't create demand 19 per household. And I can't do that because 20 of the unreliability for this particular 21 market sector. 22 DR. HENGARTNER: Statisticians have BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 218 1 worried about this kind of a problem. They 2 call it errors in variables. 3 MR. LIDDERDALE: Yes. 4 DR. HENGARTNER: And the impact of 5 various in variables, and I think economists 6 are strong on that, is that you have an 7 attenuation. That means the impact on your 8 regression coefficient is that the estimates 9 are smaller than they ought to be and maybe 10 that explains why when you just use standard 11 regression methods you look at the estimates 12 and, lo and behold, they're not significant. 13 Well, yes, what do you expect? I mean, you 14 had too much noise in the co-variants. It's 15 a dangerous thing to make those adjustments. 16 You make an adjustment based on assumptions 17 that are hard to verify but this may be one 18 of the causes of why something that you 19 expected to happen didn't. 20 MR. LIDDERDALE: But as we're 21 talking about that retail price all of that 22 to me is much more reliable data. So it's a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 219 1 good point and it's going to have some effect 2 but I'd say it's a surprising result. 3 They're not significant but I could 4 rationalize it. 5 It's something I wasn't expecting 6 but it's a nice result. I would point out 7 for natural gas and electricity, of course, 8 these 95 percent confidence intervals are 9 much better because we're talking a very 10 small share of the homes in this residential 11 energy consumption survey. 12 MR. CLEVELAND: You should probably 13 throw those data out. 14 MR. LIDDERDALE: That's probably 15 going to be my next step. 16 MR. CLEVELAND: Is there some kind 17 of proxy variable that we can think of that 18 would maybe contain some of that information 19 in it that you could use? 20 MR. LIDDERDALE: The only thing I'd 21 say is you throw in a trend variable but now 22 we get into this trend problem -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 220 1 MR. CLEVELAND: Please, we have 2 enough econometric issues here, I think, 3 without doing that. 4 MR. LIDDERDALE: But this is trying 5 to capture the trend. 6 MR. CLEVELAND: What do the error 7 terms on these things look like? You were 8 estimating a couple of these equations in 9 levels and -- 10 MR. LIDDERDALE: Well, this is the 11 level. This is the one equation that's 12 estimated a level. 13 MR. CLEVELAND: You need to test 14 whether these variables have time trends, 15 whether they're stochastic or deterministic, 16 so you know what techniques you should be 17 using. 18 MR. LIDDERDALE: The answer is yes, 19 I'm sure they do. 20 MR. CLEVELAND: So then we 21 shouldn't bother spending time looking at 22 this because you probably need to use some BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 221 1 other techniques to estimate -- 2 MR. LIDDERDALE: Well, I bring this 3 up because I know I need a different 4 technique. I bring this up because of my 5 problem with the number of homes. I could 6 resolve this problem by demand per household. 7 MS. KHANNA: No, even then you 8 can't, no. Even if you have demand per 9 household those issues don't go away, not at 10 all. 11 MR. LIDDERDALE: As a dependent 12 variable? 13 MR. CLEVELAND: As long as you're 14 estimating the levels it's the same issue. 15 MR. LIDDERDALE: But what I would 16 say is demand per household I would fully 17 expect not to see a trend problem there. 18 MS. KHANNA: You still have to test 19 it, though. 20 MR. LIDDERDALE: I expect it to be 21 stochastic. 22 MR. CLEVELAND: If you're BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 222 1 estimating levels as time series data you've 2 got to test for —————————— 3 MR. LIDDERDALE: I'm saying I would 4 expect that I wouldn't have a problem but 5 here I definitely have a problem. 6 MR. CLEVELAND: Go back to the 7 homes. You would know this better. Are 8 there other data out there that might be a 9 proxy that might be a proxy that would 10 contain some of that information? 11 MS. KHANNA: That's actually what I 12 wanted to say. Some other federal agency 13 does this, collects these data, the number of 14 new housing starts, the total number of 15 homes, and it comes out every year and even 16 more frequently. 17 MS. KIRKENDALL: Because this is 18 the number of homes that use propane as their 19 main heating fuel. 20 MR. BERNSTEIN: But isn't there 21 data at the state level on the share of 22 electricity and natural gas for heating? I BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 223 1 thought AGA used to do that. Maybe there 2 isn't. See, and then what's left over is 3 heating oil and we actually have that. 4 There's wood too, I guess, so -- 5 MS. KHANNA: How much wood, just 6 very small? I use wood, wood and natural 7 gas, but I still think it's a small number of 8 homes. Actually it's only in the more rural 9 areas, I think. In the major population 10 centers it's not wood. 11 MR. LIDDERDALE: Well, it's 12 probably banned in most metropolitan areas 13 now. 14 MR. BERNSTEIN: But I suppose the 15 question is do you really need this. If you 16 can sufficiently estimate price in the 17 wintertime by using inventory and wholesale 18 prices and heating degree-days if heating 19 degree-days seem like a good enough proxy for 20 demand the one thing I would test on heating 21 degree-days is what degree did you use as 22 your -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 224 1 MR. LIDDERDALE: Sixty-five. 2 MS. KHANNA: Sixty-five. 3 MR. BERNSTEIN: We've done some 4 interesting stuff where we've actually 5 changed that and see big differences and what 6 happens if it's 60? So I'd think about 7 looking at the sensitivity of that variable 8 at some level. I would certainly forget all 9 the summer stuff. If you're only worried 10 about winter and if you're talking about the 11 short-term energy you don't have to do it 12 monthly. Just pick your time and say okay, 13 at what point in time in the future do we 14 need to know what to expect out there in next 15 winter and do that. 16 And then the heating oil, that's 17 easy. On the propane it's almost similar. 18 You're really only caring about the other 19 relationship so also I think you don't need 20 to care so much about the number of 21 households. You need to try to uncover 22 what's driving the price difference. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 225 1 I would want to know if there's a 2 price difference in the other prices as well, 3 not just residential prices, on propane. Do 4 all the other users of propane have the same 5 types of differences? 6 MR. LIDDERDALE: On consumption if 7 we don't have to use as an explanatory 8 variable in prices of heating degree-days 9 this works better. The only reason we're 10 doing consumption is to weight prices to come 11 up with US aggregates or an annual average. 12 So the thing is I'm not going to lose a whole 13 lot of sleep over it. If I cut this to the 14 bone, take care of basic problems, I'm not 15 going to do a lot of research into the number 16 of homes. And it's -- 17 MR. BERNSTEIN: Do you need a US 18 average heating oil price since it only makes 19 sense for one region anyway? 20 MR. CLEVELAND: You watch it. 21 MR. BERNSTEIN: But it's important 22 for that region, no question, but do you need BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 226 1 a US average? 2 MR. CLEVELAND: What does a US 3 average mean? 4 MR. BERNSTEIN: And do you need a 5 US average residential propane number if 6 you're going to do it regionally anyway and 7 you don't need that household number for 8 doing it regionally then forget the US 9 average. 10 MS. KHANNA: There are some energy 11 models somewhere that's got heating oil in 12 his or her model and wants an average number. 13 MR. LIDDERDALE: No, that's an 14 excellent point. 15 MR. CLEVELAND: It's a good 16 question though, I mean, average crude oil 17 price makes sense for the US because that's 18 such a ubiquitous, geographically speaking, 19 product. But with something like home 20 heating oil where all the —————— questions 21 inherently had some geographic focus maybe 22 going through these things just to get a US BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 227 1 average may not be worth the effort. 2 MR. LIDDERDALE: If we're using 3 retail price to measure an average 4 profitability for the country for the 5 industry then you may want a US average but 6 we're not really doing that. 7 MS. ANDERSON: We typically don't 8 report that kind of information. What we 9 typically report in the wintertime is some 10 sense of what regions are paying for heating, 11 and convert that to an expenditure. But what 12 we try to derive from the national data now 13 are expenditures by regions so clearly our 14 focus is on getting regional richness about 15 heating fuel. 16 So trying to come up for an 17 arbitrary reason for a national number may be 18 completely unnecessary. You just focus on 19 the regions when you're talking about winter 20 fuel season. 21 MR. LIDDERDALE: Just because we've 22 reported it before doesn't mean we have to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 228 1 continue. 2 MS. ANDERSON: That's absolutely 3 right. 4 MR. LIDDERDALE: It would take care 5 of almost all of my problems. 6 MR. CLEVELAND: Actually we don't 7 need your position any more. 8 MR. LIDDERDALE: Oh, but I still 9 have to come up with an annual average for 10 the region. 11 MR. CLEVELAND: There are lots of 12 problems out there. 13 MS. ANDERSON: Mark, you had 14 another point you were making aside from how 15 many variables do you need to predict the 16 price and I lost track of it but I thought it 17 was a -- 18 MR. BERNSTEIN: Was it the regional 19 differences on propane and whether you saw 20 that difference amongst all the other prices? 21 I mean -- 22 MR. LIDDERDALE: The other sectors. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 229 1 MR. BERNSTEIN: The residential 2 price. 3 MS. ANDERSON: The other sectors, 4 yes. 5 MR. BERNSTEIN: The overall sector 6 price because if you find similar differences 7 among, say, petrochemical or other uses of 8 propane then you don't need to go through the 9 residential analysis. I mean, in propane 10 what do you have? You have the chemical 11 sector and the commercial sector. You 12 probably have better price data there and do 13 you see similar regional differences? 14 MR. LIDDERDALE: Excellent point. 15 I'd go ahead and point out media problems. 16 The petrochemical sector's located in 17 Houston, Texas, no outside but for a 18 commercial sector, looking at the others, 19 investigating that is certainly a good point. 20 DR. HENGARTNER: But the same 21 thing, if instead of looking just at propane 22 let's look at the ratio of the markup, for BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 230 1 example, for propane over natural gas. I 2 mean, aren't the profits and the markups the 3 same? Maybe you don't need to model all 4 these things. Maybe ———————— able to borrow 5 strength from other energy sectors instead of 6 consumption sectors. 7 MR. LIDDERDALE: Well, propane is 8 fascinating in its relationship with crude 9 oil and natural gas which I didn't get into 10 because it's related to both. Now, if crude 11 oil is moving up faster or higher than 12 natural gas, propane moves up with it but not 13 in lock step. 14 MR. BERNSTEIN: But the question is 15 between regions are there similarities to 16 other fuels, do you see similar patterns of 17 regional differences, and try to use that to 18 help explain it and then figure out how to 19 run the regression, get that understanding 20 first. 21 MS. KHANNA: And actually that ties 22 on to a question that I have. You were very BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 231 1 surprised that the retail residential prices 2 were explained only in your model by the 3 wholesale price differences. Why were you 4 surprised by that? 5 MR. LIDDERDALE: Because this cold 6 weather in the northeast I would have 7 expected, say, inventories to go low in the 8 northeast relative to the Gulf Coast which 9 would put pressure on the wholesale price but 10 it puts even more pressure on the retail 11 price. That's why I expected some response 12 in the markup and I didn't see that. 13 MR. BERNSTEIN: I think what you're 14 seeing in terms of the trading anticipation 15 there is that once they see those stocks 16 going down the wholesale price drives up. 17 There's no room for the retail price to go. 18 Once they drive the wholesale price up the 19 market won't accept more. The retail guys 20 are getting squeezed all the time. You hear 21 the same thing from refiners. The refining 22 guys say that they're squeezed all the time BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 232 1 because when everybody sees a shortage in 2 gasoline what happens is crude oil prices go 3 up, gasoline prices go up, but to the same 4 ratio and the refiners continue to get 5 squeezed. They don't make more money. 6 And I think you're having the same 7 thing in the residential as you're having in 8 the gasoline and it's the market anticipation 9 at some level driving the base price up and 10 so you don't have any room for that. So I'm 11 not surprised by that. 12 MS. KHANNA: Right, I wasn't 13 surprised either but he says he was. 14 MR. LIDDERDALE: Well, I was 15 surprised but I agree with you. Thinking 16 about it, that certainly makes sense. 17 MR. CLEVELAND: Well, after the 18 fact, it's a lot easier for us to -- 19 MR. BERNSTEIN: Before I would have 20 said, of course, demand goes up, residential 21 heating oil goes up, and the differential is 22 going to be greater. But actually I wouldn't BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 233 1 have because I've been looking at the 2 gasoline-crude oil differential lately. 3 We've been looking at —————————————— 4 MR. EDMONDS: That refinery analogy 5 is an excellent one because it's the same 6 market phenomenon. 7 MR. BERNSTEIN: We are running over 8 here. 9 DR. HENGARTNER: Yes, we are but 10 we're waiting for our testing colleagues to 11 come up here. 12 MR. LIDDERDALE: Are there any more 13 questions? 14 DR. HENGARTNER: Thank you very 15 much, nice presentation. 16 (Recess) 17 DR. HENGARTNER: Next item on the 18 agenda is the discussion by Mark Bernstein of 19 the break-out session. Mark? 20 MR. BERNSTEIN: They had some 21 issues with some of the data that they were 22 using and how to deal with it. One issue BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 234 1 came up about the potential change in the way 2 the data was collected and change in the 3 frame which could cause some problems. And 4 there was a bunch of discussion about making 5 sure you use changes or differences rather 6 than absolute levels, particularly if you got 7 time trend issues and at least a test for 8 time trend in the data. 9 We also talked about focusing on 10 the output. If you're focusing on certain 11 things you could probably get away with not 12 using some of that bad data and try to find 13 alternatives to it, given what your focus 14 objective is, separating out the winter and 15 summer effects, and looking across the 16 regions basically taking a step back and 17 looking at it in different ways, given what 18 your ultimate objective is. I know that's 19 where we came down on it. Anybody want to 20 add anything specific? So I think you got 21 it. And it might actually be easier when we 22 do it this way, hopefully. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 235 1 DR. HENGARTNER: Thank you very 2 much, Mark. We'll go right away with the 3 next item on the agenda which is a discussion 4 follow-up on frame team activities by Howard. 5 Welcome. 6 MR. BRADSHER-FREDRICK: Today I'm 7 going to talk about the most recent 8 activities of the frames team. You'll 9 remember in the 2004 sessions some of us 10 presented information about EIA frames 11 activities. Also you heard a presentation 12 from the Census Bureau this morning related 13 to frames comparisons. 14 This is an outline of my 15 presentation. I'll begin by discussing some 16 background information. Then I'll remind you 17 about previous efforts of the frames team. 18 After that I'll talk about the more recent 19 efforts of the frames team including our work 20 on frame sufficiency and the results of that 21 work and from there I'll talk about possible 22 new directions for the team and finally get BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 236 1 to the questions for the committee. 2 The EIA strategic plan provides the 3 basis for the frames team's activities, it 4 provides the impetus for increasing 5 efficiencies in frames usage, and it calls 6 for performance measures concerning 7 sufficiency of frames. The actual language 8 in the strategic plan is percent of EI survey 9 frames with sufficient industry coverage to 10 produce reliable supply, demand, and price 11 statistics. 12 I'll briefly summarize the previous 13 frames team efforts. At both of the meetings 14 last year there were presentations discussing 15 frame-checking activities at the respondent 16 level. That work included comparing National 17 Renewable Energy Lab plant-level data with 18 EIA electric power plant data. It also 19 included checking petroleum supply data with 20 MECS consumption data and comparing EIA power 21 plant data with EI coal consumption data. 22 Shawna Waugh discussed the proposed BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 237 1 work for conducting data comparisons with the 2 Census Bureau and you heard an interim report 3 about that this morning. Some data 4 comparisons at the aggregated level were also 5 discussed. We compared petroleum marketing 6 data with consumer price index data. 7 The balancing item with respect to 8 natural gas supply versus disposition data 9 was also discussed and for price data we 10 described our comparison of volumes that the 11 price data represented in comparison with 12 total volumes. This was done with natural 13 gas commercial price data. And at the last 14 fall meeting we discussed the dual system 15 methodology. As was recommended by the 16 committee, we'll be presenting our work on 17 this topic at the FCSM research conference 18 this November. 19 Now I'll discuss more recent 20 activities of the frames team. During the 21 last six months of last year we evaluated all 22 of EIA's master frames for sufficiency. We BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 238 1 employed these criteria, the existence of a 2 systematic maintenance approach, and you 3 notice that these are fairly qualitative 4 things, frame volatility over time, the use 5 of comparable frame listings from other 6 sources, the existence of a balancing item, 7 the magnitude and stability of the balancing 8 item over time, the relative concentration of 9 volumes in a few respondents, and the changes 10 in the industry or changes to legislation and 11 regulations, also independent assessment by 12 the survey manager of the frame's quality. 13 So we had to go and collect a lot 14 of information in addition to fairly a basic 15 frame information and if you've seen the 16 paper on the Intranet that Bill had out, I 17 mean, there are long appendices in the paper 18 and these are all included in Appendices A 19 through D but basically the information is 20 stability or volatility of the frame, the 21 corresponding useful listings of frame units, 22 whether EIA has access to them, the balancing BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 239 1 item, stability, and relative concentration 2 of volume in a few respondents to the skewed 3 distributions. 4 So we looked at a total of 34 5 frames and five were deemed to be out of 6 scope so we looked at 29 that we regard as 7 within the team's purview and we deemed 25 8 sufficient and four were deemed either 9 insufficient or don't know and those four 10 were the EIA 815, the Monthly Terminal 11 Blenders Report. This is a fairly new frame. 12 It's difficult to identify blenders of 13 gasoline and there is no reliable 14 comprehensive registry for comparison 15 purposes though we know that the EPA is 16 presently working on one. 17 The EIA 910, the Monthly Natural 18 Gas Marketers Report, is also a fairly new 19 frame. Again, it's difficult to identify 20 marketers and new states are being added 21 often so this frame is in a state of flux. 22 Also the EIA 886 is the Annual Survey of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 240 1 Alternative Fueled Vehicle Suppliers And 2 Users and there's been difficulty in 3 identifying and collecting data from fleets 4 owned by private nonutility companies and 5 local governments and these fleets could have 6 quite a few alternative fueled vehicles. EIA 7 is presently looking at changes to the law 8 that would facilitate easier and more 9 accurate data collections on the 886. 10 Concerning the EIA 3 this is the 11 Quarterly Coal Consumptions and Quarterly 12 Report, Manufacturing Plants. The balancing 13 item shows that this frame is fairly good but 14 there have been some questions regarding 15 whether procedures are in place to maintain 16 quality over time and, as was mentioned this 17 morning again, the Census Bureau will shortly 18 be conducting a data comparison with the 19 EIA 3 so we'll get some more information on 20 that. 21 The limitations of the frame 22 sufficiency approach, by the time we BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 241 1 completed our study we realized that there 2 could be alternative ways of proceeding that 3 might be even perhaps more useful to EIA. 4 EIA may want to determine sufficiency by 5 survey rather than by master frame. 6 Let me explain a little bit. 7 Subsets of some EIA master frames serve a 8 number of different surveys. For example, 9 the 860, the Annual Electric Generator 10 Report, serves as a master frame for a number 11 of surveys including the 920. This is the 12 Combined Heat and Power Plant Report. While 13 the team deemed the 860 sufficient it may be 14 that the 920 frame would have to be more 15 thoroughly scrutinized for sufficiency if we 16 reevaluate it separately. 17 In terms of the pros of changing 18 the approach it could be that it'd be more 19 useful to EIA to look at individual survey 20 frames. And in terms of the cons we'd be 21 looking at perhaps 67 surveys as opposed to 22 29 master frames. And also there may be some BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 242 1 question from the strategic plan about what 2 we really mean about frame sufficiency. Is 3 it to look at master frames or what is it 4 that we're really trying to accomplish with 5 this? 6 In terms of the new activities of 7 the frames team we got started in February 8 with another team essentially adding on to 9 work that's already been going on including 10 the sufficiency work but the new charter that 11 we developed has these objectives to work 12 toward a long-term goal of enhanced 13 commonality in information technology, to 14 identify opportunities to share frame 15 information, to monitor the quality of EIA 16 frames, recommend needed enhancements to 17 frames, recommend surveys to be discontinued 18 or that results tables be provided with 19 appropriate caveats, also to recommend 20 further studies needed to evaluate frame 21 sufficiencies, and to prepare updated 22 recommendations for the frame sufficiency BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 243 1 performance measure. 2 We've met several times through 3 March and we had a short interim report which 4 was essentially discussing the 5 recommendations of what the team's activity 6 should be in the future. And we prioritized 7 some things and we gave a high priority to 8 completing and maintaining an inventory of 9 EIA frames. We also thought that we should 10 conduct quarterly meetings or special 11 meetings as the need arises among affected or 12 interested offices to aid and frame 13 information sharing and we should also have 14 as a high priority to study and update frame 15 documentation standards. 16 In terms of medium priority items 17 we recommended to develop best practices for 18 system maintenance, support, and flexibility 19 of system designs with regard to frames and 20 to look into replacing the existing standard 21 on frames maintenance and develop a revised 22 version. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 244 1 The questions for the committee, 2 are there any suggestions as to how to 3 proceed in determining sufficiency of frames 4 in the future is question number one, and, 5 number two, are there any comments or advice 6 regarding the current frames team's 7 recommendations. Should I call on people? 8 MS. KIRKENDALL: Sir? 9 MR. CLEVELAND: Oh, that was from 10 last time. 11 DR. HENGARTNER: Does anybody have 12 any comments, ideas? 13 MR. BRADSHER-FREDRICK: Frame 14 sufficiency question? 15 MR. SINGPURWALLA: Can I ask a 16 question in return? I think this relates to 17 the final question in the earlier break-out 18 session, should the frame be reviewed every 19 five years, and I was wondering did the team 20 think about it, the frames team think about 21 that question, and what is your thinking? 22 MR. BRADSHER-FREDRICK: I don't BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 245 1 think we've really gotten into details of 2 that type. I mean, one of the questions, of 3 course, that we have is well, how often 4 should we even be looking at the sufficiency 5 question. I mean, it is in the strategic 6 plan so we are going to have to do this on a 7 periodic basis. 8 But, I mean, if you looked at the 9 70 pages you'd see about half of this as 10 tables and this was a big effort to put this 11 together. I mean, it may be that the first 12 time is the hardest time and things will get 13 easier with future iterations but this is 14 this is still a resource-intensive thing. 15 And while some good things may come out of it 16 it may be something we don't want to do too 17 often. 18 MS. KIRKENDALL: I think the team 19 that did the evaluation recommended that it 20 be done at least no more often than every 21 three years. They felt three years might not 22 be a bad cycle for taking a look at the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 246 1 frames. That didn't have to do so much with 2 getting the Census Bureau to look at specific 3 frames. That was an internal EIA process. 4 DR. HENGARTNER: Was there any 5 thought when you updated or changed a frame 6 to carry over the old frame at least for one 7 more cycle or have the frames overlap because 8 we have had that issue several times and it 9 comes up again and again? Each time you 10 changed a frame or you changed something the 11 whole series shifts and it would be nice to 12 have some idea on how that occurs. 13 MS. KIRKENDALL: I don't think that 14 Tanc was right on that. I really don't think 15 that it was a frame change. There was 16 probably a survey change. But I think there 17 needs to be more looking at that because 18 usually when we do these frames maintenance 19 it doesn't make that big a difference. You 20 usually just find a few more units to add. 21 MR. CLEVELAND: I used a DOE series 22 on oil and gas reserves. In 1977 there was a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 247 1 major change where historically time series 2 data on the amount of oil and gas discovered 3 in the United States by different types of 4 categories and for years, going back to the 5 '40s, the American Petroleum Institute 6 published this data. 7 In 1977 when the DOE was formed 8 there was a shift. API said DOE's going to 9 do this. We're going to stop doing it. And 10 the ways in which they collected the data 11 were fundamentally different. So if you 12 plotted the two series and they overlap by a 13 couple of years there are these big 14 differences and this was a very widely used 15 series, high profile series. 16 And so actually EIA, I believe, 17 supported two studies, one by Richard 18 Nehring, an independent consultant in oil and 19 gas, and another one internally where they 20 came up with two different fudging factors, 21 algorithms, to link the two series using 22 different assumptions. One was a more BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 248 1 statistical based and one was a subjective 2 thing where Richard Nehring went in on a 3 field by field level and tried to calibrate 4 the two. 5 And so something like that in 6 series that are really important it might be 7 worth your while to try when there's a break 8 in the series to actually pay some attention 9 to bridging the series in some kind of 10 meaningful way that statisticians can look at 11 and understand because that work was really 12 important that they did in that particular 13 series. 14 DR. HENGARTNER: And the problem is 15 you never know in advance if the changes 16 you're going to do are going to come and bite 17 you in the back. 18 DR. FEDER: And it doesn't really 19 require having a separate sample. You just 20 need to have a separate set of rates 21 calculating using the two frames so you can 22 compare the estimates without needing to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 249 1 actually go through new units. 2 DR. HENGARTNER: We just need to be 3 able to do it over a period of time, have 4 both frames in existence so you can get that 5 conversion factor. 6 Any more questions and comments? 7 DR. FEDER: I have a question. Do 8 you get some information from, let's say, on 9 a given survey or a census from another 10 regarding the sufficiency of the frame? 11 Because some of the data collections are 12 related such as suppliers and consumers and 13 so on is there a way you can assess the 14 adequacy of your frames using that way? 15 MR. BRADSHER-FREDRICK: Well, we 16 have done some of this. I mean, some of 17 these presentations, say, last year there 18 were some comparisons that were mentioned 19 like the MECS versus petroleum supply and 20 things of this type. 21 MS. KIRKENDALL: We've had some 22 feature articles that talked about that kind BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 250 1 of comparison, also the balancing item that 2 we've talked about, and you didn't 3 particularly like that but that's trying to 4 measure the same concept in different ways 5 from different surveys so then if you get 6 close balancing supply from one survey and 7 consumption from another it gives you a 8 warmer feeling about you're actually getting 9 the information you want from the two 10 surveys. I mean, in almost all of the fuel 11 areas they spend a fair amount of time 12 looking at balancing items to try to make 13 sure we -- 14 MR. BRADSHER-FREDRICK: It might be 15 interesting when Census does the EIA 3 16 because it seemed to us like the balancing 17 item showed the data to be fairly good. If 18 this is also Census's conclusion ———————— 19 DR. SITTER: You did a lot of work 20 here and overall, at least from what's 21 presented, it seems like you frames are 22 reasonably good. Is that a fair conclusion? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 251 1 MR. BRADSHER-FREDRICK: I would say 2 so. I mean, surely there are some problems, 3 as we point out, but I think they're in 4 pretty good shape but we just need to keep 5 after it. And it could be that some of these 6 suggestions we made about what the team might 7 do to try to get better cooperation between 8 offices so that people learn from each other, 9 this could be helpful too. 10 DR. SITTER: Well, I think that you 11 also learn what kinds of things, what kinds 12 of situations, you're having frame problems 13 in like new frames, for example, and certain 14 things and so I think that the important 15 thing is to be intelligent and there's no way 16 you can do this every year or every three 17 years. I mean, it's just too intensive, I 18 imagine, too costly, and there's no reason 19 to. 20 MR. BRADSHER-FREDRICK: It's also a 21 burden on the survey managers to be 22 interviewed and dredge up the information BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 252 1 that we're asking. 2 DR. SITTER: So I think that 3 basically you've got to use a quality control 4 strategy that attacks the big problems with 5 more energy, and the small ones less 6 frequently. That's the most important. 7 Otherwise you're just going to be spending 8 way too much effort on things that you're 9 pretty sure are not a problem. 10 MS. KIRKENDALL: So we might 11 conclude once we're done with this that maybe 12 our frames are okay and move on to some other 13 aspect of quality that probably isn't so 14 good. 15 DR. SITTER: Well, I think I would 16 look at it more on a survey by survey basis 17 or frame by frame basis but clearly there are 18 some well-established ones. You're pretty 19 sure the frame isn't changing quickly. You 20 can determine that fairly fast. Then 21 certainly that shouldn't be revisited as 22 often as where you know that you've got BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 253 1 something that's changing or, like you said, 2 the ones you've identified you probably could 3 have identified a priori at least three of 4 them. 5 You already knew it was hard to get 6 at these frames. These are hard frames so 7 you knew that those you should be focusing 8 on. And I think that you've probably learned 9 a little bit about certain specific things 10 where you probably didn't know you should 11 have been focusing on those. And I think 12 putting your effort where you know you have 13 problems is certainly more useful. 14 MS. KIRKENDALL: Yes, we looked at 15 all of them but any of them that have 16 regulation or somebody that regulates 17 utilities we'd go to that organization and 18 get the list of plants so it's really pretty 19 easy to keep them up to date. 20 DR. SITTER: So you don't want to 21 keep doing that. 22 MS. KIRKENDALL: So you don't worry BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 254 1 about that because we know people are keeping 2 it up to date in fairly routine ways. 3 DR. SITTER: It's like if you do 4 the six sigma stuff for quality control when 5 they come in to an industry the very first 6 thing that they say is take your quality 7 characteristics and histogram them and those 8 causing you the most grief you attack first. 9 It's very simple but you get a lot of 10 low-hanging fruit that way and I think that 11 cost versus benefit is really the thing when 12 it comes to quality. 13 DR. FEDER: Howard, I may have 14 misunderstood one of the ideas that you had 15 there but, using surveys as frames, what did 16 you mean by that? 17 MR. BRADSHER-FREDRICK: Well, we 18 really meant that some of these surveys are 19 taking pieces of various frames and even 20 doing cutoff sampling and things like that 21 and they may or may not be doing it quite the 22 right way in some cases. And, like I've BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 255 1 mentioned the 920, I think this is something 2 that a lot of people bring up, that it could 3 be that it's hard to identify. We may really 4 be missing some combined heat and power 5 plants. The 860 looks pretty good but it 6 could be that that section of the 860 isn't 7 so good. So some of the subsets may not be 8 so great. 9 DR. SITTER: You mean in actually 10 obtaining the correct subset from an uncut 11 frame? 12 MR. BRADSHER-FREDRICK: Yes, and 13 how things are drawn from the -- 14 MS. KIRKENDALL: No, the 860 is 15 both a frame and it's an annual survey and it 16 goes down to generator level and gets detail 17 about electricity generation. And the frames 18 team particularly was looking at it's just a 19 highly skewed distribution. Are you 20 capturing most of it? And we almost 21 certainly are because there are so many huge 22 generators. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 256 1 So the survey itself is fine and 2 it's a fine frame to use for monthly surveys, 3 a frame, which is the list of respondents and 4 perhaps those that are not active now. If 5 you're going to the big guys but if your 6 focus, like the 920, is on combined heat and 7 power plants that's a focus on the companies 8 that are at the small end of that frame. And 9 that's where I'd expect that frame to be less 10 complete just because it's harder to get a 11 complete list of little companies than it is 12 to get a list of the bigger companies. 13 DR. SITTER: That's an issue 14 between coverage in terms of item-wise 15 coverage versus coverage in terms of —————— 16 of the volume. 17 DR. FEDER: So what's the proposal 18 then? 19 MS. KIRKENDALL: Oh, well, my 20 proposal is that we really need an evaluation 21 of the frame for each survey. So the frame 22 for the 920 is a subset of the 860 and the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 257 1 860 should be evaluated as a frame for the 2 920 separately rather than saying is the 860 3 a good enough frame. And that's not how we 4 approached it this time but I think that's 5 not so bad. We had to start somewhere but 6 maybe in the future we need to go back. 7 Now, it turns out that that's what 8 the Census Bureau is going to be doing for us 9 anyway. So we are going to get that 10 evaluation back from them. 11 DR. SITTER: And that falls into a 12 similar category. You obviously have a lot 13 of intuition as to which one of your frames 14 are going to be a problem and you should be 15 using that and those you should be checking 16 first. 17 DR. HENGARTNER: Not only that but 18 you also know how much trouble they're going 19 to cause. And so a blanket statement saying 20 let's revise them every three years is 21 probably not the right answer. It's going to 22 depend on, as Randy said, which frames change BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 258 1 a lot and which are more stable. 2 MS. KIRKENDALL: So we need to be 3 smarter about what we look at. 4 DR. SITTER: And some of the work 5 is as a one-shot deal in some sense; that is, 6 if it is that you've got poor coverage now 7 but it's fairly stable over time then what 8 you really need to do is fix it now and then 9 you don't have to check it that often. But 10 others that may have good coverage now but 11 are not stable have to be checked a lot. It 12 just depends. 13 DR. FEDER: I don't think my 14 question is fully justified in this context 15 but I just wonder do you have any 16 longitudinal component of any of the surveys? 17 Is any unit observed for more than one time 18 deliberately to measure change or anything 19 like that? 20 MS. KIRKENDALL: Most of our 21 surveys are longitudinal panel surveys. We 22 have a wealth of data at the company level. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 259 1 If the company's been in business we've 2 probably been harassing them for years and 3 years and years, monthly and annual and -- 4 DR. FEDER: And the reason I ask 5 that is because one of the things that are 6 easy to miss is new what they call birth and 7 death of units, plants, and whatever and I 8 think in the business world there's also, of 9 course, the issue of mergers and acquisitions 10 and change in organization and splits too. 11 MR. BRADSHER-FREDRICK: That was 12 some of the data that we collected in those 13 tables, for each of the frames, births and 14 deaths and mergers, how many do you see per 15 time period, to give us some sense as to the 16 stability of the frame over time. So that 17 was a factor that we looked at in terms of 18 assessing sufficiency. 19 DR. SITTER: Is there any auxiliary 20 piece of information that you collect in your 21 surveys that would give you some information 22 on each frame changing within that particular BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 260 1 area or are you already collecting something 2 that is a reasonable indicator of an area 3 where frames are changing? 4 MS. KIRKENDALL: Well, I mean, 5 certainly companies tell us when they're 6 going out of business, when there are changes 7 in ownership, that sort of thing, especially 8 on the census surveys. 9 DR. SITTER: Doesn't give you much 10 in terms of birth. 11 MS. KIRKENDALL: Doesn't give you 12 much in terms of birth. 13 DR. SITTER: Birth would be driven 14 by certain economic situations or things 15 though, I imagine, so you might be able to 16 almost -- 17 DR. FEDER: Randy, birth could also 18 be a company that's not doing any energy- 19 related activity or anything significant 20 shifting to that sector. 21 DR. SITTER: Yes, but they don't 22 shift there for no reason. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 261 1 DR. FEDER: No, no, of course. 2 DR. SITTER: So a sector that's 3 growing is going to be more likely to have 4 births than one that's not, right? So I 5 think that they must have some indicators 6 which at least would suggest the worst-case 7 situations. You already do, obviously, 8 because at least heuristically you're making 9 statements about some of these things like, 10 oh, this wasn't unexpected and we would 11 expect this to be a frame that isn't that 12 good. And those are really being driven by 13 your general knowledge of that area and that 14 situation and I think you just need to pay 15 attention to that and attack those first. 16 DR. HENGARTNER: What's interesting 17 is you actually know about deaths in the 18 print. 19 DR. SITTER: The deaths you know 20 about. 21 DR. HENGARTNER: The deaths you 22 know. That's important information so the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 262 1 only thing we need to worry about are the 2 births. 3 MS. KIRKENDALL: People do read the 4 energy journals and whatnot in their areas to 5 try to keep up. Frequently they're written 6 up in the industry press. 7 DR. SITTER: And most of you are 8 knowledgeable about those. I think that just 9 the important thing the things you're worried 10 about are the things that probably you're 11 worrying about for a good reason and you 12 should take advantage of that and not blindly 13 evaluate frames making a statement like we 14 should evaluate all our frames every five 15 years. That's silly. 16 DR. HENGARTNER: It's probably an 17 overkill. 18 DR. NEERCHAL: Does the Census 19 Bureau put out any summaries where there have 20 businesses registered on an ongoing basis? 21 MS. KIRKENDALL: They do but they 22 don't directly relate to our frames. Shawna BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 263 1 can probably talk about this better than I 2 but when they did the refinery comparison 3 last year we had a list of 155 refineries or 4 something. The Census Bureau had 350. When 5 we finally came down to things there were 6 five that maybe they thought could have been 7 on ours and our people thought that they 8 shouldn't be and they were very small ones. 9 But it took a lot of work to get down to that 10 point. So just looking at counts from the 11 Census Bureau, they really have to get down 12 and look at unit by unit to assess it. It's 13 a lot of work. 14 DR. HENGARTNER: Well, thank you 15 very much. 16 MR. BRADSHER-FREDRICK: Well, thank 17 you. 18 DR. HENGARTNER: Well, we're in a 19 nice position. We are almost on time and in 20 front of a break. I want to remind everybody 21 that after the break there's another 22 break-out session. Mark, Cutler, Jae, Moshe, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 264 1 and Neha, and myself will go downstairs to 2 the basement. 3 (Recess) 4 MS. KIRKENDALL: Would somebody, 5 either Susan or Darius or Barbara, like to 6 summarize this session? Johnny was 7 originally assigned to do that and he's not 8 here. Or Randy. 9 DR. SITTER: I think that Susan 10 just because she's experienced and she's not 11 assigned to another one. Barbara's already 12 done one. 13 MS. SEREIKA: I'll try. How's 14 that, not knowing I had to do this one? 15 MS. FORSYTH: We'll support you, 16 Susan. 17 MS. SEREIKA: Chime in please as I 18 try to figure out what's going on. 19 MS. KIRKENDALL: With that why 20 don't we introduce Dwight French? 21 MR. FRENCH: Oh, you mean I'm 22 supposed to go up there now? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 265 1 MS. KIRKENDALL: Oh, yes, this is 2 EIA's proposed strategy for addressing 3 declining response rates in the Residential 4 Energy Consumption Survey. 5 MR. FRENCH: Thursday, 3:20 p.m. I 6 was working the violent crimes desk out of 7 the Forrestal Precinct. My associates were 8 members of the ASA Committee. My name is 9 French. We were supposed to investigate an 10 assault this afternoon but this time instead 11 of trying to figure out who did it or 12 capturing the perpetrator we're actually 13 going to try to aggravate the assault because 14 in this case the victim deserves to be 15 assaulted. The victim, nonresponse in the 16 EIA's Residential Energy Consumption Survey. 17 For those of you who are too young 18 to know what in the heck this guy is doing 19 this was supposed to be a parody on the old 20 cop show Dragnet. 21 Now the serious part of the 22 presentation begins. In fact we are going to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 266 1 look at what we intend to be a rather 2 comprehensive attack on a problem that has 3 been developing over time. Certainly this is 4 not something unique to our household survey. 5 It's happening in household surveys across 6 the federal government and certainly in the 7 private sector as well. 8 The Residential Energy Consumption 9 Survey, to give you an idea for those of you 10 who may be new to the committee, is one of 11 EIA's set of end use consumption surveys, 12 that is, surveys of the demand side of the 13 energy equation, surveys of consumers of 14 energy directly. 15 And we do our household energy 16 consumption survey as with the others with 17 the intention of gathering characteristics of 18 the population of energy consuming units 19 under study and marrying that to information 20 about their energy consumption and 21 expenditures so that you'll have a basis for 22 economic and societal understanding of what BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 267 1 is going on as opposed to just accounting for 2 energy. 3 And the RECS is our household part 4 of that set of surveys. It goes out to what 5 will be in our next survey, which is going to 6 be conducted beginning in August of this 7 year, this time a set of 4,000 households, 8 nationwide. Our sample design, as always, 9 will be a multi-stage area probability sample 10 survey and after we do the characteristics 11 collection we ask the respondents to the 12 household interview to sign a waiver allowing 13 us to go to their energy suppliers and gather 14 the consumption and expenditures information. 15 That is historically the way that the survey 16 process is run. 17 Well, we do have, as with other 18 household surveys, a problem with nonresponse 19 and since we have two surveys as part of the 20 RECS system, the household survey and the 21 supplier survey, we can get nonresponse in 22 RECS when the household interview cannot be BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 268 1 obtained for whatever reason. If we get the 2 household interview but the respondent 3 decides they don't want to sign the waiver to 4 allow us to go to the energy supplier or if 5 we get all of that the energy supplier we may 6 not get the consumption and expenditure data 7 from them for whatever reasons and we'll get 8 into that in a few minutes. 9 You will notice over time that the 10 household interview response rate has been 11 drifting down slowly over the past 20-plus 12 years. They drop at the very end from 78 and 13 1/2 to 72.8, looks a little bit more 14 concerning but I will tell you that there was 15 a cost problem with the 2001 RECS. 16 And part of the way that the cost 17 problem was addressed was that the contractor 18 did not make what we would consider to be an 19 equivalent attempt to get the last few 20 percentage points of response that one might 21 have done in our previous surveys and so I 22 would suspect that if you were talking about BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 269 1 equivalent effort you probably would have 2 ended up with 75 or 76 percent response, 3 perhaps, rather than 73 but that still 4 continues the slow, slow drift of response 5 rates. 6 And along with that we had a 7 corresponding uptick in nonsigning of 8 authorization forms. I tried to put a 9 different background on this so it would show 10 up a little bit better but we've got the 11 white background here. These are for 12 different fuels. You can't see a couple of 13 the lines but it's fair enough to say all of 14 them really track along each other together. 15 Again, there is a greater uptick in non- 16 signing rates for the authorization form in 17 the last survey in 2001 but, again, probably 18 due to a lack of conversion effort and so 19 chances are it shouldn't have looked quite as 20 bad as that under comparable effort but 21 that's what we have. 22 Similarly if you look at the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 270 1 percent of personal interview nonresponse, 2 that is, refusals and unable to contact, you 3 will notice that the trend is the same but 4 we're having proportionately a greater 5 acceleration in the rate of unable to 6 contact. And again we're all aware of that, 7 the fact that people are out of their houses 8 more, they're more difficult to reach, and so 9 forth. 10 And so when you put it all together 11 at the various stages we have to use and we 12 do use regression as an imputation technique 13 to impute for consumption and expenditures 14 when we don't actually get the consumption 15 and expenditures. And I'm emphasizing 16 consumption and expenditures here because we 17 consider that to be the core collection in 18 our survey. The characteristics go along but 19 the consumption and expenditures are really 20 the core of the survey. 21 Well, obviously nonresponse at any 22 stage of this effort affects the quality of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 271 1 the complete product so what do we want to 2 do? Well, let's start with the recruitment 3 process. As I said, we're considering a 4 number of initiatives here. 5 We're starting out with a plan for 6 a supplemental contact even before we send 7 the letters out to the household. We're just 8 going to send an introductory postcard out 9 warning people that information is coming and 10 that you are going to be part of a household 11 survey. 12 After that as a result of 13 suggestions from our esteemed new office 14 director Margot sitting over there said these 15 letters really seem bureaucratic. She wanted 16 something more uplifting and upbeat. Well, 17 we're probably not going to be quite as 18 spiritually oriented as the seminar I was 19 just at over the lunch hour by a fellow who 20 is plumping solar photovoltaics but hopefully 21 our letters will be more upbeat and 22 encouraging and eliminate bureaucratic BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 272 1 recitation of the information which we do in 2 fact have to provide to respondents. 3 We will mention gifts as a token of 4 thanks and I'll be talking about that in a 5 moment and one other thing that we are going 6 to mention quite heavily and this, of course, 7 plays into at least one part of people's 8 concerns which is security in the 21st 9 century is security of their information, 10 privacy of their information. 11 As at the end of 2002 the President 12 signed a new law into place called the 13 Confidential Information Protection and 14 Statistical Efficiency Act, CIPSEA. The 2005 15 RECS will be collected under this 16 confidentiality protection. 17 For the first time it gives us the 18 capability and the responsibility, the two 19 sides to the coin, to totally and 20 unconditionally protect any identifiable data 21 we have in-house. We've already used this 22 for the 2003 Commercial Building Survey, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 273 1 which was the first survey that we ran and 2 may be one of the first surveys in the 3 federal government that was run under CIPSEA 4 protection, and we did a lot of editing work 5 in bringing identifiable data in-house to 6 work on which we had never done before. 7 Well, what are we going to do about 8 providing incentives to complete the 9 household interview and sign the waiver form? 10 Well, we're going to have information about 11 RECS. We're going to have brochures on 12 energy saving methods. Stephanie Battles, 13 our RECS survey manager, is working on 14 collecting those types of things from the 15 Office of Energy Efficiency upstairs and 16 wherever else she might be able to get them. 17 Also when we ask people to sign the 18 waiver form we will have a pen to hand them 19 and they will be allowed to keep the pen when 20 they've signed the waiver form. And on top 21 of that we are proposing in the packages at 22 OMB right now for their consideration a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 274 1 monetary incentive for this survey. As you 2 know, OMB's record on approving monetary 3 incentives and their attitude toward monetary 4 incentives has been mixed, to say the least. 5 There were times when they felt that it was 6 an 1nappropriate thing to do but they 7 realized the realities of the circumstances 8 in which we work. They are certainly willing 9 to consider it. We do have a proposal in for 10 that for the current survey and we have not 11 gotten any indications from OMB to this point 12 that they are unfavorably disposed toward it 13 so we think that it may be quite likely that 14 they would approve it. 15 We are proposing the monetary 16 incentive as an experiment. It will be 17 unconditional. It will be given out when we 18 contact the household and set up the 19 interview. One third of the households will 20 receive nothing, one third will receive $5, 21 and one third will receive $10. 22 Just FYI, the interview is expected BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 275 1 average about 45 minutes in length. And so 2 these are the incentive levels that we've 3 tried. We will be giving cash, as I 4 understand it. We're not going to be giving 5 phone cards or anything like that. And the 6 classes in which we will implement the 7 experiment will be at the segment level. 8 I said earlier that our survey is, 9 like other household surveys, done by 10 personal interview, a multistage survey. At 11 the segment level we will have maybe three or 12 four households on average per respondents 13 per segment and we will be doing the levels 14 of incentive per segment. So we avoid the 15 neighbor effect where John says hey, I was 16 interviewed by these people. They gave me 17 $10 to complete the interview, and Fred down 18 the street says well, I was interviewed too 19 but I didn't get anything, and that doesn't 20 look too good. So we will be doing this 21 segment by segment. 22 Next thing that we're considering, BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 276 1 we are developing a new design for this 2 household interview survey. It's going to be 3 used for the next several RECS but it's 4 developed using post-2000 census information 5 to upgrade our survey to represent the 6 current population mix and distribution in 7 the United States. 8 We know from the development of 9 this design that in certain areas it is 10 likely that Spanish-speaking households will 11 pose a significant problem. There will be a 12 number of places where a lot of the 13 households will be limited or perhaps no 14 knowledge of English among the adults who are 15 eligible to complete the survey form. 16 Well, you can always try to fudge 17 your way around the problem by asking the 18 teenage children if there are any to 19 translate or do some other types of things. 20 In the past we have done a Spanish 21 translation and the interviewer has fumbled 22 around with the paper translation. This time BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 277 1 and we operate our survey, by the way, using 2 CAPI, using a Blaise questionnaire and 3 control system, we are going to have the 4 survey translated into Spanish on the CAPI 5 system. So the interviewer will be bilingual 6 in these areas. Our survey contractor, 7 National Opinion Research Center is going to 8 be hiring bilingual interviewers in those 9 areas. All they have to do is hit the 10 Spanish version, up it comes, and they run on 11 and it'll be entered into Blaise just the 12 same way as anything else so we don't have to 13 worry about supplemental coding or keying or 14 anything like that. Everything, of course, 15 goes from the Blaise system directly into a 16 central data processing and away we go. 17 You'd like to do this with Korean and 18 Japanese and some other languages too but 19 they're really not frequently occurring 20 enough to cost-effectively justify the 21 procedure at this point. 22 Of course, we know that despite our BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 278 1 best efforts people will at least initially 2 sometimes refuse. And we know from past 3 experience that there are sets of reasons why 4 people don't go along with the survey. One, 5 of course, is security and confidentiality 6 concerns, another is lack of time, another is 7 uncertainty about whether the survey is 8 really needed or what the data are used for. 9 And then there are other people who just 10 generally try to put you off. 11 Because we have something close to 12 real-time understanding from the case 13 management system of what these non- 14 respondents are like we will be able to 15 customize and send out customized conversion 16 letters to these people and they're going to 17 be worded differently depending on what we 18 understand from the interviewer the major 19 concern was by the people who refused. 20 We have not committed to implement 21 this yet but we are considering that for 22 initial soft nonresponses, that is, where BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 279 1 they don't come to the interviewer with a 2 shotgun, that we may sub-sample nonresponse 3 if we find our initial response to be low 4 enough that we want to focus our efforts and 5 concentrate on a smaller number of people 6 with more intensive follow-up. 7 And what, obviously, we're going to 8 be doing is for the committee to talk about 9 whether these methods are appropriate, what 10 additional methods, and what supplements you 11 might suggest to what we're providing here. 12 But one other thing I'd like a comment on 13 from the committee is the idea that if you 14 sub-sample nonresponse you then weight the 15 respondent's double for purposes of counting 16 your nonresponse rate in the end. If the 17 committee has anything to say about that I'd 18 be glad to hear about it. 19 Now let's go to the next issue, 20 which is missing energy supplier data. This 21 is a separate problem. There are several 22 causes. One, of course, is you can't get BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 280 1 there because the household respondent 2 refuses to sign a waiver allowing you to get 3 the information. Sometimes the supplier is 4 inaccessible and this is especially true in 5 the case of fuel oil and LPG people. 6 Sometimes you mess up the account 7 number. Somebody switches digits or 8 something on you or for some reason the 9 supplier says this isn't any account that I 10 recognize or it isn't the account for the 11 person you've got here or whatever. 12 And then, of course, there is the 13 problem of master metering in apartment 14 buildings where you have nothing that 15 corresponds to the unit that you actually 16 selected for the survey which is an 17 individual housing unit. 18 What we're going to do in this case 19 is arm the interviewers with electronic 20 scanners. And we are going to ask the 21 respondents ahead of time to have their 22 latest energy bills handy and the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 281 1 interviewers are going to scan a copy of 2 those bills in and the idea is if you can do 3 this you can get the supplier names and the 4 account numbers correct. And if any of you 5 have looked at your energy bills sometimes 6 your electric or natural gas bill will 7 actually have a year's worth of consumption 8 on it so even if something goes wrong later 9 on down the line and we don't connect with 10 the supplier somehow we still have a year's 11 worth of consumption information. 12 Bills ordinarily don't have 13 expenditure information on but they do have 14 the consumption information. And if we have 15 the consumption information we're really more 16 than halfway home with regard to C and E and 17 we can do a good job of imputing expenditures 18 from consumption if we have it. So you can 19 even think of this as a partial effort to get 20 part of our consumption data from the 21 household respondent rather than from the 22 supplier. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 282 1 Lastly, master-metered apartment 2 buildings are still a problem. We do a 3 rental agent survey in cases where we do 4 household interviews in apartment buildings. 5 It's a separate little survey that we do and 6 the interviewers will take their scanners 7 there and see if they can get information at 8 the building level, not the housing unit or 9 the household level, in that rental agent 10 survey. 11 Once again we get supplier and 12 account information identified clearly and if 13 there is consumption information on the bill 14 either that or if we can get to the supplier 15 with that account information and have the 16 agreement of the rental agent to get it we 17 can then get the appropriate information and 18 prorate that energy use for the sample 19 household. That is the operation that we 20 ordinarily go through when we have a master- 21 metered situation because obviously you're 22 never going to get the energy consumption BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 283 1 relative to that particular unit. 2 You saw the concern we have about 3 nonresponse at the beginning of the 4 presentation. We take it seriously. We want 5 to do something about it. And what we're 6 asking the committee at this point is you've 7 seen all of the major things that we're 8 planning on doing to try to address non- 9 response. Do you have any improvements? Do 10 you have any changes? Do you have any 11 additional procedures at this point you'd 12 like to suggest? That's what we're looking 13 for. And if you'd like to comment about the 14 formula for estimating nonresponse based on 15 sub-sampling of nonrespondents I'd be glad to 16 hear about that too. 17 Thank you for your attention and 18 now I guess I turn it over for comments by 19 the group. 20 MS. KIRKENDALL: To the committee. 21 DR. SITTER: Why $5 and $10? 22 MR. FRENCH: As opposed to -- BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 284 1 DR. SITTER: Anything else. 2 MR. FRENCH: An interesting 3 question. First of all there are monetary 4 concerns, budget concerns. There always are. 5 I suppose an alternative could be 0, 10, and 6 20. 7 DR. SITTER: An alternative could 8 be anything. The question is what was the 9 rationalization for the choices made? 10 MS. SEREIKA: How long is the 11 survey? Is it maybe there -- 12 MR. FRENCH: It's 45 minutes on 13 average. 14 MS. SEREIKA: It's a 45-minute 15 survey. 16 MR. FRENCH: We considered those 17 amounts to be reasonable amounts, at least 18 the upper one, the $10, a reasonable amount 19 to compensate people for their effort in 20 this. I mean, you could say you ought to pay 21 somebody $100 to do this but it's -- 22 DR. SITTER: No, I was just BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 285 1 wondering If there had been any work in 2 determining what would change somebody's 3 mind? 4 MR. FRENCH: Well, we've studied 5 the literature a little bit and, of course, 6 incentives are all over the board. There are 7 two competing things. Number one is the 8 economic incentive but number two, especially 9 with a government survey, is you don't want 10 to give them too much because there is a 11 backlash effect where sometimes people will 12 say what's the government spending all of 13 this money, giving me money to do a survey, 14 for? And so you have to balance between -- 15 DR. SITTER: No, that's all true. 16 Let me tell you where I'm coming from. 17 MR. FRENCH: To give you a 18 technical answer, as far as I know, Steph, 19 this is a fair statement. I mean, we have 20 talked about this but there is no technical 21 study that was done before the fact to 22 determine those amounts. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 286 1 MS. BATTLES: No, other than in the 2 early '80s actually we did have an incentive 3 at that time and it was like $2. And at that 4 time they did some studies and they didn't 5 find too much of a difference in response 6 rates. The big thing was they did find 7 statistically significant differences in 8 signing of the authorization form which 9 actually is the heart and soul of the other 10 RECS itself. And I think basically, though, 11 we looked at the budget consideration of how 12 much money we had. 13 DR. SITTER: Well, for example, 14 there are some industries that it's almost 15 impossible to determine what its value is to 16 society. Try to put the economic value on 17 recreational activities, for example, and 18 I've seen work on things like how do you 19 compare the commercial fisheries versus the 20 sports fisheries, which is worth more 21 economically. 22 So they do studies where they BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 287 1 interview people. I've seen one where they 2 interview a person coming in and they say 3 well, if I'd have given you $50 would you 4 have not gone fishing today, so this question 5 costs you nothing but it addresses the 6 question whether you're wasting your time 7 using 5 or $10 or whether you could have got 8 away with $5 or whether you should do fewer 9 of them and give them $20. 10 I don't know. I'm just saying that 11 what they discovered, of course, is that they 12 didn't do any pre-work on the study I looked 13 at and they blew it. They wasted half of 14 their time using too small a dollar value. I 15 guess they didn't know much about fishermen 16 but not many fishermen are going to stay home 17 if you offer them five bucks. They didn't 18 say I'll give you five bucks, would you have 19 stayed home, so they get a lot of zeroes. So 20 they learn nothing, right? Zeroes don't give 21 you any information. 22 MS. SEREIKA: So the idea is BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 288 1 they're putting a value on their free time so 2 that you can do a better estimate of how much 3 money to offer them? 4 DR. SITTER: Right, so then you do 5 a simple logistic regression. You can get an 6 estimate of how much you could have increased 7 your thing if you had in fact offered the 8 dollar value. So the reason I bring this up 9 is because, I mean, you could get turned 10 down, for example. 11 You're not allowed to offer 12 monetary incentives but you'd be allowed to 13 ask that question and if you were to ask that 14 question with different dose levels, that is, 15 different dollar values across your survey 16 randomly, then perhaps you can use that to 17 make a better case for them to actually give 18 you the money. That is, if you tried 0, 5, 19 10, 15, 20, just as a question then you fit a 20 logistic regression to it and you can get an 21 estimate of if we would have offered $10 to 22 everybody we would have got this big of an BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 289 1 increase in our response rate or our signing 2 of the waiver forms. 3 So you can get an estimate of what 4 you would have gained if you had been allowed 5 to offer a monetary incentive. 6 DR. NEERCHAL: Why do people not 7 respond? 8 DR. SITTER: Well, that was one of 9 the related things. He said security, lack 10 of time, uncertainty about the need. 11 MR. FRENCH: Uncertainty about the 12 need for the data and then just a general 13 put-off type of refusal, I don't really want 14 to do it or something like that where you 15 can't pin a reason down. 16 DR. NEERCHAL: Offering monetary 17 compensation will only address only one 18 portion of those people really. I think the 19 way I look at it is it will be nice if we can 20 connect the monetary value to the need for 21 the data. For example, you say this is how 22 much worth it is if you are spending all this BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 290 1 time answering this question and it is worth 2 so much to me. That is what you're trying to 3 indirectly tell them. 4 MR. FRENCH: Well, actually in a 5 sense we're not because what we do not want 6 with the incentive is to indicate to the 7 people that it is payment for their response. 8 The government really is not in the business 9 of paying people to give out information. 10 And what we are going to indicate to the 11 respondents is that the incentive would be a 12 token of thanks for completing the survey but 13 not payment for completing the survey. 14 And one could say well, gee, that's 15 a fine line but in fact in the government 16 sometimes treading fine lines is an important 17 thing to do indicating that thank you for 18 responding and here's the money. Now, 19 understand, this incentive is going to be 20 paid out at the beginning of the process, not 21 at the end of the process. This is an 22 unconditional incentive. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 291 1 MS. SEREIKA: Oh, it's 2 unconditional. 3 MR. SINGPURWALLA: So it will have 4 no impact on those —————— people? 5 MS. KIRKENDALL: It might. 6 MR. SINGPURWALLA: People who do 7 not answer just -- 8 MS. KIRKENDALL: Well, those you 9 can't find it won't have much impact on 10 unless you can't find them because they're 11 hiding out in their house and they don't want 12 talk to you. 13 MS. SEREIKA: Because you can't 14 even approach them. 15 MS. KIRKENDALL: Right, and then 16 the thought is if you're telling them you 17 might give them a cash incentive maybe 18 they'll be willing to come to the door. 19 MS. SEREIKA: If you flash the 20 money, maybe. 21 MR. FRENCH: Margot's got a 22 question. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 292 1 MS. ANDERSON: I wanted to follow 2 up on this issue that Randy raised about can 3 you turn part of this into an experiment. 4 We're clearly going to look at the folks that 5 don't get an incentive. We're breaking it up 6 into three groups, nothing, 5, and 10 bucks. 7 Could you put such a question on 8 the survey that even if you don't get the 9 money you could ask a question would you have 10 been more willing or can we try and get at 11 some of this information to do the continued 12 evaluation even if OMB gives you permission 13 to do this? 14 In other words how can you get 15 additional information about how much we need 16 to pay to get people to be more interested in 17 this? How could we do it even if we get the 18 permission from the OMB to give the 19 incentive? 20 DR. SITTER: Well, first of all 21 these are in-person interviews so I wouldn't 22 even mention this question except to the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 293 1 people that refuse. 2 MS. ANDERSON: Right, and then you 3 would say well, if we'd given you 10 bucks 4 would you have let me in the door? 5 DR. SITTER: See, you can choose 6 different dose levels too. For some you'd 7 say 10, some you say 20, some you say 30. 8 And it's just a zero/one response. Now you 9 have a dose response curve. You can fit a 10 model to it. 11 MS. ANDERSON: You might be able to 12 use that for the next round and whether 13 you're picking the right amount. 14 MS. SEREIKA: So you're saying you 15 actually will have it so there are forced 16 choices? They don't come up with a number in 17 their head? 18 DR. SITTER: No, no, you choose the 19 dose. You don't say to them how much it 20 would have cost me for your response. 21 MS. SEREIKA: You choose the dose. 22 DR. SITTER: You choose the dose. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 294 1 Then you learn something. 2 MS. ANDERSON: You might have a 3 little experiment going that would tell you 4 something about what people might be willing 5 to pay for this nonrespondent and presumably 6 if you get to that point OMB has already 7 agreed to give out something. Now you have 8 additional information to try and zero in on 9 how much that ought to be in subsequent 10 surveys. 11 DR. SITTER: And even if they say 12 no, we can't give you any money, you can 13 still do this. 14 MS. ANDERSON: Yes, of course. In 15 the case where they are you could still do it 16 even on a smaller —————— 17 DR. SITTER: So let's do a 18 scenario. Suppose you have 10 percent of 19 your survey people don't respond and then you 20 do this for all of them. What you end up 21 with is some curve. And with that curve you 22 can estimate if we had have offered all the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 295 1 nonrespondents 20 bucks then a third of them 2 would have responded. So now you say well, 3 of course, I couldn't have identified the 4 nonrespondents so I would have had to give 20 5 bucks to everybody or —————— so if you can 6 get at least some notion of what the cost 7 would be to you, what five bucks would have 8 bought you, what ten bucks would have bought 9 you, what twenty bucks would have bought you, 10 and it may be that the answer is don't even 11 waste their money. It might not be the 12 thing. In other words a lot of what you gain 13 from all your other activities will be much 14 higher than the monetary incentives. 15 MS. FORSYTH: I don't know if it 16 matters. This is a really tiny point but the 17 data show that incentive amounts required are 18 much higher to convert refusals than they are 19 if you start off here's $10.00, now here's 20 what I would like from you. And so if you're 21 thinking about doing this experiment with 22 those people who have already refused, I BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 296 1 mean, they are in a different mind. They're 2 not necessarily being converted. They're 3 just answering your question. But you might 4 want to worry about that same kind of -- 5 DR. SITTER: Yes, so some thought 6 about how you might get -- 7 MS. FORSYTH: How to frame the 8 task. 9 DR. SITTER: Yes, that's a bit 10 tricky. 11 MS. FORSYTH: Because you don't 12 want to be converting them because then it 13 really costs you. 14 MR. FRENCH: And the problem with 15 giving monetary incentives to people who 16 initially refuse is then you can hardly avoid 17 the perception that you are paying them to 18 respond. 19 DR. SITTER: So getting at that 20 point might be tricky. You'd have to give 21 that some really serious thought. 22 MS. FORSYTHE: But I think it's BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 297 1 just a matter of how you frame the task you 2 give them. 3 MS. KIRKENDALL: The other thing is 4 that the Census Bureau did some experiments 5 on the conversion of refusals and they paid 6 like 50 bucks or something. This was for one 7 of those that have had a deterioration on the 8 response rate to a point where it was getting 9 so that the results just weren't valid any 10 more. And so they really needed to build 11 back that response. And one concern was if 12 you only give it to nonrespondents and people 13 know each other then they'll I told them no 14 and I got 50 bucks versus the guy next door 15 who was nice and went along with them didn't 16 get anything. 17 DR. SITTER: But I think the point 18 here was in the terms of the study itself. 19 MS. FORSYTH: How you ask the 20 questions. 21 DR. SITTER: If the only people you 22 can get at are those that initially refused BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 298 1 and they are more difficult they are going to 2 cost you more. So the numbers you get might 3 suggest that it will cost you 20 bucks when 4 really it wouldn't have cost you so much. If 5 you take the same person and offered the 6 money right up front it might have only cost 7 you five but conditional upon them already 8 having said no. People are funny, right? 9 They said no, you say well, I'll give you 10 five bucks, they say no, I already said no. 11 You can't buy my vote. But if you said well, 12 we're going to thank you with five bucks 13 maybe they would have said okay. 14 MS. FORSYTH: And actually if you 15 get your experiment approved then you can 16 actually use the data to calibrate what the 17 people in the no-incentive condition are 18 telling you about the incentive that would be 19 required. So it's actually better if you do 20 this data collection with your experiment 21 than without. 22 MS. KIRKENDALL: Now, didn't you BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 299 1 pick the amounts too from some other 2 literature that you had seen, other incentive 3 experiments? 4 MR. FRENCH: Well, first of all, as 5 Stephanie mentioned, what we had paid in the 6 past and if you inflate that by inflation 7 between now and then the levels are probably 8 roughly comparable. We also looked at other 9 studies and, of course, there are surveys out 10 in the field now that pay a lot more than 11 that but they ask a lot more of people also. 12 Then you got surveys that, well, 13 like Hanes and that sort of thing where 14 people are being asked for physical 15 examinations and all that. 16 MS. KIRKENDALL: That's a little 17 different. 18 MR. FRENCH: That's different. 19 Stephanie, I know that there have been a lot 20 of interview situations where people were 21 being paid 10 and $20 and things like that 22 and some of them were for interviews that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 300 1 would take an hour, an hour and a half, even 2 longer than that, and so, I mean, we also 3 tried to weigh what we were offering versus 4 the relative level of effort. 5 MS. BATTLES: Studies have shown 6 that especially in the low-income area you 7 can offer a lower incentive for a higher 8 level of participation. We're doing it by 9 segment. Also we are trying to stratify by 10 income. This is the first time we have tried 11 this. Actually I have received some positive 12 information back from OMB as far as the 13 acceptance of this and they did also question 14 —————————————— OMB told me when ———————— 15 steak dinners and so I think we have a good 16 chance of getting this but this is the first 17 time we've done this. 18 MS. SEREIKA: I know some of the 19 studies I've been involved in instead of 20 giving them cash they've given them gift 21 certificates to the food store, things of 22 that sort that still have value to them BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 301 1 instead of just plain cash. 2 MS. BATTLES: But studies have 3 shown that money is the best, cold cash. 4 DR. SITTER: But if you really 5 believe there's a difference in the impact on 6 different income groups then are you now not 7 going to implicitly create a problem in terms 8 of your nonresponse adjustments or are you 9 going to stratify those as well? Because you 10 know what I mean? You probably have to take 11 that into account. If you're going to 12 stratify your incentives you have to keep in 13 mind that you don't want your nonresponse to 14 be related to your main response variable. 15 MR. FRENCH: And income is a pretty 16 important variable that we use to look at in 17 assessing. 18 DR. SITTER: So randomly giving the 19 same incentive across the board is going to 20 have a zero impact on me. Your 5 bucks is 21 not going to impact me at all whereas for 22 somebody 5 bucks means something and they are BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 302 1 going to do it and now your response rate is 2 related to your income which means it's 3 related to the response. 4 MS. KIRKENDALL: It already is. 5 MS. SEREIKA: It already is. 6 DR. SITTER: It already is so you 7 already adjust for it? 8 MS. BATTLES: Yes, we do. 9 DR. SITTER: So good. And just one 10 more comment, this other comment about what 11 were the reasons for it. I mean, we think of 12 it as being rational but some people are 13 different right? Oh, I'm not going sign this 14 because of security. Oh, but for five bucks. 15 DR. NEERCHAL: —————————————— there 16 is a increase in signing. Money transaction 17 and signature go together. You offer the 18 money then I feel okay with my signature 19 there. That's completely, I think, 20 consistent with people's psychology. 21 MS. ANDERSON: I wonder whether we 22 will have a better response rate given the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 303 1 degree to which energy prices have increased 2 and the more attention that all of you were 3 paying to the energy bill and I was just 4 wondering whether we might get more 5 cooperation because people may see not a 6 direct link, of course, but may be in a mind 7 to talk to somebody surveying them about 8 their use and their expenditures when they 9 are much more aware of the fact that energy 10 is costing them more than it was the last 11 time these guys knocked on their door. 12 And I don't know whether there have 13 been any studies to show that when issues are 14 hot whether that has any impact on response 15 rates. There are probably a few surveys that 16 track over time and look at common events and 17 how those might affect response rates. But 18 we might be able to get some extra bang from 19 the fact that energy prices have been so much 20 in the news and people are concerned, not 21 alarmed but certainly concerned about it. 22 MR. FRENCH: You can be sure that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 304 1 part of the interviewer training this summer 2 will be to encourage them to try to use that 3 as a hook to respondents saying this 4 information is base line information that is 5 fed to the Congress and policy people who 6 need to develop plans that affect the energy 7 that comes into your house. 8 DR. SITTER: But this does raise a 9 trap, a confounding trap, right? I mean, 10 you're going to create a monetary incentive 11 at the same time as you are changing these 12 —————— and at least certainly from my 13 perspective and from I think many peoples' 14 perspective they may be much more influenced 15 by a better education of the need and impact 16 on society than by the dollar amount but it's 17 going to show up as a blanket shift. You 18 won't know where -- 19 MS. KIRKENDALL: No, but we have a 20 zero incentive one so you should be able to 21 detect it. 22 MR. FRENCH: We are planning on BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 305 1 detecting the difference between zero, 5, and 2 10. That's why it's carried out as an 3 experiment and the other reason is that OMB 4 is believed would look more favorably on an 5 incentive if it is carried out as an 6 experiment because they want the background 7 information to add to the body of knowledge 8 on what incentives do. 9 MS. SEREIKA: And again the 10 incentive that people get will be the same 11 within a segment, right, you said, but that's 12 then randomly determined? I just wanted to 13 make sure got that straight because I heard 14 something about linking it to their income 15 level? 16 MR. FRENCH: No. 17 MS. SEREIKA: That's not the case. 18 You're going to do it randomly? 19 MR. FRENCH: It's random, yes. 20 MS. SEREIKA: Okay, good, just want 21 to make sure I got that straight. 22 MS. KIRKENDALL: But there may be BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 306 1 an income. No, they're not looking at 2 incentive to income. 3 MS. SEREIKA: That's what I want to 4 make sure for the experiment. 5 MR. FRENCH: So yes, there will be 6 low-income neighborhoods that get zero, 5, 7 and 10 and higher-income neighborhoods that 8 get zero, 5, and 10. Any other commentary? 9 DR. SITTER: One thing maybe, is 10 the soft nonresponse sub-sampling probably 11 should be done within your income strata, I 12 think. Your analysis is going to be based on 13 probably some two-phase sampling strategy, 14 that is, that the nonresponse is random. 15 Probability of nonresponse is constant within 16 sub-sampling, right, so you should probably 17 do your sub-sampling within those classes. 18 Your estimation will be —————— and you'd just 19 do a stratified estimate. Otherwise if you 20 do it randomly you can still do it but it'd 21 be more difficult so you really want to do is 22 your sub-sampling where you think the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 307 1 nonresponse probabilities are at least 2 reasonably constant so you get a better 3 estimate for less effort. 4 MS. KIRKENDALL: You didn't talk 5 about using priority mail. That was another 6 thing I have heard people say as a way to get 7 people to pay more attention when you mail 8 them something. 9 DR. SITTER: Unfortunately this is 10 being grossly overused by the advertising 11 industry making things look like a check, 12 making things look like it's important, 13 making things look like it's confidential, 14 and now we're getting pretty adept at 15 recognizing the difference between a real 16 important thing and an incentively important 17 thing. That's my opinion. 18 MS. KIRKENDALL: Well, no, the 19 Census with or some agency had a couple of 20 percent change in response rate simply by 21 using priority mail. So there has been some 22 demonstrated impact of something as simple as BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 308 1 using priority mail or some other thing that 2 makes a handout stand out as being important. 3 So you don't want a cheesy envelope. 4 MR. SINGPURWALLA: What kind of 5 questions are on the survey? 6 MR. FRENCH: Characteristics of the 7 household, how many rooms the household has, 8 how many people live in there, how many 9 windows, questions about appliances, 10 questions about which energy sources are used 11 for which particular uses in the household, 12 questions about number and types of 13 appliances. I guess, Stephanie, we're 14 putting in more questions about electronic 15 equipment this time? 16 MS. BATTLES: Computers, right. 17 MR. FRENCH: And so forth and so 18 on. 19 MR. SINGPURWALLA: Is it difficult 20 to fill out, would you say? 21 MS. FORSYTH: It sounds like it is. 22 MR. FRENCH: Probably not as BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 309 1 difficult to fill out as our Commercial 2 Building Survey but that's another story. I 3 would say it's probably moderately difficult 4 to fill out. There are some things that 5 people really don't understand about their 6 household. You would be amazed at the number 7 of people that don't know whether they heat 8 their house with electricity or natural gas 9 or whether its natural gas or LPG. Gas is 10 gas. 11 DR. NEERCHAL: Usually there is 12 only one person per household who knows it 13 and you may not get that person, right? 14 MR. FRENCH: That's true also. But 15 in addition to that we do one other thing, 16 which is an interesting thing that we stopped 17 doing in one survey, immediately regretted 18 it, and have continued to do it since then 19 and that is we actually train the 20 interviewers to measure the square footage of 21 the housing unit. We don't ask people what 22 the square footage of their housing unit is. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 310 1 MS. KIRKENDALL: That's a good 2 idea. 3 MR. FRENCH: People like me even 4 know, people like most people don't know, and 5 so we actually go out with measuring 6 equipment and do that. 7 MS. KIRKENDALL: Does your 8 interviewer check out the furnace and make 9 sure that it is what they say it is? 10 MR. FRENCH: No, they do not. 11 MS. FORSYTH: Like count the number 12 of rooms, you mean? 13 MS. KIRKENDALL: No, just go down 14 and look at the furnace. 15 MR. FRENCH: Well, actually it's an 16 interesting question because you don't nose 17 around people's houses. I mean, it's bad 18 enough to even ask them. If you are going to 19 try to measure the unit from the inside 20 because let's say it's an inside apartment 21 and how are you going to measure the building 22 to get the apartment? It's bad enough to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 311 1 even ask people to do that. We usually try 2 to measure from outside whenever we can 3 especially when it's a house. 4 MS. SEREIKA: Then that —————— 5 —————— based on the number of units or 6 something if it's a multi-unit dwelling 7 measure the outside and then try that -- 8 MR. FRENCH: Yes, or sometimes even 9 from the rental agent survey the rental agent 10 may be able to tell you that's an 800 square 11 foot apartment because they know. They rent 12 the thing. 13 MS. SEREIKA: In Pittsburgh they 14 have assessment information and they have the 15 size of houses listed. I mean, there is a 16 lot of information on the web, I'm pretty 17 sure. I wonder if that's also the case that 18 you can get information on size that way. 19 MR. FRENCH: I think we have gotten 20 some in the past, haven't we, Stephanie? 21 MS. BATTLES: We've planned this 22 time to try to use something like that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 312 1 because although sometimes on the assessments 2 you don't get if they have added on to their 3 home. We have a lot of credits built into 4 the CAPI program and so someone tells us they 5 use certain energy source and then the 6 equipment doesn't match the source a flag 7 will go up right then and there you will say 8 I are you sure and we'll track down. And we 9 also have show cards, definitions, so on and 10 so forth. 11 MR. FRENCH: To try to help people 12 as much as we can to get the information 13 because we ask things too like not just do 14 you heat your house with gas or electricity 15 but is it a forced air furnace with duct work 16 and this that and the other thing or is it a 17 radiator type system, so forth and so on. So 18 we're asking fairly sophisticated questions 19 of respondents. 20 That must mean our time is up. The 21 thing clicked. Any other comments or 22 questions? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 313 1 MR. SINGPURWALLA: Are the 2 interviewers like well trained on energy 3 issues? 4 MR. FRENCH: They go through what, 5 four days of training? A lot of that is 6 going through the questionnaire and asking 7 the questions. Are they energy experts? No. 8 Some of them may be on their own, not experts 9 but at least reasonably knowledgeable about 10 energy. 11 MR. SINGPURWALLA: What if like a 12 respondents is like, I don't know if I heat 13 my house with gas or electric. Can you show 14 me how to find out? Could the interviewer 15 help? 16 MR. FRENCH: Potentially they could 17 help but they are not really trained to do 18 that and they are supposed to go through the 19 questionnaire, not supposed to stop the 20 questionnaire and say well, let's go down and 21 look at your furnace or something like that 22 or your heating equipment, whatever. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 314 1 One of the things that we find at 2 least in some circumstances is you mentioned 3 you may not get the right person to answer 4 the questionnaire and that's true in some 5 logistic sense because the right person may 6 not be home at the right time when the 7 interviewer knocks on the door but they can 8 get an interview and if they can get an 9 interview with somebody who seems reasonably 10 competent they will do that. 11 But by the same token you may have 12 a better chance of interesting somebody who 13 has more of an interest in energy. So there 14 may be a slight tendency to get an interview 15 with the person who is more interested in 16 energy simply because they are more likely to 17 respond or if the person who doesn't know 18 anything said you really ought to talk to my 19 spouse because they are the one who knows 20 about all this stuff and come back later and 21 check with them. So they don't refuse but 22 they don't really go ahead either and then BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 315 1 maybe you can get an interview with the 2 person who knows more because the spouse says 3 they know more. 4 MS. SEREIKA: But you pretty much 5 just take whatever warm body is there? 6 MR. FRENCH: If they're willing to 7 do an interview, yes. Whatever warm body, I 8 mean, I am not going to take this four-year 9 old son. They have to be 18 or 16. It's 18 10 now? 11 MS. BATTLES: Basically they have 12 to be adults, right. 13 MR. FRENCH: But you can get the 14 18-year old high school senior who thinks 15 they know everything. Well, there's nobody 16 in the world smarter than a high school 17 senior. You know that. Just ask them. 18 They'll tell you. 19 MS. SEREIKA: But when they do this 20 census they want the head of household 21 responding for the house? 22 MS. BATTLES: Same with us. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 316 1 MS. SEREIKA: Oh, it is? So that's 2 what you desire. 3 MS. BATTLES: Either the head of 4 household or a significant other. 5 MR. PYNG LU: When you say when you 6 can ———————————— their energy bill that's 7 including all the energy source bill or just 8 the electricity? 9 MR. FRENCH: For each energy source 10 that the person uses in the household we 11 would like to get the most recent energy bill 12 and scan it. If they have electricity, 13 natural gas, and propane we will get all of 14 them or try to get all of them. 15 MR. LU: Well, monthly may be easy 16 but they probably will hesitate to give you 17 the whole year. 18 MR. FRENCH: That's correct and the 19 question was whether we're going to get just 20 one monthly bill or a whole year's worth of 21 bills and the answer is we would only expect 22 or hope to get the most recent bill. What BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 317 1 happens is if you look at your electricity 2 and natural gas bills especially you may find 3 that on the back of that bill is 12 months 4 worth of consumption. 5 And some utilities are doing that 6 in order to educate their customers about 7 what's going on with their household and how 8 much they are using and you can compare this 9 month to the same month the last year and see 10 whether it's higher or lower and even my 11 natural gas bill from Washington Gas also has 12 the average temperature this year and last 13 year so if you say well, I used more or less 14 this year than last year you can say well, if 15 a lot of it is used for space heating, and a 16 whole lot of natural gas is used for space 17 heating, you say well, maybe the temperature 18 differential had something to do with it. 19 Or if the temperature is warmer 20 this year than last heating season but you 21 still used more gas then you go to your 22 teenage children and say you took too many BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 318 1 showers and used up too much hot water or 2 something. 3 MS. KIRKENDALL: Does anybody have 4 any input for Dwight on how to compute his 5 response rate if he does a sub-sample to 6 follow up on nonrespondents? 7 MR. FRENCH: See, the idea of the 8 response rate is if you sub-sample and get an 9 additional 2 percent on your response rate 10 based on let's say a one half sub-sample of 11 the soft nonrespondents that counts towards 4 12 percent additional nonresponse because you 13 sub-sampled half the nonrespondents and each 14 of them is weighted double not only for 15 purposes of doing the estimation but for 16 purposes of computing the nonresponse rate or 17 the response rate as it were. 18 MS KIRKENDALL: The Census Bureau 19 got some grief for doing that with their 20 American Community Survey and then saying 21 that they had something like a 97 response 22 rate or a 93 response rate. It was up in the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 319 1 90s when the response before they did the 2 sampling was 70 percent and they did a 3 one-third sampling. So they evidently did a 4 pretty good job following up on the 10 5 percent that they sampled and they translated 6 that into an overall 90-something percent 7 response rate which many people thought was a 8 little misleading. 9 Well, you're following the rule 10 ———————— what Dwight just said. It sounds 11 better if it's only 1 or 2 percent that you 12 get. 13 DR. SITTER: No, that's just wrong. 14 MS KIRKENDALL: How would you 15 compute it? 16 DR. SITTER: Well, a response rate 17 is a response rate. What you're doing the 18 sub-sampling for usually is to determine 19 whether there is a difference between 20 respondents and nonrespondents so that you 21 can then assume that there is no difference 22 in estimating. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 320 1 MS. SEREIKA: That's missing. 2 DR. SITTER: So saying that you got 3 that response is nonsense so yes, they should 4 have been criticized. 5 MS. ANDERSON: They were criticized 6 but I'm not sure I ever saw what they were 7 supposed to do with the numbers. So you 8 would not use those sub-sampled ones? 9 MR. FRENCH: In other words if they 10 got 70 percent nonresponse in the first wave 11 the way you get to 97 percent response is you 12 sample 10 percent out of the 30 percent that 13 you haven't got a response from. Out of 14 those 10 percent you get 9 percent of them to 15 respond which inflates to 27 percent when you 16 triple it and you add that to the 70 percent 17 and you get 97 percent. 18 MS. ANDERSON: It looks wonderful 19 but it's really misleading. 20 MR. FRENCH: But in fact 79 percent 21 of the initial people who were in scope 22 responded. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 321 1 DR. SITTER: I don't know what you 2 are talking about. That's nonsense. That's 3 just nonsense. 4 MS. SERIEKA: So how would you 5 compute all these -- 6 DR. SITTER: Is first of all you do 7 the sub-sample of nonrespondents and what 8 you're really interested in is are the 9 characteristics of your nonrespondents 10 basically the same on the things you are 11 interested in as your respondents. Let's 12 suppose that it's true. Well, now, if you 13 had an original 70 percent and you got 9 of 14 your 10 or 90 percent response in your 15 follow-ups you have 79 percent response rate 16 and because the characteristics are the same 17 for both then you can join them together 18 reasonably so, okay. 19 You can also now estimate the total 20 by operating with the nonresponse adjustment 21 with more confidence because the nonrespond 22 adjustments assumes that the probability of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 322 1 nonresponse was at random and it wasn't 2 related to your characteristics. So it's 3 two-fold but you still only got 79 percent 4 response rate. 5 MS. FORSYTH: They get differential 6 weighting in the analysis and in the 7 estimation but you don't differentially rate 8 them in computing the response rate where 9 everyone ———— a response? 10 MR. FRENCH: Well, actually, 11 though, Randy sounded like he was arguing 12 differently a moment ago with regard to -- 13 DR. SITTER: No, the response rate 14 would still be 79 percent. 15 MR. FRENCH: The response rate is 16 but I'm saying for purposes of weighting an 17 estimation you sounded like if the non- 18 respondents are like the respondents you're 19 going to oversimplify this dramatically, take 20 your respondents and weight them up by a 100 21 over 79 to get population estimates. 22 DR. SITTER: Oh, well, I'm not BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 323 1 saying how you would use the nine percent in 2 terms of upweighting but, I mean, you have to 3 think about that but definitely now at least 4 you have got a rationale for doing the 5 upweight because the concern you have is if 6 there is a difference in the characteristics 7 of interest and you would have to be 8 something more complicated because -- 9 MR. FRENCH: Or you simply weight 10 the nonresponse cases that you were able to 11 convert up to represent all of the non- 12 respondents. 13 DR. SITTER: If the sample is large 14 enough, yes, absolutely. 15 MS. FORSYTH: And that, of course, 16 adds variance to your weights. 17 MR. FRENCH: Yes. 18 MS. FORSYTH: Which adds variance. 19 DR. SITTER: And then, of course, 20 if from the characteristics of interest 21 you've basically established that they have 22 similar characteristics then you can treat it BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 324 1 like a two-phase sample and your variance 2 estimation is also not so difficult. 3 But all they have done is do 4 exactly what you would have done with your 5 wide characteristic for the indicator of 6 nonresponse and then say well, we got 97 7 ———————————— don't understand that. So if 8 now they only get 80 percent in respondents 9 but they are going to say they got 100? We 10 got everybody in —————————————— 11 MS. KIRKINDALL: I don't know how 12 they're doing now. Actually it will be 13 interesting to find out. 14 MR. FRENCH: There are two ways of 15 computing it and these folks are saying one 16 of them is illegitimate and so -- 17 DR. SITTER: Well, I don't think it 18 takes a statistician. Anybody that has a 19 method where you've actually only got 80 20 percent response but you're saying you got 21 100 then I have got a problem with that. 22 MR. FRENCH: Well, it took a BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 325 1 statistician to figure out how to do that. 2 We won't say what kind of statistician. 3 Anything else? Well, it looks like 4 the other group has returned so thank you 5 very much. Appreciate the commentary. 6 (Recess) 7 DR. HENGARTNER: Let's start again 8 since we're one minute late. Before we go 9 on, discussion. Sorry, Susan, I almost 10 forgot. So Susan is going to discuss the 11 break-out session that we just had up here. 12 MS. SEREIKA: Right. During the 13 session EIA was telling us about some of the 14 problems they've had with declining response 15 rates for their residential energy 16 consumption surveys. The survey is they 17 actually collect information using a 18 multi-stage area probability survey from 19 4,000 households to collect information on 20 the residents in terms of what they look 21 like, for example, the number and the types 22 of appliances they have, and they also BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 326 1 collect information of consumption so they 2 can then look at and describe those things 3 for the consumers as well as then try to 4 relate them to one another. But what they 5 have had since probably at least 1980 because 6 that's the data they showed us they've shown 7 a steady declining rate of response from 8 these respondents. 9 For example, in 1980 they had a 10 response rate of 87.5 and in 2001 it declined 11 to 72.8. And they did note that in that year 12 there was also a cost problem when they were 13 trying to conduct the surveys which could 14 have led to them having a lower response rate 15 than what they really would have had had they 16 been able to be a little bit more aggressive 17 in trying to get information from the 18 respondents. 19 And the other thing they've noticed 20 is that there has also been an increase in 21 the authorization form not being signed, also 22 an increase in personal interviews not being BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 327 1 able to be contacted, refusals. They do use 2 imputation for consumption and expenditures 3 and those components are the core of their 4 survey. 5 In an effort to address these 6 concerns with declining response rate EIA is 7 suggesting a number of different initiatives. 8 First off they wanted to start with the 9 household interviewer in how they recruit 10 people. They thought before actually sending 11 out introductory letters they would first 12 start out with an introductory postcard 13 basically foretelling of the survey to come. 14 They also were suggesting the idea that the 15 introductory letters that each of the 16 respondents receive be reworked so that 17 they're more upbeat, eliminate bureaucratic 18 recitation of required information, the idea 19 of a gift, and, probably more importantly, 20 the idea of that they would protect 21 information and confidentiality of the 22 respondent. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 328 1 They're also suggesting the idea of 2 incentives in completing the household 3 interview and also getting the subject to 4 sign the waiver form. At first they were 5 doing traditionally nonmonetary incentives, 6 things like, pens, brochures. They're now 7 proposing the use of monetary incentives 8 which seems to be supported, especially if 9 they use the approach they're proposing to 10 do, and that is you're actually conducting an 11 experiment where they would randomize 12 subjects to receive either 0, 5, or $10 13 incentives, one-third each. 14 The choice of the dollar amount 15 actually is one of the things later 16 discussed. Why did we choose those dollar 17 amounts? And there appears to be some 18 evidence in the literature to support the 19 choice of the dollar amount although there 20 were still some concerns about that choice. 21 The survey, just so you know, is 22 about a 45-minute survey so, again, it should BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 329 1 be taken into consideration in the dollar 2 amount. In terms of how the randomization 3 will occur they are going to actually give 4 the incentives based on a segment level so 5 that neighbors would be receiving the same 6 amount rather than it possibly differing 7 within a segment. 8 The other initiative they have 9 underway is the fact that there may be more 10 difficulty actually interviewing Spanish- 11 speaking households and the idea being that 12 they would identify areas where that was a 13 very heavily Spanish-speaking region. They 14 would also use bilingual interviewers to get 15 the information from the subject in an effort 16 to get more accurate information and also to 17 be able to conduct the survey. 18 Reasons why some nonresponse is 19 still probably going to occur include again 20 the security and the confidentiality of 21 information. The respondent may feel there 22 is a lack of time that they have in order to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 330 1 complete it. And also they might not really 2 greatly appreciate the need of the 3 information, what it's going to be used for. 4 There is a thought of using 5 customized letters to address the concerns, 6 and also the idea of sub-sampling the soft 7 nonresponse. What else? Also there are some 8 thoughts to equip the interviewers with 9 scanners to collect information. So when 10 they're working with the respondent and the 11 respondent gives them hopefully their most 12 recent energy statements for the varieties of 13 energy they're using this information could 14 be scanned off the bill from this respondent. 15 They could also collect possibly even more 16 information of the back of the card regarding 17 the consumption over the past year. 18 When it comes to the apartment 19 buildings which are master-metered the idea 20 is that the interviewer will try to again get 21 building-level energy consumption information 22 from the rent agent and then also estimate BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 331 1 the size of the building in order to get an 2 idea of actually the cost that the apartment 3 may have generated in terms of energy 4 consumption. 5 The question that had been put 6 forth to the committee was whether we had any 7 improvement or additional procedures that we 8 might suggest. In response to that there was 9 further discussion about the choice of the 10 monetary incentives. One of reasons why the 11 choices were made besides having some 12 information ———————— to support the choice 13 was there was a budget consideration about 14 how much money would even be available from 15 monetary incentives. 16 There was some discussion about the 17 idea asking nonrespondents, I believe, how 18 much their time was worth and again giving 19 them a forced choice option about dollar 20 amounts and from that trying to then estimate 21 what would be an excellent optimum dollar 22 amount to offer people in an effort to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 332 1 improve response rates. 2 DR. HENGARTNER: That's an 3 economist, right? 4 MS. SEREIKA: The thing to keep in 5 mind is incentives are always to be viewed as 6 tokens of their appreciation not really 7 payments here. Also the idea of the 8 conversion of respondents and let's see what 9 else. 10 There is also a thought, given the 11 changes occurring right now in terms of 12 increasing energy cost, there is a hope that 13 respondents may become more responsive and 14 actually be more willing to complete the 15 survey. There is a hope in that. 16 There is also the thought of 17 possibly using priority mail in an effort to 18 improve response rate. Being that it's more 19 official looking, it may encourage people to 20 actually be more willing to respond. Toward 21 the end there was some discussion about the 22 nonrespondents, the idea of us getting some BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 333 1 information from them in an effort to 2 characterize them to see if in fact they may 3 be similar or different from the respondents 4 and therefore possibly being able to then 5 estimate what their response might be had 6 they actually responded. 7 And I think that's about it. 8 DR. HENGARTNER: Anyone else wants 9 to add to it? 10 MR. BERNSTEIN: Is Dwight here? 11 Have you gone out to seek advice from firms 12 or organizations that do have some surveys on 13 a regular basis in other areas? 14 MR. FRENCH: Well, in fact we don't 15 go out ourselves and do the surveys. We 16 contract the survey research groups to do 17 this. 18 MR. BERNSTEIN: So why wouldn't you 19 go out and —————————— an RFP to a survey 20 research firm with one of the objectives 21 being to achieve a greater response rate and 22 if they've got the experience decide how to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 334 1 best to get that response? 2 MR. FRENCH: In fact when we went 3 out with the proposal for this particular 4 task we had four people who were eligible to 5 bid on it and why certain companies are 6 eligible to bid on it is a long story which I 7 won't get into but there were four companies 8 eligible to bid on this part of EIA's 9 contract for our consumption surveys and all 10 four of them submitted bids and all four of 11 them recommended in their bid submissions 12 that we use that we use an incentive. 13 Now, they didn't specify how much 14 it ought to be but they all said for your own 15 purposes, given that they had information 16 that we had about the background of the 17 survey and what the response was, their 18 recommendation unanimously was go to an 19 incentive. 20 MR. EDMONDS: Don't they have some 21 knowledge about what incentives are most 22 effective in various situations? BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 335 1 MR. FRENCH: There were no 2 recommendations made specifically about that 3 in the proposal but the way we judged the 4 amounts to offer was based on literature, 5 based on the amount of burden we were placing 6 on people, which was an average of 45 minutes 7 per response, for the most part nonintrusive 8 questions or questions that people would feel 9 uncomfortable about answering in our 10 understanding of things versus what other 11 surveys had offered for comparable types of 12 effort or in many cases for much greater 13 efforts than ours, longer questionnaires, 14 much more intrusive type of questioning on 15 personal habits in the household. 16 That's how we determine the amount 17 in conjunction with how much money I had in 18 my pocket to try to do something like this. 19 MR. EDMONDS: Is 70 percent bad? 20 MR. FRENCH: OMB would like to see 21 80 percent response. 22 DR. FEDER: It depends which are BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 336 1 those 30 percent that are nonresponding but I 2 want to refer to the incentives, a couple of 3 things. Yes, it's a very effective way of 4 increasing response rates. I have two issues 5 with that. Number one, I think it introduces 6 a break in the series in the choice that we 7 talked about before because it does have an 8 effect on responses. Number two, a $10 9 incentive may mean a lot to one person and 10 nothing to another person so it does 11 introduce a bias and I think it's a good idea 12 that you have this one-third, one-third, 13 one-third split so you can do some analysis 14 and see the effect of the incentive. 15 MR. BERNSTEIN: But I must say 16 there is plenty of experience by survey 17 research firms on that already. I don't 18 think you need to do that. 19 DR. FEDER: But every population is 20 different. 21 MR. BERNSTEIN: But there is enough 22 breadth of experience on companies. For BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 337 1 example we have a survey research group in 2 RAND ———————————— I took some —————————— 3 there wasn't enough there but, I mean, they 4 thought that a good, experienced firm should 5 be able to come in and suggest to you in our 6 experience from doing broad household surveys 7 in a variety of areas this is what's going to 8 work. And they ought to be able to do it and 9 you ought to write that in their contract. 10 MS. KIRKENDALL: The issue is more 11 of what OMB will approve. Everyone's surveys 12 are not the same as surveys done by private 13 research firms. OMB does not approve full 14 incentives in a survey. They do approve 15 tests, experiments ———————— they're not 16 convinced yet that they're totally 17 applicable. 18 MR. BERNSTEIN: That's not true ——— 19 —————————— 20 MS. KIRKENDALL: There are some 21 exceptions in the health surveys that you 22 mentioned where you have to have somebody go BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 338 1 for a physical exam. Yes, you pay them an 2 incentive. We don't have anything that 3 invasive. 4 MR. BERNSTEIN: No, that's not 5 true, either. I'm going to have to go back. 6 I will go back and talk to our survey 7 research group —————————————— but I know we 8 have done surveys under federal government 9 contracts. They have cash incentives. 10 DR. FEDER: We do incentives on 11 government surveys. 12 MS. KIRKENDALL: If you'd find the 13 characteristics of them and actually the 14 amounts would interesting to know too. But 15 OMB used to disapprove them routinely with 16 any incentives and they've only loosened up 17 fairly recently, within the last five years 18 and for a while other than the health. 19 They've done health incentives for a long 20 time because of the nature of the survey. 21 DR. FEDER: Susan mentioned a 22 question which is unanswered, why have the BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 339 1 response rates declined, and I think an 2 analysis by subgroups would be interesting 3 regardless of the issue of incentives. I 4 know it's a general phenomenon but it's 5 something that needs to be really 6 investigated. 7 MS. KIRKENDALL: They've been 8 declining in all governments through all 9 surveys, I think, but certainly government 10 surveys. 11 DR. HENGARTNER: Phone surveys, 12 correct? 13 MS. KIRKENDALL: No, all surveys. 14 These are personal visits. 15 MR. FRENCH: Yes, this is a 16 personal interview but that's true. It's 17 been declining in all modes. 18 MS. FORSYTH: And certainly in the 19 telephone context, and I think you alluded to 20 this in your presentation, most of that non- 21 response comes from no contact at all. 22 There's a big boost in the proportions of BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 340 1 households that you never reach. 2 MR. FRENCH: But even in our 3 personal interview survey we have seen some 4 uptick in that. And that's a result of 5 smaller household sizes which means there are 6 fewer people in total possible to be home at 7 any one particular time. My wife and I are 8 no exception. We run around a little bit —— 9 —————— than we did 20 years ago. 10 MR. CLEVELAND: It's easy to 11 understand, I think. 12 MR. FRENCH: Well, but that's 13 refusal even if they are at home. 14 MR. CLEVELAND: Right. 15 MR. FRENCH: Then there is the 16 issue of you can't ever even find them at 17 home. Never mind try to convince them. 18 MS. KHANNA: I have just one side 19 comment which is you asked your respondents 20 to have their electricity bill. That's a 21 one-hour search for me. It's not a 45-minute 22 response. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 341 1 MR. FRENCH: What we will do is in 2 the information that we send to people before 3 the interviewer actually conducts the 4 interview we will ask them if they would have 5 available their latest energy bill. You have 6 to do that. 7 MS. KHANNA: I'm pretty good about 8 responding to surveys because our whole 9 profession depends on that. But if you ask 10 me to find my electric survey that's one 11 survey I might as well not -- 12 DR. SITTER: I disagree. Our 13 profession depends on us not responding to 14 surveys. I am a nonrespondent. 15 DR. HENGARTNER: Any other 16 questions or comments on this topic? I'd 17 like to move on now to the hands-on usability 18 testing of the new website, which was just 19 completed by both subgroups of the committee. 20 MS. BLESSING: Thank you. My 21 report will be short. I really want to thank 22 everybody for participating. That was really BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 342 1 a lively group. There was a lot of energy in 2 that room. The logistics, we weren't sure 3 how it was going to work with everybody 4 hearing everybody else talking and I thought 5 it went great. Like I had told you in the 6 room, we got a lot of good feedback the first 7 time and it happened again. We loved having 8 this captive audience of really smart people 9 to talk to. 10 Doing the usability testing gives 11 us a chance to really try to hone in on 12 common user tasks. These questions seem easy 13 but it takes a long time to come up with the 14 questions that are phrased correctly that try 15 to look at various parts of the site that try 16 to get you to go to different areas. We also 17 appreciate the fact that there were varying 18 levels of expertise on our website from I 19 come there almost every day I went there 20 yesterday and that's the first time and very 21 different levels of energy awareness and 22 familiarity with energy. That was good that BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 343 1 even within this homogeneous group we had 2 differences in the responses. 3 Our next steps, we've already done 4 a lot of meeting and thinking and we've 5 already done some paper testing with a 6 different group. Now we've done some live 7 testing with you guys. We're planning on 8 doing some more thinking and some testing in 9 a formal usability lab with the contractor 10 that was up there with us today. 11 And then with all that information 12 put together we're going to come up with a 13 proposal for a new site design based on all 14 of the inputs that we're getting. We also 15 had a really good opportunity to test our 16 deputy administrator today. Howard was in 17 there and he spent over an hour talking so 18 that was really good. 19 Redesigning this site is a very big 20 deal for us as you guys might expect. And we 21 have a lot of users who know our site and 22 maybe it's not the best but they know how to BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 344 1 find what they're looking for so we really 2 take it so seriously that it's taking longer 3 than we thought to get down this road. There 4 are topic versus publication issues, there 5 are design issues, there are the color 6 issues, usability issues, and then there are 7 the political issues of this area versus this 8 area versus this area and what's on the home 9 page and how it all lays out so it's a lot of 10 work for us. 11 But I feel like the feedback that 12 we've gotten and especially today, at least 13 what I heard as we only had like 10 minutes 14 to debrief, is that we're on the right track 15 and it seemed like most people liked what 16 they saw today. There were broken links, 17 there were problems, but in general it looks 18 like we're on the right track. 19 And what we plan to do is when the 20 next meeting, September-October, one of us 21 will give an update. Well, actually, I'm 22 hoping that by the next meeting the new site BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 345 1 will be up. But then we'll give you an 2 update on our progress and how it's gone. 3 And we actually have a customer 4 survey that's ongoing right now and we're 5 hoping that we have the new site up before 6 the customer survey runs out so that we'll be 7 able to see changes in people's feedback once 8 the new site comes up. But we'll able to let 9 you know all of that. 10 Does anybody have any questions of 11 me? 12 MR. BERNSTEIN: There is just one 13 thing I want to emphasize and also thought of 14 ———————————— what is lacking in the existing 15 site ———————————— new site ———————— we're 16 talking about a data portal which is going in 17 and I want to download some data to use in 18 different cuts in different ways. Right now 19 there is really no way to do that. You've 20 got to go to a particular fuel and you've got 21 to extract something in particular and go to 22 someplace else and extract something BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 346 1 different. And a data portal that allows you 2 to say okay, I want state by state data for 3 all energy consumptions, something like that. 4 Right now you have to pick and choose 5 different pieces and pull them into 6 spreadsheets and things like that so ———————— 7 aid to a researcher data portal section where 8 you're going to download data in different 9 ways, different cuts, very helpful and I 10 think a lot of people would find that very 11 helpful. 12 MS. BLESSING: Thanks. That 13 actually is a very big, big job for us. 14 MR. BERNSTEIN: I know it's not 15 easy to do —————————————— 16 MS. BLESSING: Anybody else? 17 Thank you. 18 DR. HENGARTNER: Thank you very 19 much, Colleen. We're getting towards an end 20 of our meeting. Before we adjourn I'd like 21 to invite the public if anyone is here to 22 comment on today's session. BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382 347 1 Last order of business, I want to 2 remind the committee we have dinner at the 3 Lebanese Taverna. It's on 2641 Connecticut 4 Avenue. 5 MR. WEINIG: I'll go up and get 6 there as much ahead of you as I can and take 7 a place outside. The place actually opens at 8 5:30 so you can look for me outside. 9 DR. HENGARTNER: The other thing I 10 wanted to announce is tomorrow morning we 11 start again fairly early. Let me suggest 12 that if those who stay at the hotel we could 13 meet in the lobby around 7:30 and then we can 14 go together, take the Metro, and there is 15 going to be breakfast and coffee here. So 16 thank you very much for a very productive day 17 and see you tomorrow morning. 18 (Whereupon, at 4:55 p.m., the 19 PROCEEDINGS were continued.) 20 * * * * * 21 22 BETA COURT REPORTING www.betareporting.com 202-464-2400 800-522-2382