UNITED STATES DEPARTMENT OF ENERGY MEETING OF THE AMERICAN STATISTICAL ASSOCIATION (ASA) COMMITTEE ON ENERGY STATISTICS WITH THE ENERGY INFORMATION ADMINISTRATION (EIA) Washington, D.C. Friday, October 29, 2004 2 1 PARTICIPANTS: 2 F. JAY BREIDT 3 NICOLAS HENGARTNER 4 JOHNNY BLAIR 5 MARK BURTON 6 MOSHE FEDER 7 BARBARA FORSYTH 8 NEHA KHANA 9 NAGARAJ K. NEERCHAL 10 SUSAN M. SEREIKA 11 RANDY R. SITTER 12 HOWARD BRADSHER-FREDRICK 13 ROBERT RUTCHIK 14 NANCY KIRKENDALL 15 PRESTON McDOWNEY 16 GUY CARUSO 17 TOM BROENE 18 HENRY S. BROOKS 19 BRENDA COX 20 GRACE SUTHERLAND 21 SHAWNA WAUGH 22 BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 3 1 PARTICIPANTS (CONT'D): 2 TOM LORENZ 3 PHILLIP TSENG 4 JOHN WOOD 5 HOWARD GRUENSPECHT 6 WILLIAM WEINIG 7 INDUJIT KUNDRA 8 JOE SEDRANSK 9 KAREN NORMAN 10 11 12 * * * * * 13 14 15 16 17 18 19 20 21 22 BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 4 1 C O N T E N T S 2 AGENDA: PAGE 3 Data Analysis on the EIA-826/906 5 4 Post-Stratification Methodology for 50 the 2002 Manufacturing Energy 5 Consumption Survey (MECS) 6 Time Series Edits for the Electric 118 Power EIA-906 7 If you were King 133 8 Committee Suggestions 179 9 10 11 * * * * * 12 13 14 15 16 17 18 19 20 21 22 BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 5 1 P R O C E E D I N G S 2 (8:37 a.m.) 3 CHAIRMAN BREIDT: Okay, I guess 4 we'll go ahead and start the meeting. First 5 I'd like to ask any committee member, guest, 6 EIA staff, or member of the public who was 7 not present yesterday to introduce yourself 8 from one of the microphones? 9 MR. WEINIG: If you are -- 10 CHAIRMAN BREIDT: In that category? 11 MR. WEINIG: Directly under a 12 speaker in the ceiling, it will cause the 13 thing to wind and break, so we'll use the 14 other microphone. 15 SPEAKER: Okay. 16 MR. COLE: My name is Stacey Cole. 17 I'm from the Bureau of Census, I'll be 18 speaking here today about the post 19 stratification we did for the MEC survey. 20 MR. WEINIG: Anybody else? 21 MR. SLANDA: John Slanda, U.S. 22 Bureau of the Census. I'm with Stacey Cole. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 6 1 MS. BUCCI: Susan Bucci, U.S. 2 Census Bureau and I'm with Rick. 3 MR. WEINIG: Anybody else? 4 SPEAKER: Okay. 5 SPEAKER: Okay. 6 SPEAKER: I think that's it, thank 7 you. 8 CHAIRMAN BREIDT: Okay, thank you. 9 And anyone who did not sign in yesterday is 10 asked to do so, out front at the break or 11 before you leave. Lunch for the committee 12 will be held at the conclusion of this 13 session on suggestions for the spring agenda, 14 lunch is downstairs as usual, and we'll 15 probably be discussing a variety of things 16 including future committee appointments and 17 so on. So with that I will turn it over to 18 Joe Sedransk, who will be speaking on data 19 analysis on the EIA 826 906. 20 MR. SEDRANSK: You're doing the 21 pushing, or I might as well do it. This is 22 of course joint work, so as one typically BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 7 1 says when giving the talk, the other -- the 2 co-author will answer all the questions. In 3 any rate, our work is an evaluation of 4 methodology in current use, but we're not 5 talking about things we've invented, and we 6 need to make recommendations to CNEAF about 7 several things including about methodology 8 and possibly there's been talk about changing 9 the sample. This is just rumor. 10 But anyway this is the -- 11 background is to estimate -- we're trying to 12 estimate total monthly sales and revenues for 13 all non IOU's by end user, first of all IOU 14 is investor owned utility, and the end users 15 are residential, commercial, industrial and 16 other; in all these discussions just forget 17 about other, we'll just talk about the first 18 three. 19 So there are two variables; sales 20 and revenues. Other entities such as IOU's 21 and wholesalers are censused. So we're 22 really just talking about the non IOU's BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 8 1 essentially. 861 is an annual census, it 2 covers everyone and it provides a frame for 3 the 826 which is a monthly survey used to 4 provide estimates of total sales and revenue. 5 And to give you an idea, the sample 6 size were about 150, 160, 170. There are 7 over 2000 in the population of non IOU's. 8 Sample design is cut off sample which was 9 developed in the early 90s, and at that time 10 it included the IOU's which are now censused, 11 and of course there have been additions and 12 deletions since then. 13 Sample coverage rates, I cite 14 below, they depend 76 percent to 88 percent, 15 depending on the variable and end user, 68 16 percent to 95 percent by geographic region. 17 The estimation procedure: very standard, 18 linear regression through the origin with a 19 slope different for different geographic 20 regions. 21 So Y is the current 826 value for 22 company I and region R, X is the past census BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 9 1 value for that company in the region. The 2 only thing to bring to your attention is the 3 variance, sigma squared2 proportional to X to 4 the two gamma, and the current usage has 5 gamma as 0.8. That was developed in the '90s 6 long before I had any connection with this 7 thought to be desirable and I could -- we 8 could discuss this afterwards. 9 The current method is simply, add 10 up the units, the sampled units, to get this 11 sum for a State. For example, add up the 12 sample units and make predictions for the non 13 sampled ones. It's possible that there are 14 some added -- there are some outliers and 15 that might not be included in estimating beta 16 hat for the non sampled ones but they would 17 always be added in for the -- to get the 18 totals, assuming that there has been a check 19 for outliers and that the value is correct. 20 Valuation; what we've been doing is 21 looking at Standard Exploratory Analysis 22 using scatter plots, standardized residual BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 10 1 plots and so forth. Then I want to point out 2 the ten regions that are currently being used 3 for estimation, that is related to beta R's, 4 they depend on region and the R's here bring 5 your attention to the fact that NEA is very 6 large. 7 These were developed, I think again 8 in a year -- early to the mid '90s, based on 9 temperature and climate considerations. NEA 10 is very large; it ranges from and -- possibly 11 diverse from Maine all the way down to 12 Maryland. The other thing to bring to your 13 attention is West, which is somewhere in the 14 middle of California divide; there are only 15 three sample companies in that region. 16 So with this large number of 17 regions, the sample sizes as you see, as we 18 go through here and not too -- pretty small. 19 Four regions are aggregations of the 10 that 20 we've looked at at various times. 21 Occasionally, we'll talk about all the data, 22 and sometimes about all the data except for BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 11 1 those that have been deemed outliers or 2 influential and those are the criteria we've 3 actually -- we've been using. 4 That's we've been using for 5 investigation as opposed to what is being 6 used in the current estimation. Here are a 7 couple of -- about the five scatter plots on 8 top, standardized residual plots down below. 9 Essentially if you've looked through them on 10 the website, there's very little that we can 11 find and suggest that model doesn't fit. If 12 you look carefully enough, you'll find 13 examples where there are things that you 14 might be a little unhappy about. 15 But generally the residual plots 16 look okay. Here's one, here's another one. 17 I picked ones where there are decent number 18 of points, like 14 in this case where you 19 could see something. Some of the others, you 20 can't see much. If you have only four or 21 five points, you just can't tell anything. 22 Here's another one with nine. Then I put two BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 12 1 and here's one which looks like an outlying 2 observation in the top for industrial sales 3 and it's minus 3 point -- larger than minus 4 3.5 here and, here's another one, which 5 probably -- it's probably an influential 6 point. 7 Just a quick comment; I apologize 8 that, there was a gremlin that got in, there 9 should be a second set of plots and that 10 should've been on the website. But a gremlin 11 got in and the ones, the plots, which were 12 "the after," these are "the before," this is 13 the -- you know in another words, this is all 14 the data, the ones where the things were 15 removed, never got into the data set, never 16 got into the -- okay. 17 Issues and questions; choice of 18 criteria for deleting, observe values, and 19 classifying them as outliers, we need to make 20 a recommendation about that. Second one: 21 Composition of estimation groups. I've just 22 given you some examples of 11 regions, why BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 13 1 10, why 11? It's Hawaii, Hawaii is censused, 2 and sometimes we call them 10 and sometimes 3 we call them 11, and that's my fault, it's 4 really 11. 5 Smaller number of regions, the four 6 I cited before, these were examples. Other 7 regions could be -- you could use other 8 criteria for defining the regions based on 9 temperature and climate. Another completely 10 different category is ownership, which we've 11 looked at. We don't use the IOUs at all to 12 make any predictions for the non IOUs. It 13 turns out that these seemed to be different, 14 different in terms of the regression co- 15 efficients. 16 But there's a possibility of using 17 them within the non IOUs, there are also some 18 sub-sets, including things like the federal 19 entity, the federal company -- federal 20 installations. Unfortunately they may be 21 different but we can't do much about it 22 because there aren't very many of them. I BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 14 1 think something like six to eight.(b) was 2 something I was looking for advice about; 3 it's fairly clear, in some cases how you work 4 from the ground up? 5 How would one be using ownership or 6 geography is sort of clear, but suppose you 7 just started with all these companies and so 8 you say, how am I going to form the strata 9 just from basic micro data? Not so obvious 10 to me. You can get the estimates of slopes, 11 but how trustworthy they are, not so clear. 12 So this is just methodology working from the 13 ground up to form a strata. 14 Okay, number three, deals with 15 macro-level. We have a lot of extra data that 16 we're not making any use of, and perhaps that 17 can be used to retain precision and reduce 18 the sample size. And marked here, remember 19 there are two variables and three end users, 20 and here are a couple of plots. Why don't 21 you look at the bottom one because the colors 22 are sharp and the before and after is before BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 15 1 -- all the data in "the after" is outliers. 2 Clearly there is a time series 3 pattern. What's plotted here are the 4 estimated regression co-efficients over a 24- 5 month period starting from January '02 to 6 December '03. And so each point is the 7 estimated regression co-efficient for a 8 region. So first thing to notice there's 9 clearly a time-series pattern for this. As a 10 Bayesian, what I would do -- or contemplate 11 doing, is, suppose I want to make an estimate 12 for September '03, I might use the data prior 13 to that to do a forecast, and if the forecast 14 mean and the forecast variance be the 15 empirical mean and variance of a prior 16 distribution to put together. 17 I mean this is just a suggestion; 18 we're looking for other suggestions as well. 19 I want to -- before going on, I want to 20 point, this is residential, this is 21 important, I want to point out one thing so I 22 don't have to go back to this slide. Notice BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 16 1 towards September -- let's take any cross- 2 sectional view, sometimes looks like these 3 beta hats are very -- they look like they're 4 estimating the same thing. In other periods 5 it looks like there are other groupings. 6 That there are groupings but it is 7 not everything together, and some 8 methodology, which I've been working on for 9 Years, might be useful for here, in trying to 10 improve the precision, letting the data tell 11 which sorts of regression should be combined. 12 So I'll come back to that. 13 Here's another one, commercial, 14 these are again the beta hats, same plot, not 15 as much seasonality as there was on the 16 residential. And then let's look at 17 industrial and the reason -- one reason is, 18 this is much less here as well, but note in 19 the bottom the spike for North-West. And I 20 think these plots -- the reason I'm pointing 21 this out is this is sort of pointed out, 22 something that maybe very well known to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 17 1 everybody in CNEAF, but this is sort of an 2 interesting thing and I'll come back to it in 3 a second. 4 These are comparable plots for the 5 estimates of sigma square, the residual 6 variance. It was my assumption before we 7 started that in fact sigma square might be 8 quite stable, and you could use some of this 9 to improve the estimation of sigma2. If this 10 is residential, here's commercial, there's 11 another sort of wild, but most of them behave 12 very -- they don't change much over time. 13 This is the one I wanted to point 14 to; look at the industrial, look at the 15 bottom; that's that same region we had 16 before. So, in the summer -- spring/summer 17 of '03, there was both a change in beta hat 18 and this large increase in sigma2, so 19 something went on. And this maybe fairly 20 useful, we have not tracked this down yet. 21 I've asked one person who a hypothesis about 22 what happened, but this maybe a useful thing BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 18 1 to look at. 2 Point number four is about use of 3 company level longitudinal data, and there 4 are two examples here. What is plotted here 5 is again same 24 months, each point is for a 6 region, and what's plotted is the monthly 7 value divided by the annual, divided by 12. 8 And the annual divided by 12 is to put 9 everything on the same scales, so they should 10 operate around one. And the dark thicker 11 blue line is the estimated regression co- 12 efficient for that region over the 24 months. 13 What are the reasons for this -- 14 you've possibly used utility of these plots. 15 One is to see whether the large and small 16 companies, all of them seemed out of the same 17 patterns; that would be a validation -- 18 informal validation of the model. The second 19 is if you see a lot of divergence, is maybe 20 the stratification -- post stratification 21 isn't any good. So here's the second one, 22 this looks even better from my perspective. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 19 1 This is a second one, the difference between 2 these two, these are both residential sales, 3 one is NWE and one was another. 4 We've looked at some other -- a 5 couple of others, which don't look quite as 6 good as this. But suggestions looked for 7 here about what to do with these data of this 8 sort. And the last one is what I referred to 9 before but I want to rephrase now; you have a 10 choice of a large number of estimation groups 11 and of course the advantage of that or post 12 strata, large advantage of that is greater 13 homogeneity within the groups. Disadvantage 14 is small sample sizes within each of the 15 groups. 16 Conversely you can have very few 17 estimation groups and you have a larger and 18 then you get more heterogeneity but larger 19 sample sizes. Methodology that is referred 20 to, if you've all done your homework and 21 looked at least at the paper I wrote, alluded 22 to this research which the data -- a database BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 20 1 pooling of like regression co-efficients. 2 What it is, is an extension of the Standard 3 Shrinkage Methodology. 4 Shrinkage Methodology is kind of 5 dumb; it just says which things to put 6 together. This method is dynamic and it 7 basically takes the data and tells you which 8 things to put together; it works quite well. 9 So the question here with regard to this is 10 any suggestions about that or other ways of 11 choosing between large number of estimation 12 groups or -- and with a consequent gains and 13 benefits and other things. 14 Here are the questions for the 15 committee; they're just actually a repetition 16 of the five points I just went through now, 17 and good point to stop. The first two are 18 things for which we need to make 19 recommendations, the last three, any 20 suggestions about things other than what 21 we've talked about or comments about 22 anything. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 21 1 CHAIRMAN BREIDT: Okay, thanks. 2 Let me start with quick questions and you 3 talked about deleting an outlier, so that 4 really means just setting it into -- setting 5 it outside the regression, using it as an 6 observed value? 7 MR. SEDRANSK: Yeah, we're assuming 8 that when such an outlier occurs that there 9 is contact with the company to make sure the 10 value is okay. 11 CHAIRMAN BREIDT: It's not an 12 omission? 13 MR. SEDRANSK: Yeah. 14 CHAIRMAN BREIDT: Okay. 15 MR. SEDRANSK: And that's an 16 important -- I mean, that's a -- I see 17 Scott's up there -- you know, I mean, that's 18 a standard part of practice is going back. 19 Right now there are programs that do these, 20 what we call, scatter plot edits and they 21 have been used heavily in the past, now not 22 being used very much. We think they should BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 22 1 be used a lot more. 2 CHAIRMAN BREIDT: And is this cut 3 off sample maintained or is it pretty much 4 the same thing? 5 MR. SEDRANSK: No in fact that it 6 was a cut off sample and it's now -- no, it's 7 not maintained. 8 MS. KIRKENDALL: It needs to be 9 evaluated again? 10 MR. SEDRANSK: It needs to be 11 evaluated. In fact that was what we were 12 saying before, as I was saying at the 13 beginning, there was a rumor about changing 14 this sample composition. What happens, I 15 think it would be fair to say is that, 16 somebody perceives a problem, maybe in one of 17 the sectors and in one area and just adds 18 some sample size to it and it's not done in 19 an overall principled way, so in fact, it's 20 not maintained, so this is no longer, you 21 know, it's no longer a cut off sample. 22 DR. NEERCHAL: Why does the beta BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 23 1 estimate hover around one? 2 MR. SEDRANSK: It does -- 3 DR. NEERCHAL: You explained it but 4 I didn't get it. 5 MR. SEDRANSK: Oh, they do, oh, you 6 mean in those plots in the end? 7 DR. NEERCHAL: Yes. 8 MR. SEDRANSK: Oh, yeah, what I 9 said in the company level plots, what we try 10 to do, this was the first attempt, was we 11 plotted the observed value for the month 12 divided by annual, which is an estimate of 13 the slope. But then the values went all over 14 the place, so the simple idea, divided by 12, 15 put the annual over 12 on the same scale as 16 the month. So it made the plots -- the plots 17 look better. 18 MS. KIRKENDALL: Is the regression 19 actually based on the annual over 12 also, so 20 when you put the regression you'd expect that 21 to be around one -- 22 DR. NEERCHAL: Oh. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 24 1 MS. KIRKENDALL: To show the 2 seasonality or whatever, or you could show 3 market trends, or you could show whatever is 4 going on. But in this case it sort of looks 5 like it's primarily the seasonality. 6 MR. SEDRANSK: Something I should 7 say, I know Nick is -- I'll just comment -- 8 just remark; part of this work or much of it 9 is really a warm up, it's like an exhibition 10 game for dealing with the more complex 906 11 and 920 surveys that appeared in the original 12 title. Those, the data -- the relationships 13 are not as nice or nowhere nearly as nice is 14 what they've showed here. We were trying to 15 go through the exercise for something that 16 was, looked good to get sort of what we were 17 going to do, sort of down pad. 18 And then the next thing is to turn 19 to that, which you presumably will hear about 20 in succeeding talks. This is the data, it 21 really is well behaved and when I talked 22 about the fact that there are these problems, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 25 1 it's not problems but number three, what you 2 may like to do is get some more precision, so 3 that you can reduce the sample size; that's 4 even more important that the other surveys. 5 Nick? 6 DR. HENGARTNER: You talked about 7 outliers and things like that, and I looked 8 at the plots and my first inclination would 9 be to do everything on a log-log scale. 10 Because if you look at some of the plots 11 there, I always have one guy way out there 12 that drives the regression line and just like 13 rescaling often things look -- it's just that 14 the type of plots that's what suggest that -- 15 the other thing, I was wondering you said 16 shrinkage, were you thinking of mixed effects 17 models or even hierarchal models? 18 MR. SEDRANSK: Yeah, you know that 19 was what I -- yeah, when I used shrinkage, I 20 mean -- what I was talking about very 21 specifically, let's take the example of the 22 beta hats, if you used -- if you looked, if I BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 26 1 knew where this was, maybe Lorenz can go -- 2 how do I go back? Go back on -- go back down 3 there. I'll give you an example. Now, I can 4 go forward, I know where to go as long as you 5 got me into a -- no, not pushing out, I'm not 6 going anywhere. Akay, at any rate, take a 7 cross sectional view of those beta hats, keep 8 going, Lorenz, yeah stop there, great. 9 SPEAKER: Can you speak to the 10 microphone? 11 DR. FEDER: Can you speak to the 12 microphone? 13 MR. SEDRANSK: No, I'm finished. 14 DR. HENGARTNER: But here are those 15 regions, what you have is you have one that's 16 on an annual cycle and the other one is on a 17 bi-annual cycle, and those are the regions on 18 the bi-annual cycles are the ones in which 19 you need both heating and air-conditioning. 20 MR. SEDRANSK: Yeah, okay no, no -- 21 I say no. 22 DR. HENGARTNER: Why? BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 27 1 MR. SEDRANSK: Anyway, the point I 2 was going to try to make a technical point -- 3 DR. HENGARTNER: Okay, go ahead. 4 MR. SEDRANSK: Shrinkage 5 Methodology, if you -- the standard shrinkage 6 -- I'll stay over here. Standard Shrinkage 7 Methodology, look at September '03 where 8 there is a kind of a large thing, here we 9 just say, your estimate for one region, take 10 NWE for example, will be that regression 11 estimate, weighted average and that of the 12 others, it's dumb, it doesn't realize the 13 fact that some of the data, not that all data 14 doesn't come from a single source. The 15 methodology I have developed let's the data 16 decide which things should go together. 17 DR. HENGARTNER: But you could 18 actually, since you have the time series, you 19 can think of modeling this d(t) as a function 20 of (t). And you mentioned that. 21 MR. SEDRANSK: Yes, that's right. 22 That was, okay I've told two different BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 28 1 things, right? I would do -- that's the 2 first thing is the time series effects are 3 probably -- in fact you might do it both ways 4 actually. You might have both the time 5 series and all the -- 6 DR. HENGARTNER: Yes, the next link 7 are the time series coefficient. 8 MR. SEDRANSK: Yes, in fact you 9 really want to do this both ways, you really 10 want to do the cross section and the time 11 series parts of this. But sort of not the 12 dumb shrinkage methodology -- well, shrinkage 13 methodology is same as like mixed effects; 14 you don't want to assume that the regression 15 coefficients in some of these time periods 16 come from the same source. They don't. In 17 other periods they do. So the real solution 18 to this which is really interesting 19 statistical problem is the time series and 20 the cross section. 21 DR. NEERCHAL: You mentioned there 22 are only three sample points in the --- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 29 1 MR. SEDRANSK: Lest. 2 DR. NEERCHAL: In the Lest? 3 MR. SEDRANSK: Yes. 4 DR. NEERCHAL: In the frame, how 5 many are there? 6 MR. SEDRANSK: Do we know? 7 DR. NEERCHAL: What percentage? 8 MS. KIRKENDALL: We don't know. 9 MR. SEDRANSK: Do we know off hand. 10 DR. NEERCHAL: Are there lots? 11 MS. KIRKENDALL: More than three. 12 I haven't looked at that. 13 MR. SEDRANSK: Yeah. 14 MS. KIRKENDALL: But that's 15 probably small. There's been other 16 discussion that may be California should be 17 broken into two regions. North and south are 18 very different. 19 DR. NEERCHAL: But there aren't 20 that many IOUs in California? 21 MR. SEDRANSK: Non IOUs. 22 DR. NEERCHAL: Non IOUs? BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 30 1 MR. SEDRANSK: Non IOUs. There may 2 not be, there may not be too many. 3 MS. KIRKENDALL: We have to look at 4 the data. 5 MR. SEDRANSK: Yes. There are 6 more, I'm sure there is more, I'm sure there 7 are more. 8 CHAIRMAN BREIDT: Can you say more 9 about the background behind this pooling 10 methodology? Is there kind of a -- is it 11 like a model averaging sort of procedure? 12 MR. SEDRANSK: Yes. 13 CHAIRMAN BREIDT: With prior 14 subsequent model. 15 MR. SEDRANSK: Oh what you do -- oh 16 it's very simple. You consider all the 17 partitions of all those 10 points, put a 18 prior probability on it, usually equal, and 19 then within a grouping, within the subset in 20 the partition, you assume everything is 21 exchangeable. It works very well, I develop 22 -- first person to ever -- this is -- there BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 31 1 is some literature back to the early 90s, and 2 it works very, very well. I now have a paper 3 to show how to do this for small area 4 estimation, which is really easy to do. 5 If you start with the estimates -- 6 this is actually really easy to do because 7 you -- I'm planning to do it, student who's 8 worked with me on it is very busy. And as 9 soon as we finish -- the reason this is easy 10 is if you start with the summary statistics 11 like the regression coefficient is easy. If 12 you want to read the papers, biometrical 13 paper in '92 and one about 2001 is this is a 14 sinica paper in '99. But it really works, it 15 works surprisingly well this -- 16 DR. HENGARTNER: It sounds what 17 Hartigan was also doing. 18 MR. SEDRANSK: He assumes -- Okay, 19 he assumes is there's not a mistake in what 20 he does, ours is much more general -- 21 DR. HENGARTNER: I'm glad you say 22 that. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 32 1 MR. SEDRANSK: No, no, it is, it 2 is, it is. It's the same, it's the same 3 thing, we were actually working on it at the 4 same time, started working on it at the same 5 time. He assumed something, we assumed that 6 the regression coefficients in the subset are 7 not exactly the same. He assumes they are. 8 DR. HENGARTNER: Yes. 9 MR. SEDRANSK: But once you get 10 into a subset that they have exactly the same 11 values, we allow variability in that. And 12 that's what the difference is. There's 13 unfortunately one of his students wrote a 14 thesis based on it, he's got a mistake in 15 that. 16 DR. HENGARTNER: Yes, I know. 17 MR. SEDRANSK: Oh, you know. 18 DR. HENGARTNER: But I mean in some 19 sense the Bayesian form of clustering. 20 MR. SEDRANSK: That's right. 21 DR. HENGARTNER: I mean, that's 22 really what we're doing here. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 33 1 MR. SEDRANSK: That is exactly 2 right. And in fact if you could do this 3 computationally, this would be Bayesian 4 clustering. 5 DR. HENGARTNER: Yes. 6 MR. SEDRANSK: If you want a 7 another reference which is funny, David 8 Binder who's doctoral dissertation on 9 Bayesian clustering, and has some early 10 papers and it -- was something I looked at 11 when we had started out. Now this would be 12 good. The hurdle is really that with all 13 these partitions, unless you have a way of 14 structuring the population, you get too many, 15 you just, there are just too many of these. 16 DR. HENGARTNER: Couldn't we try to 17 think in terms of the tree, and somehow -- 18 you know split your things in two and then 19 four like a CART. 20 MR. SEDRANSK: You want to try and 21 do CART. I haven't actually thought of that. 22 But that's not -- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 34 1 DR. HENGARTNER: Because if you 2 consider all the all the subsets. 3 MR. SEDRANSK: Oh, it's too many. 4 DR. HENGARTNER: Forget it, but if 5 you do it like they do in CART and things 6 like that, you actually have a prior. 7 MR. SEDRANSK: There is a paper by 8 -- people realized after I wrote my first -- 9 consoniun(?) weren't easy in biometrical. 10 After I published the first paper, did it for 11 the binomial and they -- similar work but 12 similar structure and whatever, but they 13 recognized that there are some structures 14 where you can put in the factorial designs, 15 you only look at sort of some main effects 16 and some interactions. Problems such as 17 this, not so clear how to do it. 18 DR. NEERCHAL: One thing you might 19 want to do is to start by plotting the 20 Bayesian values against those variables that 21 you're not using right now. 22 MR. SEDRANSK: Yes. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 35 1 DR. NEERCHAL: To see if there is 2 some pattern. 3 MR. SEDRANSK: Yes. See what's 4 going on. 5 DR. NEERCHAL: Is there is some, 6 some information there. 7 MS. KIRKENDALL: What value of data 8 that we're not using? 9 DR. NEERCHAL: Like some other 10 characteristics of the business, sales 11 revenue, and things like that. 12 MR. SEDRANSK: Yes. 13 DR. NEERCHAL: I don't know what 14 else do you have there, so. 15 MS. KIRKENDALL: Well, one thing, 16 there should be a relationship between sales 17 and revenues. I mean those are reported by 18 companies, right now we're doing it 19 independent estimates. 20 DR. NEERCHAL: No, no, plot the 21 beta values against some other 22 characteristics of the -- whatever you have BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 36 1 -- let's try to see if there is something you 2 can pull up and that might help you at least 3 in terms of cut down the number of subsets 4 you need to look at. 5 MR. SEDRANSK: Anyway let me switch 6 the discussion because Nancy -- Nancy is not 7 squirming but probably afterwards we do need 8 to make recommendations probably about these. 9 Does anybody know anything about the out -- 10 go back to question number one. Sort of 11 anything about looking at other than 12 influential observations and -- 13 MS. KIRKENDALL: I do want to 14 address implementation as one of the things. 15 MR. SEDRANSK: Yes. 16 MS. KIRKENDALL: That's happened in 17 the past is that the people who process, who 18 collect the data and process it, they get it 19 ready, they do whatever edits they think they 20 need to, they think it's fairly clean and 21 then it goes to an estimation program. And 22 there's a lot of information that comes out BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 37 1 of the estimation as a result of these 2 scattered plots and outliers and what not. 3 And there has not been a feedback to the 4 staff. So one way you could do something -- 5 and then they do more detailed analysis, but 6 they typically find it when somebody looks at 7 a total number and it's off it. 8 So you get an error and it goes 9 through and really messes up your total 10 estimate. And may be caught at the end, or 11 it may be caught later, and it may not be. 12 So the thought on the outlier detection was 13 just take it out and make that outlier so it 14 doesn't affect your estimate for everybody 15 else, feedback that information about the 16 outlier so people can check up on it. So 17 it's to try to minimize the impact of one 18 funny observation on the estimate of the 19 total that you're producing. 20 MR. SEDRANSK: But basically this 21 is an automatic, sort of an automated 22 procedure. Particularly this survey which is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 38 1 operating well, it doesn't get a lot of -- it 2 doesn't get as much attention in some ways as 3 other's do. So if you don't -- you've got to 4 have something that's automated. But you 5 know -- it's sort of, it's something strange, 6 really strange happens that you don't get 7 blown out of the water. 8 MS. KIRKENDALL: I mean sometimes 9 these are real data to -- there's a huge 10 changes in the company. 11 DR. NEERCHAL: If you classify 12 something as an outlier, then it will not be 13 used for the estimate prior? 14 MS. KIRKENDALL: Yes, it would just 15 not be used on data. It would be added back 16 in at the end. 17 MR. SEDRANSK: Yes. We always, we 18 always use it for -- I mean assuming it's 19 correct, you know it's deemed to be correct. 20 MS. KIRKENDALL: But then you do 21 want to check on it. 22 DR. NEERCHAL: I mean if something BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 39 1 -- some -- they have some special thing going 2 on that particular data value then it will 3 marked that as an outlier -- 4 DR. BURTON: It will still be added 5 back at the end of it as far as -- if it's 6 sure to be valid. 7 MS. KIRKENDALL: Right actually 8 that's one thing, there was some discussion 9 yesterday about over editing. EIA, we do 10 editing but we typically do not impute for 11 failed data. We try to phone on it, 12 certainly all the big companies. So while we 13 do the editing as an agency, which is across 14 most of our offices, we do not rely on 15 imputation for failed data. So we are a 16 little different from many statistical 17 agencies. 18 MR. SEDRANSK: Let me switch to a 19 different-- 20 DR. BURTON: Would your data be 21 cleaner? 22 MR. SEDRANSK: I am going to ask BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 40 1 another question which I tried to the get the 2 -- draw out the answer to. If in a single, 3 in an establishment survey and you try to do 4 stratification -- here we're doing post 5 stratification, not doing stratification -- 6 but you sort of do probably something like 7 sales or revenue are probably highly 8 correlated, profits but pretty not -- not may 9 be profits, sales and revenues, it's 10 stratify, if you had the opportunity -- you 11 would just do something like the cumulative 12 square root of F method, which is been around 13 for forty, or forty or more years. 14 This case you don't have that, 15 you're looking for common features of the 16 regression coefficients. So I just have 17 question about how you might build up strata, 18 just forget about climate and temperature. I 19 mean this is not smart, just totally forget 20 about it. How do you sort of build -- how do 21 you build up strata from the ground up, is 22 the only way I can describe it, any ideas? BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 41 1 MS. KIRKENDALL: By strata you mean 2 the estimation? 3 MR. SEDRANSK: Estimation groups in 4 post strata, yes, yes, right. Or turn it 5 around and say we're getting a new -- they 6 were in cut off samples, as far as you're 7 doing a probability design, performance 8 strata in the first place. I'm not 9 suggesting we're doing this but -- 10 MS. KIRKENDALL: Actually one thing 11 for all of these observations, we do have the 12 state that they operate in. So we can look 13 at any different groupings that we think we 14 need to. 15 MR. SEDRANSK: Yes. It's just I 16 was thinking of a just primitive method, or 17 where you start from. I mean there are 18 things you can do, you can just get estimates 19 of the regression coefficients that are just 20 monthly over annual, you have a whole batch 21 of them for each company. I just - I mean 22 I've suggested that, but I --just curious if BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 42 1 anyone has any other ideas. 2 CHAIRMAN BREIDT: But the 3 regression coefficients you get at the 4 company level are, these are based on -- 5 MR. SEDRANSK: One observation. 6 CHAIRMAN BREIDT: Yes. 7 MR. SEDRANSK: Per month. But 8 you'd actually get a bunch of months of them, 9 so you -- 10 CHAIRMAN BREIDT: Then the same 11 month across the years and that regression 12 coefficient. 13 SPEAKER: Yeah. 14 CHAIRMAN BREIDT: We're talking 15 about. 16 MR. SEDRANSK: Well, we got both 17 those, we've got the same month over the 18 years, we've got all this, we've got the time 19 series data and whatever, you could take all 20 of this and see what -- put it in the cluster 21 analysis package and see what do you get. 22 CHAIRMAN BREIDT: Right, it doesn't BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 43 1 see that you want to through all the months 2 together on some of these plots. 3 MR. SEDRANSK: I'm not sure if this 4 is important. I mean I think the temperature 5 based thing makes sense, although I think the 6 ones we've got are -- I think ones we've got 7 -- I think there are some other units. I did 8 that and I'd be surprised -- this one just 9 looks too large, that north east one. 10 DR. SITTER: Well, I mean you could 11 also use it as a validation, if we came up 12 with a method to see if it actually is a 13 decent grouping. But you can just quantify 14 the problem, you've got a variance estimator, 15 obviously? 16 MR. SEDRANSK: Yeah. 17 DR. SITTER: Okay, it's going to 18 depend on the group, throw it into your 19 favorite search algorithm for finite grouping 20 searches, clustering, you can just tick away 21 random groupings. I mean computers are fast, 22 you can quickly find out things are BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 44 1 reasonable. And I've seen them do this with 2 genetic algorithms or simulated annealing, 3 anything that doesn't require any derivatives 4 or anything. They're not really fast, but 5 you know speed isn't of the essence. 6 DR. HENGARTNER: Randy we're trying 7 to group thousands of companies, correct? 8 DR. SITTER: No, just states -- 9 MS. KIRKENDALL: There are only - 10 there are less than 200 that wer'ee actually 11 using in this regression, we have about 400 12 observations if you count the IOUs that are 13 currently not used to any regression. 14 DR. HENGARTNER: So 200 and you are 15 trying to group them in? 16 DR. SITTER: It's not too bad. I 17 mean you can even use some of the scheduling 18 methods where they want to schedule jobs to 19 computers. There are some very simple 20 algorithms that are very fast, they are not 21 going to get you the optimum, but they will 22 get you something very good. And they are BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 45 1 very simple, they just order it on some 2 reasonable proxy and then they take the 3 biggest ones, put one in each group, and then 4 they just tick the next guy and say, well, 5 which group can I put it in that will 6 increase the variance the least, and they 7 just do that bang, bang, bang, bang; it's 8 very fast. And you come up with a sub 9 optimal solution but I'm -- 10 DR. HENGARTNER: Which is pretty 11 good though -- 12 DR. SITTER: But it's pretty good 13 on average. 14 DR. HENGARTNER: I don't have 15 experience with -- you know that's the 16 Bayesian method. But I have the feeling that 17 if you tried to prove that, you are going to 18 have a lot of local minima's. 19 MR. SEDRANSK: Oh, no, I wasn't 20 even thinking of Bayesian, this was not 21 Bayesian -- 22 DR. HENGARTNER: Oh, that wasn't BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 46 1 you -- I was thinking of. 2 MR. SEDRANSK: I do all kinds of 3 things, it depends on what day it is. I 4 think it like a Bayesian, but I -- 5 MS. KIRKENDALL: One of the things 6 that I should say is this the original system 7 and the estimation methods were done by Jim 8 Knob who was hiding up there or who is trying 9 hiding up there, in the active performance in 10 alternative fuel. And he presented a lot of 11 the information to the ASA committee back in 12 the early the 90s, I think some of it was 13 when Joe was the chair of the committee. So 14 this is an example of a problem that keeps 15 coming back. 16 MR. SEDRANSK: I mean essentially 17 things are working well here, and this is 18 sort of a prototype. But there are these 19 issues, there is certainly desire to cut back 20 sample size, may be not so much in this one 21 as in 906 and 920. So can one make use of 22 this time series, information is also issues, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 47 1 you know academically it's of interest 2 whether you can make some use of it. One of 3 the problems which we did not allude to is 4 there was a major in the time series when 5 you're going to have trouble making a longer 6 time series because in 2004 there was a major 7 shift that other was sent away and replaced 8 by transportation. So that was a -- so 9 you've got a problem of different 10 distribution among those end users after 11 that. That's one of the reasons we don't go 12 after 03, then 02 and 03 are -- you know are, 13 are clear cut, so we won't have a long -- 14 MS. KIRKENDALL: Residential may 15 still be okay. 16 MR. SEDRANSK: What? 17 MS. KIRKENDALL: Residential may 18 still be okay. 19 MR. SEDRANSK: Residential probably 20 be okay. 21 CHAIRMAN BREIDT: Well, it looks 22 likes you have spatial(?) structure too. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 48 1 MR. SEDRANSK: Yes, do something 2 here. 3 CHAIRMAN BREIDT: Do something 4 here, especially in the finding the time 5 series and the spatial structure. 6 MR. SEDRANSK: Yes. 7 CHAIRMAN BREIDT: Allowing for the 8 fact that over time that their ability 9 changes a lot in states so that you have a -- 10 MR. SEDRANSK: Yes. This looks 11 like a, this is a good problem. 12 CHAIRMAN BREIDT: Yes. 13 MR. SEDRANSK: I found a similar 14 one at the census bureau also recently which 15 I now can't remember, carrying those small, 16 small area overtime, there's a - they've been 17 doing it this way, but I detected that 18 they've got some of the same structure 19 problem. 20 DR. HENGARTNER: Joe, have you 21 thought of looking at this as a geo temporal 22 process? And I'm now thinking, I know the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 49 1 location of each of the generators, right? I 2 always thought that there is some spatial 3 correlation instead of just grouping them 4 into regions, let's look at some kind of a 5 diffusion those must be correlated in some 6 ways. And we have awesome barriers like the 7 mountain ridges, whatever, I mean we actually 8 know about the geography. 9 MR. SEDRANSK: No, we haven't 10 because we've looked at really, we're 11 evaluating their methodology, it doesn't -- 12 by the way it's a good idea -- 13 DR. HENGARTNER: I mean you're 14 asking for what goes through my mind -- 15 MR. SEDRANSK: No, no, that's a 16 good -- no this is. I'm down on, --I'm down 17 geo spatial stuff, I have a student who's 18 been working on a problem and I don't want to 19 tell you what troubles he has had. These are 20 binary responses though in a geo spatial 21 setting and I can tell you things that don't 22 work, lots of things that don't work. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 50 1 CHAIRMAN BREIDT: Okay, any other 2 comments? 3 MS. KIRKENDALL: Okay, you have any 4 brilliant ideas? 5 MR. SEDRANSK: Email. 6 MS. KIRKENDALL: Email works. 7 SPEAKER: I haven't had one of 8 those in a while. 9 CHAIRMAN BREIDT: So it's now time 10 to move to our break out sessions. The next 11 session will be downstairs on the post 12 stratification methodology. 13 (Recess) 14 MR. HOUGH: Desperate to get 15 started. I was just informed that there -- I 16 guess we'll be chairing this ourselves. It's 17 a scary thought I know but -- My name's Rick 18 Hough, I am here from the Census Bureau, I am 19 the survey manager responsible for the 20 Manufacturing Energy Consumption Survey. I 21 am here to talk to you a little today with my 22 colleague, Stacey Cole, who's the brands BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 51 1 chief responsible for the R&M brands that has 2 MECS as one of its many responsibilities. We 3 want to talk to you today a little bit about 4 the post stratification methodology that we 5 implemented during the 2002 Manufacturing 6 Energy Consumption Survey. 7 Before I begin, I'd like to take a 8 quick minute and acknowledge everyone who 9 worked hard on the 2002 MECS, Susan Bucci is 10 with us today, she's my boss, she's the 11 brands chief, in charge of the brands where 12 MECS is conducted, members of my staff who 13 were responsible for data analysis, Vicky 14 Haitot, Lacy Loffin, and Eva Snap. I would 15 also like to recognize the methodologists on 16 Stacey's staff, who were responsible for 17 coming up with a lot of the details that 18 you'll see in today's presentation, Jeff 19 Dalzell, Cathy Gregor, John Slanda, who's 20 here with us today in case any technical 21 questions would arise, and Bob Struble. 22 I'd also like to take a quick BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 52 1 moment to acknowledge our members of EIA who 2 worked on the Manufacturing Energy 3 Consumption Survey. Dwight French is with us 4 today. Dwight gave the approval to go 5 forward with this process, came out to census 6 few times, listened, gave comments, and then 7 told us to go ahead, Bob Adler, who was the 8 other survey manager I saw, my co-survey 9 manager from EIA who also worked very hard, 10 and Tom Lorenz who some of you heard from 11 yesterday, survey analyst from EIA who came 12 out for census quite a bit to work with our 13 team on data analysis and preparing reports 14 and he was a very valuable member of our 15 team. 16 Just an overview of what we're 17 going to talk a little bit about today. I'm 18 going to talk to you a little bit about the 19 background and the goals of the Manufacturing 20 Energy Consumption Survey. I'm going to give 21 a brief description of the economic census, 22 talk a little bit about some of the aspects BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 53 1 of the census that affect the reasons why we 2 Post Stratified, and also some of the files 3 that are involved with Post Stratification. 4 Stacey is going to come up and talk 5 a little bit about the sample design, and 6 then he's going to get in to the actual Post 7 Stratification methodology and then we have 8 some results of what we implemented to talk 9 about. And then, finally we have some 10 questions for the panel that we'd like to 11 consider, some questions about the 12 methodology that we implemented for this 13 survey, and if time permits we'd also like to 14 talk a little bit about what we might 15 consider in the future given the time frame 16 of MECS and how it is ultimately related to 17 how the census is conducted. 18 Okay, the Manufacturing Energy 19 Consumption Survey. Basically our goal is to 20 provide detailed data on energy consumption 21 for the manufacturing sector. We provide a 22 variety of information from individual energy BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 54 1 sources, such as electricity, natural gas, 2 selected fuels. We also provide data on 3 industry as well as geographic. We report 4 statistics for some of the energy saving 5 technologies that are implemented in 6 manufacturing establishments as well as their 7 ability to switch to alternative fuels in 8 certain situations. 9 The survey was initiated in 1985, 10 it was conducted every three years until the 11 survey year 1994 and it's been conducted 12 every four years since that point. The 13 economic census, just some general 14 information, provides a detailed portrait of 15 the nation's economy. We provide data on 16 industry as well as geographic, we provide a 17 variety of maniputable formats for data users 18 such as CD-ROM's, we have an American Fact 19 Finder on our website where the data user can 20 go in and get data and create their own 21 tables. 22 The census is conducted every five BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 55 1 years; it is conducted in years that end in 2 Two and Seven. It is conducted using the 3 North American Industry Classification 4 system. This is the system that was brought 5 on to replace the old SIC. The 2002 census 6 was the second census conducted using these 7 classification definitions. 8 Okay, and the census provides a 9 comprehensive update of the classifications 10 that are contained in the Census Bureau's 11 business register. A lot of the companies 12 within the business register do not get 13 Census Bureau questionnaires every year. 14 Therefore, the update path that we have from 15 them to update their classifications is done 16 through the census. 17 I want to talk a little bit about 18 the 2002 economic census for the 19 manufacturing sector. The census bureau does 20 not mail every establishment in the country a 21 questionnaire. We have what's called a 22 non-mailed file, it is defined as single unit BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 56 1 manufacturing establishments with less than 2 five employees and these establishments were 3 not mailed questionnaires. For the year 4 2002, this file contained approximately 5 160,000 establishments. 6 The Census Bureau uses 7 administrative data from other government 8 agencies to estimate for these establishments 9 and they account for approximately 3 percent 10 of the published totals. Now, if you have a 11 non-mailed file, then you have a mailed file, 12 and the definition of the mailed file is the 13 ones you don't -- the ones you sent 14 questionnaires to. They are the single unit 15 establishments with more than five employees 16 and all multi-unit manufacturing 17 establishments were mailed questionnaires. 18 DR. HENGARTNER: Now how do you 19 have those addresses? 20 MR. HOUGH: Other government 21 agencies. 22 DR. SITTER: Homeland Security. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 57 1 DR. HENGARTNER: Good answer. 2 MS. DODDS: We do some surveys in 3 the years between the census and many of the 4 addresses come from those, and others would 5 come from other government agencies, taxing 6 agencies. 7 DR. HENGARTNER: IRS. 8 MR. HOUGH: Okay, the mailed file 9 contained approximately 200,000 manufacturing 10 establishments for the survey year 2002. 11 Okay, the data that's collected during the 12 census from the manufacturing establishments 13 -- we collect operational data on employment, 14 receipts, inventories, costs, assets, 15 purchase services, et cetera. Among the cost 16 data we collect from the establishments are 17 cost of fuels and cost of electricity. 18 And when Stacey comes up, he's 19 going to read for quite a bit to this cost of 20 energy, that is what we use to define the 21 cost of energy. We take the cost of the 22 fuels reported and the cost of electricity BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 58 1 reported by the establishments and that is 2 how the cost of energy is defined. I'm going 3 to defer now to Stacey and he's going to 4 start off by talking a little bit about the 5 sample design. 6 SPEAKER: Which one's up? 7 MR. COLE: Good Morning. Basically 8 my area was responsible for going to in 9 developing a sample design for the MEC 10 survey. The key goal of MECS is to produce 11 detailed aspects of energy consumption at the 12 national level for industries as well as 13 regional level for industries. The target 14 population is basically the mail panel of the 15 -- every portion of the census. Single 16 location establishments with more than five 17 employees and all multi unit establishments. 18 The mail file was used as just 19 said, every establishment in the mailed file 20 was classified to a specific industry code, 21 and everyone has to sign cost of energy data 22 which came from historical sources. For many BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 59 1 cases the cost of energy data came from the 2 2001 annual survey manufacturers, for other 3 cases it came from the 1997 government 4 census. So in some cases, it's very, very 5 old. 6 We used a salvation approach to go 7 through the sample, there were 473 specific 8 manufacturing industries, way too many to 9 cover in detail for MECS but fortunately 10 energy use is concentrated to a great deal. 11 There are 37 industries that are of high 12 interest to EIA, for the most part these 13 account for almost half of the energy 14 consumed in manufacturing. The balance of 15 the -- the remaining 440 -- 400 and whatever 16 is -- 436 whatever it is -- industries were 17 collapsed to higher levels, so between both 18 groups that cover the can clear the entire 19 sector of manufacturing. 20 And also there was a desire for 21 some states and some sub- national level 22 data, so we collapsed the stage to Census BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 60 1 Bureau regions. There are 267-industry group 2 by recent strata, there was one that doesn't 3 exhibit basically an empty set. There are no 4 establishments. The allocation was also 5 challenged, we have a sample of only 15,000 6 establishments. 7 The establishments both the 8 collection unit with the mail unit as well as 9 the collection unit, the collection unit as 10 well as the sample unit, and also the 11 collection unit. We have a sliding scale of 12 reliability constraints, basically we think 13 the cells that have the most amount of 14 activity are the cells of the highest 15 interests, so we tried to do a really good 16 job in the cells with a lot of activity. 17 The smaller cells have fairly high 18 C V constraints to inflation. I believe the 19 C V constraints range from 2 percent to about 20 10 percent. Again the big cells had higher 21 constraints. The section approach basically 22 is, an independent sample was drawn from each BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 61 1 one of the 267 stratum, within each stratum, 2 we've a PPS approach. The establishment size 3 does matter, establishments that have a large 4 consumption of energy had a -- a very high 5 problem with selection. Again, the energy 6 data we had for such level came from the year 7 19 -- 2001 ASM or historical data. 8 We also imposed a minimum 9 probability of.02, this helps to prevent in 10 the cases where we end up with large weights 11 and surprisingly large data, which causes 12 problems for estimation as well as the 13 variances. Estimation is really a four-stage 14 process, I only have three on here, but the 15 first stage is obviously to collect the data 16 and like most surveys, we do not get all the 17 mailed cases in. There's a small amount of 18 all that -- not a small amount, there's about 19 a 25 percent non-responses rate to a survey. 20 We mailed out 15,000 forms and we 21 got about 12,000 back in. Historically we 22 have done a non-response weight adjustment to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 62 1 MECS and this time we did the same thing. 2 The first and third board there, I think, 3 we've done historically in MECS. The middle 4 board is something basically brand new. In 5 non- response weight adjustment process, we, 6 no, go back a second, we did it by stratum. 7 With each stratum we said, we broke down in 8 to sub-stratum, we said within each stratum 9 there are certainties and non-certainties, 10 response rate varies between the two 11 subgroups, and we wanted to go through and 12 adjust their weights separately. That's also 13 an enhancement for 2002 MECS. 14 It is basically a standard form of 15 the non-response weight adjustment, where the 16 numerator is basically equal to the weighted 17 data from the entire sample on a sample frame 18 and the denominator was the observed data 19 from respondents, I'm sorry, it's the data 20 trivial to respondents. So every response in 21 the cell that did the adjustment factor is 22 equal to 1. In most cells that one didn't BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 63 1 happen. So at that point we adjust the 2 sample weights to reflect non-response. The 3 post-stratification, why we do this? We are 4 concerned about sample frame. 5 Basically we are using a 2002 mail 6 file, we are worried about the coverage of 7 the target population, we are worried of the 8 classification of the el, and we are worried 9 about the cost of energy data being accurate. 10 The coverage, we know that the mailed file 11 includes records that will ultimately not be 12 in your factory when your mail questionnaire 13 is censused, they'll come back in 14 re-classified into wholesale, retail or 15 construction or other areas. 16 Initially there were cases data 17 being mailed as retail/ wholesale for census, 18 they can be classified in manufacturing. You 19 know, that's a problem. In addition, we have 20 cases that -- the classification within 21 manufacturing will have changed between 1997 22 and 2002 and this affects the thin strata of BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 64 1 variability, and the cost of energy data is 2 also a problem because it's quite volatile 3 year to year, in addition for most of the 4 records, the last testimony of the 5 consumption is five years old. So, we had 6 all those concerns. And that prompted us to 7 go through and examine the idea of Post- 8 Stratification. 9 Fortunately, we had the Econ census 10 as a source to go through and do the post 11 stratification. It provides a comprehensive 12 collection of all factors; it allows us to 13 update the classifications of everybody in 14 the Econ directorate, and also we do collect 15 the cost of energy for all manufacturing 16 establishments. So we were able to go 17 through and basically re-assemble the frame 18 based upon the more current information, 19 which is what we did. 20 For each stratum, we identified the 21 eligible units in the stratum, we summed 22 their cost of energy to the stratum level, we BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 65 1 also developed an estimate of that from the 2 sample cases in that stratum, so we were able 3 to do a comparison between the target 4 population energy data and the sample 5 estimate of that, and here's a graph that 6 shows the comparison between the actual and 7 the estimated total. 8 For the most part, the estimates 9 are pretty close to the actual data, but 10 there are a handful of cells where the 11 estimate is quite a bit higher than the 12 actual total from the census. Look at those 13 cells and determined that -- that cause was 14 caused primarily by the big weight and big 15 data problem, had some cases we thought they 16 were quite small, but in reality they end up 17 being quite large. 18 So we had a handful of cells where 19 the estimates was quite a bit greater than 20 the actual population total. We are 21 looking for a function that will go back and 22 adjust the sample weights of the sample BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 66 1 cases, so that the weight adjusted estimate 2 will equal the controlled total. We wanted a 3 function where we didn't adjust the weights 4 of a certainty cases. They are a 5 self-representing cases during the selection 6 process and we want them to remain that way 7 during the estimation process. 8 We also wanted to avoid a problem 9 of lowering weights on non-certainties to the 10 point where they became less than one, which 11 is kind of a hard concept to understand, 12 weight less than one. So we develop a 13 process where the K is our factor, K is 14 linear, it goes to the point 1.1 and 1.1 is 15 the axis, it swings round that at a point, 16 and it's the purple area, if we have the 17 estimate as less than the population, we have 18 to adjust the weights up. 19 In the blue area, we are saying 20 basically the estimate is greater than the 21 population and we have to adjust the weights 22 downward. That's the graphic display. And BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 67 1 here's the actual formula that we used, 2 basically numerator represents the amount of 3 energy data associated with establishments 4 are not part of the sample. Basically it's 5 the cap end of the total, the first time is 6 the published total, and the second time is 7 what we observe in the MECS sample. So 8 basically the numerator is what was not in 9 the MECS sample. And the denominator is 10 basically an estimate of the numerator. 11 So hopefully, our ratio is close to 12 one, and if it's one, we are really happy. 13 The sample was -- after we get the case for 14 each stratum we go back and we adjust the 15 sample weights further by the K. Just look 16 at the top of the slide there, you can see 17 that the certainty cases, the cases where 18 WI=1, are never going to be adjusted. 19 They're going to remain -- retain their 20 sample weight of one, and never will it get 21 to miss an adjustment, but it's not a linear, 22 but not proportionate. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 68 1 In my example I have got myself -- 2 got cable.8. The first by the weight of two 3 goes done by only 10 percent to 1.8 but the 4 weight of 20 goes down by almost 20 percent 5 down to 16.2. So basically the 6 establishments with higher sample weights are 7 more impacted by the adjustment than the 8 cases with small sample weights. And I just 9 said that, okay actually in truth when we 10 classify the population and develop a control 11 total, we had almost a 1,000 establishments 12 that switched strata within manufacturing. 13 It's a little high than we expected but they 14 have often verified. 15 The overall impact on the estimate 16 of the energy at the national level was to 17 provide the down load by about 1.8 percent. 18 About half this stratum was just up and half 19 went down. So we are pretty pleased it 20 wasn't anything systematically suggesting 21 that we had a upper bias, redundant bias and 22 for the cells that were adjustable, that were BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 69 1 adjusted, the average absolute adjustments a 2 little under 4 percent well, 3.7 percent. 3 So we weren't making major 4 adjustments on most of the cells. There were 5 a handful of cells that went down by a 30 or 6 40 percent. The color adjustment, this has 7 been done in the past, this is not a new 8 activity for MECS. The sample frame is a 9 mail file, it does not include those were 10 identified after we mailed out the census and 11 not they are not part of the mail file, they 12 are not put in the estimates and in addition 13 there are cases that were mailed as retail or 14 wholesale who were reclassified in the 15 manufacturing. Those cases again not in the 16 sample frame, so they are not included in the 17 estimate right now. That's a problem that we 18 are trying to address. 19 Here is a frame, nice big graph and 20 a pretty picture, and those are the sample we 21 picked originally, again the sample of frame 22 was a mail sample. We have cases that became BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 70 1 ineligible, these are cases that died, went 2 on ---- or duplicates, cases where not 3 manufacturing. We didn't know them, that 4 they were advanced, so we drew a sample of 5 them. So our sample represents the number of 6 deaths in the actual frame, so these cases 7 can be removed from MECS, and they were. 8 But the births and the -- in the 9 transport and manufacturing were not in a 10 sample frame and yet they are still part of 11 the total population. We like to have them 12 included in the estimates but we had no 13 sample of those cases. So we basically went 14 through identified them all for about 15,000 15 births and incoming transfers who were a part 16 of the total population, which is one of the 17 cases we did have cost of energy data and we 18 developed a revised stratum cost of energy 19 control total by simply adding the energy 20 cost associated with these cases to the 21 originally control total and develop a final 22 adjustment where we had a revised control BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 71 1 total cost of energy going up by the original 2 control total. 3 And then we get a final way and we 4 applied that to establishments in each cell. 5 This is done at each one of the cells levels. 6 About 3/4 of the strata were adjusted upward. 7 The remaining quarter, they were no births or 8 there no incoming transfers, cases like 9 chemical plants and refineries, pulp mills 10 are not likely to have incoming transfers and 11 their adjustment was about 3 percent. So 12 that point we finished the process we had 13 gone through, we are adjusting our response 14 that does it for the economic census and then 15 adjusted for the coverage of the original 16 sample frame and that's where its finished as 17 we are right now. These are the questions we 18 got and I see these were in right ballpark, 19 are we committing heresy here. 20 Isn't that all that you recently 21 found and we forgot something; we miss a big 22 issue, and are your alternative that maybe BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 72 1 more appropriate. And the real one that I'm 2 looking for is, what do we tell the EIA and 3 their users about what we did, so they can 4 figure out how to use data. That's it. 5 Questions, comments? 6 MR. HOUGH: You are the chair. 7 MR. COLE: I'm the chair. Direct 8 all the questions to Rick. 9 MR. HSEN: Stacey, you astounded 10 them. 11 MR. COLE: I'm sure, I astounded, 12 oh yeah, the great minds here I'm sure I 13 astounded the great minds. 14 DR. SITTER: You chose post 15 stratification, you really calibrating, I 16 mean you using post stratification in 17 calibrating. 18 MR. COLE: That right. 19 DR. SITTER: You look at 20 calibration then? 21 MR. COLE: No, we do not look at 22 calibration methods. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 73 1 DR. SITTER: Some of the 2 calibration methods will allow you to put 3 constraints on them, which is essentially 4 what you have, that is you have a constrained 5 calibration problem. I think I can probably 6 come up with two or three references. You 7 can take a look at them. I'm not saying that 8 we better -- 9 MR. COLE: Is that done by a 10 variable, if it was a one calibration for the 11 entire stratum, I didn't -- 12 DR. SITTER: I think anything is 13 possible, as just viewing it as a calibration 14 problem may give you some insight as to where 15 you're sitting, you may in fact be in 16 calibration, if you can, I don't know. 17 MR. COLE: Okay. 18 DR. SITTER: I can give you a 19 couple of references that I know of. 20 MR. COLE: Okay, appreciate that. 21 DR. HENGARTNER: Randy, I don't 22 know that literature, how sensitive are these BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 74 1 methods to randomness, I mean the way to get 2 their attention random variable. 3 MR. COLE: Yes. 4 DR. HENGARTNER: Because the 5 frames, well, we estimated them, the frames 6 change into both. Does that have an impact? 7 DR. SITTER: Of course, what are 8 you really meaning, are they really sensitive 9 to that or something like that? 10 DR. HENGARTNER: I mean the analogy 11 is it like errors and variables problem or 12 not. 13 MR. COLE: I wouldn't classify 14 myself as an expert. John, he is my wizard. 15 MR. SLANDA: Well, we looked at the 16 weights and the impact on variances -- 17 DR. FEDER: Okay. 18 MR. SLANDA: And how they did that, 19 we treated, we looked at -- we used the total 20 cost of energy and then we used electricity 21 as the variable from the actual survey data 22 because we needed something to like a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 75 1 by-variant along with that and we did 2 variances on the -- before and after the non 3 response adjustment and after the K 4 adjustment and about 80 percent of the 5 variances went down. But our big question 6 became, how would me measures bias and that 7 was something we hadn't come up with yet and 8 we are so interested in knowing how to 9 approach and attack that problem. 10 DR. FEDER: Actually you also 11 talked about -- felt that the stage for my 12 question is, could you do stimulations to 13 examine the bias issue, because by looking at 14 your formula it's not apparent to me that 15 there is no bias here because of the 16 selection issue because you are changing the 17 weights in a disproportional way, so 18 obviously the Horowitz-Thomson is out of the 19 window, so I would recommend doing some 20 stimulation and see again, if you get good 21 results I think I certainly would say that 22 the good paper here. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 76 1 Now I -- then you revise you 2 approach, because I haven't seen something 3 quite like it but it might make sense. And 4 actually not just bias or variance, I would 5 look at them into -- together, because any 6 calibration method might introduce some bias 7 but reduce there miniscule error and this 8 after all would be after, I mean, a little 9 bit of bias is okay, I learnt from Randy. I 10 have questions for Randy although quite a bit 11 older I mean his contribution is just obvious 12 so -- 13 DR. SITTER: Yeah, a bit 14 intimidating he already had a PhD in 15 mathematics. 16 MR. SLANDA: Well, I attempted to 17 actually look into some of that simulation 18 and I was wondering, have an idea I could 19 just bounce off, see it works but I was 20 thinking of using the actual survey data to 21 get a regression but then also modeling some 22 type of error term around the regression so BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 77 1 that when we impute back, so then everybody 2 not in the sample then would have much more 3 variability as opposed to just having measure 4 of size there because if we have the measure 5 of size and we are doing simulation on that, 6 then we get funny results too. 7 DR. FEDER: One of the things that 8 you could also do -- stimulation is pretend 9 that 20 percent you assembled that responded 10 and 20 did not respond, take them out and try 11 to figure a method that would give you 12 anything close to what you actually have. So 13 -- 14 MR. SLANDA: So label about 20 15 percent of the population? 16 DR. FEDER: Yeah, I mean if you 17 stimulate no response by making some you 18 know -- you play God, you make them no 19 respond, and then you see you will have to 20 predict them the right way, you know and try 21 to make it a bit informative, make that no 22 response -- some known attributes of those -- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 78 1 DR. HENGARTNER: For example more 2 non responses of the smaller companies will 3 probably what we tend to generalize though, 4 the step, you know, the assumption that 5 moderately violated. The same thing is that, 6 we had the discussion about your estimate 7 with those weights, are you using the highest 8 estimate or the Horowitz-Thomson? 9 MR. COLE: Horowitz-Thomson. 10 MR. SLANDA: Horowitz-Thomson 11 estimate. 12 DR. HENGARTNER: Okay, there was 13 discussion and Randy suggested how it might 14 be better for some cases. 15 DR. FEDER: Randy also suggested to 16 you some calibration methods and once you 17 calibrate if you include in you calibration 18 constraints the population side, then you are 19 in fact doing high. 20 DR. HENGARTNER: Yeah but then 21 because you weights are random then you get 22 really the errors and variables problem and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 79 1 then you have to worry about attenuation 2 which is a few random weights. 3 DR. FEDER: It's a very complex 4 thing. That's why I think if you do 5 something in relation it could be some 6 cancellation of some of these errors. I 7 think it's and interesting approach affecting 8 the issue but you know, that the people that 9 represent themselves on the weight response 10 would be not change their weight but go until 11 -- 12 DR. SITTER: I'll give you why the 13 simulation is probably your best route I mean 14 I stayed so bluntly that this confront into 15 the class of constraint calibration problem, 16 if you look at the papers that I would give 17 to you none of them actually proved any 18 reason, any consistency results under any 19 reasonable framework as to what a consistency 20 result should be. Certainly they left the 21 simple size go to infinity but what you 22 really need, they discussed the consistency BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 80 1 probably more realistic framework but it 2 hasn't really been proved yet. 3 And it really comes down to that 4 constraint that is you -- because of those 5 constraints, you're disproportionably 6 adjusting the weights and that's really going 7 to play a role and that you know, if you just 8 sort of set up an obvious fixed framework as 9 N goes to infinity, you are fine but you have 10 to imagine a situation where those 11 constraints are also not staying fixed with N 12 and that I don't think anybody has actually 13 come up with a ---- it's not that they can 14 prove it, it's that you could come up with 15 such a thing but it would be sort of made up 16 with you know, what does it mean 17 realistically to say as your sample size 18 increases or your number of strata increases, 19 what happens to those constraints. 20 That's an issue, I like, in your 21 situation your constraints are the certainty 22 versus non certainty. So you wanted to set BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 81 1 up a asymptotic framework, you have to have 2 some idea of what happens with those 3 certainty versus non certainty weights as 4 your sample size increases. And that really 5 impacts some products, so you are really 6 stuck with simulations. 7 DR. FEDER: In this case your point 8 is even more important because we are dealing 9 with a very ---- population which is also the 10 measure of size, the energy consumption, so I 11 think when you do this disproportionate wages 12 adjustment, it might have an attenuated 13 effect and I think it is really worthwhile to 14 examine. 15 MR. HOUGH: I have one other 16 question to you, I certainly have time, we 17 were able to do this this time because this 18 -- MECS is currently conducted every four 19 years. The census is conducted every five. 20 This is the first time that the two surveys 21 fell in the same survey year. So we were 22 able to pull a sample from a mail file and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 82 1 stratify to a final file. 2 In the future for example, in '06, 3 we will pull the sample from the completed 4 '02, which is what we stratified to here. 5 But things like Stacy talked about will 6 happen. We do an annual survey on 7 manufactures, which is a sample of the 8 census, that kind of fills the gaps between 9 census years for manufacturing data. So the 10 question would be, if we consider doing this 11 in the future and the two surveys don't 12 coincide, is there anything we should 13 consider if we think about stratifying to 14 another sample of the census. 15 So we would want to use, say the 16 annual survey manufactures are the control 17 total? Is there anything we should consider 18 in doing that, I mean those actual control 19 totals will now contain sampling errors is 20 one thing, the estimates will themselves 21 contain sampling error and the ASM itself is 22 not really set up to give point estimates BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 83 1 although they are fairly good. 2 DR. SITTER: Well, I think you are 3 going to run into, I mean conceptually run 4 into same issues that we saw yesterday with 5 the Natural Gas essentially. I mean there is 6 going to be a change in the relationship over 7 time because you sort of have certain 8 determined your stratification on the basis 9 of old information and the relationship is 10 going to change over time. 11 So looking at it back over time and 12 what's happened you may see a pattern as to 13 how those things might change. I am speaking 14 incredibly, vaguely your problem is much more 15 complicated then the one that was presented, 16 well, ultimately that's going to be the 17 problem, it's a lag problem. 18 MR. HOUGH: Right. 19 DR. FEDER: And in '06 it will be 20 -- the census data would be as old as 21 possible because in '07 you're going to have 22 another census -- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 84 1 MR. HOUGH: Correct. 2 DR. FEDER: So, but could you 3 analysis the '06 data once you get the '07 4 census in at least it will be more up-to-date 5 than the '02 census. Or reanalysis it and at 6 least assess the impact of the uncertainty by 7 the way, but I would at least try to analysis 8 the '06 MECS with the '07 census. We have 9 been -- our surveys of individuals are a 10 problem because we use census data which is 11 only 10 years in the -- analyzing unlike in 12 Canada and so we sometimes use a 10-year-old 13 census to calibrate our population estimates. 14 And as you know, there was quite a surprise 15 because the last census -- some of the 16 demographics, clearly we face the same 17 problem. But they do projections for the 18 inter census years, I don't know if anything 19 is prevalent here. 20 DR. HENGARTNER: The problem is of 21 course because the countries addressing, 22 right, so it's -- we make our justification BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 85 1 based on signs or something like that? What 2 are you using for the first justification? 3 MR. HOUGH: For the control tables? 4 DR. HENGARTNER: Yes. 5 MR. HOUGH: Well, we would use the 6 -- rederive the cost of energy from the 7 annual survey manufacture, the most 8 reasonably the survey manufactures. So we 9 will mail the 2006 MECS at the beginning of 10 '07 and as we process that, the annual 11 survey, I mean the 2006 annual survey 12 manufactures will be processed. So by the 13 time we get to where we are ready to publish 14 our estimates, we will have an '06 ASM data 15 base to reestablish the control totals. But 16 my question was that since we were able to do 17 it this time to the census which is the 18 universe which doesn't contain any sampling 19 error and doesn't you know, it contains all 20 those smaller establishments. 21 In the ASM, the smaller 22 establishments are sampled with weights, so BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 86 1 they're representative of other smaller 2 establishments, so the control total would 3 contain some level of error. Whether or not 4 we should consider that when we think about 5 doing this again, you know, we could analysis 6 the data certainly and look at. 7 DR. FEDER: Can I go back to the 8 original question of the weight adjustment. 9 Looking at the formula, I see one thing, I 10 think someone, Nick said that it's likely or 11 that as many small companies won't respond 12 and the larger ones will but let's assume for 13 the moment that the opposite happened. So 14 many companies that had to weight of close to 15 one do not respond, the other company that 16 had the weight close to one which are more 17 like them, their weight will me adjusted much 18 if you look at the formula, which ones will 19 get adjusted, the ones that had the weight 20 far away from one. So I'm worried about bias 21 here. 22 And that's why a simulation of BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 87 1 their first, can I propose, may not be the 2 best because may be the way the population is 3 it's not so bad and my concern is not so 4 real. If you take the other kind of 5 validation by throwing away 20 percent of 6 your sample, just saying 20, some part of the 7 sampling, see the impact, you will able to do 8 it and I really like what Rick suggested 9 doing based on the weight. 10 Make it's own respondents, two 11 exercise, one with those that have weights 12 are close to one and study the impact and one 13 which would tend to be large companies. Self 14 representing, close to being self represented 15 and then another exercise when you make the 16 non-respondent fictitious with churn the 17 respondents to ones that have weights far 18 away from one and see the impact. Because in 19 theory, this estimate in my opinion would be 20 general bias just because of the phenomena -- 21 knows nothing about because it's tends to not 22 alter the weight of units that have weights BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 88 1 close to one and those might be the ones that 2 do not respond. So we are not treating them 3 in the same way, that's my concern. It might 4 not be a problem with your population, that's 5 why I -- 6 MR. HOUGH: The reality of the 7 survey is that the coverage rate is about 89 8 percent, so we are getting those large 9 companies and the majority of the 10 non-response is absolutely, definitely in the 11 higher weight and smaller establishments -- 12 so I think if we do this simulation, we 13 assume that some of the certainty cases are 14 non response, you are going to see a lot 15 different look to these ways. 16 MR. COLE: We know who the non 17 response were. And we could tell them to 18 look at their size, and we could probably go 19 through and say, well, looking at the 20 respondents can we stimulate in our response 21 to be a mirror image of the non respondents. 22 If we say that half of the small BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 89 1 cases didn't respond and look at the 2 responding small cases and say well half of 3 them had not responded either, what would 4 have been the impact on the weights. We do 5 have -- most big cases do respond, most small 6 cases don't respond. Well most of the 7 respondents are small. But it think if we go 8 through just stimulate random non response, I 9 don't think it was random. I think there was 10 a pattern with small cases not responding. 11 DR. FEDER: So use what you know 12 about the non response -- 13 MR. COLE: Yeah. 14 DR. FEDER: Like the weight 15 distribution and actually that might -- 16 MR. COLE: Good point. 17 DR. FEDER: Support this approach 18 because then you are saying that companies 19 that have big size, their weight is close to 20 one, their weight shouldn't be changed, so 21 may be it's okay, I think we need to validate 22 this but if you don't put this into BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 90 1 consideration just on general principles, 2 this is a biased approach but it's not if 3 your big companies respond, big companies 4 that have the weight close to one. 5 MR. COLE: Right. 6 DR. FEDER: And again a bias is not 7 always so bad if we tell the miniscule error. 8 That's why we have look that miniscule error 9 are notified. 10 DR. HENGARTNER: Well, except that 11 in this case the bias can be a problem 12 because errors and variables, we have found a 13 big consistency. That means you will not 14 estimate what you want to even 15 asymptotically. And so unless you are 16 careful with what you are doing and that is I 17 think what Randy is saying is that you need 18 to careful how you do your asymptotically in 19 the first place. 20 So usually bias is another small 21 problem because it's vote of one over the end 22 and I went over the sample size where BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 91 1 variance would have been throughout of the 2 sample size though ---- but in cases where we 3 need to prove consistency I feel worried 4 about that a little more. 5 MR. SLANDA: I remember one NAICS 6 by region that we are looking at, there was 7 only one or two of them that was like that 8 and they had the certainties reported and 9 then there were some non certainties that 10 were shifted in from another NAICS code 11 region I think that's what I should say and 12 they were non reporters and it ended up 13 getting or actually maybe non reporters but 14 they were non selects because they were in 15 the population but not necessarily in the 16 sample and the measure of size for the cost 17 of energy was higher than that -- for one 18 company was hirer than for any of the 19 certainties. So that cost just in that one 20 cell it caused a little problem but the other 21 cells were good. 22 DR. HENGARTNER: The other cells BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 92 1 were good, okay. 2 MR. SLANDA: We had to look at that 3 for a while. It can happen. 4 DR. HENGARTNER: You need to be 5 careful here. 6 MS. WAUGH: One of the other 7 thoughts and it s related to the fourth 8 question about metrics is that EIA publishes 9 this data along with shadow tapes of the 10 relative standard errors and I think it would 11 be useful if the committee could address what 12 tables EIA must want to publish that would 13 communicate to the users about the errors 14 associated with the data. And the other 15 thing you should know is that when they 16 publish the data they may withhold it because 17 of non-disclosure or they may not publish the 18 data due to quality, so if they have a 19 relative standard error of fifty, they won't 20 publish the data. 21 DR. SITTER: Then there's no 22 problem. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 93 1 MS. WAUGH: Well, actually our data 2 users may consider that a problem. 3 DR. SITTER: That's too hard, I 4 think. 5 DR. FEDER: One of the things that 6 I know in my limited experience, this is just 7 a suspicion is that a lot of what we do in 8 variance estimation is still based on the 9 unrealistic assumption that the weights are 10 in fact the inverse conclusion probabilities, 11 in other words the Horwitz-Thompson weights 12 and we ignore post stratification and 13 response adjustment, to some extent, and 14 implication variance although some people on 15 this committee have done some work on that, 16 but we don't practice what we preach on that, 17 and there are a lot of things that we don't 18 do, and we need to and I think a reality 19 check is really a good idea. 20 And again, I want to iterate again 21 that part of it can be done by doing the 22 exercise I've proposed, but you're right. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 94 1 But in practice people used the weights and 2 say "if I have a weight of 2 that means the 3 rate of incursion was 0.5. And if the weight 4 is smaller than 1, if someone say there is a 5 problem, how do you defer that as one of our 6 probability, one of the probability has to be 7 taken a lot. So I can't even, I mean you 8 know, we fake it, sort of, I am as a I mean 9 it's quite incomplete, it's state of the art, 10 Randy am I correct in saying that. I believe 11 that -- 12 DR. SITTER: Oh yeah. 13 DR. FEDER: State of art is of 14 variance. 15 DR. SITTER: No, I mean that in 16 some cases, they'll -- you'll use replication 17 methods like a jackknife or -- 18 DR. FEDER: Yes. 19 DR. SITTER: A bootstrap or balance 20 bootstrap and recalculate all of your 21 adjustments as well and then hope that 22 captures that. But I don't think there is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 95 1 anything close to theory, if you were to do 2 that in any realistic situation -- 3 DR. FEDER: Uh-huh. 4 DR. SITTER: Like you're suggesting 5 or even worse, I mean I've seen cases where 6 they have got non-response adjustments, item 7 non-response imputation and then some sort of 8 calibration or - called G-weights, 9 Generalized regression weights and then, you 10 know, you just, okay, we'll just throw a 11 replication method at it and recalculate them 12 each time. But you know, they can't, it 13 can't possibly -- I mean somebody I don't 14 know. Well okay then if somebody now uses 15 the data and they decide to take some of your 16 vector of outputs and calculate I don't know 17 correlation between two of them, is this 18 going to be right? 19 DR. FEDER: And the issue of weight 20 trimming, which people use to reduce the 21 error and then sometimes the need hides under 22 the carpet, the bias that it introduces, so BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 96 1 one is to look at that and again simulation 2 or replication methods can help, but it's a 3 big -- Shawna, it's big problem. I think, 4 definitely it's something that we should pay 5 more attention to and not just on the MECS 6 Survey but the general. 7 MR. HOUGH: Well, where is the 8 chair. 9 CHAIRMAN BREIDT: Okay, I will take 10 over here. Anymore questions from the 11 audience? 12 MR. FRENCH: I was going to say one 13 thing for context and that is, when you were 14 mentioning the issue of thinking about post 15 stratifying through ASM, I presume it's still 16 50-60,000 establishments somewhere in that 17 line. 18 MR. SLANDA: Yes. 19 MS. WAUGH: Yes. 20 MR. FRENCH: Whereas the MECS is 21 15,000 and four percent of 100,000 and there 22 is something else I want to say about the ASM BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 97 1 too. 2 MR. SLANDA: It's optimized. 3 MR. FRENCH: Yeah that's true it is 4 optimized for economic rather than energy so, 5 I don't know whether that would have anything 6 to do with the thought of using a post 7 stratification where the numerator is now at 8 different type of estimates based on a 9 different type of sample other than a -- 10 DR. SITTER: But it's done 11 essentially independently, correct? 12 MR. COLE: Yes. 13 MR. FRENCH: Yes. 14 DR. SITTER: And you are really 15 falling into the sort of two phase sampling 16 idea, I mean if you would do the simplest 17 possible calibration on a previous survey 18 based on a bigger sample, therefore you think 19 it is more accurate than you're really in a 20 two-phase situation. So I'm sure that that's 21 essentially what you are ending up in and now 22 sort of ignoring the variability of that BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 98 1 other survey amounts to saying that it's 2 sample size is been in order, - is in order 3 larger. If that's not true, or if you don't, 4 if you feel the variability of that maybe an 5 issue then -- 6 MR. HOUGH: Well, I'm comfortable 7 with that definitely the size of the 8 establishment and energy use are correlated 9 and the ASM will pick up the largest 10 establishments because those are the ones 11 that have the most effect on economic data. 12 So we will get a large percent of our cost of 13 energy from the ASM. 14 DR. SITTER: Uh-huh. 15 MR. HOUGH: I believe that to be an 16 accurate statement. 17 DR. SITTER: But I'm talking about 18 the translation in the variation really, you 19 know which can have to do with the weights 20 and other things, so. 21 MR. HOUGH: Right. 22 DR. SITTER: But it -- the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 99 1 calculation of it's impact is easy, because 2 they're independent. Well, easy. If it were 3 a calibration it would be easy, you have a 4 lot of constraints and so forth, but in 5 principle, you just sort of have two pieces 6 of a non-linear function, do a Taylor Series 7 expansion, you can sort of roughly, get a 8 feel for what the variance that you're 9 missing, what variance you are dropping out, 10 and you may be able to answer the question 11 what I'll be doing if that variance is 12 relative. The part I'm estimating or can I 13 estimate it, harder estimates there, things 14 like that. 15 DR. FEDER: Go ahead. 16 MS. WAUGH: Could you post stratify 17 on the certainty cases and not on the 18 non-certainty cases, since that might be 19 common to both surveys? 20 MR. HOUGH: That would be different 21 from what we did here. Actually, basically 22 what this post stratification does is it BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 100 1 measures how effective, what did you have to 2 surpass that mission Kaleen. 3 MR. COLE: How well the ---- 4 population, so referred to, they're already 5 self representing. I'm not sure you would 6 have anything you can do productive. 7 MR. HOUGH: Unless their energy -- 8 MR. COLE: This sounds like -- 9 MR. HOUGH: Unless there is such 10 as, like for example, we pull a sample, if 11 the 2005 ASM data is available to us, we take 12 the nearest piece of data to derive and 13 measure size. Now if the Survey year 2006 14 shows some spike or decrease in one of the 15 strata's of these certainty cases, then you'd 16 - - that would definitely have an effect, 17 because the initial measure or size that we 18 sample on and wait on would be from the 2005 19 ASM. 20 And if some condition and whatever 21 is going, the largest 1000 have to drop out 22 or shut down, that would affect the strata to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 101 1 the point where the control total from '06 is 2 drastically different, then that may have an 3 effect. But if the strata remains relatively 4 level from '05 to '06, the certainty cases 5 are driving those estimates. I don't know if 6 the weight effect would be, can you think of 7 anything? 8 MR. COLE: I think that the process 9 should, not only adjusting the weights of the 10 non-certainties there is no impact for 11 certainties, they are basically wiped out, in 12 a way that it's not there. 13 MR. HOUGH: Okay, it's different 14 from one doesn't mean we have a bad estimate 15 itself, it means we have bad samples on 16 certainty but the total being driven by 17 certainty, in that case, you still have a -- 18 SPEAKER: Such as what? 19 MR. HOUGH: You still have a very 20 good estimate, even though we had bad samples 21 on certainties. 22 DR. FEDER: Rick, you asked this BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 102 1 question about what to do just when the 2 survey data are old and you know, like an old 3 thing, - I want to mention some things that 4 my colleague Jim Kromy, once -- he had some 5 thoughts, he said to me, you know the census 6 data, I'm talking about the Census of 7 population is conducted every ten years and 8 what do you do, that's nine years after the 9 Census, we use the Census Projection and we 10 use them to calibrate our large scale survey 11 which has about seventy thousand respondents, 12 I'm talking about the household survey of 13 drug use, which is a distant theory has 14 unbiased estimates. 15 So why do you calibrate an unbiased 16 large scale sample which is - supposedly has 17 small min - small variants, no bias and you 18 calibrate it to census that, I know your 19 colleagues are doing a terrific job in doing 20 the interferential projections, but they are 21 old, nine years old and what we do and that's 22 what we do in practice, we calibrate it and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 103 1 we treat the Census numbers as being 2 non-random. No error on variables made, and 3 is that correct or not and there's a lots of 4 talks about it and one thing that I said why 5 don't we do the next year when we have the 6 census data go back and check our 7 assumptions. And I would suggest the same 8 thing to you. We do a lot of think-- 9 MR. HOUGH: I think that's the best 10 we've got. 11 DR. FEDER: Because we are dealing 12 with reality and uncertainty in a complex 13 statistical estimation that sometimes 14 requires maybe some refinement. But those 15 questions are not only with your situation; 16 we face them in our cases too. 17 DR. HENGARTNER: Isn't there going 18 to be a problem with your suggestion that by 19 going to next year's inventory or census, 20 some companies may not exist anymore at that 21 time? 22 DR. FEDER: But if you go back four BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 104 1 years there is a greater problem, you have 2 one either way. 3 DR. HENGARTNER: All right. 4 DR. FEDER: Yes. 5 MR. HOUGH: It's always an issue 6 with the survey, the timing. You're always 7 drawing the sample problem, the data census 8 that will be updated as we process our 9 survey. 10 DR. SITTER: Do you have a public 11 perception issue, is that one of the reasons 12 that you're calibrating or is it not a matter 13 to you, because I think that in some of the 14 cases that Moshe is talking about it's 15 probably a public perception issue. That is 16 there are some key variables and if you put 17 out estimates that's different from the 18 previous census, that is you put it out and 19 they take your weights and calculate it and 20 you are way off on some key demographic 21 characteristics or something-- 22 DR. FEDER: Well they -- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 105 1 DR. SITTER: That brings in things 2 into question. 3 DR. FEDER: Our published estimates 4 are on drug use, mental health and things 5 like that, so there's no issue there, but 6 DR. SITTER: Well, there is -- 7 DR. FEDER: But we have tried to as 8 a quality measure to see, what if you go back 9 and estimate. In fact well, I cannot talk 10 about specifics, but I've done some 11 experiments and I didn't like the results and 12 I talked about it, because, it has to do with 13 I think with the state of the art of 14 calibration which is still evolving, because, 15 I remember actually again if I might digress 16 a little bit, when I came to STATS Canada I 17 was new to statistics and taking courses from 18 Randy, I -- people were still doing 19 regression type of calibration where negative 20 weights were generated. 21 We're talking about something which 22 is quite recent, we're talking about BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 106 1 mid-nineties and people said "Hey how can I", 2 I mean you're worried about the weights 3 smaller than one, people said "How can I have 4 a negative weight?" and I remember a 5 situation, we did some projection for certain 6 region of Canada, where Dwight had turned up 7 and we said, "you know you don't have to go 8 down because of calibration", he said "no, no 9 that's wrong", so people said "you know, you 10 can do smaller estimation if you have 11 negative weights because some small areas get 12 negative estimates of total of a positive 13 quantity", so then all my colleagues who were 14 much more advanced in this topic than I was 15 came up with wrecking ratio adjustments or 16 what we call in Britain, SPREE and some other 17 methods of doing calibration that guarantees 18 positive weights. 19 Then people were happy for a while, 20 then they said, no, weight has to be greater 21 than one. Okay so I think and then you go to 22 cross extreme weights and they had leverage BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 107 1 on the various estimates, we are still moving 2 I think as a -- I call myself a young 3 statistician because I haven't been doing 4 statistics for that long. I see a lot of 5 progress in that but it's still quite in 6 evolution. 7 DR. SITTER: And more recently 8 there's a empirical likelihood methods to 9 guarantee positive weights, there is some 10 rich aggression methods, there is quite a lot 11 of work on this where you want these weight 12 adjustments to have constraints. I think 13 there is some literature there but --, and I 14 think that a lot of that literature is driven 15 by real problems but I don't think that it 16 addresses the real problems. 17 I think they are still at this 18 stage where they're talking about theoretical 19 exercises. So at least your problem is hard. 20 Well that's good news for some things, you 21 come in and you say gees did we do it right 22 and then people say wow it's hard. That's BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 108 1 good there. 2 DR. HENGARTNER: The last question 3 I have is, why can't we sync those two 4 studies? I mean they were in sync, now that 5 sync works, why don't we just keep it. 6 MS. DODDS: They just barely work. 7 MR. FRENCH: And keep the census 8 and MECS relative during the same year. 9 MR. HOUGH: We have to go back to 10 the back there. 11 MR. FRENCH: I think we've already 12 expanded the interval between the MECS year 13 to the point where we are worried about 14 relevance, so it's every 4 years now, and for 15 it to be in sync with the census, it would be 16 every 5 years -- 17 MR. HSEN: And we don't have an 18 annual manufacturing survey in the interim. 19 MR. FRENCH: Correct it's a problem 20 that, right now I don't think we know what 21 the -- 22 MS. DODDS: Yeah in fact I thought BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 109 1 your association with MECS -- 2 MR. FRENCH: Yeah and expand the 3 sample size -- 4 MR. HOUGH: I mean, this is going 5 to exist, this situation is going to exist, 6 like I said every time we do this survey, 7 this is always going to be the post 8 stratifying to a new control total, it's 9 always going to be an option, the way the 10 surveys are done and the way our surveys are 11 done. Next time we come across, we came up 12 with this method for when we had the census 13 this time, that's not going to happen for 20 14 more years. 15 DR. FEDER: But I think we are 16 getting here into the design issues, may be 17 we should put it on the committee's agenda 18 for next time because you could think about 19 many ways maybe like a rolling survey where 20 every one of those 4 years or 3 years you do 21 part of the sample so you have a way to 22 measure change, maybe a panel, a survey BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 110 1 whenever a company is observed, maybe at 2 least a couple of times we can measure change 3 more effectively, I mean there is a lot of 4 things so, maybe what we should do is put it 5 on next meeting in April, I think it'll be 6 and discuss not just the estimation issue but 7 also the design issue. 8 DR. SITTER: How stable are things 9 over the years? Have you looked at patterns 10 in terms of the relationships from census to 11 your survey? I mean in some sense having it 12 badly coordinated with the census can give 13 you sort or more data in a sense that you 14 know, you -- if things are reasonably stable 15 then you have some relationship between how 16 things deteriorate from the census over time. 17 I mean we are talking about long time windows 18 here and I don't know your industry that well 19 but did you have some data? 20 MR. HOUGH: Well the energy data is 21 going to be correlated with the production so 22 the industry has increase in revenue; we BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 111 1 usually see the energy usage go in the same 2 direction. Over the past 4 years it's been 3 going, it's been on a decline and that's the 4 result that we've had this time. 5 MR. COLE: Is that quantity-based 6 or is that value based? 7 MR. HOUGH: It has to be value 8 based, that's what it is based on -- that's 9 what the measure and size is based on values, 10 that's all -- that's all we've got but then 11 MECS collects quantity detail as well as, 12 dollar expenditure so - - 13 DR. SITTER: Yeah and your economic 14 cycles are quite often quite a bit quicker 15 than your census. 16 MR. FRENCH: And that doesn't get 17 into the issue of the fact that MECS collects 18 things like non-purchased fuels and other 19 things that's used in very different places, 20 the costs numbers which are based on 21 purchases -- 22 MR. HOUGH: We collect quite a more BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 112 1 -- we collect and publish quite a bit more 2 detail then we sample. 3 MR. FRENCH: I just want to throw 4 another monkey away. 5 MR. HOUGH: We don't want to talk 6 about what Dwight just said. 7 SPEAKER: So are we done? 8 SPEAKER: Yes, we are. 9 SPEAKER: All right. 10 CHAIRMAN BREIDT: So our first 11 summarizer will be Moshe Feder who will talk 12 about post stratification methodology -- hey. 13 DR. FEDER: With a very interesting 14 description of a problem with the MECS 15 surveys, say I've to look there, it's the 16 Manufacturers Energy consumption Survey, 17 which uses the economic census conducted 18 every five years on business they did it 19 October 7, to construct a calibration 20 weights. And in 2002, the two surveys 21 occurred at the same time because the MECS is 22 conducted now every four years, it used to be BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 113 1 every three years since 1985, and 2002 was 2 unique in the sense that every 20 years of 3 the current design was to survey, as I said, 4 the survey and the census coincides. 5 So there was an opportunity also to 6 use the economic census in calibrating the 7 weight. Now, the, what we heard is about an 8 approach to, first of all, I should go 9 through the steps of the weight adjustments 10 and there are three of them. First they do 11 the non-response adjustments, which is a 12 simpler simple ratio adjustment that to 13 calibrate the weight to the census -- to the 14 weight of census total energy consumption. 15 The next step is a post 16 stratification adjustment and that's followed 17 by a coverage adjustment, because there's a 18 typical, I guess with this kind of 19 establishment as per new companies that are 20 created, that are not covered by the survey. 21 There's also some companies that transferring 22 the manufacturing, which in the census says BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 114 1 that is also a birth. There's also death, 2 but the deaths are caught by the center 3 because when you, a survey or -- a company 4 turns out to no longer be in business or not 5 doing manufacturing that you can account for 6 that. So all those things lead to a coverage 7 adjustment. 8 And the goal of the study was to 9 come up with a certain weight adjustment, 10 that will have two features. One, it will 11 calibrate to the census total weight of 12 consumption figures and also they wanted to 13 guarantee that units that were self 14 representing in the sense that their analytic 15 weight was equal to one, their weight 16 wouldn't change. So what they proposed is a 17 new weighted weight, which is actually, it's 18 in your handout, it's equal to 1 plus K times 19 the old weight minus 1. 20 And if you look at the formula as 21 to weight is equal to one, that the 22 adjustment is one and the weight remains the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 115 1 same. And there was some concern about the 2 statistical property of those adjustments and 3 what impact it will have on the standard 4 errors. Somebody mentioned that EIA 5 publishes this every table, a scheduled table 6 of standard errors and that also used in 7 determine -- determination whether figures 8 should be released or not based on quality 9 and other consideration. So the committee 10 was asked a few questions related to that. 11 And also another important question 12 that was raised by the presenters was that 13 2002, what happened here like this perfect 14 storm of the two, two things happening at the 15 same time, the survey and the census will not 16 occur again for another 30 years, so what do 17 you do, with the censile information is old 18 like the next time MECS is going to be 19 conducted is '06. But the last census then 20 would still be '02, so what do you do about 21 that. 22 And there were some other related BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 116 1 questions and a lively discussion about what 2 would happen here. And if I may quote, one 3 of the committee members said, the problem is 4 actually hard and some of it is not clear and 5 we were also asked, have you seen this method 6 before, and the answer was not exactly like 7 that. It was mentioned that the calibration, 8 the concern about calibrated weights being 9 greater than one is actually quite common. 10 It's done in most long surveys and are 11 procedures to do it. 12 So a committee member suggested to 13 look at that and maybe use some other 14 calibration method. But the question still 15 remain, how do you evaluate the current 16 method. There was some question whether 17 there is or isn't a bias, and a simulation 18 might help here, another suggestion that was 19 made was to perhaps take the 100% of the 20 sample, set aside, say 20% of it. Use the 21 80, treat them as if they were 22 non-respondents, apply the method to the 80% BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 117 1 and validate your prediction for those 2 missing units using the new weights. I wont 3 go in to the mathematics but there were some 4 issues that if units that have the weight 5 close to one are different from the ones 6 whose weights are far greater than one, then 7 a bias would occur, so that's one way to 8 check that. 9 Let's see if, yes, there was some 10 issue here about the -- about the weights, 11 the randomness of the weights should be 12 treated as error and variables issue which 13 can lead to bias even asymptotically, as was 14 pointed out and that's a hard problem and 15 this was mentioned that in all practical 16 applications, this is the problem and there's 17 no clear solution to that. 18 And there was some other discussion 19 about using the ASM, which is independent of 20 the survey in doing the weight adjustments 21 and the calculation, it shouldn't be very 22 hard because of the independence. So that's BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 118 1 another suggestion that was made by someone 2 in the. 3 Audience. Finally, the discussion 4 in particular was the issue of the census not 5 being at the same time as the survey. It was 6 mentioned that they had the design issue and 7 maybe the survey could redesigned, maybe 8 having a part of the sample data in each 9 year, rather than having it once every four 10 years or so, what I am -- we were wondering, 11 perhaps it could be put to the agenda for the 12 next meeting in April or in the Spring. 13 CHAIRMAN BREIDT: Any questions, 14 comments or corrections? Okay, thanks Moshe. 15 So next Nagaraj will summarize Thursday's 16 break out session. 17 DR. NEERCHAL: We were looking at 18 the EIA 906 and the one that Joe was talking 19 about earlier and the data is being entered 20 through an internet data collection, that's 21 IDC, IDC systems. Another -- basically 22 monthly data is being entered and the people BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 119 1 are asked to enter their information on fuel 2 consumption, generation and stocks of fuels 3 on every month. And within -- I think within 4 45 days of the end of the month they're 5 supposed to enter the data and the -- while 6 they're entering the data you want to give 7 them comment, so to help them -- the idea is 8 to help them enter the correct data. 9 So that we can kind of cut down 10 your edit work later on and so that you get 11 the data in, person who is supplying the data 12 give you as correct a data as possible, that 13 is the overall objective. This gives rise to 14 a very complex database, because every plant 15 has several types of prime movers and each 16 prime mover has several types of fuels that 17 it can consume and so on. So we have to -- 18 they have to fill in a form like this with a 19 lot of different units going down there and 20 different fuels for each unit and so on. So 21 it gives rise to a very complex database. 22 The current system I think, the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 120 1 current edit that is being applied right now 2 is essentially data plant taking up data 3 plant and compute the mean and standard 4 deviation for that over a 24- month period. 5 And compute an interval like mean plus or 6 minus 1.5 standard deviation, and if the 7 entered data is outside that, then flag it 8 and invite them to correct the data points, 9 and give them mild warning saying that the 10 data you have entered is not correct or not 11 in the range perhaps and so on. 12 So one of the problems of this 13 system is that you know, if you have 14 autocorrelation in your data for example, you 15 might miss it, or seasonality in the data you 16 might miss it. And of course you know, there 17 are some modification and this is what I told 18 you is the simplest form of the edit. There 19 are some, you know further checks on the -- 20 this system, this method to incorporate 21 seasonality and so on but still it seems to 22 me, you know, the autocorrelation part might BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 121 1 be missed completely. 2 So that's why the proposal on the 3 -- in the paper right now is to look at 4 something like, simple exponential smoothing, 5 that we capture the autocorrelation. And you 6 know, help you pick up some outliers that you 7 may not normally consider outliers. So 8 number of questions to the committee on this 9 one is to -- I think the primary question is 10 how to choose the best alpha. That is one of 11 them and alpha will be a smoothing parameter, 12 or another -- another way of putting up the 13 same question is that, are you happy with the 14 alpha you chose? 15 And also another aspect that I know 16 whether you have a handout here, I think if 17 you look at the picture, if you compute the 18 smoothing parameter for every plant and make 19 a histogram, that sort of looks like a 20 bimodal thing like this for some of the 21 parameters. And why do we get lots of zero 22 alphas and lots of 1 alphas and what does it BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 122 1 mean and so on. 2 In this case it so happens that if 3 you have alpha that is very close to zero, 4 then you're sort of back to the mean edit 5 situation. And if alpha very close to 1, 6 that means that you don't trust the mean very 7 much because you want to startup 8 autocorrelation in the data. So some of the 9 -- some of the decision, I mean there was a 10 lot of discussion, I think, in this case I 11 think committee really acted like a 12 committee. I think we really did not have a 13 solution for any of the bullets. So we have 14 a lot of suggestions today, and that's more 15 like the all the, you know, all the non and 16 in the PHD committee, all the non-chairs do 17 that, right? Chair is the only one who 18 worries about getting it done. So -- that is 19 what we were acting like the other committee 20 members. 21 And so, the issue I think is that 22 -- you know think overall we felt very BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 123 1 comfortable with the proposal. Because 2 especially it picks up alpha and you know 3 that it is really kind of an improvement on 4 what you're doing right now. You're not 5 really completely throwing away what is being 6 done and starting something new. It is an 7 improvement on what is being done right now, 8 so it seems like it is something definitely 9 that remains to be explored. But one of the 10 critical issue here is, we don't know who is 11 right. 12 You know, suppose you apply this 13 old edit as well as the new edit and the old 14 edit kicks this out and the new edit kicks, I 15 mean does not kick it out, we don't know who 16 is right, who is wrong. That is one of the 17 critical issues. It may be worthwhile coming 18 up with a data set that you may want to say, 19 okay, we all agree that these are the correct 20 values, now let us evaluate the different 21 edit systems. It may be a worthwhile 22 exercise to spend sometime coming up with a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 124 1 "test database" where you feel comfortable in 2 accepting some values as the correct values. 3 Now, that is one of the things, and 4 as far as estimating alpha right now as one 5 of the heat I would say is that the smoothing 6 parameters are estimated using SAS. You 7 know, I'm sure there are many softwares that 8 can help you in that one but I think SAS is 9 being used, but then again you know, it seems 10 most of the simple things like kicking it out 11 and kicking it in and things like that are 12 probably done on a spreadsheet and you have 13 to combine the SAS output with the 14 spreadsheet, send it to somebody else and so 15 on. 16 So one of the suggestions might be 17 to use a simple estimate that is -- can be 18 done in like a one line for moving average 19 smoothing parameter, we can write down a 20 formula for you know, getting a decent 21 estimate for the alpha, instead of using a 22 sophisticated software. So that is something BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 125 1 that, if you do that, then your ability to 2 manipulate lot of these things will probably 3 go up. The other suggestion was to consider 4 a grid of alpha. 5 Not just you know, if the estimated 6 alpha is something, then simple estimate is 7 some 0.3, then go down a few, go up a few or, 8 you know, and see well how this, how many, 9 what percentage of the flagging and so on. 10 Yet another suggestion we got was, perhaps if 11 you look at some of the plots there are too 12 many large numbers, lots of zeros and large, 13 you know, it's kind of a oscillating thing. 14 So one might want to consider some 15 transformation of the data, so that something 16 that is not being buried somewhere, some are 17 data points. Then that is some of the 18 information we got. 19 But we ended up discussing a lot 20 about, you know, how can we incorporate some 21 of the other information that is available, 22 like, the type of fuel for example. Use of BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 126 1 major fuel used by the plant or -- 2 MS. KHANNA: Weather. 3 DR. NEERCHAL: Weather perhaps, and 4 other information that you know, some 5 auxiliary information that can be used for 6 edits and there were a lot of suggestions and 7 I think fortunately we were -- we had both 8 the people there, they seemed to be nodding 9 and saying that maybe we can try that you 10 know, we have no idea what we were saying 11 right or wrong, but I think, when you can see 12 them nodding, I said, okay, not too bad. 13 Because we didn't have the data in front of 14 us so, some of these things may be just a 15 pipe dream. 16 For example, one of the slightly 17 more complicated suggestion that came out 18 was, for certain types of plants there is a 19 special pattern that kind of goes on line and 20 come -- goes to zero for a while, comes back 21 on line for little bit and goes back to zero. 22 Now we know, and the database also tells you BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 127 1 whether that is, you know, the plant you're 2 trying to edit is of one of that type. So 3 you may want to build in a memory sort of, we 4 know, are you, speaking plan for example. 5 You know we may do -- build in another one, 6 so you just entered zero, is it, you know, 7 supposed to be zero that, so some special 8 edits like that, some addition, additional 9 suggestions on that one. 10 So overall I think committee seems 11 to be very excited with the project. I mean 12 it was lot of excitement and the, and I 13 think, specific question about model 14 selection, I think we were basically 15 skirting. That may not be that important an 16 issue to look at in this case. 17 I think this specific bullet apart, 18 whether should we consider mean absolute 19 prediction error or mean square error? I 20 mean that may not be that critical in this 21 case, I think there are lot of other issues 22 that will be, you know, better to focus your BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 128 1 energy on, that's the overall feedback based 2 on it. 3 CHAIRMAN BREIDT: Okay. Thanks 4 Nagaraj. Are there comments or questions or 5 corrections? 6 DR. FEDER: I wasn't at the meeting 7 but I looked at the handout and I want to 8 make a few comments. Just that they're all 9 focused on the time series adjustments. One 10 thing is said, if you look at the model, it 11 assumes stationarity of this, no allowance 12 for snow and there's a method by 13 Holton-Winters which, I mean it convinces(?) 14 it's in Howry's book on structural time 15 series models, which is also very simple mind 16 it, more than, very simple. It just allows 17 for a slope and there's another version, it 18 also allows for seasonality which might be 19 appropriate here. 20 I would like to not have too many 21 alphas but I would, because of their pattern 22 that Nagaraj was talking about with some BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 129 1 plants phasing in and phasing out to do 2 production, maybe stratified by something and 3 then just treat the ones that exhibit more 4 regular pattern, perhaps with or without 5 seasonality, I don't know enough about those 6 plans to say one way or the other. 7 And finally the selection of the 8 alpha are we seeing as (?)IBEC in Canada had 9 a forecasting issue when we also dealt with a 10 very short time series. Actually we ended up 11 using something like a variant of the common 12 filter but the way to do it, also with 13 Holt-Winters by using a penalty function. By 14 that I mean to following. You start with the 15 series what you know and every -- for every 16 time point you look at the Holt-Winters on, 17 the exponential smoothing, best estimates for 18 that time point, look at the difference, 19 that's called the innovation between what you 20 observed and what you predicted, you square 21 it, you edit up, that's your penalty. 22 Of course you look to minimize BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 130 1 that, you want your prediction to be what you 2 observe. And the alpha that is best -- that 3 minimizes that is your choice of alpha. It's 4 sort of an objective way of coming up with 5 alpha which is very easy to program and we 6 had the satisfactory results with that. 7 But I think having too many alphas 8 defeats the whole purpose, because it will 9 tend to favor I think the last observation 10 rather than a smooth estimate. But certainly 11 I think because of the nature of those time 12 series, I would allow for a slope and for 13 seasonality. 14 CHAIRMAN BREIDT: The exponential, 15 some of it is allowed for a local linear 16 change, so it's -- 17 DR. FEDER: Yeah that's -- that's 18 just included in the Holt-Winters. I mean -- 19 Holt- Winters is exponential -- or maybe I'm 20 wrong about the terminology. 21 CHAIRMAN BREIDT: Yeah, not on 22 perimetric change, but local linear changes, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 131 1 that's what the exponential smoothers on. 2 DR. FEDER: In the model they had 3 there was no allowance for that one -- 4 CHAIRMAN BREIDT: What I mean is a 5 stochastic change, a local linear change. 6 Not a perimetric change. 7 MS. KIRKENDALL: It is adaptive, 8 it's just -- if there is a real trend in the 9 day, then it's not good enough. But if it's 10 just some changing over time -- then it -- 11 DR. FEDER: Oh yeah. 12 MS. KIRKENDALL: It is a change. 13 We haven't seen many trends in this area. We 14 see some seasonality in the data. 15 DR. FEDER: Okay. 16 DR. HENGARTNER: Explicitly more 17 than the seasonality of them, smoothing the 18 residuals using the exponential weighting 19 might actually be a good idea. Because to 20 control better the variance, that way. 21 MS. KIRKENDALL: I think we -- 22 DR. NEERCHAL: One of the BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 132 1 suggestions the committee had was to look at 2 some seasonal versions of the moving average 3 and we're not -- 4 MS. KIRKENDALL: Yes, here I think 5 we had a brief discussion about this and one 6 of the things to remember is that, these 7 edits on at a facility level, so you'd have 8 to the seasonality on whatever structure that 9 you're talking about, each facility 10 separately. And so this is not you know, 11 like you're doing cross section across 12 facilities. So that just increases, you 13 know, and that's an enormous preposition. 14 DR. FEDER: We'd have too many 15 problems because we did with, but why can't 16 you assume the same seasonality within the 17 structure? 18 DR. HENGARTNER: Well, it is like 19 what Joe was talking about earlier today, 20 cluster residual companies. 21 CHAIRMAN BREIDT: These are issues 22 that were discussed. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 133 1 DR. HENGARTNER: Okay, I'm sorry. 2 We had a good discussion here, too. 3 CHAIRMAN BREIDT: Okay, any other 4 further comments or questions. Since we've 5 already had a warm up break, I suggest that 6 we don't take a break now and just move right 7 into the next section, that's all right, 8 which would be the discussion on -- 9 SPEAKER: Yeah, I understand, yeah 10 I understand, that's right. 11 SPEAKER: This is the last chance 12 to -- 13 CHAIRMAN BREIDT: So this is the 14 session, if you were king brought by Howard 15 Gruenspecth. 16 MR. GRUENSPECHT: That's one way to 17 do it. Anyway I was in West Virginia 18 yesterday, that's why I was not here. West 19 Virginia is a fairly affluent state, which -- 20 SPEAKER: Yeah, you would never get 21 involved in it. 22 MR. GRUENSPECHT: It is important BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 134 1 and they were having a conference and they 2 asked for representation. I did talk with 3 him last night, he indicated that this 4 meeting is so great. So future meetings, 5 hope we won't be able to talk about, but it's 6 nice to be here with you. But you know, as 7 you know we're periodically reassessing how 8 we can continue to fulfill our mission? 9 And again it's a tough place to be 10 King or Crown Prince or whatever role we play 11 here because, you know, we do have this 12 demographic issue that in fact you know where 13 lots of technical skills combine the needs in 14 the areas of expertise, it is a growing you 15 know, interdependence and nationalization of 16 the world economy and the energy 17 environments. I think that -- try and look 18 at this years blow out in the oil markets 19 where I think really demand, well, 20 particularly Asia that was not anticipated I 21 think by anybody and the data available was 22 quite poor. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 135 1 There's a lot of concern about the 2 quality of international data in this area. 3 There's a joint oil data initiative underway 4 to try to improve it. So there was lot of 5 pressure to improve the quality of data. 6 There is unrelenting demand for product 7 promos, service reports, data and model 8 improvements. I guess, yesterday you heard a 9 lot about the STEO, the regionalization of 10 the Short Term Energy Outlook, there's a lot 11 of interest in that. There's a lot of 12 interest in natural gas data, we are planning 13 a new natural gas production survey. And 14 then so what, rising demand and then a pretty 15 tough budget situation, which I'll elaborate 16 on a little bit. EIA has been doing a lot of 17 -- for in some sense we've been doing it on 18 an unsustainable basis, in that we've been 19 living beyond, we've been spending more than 20 our current budget in each of the last couple 21 of fiscal years. 22 That's been done by reducing what BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 136 1 is called priority year on cost and balances. 2 So money from earlier years that was carried 3 over, that we were spending, but if you look 4 at how we've been delivering our program, 5 we've actually been spending a fair amount 6 more than we're budgeted on, on a yearly 7 basis. That how we can continue, roughly 60% 8 of our budget supports staffing and staff 9 related costs and increases in those areas 10 are driven by Congressional action on pay. 11 Certainly we have a well-qualified 12 staff and we want them to be well paid. But 13 in the environment where last couple of years 14 that has been, maybe what, 4% each year, in 15 that area and I don't know where they will 16 come out this year, it's one of the last 17 things they'll do. It will probably 3% I'm 18 hearing, 3 or 3.5%. But this cost of 19 pressure in that area and it's also a very 20 tough budget environment. 21 EIA can try to swim upstream in 22 that tough budget environment, because we're BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 137 1 a very small fish, so that's a good news I 2 guess. If we were a big piece of the 3 domestic discretionary budget, it would be 4 very hard to be going in a different 5 direction then that whole overall budget and 6 that is a strategy that the administrator and 7 I are trying to pursue and I think that, at 8 least in the administrations budget proposal 9 for this fiscal year that started this month, 10 without a budget yet. 11 The administration only has 12 proposed a pretty substantial increase for 13 us, for about $81.1 million to $85 million, 14 which for us is a good increase. Congress 15 has not acted on it, but there's no guarantee 16 that that success, the administration would 17 propose significant budget increases for EIA 18 in 3-4 years. That Congress would provide 19 the money, we're in a tough league here, 20 we're in the interior budget. 21 So you know we're competing with 22 smoking the bear and keeping national parks BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 138 1 open and the forest service, fighting fires 2 in the West, and all kinds of, you know, 3 other programs that are important too. So we 4 think EIA's important, obviously other 5 programs have other constituencies, I think 6 they're important. So we always need to 7 think about how to deal with, you know, best 8 employing the resources that we have, how to 9 make trade offs between products if we have 10 to do that. 11 How to balance, you know, meeting 12 immediate needs and requests of today with 13 still being our capability to meet tomorrows, 14 our questions, how to deal with the 15 turnovers, some of our personnel, they're 16 interested in, there are just a lot of 17 questions. How to deal with domestic versus 18 international data? How to deal with the 19 split between surveys and analyses. How to 20 prioritize from other surveys? If indeed we 21 run into trouble, how to choose between 22 changes and periodicity and sampling BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 139 1 strategies that might save some resources 2 versus perhaps terminating some surveys. And 3 we have in the 2005 budget, even with the 4 increase from $81.1 to $85 million, proposed 5 to terminate one survey in the electricity 6 area. 7 Again there's always an issue about 8 how to deal with these questions. And we're 9 dealing with them on an everyday basis, 10 sometimes we get tunnel vision, there's a 11 real advantage in bringing folks in from the 12 outside who might have some ideas in these 13 areas. So I just like to rally throw it 14 open, there are lot of challenges before us 15 in running what I think is, a pretty good 16 program at EIA or a very good program at EIA. 17 But we do have challenges and if you were the 18 king or queen or crown prince, or whatever it 19 is, what would you be thinking about, are 20 some of these questions? 21 And we can break it up into the 22 technical expertise area, so how to trade off BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 140 1 among products, adds opportunities for cost 2 savings, which are always welcome. I think 3 our staff and you always like to talk, yeah, 4 I think it is about cognizance and a sense of 5 frontier that you try to get to, we're 6 trading off cost on the one hand and then you 7 have quality and timeliness, which are the, 8 you know, very important dimensions for EIA 9 and I think we in EIA naturally always tend 10 to think about ways to improve quality, 11 improve timeliness, well, the discussion 12 around, this program is about that. 13 We also need to think potentially 14 about ways to save money, holding quality and 15 timeliness constant. It's not as much fun 16 for out staff or maybe for you to think in 17 those terms, but again if you have ideas in 18 those area, it will be very helpful. 19 MR. BLAIR: You just touched on 20 staff and I think that obviously the issues 21 of quality and timeliness are important, but 22 I think staff frustration, staff burden, how BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 141 1 staff reacts to trying to do the same thing 2 or trying to do more unless there is also 3 something that you want to factor into the 4 equation. I know, few meetings ago, there 5 was a big discussion on the issues of 6 recruiting and maintenance and losing senior 7 managers and so forth. So I've -- I just 8 recommend that it should not get lost in the 9 equation. 10 MR. GRUENSPECHT: Yeah, it is 11 interesting. We have done some, that's 12 right, we have been very interested and over 13 the past year have introduced some new ways 14 of recruiting staff. We used it for our 15 entry level, each office as they needed 16 people would put out an individual 17 announcement, which then had to be cleared 18 through the departments personnel system. 19 And there was a very high cost for each 20 announcement. 21 Even though we realized pretty 22 early on that, at the entry levels which is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 142 1 out of, either out of under graduate or just 2 out of gradate school levels, we really 3 weren't looking for people who had, it didn't 4 matter so much if you're going it pretty well 5 in gas activity, you just have to have a lot 6 of oil and gas expertise. As along as you 7 went through with the activity, you didn't 8 really need to have a lot of, look through 9 the expertise in those levels. 10 We're really looking for, you know, 11 we'll teach you the field expertise, we 12 really want the best math staffs, the best 13 survey staffs, the best operations research 14 analysts, the best economists. So we went to 15 a EIA wide recruiting at those levels, 16 although again at the higher levels, we're 17 still recruiting office by office with 18 individual announcement. 19 I think we had a fair degree of 20 success with some of that recruiting, we 21 managed to bring some good people and I think 22 we're going to try that approach again, we've BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 143 1 also increased our outreach, because these, 2 our recruiting announcements, we're not 3 sending what information the various 4 contacts, the colleges and universities 5 indicating that we put these announcements 6 out. Again trying to, you know, build 7 interest. And we've gotten a lot of, I think 8 a lot of pretty good people. So we're able 9 to recruit, which is good. 10 We don't have some of the 11 incentives I guess, that some of the other 12 statistical agencies offer and some of the 13 others may have direct hire authority, Nancy 14 were you in with a session with some of the 15 public agencies. 16 MS. KIRKENDALL: It doesn't look 17 like, I think they're trying to, there's an 18 interagency effort to try to get direct hire 19 authority, I'm not sure right now. 20 MR. GRUENSPECHT: Right. 21 MS. KIRKENDALL: I think that the 22 agencies have been grouped, I think we'll do BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 144 1 better if we all go together and say, we need 2 direct hire authority. 3 MR. GRUENSPECHT: Then they will 4 want to bypass some of the -- 5 MR. BLAIR: Sure. No, that also, 6 instead of having sort of a ---- that one of 7 the things you want to do when you get 8 various budget problems, whatever is 9 maintained, your core competencies and so 10 that you get them. Whatever happens in the 11 future you've got to maintain that core and 12 personality is a big part of that. 13 MR. GRUENSPECHT: Do you have any 14 thoughts on the -- you know the use of 15 contractors versus in-house staff? 16 MR. BLAIR: Well, being from a 17 consulting organization -- 18 MR. GRUENSPECHT: Well, we do a lot 19 of both by the way. We're saving about 60% 20 of our costs here for the federal staff and 21 the costs directly related to that, and 22 probably you know, most of the rest is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 145 1 contracted out. 2 MR. BLAIR: Yeah I don't have a 3 general -- maybe someone else does. 4 MS. KHANNA: I wanted to say or 5 what should be done -- but I don't know 6 without making any difference in the cost of 7 doing it in-house versus doing outside, but 8 one of the problems you see with a contractor 9 is, you have no control over what's happening 10 within the contracting agency. For example, 11 if I was working with Abt consultant and you 12 know I worked in a particular project and 13 then 6 months later, Abt gets the same 14 continuation of the project, it maybe 15 somebody else. 16 And you have no control over the 17 composition of the team that actually does 18 the work, whereas if you do it in-house, not 19 only are you keeping the skills, especially 20 determined, you're trying to recruit and 21 build up the skill levels of the younger 22 staff members. You know, there's a long term BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 146 1 payback in doing it in house whereas it maybe 2 a short term payback in doing it through a 3 contract, and maybe that's something you may 4 want to think about when you take that 5 decision. 6 MR. BLAIR: Go ahead. 7 DR. BURTON: It's getting close to 8 the -- the advantages that come with 9 contracting are more pronounced when there's 10 a great deal of variability in project 11 natures and in workload. When you have a 12 very consistent set of tasks, that you're 13 doing through time, the advantages of 14 contracting out are not nearly as big and 15 it's my sense that most of what happens here 16 happens with a little very predictable 17 regularity so that that probably the 18 advantage is that a lot of front seat 19 contracting wouldn't be -- wouldn't 20 necessarily be realized in this specific 21 setting. That's kind of what I have. 22 MR. BLAIR: There are some ways to BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 147 1 control for that and I know that has given 2 some projects with different contractors is a 3 problem, you know, but keep moving staff 4 around and shifting up from one project to 5 another, but I -- I think there's some 6 contractual ways of dealing, it is with some 7 of that. The other perspective that I would 8 take on it and this is very similar to a 9 perspective we take on you know, we 10 subcontract out something as opposed to doing 11 it ourselves. 12 Again goes back to sort of, what is 13 it you're trying to develop internally as 14 skills, what competencies are you trying to 15 maintain when you contract out. Is it 16 something that, you know is an area that you 17 know, you're not trying to build in 18 internally, by means of a particular project 19 normally, but I think in the back of your 20 mind it should be, you know you're 21 contracting out something that you should be 22 building the capability for internally. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 148 1 DR. FEDER: Howard, to your 2 question, if I were king, I would still be a 3 member of this committee, but I want to quote 4 one of my colleagues from Australia, David 5 Steele. He said that, "If you have poor data 6 then the best estimation method is still not 7 going to get you very much but if you have 8 good data, even decent estimation will be -- 9 will give you almost as good." What that 10 tells me is that if I were king and I had to 11 decide where to -- 12 MR. GRUENSPECHT: You're going to 13 the priority question? 14 DR. FEDER: Yes, I would say look 15 at your data, first of all use of 16 administrative, already available data is of 17 course the cheapest and sometimes the most 18 efficient way and anything that can be done 19 there should be -- but the -- we talked today 20 about some issues on some surveys that led us 21 to think that maybe design issues should be 22 looked at. And for myself, I'm actually more BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 149 1 of a data analyst and I get to look at the 2 data when it's already been collected. It's 3 sometimes used after the data had been 4 collected and it's already done. And it 5 drives me sometimes crazy when I look at it, 6 when I see that certain data, the 7 questionnaire had been changed, you cannot 8 even look at consistency because trends are 9 sometimes much more important than levels. 10 There is a lot of interest now to 11 know why we see a spike in -- hopefully it's 12 a peak, which is going to come down, but you 13 know, cost avoidance so and so, you raise the 14 issue of international data in it's quality 15 and I think maybe there are some sources and 16 some kind of relationships that EIA can build 17 with other organizations in other countries 18 to help there. 19 And I would put my emphasis or best 20 part of my effort in improving data quality 21 prior to data analysis, just because, and of 22 course the simulation is important because it BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 150 1 helps you get more money. But good data is 2 the foundation for that and not necessarily 3 data on this and I'm saying it although 4 personally I'd like to do data analysis, but 5 that's my viewpoint. 6 DR. SITTER: Now, I think that on 7 that issue, I probably as king would be more 8 inclined to cut surveys than quality as a 9 general rule. It has two basic advantages. 10 The worst thing you can do is deteriorate on 11 average overall. Secondly you know, when you 12 state your budgets, you state your budget 13 requests on, well okay, under this budget we 14 can do this, otherwise we can't. I mean -- 15 MR. GRUENSPECHT: Well, that's what 16 we tried to do. 17 DR. SITTER: That's all you can do. 18 MR. GRUENSPECHT: I think this is 19 something that really -- I really started and 20 I -- were in the budget request it spells 21 out, at this level we drop in this survey, 22 we've not done that before. Because I think BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 151 1 that, you know that, every, lot of agencies 2 and a lot of people are been in there, sort 3 of a streamlining process for a long time 4 now. And I mean maybe early on that the 5 solution is different. Because I think that, 6 you know, you come from fat times to lean 7 times, areas start to trim, I mean you can 8 become more efficient, you can decrease costs 9 and it's a good thing. But my feel has been 10 over my time in this committee that you 11 passed that. 12 There's a lot of fact determined in 13 terms of we could do things quicker, faster, 14 more efficiently, decrease cost, and it 15 doesn't seem to really be there. And the 16 budget you're talking about doesn't, you 17 know, say, well we're going to have to have 18 to drop a position or something, these are 19 millions of dollar changes, right. When they 20 drop your budget, they drop you by enough 21 that may cost the entire, an entire survey 22 easily. So now that's my consensus. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 152 1 DR. BURTON: You're first this time 2 and then we'll be -- 3 MR. BLAIR: Kind of to second 4 Randy's comment in that, it also maybe less 5 disruptive internally in a sense if you, you 6 know, just have to cut out a piece of you 7 know, as hard as that is, rather than trying 8 to tinker with a whole bunch of things, this 9 is in terms of how well you can do it, that 10 might also be done with that. 11 DR. BURTON: I know, I'm probably 12 preaching to the choir but I was talking with 13 one of the staff yesterday and I mean a lot 14 of the reason I think the EIA has been able 15 to do as much as it is done, as well as it is 16 done it, with a actual decline in real 17 dollars, so over the last decade it is 18 because of we increased productivity this 19 largely, attributable to technology and 20 communications. Those opportunities for 21 productivity I think have been fully taken 22 advantage of, the ability to ring anymore out BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 153 1 in terms of additional productivity, my guess 2 is its simply not there. I'm telling you 3 this but I'll tell it to anybody else you'd 4 like me to. So I think if you're going to go 5 forward and keep doing what you're doing, 6 much less do anything additional, if you got 7 anymore resources. 8 DR. HENGARTNER: I mean, my 9 comments come from discussions we've had 10 yesterday and it is I think a fact that many 11 of your surveys, survey the same companies 12 for different purposes. And there is maybe a 13 little bit in efficiency of being able to 14 say, I just kind of pin each company once a 15 year or once a month, so whatever the 16 frequency is that we need to, I get all the 17 information on both oil and electricity and 18 whatever else we're asking. 19 Now the problem with that is that 20 this is going to require lots of money to 21 change the e, and the equation is, is it 22 worth the energy in the long term to do that BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 154 1 or not. And I'm in no position to advise on 2 this, but there is probably this one little 3 sliver one can -- I can see maybe getting 4 some increased efficiency. And in terms of 5 dropping surveys, it's higher, I mean it's 6 always going to be someone who uses that they 7 can well feel betrayed, but you're king. 8 DR. HENGARTNER: Those who 9 complain, you know, -- 10 MS. KHANNA: That could -- that 11 could be turned around into an opportunity 12 because the next time that someone from 13 Congress asks for some information and we 14 just don't have it because we terminated the 15 survey, we say, oops, that's what. 16 DR. HENGARTNER: Well, no -- I mean 17 I'm ---- that it dropped four months of that 18 survey. I mean DLS dropped four months of 19 the monthly survey. That's a whole, that's 20 irreplaceable. 21 SPEAKER: Rightly, four months. 22 DR. NEERCHAL: I think that's a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 155 1 good suggestion but I think it largely 2 assumes that there is a certain amount of 3 coordination going on in the company side, 4 because you might have a survey -- 5 DR. BURTON: That's a different 6 form. 7 DR. NEERCHAL: Yeah, once I was 8 dealing with, you know, second floor and the 9 other one in the fourth floor, and they don't 10 talk to each other then it's probably not 11 going to be much of a use coordinating from 12 one side. I wanted to mention a couple of 13 things about the recruiting. One of the 14 things I feel that you can really help in 15 your recruiting is to let the younger people 16 like graduate students and the younger people 17 who know the kind of work that is going on in 18 EIA. So it'll be -- I think this has been, 19 this idea has been thrown around in this 20 committee before, if the project sponsor a 21 session in ASA perhaps with the EIA work, the 22 kind of statistical work that is going on BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 156 1 here. 2 So that you really let the 3 students, the graduating students know that 4 there is an interesting work going on in the 5 agency and some things are, you know, that 6 doesn't have to be in the ASA. It could be 7 some of the smaller satellite conferences, or 8 even in the modeling area there might be some 9 conferences, where, which is suitable. So 10 you really kind of showcase your work and 11 with plenty of interesting things going on 12 here and that is one of the attraction for 13 the students when they graduate, go ahead to 14 work in. They do have lot of choices now and 15 statistics really is selling. 16 DR. SITTER: Well, on the 17 recruiting issue. So here's a question, are 18 you having trouble recruiting still? I think 19 last time it just came up, the economy was 20 blooming right. So I know that at least all 21 the students I know seemed to have as many 22 choices as they wish. We had master BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 157 1 students, master students from Vancouver, 2 turned down flights to the East Coast because 3 they just can't travel that much for 4 interview. So it's not the case, that you 5 are having trouble. 6 MR. GRUENSPECHT: I always think 7 about, the troubles that I focus on are 8 always the internal ones of moving paper 9 through the department systems. It -- my 10 sense is that we're getting some pretty good 11 applicants. 12 MS. KIRKENDALL: Yes, I think we've 13 been getting pretty good applicants. 14 MR. GRUENSPECHT: And so far we 15 think we are all right. 16 DR. SITTER: Because I have -- 17 MR. GRUENSPECHT: If you're giving 18 us a heads up, that's the environment is -- 19 DR. SITTER: It's not changing. 20 MR. GRUENSPECHT: Not changing. 21 DR. SITTER: So that just means you 22 -- I mean I haven't seen it change, I mean I BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 158 1 think that there are still lots of jobs and 2 the students have lots of choices. It just 3 sounds like you know you've an attractive 4 choice now -- 5 DR. HENGARTNER: There's always a 6 problem that you require US Citizenship. 7 MS. KIRKENDALL: Yes. 8 DR. HENGARTNER: And if you look 9 at, the school of graduate students, half of 10 them are out. 11 MS. KHANNA: More than half of 12 them. 13 DR. SITTER: Yeah, that -- that's a 14 big problem. 15 DR. HENGARTNER: That is a problem. 16 DR. SITTER: Yeah. 17 DR. HENGARTNER: And if that could 18 be alleviated or it could be somehow changed, 19 that would help a lot. 20 DR. SITTER: I mean even Los 21 Alamos, hires Canadians. 22 MS. KIRKENDALL: Well what's BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 159 1 interesting because that's what changed here 2 in the last year is that now even our 3 contractors have to be US citizens, well, uh. 4 MR. GRUENSPECHT: Yes, complicated 5 there. Certainly an issue that we're looking 6 at, very hard, there's officially -- 7 SPEAKER: Some conflicting items 8 right there, I would say but access of 9 foreign nationals to certain systems, you 10 know, in headquarters, it's a problem and in 11 our case if you can't have access to our 12 systems what're you going to do. 13 MR. GRUENSPECHT: So it's almost, 14 it's not that foreign nationals are ruled out 15 as contractors, but it's just access to 16 systems issue. 17 DR. NEERCHAL: Of course, the top 18 of the line is the airport security, you need 19 that. 20 MR. GRUENSPECHT: So there are -- 21 that is the challenge. You know, both -- 22 there are challenges in the contractor area BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 160 1 as well. It hasn't gone to the point of an 2 absolute bar, but it's a problem. --- 3b 3 MR. BLAIR: I'd like to reinforce 4 the point that Moshe made about having the 5 data versus sort of what you can devote to 6 analysis and to give that is not just your 7 analysis of course but it's the analysis of 8 other users and that having the data out 9 there, you know is important to them. 10 But I think more broadly yesterday 11 in a lot of the different parts of 12 discussions we had, we were I think as 13 committee saying, we think, you know, this is 14 I guess primarily with an OMB directive and 15 that's saying like, you know, you guys, you 16 know do a good job now and you aren't 17 appreciated sort of for what you're doing 18 though, you know, hopefully that's not in all 19 quarter and that most of you are appreciated 20 and that you have a reputation for excellence 21 and when it comes to having to make choices, 22 I still kind of go with sort of Randy's BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 161 1 approach, it's better to cut out a survey 2 however painful that maybe and whatever 3 constituency that might affect than to tinker 4 here and there. If you start to have a 5 reputation for sliding in terms of quality 6 and in terms of excellence, that takes a long 7 long time to recuperate from. 8 So, if I'm, you know, looking at 9 choices and also trying to keep in mind the 10 long-term, I think that things, you know, are 11 very hard to reverse once they start to -- 12 once they start to slide. 13 MS. KIRKENDALL: I think over the 14 last ten years a lot of those changes in 15 sample sizes is to get a lower precision. 16 Those have already been done. So we are much 17 lower than we've got now, you don't know if I 18 have anything much left. 19 MR. BLAIR: Yeah, if you're not to 20 the point of diminishing returns you're 21 awfully close to. 22 CHAIRMAN BREIDT: John? BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 162 1 MR. WOOD: Just on the diminishing 2 returns, you know, you casually said there 3 might be some efficiencies in going to a 4 company to get all the EIA data. Many of the 5 companies, let's say Exxon Mobil have many 6 times the size of EIA and vastly many times 7 the size of the budget. There would be some 8 efficiency, if you were going to run an oil 9 survey and a big oil production survey and a 10 gas production survey, that would be 11 efficient to go to the production accounting 12 department, but if you were going to go to 13 Exxon Mobil for all the types of information 14 that EIA actually surveys, you would add, I 15 would say significant cost and an efficiency 16 to do so because you know they have this 17 diverse interest on most of all the interior 18 department and all the department of energy, 19 etcetera. So there are limited areas where 20 you could still potentially gain something, 21 or just a blanket go to Exxon and ask them 22 all the questions that you want to know is BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 163 1 not very practical. 2 DR. SITTER: No, I just -- it 3 didn't fall down when I told it to you, you 4 know what I say. 5 CHAIRMAN BREIDT: I just wanted to 6 second or maybe third what Randy and Johnny 7 have been saying that we really don't want to 8 die by a thousand horizontal cuts, you need 9 sharp vertical cuts, you want to share that 10 pain, so you want someone else to suffer for 11 it, and see how they react to it and I don't 12 know how -- 13 MR. GRUENSPECHT: But what I'm 14 asking, how do you prioritize what you pick, 15 is that the answer? 16 CHAIRMAN BREIDT: Well, yeah and 17 for example some of your surveys are now used 18 in derivative products; I mean make them 19 hurt, so -- 20 MS. KHANNA: But that's what the 21 economists are saying. 22 MR. GRUENSPECHT: What a good BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 164 1 example that is, they're actually making us 2 hurt because of the fact they're being used 3 in a way that we had not anticipated, we have 4 to go back and look at our processes and 5 procedures and some of the things that we've 6 done that, you know make our work more 7 efficient like having a lot of people work on 8 something part time. 9 If it gets to be very financially 10 important, then confidentiality is well too, 11 maybe we should analyze this more, well then 12 you have, you know, less efficiency in the 13 group, you know you take one system that was 14 producing that survey and others, you say, 15 no, this only has to be done separately, so 16 the number of people who has access to it has 17 to be reduced to the minimum, well then, you 18 know, yes, it's become more important, 19 professionally those changes are making it 20 more costly for us and we have to do more to 21 be sure that this thing, you know gets out at 22 exactly you know at the time that the market BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 165 1 is expecting it and that everyone gets it at 2 the same time. 3 These are all things that we have 4 never worried about. I mean -- you guys are 5 more general than we are but we went over to 6 NAS, and over to the Department of 7 Agriculture, it was, you know their system of 8 lock-up and -- I don't know if any of you 9 have been, it's really a treat, but it's -- I 10 would say it's quite deluxe compared to the 11 systems that we either have or are 12 contemplating. 13 Now, they have separate switch, 14 separate telecommunication switches and 15 almost -- it's almost like going into a 16 submarine where you close the water tight 17 doors and it just -- you know we are putting 18 out surveys every week, that you say now 19 these derivatives are startling and all but 20 almost that it's worked the other way, you 21 know, have it, the fact that is, that these 22 products have come into the market, have BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 166 1 added sometimes more things that we have to 2 do and have required us maybe to give up a 3 little bit of efficiency to improve security. 4 So at least in the short term it's 5 been adding -- this is so important it merits 6 additional resources. So, it's interesting 7 but your right, those things are potential 8 opportunities but I think it would -- again 9 I'm not the king, the king's over here but I 10 have a feeling that the weekly petroleum 11 inventory survey will not be the first one on 12 our list. 13 CHAIRMAN BREIDT: Right. 14 MR. GRUENSPECHT: Although it will 15 probably be the one that would generate the 16 most -- if you're trying to generate maximum 17 pain. 18 CHAIRMAN BREIDT: Right, I just 19 wonder if there are some slightly subtle ways 20 to do the same thing, you know, it's not 21 quite that in your face but it's similar -- 22 DR. NEERCHAL: Could I ask a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 167 1 related question, do you -- I think there was 2 a session -- related session, couple of 3 meetings ago, do you have a fairly good 4 understanding of who they uses for each 5 survey is and I think you know, you might 6 give a power ranking on each of them -- each 7 of your user, so you know where to push the 8 button if you ever need to -- I think, I also 9 agree, I think that you need to, people will 10 know how an important survey is when it is 11 gone, like we have been complaining about the 12 BLS survey, and you see it is gone, you know 13 how important it was. So once in a while you 14 just let people now, what it feels like if 15 you're not there. 16 So, it maybe an important thing for 17 you to know, who the customers are for each 18 survey and you know, who are the important 19 players in each group. 20 MS. KIRKENDALL: We have spotty 21 data on that, for some offices they have been 22 very good in keeping tabs on who is using BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 168 1 their data profiles and other offices haven't 2 paid too much attention. Dwight French's 3 office survey, energy consumption survey, has 4 always done a good job knowing who the users 5 are. The outside users, the outside federal 6 users especially. 7 MS. KHANNA: Well, I think we are 8 all saying this ma'am, I'm just going to 9 restate. One of the things that we want to 10 keep in mind is what are the goals of the 11 exercise? Is it really -- this is our budget 12 for this year and we have to live with it and 13 so then we trim or is this our budget 14 projections for next year and there is chance 15 to actually getting an increase and then we 16 cut where it hurts, you know because the -- I 17 mean your strategy is going to depend on the 18 goal and so I think we need to first -- each 19 time actually define the goal before we 20 decide what to do. 21 DR. BURTON: Howard, there is 22 another question too and I think we've talked BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 169 1 about this in the past, I can't recall for 2 sure. I mean certainly you all develop 3 products that people would pay for and is 4 there an opportunity to develop revenues 5 through those products. I like being able to 6 get all the stuff for free, but I would damn 7 sure pay for it. 8 DR. FEDER: I was going to say 9 exactly what you said -- 10 DR. HENGARTNER: The stock market 11 there's the option product, given a 12 subscription, those who want it at 9:30 on 13 the Wednesdays and everybody else can have it 14 hour later for free. 15 DR. BURTON: Having it three 16 minutes later would be -- 17 DR. HENGARTNER: Those who want it 18 at 9:30 on Wednesday, you know there is a 19 subscription and that's it. It will get the 20 data to the researchers. 21 MR. GRUENSPECHT: Essentially, I 22 mean some of the stuff has a lot of -- BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 170 1 tremendous market value, others delivers on 2 mission of the agencies that probably do not 3 have a lot of market value but once you go 4 down the path of user fees, which is I guess 5 what's being described. I mean it is an 6 interesting path but it's a path where there 7 maybe whole parts of the program that would 8 not -- 9 DR. HENGARTNER: No, no but you 10 have to be careful, user fees from the 11 private sector pay and the government pays 12 the rest, because the government also 13 requires the data. 14 MR. GRUENSPECHT: Right. 15 DR. HENGARTNER: And that's the 16 current model, that's where we do it. 17 MR. GRUENSPECHT: But we don't have 18 any user fees. 19 DR. HENGARTNER: You have no user 20 fees? 21 MR. GRUENSPECHT: Right. 22 DR. HENGARTNER: Well, what we're BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 171 1 suggesting is make the private sector who 2 receives an economic value in portions of 3 your data might be willing to pay. 4 MS. KHANNA: How do you know, I 5 mean I could download the data and e-mail it 6 to my friend. 7 DR. FEDER: You sign in there, I 8 mean -- you say that you're not -- 9 MS. KHANNA: But everybody does it 10 anyway. 11 DR. BURTON: Oh, only in academics 12 we do that, firms will do that, the risk of 13 -- 14 DR. FEDER: For instance I 15 downloaded the paper -- yeah downloaded it 16 but don't send it outside, because -- you're 17 right companies really would pull with risk 18 and be there. 19 DR. SITTER: And they'd get it at 20 9:31. 21 DR. FEDER: True. 22 MR. GRUENSPECHT: There is really a BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 172 1 tremendous secondary industry of taking EIA 2 data and repackaging it. 3 DR. BURTON: I know. 4 MR. GRUENSPECHT: About the data 5 but -- 6 MR. BLAIR: I wanted to comment 7 that I think relates to a number of these 8 things, whether you're cutting back 9 particular products or whole surveys, or 10 you're doing tinkering inside to try to make 11 adjustments and that is visibility, don't do 12 these things without the outside world kind 13 of being aware of what's going on and what's 14 in the pipeline, particularly it's, you know, 15 you're making small changes inside, nobody 16 sees that, nobody cares you know as long as 17 they are getting so. 18 But if you've got things in the 19 works or you're considering something like a 20 feed based program or whatever I don't know 21 the politics and administration issues but 22 wherever possible, I think that, you know, BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 173 1 you want to be visible, you don't want this 2 to be an internal thing that, you know, you 3 suffered, to let your users and the other 4 people related, suffer along with you as 5 you're making these decisions and perhaps 6 advocate for you and decisions are under 7 considerations that people have feelings 8 about. Well, one of the -- 9 DR. FEDER: Sorry, again summing 10 from my Canadian experience, when provinces 11 need -- province specific estimates or they 12 want more position to augment a simple size, 13 setting a province or some transfer of 14 funding from the province, I don't know if -- 15 MR. GRUENSPECHT: Yes, we've had 16 some of that, I think we had working with 17 certain States -- 18 MR. FRENCH: We've mostly worked 19 with other federal agencies to do things to 20 add questions on surveys and do things 21 generally. Now with our consumption survey 22 at the time we're doing the re-designs of BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 174 1 surveys to try to have the capability of 2 going down the smaller geographic areas, so 3 if there's somebody who does want to pay for 4 something to supplemental survey for a 5 particular area, we're more capable of doing 6 that, but we thought a few years ago that 7 that might be a distinct possibility as there 8 was some States that maybe interested in 9 doing that. But I think States feel squeezed 10 and we're not hearing that the same way as we 11 were a few years ago. Well, we can still do 12 our re-designs to try to provide as much 13 capability for that as possible, if somebody 14 comes out of his work. 15 MR. WOOD: By the way, maybe 15 16 years ago at an ASA meeting, we had as a 17 topic, kind of the value of publicly 18 available data and it was one of the MIT 19 professors and it was just a brilliant 20 presentation, but the fundamental conclusion 21 was the same as we already heard this 22 morning, that if you've got some important BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 175 1 information, it's worth a million dollar to 2 an individual to know it, 24 hours prior to 3 the rest of the world. 4 That is, it loses it's monetary 5 value, even though it keeps it's societal 6 value if you release it to everyone at the 7 same time and it looks like we could have our 8 cake and icing and eat it and everything is 9 good, I like that idea, there's a subscriber 10 list, if you want to know it on Thursday and 11 it's a million dollars a year. And the 12 others gets to it on Friday, and we do do 13 sort of something like that with other 14 federal agencies who fund individual work 15 that they want done, give us money to do it 16 but I think, there might be a few 17 constitutional amendments. 18 There might be some ethical 19 question that might be to put on board, but 20 you know and who really -- what organization, 21 what group really wants to know that 22 information and can they bring pressure on BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 176 1 the people that set our budget. You know we 2 don't go all the way and start really 3 delicate. An agency that has the mechanism 4 in fact to sell it, I think this might be 5 pushing the envelop you know if you had a 6 subscription survey but that's the way huge 7 numbers of consulting firms are actually 8 doing. 9 Many of them are in that secondary 10 market and every single slide in a million 11 dollar customer oriented cost survey and EIA 12 is that data collector and often the analyses 13 that we send out, yet people pay a great deal 14 of money to revive. 15 MR. GRUENSPECHT: If BLS could 16 follow that model they'd be a real cost 17 center for the government. 18 DR. SITTER: I think that this 19 issue of visibility in your customers and who 20 is willing to pay and so forth really tides 21 in well with the discussion on your self 22 evaluation and external review process that BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 177 1 was discussed yesterday. One of the main 2 ideas about what you do well and don't do 3 well when presenting is, you know, who is 4 using your data and the other thing is how 5 much do they value your data and I think that 6 if your going to go through the exercise 7 anyway, it might be a good idea to you know 8 have some of your big customers, take a look 9 at you and talk this things over with them as 10 part of your OMG thing and maybe they'll tell 11 you that they really value your data and when 12 you start talking about well we've really got 13 problems, maybe they will be some help for 14 you there both in your OMG evaluation and at 15 the same you'll get some input, maybe you're 16 wrong, maybe they don't value your data that 17 much. Good to know. 18 MR. BLAIR: I like that idea a lot 19 and I think if there was some way to make it 20 part of the process in the sense that almost 21 -- like you know, part of an advisory board 22 or whatever that, you know would comment and BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 178 1 those kinds of things, build that into the 2 process that they're talking about in terms 3 of improving the agency's effectiveness, that 4 there could be, you know, depending on how 5 you get on the board but you know, 6 potentially powerful source, I think of 7 support for you. 8 MR. GRUENSPECHT: I don't think you 9 made Guy's life any easier, but by that 10 criterion I think we do reach out, but I 11 think there are a lot of good ideas here, 12 some of which would have significant 13 implementation problems, but others I think 14 that we could pursue, and it's really 15 appreciated. Guy, do you have any? 16 MR. CARUSO: No, I appreciate it. 17 MR. GRUENSPECHT: Well, thanks a 18 lot, those are the little, you know beyond 19 your stripes, narrow functional area 20 expertise but I think it's something we need 21 to be looking at. I appreciate your -- 22 MS. KHANNA: It doesn't matter we BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 179 1 like to talk anyway. 2 MR. GRUENSPECHT: What's that? 3 MS. KHANNA: It doesn't matter we 4 like to talk anyway, we're researchers. 5 MR. GRUENSPECHT: And being really 6 patient, I appreciate it, thank you very 7 much. 8 CHAIRMAN BREIDT: Okay, we're right 9 on time here, maybe just a few minutes early. 10 There's an agenda item on committee 11 suggestions for the spring '05 meeting, but 12 we'll defer that till lunch. So, what we're 13 doing now is I'll invite public comment. 14 Yeah, see the public as we make 15 this history. So, before someone has -- 16 yeah, can you come to the mike, please? And 17 if you can just say your name and 18 affiliation? 19 MS. CARROLL: My name is Carolyn 20 Carroll and I'm affiliated with a small 21 statistical consulting company, Stattac. 22 What I would like to say and I could probably BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 180 1 do that -- what I would like to say is that I 2 was struck by how willing and cooperative and 3 collegial this group was and how interested 4 the group was in trying to do better and also 5 how hard this group works. I come out of a 6 couple of different communities, I've spend a 7 lot of time in the Department of Defense and 8 a lot of time in industry, IBM being one of 9 the places and so I wanted to say that this 10 sort of willingness to work through things I 11 think it's very positive and that there are a 12 number of communities that have some of those 13 same interest and that by talking you can 14 find ways to go forward. So that's what I 15 think. 16 CHAIRMAN BREIDT: Thank you. And 17 Neha you had something? 18 MS. KHANNA: Yes, before the 19 meeting is officially adjourned, I want to 20 take the liberty of speaking on behalf of 21 every member of the committee to say thank 22 you to you Jay. BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 181 1 CHAIRMAN BREIDT: Well, thank you 2 very much. Before we did adjourn I wanted to 3 say thank you as well, so. First I wanted to 4 thank the members of the committee for 5 agreeing to participate in the committee and 6 being very robust and flexible and dealing 7 with all of the different kinds of problems 8 we've dealt with over the years. It's really 9 been a privilege to work with such a talented 10 and diverse group, I've learned a lot, I 11 don't normally interact with econometricians 12 and survey methodologist, my normal herd is 13 math-stat type people but it's nice to get 14 out and meet new people, so that's been a 15 pleasure. 16 It's also really been a pleasure to 17 work with EIA. I didn't really know anything 18 about the agency before serving on this 19 committee but I learned a lot about it and I 20 now feel this allegiance that Nancy and Randy 21 were talking about yesterday, so. In the six 22 years I've been on this committee, we've BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 182 1 talked about many many different kinds of 2 problems but almost every time it was not a 3 question of how do we do this but how do we 4 do this better and it's just an obvious 5 dedication to quality throughout this agency 6 that's really impressive and a pleasure to 7 work with. 8 So, I would like to thank the 9 people who -- the EIA staff who regularly 10 interact with committee but they all slipped 11 off right when -- but you know there's many 12 many people that regularly interact and it's 13 -- we see them again and again and get to 14 know them, which is nice and then of course I 15 want to thank Nancy for all her work and Bill 16 in particular for all the logistical support, 17 handling everything making this Chairmanship 18 job very easy, so. So, then it's been real 19 pleasure, thanks very much and we're 20 adjourned. 21 (Whereupon, at 11:53 a.m., the 22 PROCEEDINGS were adjourned.) BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382 183 1 * * * * * 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 BETA REPORTING & VIDEOGRAPHY SERVICES www.betareporting.com (202) 638-2400 1-800-522-2382 (703) 684-2382