Back to Methodology
Response Rates and Nonresponse Bias in the 2018 CBECS Buildings Survey
Release date: April 12, 2023
The starting sample for the 2018 CBECS was about 16,000 buildings. The final count of completed building cases was 6,288, plus another 208 strip shopping center building cases, for a total of 6,436 buildings in the final data set. About 20% of the sample was not eligible for the survey .
We can calculate different types of response rates for a survey. The CBECS has traditionally calculated an unweighted response rate to show the proportion of eligible sample units that cooperated in the study. Those response rates were calculated by dividing the number of responding eligible sample units by the number of all eligible units in the survey. In 2018, the calculation of an unweighted response rate was complicated by the fact that twice during the field period, we subsampled the nonresponding cases to focus remaining data collection resources toward some sample units while excluding others. Consequently, we term the 2018 CBECS response rate a partially weighted response rate because it is unweighted in terms of using the base weights in the calculation, but weighted to account for the second-phase subsampling procedures.
Campus subsampling
One second-phase sampling procedure occurred in situations in which respondents for campuses with multiple sampled buildings informed data collectors that they would not be able to complete the survey for all the sampled buildings on their campuses. For three universities and three office parks, the number of originally sampled buildings was subsampled to reduce respondent burden.
Closeout procedure subsampling
The other second-phase sampling procedure occurred late in the field period as a way to save resources and concentrate efforts on some of the buildings without a final disposition code (in-progress buildings). In-progress buildings that were considered to be of lower priority and less likely to respond were subsampled, meaning we continued to try to collect data for a portion of a buildings, and we stopped trying for the remaining portion.
Response rates
The response rate formula for 2018 CBECS is a partially weighted response rate that accounts for second-phase subsampling procedures. This partially weighted response rate is akin to an unweighted response rate in a study that does not implement a second-phase sample. Although the CBECS program has typically provided AAPOR Response Rate 51, there is no AAPOR equivalent for the 2018 CBECS partially weighted response rate.2
The weight used to adjust for second-phase sampling and used in calculating partially weighted response rates is the product of the inverse of the campus subsampling probability of selection and the inverse of the closeout procedure probability of selection. A case not eligible for a second-phase sampling procedure was given a probability of selection of one for that procedure. The partially weighted response rate (PWRR) was calculated as:
PWRR = R / (R + ENR + UNK*e)
where
R = count of respondents, partially weighted for the second-stage subsampling procedures;
ENR = count of eligible nonrespondents, partially weighted for the second-stage subsampling procedures;
UNK = count of cases of unknown eligibility, partially weighted for the second-stage subsampling procedures;
e = (R + ENR)/ (R + ENR + IE); and
IE = count of in-scope cases determined to be ineligible, partially weighted for the second-stage subsampling procedures.
Respondents are cases for which we have completed interviews (R). Nonrespondents are cases for which we do not have survey data; some nonrespondents were eligible for CBECS while others were not. Out of scope cases are those that were determined during the screener phase (without talking with a respondent) to be outside the scope of interest as defined for the purposes of the CBECS; these buildings are excluded from the response rate calculation. Ineligible buildings are those cases that were determined during the interview (from information provided by the respondent) to be too small or not sufficiently commercial to be included (IE). A small proportion of nonrespondents had completed enough of the interview to determine that they were eligible (ENR). Because we have no interview data for most nonresponding buildings, the true number of ineligible sample units is unknown. For the response rate, the number of eligible units among those with unknown eligibility is estimated based on the eligibility rate (e) among sample units for which we do have interview data (UNK).
The overall partially weighted response rate (including complete strip shopping center cases) was 54.7%.
The overall partially weighted response rate for establishment interviews was 48.0%.
The overall partially weighted response rate for strip shopping center management interviews was 38.4%.
Nonresponse bias
Analysis of nonresponse bias is just as important as calculating response rates for understanding how well the results of data collection lead to precise estimates of building characteristics and energy consumption. A detailed nonresponse bias study was performed for the 2018 CBECS to assess if respondents and nonrespondents differed in meaningful ways with respect to energy consumption. The analyses performed in the study showed little evidence of nonresponse bias in the final estimates. The study is available upon request.
Endnotes
1The American Association for Public Opinion Research. 2016. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 9th edition. AAPOR.
2We have adapted the response rate on the top of page 68 in the American Association for Public Opinion Research (AAPOR) Standard Definitions, wherein we set the base weight (wi = 1), which corresponds to an unweighted analysis, and then applied the inverse of the subsampling rate.
