1. bookVolume 36 (2020): Issue 3 (September 2020)
    Special Issue on Nonresponse
Journal Details
License
Format
Journal
First Published
01 Oct 2013
Publication timeframe
4 times per year
Languages
English
access type Open Access

A Validation of R-Indicators as a Measure of the Risk of Bias using Data from a Nonresponse Follow-Up Survey

Published Online: 24 Jul 2020
Page range: 675 - 701
Received: 01 Jul 2018
Accepted: 01 May 2020
Journal Details
License
Format
Journal
First Published
01 Oct 2013
Publication timeframe
4 times per year
Languages
English

R-indicators are increasingly used as nonresponse bias indicators. However, their effectiveness depends on the auxiliary data used in their estimation. Because of this, it is not always clear for practitioners what the magnitude of the R-indicator implies for bias in other survey variables, or how adjustment on auxiliary variables will affect it. In this article, we investigate these potential limitations of R-indicators in a case study using data from the Swiss European Social Survey (ESS5), which included a nonresponse follow-up (NRFU) survey. First, we analyse correlations between estimated response propensities based on auxiliary data from the register-based sampling frame, and responses to survey questions also included in the NRFU. We then examine how these relate to bias detected by the NRFU, before and after adjustment, and to predictions of the risk of bias provided by the R-indicator. While the results lend support for the utility of R-indicators as summary statistics of bias risk, they suggest a need for caution in their interpretation. Even where auxiliary variables are correlated with target variables, more bias in the former (resulting in a larger R-indicator) does not automatically imply more bias in the latter, nor does adjustment on the former necessarily reduce bias in the latter.

Keywords

AAPOR, American Association for Public Opinion Research. 2016. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 9th edition.Search in Google Scholar

Bethlehem, J.G. 2002. “Weighting Non-response Adjustment Based on Auxiliary Information.” In Survey Non-response, edited by R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little, 41–54. New York: Wiley.Search in Google Scholar

Beullens, K. and G. Loosveldt. 2012. “Should High response Rates Really be the Primary Objective?” Survey Practice 5(3): 1–5. DOI: https://doi.org/10.29115/SP-2012-0019.Search in Google Scholar

Beullens, K., G. Loosveldt, C. Vandenplas, and I. Stoop. 2018. “Response Rates in the European Social Survey: Increasing, Decreasing, or a Matter of Fieldwork Efforts?” Survey Methods: Insights from the Field. DOI: https://doi.org/10.13094/SMIF-2018-00003.Search in Google Scholar

Brick, J.M. and M.E. Jones. 2008. “Propensity to respond and nonresponse bias.” METRON – International Journal of Statistics, LXVI(1), 51–73.Search in Google Scholar

Brick, J.M. and D. Williams. 2013. “Explaining rising Non-response Rates in Cross-sectional Surveys.” The ANNALS of the American Academy of Political and Social Science 645: 36–59. DOI: https://doi.org/10.1177/0002716212456834.Search in Google Scholar

Brick, J.M. and R. Tourangeau. 2017. “Responsive Survey Designs for Reducing Non-response Bias.” Journal of Official Statistics 33(3): 735–752. DOI: http://dx.doi.org/10.1515/JOS-2017-0034.Search in Google Scholar

Cobben, F. 2009. Non-response in Sample Surveys. Methods for Analysis and Adjustment. PhD thesis, The Hague: Statistics Netherlands. Available at: https://hdl.handle.-net/11245/1.312964 (accessed June 2020).Search in Google Scholar

Cornesse, C. and M. Bosnjak. 2018. “Is there an association between survey characteristics and representativeness? A meta-analysis.” Survey Research Methods 12(1): 1–13. DOI: https://doi.org/10.18148/srm/2018.v12i1.7205.Search in Google Scholar

De Heij, V., B. Schouten, and N. Shlomo. 2015. RISQ manual 2.1. Tools in SAS and R for the computation of R-indicators, partial R-indicators and partial coefficients of variation. RISQ Project. Available at: www.risq-project.eu.Search in Google Scholar

De Leeuw, E. and W. de Heer. 2002. “Trends in Household Survey Non-response: A Longitudinal and International Comparison.” In Survey Non-response, edited by R.M. Groves, D.A. Dillman, J.L. Eltinge, and R.J.A. Little, 41–54. New York: Wiley.Search in Google Scholar

Ernst Stähli, M., D. Joye, M. Sapin, A. Pollien, M. Ochsner, and A. van den Hende. 2018. “Non Response Surveys (NRS): ESS 2006, EVS 2008, ESS 2010, MOSAiCH 2011, ESS 2012, ESS 2014.” [Dataset]. Distributed by FORS, Lausanne. https://doi.org/10.23662/FORS-DS-697-1.Search in Google Scholar

ESS Round 5: European Social Survey 2016: ESS-5 2010 Documentation Report. Edition 4.1. Bergen, European Social Survey Data Archive, NSD – Norwegian Centre for Research Data for ESS ERIC. Available at: https://www.europeansocialsurvey.org/docs/round5/survey/ESS5_data_documentation_report_e04_2.pdf (accessed June 2020).Search in Google Scholar

ESS Round 5: European Social Survey Round 5 Data. 2010. Data file edition 3.4. NSD – Norwegian Centre for Research Data, Norway – Data Archive and distributor of ESS data for ESS ERIC. Available at: https://www.europeansocialsurvey.org/download.html?file=ESS5CH&c=CH&y=2010.Search in Google Scholar

Groves, R.M. 2006. “Non-response Rates and Non-response Bias in Household surveys.” Public Opinion Quarterly 70, Special Issue: 646–675. DOI: https://doi.org/10.1093/poq/nfl033.Search in Google Scholar

Groves, R.M. and S. Heeringa. 2006. “Responsive design for household surveys: tools for actively controlling survey errors and costs.” Journal of the Royal Statistical Society: Series A (Statistics in Society), 169(3): 439–457. DOI: https://doi.org/10.1111/j.1467-985X.2006.00423.xSearch in Google Scholar

Groves, R.M. and E. Peytcheva. 2008. “The Impact of Non-response Rates on Non-response Bias – A Meta-Analysis.” Public Opinion Quarterly 72: 167–189. DOI: https://doi.org/10.1093/poq/nfn011.Search in Google Scholar

Groves, R.M. and M.P. Couper. 1998. Non-Response in Household Interview Survey. New York: John Wiley & Sons.Search in Google Scholar

Groves, R.M., J.M. Brick, M.P. Couper, W. Kalsbeek, B. Harris-Kojetin, F. Kreuter, B.-E. Pennell, T. Raghunathan, B. Schouten, T. Smith, R. Tourangeau, A. Bowers, M. Jans, C. Kennedy, R. Levenstein, K. Olson, E. Peytcheva, S. Ziniel, and J. Wagner. 2008. “Issues Facing the Field: Alternative Practical Measures of Representativeness of Survey Respondent Pools.” Survey Practice 1(3): 1–6. DOI: https://doi.org/10.29115/SP-2008-0013.Search in Google Scholar

Gummer, T. and J.E. Blummenstiel. 2018. “Experimental Evidence on Reducing Nonresponse Bias through Case Prioritization: The Allocation of Interviewers.” Field Methods 30(2): 124–139. DOI: https://doi.org/10.1177/1525822X18757967.Search in Google Scholar

Kreuter, F. 2013. “Facing the Nonresponse Challenge.” The Annals of the American Academy of Political and Social Science 645(1): 23–35. DOI: https://doi.org/10.1177/0002716212456815.Search in Google Scholar

Kreuter, F. and K. Olson. 2011. “Multiple Auxiliary Variables in Nonresponse Adjustment.” Sociological Methods & Research 40(2): 311–332. DOI: https://doi.org/10.1177/0049124111400042.Search in Google Scholar

Lin, I.-F. and N.C. Schaeffer. 1995. “Using Survey Participants to Estimate the Impact of Nonparticipation.” Public Opinion Quarterly 59(2): 236–258, DOI: https://doi.org/10.1086/269471.Search in Google Scholar

Little, R.J.A. and S. Vartivarian. 2005. “Does Weighting for Non-Response increase the Variance of Survey Means?” Survey Methodology 31(2): 161–68. DOI: https://doi.org/10.2307/1403140.Search in Google Scholar

Little, R.J.A. 1986. “Survey Nonresponse Adjustments for Estimates of Means.” International Statistical Review / Revue Internationale de Statistique 54(2), 139–157. JSTOR. DOI: https://doi.org/10.2307/1403140.Search in Google Scholar

Little, R.J.A. and D.B. Rubin. 2014. Statistical Analysis with Missing Data. London, UK: John Wiley & Sons.Search in Google Scholar

Luiten, A. and B. Schouten, 2013. “Tailored Fieldwork Design to increase Representative Household Survey Response: An Experiment in the Survey of Consumer Satisfaction.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 176(1): 169–189. DOI: https://doi.org/10.1111/j.1467-985X.2012.01080.x.Search in Google Scholar

Matsuo, H., J. Billiet, G. Loosveldt, F. Berglund, and Ø. Kleven. 2010. “Measurement and Adjustment of Non-response Bias based on Non-response Survey: the Case of Belgium and Norway in the European Social Survey Round 3.” Survey Research Methods 4(3): 165–178. DOI: http://dx.doi.org/10.18148/srm/2010.v4i3.3774.Search in Google Scholar

McCaffrey, D.F., G. Ridgeway, and A.R. Morral. 2004. “Propensity Score Estimation with Boosted Regression for Evaluating Causal Effects in Observational Studies.” Psychological Methods 9: 403–425. DOI: https://doi.org/10.1037/1082-989X.9.4.403.Search in Google Scholar

Moore, J.C., G.B. Durrant, and P.W. Smith. 2018. “Data Set Representativeness during Data Collection in Three UK Social Surveys: Generalizability and the Effects of Auxiliary Covariate Choice.” Journal of the Royal Statistical Society: Series A (Statistics in Society) 181(1): 229–248. DOI: https://doi.org/10.1111/rssa.12256.Search in Google Scholar

Nagelkerke, N.J. 1991. “A Note on a General Definition of the Coefficient of Determination.” Biometrika 78(3): 691–692. DOI: https://doi.org/10.1093/biomet/78.3.691.Search in Google Scholar

Nishimura, R., J. Wagner, and M. Elliott. 2016. “Alternative Indicators for the Risk of Non-response Bias: A Simulation Study.” International Statistical Review 84(1): 43–62. DOI: https://doi.org/10.1111/insr.12100.Search in Google Scholar

Olmos, A. and P. Govindsamy. 2015. “A Practical Guide for using Propensity Score Weighting in R.” Practical Assessment, Research & Evaluation, 20(13). Available at: http://pareonline.net/getvn.asp?v=20&n=13. (accessed June 2020).Search in Google Scholar

Olson, K. 2006. “Survey participation, Non-Response Bias, Measurement Error Bias, and Total Bias.” Public Opinion Quarterly 70(5): 737–758. DOI: https://doi.org/10.1093/-poq/nfl038.Search in Google Scholar

Olson, K. 2013. “Do Non-response Follow-ups Improve or Reduce Data Quality?: A Review of the Existing Literature.” Journal of the Royal Statistical Society: Series A: Statistics in Society 176(1): 129–145. DOI: http://doi.org/10.1111/j.1467-985X.2012.01042.x.Search in Google Scholar

Peytchev, A., E. Peytcheva, and R.M. Groves. 2010. “Measurement Error, Unit Nonresponse, and Self-Reports of Abortion Experiences.” Public Opinion Quarterly 74: 319–327. DOI: https://doi.org/10.1093/poq/nfq002.Search in Google Scholar

Peytcheva, E. and R.M. Groves. 2009. “Using Variation in Response Rates of Demographic Subgroups as Evidence of Non-response Bias in Survey Estimates.” Journal of Official Statistics 25: 193–201. Available at: https://www.researchgate.net/publication/282119961_Using_Variation_in_Response_Rates_of_Demographic_Subgroups_as_Evidence_of_Nonresponse_Bias_in_Survey_Estimates (accessed June 2020).Search in Google Scholar

Roberts, C. and C. Vandenplas. 2017. “Estimating Components of Mean Squared Error to Evaluate the Benefits of Mixing Data Collection Modes.” Journal of Official Statistics 33(2): 303–334. DOI: https://doi.org/10.1515/jos-2017-0016.Search in Google Scholar

Roberts, C., C. Vandenplas, and M. Ernst Stähli. 2014a. “Using Register Data to assess the Impact of Response Enhancement Methods on the Risk of Non-response Bias.” Survey Research Methods 8(2): 67–80. DOI: http://dx.doi.org/10.18148/srm/2014.v8i2.5459.Search in Google Scholar

Roberts, C., N. Allum, and P. Sturgis. 2014b. “Non-response and Measurement error in an online panel: Does additional Effort to Recruit Reluctant Respondents Result in Poorer Quality Data?” In Online Panel Research: A Data Quality Perspective. edited by M. Callegaro, R. Baker, J. Bethlehem, A.S. Göritz, J.A. Krosnick, and P.J. Lavrakas. Hoboken: Wiley, Survey Methodology Series.Search in Google Scholar

Sakshaug, J.F. and M. Antoni. 2018. “Evaluating the Utility of Indirectly Linked Federal Administrative Records for Nonresponse Bias Adjustment.” Journal of Survey Statistics and Methodology 7(2): 227–249. DOI: https://doi.org/10.1093/jssam/smy009.Search in Google Scholar

Särndal, C.-E. and S. Lundström. 2010. “Design for Estimation: Identifying Auxiliary Vectors to Reduce Nonresponse Bias.” Survey Methodology 36(2): 131–144.Search in Google Scholar

Schouten, B. 2018. “Statistical inference based on randomly generated auxiliary variables.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 80(1): 33–56. DOI: https://doi.org/10.1111/rssb.12242.Search in Google Scholar

Schouten, B. and N. Shlomo. 2017. “Selecting Adaptive Survey Design Strata with Partial R-indicators.” International Statistical Review 85(1): 143–163. DOI: https://doi.org/10.1111/insr.12159.Search in Google Scholar

Schouten, B. and F. Cobben. 2007. “R-Indexes for the Comparison of Different Fieldwork Strategies and Data Collection Modes.” Statistics Netherlands. Discussion Paper, 07002, CBC, Voorburg. Available at: http://hummedia.manchester.ac.uk/institutes/cmist/risq/schouten-cobben-2007-a.pdf (accessed June 2020).Search in Google Scholar

Schouten, B., F. Cobben, and J. Bethlehem. 2009. “Indicators for the Representativeness of Survey Response.” Survey Methodology 35(1): 101–113. Available at: https://pdfs.semanticscholar.org/aa59/4bf03a7cc219ccc6da01d1e3cb14a125d67a.pdf (accessed June 2020).Search in Google Scholar

Schouten, B., F. Cobben, P. Lundquist, and J. Wagner. 2016. “Does more Balanced Response imply less Non-response Bias?” Journal of the Royal Statistical Society, Series A (Statistics in Society) 179(3): 727–748. DOI: https://doi.org/10.1111/rssa.12152.Search in Google Scholar

Schouten, B., J. Bethlehem, K. Beullens, Ø. Kleven, G. Loosveldt, A. Luiten, K. Rutar, N. Shlomo, and C. Skinner. 2012. “Evaluating, Comparing, Monitoring, and Improving Representativeness of Survey Response Through R-Indicators and Partial R-Indicators.” International Statistical Review 80(3): 382–399. DOI: https://doi.org/10.1111/j.1751-5823.2012.00189.x.Search in Google Scholar

Schouten, B., N. Shlomo, and C. Skinner. 2010. “Indicators for Representative Response.” Paper presented at Quality in Official Statistics Conference 2010, Helinski, Finland. Available at: https://q2010.stat.fi/ (accessed June 2020).Search in Google Scholar

Schouten, B., N. Shlomo, and C. Skinner. 2011. “Indicators for Monitoring and Improving Representativeness of Response.” Journal of Official Statistics 27(2): 231–253. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/indicators-for-monitoring-and-improving-representativeness-of-response.pdf (accessed June 2020).Search in Google Scholar

Stoop, I.A. 2004. “Surveying Nonrespondents.” Field Methods 16(1): 23–54. DOI: https://doi.org/10.1177/1525822X03259479.Search in Google Scholar

Stoop, I., J. Billiet, A. Koch, and R. Fitzerald. 2010. Improving Survey Response: Lessons Learned from the European Social Survey. London, UK: John Wiley and Sons Ltd.Search in Google Scholar

Tancreto, J.G. and M. Bentley. 2005. “Determining the Effectiveness of Multiple Non-response Follow-up Contact Attempts on Response and Data Quality.” In Proceedings of the Section on Survey Research Methods: American Statistical Association, 2005. 3626–3632. Minneapolis, MN: American Statistical Association. Available at: http://www.asasrms.org/Proceedings/y2005f.html (accessed June 2020).Search in Google Scholar

Vandenplas, C., D. Joye, M. Ernst Stähli, and A. Pollien. 2015. “Identifying Pertinent Variables for Nonresponse Follow-Up Surveys. Lessons Learned from 4 Cases in Switzerland.” Survey Research Methods 9(3): 141–158. DOI: https://doi.org/10.18148/srm/2015.v9i3.6138.Search in Google Scholar

Voogt, R.J.J. and W.E. Saris, 2005. “Mixed Mode Designs: Finding the Balance Between Non-response Bias and Mode Effects.” Journal of Official Statistics, 21(3): 367–387. Available at: https://www.scb.se/contentassets/ca21efb41fee47d293bbee5bf7be7fb3/-mixed-mode-designs-finding-the-balance-between-nonresponse-bias-and-mode-effects.pdf (accessed June 2020).Search in Google Scholar

Wagner, J. 2012. “A Comparison of Alternative Indicators for the Risk of Nonresponse Bias.” Public Opinion Quarterly 76(3): 555–575. DOI: https://doi.org/10.1093/poq/nfs032.Search in Google Scholar

Williams, D. and J.M. Brick. 2017. “Trends in U.S. Face-To-Face Household Survey Nonresponse and Level of Effort.” Journal of Survey Statistics and Methodology 6(2): 186–211. DOI: https://doi.org/10.1093/jssam/smx019.Search in Google Scholar

Yan, T., R. Tourangeau, and Z. Arens. 2004. “When less is more: Are reluctant respondents poor reporters?” In Proceedings of the Section on Survey Research Methods: American Statistical Association, 2004. 4632–4651. Toronto: American Statistical Association. Available at: http://www.asasrms.org/Proceedings/y2004/files/Jsm2004-000169.pdf (accessed July 2020).Search in Google Scholar

Recommended articles from Trend MD

Plan your remote conference with Sciendo