Ivanovska, A., Bošnjak, M., & Vehovar, V. (2026). Data quality in estimates from probability-based online panels: Systematic review and meta-analysis. Acta Informatica Pragensia - Journal of Applied Informatics, 15(1). 

Abstract

Background: General population surveys now increasingly use nonprobability samples from access panels instead of probability-based methods, which often leads to lower-quality estimates. In response, many official and academic surveys have adopted probability-based online panels (PBOPs), which use probability sampling and retain participants for follow-up surveys. While these panels reduce costs compared to one-time surveys, they still face low response rates and other challenges that may affect data quality.

Objective: This study aimed to assess the accuracy of PBOPs by synthesising evidence on relative bias (RB), and to examine how RB varies by country, domain, measurement level, and item sensitivity.

Methods: A systematic review yielded 44 eligible studies from 12 countries, and 1,897 effect sizes of absolute RB from studies that compared PBOP estimates to benchmarks. A three-level random effects meta-analytic model accounted for variance across studies, within studies and sampling variance. Moderator analyses evaluated the influence of country, item topic, measurement level and sensitivity on RB. Sensitivity analyses excluded the top 5% of RB outliers to test robustness.

Results: The pooled RB was 23.14% (95% CI: 18.38%–27.91%) and heterogeneous. Most variance was attributed to within-study item-level differences. Country and topic did not significantly moderate RB. Items with high topic sensitivity had significantly higher RB (+19.33%) than items with no sensitivity.  Ordinal items had significantly lower RB than nominal (–14.90%). However, when sensitivity and measurement level were modelled together, substantial residual heterogeneity remained.

Conclusion: While PBOPs offer cost and logistical advantages, they require careful design considerations to lower substantial bias, especially regarding item sensitivity and measurement scale. PBOPs may not be suitable for certain question types, like sensitive or low-prevalence behaviours, especially when high accuracy is needed. Improved methodological planning and innovations are needed to improve PBOP data quality.

https://doi.org/10.18267/j.aip.279

Schäufele-Elbers, I., Bošnjak, M., Gastaldello, G., & Schamel, G. (2025). Nudging meat off the plate in food service? A preregistered systematic review and meta-analysis of field intervention studies. Journal of Environmental Psychology, 108, 1-13.

Abstract

This meta-analysis investigates real-world nudging interventions to reduce meat consumption, uniquely focusing on 33 field experiments with 78 effect sizes that use revealed preferences to minimize attitude-behavior bias. Key findings indicate that the effectiveness of nudging interventions is highly heterogeneous (I² = 99.97 %), with an estimated prediction interval ranging from −50 % to 28 %. Therefore, our focus is on the conditions that moderate potential nudging effects in food service: Interventions targeting decision structure yield a substantial effectiveness with reductions in meat consumption up to 30 %, among them, choice defaults with a 54 % reduction on average. Conversely, strategies based on information provision or decision assistance were found to be ineffective in promoting behavioral change. Our study also identifies a concern: approximately 30 % of interventions resulted in a backfire effect, leading to increased meat consumption. This proportion is higher than reported in some previous meta-analyses and could be attributed to less controlled field study environments or psychological reactance. Future research should continue to explore the complex interplay of moderating variables, investigate the long-term persistence of default effects, examine potential compensation behaviors, and analyze how individual values and attitudes might further moderate nudging success. Understanding these nuances will be crucial for designing and implementing truly effective and context-sensitive strategies to promote more sustainable and healthy dietary patterns.

https://doi.org/10.1016/j.jenvp.2025.102830

Gastaldello, G., Bošnjak, M., Schäufele-Elbers, I., & Schamel, G. (2025). Are consumers willing to pay more for sustainable wine? A pre-registered systematic review and meta-analysis. Food Quality and Preference, 134.

Abstract

The importance of sustainability in the wine sector has gradually increased in recent years, due to significant pressure from the market, regulators and a changing climate. Given significant investments and risks that wineries face to transition towards sustainable production practices, numerous studies have investigated consumers' willingness to pay (WTP) for sustainably produced wine. Most of these studies focus on the environmental pillar of sustainability, while attention to the social pillar is more recent. In this context, our paper aims to fill an important research gap: no meta-analysis has analysed the WTP for sustainably produced wine, considering social and environmental dimensions as well as differentiating between pure claims, actual practices and certified actions. Our results are heterogeneous indicating a WTP of 15 % (95 CI, 13 %–18 %) for sustainably produced wine based on 146 effect sizes. That is, price premia depend on a number of methodological and substantive moderators, with the latter primarily explaining WTP heterogeneity. Key moderators include the data collection year, the use of certified labels vs generic claims and declarations regarding sustainable practices. To a lesser extent, organic certifications, the study location, respondents' wine consumption frequency and the number of sustainability attributes examined are also influential. Our results can guide researchers in the field to optimize study designs, wineries to valorise their sustainability efforts and policy makers to support the wine sector's sustainable transition.

https://doi.org/10.1016/j.foodqual.2025.105655

Kraemer, F., Lugtig, P., Struminskaya, B., Silber, H., Weiß, B., & Bošnjak, M. (2025). Monitoring attitudes over time - Real change or the result of repeated interviewing? Sociological Methods and Research.

Abstract

Panel data are often used to study change and stability in social patterns. However, repeated interviewing may affect respondents’ attitudes in a panel study by triggering reflection processes on the surveyed topics (cognitive stimulus hypothesis). Using data from a survey experiment within a probability-based and a nonprobability panel in Germany, we investigate change—and the mechanisms underlying change—in respondents’ abortion attitudes over six panel waves. The experiment manipulated the frequency of receiving identical attitude questions. We estimate multiple-group and longitudinal structural equation models to differentiate change in the measurement of abortion attitudes from “real” attitude change. Results show that repeatedly administering the same abortion questions increases the reliability of respondents’ reported attitudes and the stability of their latent attitudes toward abortion. However, we find no evidence of an increase in attitude certainty and knowledgeability on abortion and only tentative evidence of improved response behavior (increased attitude reliability) due to general survey experience.

https://doi.org/10.1177/00491241251372503

Bartol, J., Bošnjak, M., Vehovar, V., & Petrovčič, A. (2025). Big five personality dimensions and disintegration as antecedents of vertical and horizontal internet privacy concerns. Cyberpsychology: Journal of Psychosocial Research on Cyberspace, 19(3), 1-26.

Abstract

Given the role that internet privacy concerns (IPCs) have in individuals’ online behaviors, it is essential to understand how personality shapes people’s concerns about privacy online. The five-factor model of human personality—with the Big Five personality dimensions of openness, conscientiousness, extraversion, agreeableness, and neuroticism—has been widely used to study the formation of internet users’ privacy concerns. However, existing literature has investigated only vertical (institutional) IPCs, while there is no corresponding research on horizontal (peer) IPCs. The role of a recently proposed personality trait, disintegration, which captures individuals’ proneness to perceive connections among unrelated phenomena and thus complements existing personality models, has also not been examined yet. This preregistered study addressed these gaps by investigating the associations of the Big Five personality dimensions and the disintegration trait with vertical and horizontal IPCs among internet users. The hypotheses and research questions were addressed through path analysis on a sample of internet users (N = 797) aged 18 years or older, recruited through probability-based sampling (response rate 42%). The results showed that agreeableness was associated with greater vertical IPCs, whereas openness with lower and disintegration with higher horizontal IPCs. The association between disintegration and vertical IPCs was moderated by previous privacy violations. The findings suggest that personality traits have different associations with vertical and horizontal IPCs, warranting further investigation of the role of the disintegration trait in individuals’ online privacy perceptions and behaviors.

Ivanovska, A., Bošnjak, M., & Vehovar, V. (2025). Relative bias in health estimates from probability-based online panels: Systematic review and meta-analysis. Journal of Health and Rehabilitation Sciences, 4(1), 1-14.

Abstract

Introduction: Health surveys require the highest data quality, especially when they inform public health policies. With recent technological developments, probability-based online panels (PBOPs) are becoming an attractive cost-effective alternative to traditional surveys. They are also beginning to be used for official health statistics. However, PBOPs still face concerns about bias, especially for health-related estimates.

Method: Using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) approach, we conducted a systematic review and meta-analysis of PBOP PBOP health survey data quality, with relative bias (RB) of the estimates as the effect size. We analysed 137 health-related survey items from 14 studies and used a linear regression model to examine factors that moderate RB.

Results: The RB varied considerably across the subjects, and its overall median was 12.7%. The highest RBs were exhibited by disabilities (23.6%), mental illnesses (23.2%), personal mental health conditions (20.8%) and drug use (20.7%), and the lowest, by doctor’s treatment (2.24%). The measurement levels with ordinal scales (25.8%) showed higher RB, and certain country effects were also observed.

Conclusion: This moderate bias of the health estimates raises concerns about the accuracy of PBOP estimates regarding sensitive health topics. Therefore, PBOP should be used cautiously for official health statistics; and when designing PBOP surveys for health subjects, the item and study characteristics should be included as methodological considerations.

https://doi.org/10.33700/jhrs.4.1.137

Azua, C. & Bošnjak, M. (2024). Effectiveness, efficiency, and satisfaction: A systematic review and meta-analysis on the strength of associations among usability dimensions in informational websites

Abstract

A systematic review and meta-analysis of informational websites usability studies conducted between 2005-2020 is presented. The sample sizes of the studies identified were on average small, primarily consisting of homogeneous samples. Digital libraries were the most tested websites, followed by news and information websites. The purposes of the studies varied, including the development or evaluation of usability measures, website development, and investigation of usability aspects in general. The meta-analysis estimated the average intercorrelation between the three usability dimensions effectiveness, efficiency, and satisfaction amounting to r = .6. Contrary to a previous meta-analysis, this study suggests that changes in one dimension may lead to corresponding changes in the other two. However, the meta-analysis encountered limitations. The small number of effect sizes available prevented the identification of potential sources of heterogeneity through moderator analyses. Incomplete reporting and questionable data-sharing practices hindered the utilization of primary study findings.

https://doi.org/10.31219/osf.io/qcmt4

Spitzer, L., Bošnjak, M. & Müller, S. (2024). Testing the usability of the Psychological Research Preregistration-Quantitative (PRP-QUANT) template. Meta-Psychology, 8(2024).

Abstract

The Psychological Research Preregistration-Quantitative (PRP-QUANT) Template provides researchers with a comprehensive list of elements to consider when planning a psychological study. We assessed its usability and researchers’ intention to use it. We conducted a usability test (study 1) and surveyed researchers who submitted or reviewed a preregistration created with the template (study 2, authors: N = 19, reviewers: N = 29) regarding their impression of the template. For the usability test, we recruited participants via the mailing lists of the German Psychological Society, the American Psychological Association, and the British Psychological Society, and social media. Participants answered selected template and web probing items and provided an overall rating (N = 88). Based on the Unified Theory of Acceptance and Use of Technology (UTAUT), we expected that the intention to use the template is influenced by performance expectancy (moderated by age), effort expectancy (moderated by age and experience), and social influence (moderated by age, experience, and voluntariness, N = 60). The results suggest that the PRP-QUANT Template is suitable for different research areas within psychology, is evaluated as effective, and perceived positively. Performance expectancy and all predictors combined significantly predicted researchers’ intention to use the template. A plain language summary in English and German is available in Text S1 in the supplemental material.

https://doi.org/10.15626/MP.2023.4039

Thölking, T., Knezevic, G., Lazarevic, L., &  Bošnjak, M. (2024). Internal consistency of the DELTA scale assessing psychotic-like experiences: A reliability generalization meta-analysis. Zeitschrift für Psychologie, 232(4), 279-290. 

Abstract

This study’s objective was to conduct a reliability generalization meta-analysis of the DELTA scale capturing Disintegration, i.e., proneness to psychotic-like experiences and behaviors. Eligible studies applied various versions of the DELTA scale; 573 α coefficients nested in 76 reports were identified. The pooled effect of all included α coefficients was calculated by applying a 4-level meta-analytic random-effects model. Test length, sample type, demographic sample characteristics (mean age, share of females), language, the proportion of reverse-keyed items, publication status, documentation quality of reliability estimates (reported vs. post hoc computed reliabilities), and DELTA score M and SDs were tested as moderators. The model yielded an average Cronbach’s α for DELTA and its facets of .86 (95% CI [.84, .88]), with medium and long DELTA versions reaching .96 (95% CI [.94, .96]) and .97 (95% CI [.95, .98]), respectively. Moderators explaining the heterogeneity among α coefficients were as follows: test length (expectedly, higher reliability estimates for longer tests), mean scores and SD of the DELTA scores (lower α estimates for larger means and lower SDs), and language (highest α estimates for the German language). Overall, the DELTA scales used in past research showed high-reliability estimates.

https://doi.org/10.1027/2151-2604/a000581

Background: Some COVID-19 testing centres have reported manipulated test numbers for antigen tests/rapid tests. This study compares statistical approaches with traditional fraud detection methods. The extent of agreement between traditional and statistical methods was analysed, as well as the extent to which statistical approaches can identify additional cases of potential fraud. Methods: Outlier detection marking a high number of tests, modeling of the positivity rate (Poisson Regression), deviation from distributional assumptions regarding the first digit (Benford’s Law) and the last digit of the number of reported tests. The basis of the analyses were billing data (April 2021 to August 2022) from 907 testing centres in a German city. Results: The positive agreement between the conventional and statistical approaches (‘sensitivity’) was between 8.6% and 24.7%, the negative agreement (‘specificity’) was between 91.3% and 94.6%. The proportion of potentially fraudulent testing centres additionally identified by statistical approaches was between 7.0% and 8.7%. The combination of at least two statistical methods resulted in an optimal detection rate of test centres with previously undetected initial suspicion. Conclusions: The statistical approaches were more effective and systematic in identifying potentially fraudulent testing centres than the conventional methods. Testing centres should be urged to map paradata (e.g. timestamps of testing) in future pandemics.

English Version

German Version