Funding German Research Foundation (DFG)
Project Duration November 2019 - November 2022
Partners GESIS – Leibniz Institute for the Social Sciences, Utrecht University
Project Team Dr. Henning Silber (GESIS), Prof. Dr. Michael Bosnjak (Trier University), Asst. Prof. Dr. Bella Struminskaya (Utrecht University), Dr. Bernd Weiß (GESIS), Fabienne Kraemer (GESIS), Andrea Sarafoglou (GESIS)

Studies

Data is collected within two panel surveys:

• Probability-based mixed-mode panel survey (i.e., GESIS Panel)
• Non-probability online panel survey

Samples

• Probabilistic sample of the German-speaking population aged 18+ (at the time of recruitment) and permanently resident in Germany
• Quota sample of official panel members of the opt-in online panel provided by respondi AG aged 18+ and permanently resident in Germany

Data Collection

October 2020 - February 2022 (six panel waves in total)

Two data collection intervals:

• short: every month (non-probability online panel survey)
• long: every two/three months (probability-based mixed-mode panel survey + non-probability online panel survey)

The project “Mechanisms of Panel Conditioning in Longitudinal Studies: Reflection, Satisficing, and Social Desirability” is a joint research project realized in cooperation with GESIS – Leibniz Institute for the Social Sciences and Utrecht University that addresses the problem of panel conditioning effects – a measurement error in longitudinal studies that threatens the validity and quality of panel data.

Longitudinal surveys in general are highly relevant to research across disciplines. They allow researchers to study intra-individual stability and change over time and to make statements about causal relationships. By now, Germany hosts multiple large panel infrastructures of national and international importance. Nevertheless, concerns about their quality have been raised along with the request to intensify methodological research to provide information on the quality of panel data and their susceptibility to bias (Leopoldina 2016). The PaCo-project aims to comprehensively investigate the magnitude of bias that comes from panel conditioning as well as to identify its underlying mechanisms.

Panel conditioning effects describe learning effects that occur when the participation in prior waves of the same study affects respondents’ answers in subsequent interviews causing artificial changes which cannot be differentiated from real changes in respondents’ behavior, attitudes, and knowledge over time. This endangers the most important aims of longitudinal research, namely the valid measurement of stability and change.

Goals

The project’s major goal is to gain a deeper understanding of the magnitude (i.e., effect size) of PCE and the mechanisms that cause PCE in order to advance the theoretical and empirical knowledge about panel conditioning. PCE have received growing attention within the research, however, little is still known about their magnitude for different question types (i.e., knowledge, attitudes, and behavior) and dosage conditions (i.e., the frequency with which identical questions are asked across panel waves) as well as the significance of different theoretically proposed mechanisms that cause panel conditioning. The project’s findings will serve as a basis for future recommendations and best practice advice for researchers conducting and working with data of longitudinal studies. Accordingly, there are three main objectives:

• to provide evidence for the relative significance of different mechanisms leading to PCE: changes in knowledge, attitudes, behaviors or changes in reporting of these characteristics due to reflection, satisficing and SD-bias.
• to identify the magnitude (i.e., effect size) and significance of PCE for the different question types commonly used in social surveys (i.e., attitudes, behavior, and knowledge questions), and different dosage conditions, that is the different number of times identical questions are asked (conditioning frequency), and the different time intervals between survey waves (conditioning interval).
• to give recommendations for correction methods that will function as best practice advice. These recommendations will help to decrease measurement inaccuracy for different question types in panel studies.

The project will also have a decisive impact on evidence-based practice because it will help large-scale longitudinal studies with substantial public policy implications to account for potential measurement errors.

Probability-based panel

PaCo data collected within the probability-based panel will be provided by the GESIS Panel (see waves he - id). We will announce the PaCo data release by the GESIS Panel in our newsbox. For more information on the GESIS Panel and how to get access to the data, see https://www.gesis.org/gesis-panel/.

Non-probability panel

The PaCo data collected within the non-probability panel will be accessible over PsychArchives [hyperlink to be announced]. The published data will comprise the cumulative dataset on all fielded panel waves of both panel intervals (1 month vs. 2-3 months). Release of the final dataset will be announced via our newsbox and information on data accessibility will be provided on this website shortly after.

The documentation comprises the project’s study description with a detailed methods report, the questionnaires of each survey wave along with the codebook that describes the existing variables in the data set as well as the field reports of the non-probability panel, and other additional documents such as a cheatsheet and a separate paradata documentation. The documentation will be available soon and can be accessed under the respective links below.

• Study Description & Methods Report (incl. DFG Proposal): [to be announced]
• Wave Reports & Data Processing: [to be announced]
• Cheatsheet: [to be announced]
• Questionnaires: [to be announced]
• Codebook: [to be announced]
• Additional Documents: [to be announced]

Preprints

Kraemer, F., Silber, H., Struminskaya, B., Bosnjak, M., Koßmann, J., & Weiß, B. (2022).  Panel Conditioning in a German Probability-Based Longitudinal Study: A Comparison of Respondents with Different Levels of Survey Experience. https://doi.org/10.31235/osf.io/vd5xp

Kraemer, F., Silber, H., Struminskaya, B., Sand, M., Bosnjak, M., Koßmann, J., & Weiß, B. (2022).  Satisficing Response Behavior Across Time: Assessing Negative Panel Conditioning Using an Experimental Design with Six Repetitions. https://doi.org/10.31235/osf.io/7atfm

Presentations

Kraemer, F., Silber H., Struminskaya, B., Sand, M., Bosnjak, M., Koßmann, J., & Weiß, B. (2022, September).  Do Response Effects Change Over Time? Experimental Results From Six Waves of a German Online Panel Survey. Presentation given at the General Online Research Conference (GOR 22), September 9, 2022, Berlin (Germany). [presentation]

Kluge, R., Krämer, F., Silber, H., Bosnjak, M., Koßmann, J., Struminskaya, B. & Weiß, B. (2022, September).  A Closer Look at Face-Saving Response Options to Reduce Vote Overreporting: Disentangling Social Desirability Bias, Memory Failure, and Response Order Effects. Presentation given at the General Online Research Conference (GOR 22), September 8, 2022, Berlin (Germany). [presentation]

Kraemer, F., Silber H., Struminskaya, B., Sand, M., Bosnjak, M., Koßmann, J., & Weiß, B. (2022, May).  Do Response Effects Change Over Time? Experimental Results From Six Waves of a German Online Panel Survey. Presentation given at the 77th Annual AAPOR Conference, May 13, 2022, Chicago (USA). [presentation]

Kraemer, F., Silber, H., Struminskaya, B., Bosnjak, M., Koßmann, J. & Weiß, B. (2021, November).  Satisficing im Zeitverlauf: Eine Analyse von negativer Panel-Konditionierung unter Verwendung eines experimentellen Designs mit sechs Panelwellen. Vortrag auf der Gemeinsamen Tagung „Paneldaten: Methodische Qualitätsansprüche, praktische Datenanforderungen und politische Informationsbedarfe“ der Arbeitsgemeinschaft sozialwissenschaftlicher Institute (ASI e.V.) und der DGS-Sektion „Methoden der empirischen Sozialforschung“, November 19, 2021, online. [presentation]

Kraemer, F., Bosnjak, M., Koßmann, J., Silber, H., Struminskaya, B., & Weiß, B. (2021, September).  Satisficing Behavior across Time: Assessing Negative Panel Conditioning Using a Randomized Experiment. Presentation given at the General Online Research (GOR 21) Conference, September 10, 2021, online. [presentation] https://www.youtube.com/watch?v=2x8AlnlXeV0

Kraemer, F., Silber, H., Struminskaya, B., Bosnjak, M., Koßmann, J., & Weiß, B. (2021, July). Assessing Panel Conditioning in a German Probability-based Panel: A Comparison of Respondents with Different Levels of Experience. Presentation given at the European Survey Research Association (ESRA) Conference, July 16, 2021, online. [presentation]

Silber, H., Struminskaya, B., Sand, M., Bosnjak, M., Koßmann, J., Kraemer, F., & Weiß, B. (2021, July). Methodological Aspects of Measuring Stability and Change in Personal Culture: Testing the Models of Settled Dispositions and Active Updating Using a Randomized Experiment. Presentation given at the European Survey Research Association (ESRA) Conference, July 16, 2021, online. [presentation]

Kraemer, F., Silber, H., Struminskaya, B., Bosnjak, M., Koßmann, J., & Weiß, B. (2021, March).  Assessing Panel Conditioning in the GESIS Panel. Comparing Respondents with Different Levels of Experience. Presentation given at the Current Innovations in Probability-based Household Internet Panel Research conference (CIPHER), March 4, 2021, online. [presentation]  https://www.youtube.com/embed/yXJ0mHHL1to

Poster

Kraemer, F., Koßmann, J., Bosnjak, M., Silber, H., Struminskaya, B., & Weiß, B. (2020, September).  Panel Conditioning in a Mixed-Mode Probability Panel: Comparing Novice and Experienced Respondents. Poster presented at the General Online Research Conference (GOR 20), September 10, 2020, online. [poster]  https://www.youtube.com/watch?v=e9_eAXcCy8c

Principal Investigators

• Dr. Henning Silber
  GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany
  +49 (0)621 1246-595
  henning.silber(at)gesis.org
  https://orcid.org/0000-0002-3568-3257
• Prof. Dr. Michael Bosnjak
  University of Trier, Trier, Germany
  +49 (0)651 201-2706
  bosnjakuni-trierde
  https://orcid.org/0000-0002-1431-8461
• Asst. Prof. Dr. Bella Struminskaya
  Utrecht University, Utrecht, Netherlands
  b.struminskaya(at)uu.nl
• Dr. Bernd Weiß
  GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany
  bernd.weiss(at)gesis.org
  https://orcid.org/0000-0002-1176-8408

Project Members

• Fabienne Kraemer
  GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany
  fabienne.kraemer(at)gesis.org
  https://orcid.org/0000-0002-0144-5824

Student Research Assistants

• Andrea Sarafoglou
  GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany

Collaborators

Dr. Matthias Sand (GESIS)
Rebekka Kluge (GESIS)

• January, 2022: conference presentations accepted at GOR22 and AAPOR2022
• December, 2021: overall data collection completed
• November, 2021:  field start of last wave in long interval
• November, 2021: data collection of sensitivity degree study completed
• November, 2021: presentation at ASI/DGS conference
• October, 2021:  1st Wave (Prob Panel) published in GESIS Archive
• September, 2021:  5th wave in long interval completed
• September, 2021:  presentation at GOR conference
• July, 2021:  presentations at ESRA conference
• May/June, 2021:  collecting 4th wave in long interval, about 90% completion rate
• April, 2021:  data collection in short interval completed
• March, 2021:  presentation at CIPHER conference
• September, 2020:  poster talk at GOR conference