Mechanisms of Panel Conditioning in Longitudinal Studies: Reflection, Satisficing, and Social Desirability (PaCo)

A Joint Research Project by GESIS, Trier University, and Utrecht University

 

Overview

Funding German Research Foundation (DFG)
Project Duration November 2019 - Dezember 2024
Partners GESIS – Leibniz Institute for the Social Sciences, Trier University, Utrecht University
Project Team Dr. Henning Silber (GESIS), Prof. Dr. Michael Bosnjak (Trier University), Asst. Prof. Dr. Bella Struminskaya (Utrecht University), Dr. Bernd Weiß (GESIS), Fabienne Kraemer (GESIS)

Studies

Data is collected within two panel surveys:

  • Probability-based mixed-mode panel survey (i.e., GESIS Panel)
  • Non-probability online panel survey

Samples

  • Probabilistic sample of the German-speaking population aged 18+ (at the time of recruitment) and permanently resident in Germany
  • Quota sample of official panel members of the opt-in online panel provided by respondi AG aged 18+ and permanently resident in Germany

Data Collection

October 2020 - February 2022 (six panel waves in total)

Two data collection intervals:

  • short: every month (non-probability online panel survey)
  • long: every two/three months (probability-based mixed-mode panel survey + non-probability online panel survey)

Panel Conditioning Project

Overview

Funding German Research Foundation (DFG)
Project Duration November 2019 - November 2022
Partners GESIS – Leibniz Institute for the Social Sciences, Utrecht University
Project Team Dr. Henning Silber (GESIS), Prof. Dr. Michael Bosnjak (Trier University), Asst. Prof. Dr. Bella Struminskaya (Utrecht University), Dr. Bernd Weiß (GESIS), Fabienne Kraemer (GESIS), Andrea Sarafoglou (GESIS)

Studies

Data is collected within two panel surveys:

  • Probability-based mixed-mode panel survey (i.e., GESIS Panel)
  • Non-probability online panel survey

Samples

  • Probabilistic sample of the German-speaking population aged 18+ (at the time of recruitment) and permanently resident in Germany
  • Quota sample of official panel members of the opt-in online panel provided by respondi AG aged 18+ and permanently resident in Germany

Data Collection

October 2020 - February 2022 (six panel waves in total)

Two data collection intervals:

  • short: every month (non-probability online panel survey)
  • long: every two/three months (probability-based mixed-mode panel survey + non-probability online panel survey)

Project Description

The project “Mechanisms of Panel Conditioning in Longitudinal Studies: Reflection, Satisficing, and Social Desirability” is a joint research project realized in cooperation with GESIS – Leibniz Institute for the Social Sciences and Utrecht University that addresses the problem of panel conditioning effects – a measurement error in longitudinal studies that threatens the validity and quality of panel data.

Longitudinal surveys in general are highly relevant to research across disciplines. They allow researchers to study intra-individual stability and change over time and to make statements about causal relationships. By now, Germany hosts multiple large panel infrastructures of national and international importance. Nevertheless, concerns about their quality have been raised along with the request to intensify methodological research to provide information on the quality of panel data and their susceptibility to bias (Leopoldina 2016). The PaCo-project aims to comprehensively investigate the magnitude of bias that comes from panel conditioning as well as to identify its underlying mechanisms.

Panel conditioning effects describe learning effects that occur when the participation in prior waves of the same study affects respondents’ answers in subsequent interviews causing artificial changes which cannot be differentiated from real changes in respondents’ behavior, attitudes, and knowledge over time. This endangers the most important aims of longitudinal research, namely the valid measurement of stability and change.

Goals

The project’s major goal is to gain a deeper understanding of the magnitude (i.e., effect size) of PCE and the mechanisms that cause PCE in order to advance the theoretical and empirical knowledge about panel conditioning. PCE have received growing attention within the research, however, little is still known about their magnitude for different question types (i.e., knowledge, attitudes, and behavior) and dosage conditions (i.e., the frequency with which identical questions are asked across panel waves) as well as the significance of different theoretically proposed mechanisms that cause panel conditioning. The project’s findings will serve as a basis for future recommendations and best practice advice for researchers conducting and working with data of longitudinal studies. Accordingly, there are three main objectives:

  • to provide evidence for the relative significance of different mechanisms leading to PCE: changes in knowledge, attitudes, behaviors or changes in reporting of these characteristics due to reflection, satisficing and SD-bias.
  • to identify the magnitude (i.e., effect size) and significance of PCE for the different question types commonly used in social surveys (i.e., attitudes, behavior, and knowledge questions), and different dosage conditions, that is the different number of times identical questions are asked (conditioning frequency), and the different time intervals between survey waves (conditioning interval).
  • to give recommendations for correction methods that will function as best practice advice. These recommendations will help to decrease measurement inaccuracy for different question types in panel studies.

The project will also have a decisive impact on evidence-based practice because it will help large-scale longitudinal studies with substantial public policy implications to account for potential measurement errors.

Research Design

Data is collected within two different panel surveys, that is a) a probability-based mixed-mode panel (i.e., the GESIS Panel) and b) a non-probability online panel (i.e., the Study on Science and Society).

For both studies, the project employs an experimental design to investigate the impact of different dosage conditions, that is, different intervals between survey waves and the frequency with which identical questions are asked across waves, on PCE.

The study within the probability-based panel survey is characterized by a 3-factorial experiment manipulating the frequency of answering identical questions (1 time vs. 3 times vs. 6 times) over the course of the study. Intervals between singel survey waves have not been manipulated as respondents of the GESIS Panel are uniformly surveyed every 2 to 3 months. However, the study within the non-probability panel includes a partly-crossed 2x6-factorial experiment manipulating the interval between single survey waves (1 month vs. 2-3 months) as well as the frequency with which identical questions are asked over the course of the study (from 1 to 6 times).

To manipulate the frequency of receiving identical questions (i.e., target questions) over the course of the study, additional questionnaire versions with filler items have been constructed. Thus, dependent on their assignment to an experimental group, respondents either received the questionnaire with target items in the conditioning state or a filler questionnaire. The questionnaires with filler items varied with each survey to prevent confounding conditioning effects on the content of the filler items.

In total, the project’s research design results in 8 experimental groups which are a) fielded in different waves dependent on their assignment to the conditioning interval manipulation (1 month vs. 2-3 months) b) receive different questionnaire versions depending on their assignment to the conditioning frequency manipulation (1 to 6 times). Three of the 8 experimental groups (i.e., Group 3, Group 6, and Group 8) are both represented in the probability-based as well as the non-probability study to enable the investigation of dosage effects on the magnitude of PCE for different samples.

The figure below illustrates the specific research design showing the wave schedule as well as the administered questionnaire versions for the single experimental groups.

Panel Sample

Data

Probability-based panel

PaCo data collected within the probability-based panel will be provided by the GESIS Panel (see waves he - id). We will announce the PaCo data release by the GESIS Panel in our newsbox. For more information on the GESIS Panel and how to get access to the data, see https://www.gesis.org/gesis-panel/.

Non-probability panel

The PaCo data collected within the non-probability panel will be accessible over PsychArchives [hyperlink to be announced]. The published data will comprise the cumulative dataset on all fielded panel waves of both panel intervals (1 month vs. 2-3 months). Release of the final dataset will be announced via our newsbox and information on data accessibility will be provided on this website shortly after.

Documentation

The documentation comprises the project’s study description with a detailed methods report, the questionnaires of each survey wave along with the codebook that describes the existing variables in the data set as well as the field reports of the non-probability panel, and other additional documents such as a cheatsheet and a separate paradata documentation. The documentation will be available soon and can be accessed under the respective links below.

  • Study Description & Methods Report (incl. DFG Proposal): [to be announced]
  • Wave Reports & Data Processing: [to be announced]
  • Cheatsheet: [to be announced]
  • Questionnaires: [to be announced]
  • Codebook: [to be announced]
  • Additional Documents: [to be announced]

Project Deliverables

Preprints

Kraemer, F., Silber, H., Struminskaya, B., Bosnjak, M., Koßmann, J., & Weiß, B. (2022).  Panel Conditioning in a German Probability-Based Longitudinal Study: A Comparison of Respondents with Different Levels of Survey Experience. https://doi.org/10.31235/osf.io/vd5xp

Kraemer, F., Silber, H., Struminskaya, B., Sand, M., Bosnjak, M., Koßmann, J., & Weiß, B. (2022).  Satisficing Response Behavior Across Time: Assessing Negative Panel Conditioning Using an Experimental Design with Six Repetitions. https://doi.org/10.31235/osf.io/7atfm

 

Presentations

Kraemer, F., Silber H., Struminskaya, B., Sand, M., Bosnjak, M., Koßmann, J., & Weiß, B. (2022, September).  Do Response Effects Change Over Time? Experimental Results From Six Waves of a German Online Panel Survey. Presentation given at the General Online Research Conference (GOR 22), September 9, 2022, Berlin (Germany). [presentation]

Kluge, R., Krämer, F., Silber, H., Bosnjak, M., Koßmann, J., Struminskaya, B. & Weiß, B. (2022, September).  A Closer Look at Face-Saving Response Options to Reduce Vote Overreporting: Disentangling Social Desirability Bias, Memory Failure, and Response Order Effects. Presentation given at the General Online Research Conference (GOR 22), September 8, 2022, Berlin (Germany). [presentation]

Kraemer, F., Silber H., Struminskaya, B., Sand, M., Bosnjak, M., Koßmann, J., & Weiß, B. (2022, May).  Do Response Effects Change Over Time? Experimental Results From Six Waves of a German Online Panel Survey. Presentation given at the 77th Annual AAPOR Conference, May 13, 2022, Chicago (USA). [presentation]

Kraemer, F., Silber, H., Struminskaya, B., Bosnjak, M., Koßmann, J. & Weiß, B. (2021, November).  Satisficing im Zeitverlauf: Eine Analyse von negativer Panel-Konditionierung unter Verwendung eines experimentellen Designs mit sechs Panelwellen. Vortrag auf der Gemeinsamen Tagung „Paneldaten: Methodische Qualitätsansprüche, praktische Datenanforderungen und politische Informationsbedarfe“ der Arbeitsgemeinschaft sozialwissenschaftlicher Institute (ASI e.V.) und der DGS-Sektion „Methoden der empirischen Sozialforschung“, November 19, 2021, online. [presentation]

Kraemer, F., Bosnjak, M., Koßmann, J., Silber, H., Struminskaya, B., & Weiß, B. (2021, September).  Satisficing Behavior across Time: Assessing Negative Panel Conditioning Using a Randomized Experiment. Presentation given at the General Online Research (GOR 21) Conference, September 10, 2021, online. [presentation] https://www.youtube.com/watch?v=2x8AlnlXeV0

Kraemer, F., Silber, H., Struminskaya, B., Bosnjak, M., Koßmann, J., & Weiß, B. (2021, July). Assessing Panel Conditioning in a German Probability-based Panel: A Comparison of Respondents with Different Levels of Experience. Presentation given at the European Survey Research Association (ESRA) Conference, July 16, 2021, online. [presentation]

Silber, H., Struminskaya, B., Sand, M., Bosnjak, M., Koßmann, J., Kraemer, F., & Weiß, B. (2021, July). Methodological Aspects of Measuring Stability and Change in Personal Culture: Testing the Models of Settled Dispositions and Active Updating Using a Randomized Experiment. Presentation given at the European Survey Research Association (ESRA) Conference, July 16, 2021, online. [presentation]

Kraemer, F., Silber, H., Struminskaya, B., Bosnjak, M., Koßmann, J., & Weiß, B. (2021, March).  Assessing Panel Conditioning in the GESIS Panel. Comparing Respondents with Different Levels of Experience. Presentation given at the Current Innovations in Probability-based Household Internet Panel Research conference (CIPHER), March 4, 2021, online. [presentationhttps://www.youtube.com/embed/yXJ0mHHL1to

 

Poster

Kraemer, F., Koßmann, J., Bosnjak, M., Silber, H., Struminskaya, B., & Weiß, B. (2020, September).  Panel Conditioning in a Mixed-Mode Probability Panel: Comparing Novice and Experienced Respondents. Poster presented at the General Online Research Conference (GOR 20), September 10, 2020, online. [posterhttps://www.youtube.com/watch?v=e9_eAXcCy8c

Project Team

Principal Investigators

Project Members

Student Research Assistants

  • Andrea Sarafoglou
    GESIS – Leibniz Institute for the Social Sciences, Mannheim, Germany

 

Collaborators

Dr. Matthias Sand (GESIS)
Rebekka Kluge (GESIS)

Archived News

  • January, 2022: conference presentations accepted at GOR22 and AAPOR2022
  • December, 2021: overall data collection completed
  • November, 2021:  field start of last wave in long interval
  • November, 2021: data collection of sensitivity degree study completed
  • November, 2021: presentation at ASI/DGS conference
  • October, 2021:  1st Wave (Prob Panel) published in GESIS Archive
  • September, 2021:  5th wave in long interval completed
  • September, 2021:  presentation at GOR conference
  • July, 2021:  presentations at ESRA conference
  • May/June, 2021:  collecting 4th wave in long interval, about 90% completion rate
  • April, 2021:  data collection in short interval completed
  • March, 2021:  presentation at CIPHER conference
  • September, 2020:  poster talk at GOR conference