ODISSEI Lunch Lecture: Reconciliation of inconsistent data sources using hidden Markov models

On Tuesday 26 October from 12:00-13:00 hours, dr. Paulina Pankowska (VU Amsterdam) presents her research on the use of hidden Markov models to produce consistent statistics with inconsistent sources.

We warmly invite you to attend this lecture, learn more about Pankowska’s research, and participate in the Q&A and discussion after the lecture. More information about the lecture can be found below. Automatic registration has closed, but please send an email to communications@odissei-data.nl to receive the link to the session.

In her PhD research, Pankowska examined how National Statistical Institutes (NSI’s) can use hidden Markov models to produce consistent official statistics for categorical, longitudinal variables using inconsistent sources. She used linked survey (LFS) and administrative (Employment Register) data from CBS on employment contract type.

The use of hidden Markov Models can be a complicated and expensive procedure. Therefore, it is preferable to use the error parameter estimates as a correction factor for a number of years. However, this might lead to biased structural estimates if measurement error changes over time or if the data collection process changes. Pankowska’s results on these issues are highly encouraging and imply that the suggested method is appropriate for NSI’s. Specifically, linkage error only leads to substantial bias in very extreme scenarios. Moreover, measurement error parameters are largely stable over time if no major changes in the data collection process occur. However, when a substantial change in the data collection process occurs, such as a switch from dependent (DI) to independent (INDI) interviewing, re-using measurement error estimates is not advisable. These results are more informative for those who are looking to improve the quality of their data and reduce measurement error by using multiple sources on the same variable for the same sample.

Dr. Paulina Pankowska is currently a postdoctoral researcher at the Communication Science and Sociology departments at the Vrije Universiteit Amsterdam. She is working on the development of an online participant recruitment platform for Social Science and Humanities research in the Netherlands and focuses on aspects related to the quality of data collected using such a platform. She is also the task leader for the ODISSEI Benchmarking project that aims to design and set up a social science benchmark. Finally, she is a senior quantitative methodologist at the BaM (Becoming a Minority) Project, which looks at the lives of people without a migration background living in ethnically diverse neighborhoods.


About the ODISSEI Lunch Lecture Series

The ODISSEI Lunch Lectures highlight methodological issues and innovations in Social Science. The Lunch Lecture following this one will take place on 9 December.

Relevant links


Photo by Chris Liverani on Unsplash