ODISSEI Lunchlezing: Reconciliation of inconsistent data sources using hidden Markov models

Op dinsdag 26 oktober van 12:00-13:00 uur presenteert dr. Paulina Pankowska (Vrije Universiteit) haar onderzoek naar het gebruik van verborgen Markov modellen om inconsistente data tot consistente statistiek te verwerken.

We nodigen je van harte uit om deze online lezing bij te wonen en meer te weten te komen over Pankowska’s onderzoek. Meer informatie over de lezing is hieronder te lezen (Engels). De automatische registratie is gesloten, stuur graag een mailtje naar communications@odissei-data.nl om de link naar de bijeenkomst te ontvangen.


In her PhD research, Pankowska examined how National Statistical Institutes (NSI’s) can use hidden Markov models to produce consistent official statistics for categorical, longitudinal variables using inconsistent sources. She used linked survey (LFS) and administrative (Employment Register) data from CBS on employment contract type.

The use of hidden Markov Models can be a complicated and expensive procedure. Therefore, it is preferable to use the error parameter estimates as a correction factor for a number of years. However, this might lead to biased structural estimates if measurement error changes over time or if the data collection process changes. Pankowska’s results on these issues are highly encouraging and imply that the suggested method is appropriate for NSI’s. Specifically, linkage error only leads to substantial bias in very extreme scenarios. Moreover, measurement error parameters are largely stable over time if no major changes in the data collection process occur. However, when a substantial change in the data collection process occurs, such as a switch from dependent (DI) to independent (INDI) interviewing, re-using measurement error estimates is not advisable. These results are more informative for those who are looking to improve the quality of their data and reduce measurement error by using multiple sources on the same variable for the same sample.

Dr. Paulina Pankowska is currently a postdoctoral researcher at the Communication Science and Sociology departments at the Vrije Universiteit Amsterdam. She is working on the development of an online participant recruitment platform for Social Science and Humanities research in the Netherlands and focuses on aspects related to the quality of data collected using such a platform. She is also the task leader for the ODISSEI Benchmarking project that aims to design and set up a social science benchmark. Finally, she is a senior quantitative methodologist at the BaM (Becoming a Minority) Project, which looks at the lives of people without a migration background living in ethnically diverse neighborhoods.

Over de ODISSEI Lunchlezingen

De ODISSEI Lunchlezingen richten zich op methodologische uitdagingen en innovatie in de sociale wetenschappen. De lunchlezing die op deze volgt, vindt plaats op 9 december.

Relevante links

Foto door Chris Liverani op Unsplash