The Psychological Society of South Africa (PsySSA) held its 2018 conference at Emperor’s Palace from 12 – 14 September 2018. Dr Celeste Combrinck from the Science Mathematics and Technology Education Department (SMTE) presented on a study designed to investigate sampling error and psychometric validity in large scale assessment. Findings from South Africa’s participation in the Progress in International Reading Literacy Study (PIRLS) were used in the analysis. PIRLS utilised stratified random sampling to reduce bias but standard errors remain large for South Africa. The study is exploratory and being conducted in conjunction with the Department of Basic Education (DBE). The conclusion of the presentation was that even when random sampling and stratification are used to select schools, classes and participants, this does not mean that the units are inherently comparable. When indicators are used to predict outcome variables, researchers should consider weighting each group or predictor or controlling for differences between groups by creating dummy variables in regression models.