Ground Truth - The Uses and Abuses of Human Annotation in Text Analysis - Professor Kenneth Benoit
Alan Turing Institute via YouTube
Overview
Syllabus
Intro
the curious notion of "ground truth"
the curious notion of ground truth
sources of ground truth
examples in text analysis
sometimes it's far from obvious
how do we validate latent positions?
a recoding experiment
intercoder agreement
misclassification
more recent investigations
text annotation using the crowd
example: labelling immigration sentences
example: economic and social policy
start with expert coding, for benchmark
deployment on Crowdflower
inter-coder agreement
comparing experts to crowd-coders
correlation of aggregate measures
experts were just a less variable crowd
example: immigration policy
test for immigration policy
quality control: very important
conclusions
Taught by
Alan Turing Institute