Lecturer: Edwin Simpson, is a lecturer (equivalent to assistant professor) at the University of Bristol, working on interactive natural language processing. His research focusses on learning from small and unreliable data, including user feedback, and adapts Bayesian approaches to topics such as argumentation, summarisation and sequence labelling. Previously, he was a post-doc at TU Darmstadt, Germany and completed his PhD at the University of Oxford on Bayesian methods for aggregating crowdsourced data.
Abstract: Current machine learning methods are data hungry. Crowdsourcing is a common solution to acquiring annotated data at large scale for a modest price. However, the quality of the annotations is highly variable and annotators do not always agree on the correct label for each data point. This talk presents techniques for aggregating crowdsourced annotations using preference learning and classifier combination to estimate gold-standard rankings and labels, which can be used as training data for ML models. We apply approximate Bayesian approaches to handle noise, small amounts of data per annotator, and provide a basis for active learning. While these techniques are applicable to any kind of data, we demonstrate their effectiveness for natural language processing tasks.