Collaborative knowledge acquisition through intelligent crowdsourcing.

In this paper we tackle the problem of crowdsourced knowledge acquisition in an online setting. Given arbitrary questions with unknown answers, we elicit responses from multiple people and, from these responses, pick the one most likely to be correct. We propose two online algorithms that measure the difficulty of each question and the ability of each user that answered that question to gauge the probability of a response being correct. The first model uses a semi-supervised version of a user's prior to compute the correctness of a response. The second model modifies the Rasch model to make it online and unsupervised. The semi-supervised prior achieves a high accuracy (95.5%) but needs to be trained on labeled data. The modified Rasch model achieves a lower accuracy (93.9%) but still beats the baseline (93.4%) that picks the most popular response and requires no training data.