the UID group: TurKit, iterative tasks on Mechanical Turk

  1. TurKit team members Greg Little, project leader and EECS senior Elena Tatarchenko, completing her Advanced Undergraduate Project.
    TurKit team members Greg Little, project leader and EECS senior Elena Tatarchenko, completing her Advanced Undergraduate Project.
  2. TurKit homepage featuring download, video, blog and other useful links and information.
    TurKit homepage featuring download, video, blog and other useful links and information.
  3. TurKit team members Greg Little, project leader and EECS senior Elena Tatarchenko, completing her Advanced Undergraduate Project.
    TurKit team members Greg Little, project leader and EECS senior Elena Tatarchenko, completing her Advanced Undergraduate Project.
  4. Figure 5 from:
    Figure 5 from: "TurKit: Human Computation Algorithms on Mechanical Turk." UIST 2010 authors: Greg Little, Lydia B. Chilton, Max Goldman, and Robert C. Miller. This is the TurKit web user interface, an online IDE for writing TurKit scripts, running them, and automatically rerunning them. Projects appear on the left, an editor appears in the center, and output appears to the right. There is also an execution trace pane showing the history of recorded actions. The run controls area has options for switching between sandbox and normal mode on Mechanical Turk, running the script, letting the script rerun automatically, and resetting the script. The lower-right contains a link to the TurKit API reference, as well as example projects which can be cloned as a starting point for writing scripts.
  5. TurKit team members, Greg Little, team lead discussing the project with Elena Tatarchenko, Elena
    TurKit team members, Greg Little, team lead discussing the project with Elena Tatarchenko, Elena
  6. TurKit team members Greg Little, project leader and EECS senior Elena Tatarchenko, completing her Advanced Undergraduate Project.
    TurKit team members Greg Little, project leader and EECS senior Elena Tatarchenko, completing her Advanced Undergraduate Project.

Meet Greg Little, recent EECS PhD graduate who has been Project Lead for TurKit in the Miller UID group working with Elena Tatarchenko, now an MEng student under Prof. Rob Miller.

Greg, could you give a brief description of how TurKit works and what was the motivation behind its conception?

Greg Little:
“TurKit is basically a JavaScript programming API for Mechanical Turk with one key feature: it remembers results from tasks done on Mechanical Turk between runs of a program, so that they do not need to be rerun. This allows a programmer to debug and re-execute their code without re-waiting for tasks to complete.

We wanted to experiment with more complicated algorithms on Mechanical Turk, and TurKit was our in-house tool for doing so. The goal of the project was to demonstrate that more complicated workflows could solve interesting problems.

And, how did TurKit get started?

Greg Little:
“Well, the original motivation was to try and make programming easier by allowing people to outsource very small bits of their code. However, we did not have access to a labor market with on-demand programming experts. Mechanical Turk had on-demand human labor, so we chose problems that most humans could solve, like writing and brainstorming.”

Did you say there are a lot of users of TurKit already? How do you ‘market’ TurKit?

Greg Little:
“I’m not sure about “a lot”. I think 200 people have downloaded it, and I’m not sure how many of those have actually used it. The first marketing TurKit got was a rejected UIST submission that got handed around the UIST community (essentially word of mouth). We also published a blog of Mechanical Turk experiments called Deneme, where most of the experiments used TurKit. Finally, somehow TurKit had pretty good placement on Google — at least one person reported discovering TurKit when searching with the terms “Mechanical Turk API”.

Do you have competitors or are you covering a unique need (in a unique way)?

Greg Little:
“Yes [we have competitors]. Many researchers want to create their own toolkit for using Mechanical Turk. A couple other toolkits I’ve heard of have popped up with novel approaches, including a MapReduce inspired framework, as well as a visual Yahoo Pipes style dataflow model.”

What are the ‘perks’ you gain as members of Prof. Miller’s User Interface Design Group — aside from getting support from this association? ie., do you all meet to describe, trouble shoot?

Greg Little:
“Prof. Miller has a daily meeting we call ‘tea’. This is an informal meeting where people can discuss their research, what they are thinking about, or cool things they’ve seen online. As a result, I feel like I know what everyone is doing, and I have received lots of useful feedback on my own ideas. Because the meeting is informal, I feel that people are willing to brainstorm and discuss ideas before they have matured, which I believe is a good time to receive constructive feedback.”

What has been your own personal path that enabled you to be part of this group?

Greg Little:
“My background was in programming languages and artificial intelligence. In fact, I did not know that the field of HCI existed. However, Prof. Miller also has a background in programming languages, and the group has done a lot of work in the space of end-user programming (e.g. making programming more accessible to end-users).”

Elena, what has been your role as you worked with Greg on the TurKit project?

Elena Tatarchenko:
“TurKit uses a crash and rerun programming model to deal with the high cost and high human latency operations. The goal of my project was to come up with an alternative programming model for TurKit that would actually store the state of execution. To do this, I added support for continuations to the TurKit API. The user is able to create a continuation, serialize it, and when the user chooses to resume execution to deserialize the continuation and continue execution.”

How have you found the experience of being part of Prof. Miller’s UID group?

Elena Tatarchenko:
“We have daily meetings to discuss different topics in the field and particular research topics that we are interested in. We brainstorm ideas for that area and try to come up with new approaches to explore it further. Professor Miller is easily available for meetings and takes a very active role in his group.”

What was the path that led you to become part of Prof. Miller’s UID group working on TurKit?

Elena Tatarchenko:
“My own personal path involved taking two of his classes and doing well in them, 6.005 and 6.813. After exploring different fields in computer science, I decided that UI research was what I wanted to do for my AUP and MEng. Professor Miller was very open to the idea and that’s how I got started.”

Leave a Reply

Replies which add to the content of this article in the EECS Newsletter are welcome. The Department reserves the right to moderate all comments. If you would like to provide any updated information please send an email to newsletter@eecs.mit.edu.