the UID group: Sikuli, picture-driven computing

  1. Sikuli project lead Sean Tsung-Hsiang Chang working with UROP student Maria Rodriguez.
    Sikuli project lead Sean Tsung-Hsiang Chang working with UROP student Maria Rodriguez.
  2. homepage for Sikuli. http://sikuli.csail.mit.edu/  Sikuli is a new way to automate GUI interactions using screenshot patterns to direct mouse and keyboard events. Users can programmatically control a web page, a desktop application running on Windows/Linux/Mac OS X, or even an iphone application running in an emulator as long as you can see them on the screen.
    homepage for Sikuli. http://sikuli.csail.mit.edu/ Sikuli is a new way to automate GUI interactions using screenshot patterns to direct mouse and keyboard events. Users can programmatically control a web page, a desktop application running on Windows/Linux/Mac OS X, or even an iphone application running in an emulator as long as you can see them on the screen.
  3. Sikuli project lead Sean Tsung-Hsiang Chang working with UROP student Maria Rodriguez.
    Sikuli project lead Sean Tsung-Hsiang Chang working with UROP student Maria Rodriguez.
  4. A screen shot of the Sikuli IDE
    A screen shot of the Sikuli IDE
  5. Sikuli project lead Sean Tsung-Hsiang Chang working with UROP student Maria Rodriguez.
    Sikuli project lead Sean Tsung-Hsiang Chang working with UROP student Maria Rodriguez.

Meet Sikuli Project Lead Sean Tsung-Hsiang Chang, PhD candidate in the User Interface Design Group under Professor Rob Miller. Along with EECS senior (6-3 2011) Maria Rodriguez (now MEng. student in EECS), Sean and off-site Project Lead Tom Yeh continue refining Sikuli. The name Sikuli means “God’s eye” in the language of Mexico’s Huichol Indians. Read more in the MIT News Office January 20, 2010 article. View (and test) the beta version of Sikuli.

Sean, could you give an ‘elevator’ pitch for Sikuli?

Sean Chang:
“Sikuli is a programming platform for automating graphical user interfaces (GUIs) using screenshots. Sikuli allows users to refer to GUI widgets by capturing a screen shot of them. With this technique, users can programmatically control any GUIs they see on the screen without the support from the underlying applications.”

What is your goal in developing Sikuli?

Sean Chang:
“My goal is to make Sikuli easy enough for end-users to automate their everyday and repetitive jobs, but also powerful enough for professional developers to script and test their applications automatically.”

How did this project get started?

Sean Chang:
“Sikuli started in 2008, when I got into MIT and joined Prof. Miller’s group. Tom Yeh (who was a PhD student in MIT) had an idea of using screenshots of GUI widgets to search documentation then. I worked with Tom to build a prototype of that idea, and later we came up with an new idea that uses screenshots for GUI automation, which became the current Sikuli.”

If there are a lot of users of Sikuli already, how did you ‘market’ Sikuli’ or did you need to?

Sean Chang:
“We didn’t do much marketing by ourselves. The number of Sikuli users was boosted a lot after MIT News published an article for us. That article was cited by Slashdot and LifeHacker, which brought us thousands of visitors and beta users in a few days. Based on the news effect, we quickly fixed lots of bugs and released three update versions in a month and then started building a community that helps us to write documents and answer questions from so many users.”

Do you have competitors or are you covering a unique need (in a unique way)?

Sean Chang:
“There is some software that addresses GUI automation, but it either requires support from the system to be controlled, or uses methods that are too brittle to changes in the GUI. Besides, no other existing software provides a user interface like ours — one that directly uses and shows screenshots as first-class objects in the programming environment.”

What has being a part of the Miller Group meant to you?

Sean Chang:
“Prof. Miller’s group has much experience in end-user programming systems. Based on that experience, Rob gave us many useful insights and kept us on the right direction, so we don’t need to waste too much time on trial and error.”

What has been your own personal path that enabled you to be part of Prof. Miller’s group?

Sean Chang:
“I have been interested in programming and software engineering for a long time, particularly in making programming easier so everyone can use computers in a more efficient way. Before I got into MIT, I saw programming language researchers working hard on raising the bar of programming and software engineering people trying to figure out how to build giant software effectively. Then I realized the problem was more high-level. Programming is essentially a user interface for commanding computers. It’s just the most powerful and complex one. Therefore I turned to the field of human-computer interaction and found out so much interesting research regarding this problem is going on here. That’s why I chose Prof. Miller’s group, since he’s one of the pioneers in this field.”

Maria, how did you come to be associated with Prof. Miller’s Group and specifically with the Sikuli project?

Maria Rodriguez:
“I took Professor Miller’s software engineering class (6.005) and user interface design class (6.813) and I really enjoyed them. Professor Miller approached me to work on a project [as a UROP] in his group and I chose Sikuli because I thought the experience of being part of a team of developers for a system with many real-world users would be valuable. In the future, I plan to work in the field of bioinformatics and build analysis tools to assist biologists.”

What aspects of your work on Sikuli will you find useful in moving on to bio computation? Will you be entering the new 6-7 degree?

Maria Rodriguez
“I wish I could do the new 6-7 degree! I am staying another year to pursue my master’s in EECS and then I plan to apply to MBA programs. Working on Sikuli has given me more experience in software engineering, which I think is useful for any field, including biological research.”

Leave a Reply

Replies which add to the content of this article in the EECS Newsletter are welcome. The Department reserves the right to moderate all comments. If you would like to provide any updated information please send an email to newsletter@eecs.mit.edu.