AI Art Turing Test
December 2024
Scott Alexander put together an “AI Art Turing Test”. He collected 50 pieces of art, about half made by humans and half by AI, and asked his readers if they could tell which was which. The test was administered as a Google Form and the correct answers were in a comment in the original post.
This was an amazing test, and I had lots of fun taking it, but it was somewhat cumbersome to write down your own guesses and compare them to the answers and to Scott’s notes in his follow up post.
I wanted a version of this great test without these drawbacks, so I collected the 50 images, the answers, and Scott’s notes, and made a new website, ai-art-turing-test.com.
The answers can be shown after each guess or, for the epistemically rigorous, shown only at the end. The site doesn’t record your score—it’s entirely client-side. Scott also released an Excel spreadsheet with the guesses of the 11,061 readers who took his test. I used the data to tell the test-taker what percent of people got each question right, and where they stand in the bell curve of all test-takers:
One issue was what to do with the questions that the user skipped. Scott didn’t include these in his denominator, which means that you could increase your score by skipping images that you were unsure about. I decided to grade skipped answers as wrong. That’s what normally happens on a test at school, so people should expect that. It also makes it easier to compare scores from different people.
The source code is available on GitHub.