An experiment was conducted to investigate if AI labs are optimizing their models specifically for Simon Willison's pelican-riding-a-bicycle benchmark.

Over 1,000 SVGs were generated using seven advanced models and evaluated by an LLM judge. The analysis utilized Claude Fable 5 and included a grid of 48 prompts, combining 8 animals and 6 vehicles. The pelican-on-a-bicycle prompt is part of this grid and has become a well-known informal benchmark in AI discussions, especially on platforms like Hacker News.

Source: dylancastillo.co