# Nonobench

> Nonobench is a benchmark of how well large language models solve nonogram (picross) puzzles. Results last updated 2026-09-27T15:35:03.611Z.

## Method

Each model gets the same system prompt and a puzzle's row and column clues, and must answer with the filled grid. An answer is correct when it satisfies every row and column clue; some puzzles have more than one valid solution. There are 40 puzzles in two tiers:

- Standard: 10 each of 5x5, 10x10 and 15x15. Answers are one string of `1` (filled) and `0` (empty), row by row. Overall accuracy covers these 30 puzzles.
- Hard mode: 10 random 20x20 puzzles, each with a single solution. Answers are written one row per line. Each family runs it at its best Standard effort level, and its results stay separate from the overall score. Query it with `size=20x20`.

## Access the data

No authentication. Everything is read-only.

- REST API: `https://www.nonobench.com/api/v1`. OpenAPI spec: https://www.nonobench.com/api/openapi.json
  - `GET /api/v1/leaderboard?size=10x10&provider=openai&effort=best&version=1.2`: models ranked by accuracy, with shared ranks for equal displayed scores. Filters: `provider`, `family` (comma-separated ids), `version` (comma-separated 1.0, 1.1, 1.2), `effort` (best, all, or a level), `reasoning` and `open_weights` (true/false), `size`, `min_correct` (non-negative integer). API defaults: effort all, min_correct 0 (includes variants that solved none). Use `min_correct=1` to hide them. Leaderboard and model responses include `version`.
  - Empty `provider`, `family`, or `version` values mean no filter; spaces around comma-separated ids are ignored. Empty `effort` means all. Unknown open-weight status is excluded by both weights filters.
  - `GET /api/v1/providers`: provider ids, names and families
  - `GET /api/v1/families`: family ids, display names, efforts and best variants
  - `POST /api/v1/compare` with `{"models":["Claude Sonnet 4.5","GLM 5"]}`: compare family-best variants or exact variant ids
  - `GET /api/v1/models/{model}`: one model, per grid size (accuracy, cost, latency, tokens)
  - `GET /api/v1/models/{model}/puzzles`: outcomes for all 40 puzzles, including which were solved
  - `GET /api/v1/puzzles?size=5x5`: the puzzles with ids and clues
  - `GET /api/v1/puzzles/{id}?include_solution=true`: one puzzle and the exact prompt text
  - `GET /api/v1/puzzles/{id}/results?family=gpt-6-sol&effort=best&include_answers=true`: per-model outcomes; filters match the leaderboard and answers are opt-in
  - `POST /api/v1/puzzles/{id}/check` with `{"grid": "0110..."}`: check a grid against the clues
  - `GET /api/v1/runs?model=&puzzle=&size=&include_output=true&limit=100&offset=0`: individual runs
- MCP server (Streamable HTTP, stateless): `https://www.nonobench.com/mcp`. Tools: get_leaderboard, list_providers, list_families, compare_models, get_model_results, get_model_puzzles, list_puzzles, get_puzzle, get_puzzle_results, check_solution, list_runs
- Bulk downloads: https://www.nonobench.com/results-raw.json (every run with prompt and output, ~11 MB)
- Puzzle outcomes and parsed grids: https://www.nonobench.com/puzzle-results.json
- Source and benchmark runner: https://github.com/mauricekleine/nonobench
- Made by [Maurice Kleine](https://www.mauricekleine.com/)

## Pages

- [Leaderboard](https://www.nonobench.com/): results by model and grid size. Also available as markdown at https://www.nonobench.com/index.md
- [Puzzle explorer](https://www.nonobench.com/puzzles): browse the puzzles. Markdown: https://www.nonobench.com/puzzles.md
- [Puzzle insights](https://www.nonobench.com/puzzles/overview): difficulty ranking, model heatmap, and links to answer overlays

## Credits

- maker: [maurice kleine](https://www.mauricekleine.com/)