The whole of this group is a release asset
This site holds 4 of the 261 question directories these 11 runs wrote; the archive holds every one:
- archive
- minidev-sqlite-gpt-35-turbo.tar.gz
- size
- 14,720,914 bytes
- sha256
- d01c13ed83f1e3107e7c14130ca44c9df9ce9a8e19d218f42de33d0de0e8ff38
The sums of the 11 runs
One prediction file, audited once per database it names, because a connection here is one file. Each run below states its own counts; these are those counts added up, which they can be because no question is in two of the runs.
| what | how many |
|---|---|
| audited | 498 |
| compared | 409 |
| EQUAL | 164 |
| NOT_EQUAL | 245 |
| ERROR | 89 |
| probes fired | 53 |
| credited by BIRD and NOT_EQUAL | 27 |
The runs
Every group of minidev-sqlite is indexed a level up.
| database | date | engine | question set | question set digest |
|---|---|---|---|---|
| california_schools | 2026-09-14T07:31:39.496591+00:00 | sqlite | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json | sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b |
| card_games | 2026-09-14T08:32:00.442520+00:00 | sqlite | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json | sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b |
| codebase_community | 2026-09-14T09:56:28.152711+00:00 | sqlite | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json | sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b |
| debit_card_specializing | 2026-09-14T07:31:45.544531+00:00 | sqlite | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json | sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b |
| european_football_2 | 2026-09-14T07:33:33.154928+00:00 | sqlite | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json | sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b |
| financial | 2026-09-14T07:33:33.154358+00:00 | sqlite | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json | sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b |
| formula_1 | 2026-09-14T07:33:33.154562+00:00 | sqlite | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json | sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b |
| student_club | 2026-09-14T07:34:02.009812+00:00 | sqlite | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json | sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b |
| superhero | 2026-09-14T07:34:02.009812+00:00 | sqlite | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json | sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b |
| thrombosis_prediction | 2026-09-14T07:34:02.009904+00:00 | sqlite | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json | sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b |
| toxicology | 2026-09-14T07:34:02.867769+00:00 | sqlite | https://bird-bench.oss-cn-beijing.aliyuncs.com/minidev.zip (sha256 cc48ba16838204e4e214512030cb572eeb5f7bcdd999bae4b9b6ff12ec13b92f, downloaded 2026-09-07), member minidev/MINIDEV/mini_dev_sqlite.json | sha256:4ba5fa8de55856222f484d380d2ba872b380bf79d825de70478e2120cb0fc43b |
Data and credit
The audited question sets are BIRD Mini-Dev and BIRD dev, both licensed under CC BY-SA 4.0. This site shows their question text, evidence text and gold SQL as they ship, beside results this project measured. The rest of the site is this project's own work under Apache-2.0, and NOTICE is the precise list of which files carry which.