R-ORD GOLD-ONLY not-a-function-of-the-data
formula_1 · dev_20251106-00000-of-00001 from https://huggingface.co/datasets/birdsql/bird_sql_dev_20251106/resolve/3c11fb193e5439b338e23677fa0aae11e8b85db9/data/dev_20251106-00000-of-00001.json (commit 3c11fb19, downloaded 2026-09-07)
Name the top 3 drivers and the points they scored in the 2017 Chinese Grand Prix.
This question was audited without a prediction beside it, so there is nothing to compare the gold with. The probes below read the gold alone.
read by hand, 2026-09-07: harmless the probe fired on something real and the answer is the same either way
the top two drivers hold the same points, so a shuffled copy returns the same three drivers in another order and BIRD compares them as a set
a maintainer's reading of this question, out of classification.json, copied from plans/reports/bird-dev-sqlite-260907/classification.json. It is not a verdict and nothing above it was computed from it.
SELECT T3.forename, T3.surname, T2.points FROM races AS T1 INNER JOIN driverStandings AS T2 ON T2.raceId = T1.raceId INNER JOIN drivers AS T3 ON T3.driverId = T2.driverId WHERE T1.name = 'Chinese Grand Prix' AND T1.year = 2017 ORDER BY T2.points DESC LIMIT 3sha256:2755e605ebc990d2cfdd0e605611096102aa7fe600d8e50b0f00b671741896e7
from evidence-gold.json, 3 rows
| forenameTEXT | surnameTEXT | pointsREAL |
|---|---|---|
| Sebastian | Vettel | 43.0 |
| Lewis | Hamilton | 43.0 |
| Max | Verstappen | 25.0 |
A smell is a mechanical reason to read this gold statement again. It is a heuristic: it does not state that the statement is wrong, and a maintainer decides.
this statement orders by a text column holding only numbers, and ordering it as a number gives a different answer, so the gold may be sorting 9.5 above 10
no ORDER BY key resolves to a text column
{
"heuristic": true,
"reason": "no ORDER BY key resolves to a text column",
"keys": [
{
"key": "T2.points",
"column": "driverStandings.points",
"declared_type": "REAL",
"not_applicable": "the column is not declared as text"
}
]
}this statement cuts its result at a LIMIT that does not decide which rows come back, so a different but equally correct statement can return other rows and score zero
{
"heuristic": true,
"cut": 3,
"offset": 0,
"distinct_kept": false,
"unbounded_sql": "SELECT T3.forename, T3.surname, T2.points, T2.points AS attestql_ordering_key_0 FROM races AS T1 INNER JOIN driverStandings AS T2 ON T2.raceId = T1.raceId INNER JOIN drivers AS T3 ON T3.driverId = T2.driverId WHERE T1.name = 'Chinese Grand Prix' AND T1.year = 2017 ORDER BY T2.points DESC",
"unbounded_rows": 20,
"projected_columns": [
"forename",
"surname",
"points"
],
"ordering_key_columns": [
"attestql_ordering_key_0"
],
"ordering_keys": [
{
"key": "T2.points",
"direction": "desc",
"nulls": "last",
"nulls_first_in_effect": false,
"returned_rows_null_in_this_key": 0,
"fires": false
}
],
"case": null
}rerun over the same rows in another physical order this statement gives another answer, so its result depends on how the rows are stored and not only on the data
from smells.json, 3 rows
| Lewis | Hamilton | 43.0 |
| Sebastian | Vettel | 43.0 |
| Max | Verstappen | 25.0 |
{
"heuristic": true,
"rule": "R-ORD",
"baseline_result_hash": "sha256:2755e605ebc990d2cfdd0e605611096102aa7fe600d8e50b0f00b671741896e7",
"baseline_result": {
"columns": [
{
"name": "forename",
"declared_type": "TEXT"
},
{
"name": "surname",
"declared_type": "TEXT"
},
{
"name": "points",
"declared_type": "REAL"
}
],
"row_count": 3,
"truncated": false,
"rows_shown": 3,
"rows": [
[
{
"type": "str",
"value": "Sebastian"
},
{
"type": "str",
"value": "Vettel"
},
{
"type": "dec",
"value": "43.0"
}
],
[
{
"type": "str",
"value": "Lewis"
},
{
"type": "str",
"value": "Hamilton"
},
{
"type": "dec",
"value": "43.0"
}
],
[
{
"type": "str",
"value": "Max"
},
{
"type": "str",
"value": "Verstappen"
},
{
"type": "dec",
"value": "25.0"
}
]
],
"result_hash": "sha256:2755e605ebc990d2cfdd0e605611096102aa7fe600d8e50b0f00b671741896e7"
},
"planner_statistics": {},
"shuffle": {
"seed": "1",
"row_limit": 300000,
"tables": [
"races",
"driverStandings",
"drivers"
],
"tables_not_shuffled": [],
"tables_skipped_for_size": {
"lapTimes": 420369
},
"tables_not_reached_by_a_copy": {}
},
"shuffled_copies": {
"run": true,
"verdict": "not_equal",
"differs": true,
"result_hash": "sha256:8f8106b90a887b3417f9d144b92d79fc7fc42b992b83f48e8229835c3f6fa525",
"result": {
"columns": [
{
"name": "forename",
"declared_type": "TEXT"
},
{
"name": "surname",
"declared_type": "TEXT"
},
{
"name": "points",
"declared_type": "REAL"
}
],
"row_count": 3,
"truncated": false,
"rows_shown": 3,
"rows": [
[
{
"type": "str",
"value": "Lewis"
},
{
"type": "str",
"value": "Hamilton"
},
{
"type": "dec",
"value": "43.0"
}
],
[
{
"type": "str",
"value": "Sebastian"
},
{
"type": "str",
"value": "Vettel"
},
{
"type": "dec",
"value": "43.0"
}
],
[
{
"type": "str",
"value": "Max"
},
{
"type": "str",
"value": "Verstappen"
},
{
"type": "dec",
"value": "25.0"
}
]
],
"result_hash": "sha256:8f8106b90a887b3417f9d144b92d79fc7fc42b992b83f48e8229835c3f6fa525"
}
},
"plan_variant": {
"run": false,
"reason": "the plan variant was not asked for"
}
}this statement returns the same whole row more than once and never says DISTINCT, so a statement answering the same question once per row disagrees on multiplicity alone
{
"heuristic": true,
"rows": 3,
"distinct_rows": 3,
"repeated_rows": 0,
"largest_repeat": 1,
"result_bounded": false,
"distinct_stated": false,
"set_operation": false
}SELECT T3.forename, T3.surname, T2.points FROM races AS T1 INNER JOIN driverStandings AS T2 ON T2.raceId = T1.raceId INNER JOIN drivers AS T3 ON T3.driverId = T2.driverId WHERE T1.name = 'Chinese Grand Prix' AND T1.year = 2017 ORDER BY T2.points DESC LIMIT 3result_hash sha256:2755e605ebc990d2cfdd0e605611096102aa7fe600d8e50b0f00b671741896e7 recomputed from this JSON: match
record_hash sha256:8d494bbc6204b0cad909b6bde402d2b4ae115d352b3d92f69fefa41c3956c716 recomputed from this JSON: match
from evidence-gold.json, 3 rows
| forenameTEXT | surnameTEXT | pointsREAL |
|---|---|---|
| Sebastian | Vettel | 43.0 |
| Lewis | Hamilton | 43.0 |
| Max | Verstappen | 25.0 |
re-run this statement read-only against SQLite 3.53.4 | file=/private/tmp/attestql-runs/data/dev/dev_databases/formula_1/formula_1.sqlite | size=22360064 under the session settings and over the data this record's fixture digest names, and compare the two results under R-ORD