Antigravity Model Benchmark Leaderboard

Automated SWE-Bench coding evaluation & Pareto efficiency matrix • Run run_full_sweep • Generated 2026-09-25 20:44:13

Models Evaluated
17
Total Trials Executed
102
Top Resolution Rate
100.0%
Fastest Solved Avg
22.3s

Model Leaderboard & Efficiency Rankings

Model Name ▴▾ Pass Rate ▴▾ Solved / Total ▴▾ Avg Time (s) ▴▾ Avg Total Tokens ▴▾ Avg Thinking Tokens ▴▾ Avg Cost / Task ▴▾ Pareto Frontier ▴▾
gemini-3.6-flash-high 100.0% 6 / 6 22.3s 51,339 4,629 $0.0106 Pareto Optimal
gemini-3.1-pro-low 100.0% 6 / 6 26.0s 24,165 1,636 $0.0403 -
gemini-3.6-flash-low 100.0% 6 / 6 26.4s 38,168 1,770 $0.0073 Pareto Optimal
gemini-3.7-flash-medium 100.0% 6 / 6 27.2s 85,019 2,381 $0.0146 -
gemini-3.7-flash-low 100.0% 6 / 6 28.7s 78,402 1,552 $0.0132 -
gemini-3.8-flash-low 100.0% 6 / 6 29.4s 89,735 1,616 $0.0150 -
gemini-flash-demo 100.0% 6 / 6 34.2s 91,170 4,139 $0.0165 -
gemini-3.1-pro-high 100.0% 6 / 6 38.4s 29,331 3,403 $0.0536 -
gemini-3.7-flash-high 100.0% 6 / 6 39.0s 105,788 4,543 $0.0189 -
gemini-3.6-flash-medium 100.0% 6 / 6 40.7s 49,615 3,545 $0.0098 -
agy-demo 100.0% 6 / 6 53.7s 52,993 1,001 $0.0753 -
gemini-3.8-flash-medium 100.0% 6 / 6 70.6s 140,993 7,927 $0.0261 -
gemini-3.8-flash-demo 100.0% 6 / 6 84.1s 155,949 11,669 $0.0300 -
gemini-3.8-flash-high 100.0% 6 / 6 86.7s 114,162 7,232 $0.0215 -
gpt-oss-120b-medium 50.0% 3 / 6 165.0s 39,527 0 $0.0206 -
claude-sonnet-4-6 0.0% 0 / 6 120.4s 0 0 $0.0000 -
claude-opus-4-6-thinking 0.0% 0 / 6 124.7s 0 0 $0.0000 -

Task Matrix Results

Task ID Model Status Time Total Tokens Thinking Tokens Cost ($)
task_01_kv_ttl agy-demo PASSED 42.9s 56,992 564 $0.0784
task_02_frontmatter agy-demo PASSED 91.7s 56,358 2,345 $0.0861
task_03_limiter agy-demo PASSED 43.4s 54,979 703 $0.0752
task_04_refactor agy-demo PASSED 56.1s 54,126 409 $0.0745
task_05_emitter agy-demo PASSED 45.0s 56,795 877 $0.0799
task_06_semver agy-demo PASSED 43.1s 38,707 1,105 $0.0578
task_01_kv_ttl claude-opus-4-6-thinking FAILED 123.4s 0 0 $0.0000
task_02_frontmatter claude-opus-4-6-thinking FAILED 130.2s 0 0 $0.0000
task_03_limiter claude-opus-4-6-thinking FAILED 132.2s 0 0 $0.0000
task_04_refactor claude-opus-4-6-thinking FAILED 127.2s 0 0 $0.0000
task_05_emitter claude-opus-4-6-thinking FAILED 126.2s 0 0 $0.0000
task_06_semver claude-opus-4-6-thinking FAILED 108.7s 0 0 $0.0000
task_01_kv_ttl claude-sonnet-4-6 FAILED 124.5s 0 0 $0.0000
task_02_frontmatter claude-sonnet-4-6 FAILED 118.2s 0 0 $0.0000
task_03_limiter claude-sonnet-4-6 FAILED 128.2s 0 0 $0.0000
task_04_refactor claude-sonnet-4-6 FAILED 128.7s 0 0 $0.0000
task_05_emitter claude-sonnet-4-6 FAILED 112.2s 0 0 $0.0000
task_06_semver claude-sonnet-4-6 FAILED 110.3s 0 0 $0.0000
task_01_kv_ttl gemini-3.1-pro-high PASSED 33.9s 36,398 2,742 $0.0595
task_02_frontmatter gemini-3.1-pro-high PASSED 66.4s 40,555 6,825 $0.0823
task_03_limiter gemini-3.1-pro-high PASSED 26.4s 22,232 1,878 $0.0381
task_04_refactor gemini-3.1-pro-high PASSED 27.3s 24,821 2,017 $0.0419
task_05_emitter gemini-3.1-pro-high PASSED 31.2s 23,283 2,416 $0.0431
task_06_semver gemini-3.1-pro-high PASSED 45.3s 28,694 4,542 $0.0566
task_01_kv_ttl gemini-3.1-pro-low PASSED 29.6s 35,140 2,038 $0.0559
task_02_frontmatter gemini-3.1-pro-low PASSED 34.7s 25,953 2,927 $0.0484
task_03_limiter gemini-3.1-pro-low PASSED 21.6s 18,857 1,711 $0.0324
task_04_refactor gemini-3.1-pro-low PASSED 20.1s 21,029 678 $0.0323
task_05_emitter gemini-3.1-pro-low PASSED 25.6s 20,935 1,070 $0.0344
task_06_semver gemini-3.1-pro-low PASSED 24.3s 23,078 1,394 $0.0385
task_01_kv_ttl gemini-3.6-flash-high PASSED 13.0s 49,928 2,144 $0.0091
task_02_frontmatter gemini-3.6-flash-high PASSED 37.4s 74,612 13,356 $0.0185
task_03_limiter gemini-3.6-flash-high PASSED 19.4s 41,002 2,074 $0.0077
task_04_refactor gemini-3.6-flash-high PASSED 23.3s 47,999 1,941 $0.0087
task_05_emitter gemini-3.6-flash-high PASSED 14.4s 45,391 2,539 $0.0087
task_06_semver gemini-3.6-flash-high PASSED 26.3s 49,104 5,717 $0.0108
task_01_kv_ttl gemini-3.6-flash-low PASSED 27.3s 43,170 701 $0.0075
task_02_frontmatter gemini-3.6-flash-low PASSED 35.2s 44,610 4,960 $0.0098
task_03_limiter gemini-3.6-flash-low PASSED 21.9s 27,347 1,079 $0.0050
task_04_refactor gemini-3.6-flash-low PASSED 26.4s 45,356 542 $0.0077
task_05_emitter gemini-3.6-flash-low PASSED 17.1s 29,843 0 $0.0055
task_06_semver gemini-3.6-flash-low PASSED 30.2s 38,682 3,335 $0.0080
task_01_kv_ttl gemini-3.6-flash-medium PASSED 26.8s 51,076 2,483 $0.0094
task_02_frontmatter gemini-3.6-flash-medium PASSED 48.8s 59,436 6,771 $0.0130
task_03_limiter gemini-3.6-flash-medium PASSED 43.2s 45,625 1,752 $0.0083
task_04_refactor gemini-3.6-flash-medium PASSED 43.0s 49,641 1,426 $0.0088
task_05_emitter gemini-3.6-flash-medium PASSED 39.3s 39,871 2,866 $0.0081
task_06_semver gemini-3.6-flash-medium PASSED 42.8s 52,040 5,973 $0.0113
task_01_kv_ttl gemini-3.7-flash-high PASSED 25.7s 93,453 2,042 $0.0158
task_02_frontmatter gemini-3.7-flash-high PASSED 48.0s 122,065 8,765 $0.0236
task_03_limiter gemini-3.7-flash-high PASSED 45.7s 116,988 2,554 $0.0196
task_04_refactor gemini-3.7-flash-high PASSED 40.2s 112,375 2,492 $0.0190
task_05_emitter gemini-3.7-flash-high PASSED 38.0s 101,801 5,041 $0.0184
task_06_semver gemini-3.7-flash-high PASSED 36.0s 88,044 6,363 $0.0170
task_01_kv_ttl gemini-3.7-flash-low PASSED 14.3s 50,001 439 $0.0083
task_02_frontmatter gemini-3.7-flash-low PASSED 25.2s 83,962 2,892 $0.0148
task_03_limiter gemini-3.7-flash-low PASSED 20.0s 75,340 1,069 $0.0124
task_04_refactor gemini-3.7-flash-low PASSED 20.0s 88,986 0 $0.0140
task_05_emitter gemini-3.7-flash-low PASSED 61.5s 96,859 1,407 $0.0161
task_06_semver gemini-3.7-flash-low PASSED 31.2s 75,262 3,507 $0.0137
task_01_kv_ttl gemini-3.7-flash-medium PASSED 19.2s 90,243 1,144 $0.0148
task_02_frontmatter gemini-3.7-flash-medium PASSED 38.6s 101,344 5,394 $0.0187
task_03_limiter gemini-3.7-flash-medium PASSED 19.6s 75,945 690 $0.0123
task_04_refactor gemini-3.7-flash-medium PASSED 27.1s 83,380 1,401 $0.0139
task_05_emitter gemini-3.7-flash-medium PASSED 24.4s 86,064 2,444 $0.0148
task_06_semver gemini-3.7-flash-medium PASSED 34.3s 73,135 3,210 $0.0133
task_01_kv_ttl gemini-3.8-flash-demo PASSED 41.3s 114,131 3,127 $0.0197
task_02_frontmatter gemini-3.8-flash-demo PASSED 156.8s 195,892 29,848 $0.0449
task_03_limiter gemini-3.8-flash-demo PASSED 64.2s 125,748 3,072 $0.0212
task_04_refactor gemini-3.8-flash-demo PASSED 75.8s 190,053 4,817 $0.0321
task_05_emitter gemini-3.8-flash-demo PASSED 55.2s 111,852 8,796 $0.0217
task_06_semver gemini-3.8-flash-demo PASSED 111.3s 198,016 20,355 $0.0407
task_01_kv_ttl gemini-3.8-flash-high PASSED 41.7s 105,593 3,303 $0.0185
task_02_frontmatter gemini-3.8-flash-high PASSED 180.4s 0 0 $0.0000
task_03_limiter gemini-3.8-flash-high PASSED 66.6s 153,188 4,475 $0.0263
task_04_refactor gemini-3.8-flash-high PASSED 58.7s 132,916 4,795 $0.0234
task_05_emitter gemini-3.8-flash-high PASSED 91.2s 155,303 17,195 $0.0324
task_06_semver gemini-3.8-flash-high PASSED 81.5s 137,969 13,625 $0.0285
task_01_kv_ttl gemini-3.8-flash-low PASSED 20.1s 76,747 0 $0.0122
task_02_frontmatter gemini-3.8-flash-low PASSED 51.2s 118,421 7,786 $0.0226
task_03_limiter gemini-3.8-flash-low PASSED 27.4s 91,578 0 $0.0144
task_04_refactor gemini-3.8-flash-low PASSED 40.5s 106,620 1,908 $0.0177
task_05_emitter gemini-3.8-flash-low PASSED 17.5s 71,597 0 $0.0114
task_06_semver gemini-3.8-flash-low PASSED 19.5s 73,445 0 $0.0117
task_01_kv_ttl gemini-3.8-flash-medium PASSED 57.2s 128,593 2,176 $0.0217
task_02_frontmatter gemini-3.8-flash-medium PASSED 141.2s 184,146 23,585 $0.0408
task_03_limiter gemini-3.8-flash-medium PASSED 52.8s 131,037 2,424 $0.0219
task_04_refactor gemini-3.8-flash-medium PASSED 56.1s 123,108 3,322 $0.0212
task_05_emitter gemini-3.8-flash-medium PASSED 59.3s 113,778 8,757 $0.0220
task_06_semver gemini-3.8-flash-medium PASSED 56.9s 165,295 7,295 $0.0293
task_01_kv_ttl gemini-flash-demo PASSED 26.4s 88,905 1,774 $0.0150
task_02_frontmatter gemini-flash-demo PASSED 50.0s 97,114 9,835 $0.0201
task_03_limiter gemini-flash-demo PASSED 27.5s 80,010 1,683 $0.0135
task_04_refactor gemini-flash-demo PASSED 32.5s 98,129 1,265 $0.0162
task_05_emitter gemini-flash-demo PASSED 30.4s 86,175 2,859 $0.0151
task_06_semver gemini-flash-demo PASSED 38.3s 96,689 7,420 $0.0189
task_01_kv_ttl gpt-oss-120b-medium PASSED 120.8s 95,319 0 $0.0498
task_02_frontmatter gpt-oss-120b-medium TIMEOUT 180.4s 0 0 $0.0000
task_03_limiter gpt-oss-120b-medium PASSED 180.3s 0 0 $0.0000
task_04_refactor gpt-oss-120b-medium TIMEOUT 180.4s 0 0 $0.0000
task_05_emitter gpt-oss-120b-medium PASSED 147.7s 141,844 0 $0.0739
task_06_semver gpt-oss-120b-medium TIMEOUT 180.3s 0 0 $0.0000