{
  "version": 1,
  "startedAt": "2026-09-12T20:12:10.195Z",
  "finishedAt": "2026-09-12T20:15:05.451Z",
  "repetitions": 2,
  "timeoutMs": 60000,
  "methodology": {
    "language": "English",
    "concurrency": 1,
    "temperature": 0,
    "maxTokens": 2048,
    "streaming": true,
    "retries": 0,
    "grading": "Exact JSON structure and values; optional JSON fence. All-or-nothing per task. Failed requests count as incorrect.",
    "scope": "Production Nebula API routes, including server-side routing/fallbacks. No browser rendering, chat history, tools, translation or client-side retries. Self-reported upstream model IDs are not independently verified."
  },
  "tasks": [
    {
      "id": "invoice",
      "category": "Cálculo",
      "prompt": "An invoice has 3 items at $19.50 each and 2 at $8.25 each. Apply a 10% discount to the subtotal, then 8% tax to the discounted amount. Return only JSON with numeric keys subtotal, discounted, total; round money to two decimals.",
      "expected": {
        "subtotal": 75,
        "discounted": 67.5,
        "total": 72.9
      }
    },
    {
      "id": "extract",
      "category": "Extracción",
      "prompt": "Extract this order: Order NB-204 was placed by Ana Ruiz on 2026-09-10. It contains two blue notebooks and one black pen. Delivery date is not provided. Return only JSON with keys order_id, customer, item_count (total units), delivery_date (null if unknown).",
      "expected": {
        "order_id": "NB-204",
        "customer": "Ana Ruiz",
        "item_count": 3,
        "delivery_date": null
      }
    },
    {
      "id": "schedule",
      "category": "Lógica",
      "prompt": "Four jobs A, B, C, D run one at a time. B must be before A. D must be after A. C must be before B. Return only JSON with key order containing the only valid sequence as an array of job letters.",
      "expected": {
        "order": [
          "C",
          "B",
          "A",
          "D"
        ]
      }
    },
    {
      "id": "code",
      "category": "Código",
      "prompt": "Trace this Python without running it: xs = [2, 3, 2, 5]; seen = set(); out = []; for x in xs: if x not in seen: seen.add(x); out.append(x * 2). Both seen.add and out.append are inside the if, which is inside the loop. Return only JSON with key out and the final list.",
      "expected": {
        "out": [
          4,
          6,
          10
        ]
      }
    },
    {
      "id": "evidence",
      "category": "Lectura",
      "prompt": "Use only these facts: Basic costs $9 monthly and allows 3 projects. Team costs $24 monthly and allows 12 projects. Both include export. The text says nothing about offline access. Return only JSON with keys cheapest_for_8_projects (plan name), monthly_cost (number), offline_supported (null if not established).",
      "expected": {
        "cheapest_for_8_projects": "Team",
        "monthly_cost": 24,
        "offline_supported": null
      }
    },
    {
      "id": "filter",
      "category": "Instrucciones",
      "prompt": "Records: [{\"id\":\"A\",\"active\":true,\"score\":7},{\"id\":\"B\",\"active\":false,\"score\":9},{\"id\":\"C\",\"active\":true,\"score\":9},{\"id\":\"D\",\"active\":true,\"score\":4}]. Keep active records with score at least 7. Sort by score descending. Return only JSON with key ids containing their IDs in that order.",
      "expected": {
        "ids": [
          "C",
          "A"
        ]
      }
    }
  ],
  "runs": [
    {
      "model": "Nebula Synth 5.1",
      "requestedModel": "anthropic/claude-opus-4-6",
      "task": "invoice",
      "repetition": 1,
      "timestamp": "2026-09-12T20:12:13.065Z",
      "ok": true,
      "correct": false,
      "firstTokenMs": 2015,
      "totalMs": 3329,
      "servedModel": "claude-opus-4-6",
      "answer": "\n\n```json\n{\n  \"subtotal\": 75.00,\n  \"discounted\": 67.50,\n  \"total\": 72.90\n}\n```\n\n**Breakdown:**\n\n- **Subtotal:** (3 × $19.50) + (2 × $8.25) = $58.50 + $16.50 = **$75.00**\n- **Discounted:** $75.00 × 0.90 = **$67.50**\n- **Total:** $67.50 × 1.08 = **$72.90**",
      "error": null
    },
    {
      "model": "Nebula Spark 5",
      "requestedModel": "gpt-5.6-sol",
      "task": "invoice",
      "repetition": 1,
      "timestamp": "2026-09-12T20:12:16.419Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 4160,
      "totalMs": 4368,
      "servedModel": "glm-5.3",
      "answer": "```json\n{\n  \"subtotal\": 75.00,\n  \"discounted\": 67.50,\n  \"total\": 72.90\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Dash 4.6",
      "requestedModel": "glm-5.3",
      "task": "invoice",
      "repetition": 1,
      "timestamp": "2026-09-12T20:12:20.805Z",
      "ok": false,
      "correct": false,
      "firstTokenMs": null,
      "totalMs": 126,
      "servedModel": null,
      "answer": "",
      "error": "HTTP 503"
    },
    {
      "model": "Nebula Nano-T",
      "requestedModel": "gpt-5.2-chat",
      "task": "invoice",
      "repetition": 1,
      "timestamp": "2026-09-12T20:12:20.954Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 1356,
      "totalMs": 1462,
      "servedModel": "claude-sonnet-4-6",
      "answer": "```json\n{\n  \"subtotal\": 75.00,\n  \"discounted\": 67.50,\n  \"total\": 72.90\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Spark 5",
      "requestedModel": "gpt-5.6-sol",
      "task": "extract",
      "repetition": 1,
      "timestamp": "2026-09-12T20:12:22.432Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 2894,
      "totalMs": 3168,
      "servedModel": "glm-5.3",
      "answer": "```json\n{\n  \"order_id\": \"NB-204\",\n  \"customer\": \"Ana Ruiz\",\n  \"item_count\": 3,\n  \"delivery_date\": null\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Dash 4.6",
      "requestedModel": "glm-5.3",
      "task": "extract",
      "repetition": 1,
      "timestamp": "2026-09-12T20:12:25.619Z",
      "ok": false,
      "correct": false,
      "firstTokenMs": null,
      "totalMs": 121,
      "servedModel": null,
      "answer": "",
      "error": "HTTP 503"
    },
    {
      "model": "Nebula Nano-T",
      "requestedModel": "gpt-5.2-chat",
      "task": "extract",
      "repetition": 1,
      "timestamp": "2026-09-12T20:12:25.756Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 2851,
      "totalMs": 58403,
      "servedModel": "claude-sonnet-4-6",
      "answer": "```json\n{\n  \"order_id\": \"NB-204\",\n  \"customer\": \"Ana Ruiz\",\n  \"item_count\": 3,\n  \"delivery_date\": null\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Synth 5.1",
      "requestedModel": "anthropic/claude-opus-4-6",
      "task": "extract",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:24.225Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 4258,
      "totalMs": 5922,
      "servedModel": "claude-opus-4-6",
      "answer": "\n\n```json\n{\n  \"order_id\": \"NB-204\",\n  \"customer\": \"Ana Ruiz\",\n  \"item_count\": 3,\n  \"delivery_date\": null\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Dash 4.6",
      "requestedModel": "glm-5.3",
      "task": "schedule",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:30.167Z",
      "ok": false,
      "correct": false,
      "firstTokenMs": null,
      "totalMs": 403,
      "servedModel": null,
      "answer": "",
      "error": "HTTP 503"
    },
    {
      "model": "Nebula Nano-T",
      "requestedModel": "gpt-5.2-chat",
      "task": "schedule",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:30.585Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 1632,
      "totalMs": 1661,
      "servedModel": "claude-sonnet-4-6",
      "answer": "```json\n{\"order\": [\"C\", \"B\", \"A\", \"D\"]}\n```",
      "error": null
    },
    {
      "model": "Nebula Synth 5.1",
      "requestedModel": "anthropic/claude-opus-4-6",
      "task": "schedule",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:32.266Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 1610,
      "totalMs": 1905,
      "servedModel": "claude-opus-4-6",
      "answer": "```json\n{\"order\": [\"C\", \"B\", \"A\", \"D\"]}\n```",
      "error": null
    },
    {
      "model": "Nebula Spark 5",
      "requestedModel": "gpt-5.6-sol",
      "task": "schedule",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:34.187Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 3474,
      "totalMs": 3683,
      "servedModel": "glm-5.3",
      "answer": "{\"order\":[\"C\",\"B\",\"A\",\"D\"]}",
      "error": null
    },
    {
      "model": "Nebula Nano-T",
      "requestedModel": "gpt-5.2-chat",
      "task": "code",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:37.888Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 1668,
      "totalMs": 1669,
      "servedModel": "claude-sonnet-4-6",
      "answer": "{\"out\": [4, 6, 10]}",
      "error": null
    },
    {
      "model": "Nebula Synth 5.1",
      "requestedModel": "anthropic/claude-opus-4-6",
      "task": "code",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:39.572Z",
      "ok": true,
      "correct": false,
      "firstTokenMs": 1577,
      "totalMs": 4181,
      "servedModel": "claude-opus-4-6",
      "answer": "Let me trace through the loop step by step:\n\n**Iteration 1:** `x = 2`\n- `2 not in seen` (seen = `{}`) → **True**\n- `seen.add(2)` → seen = `{2}`\n- `out.append(2 * 2)` → out = `[4]`\n\n**Iteration 2:** `x = 3`\n- `3 not in seen` (seen = `{2}`) → **True**\n- `seen.add(3)` → seen = `{2, 3}`\n- `out.append(3 * 2)` → out = `[4, 6]`\n\n**Iteration 3:** `x = 2`\n- `2 not in seen` (seen = `{2, 3}`) → **False**\n- Skip\n\n**Iteration 4:** `x = 5`\n- `5 not in seen` (seen = `{2, 3}`) → **True**\n- `seen.add(5)` → seen = `{2, 3, 5}`\n- `out.append(5 * 2)` → out = `[4, 6, 10]`\n\n```json\n{\"out\": [4, 6, 10]}\n```",
      "error": null
    },
    {
      "model": "Nebula Spark 5",
      "requestedModel": "gpt-5.6-sol",
      "task": "code",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:43.769Z",
      "ok": true,
      "correct": false,
      "firstTokenMs": 56,
      "totalMs": 60,
      "servedModel": "gpt-5.6-sol",
      "answer": "Soy Kiro, potenciado por GPT-5.6 Sol, un modelo GPT de OpenAI integrado en Nebula AI para fines educativos y de investigacion.",
      "error": null
    },
    {
      "model": "Nebula Dash 4.6",
      "requestedModel": "glm-5.3",
      "task": "code",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:43.847Z",
      "ok": false,
      "correct": false,
      "firstTokenMs": null,
      "totalMs": 1188,
      "servedModel": null,
      "answer": "",
      "error": "HTTP 503"
    },
    {
      "model": "Nebula Synth 5.1",
      "requestedModel": "anthropic/claude-opus-4-6",
      "task": "evidence",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:45.050Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 3343,
      "totalMs": 5191,
      "servedModel": "claude-opus-4-6",
      "answer": "```json\n{\n  \"cheapest_for_8_projects\": \"Team\",\n  \"monthly_cost\": 24,\n  \"offline_supported\": null\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Spark 5",
      "requestedModel": "gpt-5.6-sol",
      "task": "evidence",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:50.251Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 3816,
      "totalMs": 4186,
      "servedModel": "glm-5.3",
      "answer": "```json\n{\n  \"cheapest_for_8_projects\": \"Team\",\n  \"monthly_cost\": 24,\n  \"offline_supported\": null\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Dash 4.6",
      "requestedModel": "glm-5.3",
      "task": "evidence",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:54.447Z",
      "ok": false,
      "correct": false,
      "firstTokenMs": null,
      "totalMs": 119,
      "servedModel": null,
      "answer": "",
      "error": "HTTP 503"
    },
    {
      "model": "Nebula Nano-T",
      "requestedModel": "gpt-5.2-chat",
      "task": "evidence",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:54.582Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 1359,
      "totalMs": 1427,
      "servedModel": "claude-sonnet-4-6",
      "answer": "```json\n{\n  \"cheapest_for_8_projects\": \"Team\",\n  \"monthly_cost\": 24,\n  \"offline_supported\": null\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Spark 5",
      "requestedModel": "gpt-5.6-sol",
      "task": "filter",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:56.023Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 3779,
      "totalMs": 3845,
      "servedModel": "glm-5.3",
      "answer": "{\"ids\":[\"C\",\"A\"]}",
      "error": null
    },
    {
      "model": "Nebula Dash 4.6",
      "requestedModel": "glm-5.3",
      "task": "filter",
      "repetition": 1,
      "timestamp": "2026-09-12T20:13:59.878Z",
      "ok": false,
      "correct": false,
      "firstTokenMs": null,
      "totalMs": 122,
      "servedModel": null,
      "answer": "",
      "error": "HTTP 503"
    },
    {
      "model": "Nebula Nano-T",
      "requestedModel": "gpt-5.2-chat",
      "task": "filter",
      "repetition": 1,
      "timestamp": "2026-09-12T20:14:00.026Z",
      "ok": true,
      "correct": false,
      "firstTokenMs": 1465,
      "totalMs": 4049,
      "servedModel": "claude-sonnet-4-6",
      "answer": "Here is the result:\n\n```json\n{\"ids\":[\"C\",\"A\"]}\n```\n\n**Steps:**\n1. **Filter** active records with score ≥ 7: **A** (active, 7), **C** (active, 9) — B is excluded (inactive), D is excluded (score 4).\n2. **Sort** by score descending: C (9) → A (7).",
      "error": null
    },
    {
      "model": "Nebula Synth 5.1",
      "requestedModel": "anthropic/claude-opus-4-6",
      "task": "filter",
      "repetition": 1,
      "timestamp": "2026-09-12T20:14:04.091Z",
      "ok": true,
      "correct": false,
      "firstTokenMs": 1461,
      "totalMs": 3144,
      "servedModel": "claude-opus-4-6",
      "answer": "\n\n```json\n{\"ids\": [\"C\", \"A\"]}\n```\n\n**Reasoning:**\n\n1. **Filter active records:** A (score 7), C (score 9), D (score 4) — B is excluded (active: false).\n2. **Filter score ≥ 7:** A (score 7), C (score 9) — D is excluded (score 4).\n3. **Sort by score descending:** C (9), A (7).",
      "error": null
    },
    {
      "model": "Nebula Spark 5",
      "requestedModel": "gpt-5.6-sol",
      "task": "invoice",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:07.250Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 3695,
      "totalMs": 4063,
      "servedModel": "glm-5.3",
      "answer": "```json\n{\n  \"subtotal\": 75.00,\n  \"discounted\": 67.50,\n  \"total\": 72.90\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Dash 4.6",
      "requestedModel": "glm-5.3",
      "task": "invoice",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:11.337Z",
      "ok": false,
      "correct": false,
      "firstTokenMs": null,
      "totalMs": 122,
      "servedModel": null,
      "answer": "",
      "error": "HTTP 503"
    },
    {
      "model": "Nebula Nano-T",
      "requestedModel": "gpt-5.2-chat",
      "task": "invoice",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:11.480Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 1651,
      "totalMs": 2109,
      "servedModel": "claude-sonnet-4-6",
      "answer": "```json\n{\n  \"subtotal\": 75.00,\n  \"discounted\": 67.50,\n  \"total\": 72.90\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Synth 5.1",
      "requestedModel": "anthropic/claude-opus-4-6",
      "task": "invoice",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:13.603Z",
      "ok": true,
      "correct": false,
      "firstTokenMs": 1545,
      "totalMs": 2832,
      "servedModel": "claude-opus-4-6",
      "answer": "\n\n```json\n{\n  \"subtotal\": 75.00,\n  \"discounted\": 67.50,\n  \"total\": 72.90\n}\n```\n\n**Breakdown:**\n\n- **Subtotal:** (3 × $19.50) + (2 × $8.25) = $58.50 + $16.50 = **$75.00**\n- **Discounted:** $75.00 × 0.90 = **$67.50**\n- **Total:** $67.50 × 1.08 = **$72.90**",
      "error": null
    },
    {
      "model": "Nebula Dash 4.6",
      "requestedModel": "glm-5.3",
      "task": "extract",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:16.451Z",
      "ok": false,
      "correct": false,
      "firstTokenMs": null,
      "totalMs": 386,
      "servedModel": null,
      "answer": "",
      "error": "HTTP 503"
    },
    {
      "model": "Nebula Nano-T",
      "requestedModel": "gpt-5.2-chat",
      "task": "extract",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:16.853Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 2860,
      "totalMs": 4550,
      "servedModel": "claude-sonnet-4-6",
      "answer": "```json\n{\n  \"order_id\": \"NB-204\",\n  \"customer\": \"Ana Ruiz\",\n  \"item_count\": 3,\n  \"delivery_date\": null\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Synth 5.1",
      "requestedModel": "anthropic/claude-opus-4-6",
      "task": "extract",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:21.414Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 1780,
      "totalMs": 2044,
      "servedModel": "claude-opus-4-6",
      "answer": "\n\n```json\n{\n  \"order_id\": \"NB-204\",\n  \"customer\": \"Ana Ruiz\",\n  \"item_count\": 3,\n  \"delivery_date\": null\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Spark 5",
      "requestedModel": "gpt-5.6-sol",
      "task": "extract",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:23.469Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 2822,
      "totalMs": 3418,
      "servedModel": "glm-5.3",
      "answer": "```json\n{\n  \"order_id\": \"NB-204\",\n  \"customer\": \"Ana Ruiz\",\n  \"item_count\": 3,\n  \"delivery_date\": null\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Nano-T",
      "requestedModel": "gpt-5.2-chat",
      "task": "schedule",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:26.904Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 1381,
      "totalMs": 1401,
      "servedModel": "claude-sonnet-4-6",
      "answer": "```json\n{\"order\": [\"C\", \"B\", \"A\", \"D\"]}\n```",
      "error": null
    },
    {
      "model": "Nebula Synth 5.1",
      "requestedModel": "anthropic/claude-opus-4-6",
      "task": "schedule",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:28.321Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 1504,
      "totalMs": 1726,
      "servedModel": "claude-opus-4-6",
      "answer": "```json\n{\"order\": [\"C\", \"B\", \"A\", \"D\"]}\n```",
      "error": null
    },
    {
      "model": "Nebula Spark 5",
      "requestedModel": "gpt-5.6-sol",
      "task": "schedule",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:30.061Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 4482,
      "totalMs": 4638,
      "servedModel": "glm-5.3",
      "answer": "{\"order\":[\"C\",\"B\",\"A\",\"D\"]}",
      "error": null
    },
    {
      "model": "Nebula Dash 4.6",
      "requestedModel": "glm-5.3",
      "task": "schedule",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:34.714Z",
      "ok": false,
      "correct": false,
      "firstTokenMs": null,
      "totalMs": 402,
      "servedModel": null,
      "answer": "",
      "error": "HTTP 503"
    },
    {
      "model": "Nebula Synth 5.1",
      "requestedModel": "anthropic/claude-opus-4-6",
      "task": "code",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:35.132Z",
      "ok": true,
      "correct": false,
      "firstTokenMs": 1368,
      "totalMs": 3947,
      "servedModel": "claude-opus-4-6",
      "answer": "Let me trace through the loop step by step:\n\n**Iteration 1:** `x = 2`\n- `2 not in seen` (seen = `{}`) → **True**\n- `seen.add(2)` → seen = `{2}`\n- `out.append(2 * 2)` → out = `[4]`\n\n**Iteration 2:** `x = 3`\n- `3 not in seen` (seen = `{2}`) → **True**\n- `seen.add(3)` → seen = `{2, 3}`\n- `out.append(3 * 2)` → out = `[4, 6]`\n\n**Iteration 3:** `x = 2`\n- `2 not in seen` (seen = `{2, 3}`) → **False**\n- Skip\n\n**Iteration 4:** `x = 5`\n- `5 not in seen` (seen = `{2, 3}`) → **True**\n- `seen.add(5)` → seen = `{2, 3, 5}`\n- `out.append(5 * 2)` → out = `[4, 6, 10]`\n\n```json\n{\"out\": [4, 6, 10]}\n```",
      "error": null
    },
    {
      "model": "Nebula Spark 5",
      "requestedModel": "gpt-5.6-sol",
      "task": "code",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:39.095Z",
      "ok": true,
      "correct": false,
      "firstTokenMs": 46,
      "totalMs": 46,
      "servedModel": "gpt-5.6-sol",
      "answer": "Soy Kiro, potenciado por GPT-5.6 Sol, un modelo GPT de OpenAI integrado en Nebula AI para fines educativos y de investigacion.",
      "error": null
    },
    {
      "model": "Nebula Dash 4.6",
      "requestedModel": "glm-5.3",
      "task": "code",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:39.151Z",
      "ok": false,
      "correct": false,
      "firstTokenMs": null,
      "totalMs": 117,
      "servedModel": null,
      "answer": "",
      "error": "HTTP 503"
    },
    {
      "model": "Nebula Nano-T",
      "requestedModel": "gpt-5.2-chat",
      "task": "code",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:39.285Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 1130,
      "totalMs": 1153,
      "servedModel": "claude-sonnet-4-6",
      "answer": "{\"out\": [4, 6, 10]}",
      "error": null
    },
    {
      "model": "Nebula Spark 5",
      "requestedModel": "gpt-5.6-sol",
      "task": "evidence",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:40.453Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 2698,
      "totalMs": 3047,
      "servedModel": "glm-5.3",
      "answer": "```json\n{\n  \"cheapest_for_8_projects\": \"Team\",\n  \"monthly_cost\": 24,\n  \"offline_supported\": null\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Dash 4.6",
      "requestedModel": "glm-5.3",
      "task": "evidence",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:43.510Z",
      "ok": false,
      "correct": false,
      "firstTokenMs": null,
      "totalMs": 1179,
      "servedModel": null,
      "answer": "",
      "error": "HTTP 503"
    },
    {
      "model": "Nebula Nano-T",
      "requestedModel": "gpt-5.2-chat",
      "task": "evidence",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:44.710Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 1393,
      "totalMs": 1460,
      "servedModel": "claude-sonnet-4-6",
      "answer": "```json\n{\n  \"cheapest_for_8_projects\": \"Team\",\n  \"monthly_cost\": 24,\n  \"offline_supported\": null\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Synth 5.1",
      "requestedModel": "anthropic/claude-opus-4-6",
      "task": "evidence",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:46.180Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 3229,
      "totalMs": 5051,
      "servedModel": "claude-opus-4-6",
      "answer": "```json\n{\n  \"cheapest_for_8_projects\": \"Team\",\n  \"monthly_cost\": 24,\n  \"offline_supported\": null\n}\n```",
      "error": null
    },
    {
      "model": "Nebula Dash 4.6",
      "requestedModel": "glm-5.3",
      "task": "filter",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:51.243Z",
      "ok": false,
      "correct": false,
      "firstTokenMs": null,
      "totalMs": 1176,
      "servedModel": null,
      "answer": "",
      "error": "HTTP 503"
    },
    {
      "model": "Nebula Nano-T",
      "requestedModel": "gpt-5.2-chat",
      "task": "filter",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:52.435Z",
      "ok": true,
      "correct": false,
      "firstTokenMs": 2736,
      "totalMs": 5356,
      "servedModel": "claude-sonnet-4-6",
      "answer": "Here is the result:\n\n```json\n{\"ids\":[\"C\",\"A\"]}\n```\n\n**Steps:**\n1. **Filter** active records with score ≥ 7: **A** (active, 7), **C** (active, 9) — B is excluded (inactive), D is excluded (score 4).\n2. **Sort** by score descending: C (9) → A (7).",
      "error": null
    },
    {
      "model": "Nebula Synth 5.1",
      "requestedModel": "anthropic/claude-opus-4-6",
      "task": "filter",
      "repetition": 2,
      "timestamp": "2026-09-12T20:14:57.808Z",
      "ok": true,
      "correct": false,
      "firstTokenMs": 1441,
      "totalMs": 3200,
      "servedModel": "claude-opus-4-6",
      "answer": "\n\n```json\n{\"ids\": [\"C\", \"A\"]}\n```\n\n**Reasoning:**\n\n1. **Filter active records:** A (score 7), C (score 9), D (score 4) — B is excluded (active: false).\n2. **Filter score ≥ 7:** A (score 7), C (score 9) — D is excluded (score 4).\n3. **Sort by score descending:** C (9), A (7).",
      "error": null
    },
    {
      "model": "Nebula Spark 5",
      "requestedModel": "gpt-5.6-sol",
      "task": "filter",
      "repetition": 2,
      "timestamp": "2026-09-12T20:15:01.018Z",
      "ok": true,
      "correct": true,
      "firstTokenMs": 4016,
      "totalMs": 4420,
      "servedModel": "glm-5.3",
      "answer": "{\"ids\":[\"C\",\"A\"]}",
      "error": null
    }
  ],
  "models": [
    {
      "label": "Nebula Synth 5.1",
      "attempts": 12,
      "correct": 6,
      "successful": 12,
      "medianFirstTokenMs": 1593.5,
      "medianTotalMs": 3264.5
    },
    {
      "label": "Nebula Spark 5",
      "attempts": 12,
      "correct": 10,
      "successful": 12,
      "medianFirstTokenMs": 3584.5,
      "medianTotalMs": 3764
    },
    {
      "label": "Nebula Dash 4.6",
      "attempts": 12,
      "correct": 0,
      "successful": 0,
      "medianFirstTokenMs": null,
      "medianTotalMs": null
    },
    {
      "label": "Nebula Nano-T",
      "attempts": 12,
      "correct": 10,
      "successful": 12,
      "medianFirstTokenMs": 1548.5,
      "medianTotalMs": 1665
    }
  ]
}