Dashboard Console Reconnection Bug Analysis

Details

Raw Data

{
  "id": "6dac8b23-6784-41f8-84cc-3bd280d6364c",
  "type": "general",
  "status": "completed",
  "title": "Dashboard Console Reconnection Bug Analysis",
  "description": "Investigating RECONNECTING stuck state and 503 errors after turns",
  "context": {
    "output": "<div style=\"background:#1e1e1e;color:#e0e0e0;padding:20px;border-radius:8px;font-family:'Segoe UI','Roboto',sans-serif;\"><h2 style=\"color:#fff;margin:0 0 16px 0;\">🔍 Dashboard Console Reconnection Bug Analysis</h2><div style=\"background:#3b1e1e;padding:16px;border-radius:6px;margin-bottom:16px;border-left:4px solid #ef4444;\"><h3 style=\"margin:0 0 8px 0;color:#ef4444;\">Problem Summary</h3><ul style=\"margin:0;padding-left:20px;line-height:1.8;\"><li>After significant turns complete, UI gets stuck at \"RECONNECTING\"</li><li>Server requires restart to recover</li><li>503 errors occur for 1-2 minutes after restart</li><li>Started happening yesterday (Jan 5, 2026)</li></ul></div><div style=\"background:#2d2d2d;padding:16px;border-radius:6px;margin-bottom:16px;\"><h3 style=\"margin:0 0 12px 0;color:#61afef;\">📊 What Changed Yesterday</h3><p style=\"color:#9ca3af;margin:0 0 12px 0;\">Commits on Jan 5 (yesterday):</p><table style=\"width:100%;border-collapse:collapse;font-size:0.9rem;\"><tr style=\"background:#3d3d3d;\"><td style=\"padding:8px;color:#98c379;font-family:monospace;\">dfb1d73</td><td style=\"padding:8px;\">Hot Sessions Documentation</td></tr><tr><td style=\"padding:8px;color:#98c379;font-family:monospace;\">53bf53e</td><td style=\"padding:8px;\">Quick Turn: Prompt caching</td></tr><tr style=\"background:#3d3d3d;\"><td style=\"padding:8px;color:#98c379;font-family:monospace;\">d31001a</td><td style=\"padding:8px;\">Quick Turn Phase 3: UX improvements</td></tr><tr><td style=\"padding:8px;color:#e5c07b;font-family:monospace;\">eb3abf8</td><td style=\"padding:8px;\"><strong>Quick Turn Phase 2: Multi-session + pool</strong> ⚠️</td></tr></table></div><div style=\"background:#2d2d2d;padding:16px;border-radius:6px;margin-bottom:16px;\"><h3 style=\"margin:0 0 12px 0;color:#e5c07b;\">⚠️ Likely Root Cause: Hot Session Pool</h3><p style=\"color:#e0e0e0;margin:0 0 12px 0;\">The Hot Session architecture (eb3abf8) introduced several new components:</p><ol style=\"margin:0;padding-left:24px;line-height:2;\"><li><strong>HotClaudeSession</strong> - Persistent Claude processes that stay alive between prompts</li><li><strong>Session Pool</strong> - Pre-warmed sessions for instant responses</li><li><strong>stdin/stdout streaming</strong> - Different protocol than one-shot spawning</li></ol><p style=\"color:#9ca3af;margin:16px 0 0 0;\">The issue appears to be that when a hot session completes a turn, the SSE stream may not be properly signaling completion to the client, causing the \"RECONNECTING\" state.</p></div><div style=\"background:#2d2d2d;padding:16px;border-radius:6px;margin-bottom:16px;\"><h3 style=\"margin:0 0 12px 0;color:#98c379;\">🔬 Architecture Analysis</h3><pre style=\"background:#1e1e1e;padding:12px;border-radius:4px;overflow-x:auto;font-size:0.85rem;color:#e0e0e0;\">Client (konui) → POST /api/konsole/dashboard/stream\n                     ↓\n              konui/router.ts → konsole /sessions/{id}/hot-prompt\n                                     ↓\n                            konsole/stream.ts → processHotPrompt()\n                                     ↓\n                            hot-claude.ts → HotClaudeSession\n                                     ↓\n                            Claude CLI (--input-format stream-json)\n</pre><p style=\"color:#9ca3af;margin:12px 0 0 0;\">Key observation from screenshot #3: The turn <strong>did complete successfully</strong> (shows \"Done\", \"388.4s total\", \"37 tools\") but UI still shows \"RECONNECTING\".</p></div><div style=\"background:#2d2d2d;padding:16px;border-radius:6px;margin-bottom:16px;\"><h3 style=\"margin:0 0 12px 0;color:#c678dd;\">🎯 Suspected Issues</h3><ol style=\"margin:0;padding-left:24px;line-height:2;\"><li><strong>SSE stream not closing properly</strong> - The <code>controller.close()</code> in finally block may not be propagating</li><li><strong>EventSource reconnect loop</strong> - When POST stream ends, EventSource reconnection may be stuck</li><li><strong>503 on startup</strong> - Hot pool replenishment on restart may block health checks</li><li><strong>dashboardStreams.complete()</strong> may not be called in all exit paths</li></ol></div><div style=\"background:#2d2d2d;padding:16px;border-radius:6px;margin-bottom:16px;\"><h3 style=\"margin:0 0 12px 0;color:#56b6c2;\">📋 Investigation Steps</h3><ol style=\"margin:0;padding-left:24px;line-height:2;\"><li>Check if <code>dashboardStreams.complete(requestId)</code> is called after hot prompt finishes</li><li>Review EventSource reconnection logic in console-streaming.ts</li><li>Add logging to hot-claude.ts <code>pendingResponse</code> resolution</li><li>Check if hot session pool initialization blocks health endpoint</li><li>Test with <code>KONSOLE_HOT_POOL_SIZE=0</code> to disable pool</li></ol></div><div style=\"background:#1e3a5f;padding:16px;border-radius:6px;\"><h4 style=\"margin:0 0 8px 0;color:#fff;\">Quick Fix to Try</h4><p style=\"color:#e0e0e0;margin:0;\">Set <code>KONSOLE_HOT_POOL_SIZE=0</code> in <code>/konnectvol/konsole/.env</code> and restart konsole. This disables the hot session pool and uses classic one-shot spawning which was stable before.</p></div></div>",
    "requestedAt": "2026-01-06T04:21:00.000Z",
    "requestId": "e9b14cdb-0563-4942-a904-e0efd3349e15",
    "choices": [
      {
        "label": "Disable hot pool",
        "value": "Set KONSOLE_HOT_POOL_SIZE=0 in /konnectvol/konsole/.env and restart konsole to test if that fixes the reconnection issue",
        "primary": true
      },
      {
        "label": "Investigate code",
        "value": "Investigate the hot-claude.ts and console-streaming.ts code to find the exact bug causing RECONNECTING to get stuck"
      },
      {
        "label": "Add logging",
        "value": "Add debug logging to track SSE stream completion and EventSource reconnection events"
      },
      {
        "label": "Check logs",
        "value": "Check the konsole and konui logs for errors around the time of the RECONNECTING issue"
      }
    ],
    "turnTiming": {
      "totalMs": 159286,
      "ttfeMs": 55,
      "thinkingMs": 159231,
      "toolExecutionMs": 0,
      "toolCallCount": 0,
      "thinkingPct": 100,
      "toolsPct": 0,
      "ttfePct": 0
    }
  },
  "createdBy": "claude",
  "createdAt": "2026-01-05T14:23:45.463Z",
  "updatedAt": "2026-01-05T14:23:52.167Z",
  "requestId": "e9b14cdb-0563-4942-a904-e0efd3349e15",
  "scope": "vibetools",
  "tags": [
    "konsole",
    "konui",
    "bug",
    "sse",
    "performance"
  ],
  "targetUser": "claude"
}