# Hermes-Xiaozhi Voice Companion — Project

## Purpose
Build the **Hermes-Xiaozhi Bridge**: a small glue layer that turns a Xiaozhi ESP32
into a Thai voice companion backed by local Hermes + Qwen 3.8 (V100) and local
Thai STT/TTS (RTX 5060 Ti). The bridge does NOT reimplement LLM/STT/TTS/agent
stacks — it adapts the Xiaozhi WebSocket protocol to internal services.

Owner intent source: `hermes-xiaozhi-voice-companion-plan.md` (attached).

## Required outcomes
- REQ-001 — Bridge accepts a Xiaozhi device over WebSocket (`/ws/xiaozhi`),
  completes the documented handshake, and holds a live conversational session.
- REQ-002 — Audio in: Opus → PCM, with end-of-speech detection (VAD).
- REQ-003 — Thai STT behind an `STTEngine` interface (Typhoon primary,
  faster-whisper fallback). No model hard-coded.
- REQ-004 — Hermes voice profile: reasoning **none**, tools = `session_search`
  only, filesystem/shell/code-exec disabled. Read-oriented, not a worker.
- REQ-005 — Multi-turn voice session with bounded working context (~8k–16k).
- REQ-006 — Thai TTS behind a `TTSEngine` interface (JaiTTS primary),
  Thai-aware chunking, streaming (first audio before full answer).
- REQ-007 — Barge-in: user speech interrupts AI speech (cancel TTS task + queue).
- REQ-008 — Cross-session recall via Hermes `session_search` (no new vector DB).
- REQ-009 — Device authentication (device_id + token_hash, allowlist) before any
  session. Hermes/Qwen/vLLM never exposed to the internet directly.
- REQ-010 — GPU resource manager with manual modes: `voice-full`,
  `voice-lite`, `voice-off` (auto later).
- REQ-011 — Real latency logging; headline metric = End-of-Speech → First
  Audible Audio.
- REQ-012 — Cloudflare-friendly: bridge binds LAN/localhost; tunnel is external
  (no Cloudflare-specific logic in the bridge).

## Constraints
- CON-001 — Python. STT/TTS ecosystem is Python-first.
- CON-002 — All backends (STT/TTS/GPU/opus) are lazy/optional imports: the
  bridge must import and run (and be tested) on a machine with **no** GPU,
  no device, and no native opus. Heavy deps load only when selected.
- CON-003 — No new LLM server, vector DB, memory framework, agent framework,
  audio protocol, Hermes fork, firmware rewrite, or STT/TTS framework.
- CON-004 — Protocol/audio constants are **PoC assumptions** until verified
  against the real Xiaozhi firmware repo. They are config-driven, not magic.

## Prohibitions
- MUST-NOT-001 — Do not hard-code an STT/TTS model name in the bridge code path.
- MUST-NOT-002 — Do not expose Hermes/Qwen/vLLM HTTP APIs to the internet.
- MUST-NOT-003 — Do not let the Voice profile write files, run shell, or execute
  code (read/recall only).

## Acceptance criteria
- AC-001 — `pytest` passes on a machine with no GPU/device/native-libs
  (mock backends). This is the CI-verifiable bar on this host.
- AC-002 — `python -m app.main` imports and starts a FastAPI app exposing
  `/health` and `/ws/xiaozhi` (device handshake reachable).
- AC-003 — Handshake, auth, chunker, latency metrics, VAD end-of-speech,
  TTS queue barge-in cancel, and GPU mode selection are each unit-tested.
- AC-004 — Voice profile asserts reasoning==none and tools=={session_search}.
- AC-005 — A full "spoken turn" runs end-to-end against mock STT/LLM/TTS in a test
  (speech → text → answer text → TTS chunks → audio out) with barge-in cancel.

## Governing documents
- `plan.md` — execution plan & status (authoritative operational state).
- The attached plan (`hermes-xiaozhi-voice-companion-plan.md`) — requirements source.

## Decisions
- DEC-001 — Repo root = `/Users/kunthawat/Gitea/hermes-xiaozhi-bridge/`, Python 3.14,
  venv at `.venv/`.
- DEC-002 — Protocol layer is a documented PoC (config-driven). Mark every
  firmware-dependent constant with `# FIRMWARE:` so Phase 0 verification is a
  grep, not an archaeology dig.
- DEC-003 — Provide `Mock*` backends (STT/LLM/TTS/GPU/opus) wired via config
  `engines.*: mock` so the whole pipeline is testable on a CPU-only Mac.

## Open questions
- OQ-001 — Exact Xiaozhi firmware audio framing & hello schema (verify in firmware repo).
- OQ-002 — Hermes voice-turn API surface (local OpenAI-compatible HTTP? CLI? internal?) —
  determines `app/hermes/client.py` transport.
- OQ-003 — Typhoon ASR & JaiTTS serving endpoints on the voice-server.

## Change log
- 2026-10-03 — Project initiated from attached plan. REQ/AC/CON set above.

## CON-003 OVERRIDE (2026-10-03, owner directive)
- Owner: build our own xiaozhi v2.5.0 firmware via ESP-IDF v6.1, configure before flash.
- Rationale: factory bin bakes OTA_URL=api.tenclass.net + LANGUAGE_ZH_CN; device can't reach our bridge / Thai.
- Build config: BOARD_TYPE_SPOTPEAR_ESP32_S3_1_54_MUMA, OTA_URL=https://zhi.moreminimore.com/ota, LANGUAGE_TH_TH.
- NOT a source rewrite — stock xiaozhi source, custom Kconfig defaults (OTA server + language only).
- New: bridge must serve OTA endpoint (/ota) + firmware-protocol WS auth (Authorization Bearer + Device-Id header).