diff --git a/PAUSE_SIGNAL.md b/PAUSE_SIGNAL.md new file mode 100644 index 0000000..1e26c1d --- /dev/null +++ b/PAUSE_SIGNAL.md @@ -0,0 +1,32 @@ +# Pause Signal Rework + +## Problem + +The current pause gesture is **arms-out (T-pose)**: both wrists at shoulder height. + +Issues: +- **Accidental triggers** — players raise arms during gameplay and inadvertently pause +- **Unintuitive** — new players don't know or remember to hold a T-pose to pause + +## Current Implementation + +- `armsOutDetector.ts` — checks both wrists within `ARMS_Y_TOLERANCE` (0.15) of shoulder Y +- 2-second cooldown after arms drop before it can re-trigger +- Arms-out is dual-purpose: it both **pauses gameplay** and is the **calibration-ready signal** + +## Alternative Signals + +| Gesture | Pros | Cons | +|---|---|---| +| **Hands on head** (both wrists above head) | Very intentional, hard to do by accident | Might be tiring if held | +| **Crossed arms** (wrists near opposite shoulders) | Familiar "stop" posture, unlikely during dance | Harder to detect reliably with occlusion | +| **Hand over face** (one wrist covering nose) | Simple, quick | Could conflict with wiping sweat | +| **Both hands raised high** (surrender pose) | Easy to detect, distinct from dance moves | Could overlap with jump celebrations | +| **Timeout / stillness** (no movement for N seconds) | No specific pose needed | Slow, frustrating to wait | +| **Palm facing camera** (open hand toward webcam) | Intuitive "stop" signal | Requires hand landmark model, more compute | + +## Considerations + +- Whatever replaces T-pose for pause must not conflict with the calibration flow (which also uses arms-out to signal "ready") +- Should be easy to explain in the tutorial overlay +- Must be distinct from stepping, jumping, and natural dance movement diff --git a/REFACTOR.md b/REFACTOR.md deleted file mode 100644 index 5ff7543..0000000 --- a/REFACTOR.md +++ /dev/null @@ -1,357 +0,0 @@ -# Refactor Proposal: Modular MediaPipe + Gesture Callbacks - -## Current State - -| File | Lines | Responsibility | -|------|-------|---------------| -| `gestureDetector.ts` | 511 | State machine + all gesture detection + step detection + graph sampling + logging | -| `App.tsx` | 457 | MediaPipe init, video management, animation loop, canvas drawing, gesture wiring, UI | -| `Settings.tsx` | 254 | Settings panel (fine as-is) | -| `StepGraphs.tsx` | 153 | Debug visualization (fine as-is) | -| `arScene.ts` | 64 | Three.js setup (fine as-is) | -| `types.ts` | 49 | Settings types + persistence | -| `worldToCamera.ts` | 61 | Unused (dead code) | -| `DebugGraph.tsx` | 136 | Unused (dead code) | -| `debugLog.ts` | 16 | In-memory log buffer | - -### Problems - -1. **gestureDetector.ts is a 511-line monolith** -- calibration state machine, jump detection, leg raises, palms, arms-out, step detection, graph sampling, and baseline adaptation all live in one function. - -2. **App.tsx does too much** -- MediaPipe lifecycle, webcam/file source management, the animation loop, canvas drawing, and React UI state are all in one component with 7 refs. - -3. **No callback/event system** -- gesture signals pass through mutable `GestureState` + return values. There's no way to subscribe to "a step was detected" without polling the ref every frame. - -4. **Dead code** -- `worldToCamera.ts` and `DebugGraph.tsx` are imported nowhere. - ---- - -## Proposed Architecture - -``` -src/ - types.ts # All shared types (expanded) - debugLog.ts # Unchanged - - pose/ - PoseDetector.ts # MediaPipe lifecycle (init, detect, dispose) - VideoSource.ts # Webcam/file source: attach -> loadeddata -> play, debug.webm probe - usePoseLoop.ts # Hook: owns rAF loop, calls engine.update() + arScene directly (no React in hot path) - - gesture/ - types.ts # GestureSignal, CalibrationStatus, per-detector state, callbacks - CalibrationMachine.ts # Framing -> waiting -> calibrating -> pos-center -> ready/paused - GestureTelemetry.ts # Graph/log sampling — subscribes to callbacks, collects GraphSample/StepGraphSample/StepLogEntry - detectors/ - jumpDetector.ts # Jump detection (both ankles rise) - stepDetector.ts # Step detection (toe XZ displacement + baseline adaptation) - palmsDetector.ts # Palms together (wrist distance) - armsOutDetector.ts # Arms out (wrist at shoulder height) - GestureEngine.ts # Orchestrates calibration + detectors, fires callbacks. Config via constructor + setConfig() - useGesture.ts # Hook: creates engine, exposes read-only state + callbacks to React. Owns display timer (not engine) - - ar/ - arScene.ts # Three.js setup (remove unused resizeARScene export) - - components/ - App.tsx # Slim orchestrator: wires hooks together, renders UI - Settings.tsx # Unchanged - StepGraphs.tsx # Unchanged - GestureOverlay.tsx # Gesture label + calibration hints (extracted from App). Owns 600ms display timer - TransportBar.tsx # Video playback controls (extracted from App) - PoseCanvas.tsx # Canvas drawing: landmarks, connectors, framing box, debug text -``` - ---- - -## Key Design Decisions - -### 1. Callback-based gesture events - -The core ask. Instead of polling a mutable ref, consumers register callbacks: - -```ts -// gesture/types.ts -export interface GestureCallbacks { - onGesture?: (signal: GestureSignal) => void // any gesture fires (immediate, no display timer) - onStep?: (direction: StepDirection, foot: 'left' | 'right') => void - onJump?: () => void - onPalms?: () => void - onArmsOut?: () => void - onCalibrationChange?: (status: CalibrationStatus) => void - onPause?: () => void - onResume?: () => void -} - -export type StepDirection = 'left' | 'right' | 'forward' | 'back' -``` - -The `GestureEngine` accepts callbacks at construction and fires them when gestures trigger. The `useGesture` hook exposes this to React: - -```ts -// Usage in App.tsx -const gesture = useGesture(poseResults, settings, { - onStep: (dir, foot) => console.log(`Step ${dir} with ${foot} foot`), - onJump: () => console.log('Jump!'), - onCalibrationChange: (status) => console.log(`Calibration: ${status}`), -}) -``` - -### 2. PoseDetector class wraps MediaPipe lifecycle - -Extracts the MediaPipe init/detect/dispose cycle out of App.tsx into a plain class: - -```ts -// pose/PoseDetector.ts -export class PoseDetector { - static async create(settings: PoseSettings): Promise - detectFrame(video: HTMLVideoElement, timestamp: number): PoseResult - dispose(): void -} - -export interface PoseResult { - landmarks2D: NormalizedLandmark[][] - worldLandmarks: Landmark[][] -} -``` - -### 3. VideoSource class wraps webcam/file management - -Owns the `MediaStream`, video element setup, and recording logic: - -```ts -// pose/VideoSource.ts -export class VideoSource { - static async create( - video: HTMLVideoElement, - mode: 'webcam' | 'file', - fileUrl?: string - ): Promise - get stream(): MediaStream | null - dispose(): void -} -``` - -### 4. usePoseLoop hook ties it together - -Single hook that manages the animation frame loop: - -```ts -// pose/usePoseLoop.ts -export function usePoseLoop( - videoRef: RefObject, - settings: PoseSettings, - fileUrl: string | null, - onFrame: (result: PoseResult, fps: number) => void, -): { loading: boolean; error: string | null } -``` - -### 5. Individual detectors are pure functions - -Each detector is a small, focused, testable function that owns its own state type: - -```ts -// gesture/detectors/jumpDetector.ts -export interface JumpState { wasAirborne: boolean } -export function createJumpState(): JumpState -export function detectJump( - state: JumpState, - leftRise: number, - rightRise: number, -): boolean // true = just detected -``` - -```ts -// gesture/detectors/stepDetector.ts -export interface StepState { - activeStep: GestureSignal - steppedFoot: 'left' | 'right' | null - heelBaseL: Point2D | null - heelBaseR: Point2D | null -} -export function createStepState(): StepState -export function detectStep(...): StepResult | null -export function checkStepReset(...): boolean -export function adaptBaseline(base: Point2D, current: Point2D, alpha?: number): Point2D -``` - -```ts -// gesture/detectors/palmsDetector.ts -export interface PalmsState { wasPalms: boolean } -export function createPalmsState(): PalmsState -export function detectPalms(state: PalmsState, worldLandmarks: Landmark3D[]): boolean - -// gesture/detectors/armsOutDetector.ts -export interface ArmsOutState { wasArmsOut: boolean; armsDownSince: number | null } -export function createArmsOutState(): ArmsOutState -export function detectArmsOut(state: ArmsOutState, worldLandmarks: Landmark3D[]): boolean -``` - -### 6. CalibrationMachine is a standalone state machine - -Separated from gesture detection. Owns ankle baseline collection, heel calibration samples, and foot spread measurement: - -```ts -// gesture/CalibrationMachine.ts -export interface CalibratedBaselines { - leftY: number // 2D ankle Y baseline - rightY: number - heelBaseL: Point2D // hip-relative XZ for step detection - heelBaseR: Point2D - footSpread: number // meters -} - -export class CalibrationMachine { - get status(): CalibrationStatus - get framingHint(): string | null - get baselines(): CalibratedBaselines | null - - /** Call each frame. Returns true if state changed. */ - update( - landmarks2D: NormalizedLandmark[], - worldLandmarks: Landmark3D[], - ): boolean - - reset(): void -} -``` - -### 7. GestureEngine orchestrates everything - -Replaces the current `detectGesture()` monolith. Owns the calibration machine and runs all detectors in sequence. Config passed at construction, updated via setter: - -```ts -// gesture/GestureEngine.ts -export class GestureEngine { - constructor(config: StepConfig, callbacks?: GestureCallbacks) - - /** Call each frame with latest pose data. Returns current signal for display. */ - update( - landmarks2D: NormalizedLandmark[], - worldLandmarks: Landmark3D[], - ): GestureSignal - - /** Read-only access for synchronous canvas drawing (e.g. calibrationStatus for red box) */ - get state(): Readonly - - setConfig(config: StepConfig): void - setCallbacks(callbacks: GestureCallbacks): void -} -``` - ---- - -## What Stays the Same - -- All detection algorithms (thresholds, hysteresis, baseline adaptation) -- Calibration flow (framing -> waiting -> calibrating -> pos-center -> ready/paused) -- Settings panel and persistence -- Three.js AR overlay -- Step graph visualization -- Debug logging -- Recording functionality - ---- - -## What Gets Deleted - -- `worldToCamera.ts` -- unused dead code -- `DebugGraph.tsx` -- unused dead code - ---- - -## Migration Order - -The refactor is incremental — app stays working at each step: - -1. **Delete dead code** -- `worldToCamera.ts`, `DebugGraph.tsx`, unused `resizeARScene` export, `left-leg`/`right-leg` from `GestureSignal` type + App.tsx UI references, `wasLeftRaised`/`wasRightRaised` from `GestureState` -2. **Extract types** -- move gesture types from `gestureDetector.ts` to `gesture/types.ts`, add `GestureCallbacks` interface, per-detector state types -3. **Extract individual detectors** -- pull `jumpDetector`, `stepDetector`, `palmsDetector`, `armsOutDetector` out as pure functions with own state types -4. **Extract CalibrationMachine** -- pull calibration state logic + baseline collection into its own module -5. **Build GestureEngine as thin wrapper** -- initially delegates to existing `detectGesture()` to keep things working, then incrementally migrate logic to use calibration machine + detectors + callback dispatch -6. **Extract GestureTelemetry** -- move graph/log sampling out of engine, subscribe to callbacks -7. **Move display timer to React** -- remove `signalShowUntil` from engine, let `useGesture`/`GestureOverlay` own the 600ms display timer -8. **Extract PoseDetector** -- wrap MediaPipe init/detect/dispose -9. **Extract VideoSource** -- wrap webcam/file source management including attach -> loadeddata -> play lifecycle and debug.webm auto-probe -10. **Build usePoseLoop hook** -- owns rAF loop, calls `gestureEngine.update()` + `arScene.update()` directly (no React in hot path), batched setState for UI -11. **Build useGesture hook** -- wraps GestureEngine for React, exposes read-only state + callbacks -12. **Extract UI components** -- `GestureOverlay`, `TransportBar`, `PoseCanvas` -13. **Slim down App.tsx** -- wire hooks and components together - ---- - -## Expected Result - -| Before | After | -|--------|-------| -| `App.tsx` 457 lines | `App.tsx` ~100 lines | -| `gestureDetector.ts` 511 lines | `GestureEngine.ts` ~80 lines + 4 detectors ~40-60 lines each + `CalibrationMachine.ts` ~120 lines + `GestureTelemetry.ts` ~60 lines | -| No callback system | `GestureCallbacks` interface with per-gesture hooks | -| 2 dead files | Deleted | -| All logic in 2 big files | 15+ small, focused modules | - ---- - -## Review Notes - -### Issues Found - -**1. Leg raise detection doesn't exist yet — nothing to extract** - -The proposal lists `legRaiseDetector.ts` as something to extract, but `emit('left-leg')` / `emit('right-leg')` is never called in the current code. The `wasLeftRaised`/`wasRightRaised` state fields are declared but never written to. The `leftRaised`/`rightRaised` variables are computed (line 406-407) but only used to gate baseline adaptation (lines 503-508), not to fire a gesture signal. The type union and App.tsx UI reference `left-leg`/`right-leg`, so the intent was there, but the implementation was never finished. The refactor should either implement it or remove the dead types/UI — not list it as an extraction. - -**2. Graph/log sampling has no home** - -`GraphSample`, `StepGraphSample`, and `StepLogEntry` collection is ~40% of the current `detectGesture()` code. The proposal doesn't say which module owns this. It can't live in the individual detectors (they're supposed to be pure). It shouldn't live in `GestureEngine` (that's orchestration, not observability). Suggest a dedicated `gesture/GestureTelemetry.ts` that subscribes to the callback system, or have `GestureEngine` expose a `samples` getter and do collection internally as a separate concern. - -**3. `signalShowUntil` mixes UI and detection** - -The current code gates all detection behind `if (now < state.signalShowUntil) return state.activeSignal` — a 600ms display timer that suppresses new gestures. This is a UI concern (how long to show a label) leaking into detection logic. The proposal doesn't address where this lives. Suggestion: remove it from the engine entirely. Let `GestureEngine` fire callbacks immediately and let the React layer (`useGesture` or `GestureOverlay`) own the display timer. - -**4. `arScene.ts` has dead code too** - -The proposal says arScene.ts is "fine as-is", but `resizeARScene` (line 49) is exported and never called. Minor, but should be included in the dead code cleanup. - -**5. `stepConfig` passed per-frame is noisy** - -`GestureEngine.update()` takes `stepConfig` every frame, but it only changes when the user edits settings. Pass it at construction and add a `setConfig()` method instead. Same applies to any other settings the engine needs. - -**6. `VideoSource` API is too thin** - -The proposed `VideoSource.create(video, mode, fileUrl?)` doesn't account for: -- The `loadeddata` promise needed before `video.play()` -- Auto-probing `/debug.webm` on startup -- Calling `video.play()` after source attachment - -The real lifecycle is: attach source → wait for loadeddata → play. `VideoSource` needs to own all three steps or the caller ends up reimplementing the complexity. - -**7. Data flow between hooks is undefined** - -`usePoseLoop` emits `(result, fps)` via `onFrame`. `useGesture` takes `poseResults`. How do they connect? If `onFrame` calls into `useGesture`, then `useGesture` can't take `poseResults` as a prop — it needs to be a callback target. If App.tsx wires them via state, you get a frame of latency and re-render per frame. The proposal should specify: does `usePoseLoop` call `gestureEngine.update()` directly inside the animation loop, or does React mediate? The former is better for performance (no React in the hot path). - -**8. Step 5 in migration is a risky big-bang swap** - -"Build GestureEngine — compose calibration machine + detectors + callback dispatch, replacing `detectGesture()`" replaces the working monolith in one shot. Safer: first build `GestureEngine` as a thin wrapper that delegates to the existing `detectGesture()`, then incrementally migrate logic out of it. - -### Suggestions - -**A. Split `GestureState` per-detector** - -The current 20+ field bag makes it hard to reason about what state belongs to which detector. Each detector should own its state: - -```ts -interface JumpState { wasAirborne: boolean } -interface StepState { activeStep: GestureSignal; steppedFoot: 'left' | 'right' | null; heelBaseL: Point2D | null; ... } -interface PalmsState { wasPalms: boolean } -// etc. -``` - -`GestureEngine` composes these. This makes the state boundaries explicit and each detector truly self-contained. - -**B. Decide on frame-loop ownership** - -The cleanest architecture: `usePoseLoop` owns the `requestAnimationFrame` loop and calls `gestureEngine.update()` + `arScene.update()` directly — no React state in the hot path. React only gets notified for UI updates (gesture label, calibration status) via batched setState. The `onFrame` callback is for the canvas drawing, not for gesture detection. - -**C. Callbacks as the primary interface** - -Callbacks are the right call — they make the API push-based and decouple consumers from engine internals. Just make sure the engine still exposes a read-only `state` getter for the few cases where the animation loop needs synchronous access (e.g. canvas drawing checks `calibrationStatus` to decide whether to draw the red box). Callbacks notify, state answers "what's true right now". diff --git a/src/DebugApp.tsx b/src/DebugApp.tsx index 1d3cda7..0976c72 100644 --- a/src/DebugApp.tsx +++ b/src/DebugApp.tsx @@ -66,7 +66,7 @@ export function DebugApp() { const { loading, error, fps, gesture, calibrationStatus, framingHint, - stepGraphSamples, worldLandmarks, telemetry, + stepGraphSamples, landmarks2D, worldLandmarks, telemetry, } = usePoseLoop(videoRef, canvasRef, arCanvasRef, settings, null, { onStep: (direction) => arrowGameRef.current?.handleStep(direction), }) @@ -144,7 +144,7 @@ export function DebugApp() { thresholdZ={settings.stepThresholdZ} reset={settings.stepReset} /> - +