16 KiB

Raw Blame History

Refactor Proposal: Modular MediaPipe + Gesture Callbacks

Current State

File	Lines	Responsibility
`gestureDetector.ts`	511	State machine + all gesture detection + step detection + graph sampling + logging
`App.tsx`	457	MediaPipe init, video management, animation loop, canvas drawing, gesture wiring, UI
`Settings.tsx`	254	Settings panel (fine as-is)
`StepGraphs.tsx`	153	Debug visualization (fine as-is)
`arScene.ts`	64	Three.js setup (fine as-is)
`types.ts`	49	Settings types + persistence
`worldToCamera.ts`	61	Unused (dead code)
`DebugGraph.tsx`	136	Unused (dead code)
`debugLog.ts`	16	In-memory log buffer

Problems

gestureDetector.ts is a 511-line monolith -- calibration state machine, jump detection, leg raises, palms, arms-out, step detection, graph sampling, and baseline adaptation all live in one function.
App.tsx does too much -- MediaPipe lifecycle, webcam/file source management, the animation loop, canvas drawing, and React UI state are all in one component with 7 refs.
No callback/event system -- gesture signals pass through mutable GestureState + return values. There's no way to subscribe to "a step was detected" without polling the ref every frame.
Dead code -- worldToCamera.ts and DebugGraph.tsx are imported nowhere.

Proposed Architecture

src/
  types.ts                    # All shared types (expanded)
  debugLog.ts                 # Unchanged

  pose/
    PoseDetector.ts           # MediaPipe lifecycle (init, detect, dispose)
    VideoSource.ts            # Webcam/file source: attach -> loadeddata -> play, debug.webm probe
    usePoseLoop.ts            # Hook: owns rAF loop, calls engine.update() + arScene directly (no React in hot path)

  gesture/
    types.ts                  # GestureSignal, CalibrationStatus, per-detector state, callbacks
    CalibrationMachine.ts     # Framing -> waiting -> calibrating -> pos-center -> ready/paused
    GestureTelemetry.ts       # Graph/log sampling — subscribes to callbacks, collects GraphSample/StepGraphSample/StepLogEntry
    detectors/
      jumpDetector.ts         # Jump detection (both ankles rise)
      stepDetector.ts         # Step detection (toe XZ displacement + baseline adaptation)
      palmsDetector.ts        # Palms together (wrist distance)
      armsOutDetector.ts      # Arms out (wrist at shoulder height)
    GestureEngine.ts          # Orchestrates calibration + detectors, fires callbacks. Config via constructor + setConfig()
    useGesture.ts             # Hook: creates engine, exposes read-only state + callbacks to React. Owns display timer (not engine)

  ar/
    arScene.ts                # Three.js setup (remove unused resizeARScene export)

  components/
    App.tsx                   # Slim orchestrator: wires hooks together, renders UI
    Settings.tsx              # Unchanged
    StepGraphs.tsx            # Unchanged
    GestureOverlay.tsx        # Gesture label + calibration hints (extracted from App). Owns 600ms display timer
    TransportBar.tsx          # Video playback controls (extracted from App)
    PoseCanvas.tsx            # Canvas drawing: landmarks, connectors, framing box, debug text

Key Design Decisions

1. Callback-based gesture events

The core ask. Instead of polling a mutable ref, consumers register callbacks:

// gesture/types.ts
export interface GestureCallbacks {
  onGesture?: (signal: GestureSignal) => void        // any gesture fires (immediate, no display timer)
  onStep?: (direction: StepDirection, foot: 'left' | 'right') => void
  onJump?: () => void
  onPalms?: () => void
  onArmsOut?: () => void
  onCalibrationChange?: (status: CalibrationStatus) => void
  onPause?: () => void
  onResume?: () => void
}

export type StepDirection = 'left' | 'right' | 'forward' | 'back'

The GestureEngine accepts callbacks at construction and fires them when gestures trigger. The useGesture hook exposes this to React:

// Usage in App.tsx
const gesture = useGesture(poseResults, settings, {
  onStep: (dir, foot) => console.log(`Step ${dir} with ${foot} foot`),
  onJump: () => console.log('Jump!'),
  onCalibrationChange: (status) => console.log(`Calibration: ${status}`),
})

2. PoseDetector class wraps MediaPipe lifecycle

Extracts the MediaPipe init/detect/dispose cycle out of App.tsx into a plain class:

// pose/PoseDetector.ts
export class PoseDetector {
  static async create(settings: PoseSettings): Promise<PoseDetector>
  detectFrame(video: HTMLVideoElement, timestamp: number): PoseResult
  dispose(): void
}

export interface PoseResult {
  landmarks2D: NormalizedLandmark[][]
  worldLandmarks: Landmark[][]
}

3. VideoSource class wraps webcam/file management

Owns the MediaStream, video element setup, and recording logic:

// pose/VideoSource.ts
export class VideoSource {
  static async create(
    video: HTMLVideoElement,
    mode: 'webcam' | 'file',
    fileUrl?: string
  ): Promise<VideoSource>
  get stream(): MediaStream | null
  dispose(): void
}

4. usePoseLoop hook ties it together

Single hook that manages the animation frame loop:

// pose/usePoseLoop.ts
export function usePoseLoop(
  videoRef: RefObject<HTMLVideoElement>,
  settings: PoseSettings,
  fileUrl: string | null,
  onFrame: (result: PoseResult, fps: number) => void,
): { loading: boolean; error: string | null }

5. Individual detectors are pure functions

Each detector is a small, focused, testable function that owns its own state type:

// gesture/detectors/jumpDetector.ts
export interface JumpState { wasAirborne: boolean }
export function createJumpState(): JumpState
export function detectJump(
  state: JumpState,
  leftRise: number,
  rightRise: number,
): boolean  // true = just detected

// gesture/detectors/stepDetector.ts
export interface StepState {
  activeStep: GestureSignal
  steppedFoot: 'left' | 'right' | null
  heelBaseL: Point2D | null
  heelBaseR: Point2D | null
}
export function createStepState(): StepState
export function detectStep(...): StepResult | null
export function checkStepReset(...): boolean
export function adaptBaseline(base: Point2D, current: Point2D, alpha?: number): Point2D

// gesture/detectors/palmsDetector.ts
export interface PalmsState { wasPalms: boolean }
export function createPalmsState(): PalmsState
export function detectPalms(state: PalmsState, worldLandmarks: Landmark3D[]): boolean

// gesture/detectors/armsOutDetector.ts
export interface ArmsOutState { wasArmsOut: boolean; armsDownSince: number | null }
export function createArmsOutState(): ArmsOutState
export function detectArmsOut(state: ArmsOutState, worldLandmarks: Landmark3D[]): boolean

6. CalibrationMachine is a standalone state machine

Separated from gesture detection. Owns ankle baseline collection, heel calibration samples, and foot spread measurement:

// gesture/CalibrationMachine.ts
export interface CalibratedBaselines {
  leftY: number              // 2D ankle Y baseline
  rightY: number
  heelBaseL: Point2D         // hip-relative XZ for step detection
  heelBaseR: Point2D
  footSpread: number         // meters
}

export class CalibrationMachine {
  get status(): CalibrationStatus
  get framingHint(): string | null
  get baselines(): CalibratedBaselines | null

  /** Call each frame. Returns true if state changed. */
  update(
    landmarks2D: NormalizedLandmark[],
    worldLandmarks: Landmark3D[],
  ): boolean

  reset(): void
}

7. GestureEngine orchestrates everything

Replaces the current detectGesture() monolith. Owns the calibration machine and runs all detectors in sequence. Config passed at construction, updated via setter:

// gesture/GestureEngine.ts
export class GestureEngine {
  constructor(config: StepConfig, callbacks?: GestureCallbacks)

  /** Call each frame with latest pose data. Returns current signal for display. */
  update(
    landmarks2D: NormalizedLandmark[],
    worldLandmarks: Landmark3D[],
  ): GestureSignal

  /** Read-only access for synchronous canvas drawing (e.g. calibrationStatus for red box) */
  get state(): Readonly<GestureState>

  setConfig(config: StepConfig): void
  setCallbacks(callbacks: GestureCallbacks): void
}

What Stays the Same

All detection algorithms (thresholds, hysteresis, baseline adaptation)
Calibration flow (framing -> waiting -> calibrating -> pos-center -> ready/paused)
Settings panel and persistence
Three.js AR overlay
Step graph visualization
Debug logging
Recording functionality

What Gets Deleted

worldToCamera.ts -- unused dead code
DebugGraph.tsx -- unused dead code

Migration Order

The refactor is incremental — app stays working at each step:

Delete dead code -- worldToCamera.ts, DebugGraph.tsx, unused resizeARScene export, left-leg/right-leg from GestureSignal type + App.tsx UI references, wasLeftRaised/wasRightRaised from GestureState
Extract types -- move gesture types from gestureDetector.ts to gesture/types.ts, add GestureCallbacks interface, per-detector state types
Extract individual detectors -- pull jumpDetector, stepDetector, palmsDetector, armsOutDetector out as pure functions with own state types
Extract CalibrationMachine -- pull calibration state logic + baseline collection into its own module
Build GestureEngine as thin wrapper -- initially delegates to existing detectGesture() to keep things working, then incrementally migrate logic to use calibration machine + detectors + callback dispatch
Extract GestureTelemetry -- move graph/log sampling out of engine, subscribe to callbacks
Move display timer to React -- remove signalShowUntil from engine, let useGesture/GestureOverlay own the 600ms display timer
Extract PoseDetector -- wrap MediaPipe init/detect/dispose
Extract VideoSource -- wrap webcam/file source management including attach -> loadeddata -> play lifecycle and debug.webm auto-probe
Build usePoseLoop hook -- owns rAF loop, calls gestureEngine.update() + arScene.update() directly (no React in hot path), batched setState for UI
Build useGesture hook -- wraps GestureEngine for React, exposes read-only state + callbacks
Extract UI components -- GestureOverlay, TransportBar, PoseCanvas
Slim down App.tsx -- wire hooks and components together

Expected Result

Before	After
`App.tsx` 457 lines	`App.tsx` ~100 lines
`gestureDetector.ts` 511 lines	`GestureEngine.ts` ~80 lines + 4 detectors ~40-60 lines each + `CalibrationMachine.ts` ~120 lines + `GestureTelemetry.ts` ~60 lines
No callback system	`GestureCallbacks` interface with per-gesture hooks
2 dead files	Deleted
All logic in 2 big files	15+ small, focused modules

Review Notes

Issues Found

1. Leg raise detection doesn't exist yet — nothing to extract

The proposal lists legRaiseDetector.ts as something to extract, but emit('left-leg') / emit('right-leg') is never called in the current code. The wasLeftRaised/wasRightRaised state fields are declared but never written to. The leftRaised/rightRaised variables are computed (line 406-407) but only used to gate baseline adaptation (lines 503-508), not to fire a gesture signal. The type union and App.tsx UI reference left-leg/right-leg, so the intent was there, but the implementation was never finished. The refactor should either implement it or remove the dead types/UI — not list it as an extraction.

2. Graph/log sampling has no home

GraphSample, StepGraphSample, and StepLogEntry collection is ~40% of the current detectGesture() code. The proposal doesn't say which module owns this. It can't live in the individual detectors (they're supposed to be pure). It shouldn't live in GestureEngine (that's orchestration, not observability). Suggest a dedicated gesture/GestureTelemetry.ts that subscribes to the callback system, or have GestureEngine expose a samples getter and do collection internally as a separate concern.

3. signalShowUntil mixes UI and detection

The current code gates all detection behind if (now < state.signalShowUntil) return state.activeSignal — a 600ms display timer that suppresses new gestures. This is a UI concern (how long to show a label) leaking into detection logic. The proposal doesn't address where this lives. Suggestion: remove it from the engine entirely. Let GestureEngine fire callbacks immediately and let the React layer (useGesture or GestureOverlay) own the display timer.

4. arScene.ts has dead code too

The proposal says arScene.ts is "fine as-is", but resizeARScene (line 49) is exported and never called. Minor, but should be included in the dead code cleanup.

5. stepConfig passed per-frame is noisy

GestureEngine.update() takes stepConfig every frame, but it only changes when the user edits settings. Pass it at construction and add a setConfig() method instead. Same applies to any other settings the engine needs.

6. VideoSource API is too thin

The proposed VideoSource.create(video, mode, fileUrl?) doesn't account for:

The loadeddata promise needed before video.play()
Auto-probing /debug.webm on startup
Calling video.play() after source attachment

The real lifecycle is: attach source → wait for loadeddata → play. VideoSource needs to own all three steps or the caller ends up reimplementing the complexity.

7. Data flow between hooks is undefined

usePoseLoop emits (result, fps) via onFrame. useGesture takes poseResults. How do they connect? If onFrame calls into useGesture, then useGesture can't take poseResults as a prop — it needs to be a callback target. If App.tsx wires them via state, you get a frame of latency and re-render per frame. The proposal should specify: does usePoseLoop call gestureEngine.update() directly inside the animation loop, or does React mediate? The former is better for performance (no React in the hot path).

8. Step 5 in migration is a risky big-bang swap

"Build GestureEngine — compose calibration machine + detectors + callback dispatch, replacing detectGesture()" replaces the working monolith in one shot. Safer: first build GestureEngine as a thin wrapper that delegates to the existing detectGesture(), then incrementally migrate logic out of it.

Suggestions

A. Split GestureState per-detector

The current 20+ field bag makes it hard to reason about what state belongs to which detector. Each detector should own its state:

interface JumpState { wasAirborne: boolean }
interface StepState { activeStep: GestureSignal; steppedFoot: 'left' | 'right' | null; heelBaseL: Point2D | null; ... }
interface PalmsState { wasPalms: boolean }
// etc.

GestureEngine composes these. This makes the state boundaries explicit and each detector truly self-contained.

B. Decide on frame-loop ownership

The cleanest architecture: usePoseLoop owns the requestAnimationFrame loop and calls gestureEngine.update() + arScene.update() directly — no React state in the hot path. React only gets notified for UI updates (gesture label, calibration status) via batched setState. The onFrame callback is for the canvas drawing, not for gesture detection.

C. Callbacks as the primary interface

Callbacks are the right call — they make the API push-based and decouple consumers from engine internals. Just make sure the engine still exposes a read-only state getter for the few cases where the animation loop needs synchronous access (e.g. canvas drawing checks calibrationStatus to decide whether to draw the red box). Callbacks notify, state answers "what's true right now".

16 KiB Raw Blame History