Improve debug views: add 2D graph, fix flipped axes, hide invisible landmarks, show data pre-calibration
- Delete REFACTOR.md, add PAUSE_SIGNAL.md for pause gesture rework planning - Fix vertically flipped FRONT/SIDE 3D views (remove incorrect Y negation) - Hide landmarks with visibility < 0.3 and exclude from auto-scale - Show step graphs before calibration completes (use raw position as baseline) - Add 4th debug view showing 2D normalized landmarks - Expose landmarks2D from usePoseLoop - Flip step graph Y axis to match intuitive direction Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>main
name
parent
ce7cf3a042
commit
5007dd9440
|
|
@ -0,0 +1,32 @@
|
|||
# Pause Signal Rework
|
||||
|
||||
## Problem
|
||||
|
||||
The current pause gesture is **arms-out (T-pose)**: both wrists at shoulder height.
|
||||
|
||||
Issues:
|
||||
- **Accidental triggers** — players raise arms during gameplay and inadvertently pause
|
||||
- **Unintuitive** — new players don't know or remember to hold a T-pose to pause
|
||||
|
||||
## Current Implementation
|
||||
|
||||
- `armsOutDetector.ts` — checks both wrists within `ARMS_Y_TOLERANCE` (0.15) of shoulder Y
|
||||
- 2-second cooldown after arms drop before it can re-trigger
|
||||
- Arms-out is dual-purpose: it both **pauses gameplay** and is the **calibration-ready signal**
|
||||
|
||||
## Alternative Signals
|
||||
|
||||
| Gesture | Pros | Cons |
|
||||
|---|---|---|
|
||||
| **Hands on head** (both wrists above head) | Very intentional, hard to do by accident | Might be tiring if held |
|
||||
| **Crossed arms** (wrists near opposite shoulders) | Familiar "stop" posture, unlikely during dance | Harder to detect reliably with occlusion |
|
||||
| **Hand over face** (one wrist covering nose) | Simple, quick | Could conflict with wiping sweat |
|
||||
| **Both hands raised high** (surrender pose) | Easy to detect, distinct from dance moves | Could overlap with jump celebrations |
|
||||
| **Timeout / stillness** (no movement for N seconds) | No specific pose needed | Slow, frustrating to wait |
|
||||
| **Palm facing camera** (open hand toward webcam) | Intuitive "stop" signal | Requires hand landmark model, more compute |
|
||||
|
||||
## Considerations
|
||||
|
||||
- Whatever replaces T-pose for pause must not conflict with the calibration flow (which also uses arms-out to signal "ready")
|
||||
- Should be easy to explain in the tutorial overlay
|
||||
- Must be distinct from stepping, jumping, and natural dance movement
|
||||
357
REFACTOR.md
357
REFACTOR.md
|
|
@ -1,357 +0,0 @@
|
|||
# Refactor Proposal: Modular MediaPipe + Gesture Callbacks
|
||||
|
||||
## Current State
|
||||
|
||||
| File | Lines | Responsibility |
|
||||
|------|-------|---------------|
|
||||
| `gestureDetector.ts` | 511 | State machine + all gesture detection + step detection + graph sampling + logging |
|
||||
| `App.tsx` | 457 | MediaPipe init, video management, animation loop, canvas drawing, gesture wiring, UI |
|
||||
| `Settings.tsx` | 254 | Settings panel (fine as-is) |
|
||||
| `StepGraphs.tsx` | 153 | Debug visualization (fine as-is) |
|
||||
| `arScene.ts` | 64 | Three.js setup (fine as-is) |
|
||||
| `types.ts` | 49 | Settings types + persistence |
|
||||
| `worldToCamera.ts` | 61 | Unused (dead code) |
|
||||
| `DebugGraph.tsx` | 136 | Unused (dead code) |
|
||||
| `debugLog.ts` | 16 | In-memory log buffer |
|
||||
|
||||
### Problems
|
||||
|
||||
1. **gestureDetector.ts is a 511-line monolith** -- calibration state machine, jump detection, leg raises, palms, arms-out, step detection, graph sampling, and baseline adaptation all live in one function.
|
||||
|
||||
2. **App.tsx does too much** -- MediaPipe lifecycle, webcam/file source management, the animation loop, canvas drawing, and React UI state are all in one component with 7 refs.
|
||||
|
||||
3. **No callback/event system** -- gesture signals pass through mutable `GestureState` + return values. There's no way to subscribe to "a step was detected" without polling the ref every frame.
|
||||
|
||||
4. **Dead code** -- `worldToCamera.ts` and `DebugGraph.tsx` are imported nowhere.
|
||||
|
||||
---
|
||||
|
||||
## Proposed Architecture
|
||||
|
||||
```
|
||||
src/
|
||||
types.ts # All shared types (expanded)
|
||||
debugLog.ts # Unchanged
|
||||
|
||||
pose/
|
||||
PoseDetector.ts # MediaPipe lifecycle (init, detect, dispose)
|
||||
VideoSource.ts # Webcam/file source: attach -> loadeddata -> play, debug.webm probe
|
||||
usePoseLoop.ts # Hook: owns rAF loop, calls engine.update() + arScene directly (no React in hot path)
|
||||
|
||||
gesture/
|
||||
types.ts # GestureSignal, CalibrationStatus, per-detector state, callbacks
|
||||
CalibrationMachine.ts # Framing -> waiting -> calibrating -> pos-center -> ready/paused
|
||||
GestureTelemetry.ts # Graph/log sampling — subscribes to callbacks, collects GraphSample/StepGraphSample/StepLogEntry
|
||||
detectors/
|
||||
jumpDetector.ts # Jump detection (both ankles rise)
|
||||
stepDetector.ts # Step detection (toe XZ displacement + baseline adaptation)
|
||||
palmsDetector.ts # Palms together (wrist distance)
|
||||
armsOutDetector.ts # Arms out (wrist at shoulder height)
|
||||
GestureEngine.ts # Orchestrates calibration + detectors, fires callbacks. Config via constructor + setConfig()
|
||||
useGesture.ts # Hook: creates engine, exposes read-only state + callbacks to React. Owns display timer (not engine)
|
||||
|
||||
ar/
|
||||
arScene.ts # Three.js setup (remove unused resizeARScene export)
|
||||
|
||||
components/
|
||||
App.tsx # Slim orchestrator: wires hooks together, renders UI
|
||||
Settings.tsx # Unchanged
|
||||
StepGraphs.tsx # Unchanged
|
||||
GestureOverlay.tsx # Gesture label + calibration hints (extracted from App). Owns 600ms display timer
|
||||
TransportBar.tsx # Video playback controls (extracted from App)
|
||||
PoseCanvas.tsx # Canvas drawing: landmarks, connectors, framing box, debug text
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### 1. Callback-based gesture events
|
||||
|
||||
The core ask. Instead of polling a mutable ref, consumers register callbacks:
|
||||
|
||||
```ts
|
||||
// gesture/types.ts
|
||||
export interface GestureCallbacks {
|
||||
onGesture?: (signal: GestureSignal) => void // any gesture fires (immediate, no display timer)
|
||||
onStep?: (direction: StepDirection, foot: 'left' | 'right') => void
|
||||
onJump?: () => void
|
||||
onPalms?: () => void
|
||||
onArmsOut?: () => void
|
||||
onCalibrationChange?: (status: CalibrationStatus) => void
|
||||
onPause?: () => void
|
||||
onResume?: () => void
|
||||
}
|
||||
|
||||
export type StepDirection = 'left' | 'right' | 'forward' | 'back'
|
||||
```
|
||||
|
||||
The `GestureEngine` accepts callbacks at construction and fires them when gestures trigger. The `useGesture` hook exposes this to React:
|
||||
|
||||
```ts
|
||||
// Usage in App.tsx
|
||||
const gesture = useGesture(poseResults, settings, {
|
||||
onStep: (dir, foot) => console.log(`Step ${dir} with ${foot} foot`),
|
||||
onJump: () => console.log('Jump!'),
|
||||
onCalibrationChange: (status) => console.log(`Calibration: ${status}`),
|
||||
})
|
||||
```
|
||||
|
||||
### 2. PoseDetector class wraps MediaPipe lifecycle
|
||||
|
||||
Extracts the MediaPipe init/detect/dispose cycle out of App.tsx into a plain class:
|
||||
|
||||
```ts
|
||||
// pose/PoseDetector.ts
|
||||
export class PoseDetector {
|
||||
static async create(settings: PoseSettings): Promise<PoseDetector>
|
||||
detectFrame(video: HTMLVideoElement, timestamp: number): PoseResult
|
||||
dispose(): void
|
||||
}
|
||||
|
||||
export interface PoseResult {
|
||||
landmarks2D: NormalizedLandmark[][]
|
||||
worldLandmarks: Landmark[][]
|
||||
}
|
||||
```
|
||||
|
||||
### 3. VideoSource class wraps webcam/file management
|
||||
|
||||
Owns the `MediaStream`, video element setup, and recording logic:
|
||||
|
||||
```ts
|
||||
// pose/VideoSource.ts
|
||||
export class VideoSource {
|
||||
static async create(
|
||||
video: HTMLVideoElement,
|
||||
mode: 'webcam' | 'file',
|
||||
fileUrl?: string
|
||||
): Promise<VideoSource>
|
||||
get stream(): MediaStream | null
|
||||
dispose(): void
|
||||
}
|
||||
```
|
||||
|
||||
### 4. usePoseLoop hook ties it together
|
||||
|
||||
Single hook that manages the animation frame loop:
|
||||
|
||||
```ts
|
||||
// pose/usePoseLoop.ts
|
||||
export function usePoseLoop(
|
||||
videoRef: RefObject<HTMLVideoElement>,
|
||||
settings: PoseSettings,
|
||||
fileUrl: string | null,
|
||||
onFrame: (result: PoseResult, fps: number) => void,
|
||||
): { loading: boolean; error: string | null }
|
||||
```
|
||||
|
||||
### 5. Individual detectors are pure functions
|
||||
|
||||
Each detector is a small, focused, testable function that owns its own state type:
|
||||
|
||||
```ts
|
||||
// gesture/detectors/jumpDetector.ts
|
||||
export interface JumpState { wasAirborne: boolean }
|
||||
export function createJumpState(): JumpState
|
||||
export function detectJump(
|
||||
state: JumpState,
|
||||
leftRise: number,
|
||||
rightRise: number,
|
||||
): boolean // true = just detected
|
||||
```
|
||||
|
||||
```ts
|
||||
// gesture/detectors/stepDetector.ts
|
||||
export interface StepState {
|
||||
activeStep: GestureSignal
|
||||
steppedFoot: 'left' | 'right' | null
|
||||
heelBaseL: Point2D | null
|
||||
heelBaseR: Point2D | null
|
||||
}
|
||||
export function createStepState(): StepState
|
||||
export function detectStep(...): StepResult | null
|
||||
export function checkStepReset(...): boolean
|
||||
export function adaptBaseline(base: Point2D, current: Point2D, alpha?: number): Point2D
|
||||
```
|
||||
|
||||
```ts
|
||||
// gesture/detectors/palmsDetector.ts
|
||||
export interface PalmsState { wasPalms: boolean }
|
||||
export function createPalmsState(): PalmsState
|
||||
export function detectPalms(state: PalmsState, worldLandmarks: Landmark3D[]): boolean
|
||||
|
||||
// gesture/detectors/armsOutDetector.ts
|
||||
export interface ArmsOutState { wasArmsOut: boolean; armsDownSince: number | null }
|
||||
export function createArmsOutState(): ArmsOutState
|
||||
export function detectArmsOut(state: ArmsOutState, worldLandmarks: Landmark3D[]): boolean
|
||||
```
|
||||
|
||||
### 6. CalibrationMachine is a standalone state machine
|
||||
|
||||
Separated from gesture detection. Owns ankle baseline collection, heel calibration samples, and foot spread measurement:
|
||||
|
||||
```ts
|
||||
// gesture/CalibrationMachine.ts
|
||||
export interface CalibratedBaselines {
|
||||
leftY: number // 2D ankle Y baseline
|
||||
rightY: number
|
||||
heelBaseL: Point2D // hip-relative XZ for step detection
|
||||
heelBaseR: Point2D
|
||||
footSpread: number // meters
|
||||
}
|
||||
|
||||
export class CalibrationMachine {
|
||||
get status(): CalibrationStatus
|
||||
get framingHint(): string | null
|
||||
get baselines(): CalibratedBaselines | null
|
||||
|
||||
/** Call each frame. Returns true if state changed. */
|
||||
update(
|
||||
landmarks2D: NormalizedLandmark[],
|
||||
worldLandmarks: Landmark3D[],
|
||||
): boolean
|
||||
|
||||
reset(): void
|
||||
}
|
||||
```
|
||||
|
||||
### 7. GestureEngine orchestrates everything
|
||||
|
||||
Replaces the current `detectGesture()` monolith. Owns the calibration machine and runs all detectors in sequence. Config passed at construction, updated via setter:
|
||||
|
||||
```ts
|
||||
// gesture/GestureEngine.ts
|
||||
export class GestureEngine {
|
||||
constructor(config: StepConfig, callbacks?: GestureCallbacks)
|
||||
|
||||
/** Call each frame with latest pose data. Returns current signal for display. */
|
||||
update(
|
||||
landmarks2D: NormalizedLandmark[],
|
||||
worldLandmarks: Landmark3D[],
|
||||
): GestureSignal
|
||||
|
||||
/** Read-only access for synchronous canvas drawing (e.g. calibrationStatus for red box) */
|
||||
get state(): Readonly<GestureState>
|
||||
|
||||
setConfig(config: StepConfig): void
|
||||
setCallbacks(callbacks: GestureCallbacks): void
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## What Stays the Same
|
||||
|
||||
- All detection algorithms (thresholds, hysteresis, baseline adaptation)
|
||||
- Calibration flow (framing -> waiting -> calibrating -> pos-center -> ready/paused)
|
||||
- Settings panel and persistence
|
||||
- Three.js AR overlay
|
||||
- Step graph visualization
|
||||
- Debug logging
|
||||
- Recording functionality
|
||||
|
||||
---
|
||||
|
||||
## What Gets Deleted
|
||||
|
||||
- `worldToCamera.ts` -- unused dead code
|
||||
- `DebugGraph.tsx` -- unused dead code
|
||||
|
||||
---
|
||||
|
||||
## Migration Order
|
||||
|
||||
The refactor is incremental — app stays working at each step:
|
||||
|
||||
1. **Delete dead code** -- `worldToCamera.ts`, `DebugGraph.tsx`, unused `resizeARScene` export, `left-leg`/`right-leg` from `GestureSignal` type + App.tsx UI references, `wasLeftRaised`/`wasRightRaised` from `GestureState`
|
||||
2. **Extract types** -- move gesture types from `gestureDetector.ts` to `gesture/types.ts`, add `GestureCallbacks` interface, per-detector state types
|
||||
3. **Extract individual detectors** -- pull `jumpDetector`, `stepDetector`, `palmsDetector`, `armsOutDetector` out as pure functions with own state types
|
||||
4. **Extract CalibrationMachine** -- pull calibration state logic + baseline collection into its own module
|
||||
5. **Build GestureEngine as thin wrapper** -- initially delegates to existing `detectGesture()` to keep things working, then incrementally migrate logic to use calibration machine + detectors + callback dispatch
|
||||
6. **Extract GestureTelemetry** -- move graph/log sampling out of engine, subscribe to callbacks
|
||||
7. **Move display timer to React** -- remove `signalShowUntil` from engine, let `useGesture`/`GestureOverlay` own the 600ms display timer
|
||||
8. **Extract PoseDetector** -- wrap MediaPipe init/detect/dispose
|
||||
9. **Extract VideoSource** -- wrap webcam/file source management including attach -> loadeddata -> play lifecycle and debug.webm auto-probe
|
||||
10. **Build usePoseLoop hook** -- owns rAF loop, calls `gestureEngine.update()` + `arScene.update()` directly (no React in hot path), batched setState for UI
|
||||
11. **Build useGesture hook** -- wraps GestureEngine for React, exposes read-only state + callbacks
|
||||
12. **Extract UI components** -- `GestureOverlay`, `TransportBar`, `PoseCanvas`
|
||||
13. **Slim down App.tsx** -- wire hooks and components together
|
||||
|
||||
---
|
||||
|
||||
## Expected Result
|
||||
|
||||
| Before | After |
|
||||
|--------|-------|
|
||||
| `App.tsx` 457 lines | `App.tsx` ~100 lines |
|
||||
| `gestureDetector.ts` 511 lines | `GestureEngine.ts` ~80 lines + 4 detectors ~40-60 lines each + `CalibrationMachine.ts` ~120 lines + `GestureTelemetry.ts` ~60 lines |
|
||||
| No callback system | `GestureCallbacks` interface with per-gesture hooks |
|
||||
| 2 dead files | Deleted |
|
||||
| All logic in 2 big files | 15+ small, focused modules |
|
||||
|
||||
---
|
||||
|
||||
## Review Notes
|
||||
|
||||
### Issues Found
|
||||
|
||||
**1. Leg raise detection doesn't exist yet — nothing to extract**
|
||||
|
||||
The proposal lists `legRaiseDetector.ts` as something to extract, but `emit('left-leg')` / `emit('right-leg')` is never called in the current code. The `wasLeftRaised`/`wasRightRaised` state fields are declared but never written to. The `leftRaised`/`rightRaised` variables are computed (line 406-407) but only used to gate baseline adaptation (lines 503-508), not to fire a gesture signal. The type union and App.tsx UI reference `left-leg`/`right-leg`, so the intent was there, but the implementation was never finished. The refactor should either implement it or remove the dead types/UI — not list it as an extraction.
|
||||
|
||||
**2. Graph/log sampling has no home**
|
||||
|
||||
`GraphSample`, `StepGraphSample`, and `StepLogEntry` collection is ~40% of the current `detectGesture()` code. The proposal doesn't say which module owns this. It can't live in the individual detectors (they're supposed to be pure). It shouldn't live in `GestureEngine` (that's orchestration, not observability). Suggest a dedicated `gesture/GestureTelemetry.ts` that subscribes to the callback system, or have `GestureEngine` expose a `samples` getter and do collection internally as a separate concern.
|
||||
|
||||
**3. `signalShowUntil` mixes UI and detection**
|
||||
|
||||
The current code gates all detection behind `if (now < state.signalShowUntil) return state.activeSignal` — a 600ms display timer that suppresses new gestures. This is a UI concern (how long to show a label) leaking into detection logic. The proposal doesn't address where this lives. Suggestion: remove it from the engine entirely. Let `GestureEngine` fire callbacks immediately and let the React layer (`useGesture` or `GestureOverlay`) own the display timer.
|
||||
|
||||
**4. `arScene.ts` has dead code too**
|
||||
|
||||
The proposal says arScene.ts is "fine as-is", but `resizeARScene` (line 49) is exported and never called. Minor, but should be included in the dead code cleanup.
|
||||
|
||||
**5. `stepConfig` passed per-frame is noisy**
|
||||
|
||||
`GestureEngine.update()` takes `stepConfig` every frame, but it only changes when the user edits settings. Pass it at construction and add a `setConfig()` method instead. Same applies to any other settings the engine needs.
|
||||
|
||||
**6. `VideoSource` API is too thin**
|
||||
|
||||
The proposed `VideoSource.create(video, mode, fileUrl?)` doesn't account for:
|
||||
- The `loadeddata` promise needed before `video.play()`
|
||||
- Auto-probing `/debug.webm` on startup
|
||||
- Calling `video.play()` after source attachment
|
||||
|
||||
The real lifecycle is: attach source → wait for loadeddata → play. `VideoSource` needs to own all three steps or the caller ends up reimplementing the complexity.
|
||||
|
||||
**7. Data flow between hooks is undefined**
|
||||
|
||||
`usePoseLoop` emits `(result, fps)` via `onFrame`. `useGesture` takes `poseResults`. How do they connect? If `onFrame` calls into `useGesture`, then `useGesture` can't take `poseResults` as a prop — it needs to be a callback target. If App.tsx wires them via state, you get a frame of latency and re-render per frame. The proposal should specify: does `usePoseLoop` call `gestureEngine.update()` directly inside the animation loop, or does React mediate? The former is better for performance (no React in the hot path).
|
||||
|
||||
**8. Step 5 in migration is a risky big-bang swap**
|
||||
|
||||
"Build GestureEngine — compose calibration machine + detectors + callback dispatch, replacing `detectGesture()`" replaces the working monolith in one shot. Safer: first build `GestureEngine` as a thin wrapper that delegates to the existing `detectGesture()`, then incrementally migrate logic out of it.
|
||||
|
||||
### Suggestions
|
||||
|
||||
**A. Split `GestureState` per-detector**
|
||||
|
||||
The current 20+ field bag makes it hard to reason about what state belongs to which detector. Each detector should own its state:
|
||||
|
||||
```ts
|
||||
interface JumpState { wasAirborne: boolean }
|
||||
interface StepState { activeStep: GestureSignal; steppedFoot: 'left' | 'right' | null; heelBaseL: Point2D | null; ... }
|
||||
interface PalmsState { wasPalms: boolean }
|
||||
// etc.
|
||||
```
|
||||
|
||||
`GestureEngine` composes these. This makes the state boundaries explicit and each detector truly self-contained.
|
||||
|
||||
**B. Decide on frame-loop ownership**
|
||||
|
||||
The cleanest architecture: `usePoseLoop` owns the `requestAnimationFrame` loop and calls `gestureEngine.update()` + `arScene.update()` directly — no React state in the hot path. React only gets notified for UI updates (gesture label, calibration status) via batched setState. The `onFrame` callback is for the canvas drawing, not for gesture detection.
|
||||
|
||||
**C. Callbacks as the primary interface**
|
||||
|
||||
Callbacks are the right call — they make the API push-based and decouple consumers from engine internals. Just make sure the engine still exposes a read-only `state` getter for the few cases where the animation loop needs synchronous access (e.g. canvas drawing checks `calibrationStatus` to decide whether to draw the red box). Callbacks notify, state answers "what's true right now".
|
||||
|
|
@ -66,7 +66,7 @@ export function DebugApp() {
|
|||
|
||||
const {
|
||||
loading, error, fps, gesture, calibrationStatus, framingHint,
|
||||
stepGraphSamples, worldLandmarks, telemetry,
|
||||
stepGraphSamples, landmarks2D, worldLandmarks, telemetry,
|
||||
} = usePoseLoop(videoRef, canvasRef, arCanvasRef, settings, null, {
|
||||
onStep: (direction) => arrowGameRef.current?.handleStep(direction),
|
||||
})
|
||||
|
|
@ -144,7 +144,7 @@ export function DebugApp() {
|
|||
thresholdZ={settings.stepThresholdZ}
|
||||
reset={settings.stepReset}
|
||||
/>
|
||||
<Pose3DView landmarks={worldLandmarks} />
|
||||
<Pose3DView landmarks={worldLandmarks} landmarks2D={landmarks2D} />
|
||||
<button
|
||||
type="button"
|
||||
className="copy-log-btn"
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ function drawGraph(
|
|||
const allVals = samples.flatMap((s) => [getLeft(s), getRight(s)])
|
||||
const maxAbs = Math.max(threshold * 1.5, ...allVals.map(Math.abs))
|
||||
|
||||
const toY = (v: number) => h / 2 - (v / maxAbs) * (h / 2 - 4)
|
||||
const toY = (v: number) => h / 2 + (v / maxAbs) * (h / 2 - 4)
|
||||
const toX = (i: number) => (i / (samples.length - 1)) * w
|
||||
|
||||
// Highlight active step segments
|
||||
|
|
|
|||
|
|
@ -1,9 +1,10 @@
|
|||
import { useEffect, useRef } from 'react'
|
||||
import type { Landmark3D } from '../gesture/types'
|
||||
import type { Landmark2D, Landmark3D } from '../gesture/types'
|
||||
import { PoseLandmarker } from '@mediapipe/tasks-vision'
|
||||
|
||||
interface Props {
|
||||
landmarks: Landmark3D[]
|
||||
landmarks2D?: Landmark2D[]
|
||||
size?: number
|
||||
}
|
||||
|
||||
|
|
@ -14,8 +15,8 @@ type View = {
|
|||
}
|
||||
|
||||
const VIEWS: View[] = [
|
||||
{ title: 'FRONT (X\u00b7Y)', getX: (l) => l.x, getY: (l) => -l.y },
|
||||
{ title: 'SIDE (Z\u00b7Y)', getX: (l) => l.z, getY: (l) => -l.y },
|
||||
{ title: 'FRONT (X\u00b7Y)', getX: (l) => l.x, getY: (l) => l.y },
|
||||
{ title: 'SIDE (Z\u00b7Y)', getX: (l) => l.z, getY: (l) => l.y },
|
||||
{ title: 'TOP (X\u00b7Z)', getX: (l) => l.x, getY: (l) => l.z },
|
||||
]
|
||||
|
||||
|
|
@ -74,11 +75,13 @@ function drawView(
|
|||
return
|
||||
}
|
||||
|
||||
const VIS_THRESHOLD = 0.3
|
||||
const visible = landmarks.map((l) => (l.visibility ?? 1) >= VIS_THRESHOLD)
|
||||
const projected = landmarks.map((l) => ({ x: view.getX(l), y: view.getY(l) }))
|
||||
|
||||
// Auto-scale: fit skeleton into canvas with padding
|
||||
const xs = projected.map((p) => p.x)
|
||||
const ys = projected.map((p) => p.y)
|
||||
// Auto-scale: fit only visible landmarks into canvas with padding
|
||||
const xs = projected.filter((_, i) => visible[i]).map((p) => p.x)
|
||||
const ys = projected.filter((_, i) => visible[i]).map((p) => p.y)
|
||||
const minX = Math.min(...xs), maxX = Math.max(...xs)
|
||||
const minY = Math.min(...ys), maxY = Math.max(...ys)
|
||||
const range = Math.max(maxX - minX, maxY - minY) || 0.001
|
||||
|
|
@ -105,6 +108,7 @@ function drawView(
|
|||
// Connections
|
||||
ctx.lineWidth = 1.5
|
||||
for (const conn of PoseLandmarker.POSE_CONNECTIONS) {
|
||||
if (!visible[conn.start] || !visible[conn.end]) continue
|
||||
const a = toScreen(projected[conn.start])
|
||||
const b = toScreen(projected[conn.end])
|
||||
ctx.strokeStyle = connectionColor(conn.start, conn.end)
|
||||
|
|
@ -116,15 +120,13 @@ function drawView(
|
|||
|
||||
// Landmarks
|
||||
for (let i = 0; i < projected.length; i++) {
|
||||
if (!visible[i]) continue
|
||||
const p = toScreen(projected[i])
|
||||
const vis = landmarks[i].visibility ?? 1
|
||||
ctx.globalAlpha = vis < 0.3 ? 0.2 : 1
|
||||
ctx.fillStyle = landmarkColor(i)
|
||||
ctx.beginPath()
|
||||
ctx.arc(p.x, p.y, 3, 0, Math.PI * 2)
|
||||
ctx.fill()
|
||||
}
|
||||
ctx.globalAlpha = 1
|
||||
|
||||
// Title
|
||||
ctx.font = 'bold 14px monospace'
|
||||
|
|
@ -132,10 +134,13 @@ function drawView(
|
|||
ctx.fillText(view.title, 8, 20)
|
||||
}
|
||||
|
||||
export function Pose3DView({ landmarks, size = 220 }: Props) {
|
||||
const VIEW_2D: View = { title: '2D (X\u00b7Y)', getX: (l) => l.x, getY: (l) => l.y }
|
||||
|
||||
export function Pose3DView({ landmarks, landmarks2D, size = 220 }: Props) {
|
||||
const ref0 = useRef<HTMLCanvasElement>(null)
|
||||
const ref1 = useRef<HTMLCanvasElement>(null)
|
||||
const ref2 = useRef<HTMLCanvasElement>(null)
|
||||
const ref3 = useRef<HTMLCanvasElement>(null)
|
||||
|
||||
useEffect(() => {
|
||||
const refs = [ref0, ref1, ref2]
|
||||
|
|
@ -145,7 +150,12 @@ export function Pose3DView({ landmarks, size = 220 }: Props) {
|
|||
const ctx = canvas.getContext('2d')!
|
||||
drawView(ctx, size, size, landmarks, VIEWS[i])
|
||||
}
|
||||
}, [landmarks, size])
|
||||
|
||||
if (ref3.current && landmarks2D) {
|
||||
const lm3d = landmarks2D.map((l) => ({ x: l.x, y: l.y, z: 0, visibility: l.visibility }))
|
||||
drawView(ref3.current.getContext('2d')!, size, size, lm3d, VIEW_2D)
|
||||
}
|
||||
}, [landmarks, landmarks2D, size])
|
||||
|
||||
return (
|
||||
<div style={{
|
||||
|
|
@ -159,6 +169,7 @@ export function Pose3DView({ landmarks, size = 220 }: Props) {
|
|||
<canvas ref={ref0} width={size} height={size} />
|
||||
<canvas ref={ref1} width={size} height={size} />
|
||||
<canvas ref={ref2} width={size} height={size} />
|
||||
<canvas ref={ref3} width={size} height={size} />
|
||||
</div>
|
||||
)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -14,23 +14,25 @@ export class GestureTelemetry {
|
|||
sample(engine: GestureEngine, worldLandmarks: Landmark3D[]): void {
|
||||
const now = performance.now()
|
||||
|
||||
// Step graph + log (only during gameplay with baselines)
|
||||
if (engine.calibrationStatus === 'ready' && worldLandmarks?.length >= 33) {
|
||||
// Step graph + log (sample whenever we have landmarks)
|
||||
if (worldLandmarks?.length >= 33) {
|
||||
const heelBaseL = engine.heelBaseL
|
||||
const heelBaseR = engine.heelBaseR
|
||||
if (heelBaseL && heelBaseR) {
|
||||
const lt = worldLandmarks[31], rt = worldLandmarks[32]
|
||||
const sample: StepGraphSample = {
|
||||
time: now,
|
||||
leftX: lt.x - heelBaseL.x,
|
||||
rightX: rt.x - heelBaseR.x,
|
||||
leftZ: lt.z - heelBaseL.z,
|
||||
rightZ: rt.z - heelBaseR.z,
|
||||
activeStep: engine.activeStep,
|
||||
}
|
||||
this.stepGraph.push(sample)
|
||||
if (this.stepGraph.length > GRAPH_HISTORY) this.stepGraph.shift()
|
||||
const lt = worldLandmarks[31], rt = worldLandmarks[32]
|
||||
const baseL = heelBaseL ?? { x: lt.x, z: lt.z }
|
||||
const baseR = heelBaseR ?? { x: rt.x, z: rt.z }
|
||||
const sample: StepGraphSample = {
|
||||
time: now,
|
||||
leftX: lt.x - baseL.x,
|
||||
rightX: rt.x - baseR.x,
|
||||
leftZ: lt.z - baseL.z,
|
||||
rightZ: rt.z - baseR.z,
|
||||
activeStep: engine.activeStep,
|
||||
}
|
||||
this.stepGraph.push(sample)
|
||||
if (this.stepGraph.length > GRAPH_HISTORY) this.stepGraph.shift()
|
||||
|
||||
if (heelBaseL && heelBaseR) {
|
||||
this.stepLog.push({
|
||||
t: now,
|
||||
ltX: lt.x, ltZ: lt.z, rtX: rt.x, rtZ: rt.z,
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@ export interface PoseLoopState {
|
|||
calibrationStatus: CalibrationStatus
|
||||
framingHint: string | null
|
||||
stepGraphSamples: StepGraphSample[]
|
||||
landmarks2D: Landmark2D[]
|
||||
worldLandmarks: Landmark3D[]
|
||||
engine: GestureEngine | null
|
||||
telemetry: GestureTelemetry | null
|
||||
|
|
@ -52,6 +53,7 @@ export function usePoseLoop(
|
|||
const [calibrationStatus, setCalibrationStatus] = useState<CalibrationStatus>('framing')
|
||||
const [framingHint, setFramingHint] = useState<string | null>(null)
|
||||
const [stepGraphSamples, setStepGraphSamples] = useState<StepGraphSample[]>([])
|
||||
const [landmarks2DState, setLandmarks2DState] = useState<Landmark2D[]>([])
|
||||
const [worldLandmarksState, setWorldLandmarksState] = useState<Landmark3D[]>([])
|
||||
|
||||
const engineRef = useRef<GestureEngine | null>(null)
|
||||
|
|
@ -217,6 +219,7 @@ export function usePoseLoop(
|
|||
setFramingHint(engine.framingHint)
|
||||
if (frameCount % 3 === 0) {
|
||||
setStepGraphSamples([...telemetry.stepGraph])
|
||||
if (landmarks2D.length >= 33) setLandmarks2DState([...landmarks2D])
|
||||
if (worldLandmarks.length >= 33) setWorldLandmarksState([...worldLandmarks])
|
||||
}
|
||||
|
||||
|
|
@ -251,7 +254,7 @@ export function usePoseLoop(
|
|||
}, [settings, fileUrl]) // eslint-disable-line react-hooks/exhaustive-deps
|
||||
|
||||
return {
|
||||
loading, error, fps, gesture, calibrationStatus, framingHint, stepGraphSamples, worldLandmarks: worldLandmarksState,
|
||||
loading, error, fps, gesture, calibrationStatus, framingHint, stepGraphSamples, landmarks2D: landmarks2DState, worldLandmarks: worldLandmarksState,
|
||||
engine: engineRef.current,
|
||||
telemetry: telemetryRef.current,
|
||||
videoSource: videoSourceRef.current,
|
||||
|
|
|
|||
Loading…
Reference in New Issue