Behavioural capture stack

What is read from the video?

Inputs: Any modern video: venue AV, GoPro, iPhone, webcam. No wearables.
Computation: Frames are sampled, not read one by one, so compute stays bounded on a long session. A 2 minute clip runs at 12 frames a second, a 10 minute clip at 2, an hour at one frame per 4 seconds. On each sampled frame MediaPipe FaceMesh reads 468 landmarks per detected face, MediaPipe Pose reads body keypoints, and Whisper-derived features read pitch, pace and loudness. Face embeddings are not stored. Only the scoring numbers survive the pass.
Output: One record per sampled frame: the face slot, the smile, surprise and neutral probabilities, and an attending flag. Attending is true when the head points at the stage. That means within 25 degrees up or down, and 30 degrees left or right.
References: MediaPipe FaceMesh, 468 landmarks (Google Research, 2019)

Synchrony Score

Did the audience react together or one face at a time?

Inputs: Face arrays from each sampled frame. Faces are matched by slot index inside a frame; identity is not tracked across the room.
Computation: Rolling 30 second windows. In each window, correlate the smile-probability track of each pair of attending faces with at least 3 frames, using Pearson r. Report the aggregate mean pairwise r per window and for the session. An r of 1 means the pair moved together.
Output: One r per window, from -1 to 1. The UI shows (r + 1) / 2, on a 0 to 1 scale.
When it abstains: A window with fewer than two attending faces reports zero pairs, not an estimate.
References: Inter-subject facial response synchrony (PMC11109022, 2024)

What share of seconds did the room point at the stage?

Inputs: Face arrays from each sampled frame, with the attending flag set from head angle relative to a stage-facing camera. Head angle, not gaze.
Computation: Group faces by second. For each second, divide attending faces by faces detected. A second is held when that fraction reaches 0.6. Report the held rate, the mean fraction, and the longest unbroken run.
Output: An attention fraction per second. A held rate. The longest sustained run, in seconds. In code: attentionFraction, heldRate, longestSustainedSec.
References: Standard head-pose attention proxy (per-frame pitch/yaw thresholding).

Where did the audible reactions land and how long did they hold?

Inputs: The AudioPeak stream. Each peak carries a timestamp, an RMS loudness, how long it held above threshold, and a guessed type: laugh, applause, ovation, or reaction.
Computation: Count peaks by type. Rank them by duration and keep the top N. Bin into 5 second buckets, tracking weighted duration and peak amplitude. Report the 95th percentile amplitude, the level the loudest 5 percent of peaks pass.
Output: Counts by type, a sustained ranking, the 95th percentile amplitude, and a per-bin payload for the heat bar.
References: Affectiva audience response methodology (acoustic envelope segmentation).

Compared to the prior session, what moved?

Inputs: Two sessions for the same speaker, each with a SessionSummary and its engagement buckets.
Computation: Take the change in overall score, mean attention, and reaction totals. Compute Cohen's d across the bucket engagement scores on pooled standard deviation. That is the size of a change in standard deviations: meaningful at 0.5, a trend at 0.2, noise below.
Output: The change on each metric, Cohen's d, and the band it falls in.
When it abstains: A session needs at least two buckets. With one short session the panel shows no prior, not a number.
References: Cohen's d effect size convention (Cohen, 1988).