Imitation Quality

Bitbox can quantify how closely one set of signals follows a reference set of signals. This functionality is useful for measuring imitation quality in facial expressions, head movements, or body actions (coming soon).

This function accepts landmarks (2D or 3D), head pose, or facial expressions (global or local). It computes a windowed, lagged cross-correlation between all pairs of signals.

from bitbox.social import imitation

# output directory
output_dir = 'output'

# define a face processor
processor = FP(runtime='bitbox:latest')

# get all facial signals for a participant
processor.io(input_file='participant.mp4', output_dir=output_dir)
rect_p, land_p, exp_glob_p, pose_p, land_can_p, exp_loc_p = processor.run_all(normalize=True)

# get all facial signals for the reference model
processor.io(input_file='model.mp4', output_dir=output_dir)
rect_r, land_r, exp_glob_r, pose_r, land_can_r, exp_loc_r = processor.run_all(normalize=True)

# quantify imitation performance
corr_mean, corr_std, corr_lag = imitation(exp_glob_p, exp_glob_r, width=1.1, step=0.5, fps=30)

Correlation Mean: The average correlation between two signals across all time windows.

Correlation Std: The standard deviation of correlations across all time windows.

Lag: The average optimal lag that maximizes the correlation across all time windows.

All three outputs are matrices where rows correspond to the first set of signals (e.g., 79 global expression units for the participant) and columns to the second (e.g., 79 global expression units for the model).

Bitbox automatically learns the optimal lag between signals. In imitation mode, lag is restricted to one direction — a time point in the first signal can match the same or an earlier point in the reference signal (you can imitate a model slightly behind, but not ahead). To disable this causality constraint and allow lags in both directions, set casuality=False. Note that lags are defined relative to the second signal; if you need bidirectional lags, use the coordination function instead.

# allow lags both in past and future
corr_mean, corr_std, corr_lag = imitation(exp_glob_p, exp_glob_r, casuality=False)

The window width (width) and step size (step) are defined in seconds, so ensure that the frame rate (fps) matches your signal’s sampling rate.

By default, positive and negative correlations are treated the same (the maximum absolute correlation is used per window). To ignore negative correlations and consider only positive ones, set polarity=False.

# Only positive correlations are preferred
corr_mean, corr_std, corr_lag = imitation(exp_glob_p, exp_glob_r, polarity=False)

If you want to access detailed correlation values and lags for every individual window rather than aggregated statistics, you can set stats=False. In that case, the function returns 3D arrays (Nwindows x Nsignals1 x Nsignals2) that contain the pairwise correlation values (or corresponding lags) for each window. This allows you to analyze how the imitation quality changes dynamically throughout the video.

# Get correlation values and lags for each window
corr_vals, corr_lags = imitation(exp_glob_p, exp_glob_r, stats=False)