Direct inference
Direct inference evaluates an instruction-tuned LEAF checkpoint without training a dataset-specific classifier head. Predictions are made by comparing the normalized EEG embedding with frozen text-label prototypes.
Standard direct inference
c_direct_inference.py evaluates every dataset registered in configs/tasks.yaml.
Instruction levels
| Level | Instruction embedding |
|---|---|
| 0 | random Default string; no label blending |
| 1 | random task-family instruction; no label blending |
| 2 | task-family instruction averaged with the normalized mean of candidate-label embeddings, then normalized |
The random instruction choice is seeded by --seed. Each dataset is evaluated against only its own ordered label prototypes.
Metrics and outputs
c_direct_inference.py reads only the standard test partitions from load_datasets.py, loading each dataset once for all requested levels. It reports pooled-test balanced accuracy and Cohen's kappa. Predictions are scored batch by batch without retaining all EEG embeddings.
The default output is one combined CSV and a matching JSON file beside the checkpoint:
checkpoints/direct_inference/leaf-v1.0-instruct-mpnet-base_per_dataset.csv
checkpoints/direct_inference/leaf-v1.0-instruct-mpnet-base_per_dataset.json
Each CSV row contains dataset, n_samples, and levelN_bAcc / levelN_kappa for each selected level. The MEAN row is the equal-weight average over completed datasets. The JSON records the checkpoint, model configuration, device, batch size, seed, ordered labels, actual instruction texts, and completion status. Results are updated atomically after each dataset, so an interrupted run retains its completed rows. Existing output files are not overwritten; choose a new --output to rerun.
Command
python c_direct_inference.py \
--config configs/LEAF_mpnet.yaml \
--ckpt leaf-v1.0-instruct-mpnet-base.ckpt \
--gpu 0 --bs 256 --level 0,1,2 --seed 42
# Selected datasets and task-and-class conditioning only
python c_direct_inference.py \
--ckpt leaf-v1.0-instruct-mpnet-base.ckpt \
--ds MI_BCIC_IV2a,MI_OpenBMI --level 2 \
--gpu 0 --output checkpoints/direct_inference/selected.csv--ckpt accepts an absolute or relative path, or a filename under checkpoints/. --gpu accepts one GPU index or -1 for CPU. --bs must be positive. --dim and --emb optionally override the configured architecture and text model and must match the checkpoint. All registered datasets and all three levels are evaluated by default.
Instruction selection is computed in a fixed level-first order over all registered datasets, then restricted to the requested subset. With the same seed, selecting fewer datasets or levels does not change the chosen instructions.
Global embedding export
export_leaf_embeddings.py exports normalized EEG embeddings, true labels, predicted labels, label names, instruction text, checkpoint path, and instruction level for each test dataset. It writes one .npz per dataset plus summary.csv under:
ckpt/<architecture>/<tuning-run>/global_inference/level<level>/
It supports a comma-separated dataset subset and resumable output. For inputs longer than 1,000 samples, such as FACED, it caps effective batch size at 64 to reduce CUDA activation pressure.
Held-out zero-shot inference
zero_shot_heldout.py evaluates the frozen model on MI_Dreyer2023 and MI_Weibo2014, neither of which appears in instruction tuning. It uses only the Left and Right prototypes and reports balanced accuracy, kappa, weighted F1, AUROC, and confusion counts. See held-out datasets.