LEAF LEAFDocumentation
Project

Supervised fine-tuning

The paper evaluates two fine-tuning paths starting from the same instruction-tuned LEAF checkpoint. Both adapt the model using labeled training trials from one target dataset. Direct inference with c_direct_inference.py instead uses the checkpoint without further parameter updates.

Two LEAF fine-tuning paths: (A) EEG encoder with a new MLP classifier; (B) EEG encoder and Q-Former with cosine-similarity logits against fixed text prototypes

Fine-tuning from a shared instruction-tuned checkpoint. (A) adds a dataset-specific MLP; (B) retains the EEG–language classification path without adding trainable parameters.

A. EEG encoder + MLP (main paper results)

d_downstream_add_mlp.py restores the EEG Tower from the instruction-tuned checkpoint and attaches a randomly initialized FlattenClassifier. It concatenates the contextual and causal branch tokens, flattens them, and applies an MLP with hidden widths 2 × dim_token and dim_token, GELU activations, and dropout of 0.3. The output has one logit per dataset class.

The encoder parameters in this prediction path and the new MLP are jointly optimized with cross-entropy. The Q-Former is not instantiated, and no instruction embeddings or textual class prototypes are used for classification. At test time, the predicted class is the largest MLP logit. This is the supervised transfer setting used for LEAF in the paper's main baseline comparisons.

B. EEG encoder + Q-Former + cosine cross-entropy

d_downstream_qformer_ce.py restores the complete instruction-tuned LEAF model, including the Q-Former. For each EEG trial, the instruction-conditioned output is L2-normalized and compared with fixed, normalized label-text embeddings. The class logits are cosine_similarity; cross-entropy updates the existing EEG encoder and Q-Former. No new trainable parameters are added. The text embeddings remain fixed.

Fine-tuning uses task-and-class conditioning: the task instruction is combined with the normalized mean of the candidate-label prototypes. This uses the candidate class set, not the trial's ground-truth label. The existing encoder and Q-Former are jointly fine-tuned. At test time, the class with the largest cosine logit is selected.

Run fine-tuning

Run these commands from LEAF/ after preparing the datasets and downloading the instruction-tuned checkpoint as described in the repository quick start. Both scripts use the dataset-specific train/validation/test partitions in load_datasets.py; these include cross-subject and within-subject trial splits depending on the dataset. Class indices must follow the target-label order in configs/tasks.yaml. Path B also requires the matching cached text embeddings in text_embeddings/.

# (A) Encoder + a new MLP head
python d_downstream_add_mlp.py \
  --config configs/LEAF_mpnet.yaml \
  --ckpt leaf-v1.0-instruct-mpnet-base.ckpt \
  --ds MI_BCIC_IV2a --gpu 0 --bs 128 --epochs 100 --seed 0

# (B) Existing encoder + Q-Former, with fixed text prototypes
python d_downstream_qformer_ce.py \
  --config configs/LEAF_mpnet.yaml \
  --ckpt leaf-v1.0-instruct-mpnet-base.ckpt \
  --ds MI_BCIC_IV2a --gpu 0 --bs 128 --epochs 100 --seed 0

These commands explicitly use the paper's stated 100-epoch, batch-size-128 fine-tuning settings. Without overrides, the new MLP script defaults to batch size 512 and 100 epochs, while the Q-Former script defaults to batch size 128 and reads the epoch count from downstream.max_epochs (currently 200). With configs/LEAF_mpnet.yaml, both use AdamW, weight decay 1e-3, one warmup epoch, and cosine decay to 0.1 times the peak learning rate. Peak learning rates are 1e-3 for the tokenizer and the new MLP, when present, and 1e-4 for the remaining parameters. GPU training uses bf16 mixed precision by default; --gpu -1 uses CPU fp32.

Both new scripts select the checkpoint with the highest validation balanced accuracy, breaking ties with the lowest validation loss, then evaluate the test set once. They save best.ckpt, run.json, and metrics.npz under checkpoints/mlp_finetuning/ or checkpoints/qformer_finetuning/ for the released checkpoint; Path B also saves text_readout.npz. Use --outdir to choose a new destination. Repeat with different --seed values for multiple runs; existing output directories are not overwritten.

Fine-tuning results

The table below reproduces the camera-ready paper's fine-tuning comparison table. Balanced accuracy (B.Acc) and Cohen's kappa are reported on the 0–1 scale as mean ± standard deviation across three seeds; the Q-Former runs use seeds 0, 1, and 2. The MLP values are the LEAF results from the paper's main tables. The average row weights all 16 datasets equally.

Encoder + MLP averages 64.14% balanced accuracy and 0.4474 kappa; Q-Former + cosine CE averages 62.31% and 0.4344. The MLP path is higher by 1.83 percentage points in average balanced accuracy and has higher mean balanced accuracy on 12 of these 16 datasets. The Q-Former path improves balanced accuracy on BCIC-IV-2a, Cho2017, Shin2017A, and PhysioNet-MI while retaining the text-prototype readout. These are reported means, not claims of statistical significance.

ID Dataset (A) Encoder + MLP B.Acc (A) Kappa (B) Q-Former + cosine CE B.Acc (B) Kappa
1 BCIC-IV-2a 0.6381 ± 0.0120 0.5175 ± 0.0160 0.6620 ± 0.0136 0.5494 ± 0.0182
2 OpenBMI-MI 0.8134 ± 0.0031 0.6268 ± 0.0062 0.8017 ± 0.0018 0.6035 ± 0.0036
3 BCIC-Upperlimb 0.5483 ± 0.0038 0.3225 ± 0.0057 0.4828 ± 0.0038 0.2242 ± 0.0058
4 SHU-MI 0.6134 ± 0.0008 0.2268 ± 0.0016 0.5897 ± 0.0053 0.1792 ± 0.0106
5 HighGamma 0.7982 ± 0.0035 0.5964 ± 0.0070 0.7899 ± 0.0104 0.5797 ± 0.0208
6 Cho2017 0.7908 ± 0.0060 0.5816 ± 0.0120 0.8030 ± 0.0056 0.6059 ± 0.0111
7 Shin2017A 0.7256 ± 0.0088 0.4512 ± 0.0176 0.7509 ± 0.0307 0.5019 ± 0.0614
8 PhysioNet-MI 0.6992 ± 0.0035 0.3984 ± 0.0070 0.7535 ± 0.0025 0.5070 ± 0.0050
9 FACED 0.5819 ± 0.0023 0.5293 ± 0.0026 0.4071 ± 0.0040 0.3292 ± 0.0050
10 SEED 0.7011 ± 0.0068 0.5511 ± 0.0102 0.6870 ± 0.0035 0.5347 ± 0.0045
11 SEED-IV 0.4630 ± 0.0034 0.2660 ± 0.0044 0.4511 ± 0.0048 0.2702 ± 0.0056
12 SEED-V 0.4126 ± 0.0023 0.2640 ± 0.0029 0.4010 ± 0.0076 0.2556 ± 0.0106
13 SEED-VII 0.3356 ± 0.0012 0.2246 ± 0.0014 0.3286 ± 0.0032 0.2175 ± 0.0038
14 OpenBMI-SSVEP 0.9462 ± 0.0043 0.9283 ± 0.0057 0.8971 ± 0.0047 0.8628 ± 0.0063
15 BCIC-Speech 0.5453 ± 0.0023 0.4316 ± 0.0029 0.5338 ± 0.0078 0.4172 ± 0.0098
16 Mental Workload 0.6493 ± 0.0211 0.2420 ± 0.0370 0.6296 ± 0.0334 0.3122 ± 0.0921
Average 0.6414 0.4474 0.6231 0.4344