GAZE-GUIDED MEDICAL IMAGE SEGMENTATION

GazeLIFT

Gaze-Guided Few-Shot Adaptation
for Medical Image Segmentation

Mohammed Oussama Benyahia1, Marouane Tliba1, Mohamed Amine Kerkouri1, Taifour Yousra1, Bin Wang2, Zuheng Ming1, Azeddine Beghdadi1, Ulas Bagci2, Aladine Chetouani1

1 L2TI, Institut Galilée, Université Sorbonne Paris Nord, France
2 Northwestern University, USA

GazeLIFT uses fixation locations and durations to condition frozen DINOv3 features for polyp and prostate segmentation.

0.818Mtrainable parameters
93.35%Kvasir-SEG Dice
89.24%NCI-ISBI Dice

Main configuration, seed 42. Parameter count excludes the frozen DINOv3 encoder, which is required at inference.

METHOD

Model architecture

GazeLIFT conditions multi-depth DINOv3 features using duration-weighted gaze and foreground–background prototype contrast. A learnable lifting-wavelet branch supplies image features. The decoder combines these inputs to predict a mask.

GazeLIFT architecture
GazeLIFT architecture with frozen DINOv3, gaze conditioning, lifting wavelets and compact decoder
Four frozen feature depths, gaze-conditioned adaptation and eight learnable wavelet parameters. Expert fixation records are required at inference.
01

Feature extraction

Frozen DINOv3 features are extracted at blocks 6, 12, 18 and 24.

02

Gaze conditioning

Fixation locations and durations form heatmaps that guide prototype contrast and feature adaptation.

03

Mask prediction

Wavelet features and RGB information complement the semantic features in the segmentation decoder.

RESULTS

Accuracy and training-run variability

Across seeds 40–54, GazeLIFT improves the mean test Dice over the without-gaze configuration on both datasets. Values below report mean ± SD across training runs; confidence intervals describe the estimated run-level mean.

DatasetWithout gazeGazeLIFTPaired gain (95% CI), pp
Kvasir-SEG90.28 ± 0.54
95% CI [90.03, 90.56]
93.22 ± 0.44
95% CI [93.00, 93.43]
+2.94 [2.69, 3.18]
NCI-ISBI84.25 ± 0.88
95% CI [83.81, 84.67]
87.94 ± 0.79
95% CI [87.54, 88.31]
+3.70 [3.11, 4.29]

Test sets: 100 Kvasir-SEG images and 117 NCI-ISBI slices. These intervals quantify training-run variability on fixed test sets; they do not establish performance on new clinical populations. Paired gains use unrounded run scores.

TRAINING FRACTIONS

Adaptation with few training masks

In the separate fixed-update experiment, nine Kvasir-SEG training masks yield 83.16% Dice and six NCI-ISBI training masks yield 80.85%. The six prostate slices span six patients. Fixed validation sets contain an additional 90 images and 73 slices, respectively.

Training-fraction experiment
Dice performance as a function of training fraction
This experiment uses a separate fixed-update protocol. Its 100% reference scores are 93.35% and 89.23%.
QUALITATIVE COMPARISON

Target coverage and boundary alignment

One test case per dataset
One polyp and one prostate case comparing GazeLIFT with nnU-Net plus gaze, GazeRefine and GazeMedSAMv2
Yellow markers and lines show expert gaze; marker size encodes fixation duration. Green indicates ground truth and cyan indicates predictions. All methods receive the same fixation records. Per-case Dice scores (%) appear on the predictions.

In the displayed polyp case, GazeLIFT follows the target contour while comparator predictions include additional regions. In the prostate case, GazeLIFT covers more of the annotated gland, while comparator predictions leave larger parts uncovered.

RESOURCES

Code and demo

Interactive segmentation

Run the trained model with an image and gaze fixations. The demo accepts fixation clicks and imported gaze records.

Open demo ↗
Enlarged research figure