Iris Landmark Localization

Dense iris landmark regression trained on synthesized eye images — end-to-end from synthetic data generation to real-world inference.

A full end-to-end implementation of dense iris landmark localization using a CNN trained exclusively on synthesized eye images and evaluated on real cropped eyes.

Repository: digital-nomad-cheng/Iris_Landmarks_PyTorch

flowchart LR
    A["UnityEyes Synthesizer"] --> B["gen_dataset Scripts"]
    B --> C["Annotated Dataset"]
    C --> D["CNN Training"]
    D --> E["ONNX Export"]
    E --> F["onnxsim"]
    F --> G["ncnn .param/.bin"]

    style A fill:#4a90d9,color:#fff
    style B fill:#7b61ff,color:#fff
    style C fill:#56b6c2,color:#fff
    style D fill:#e06c75,color:#fff
    style E fill:#f0a500,color:#fff
    style F fill:#d35400,color:#fff
    style G fill:#27ae60,color:#fff

The Job-Landing Feature: Synthetic-to-Real Transfer

Most practitioners fine-tune pre-trained models on labelled data. This project takes a harder path:

1. Synthesize the data — use the UnityEyes renderer to generate thousands of parameterically varied eye images with ground-truth iris landmarks.
2. Script the annotations — gen_dataset/ scripts convert renderer outputs into training-ready landmark coordinates.
3. Design a custom loss — loss.py implements a regression loss tailored for facial landmark localization (robust against outliers in synthetic-to-real domain shifts).
4. Train from scratch — training/train.py + config.py give full control over architecture, schedule, and augmentation.
5. Generalise to real eyes — the trained model runs inference on real cropped eye images, demonstrating domain transfer without any real-data labels.

This skill set — building data pipelines, writing loss functions, and closing the sim-to-real gap — is directly relevant to roles in face analytics, AR/VR gaze tracking, automotive ADAS, and robotics perception.

Dataset

Two options for preparing training data:

• DIY synthesis: Run the UnityEyes renderer and use gen_dataset/ to produce annotations.
• Pre-built dataset: Download the provided dataset (Baidu Pan, password 990n) and drop images into data/.

Edge Inference with ncnn

The onnx2ncnn/ folder contains the full deployment pipeline for running the iris landmark model on mobile and edge devices:

1. Architecture optimisation — nets/optimized_landmark.py redesigns the backbone using MobileNetV2-style depthwise separable convolutions and inverted residual blocks, cutting FLOPs while preserving accuracy. Input resolution: 100 × 112.
2. Export to ONNX — the trained PyTorch model is exported via torch.onnx.export.
3. Model simplification — onnxsim eliminates redundant nodes, producing a clean graph for conversion.
4. Conversion to ncnn — onnx2ncnn.sh calls the onnx2ncnn tool to produce iris_lnet.param (graph definition) and iris_lnet.bin (weights), ready to be loaded by any ncnn C++ or Android/iOS application.

🔗 View onnx2ncnn/

This makes the project a complete research-to-deployment story: from synthetic data and model training all the way to an optimised binary running on an edge CPU.

Tech Stack

Python · PyTorch · OpenCV · NumPy · ncnn · ONNX, Synthetic Data

References

Resource	Notes
UnityEyes	Parametric eye image synthesizer (Univ. of Cambridge)
Google MediaPipe Iris (Bazarevsky et al., 2020)	Production iris landmark model — the target to compare against
ncnn	Tencent’s high-performance neural network inference framework for mobile
onnx-simplifier	Graph simplification tool used before ncnn conversion