SCOPE-4D: Endoscopic 4D Geometry Foundation Models

Abstract

Geometric understanding supports endoscopic navigation and robotic assistance, but learning reliable endoscopic geometry faces two challenges: scarce geometric annotations and ambiguity between camera motion and tissue deformation. We present SCOPE-4D, an endoscopic 4D geometry foundation model that jointly predicts camera parameters, dense geometry, and 3D tissue trajectories from monocular RGB video in a single forward pass. Our curation and annotation pipeline constructs SCOPE-5K, a collection of approximately 5,000 clips spanning real and synthetic gastrointestinal endoscopy and laparoscopy. The collection provides rich geometric supervision and includes newly collected phantom and real-colonoscopy evaluation sets. Geometric supervised fine-tuning on SCOPE-5K learns endoscopic priors that improve camera and depth estimation. Common–Residual Motion (CRM) further constrains local deformation relative to common tissue movement. Together with geometric supervision, CRM and trajectory supervision further improve camera and depth estimation over geometric fine-tuning alone while enabling dense 3D tissue tracking. Evaluations on public and newly collected benchmarks demonstrate strong in-domain and out-of-domain geometry, superior 3D tracking, and more stable long-sequence colon reconstruction. A blinded user study further supports the perceived reconstruction quality on real clinical video. Together, these results demonstrate the value of large-scale endoscopic supervision and motion constraints for joint geometry estimation and tissue tracking.

Overview

Method

SCOPE-4D builds on a feed-forward multi-view transformer. Geometric fine-tuning on SCOPE-5K supervises camera and dense depth. CRM represents each 3D point's motion as a shared common motion plus a local residual. Together with geometric supervision, CRM and trajectory supervision further improve camera and depth estimation over geometric fine-tuning alone while enabling dense 3D tissue tracking.

SCOPE-5K data

Real endoscopic video rarely comes with geometry. Our pipeline (panel a of the method figure) turns raw videos into clips with camera and depth labels: screening and clipping, per-clip structure-from-motion with tracking-based correspondences, dense labels from a camera-conditioned geometry model, and an automatic quality-control stage that accepts, flags or rejects each clip.

Data composition

Results

Geometry and camera stability Hamlyn, out-of-domain

3D tissue tracking StereoMIS

Liver breathing

Instrument pulling tissue

Long-sequence colon reconstruction RealSynCol, 701 views

Interactive demo

Comparison

Predicted trajectory GT trajectory

Quantitative results

Ablation: training data

Training data	In-domain average				Hamlyn (OOD)		SCOPE-colon-phantom (OOD)
	Pose		Depth		Depth		Pose
	AUC@5°↑	AUC@30°↑	AbsRel↓	δ1.25↑	AbsRel↓	δ1.25↑	RTE↓	ARE*↓
No finetune (VGGT-Ω weights)	9.55	53.03	0.077	94.03	0.135	80.82	10.79	8.22
SC+SIM+C3VD+EM-S+SA	35.99	74.08	0.031	99.50	0.143	80.93	11.28	7.12
+ StereoMIS	33.10	70.02	0.032	99.53	0.142	81.40	10.51	6.83
+ EndoMapper-Real	36.62	74.07	0.032	99.57	0.128	83.04	10.37	7.37
Full dataset	36.02	73.68	0.030	99.59	0.129	83.77	10.71	7.37

Data composition, geometric fine-tuning only. Hamlyn depth excludes instruments. SC: SCARED, SIM: SimCol3D, EM-S: EndoMapper-Sim, SA: Sano.

Ablation: motion representation

Variant	Tracking		Pose		Depth
	EPE↓	APD↑	AUC@5°↑	AUC@30°↑	AbsRel↓	δ1.25↑
SoM (baseline)	5.94	46.58	5.67	26.86	0.050	98.82
w/ zero-motion anchor	5.30	50.11	6.09	27.17	0.048	98.82
Full CRM	5.27	52.63	6.96	27.66	0.049	98.85

Long-sequence reconstruction

Method	ATE↓	RTE↓	RRE↓
VGGT	93.21	23.65	15.11
VGGT-Ω	85.92	21.67	31.21
SM4RT	88.95	21.29	16.98
Endo3R	77.87	14.75	22.92
SCOPE-4DSFT (Ours)	73.52	10.38	13.38
SCOPE-4D (Ours)	71.17	10.94	13.61

User study

Method	Camera trajectory↓	Point cloud↓	Depth map↓
VGGT-Ω	1.93 [1.65, 2.25]	1.95 [1.65, 2.28]	1.98 [1.67, 2.32]
SM4RT	2.90 [2.60, 3.08]	3.22 [2.80, 3.57]	3.33 [2.87, 3.73]
Endo3R	3.83 [3.53, 4.00]	3.52 [3.23, 3.78]	3.37 [3.00, 3.70]
SCOPE-4D (Ours)	1.33 [1.12, 1.62]	1.32 [1.13, 1.55]	1.32 [1.13, 1.53]

SCOPE-colon-real: mean rank from six clinical and biomedical experts on ten clips (1 best, 4 worst), 95% bootstrap intervals.

Study interface