Video dataset for surgical phase, keypoint, and instrument recognition in laparoscopic surgery (PhaKIR)

Abstract

This data article describes the Surgical Procedure Phase, Keypoint, and Instrument Recognition (PhaKIR) dataset, a collection of eight complete laparoscopic cholecystectomy videos acquired during routine minimally invasive procedures at three German medical centers. The recordings were captured with different monocular endoscopic camera systems at 25 frames per second and a resolution of 1920 × 1080 pixels, with durations ranging from 28 to 58 min. Segments showing regions outside the abdominal cavity were removed for anonymization, and the corresponding cut indices are provided.

The dataset offers unified annotations for three interrelated tasks on the same complete surgical sequences: surgical phase recognition, annotated for every frame at 25 fps (485,875 frames), instrument keypoint estimation (19,435 frames), and instrument instance segmentation (19,435 frames), the latter two annotated at one frame per second (every 25th frame). Surgical phases follow the seven-phase Cholec80 scheme, extended by an undefined label for transitional frames. The instrument annotations distinguish 19 instrument classes as well as individual instances of the same class, with keypoint annotations comprising two to four class-dependent points labeled with COCO-style visibility states. Segmentation and keypoint labels were created manually using the Computer Vision Annotation Tool (CVAT), whereas phase labels were derived from documented phase-transition timestamps, and all annotations underwent a multi-stage review by a medically trained team. The data are provided in open formats (MP4, CSV, JSON, PNG) together with a frame-extraction script under the CC BY-NC-SA license and are available through controlled access upon request via the Zenodo platform.

Because the dataset combines procedural context, instrument pose, and pixel-accurate instance segmentations within full-length recordings from multiple institutions, it can be reused for single-task or multi-task model development, for temporally aware approaches that exploit motion continuity, and for cross-institutional generalization protocols such as leave-one-hospital-out evaluation. The dataset served as the training resource for the PhaKIR Challenge at the Endoscopic Vision (EndoVis) Challenge at MICCAI 2024.

Specifications Table

Subject	Computer Sciences
Specific subject area	Surgical phase, keypoint, and instrument recognition in laparoscopic surgery
Type of data	Video (MP4), Image (PNG), Table (CSV), Annotation file (JSON). Raw (surgical videos) and processed (extracted frames, segmentation masks, keypoint and phase annotations).
Data collection	Eight complete laparoscopic cholecystectomy videos were recorded during routine minimally invasive procedures on adult patients at three German hospitals. Recordings were acquired using different monocular endoscopic camera systems at 25 fps and 1920 × 1080 resolution. Sequences captured outside the abdominal cavity were removed for anonymization. Instrument instance segmentation and keypoint annotations were created manually using the Computer Vision Annotation Tool (CVAT). Surgical phase labels were derived from documented phase-transition timestamps.
Data source location	Data were collected at three medical centers in Germany: TUM University Hospital Rechts der Isar, Munich, Germany (n = 6); Heidelberg University Hospital, Heidelberg, Germany (n = 1); and Weilheim-Schongau Hospital, Weilheim, Germany (n = 1).
Data accessibility	Repository name: PhaKIR Dataset – Surgical Procedure Phase, Keypoint, and Instrument Recognition. Data identification number: 10.5281/zenodo.15740619 Direct URL to data: https://doi.org/10.5281/zenodo.15740619. Instructions: The dataset is available through controlled access upon request via the DOI link above. Requests are processed by the corresponding authors (see “Access conditions and licensing”).
Related research article	T. Rueckert, D. Rauber, R. Maerkl, L. Klausmann, S.R. Yildiran, M. Gutbrod, et al., Comparative validation of surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation in endoscopy: Results of the PhaKIR 2024 challenge, Med. Image Anal. 109 (2026) 103945. https://doi.org/10.1016/j.media.2026.103945

1. Value of the Data

• •To the authors' knowledge, PhaKIR is the first multi-institutional dataset that provides frame-level annotations for surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation on identical and complete laparoscopic cholecystectomy sequences, enabling the joint study of instrument type, spatial location, instance identity, and procedural context within a single resource.
• •The dataset contains eight complete cholecystectomy procedures rather than isolated frames or short clips. Phase labels cover every frame at 25 fps (485,875 frames), while instance segmentation and keypoint labels are provided at 1 fps (19,435 frames per task), with all annotation types referring to the same complete sequences. This supports the development and benchmarking of temporally aware models that exploit motion continuity and long-range workflow dependencies.
• •Recordings were acquired at three German medical centers using different endoscopic camera systems, introducing inter-institutional variability in imaging conditions, instrumentation, and surgical practice. This composition allows researchers to assess model generalization across institutions, for example through leave-one-hospital-out or leave-one-video-out protocols defined at the video level.
• •The unified annotation scheme supports both single-task and multi-task learning. Researchers can train and evaluate individual models for phase recognition, instrument instance segmentation, or keypoint estimation, jointly optimize all three tasks, or reproduce the experimental setup of the PhaKIR Challenge at MICCAI 2024, for which the dataset served as the training resource.
• •The annotations distinguish 19 instrument classes as well as individual instances of the same class, while the keypoint annotations comprise two to four class-dependent points (instrument tip, shaft-to-tip junction, and entry endpoint) labeled with COCO-style visibility states (visible, occluded, not available). These annotations can be reused to investigate instance-level recognition, occlusion-aware pose estimation, and the association between instrument presence and surgical phase.
• •The dataset is released in open, FAIR-aligned formats (MP4, CSV, JSON, PNG) under the CC BY-NC-SA license and is available through controlled access upon request via the Zenodo platform. It can be combined with complementary public resources, such as Cholec80 for additional phase annotations or EndoVis segmentation datasets, supporting cross-dataset evaluation, transfer learning, and studies on domain shift across institutions and recording setups. The standardized file layout and the versioned repository record furthermore allow the resource to be extended, for example with recordings from further institutions or additional procedure types, without breaking compatibility with existing references.

2. Background

This data article documents the Surgical Procedure Phase, Keypoint, and Instrument Recognition (PhaKIR) dataset, compiled for the PhaKIR sub-challenge organized within the Endoscopic Vision (EndoVis) Challenge at MICCAI 2024 and reported in the companion research article [1]. The challenge addressed a recurring limitation in surgical data science: methods for instrument recognition and surgical workflow analysis are typically trained and evaluated on datasets that target a single task, omit complete procedure sequences, and originate from a single medical center, which constrains the study of temporal context and cross-institutional generalization [[2], [3], [4], [5], [6], [7], [8]]. A tabular comparison of the related challenge datasets is provided in Table 1 of the companion research article [1], and a comprehensive review of the datasets available in this field is given in [2]. The dataset was therefore designed to combine three interrelated tasks — surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation — within full-length laparoscopic cholecystectomy recordings from several institutions. Whereas the companion article [1] presents the challenge design and the comparative evaluation of the submitted methods, the present article focuses on the data resource itself. It provides a detailed account of the acquisition setup, the annotation protocols for all three tasks, the ground-truth generation, the file organization, and the access conditions, thereby supporting transparent reuse of the data independently of the original challenge.

3. Data Description

The PhaKIR dataset comprises eight complete laparoscopic cholecystectomy videos recorded at three medical centers, accompanied by frame-level annotations for three interrelated tasks: surgical phase recognition, instrument keypoint estimation, and instrument instance segmentation. An overview of the source video data and the corresponding annotations for the three tasks across the three contributing centers is shown in Fig. 1. All videos were acquired at 25 frames per second (fps) and a resolution of 1920 × 1080 pixels, with durations between 28 and 58 min. Surgical phase labels are provided for every frame (485,875 frames in total), whereas instrument keypoint and instance segmentation annotations are provided for every 25th frame, corresponding to 1 fps (19,435 annotated frames per task). All annotation types cover the identical, complete surgical sequences. The source hospital, duration, raw frame count, and per-task annotation count of each video are listed in Table 1; grouping these entries by hospital yields the corresponding per-center figures. The dataset is available through controlled access upon request via the Zenodo platform [9]. The access procedure is described in the section "Publication and access".

Fig. 1.

Overview of the PhaKIR dataset, illustrating the source video data and the annotations for the three tasks — surgical phase recognition, instrument instance segmentation, and instrument keypoint estimation — across the three contributing medical centers.

Table 1.

Video numbers together with their source hospital, duration in minutes, number of raw frames, and number of annotated frames for the phase recognition task as well as for the instrument instance segmentation and keypoint estimation tasks. Hospital abbreviations: MRI = TUM University Hospital Rechts der Isar, Munich; UKHD = Heidelberg University Hospital; KWS = Weilheim-Schongau Hospital. Videos are not numbered consecutively because the intermediate recordings were reserved for the test set.

Video	1	2	3	4	5	7	11	13	Total
Hospital	MRI	MRI	MRI	MRI	MRI	MRI	UKHD	KWS
Duration (min)	39:07	48:12	32:39	49:10	28:31	57:03	32:54	36:19	323:55
#Frames (raw)	58,675	72,300	48,975	73,750	42,775	85,575	49,350	54,475	485,875
#Annotated (phase)	58,675	72,300	48,975	73,750	42,775	85,575	49,350	54,475	485,875
#Annotated (seg., kp.)	2347	2892	1959	2950	1711	3423	1974	2179	19,435

The dataset archive is organized at the root level into eight video archives — Video_01.zip, Video_02.zip, Video_03.zip, Video_04.zip, Video_05.zip, Video_07.zip, Video_11.zip, and Video_13.zip. The archives are not numbered consecutively, as the intermediate recordings were reserved for the (non-public) PhaKIR test set. The frame-extraction script split_video_in_frames.py is provided separately in a dedicated GitHub repository (https://github.com/remic-othr/PhaKIR_Dataset) rather than within the archive, so that updates to the script do not trigger a new dataset version. The complete directory layout is shown in Fig. 2. Each video archive follows an identical internal structure and contains five files, where the placeholder xx denotes the two-digit video identifier:

• •Video_xx_Cuts.csv — frame indices marking the segments removed during anonymization, aligned with the video frame numbering;
• •Video_xx.mp4 — the surgical video at 25 fps and 1920 × 1080 pixels;
• •Video_xx_Keypoints.json — instrument keypoint annotations together with their visibility states;
• •Video_xx_Phases.csv — frame-level surgical phase labels;
• •Video_xx_Masks.zip — instrument instance segmentation masks stored as PNG images.

Fig. 2.

Structure of the dataset. The eight video archives are displayed on the left, the middle column shows the individual elements of each video archive, and the fine-grained folder structure for the extracted frames and masks is displayed on the right. The elements marked in blue are generated after frame extraction using the frame-extraction script, which is hosted separately in a GitHub repository (https://github.com/remic-othr/PhaKIR_Dataset).

The content, format, and naming conventions of these files are summarized in Table 2.

Table 2.

Summary of the files contained in each video archive, their content, format, and conventions.

Filename	Content	Format	Notes
Video_xx_Cuts.csv	Anonymization cut indices	CSV	Aligns with video frame numbering
Video_xx.mp4	Surgical video	MP4	25 fps, 1920 × 1080; 1000-frame subfolders; zero-padded filenames.
Video_xx_Keypoints.json	Keypoint annotations	JSON	Per frame → per instrument → coordinates + visibility.
Video_xx_Phases.csv	Surgical phase labels	CSV	Frame index + phase label (7 phases + undefined).
Video_xx_Masks.zip	Segmentation masks	ZIP (PNG)	Subfolders of 1000; filenames match frames; color encoding in Table 4.

Individual frames are obtained by applying split_video_in_frames.py to Video_xx.mp4. Extracted frames are written to subfolders of 1000 images each (named 0, 1000, …), with filenames zero-padded to six digits (e.g., frame_055649.png). The segmentation masks contained in Video_xx_Masks.zip follow the same subfolder organization of 1000 files each, with filenames composed of the corresponding six-digit frame index and the suffix _mask.png (e.g., 055650_mask.png). Frame indices are zero-based throughout the dataset: the first frame of a video has index 0, and the same index is used in the extracted frame filenames, the mask filenames, the phase CSV files, the keypoint JSON files, and the cut-index CSV files, so that all annotation types can be aligned by frame index without any offset. All data are provided in open formats (CSV, JSON, PNG, MP4).

Each Video_xx_Phases.csv lists, for every frame, the frame index and the assigned phase label. The phase labels comprise the seven phases defined in the Cholec80 dataset [10] — Preparation, Calot Triangle Dissection, Clipping and Cutting, Gallbladder Dissection, Gallbladder Packaging, Cleaning and Coagulation, and Gallbladder Retraction — together with an additional undefined label for transitional frames. Frames carrying this label were excluded from the evaluation in the PhaKIR challenge [1]; we recommend adopting this convention for results intended to be comparable with the challenge. The relative duration of each phase per video is shown in Fig. 3.

Fig. 3.

Relative duration of each surgical phase per video. The phases are arranged in order of their most frequent occurrence across all recordings.

Each Video_xx_Keypoints.json stores keypoint annotations in JSON format, structured per frame and, within each frame, per instrument instance, with each entry providing the keypoint coordinates and visibility state. The number of keypoints is class-dependent and ranges from two to four per instrument: End Point (EP), Shaft Point (SP), Tip 1 (T1), and Tip 2 (T2). Each keypoint additionally carries a visibility state following the COCO protocol [11] (visible, occluded, or not available); keypoints that are not applicable to a given instrument are marked as not available. The number and type of keypoints associated with each instrument class are listed in Table 3.

Table 3.

Surgical instruments grouped by the number and type of annotated keypoints. Keypoint labels are abbreviated as T1 (Tip 1), T2 (Tip 2), SP (Shaft Point), and EP (End Point).

Keypoints	Instruments
4 Keypoints (T1, T2, SP, EP)	Bipolar-Clamp, Blunt-Grasper, Blunt-Grasper-Curved, Blunt-Grasper-Spec., Clip-Applicator, Grasper, Hook-Clamp, Overholt, PE-Forceps, Scissor, Sponge-Clamp
3 Keypoints (T1, SP, EP)	Argonbeamer, Dissection-Hook, HF-Coag.-Probe, Suction-Rod
2 Keypoints (T1, EP)	Drainage, Needle-Probe, Palpation-Probe, Trocar-Tip

The instrument instance segmentation masks are encoded as three-channel PNG images. The red and green channels encode the instrument class, while the blue channel encodes the instrument instance; instances of the same class therefore share identical red and green values and differ only in the blue channel. The per-class RGB codes are listed in Table 4.

Table 4.

Designation of the surgical instrument types together with the RGB color codes used in the segmentation masks and a visualization of the respective color, sorted in ascending alphabetical order by instrument name. The R and G values encode the instrument class; for the B channel, the values of the instances appearing first in a video for an instrument are shown.

The dataset distinguishes 19 instrument classes. The appearance of the individual instrument types is depicted in Fig. 4, and the number of frames in which each instrument type occurs is shown in Fig. 5.

Fig. 4.

Instrument types included in the dataset. Where more than one instrument is present in an image, the instruments are listed from left to right in the order in which they appear: Grasper, PE-Forceps (a); Clip-Applicator (b); Scissor (c); Trocar-Tip, Suction-Rod, Palpation-Probe, HF-Coag.-Probe (d); Needle-Probe (e); Argonbeamer (f); Blunt-Grasper-Spec., Bipolar-Clamp (g); Blunt-Grasper, Blunt-Grasper-Curved (h); Blunt-Grasper-Spec., Dissection-Hook, Trocar-Tip (i); Sponge-Clamp (j); Drainage (k); Overholt (l).

Fig. 5.

Number of frames in which each instrument type occurs, sorted in descending order of frequency. The counts refer to the eight recordings released here; the corresponding counts for the withheld test recordings are reported in [1].

4. Experimental Design, Materials and Methods

The PhaKIR dataset was created following the four-stage workflow summarized in Fig. 6: acquisition of the surgical videos, manual annotation of the three tasks, multi-stage validation of the annotations, and publication of the dataset under a controlled-access procedure. The subsections below follow this order and specify the tools, software, and scripts used in each stage.

Fig. 6.

Creation pipeline of the PhaKIR dataset, from video acquisition and anonymization through manual annotation of the three tasks and multi-stage validation to publication under controlled access. The lower row indicates the data artifacts resulting from each stage.

4.1. Data acquisition

A total of n = 8 videos were recorded during routine minimally invasive cholecystectomies performed on adult patients of different genders. Recordings were acquired with different endoscopic cameras at 25 frames per second (fps) and a resolution of 1920 × 1080 pixels, with durations ranging from 28 to 58 min. The videos originated from three German hospitals: TUM University Hospital Rechts der Isar (MRI; n = 6), Heidelberg University Hospital (UKHD; n = 1), and Weilheim-Schongau Hospital (KWS; n = 1). The single Heidelberg recording corresponds to HeiChole2.mp4, originally released as part of the HeiChole benchmark [8], and was re-annotated for protocol consistency. Sequences depicting regions outside the abdominal cavity were removed during anonymization; the corresponding cut indices are provided for each video in Video_xx_Cuts.csv. The source hospital, duration, and per-task annotation counts of all videos are listed in Table 1.

4.2. Annotation

4.2.1. Annotation team

Annotations were created by four annotators with medical expertise: one senior surgeon and three medically trained students.

4.2.2. Frame extraction for annotation

For surgical phase recognition, every frame was annotated at the native rate of 25 fps, resulting in 485,875 labeled frames. For instrument instance segmentation and instrument keypoint estimation, one frame per second (every 25th frame) was annotated, resulting in 19,435 labeled frames per task. All three tasks cover the identical and complete surgical sequences.

4.2.3. Task 1 - surgical phase recognition

Each frame was assigned to one of the seven surgical phases defined in the Cholec80 dataset [10], with an additional undefined label to capture transitions. Phase-transition timestamps were documented during manual review of the full-length recordings, without timeline restrictions. The start of a phase was defined as the first appearance of the characteristic instruments performing the essential actions of that phase, and its end as the disappearance of all such instruments. The documented timestamps were converted into CSV files, and all frames between the start and end markers of a phase were automatically assigned to the corresponding label, yielding the frame-level phase annotations stored in Video_xx_Phases.csv. The Heidelberg recording was fully re-annotated according to this protocol.

4.2.4. Task 2 - instrument instance segmentation

Instruments were segmented in every 25th frame and assigned to one of 19 instrument classes, with each pixel labeled as either instrument or background, and multiple instances of the same class distinguished. Segmentation was performed with the Computer Vision Annotation Tool (CVAT) [12] using polygon contours; only the visible parts of the instruments were labeled. During annotation, different instances of the same class were temporarily separated using pseudo-classes to support ground-truth generation. The manual annotations were subsequently converted into segmentation masks by means of Python scripts. Each mask is encoded as a three-channel PNG image in which the red and green channels encode the instrument class and the blue channel encodes the instrument instance; instances of the same class therefore share identical red and green values and differ only in the blue channel. The per-class RGB codes are listed in Table 4, and the resulting masks are provided in Video_xx_Masks.zip.

4.2.5. Task 3 - instrument keypoint estimation

For every 25th frame, keypoints describing the instrument pose were annotated together with their visibility status. For each instrument, a set of predefined keypoints was specified: the End Point, where the instrument enters the image border; the Shaft Point, marking the shaft-to-tip junction; and the instrument tip, which for opening tools was represented by two separate points (Tip 1 and Tip 2). The number of keypoints was class-dependent and ranged from two to four per instrument; the number and type of keypoints associated with each instrument class are listed in Table 3. Keypoints were annotated in CVAT [12] using the point function. Each keypoint was assigned a visibility status following the COCO protocol [11] (visible, occluded, or not available), and keypoints not applicable to a given instrument were marked as not available. When placing the keypoints, the annotators had access to the full video sequences, including both preceding and subsequent frames; the temporal continuity and expected motion of the instruments were taken into account to refine the placement, particularly when the target region was partially occluded. The annotations were stored in JSON format, structured per frame and, within each frame, per instrument instance, in Video_xx_Keypoints.json.

4.3. Validation of Annotations

4.3.1. General approach

Prior to annotation, the team was provided with a detailed annotation protocol containing task-specific guidelines and illustrative examples, developed jointly with medical experts and refined during the annotation process. Each video was first annotated by a single annotator, who then re-inspected the annotations to correct potential oversights. Two additional team members subsequently verified the annotations sequentially according to the protocol, so that each video was reviewed three times after its initial annotation. The annotation protocol is published as Appendix B (Supplementary file S1) of the companion research article [1], which is openly accessible. Adjudication followed a sequential rather than a consensus scheme: each reviewer applied corrections directly to the annotations of the preceding pass, with the protocol serving as the binding reference, so that the state after the third review pass constitutes the released ground truth. The task-specific rules governing ambiguous cases are part of the protocol, which was refined during the annotation process whenever a case was not yet covered.

4.3.2. Task 1 - surgical phase recognition

The annotated timestamps were repeatedly checked following the procedure described above. In addition, plausibility checks were performed by verifying the presence of the expected instruments in the corresponding phases, as certain instrument classes occur only in specific phases. No quantitative correction statistics were recorded for this task.

4.3.3. Task 2 - instrument instance segmentation

The verification of the original annotations was supplemented by visual inspection. The annotations were visualized and combined side by side with the original frames into composite images, from which a correction video was generated and subjected to the three-stage validation procedure. Monitoring the color-coded instrument representations over time allowed inconsistencies to be detected, such as changes in the assigned class of an instrument between consecutive frames. Between the second and third review passes, correction rates were quantified using the multi-instance, multi-class Dice similarity coefficient (DSC) applied analogously to the PhaKIR challenge evaluation [1], which yielded an annotation agreement of 83.64%. The principal source of disagreement was the classification of instruments, for which a misclassification entered the result as a DSC of zero for the affected instrument instance.

4.3.4. Task 3 - instrument keypoint estimation

Keypoint annotations were likewise verified by visual inspection. As the keypoints were annotated in CVAT on top of the segmentation labels, the segmentations and their classifications were implicitly re-examined during this step. The keypoints were annotated by a team member who had not contributed to the original segmentations or to the segmentation correction runs. The input images were combined with the keypoint annotations, compiled into videos, and subjected to the three-stage validation procedure. No quantitative correction rates were tracked for this task; quality control relied on repeated visual inspection and temporal consistency checks under occlusion.

4.3.5. Community use during the challenge

The dataset served as the training resource of the PhaKIR challenge at MICCAI 2024 [1], where it was downloaded and inspected by 66 registered teams. Rare annotation errors identified and reported by participants during the challenge were corrected, and the dataset was updated accordingly. No systematic errors were reported during the challenge.

4.4. Publication and Access

4.4.1. Assembly of the released dataset

For each video, the raw video file, the cut indices, and the annotations for all three tasks were bundled into a single archive (Video_xx.zip). To reduce storage requirements, the complete videos were distributed rather than pre-extracted frames; a separate frame-extraction script (see below) allows users to control the frame compression. The resulting root-level organization of the dataset is shown in Fig. 2.

4.4.2. Access conditions and licensing

Because the recordings constitute sensitive surgical patient data, the dataset is not offered as an unrestricted download but as a restricted-access record on Zenodo [9]. Access is requested through the request function of the record, stating the requesting institution and the intended use. Institutional affiliation is recorded for documentation purposes but is not an approval criterion. Requests are granted if the stated purpose is non-commercial research consistent with the CC BY-NC-SA 4.0 license [13]; no separate data-use agreement is required beyond this statement. Requests are reviewed by the corresponding authors and are typically processed within one working day, after which approved users obtain download access to the complete dataset. Access control and licensing serve complementary purposes: the request step documents who receives the data, whereas the license defines the permitted reuse — non-commercial use only, citation of this article, the companion research article [1], and the HeiChole benchmark [8], and distribution of derived material under identical terms. Incomplete requests are not declined outright but returned for clarification and may be resubmitted.

A single access request covers the complete record, that is, all video files, phase annotations, keypoint annotations, segmentation masks, and cut indices; no individual component is subject to a separate request. The frame-extraction script is not part of the restricted record and is available without registration from the public GitHub repository (see "Source code and frame extraction"). Apart from this script, no further software is required to use the dataset; the evaluation metrics and the evaluation protocol applied in the PhaKIR challenge are defined in the companion research article [1]. The dataset is archived on Zenodo, which provides persistent identification and long-term availability. The DOI cited in this article is the concept DOI, which always resolves to the most recent version of the record. Individual versions additionally receive their own DOI, so that revised or extended releases can be published without invalidating existing references.

4.4.3. Source code and frame extraction

The frame-extraction script split_video_in_frames.py is provided in a dedicated GitHub repository (https://github.com/remic-othr/PhaKIR_Dataset) rather than within the Zenodo archive, so that updates to the script do not trigger a new dataset version. It splits the video sequences into individual frames and allows the compression rate of the resulting frames to be controlled. The script is written in Python and requires no external dependencies beyond the standard library. Applied to Video_xx.mp4, it writes the extracted frames into subfolders of 1000 images each, with filenames zero-padded to six digits (e.g., frame_055649.png); the segmentation masks in Video_xx_Masks.zip follow the same subfolder organization and filename convention (e.g., 055650_mask.png). The script is released under the same license as the dataset (CC BY-NC-SA 4.0) [13].

Limitations

Several limitations of the dataset should be considered. First, the number of procedures is limited to eight complete recordings, which may not capture the full range of variability present in surgical practice. Second, all recordings depict laparoscopic cholecystectomies, so other types of surgical procedures are not represented. Third, the data originate exclusively from German medical centers, and surgical practices for cholecystectomy may differ between countries. Fourth, the dataset contains only RGB video; no further multimodal information (e.g., depth, kinematic, or sensor data) is included. Fifth, the frequency of occurrence of the instrument classes is highly imbalanced, with some classes appearing in several thousand frames and others in fewer than one hundred. Sixth, while surgical phase labels are provided for every frame, instrument keypoint and instance segmentation annotations are available only at one frame per second (every 25th frame). Finally, for the instrument instance segmentation task, only the visible parts of the instruments were annotated.

These limitations also delineate the directions in which the resource can be extended. The most immediate one is a broader institutional basis: in the PhaKIR challenge, the submitted methods performed consistently worse on the test recordings from the two centers that were less represented in the training data, which was attributed to the predominance of a single center in that data [1]. Extending the dataset with recordings from additional centers, including centers outside Germany, and with a more balanced number of procedures per center would therefore allow leave-one-hospital-out protocols to be evaluated with several recordings per held-out institution. A second direction is the inclusion of further laparoscopic procedure types beyond cholecystectomy. The annotation protocol, the mask encoding, and the keypoint definitions are not specific to cholecystectomy and can be transferred to other interventions, requiring only an extension of the instrument vocabulary, so that cross-procedure transfer and domain shift could be studied within a consistent annotation scheme. Beyond these two aspects, the targeted acquisition of sequences containing rarely observed instrument classes would mitigate the class imbalance, a higher annotation frequency for the segmentation and keypoint tasks would support dense temporal modeling and instrument tracking, and complementary modalities such as depth or kinematic data would broaden the range of addressable research questions.

CRediT Author Statement

Tobias Rueckert: Conceptualization, Methodology, Software, Validation, Formal analysis, Data curation, Writing – original draft, Writing – review & editing, Visualization, Project administration. Raphaela Maerkl: Conceptualization, Methodology, Validation, Data curation. David Rauber: Conceptualization, Methodology, Software, Validation, Data curation. Leonard Klausmann: Conceptualization, Methodology, Software, Validation, Resources. Max Gutbrod: Conceptualization, Methodology, Software, Validation, Data curation. Daniel Rueckert: Conceptualization, Methodology, Writing – review & editing, Supervision. Hubertus Feussner: Conceptualization, Methodology, Data curation, Writing – review & editing, Supervision, Funding acquisition. Dirk Wilhelm: Conceptualization, Methodology, Data curation, Writing – review & editing, Supervision, Funding acquisition. Christoph Palm: Conceptualization, Methodology, Writing – original draft, Writing – review & editing, Supervision, Project administration, Funding acquisition.

Ethics Statement

The dataset comprises endoscopic video recordings acquired during routine laparoscopic cholecystectomies. The underlying research project was approved by the institutional ethics committee of the Technical University of Munich (TUM; approval code 337/21 S-EB) and conducted in accordance with the Declaration of Helsinki. Informed consent was obtained from all participating patients for the anonymized use of their recordings for research purposes.

Only fully anonymized data were released. The recordings capture exclusively the intra-abdominal surgical field; no patient-identifying information, demographic records, or other clinical metadata were collected or stored. To further protect the anonymity of patients and operating-room staff, all segments recorded outside the abdominal cavity were removed prior to publication, with the corresponding frame indices documented in the Video_xx_Cuts.csv files. The released data therefore contain no personal data within the meaning of applicable data-protection regulations.

The authors have read and follow the ethical requirements for publication in Data in Brief.

Declaration of Generative AI and AI-Assisted Technologies in the Manuscript Preparation Process

During the preparation of this work the authors used Claude (Claude Opus 4.8, Anthropic) to assist with language refinement and to improve the clarity and readability of the manuscript. The tool was used exclusively for linguistic and stylistic purposes and not for the generation of scientific content, data, analyses, or interpretations. After using this tool, the authors carefully reviewed and verified the entire manuscript, edited the content as needed, and take full responsibility for the content of the publication.

Data Availability

• ZenodoPhaKIR Dataset - Surgical Procedure Phase, Keypoint, and Instrument Recognition (Original data)