Transformers 中 SAM3 Video 实战：文本提示驱动的视频可提示概念分割（PCS）与检测-跟踪融合架构解析

Transformers 中 SAM3 Video 实战文本提示驱动的视频可提示概念分割PCS与检测-跟踪融合架构解析【免费下载链接】transformers Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.项目地址: https://gitcode.com/GitHub_Trending/tra/transformersSAM3 Video 是 Transformers 库中将 Meta 的 SAM 3Segment Anything Model 3用于视频场景的模型实现输入一句文本提示如 yellow school bus它会对视频中所有匹配该概念的目标实例给出分割掩码、边界框与置信度并在全片范围内保持每个实例的身份一致。本文以官方模型文档 sam3_video.md 为主体完整覆盖预加载视频、多提示、流式推理与自定义分辨率四类用法并深入 modeling_sam3_video.py、configuration_sam3_video.py、processing_sam3_video.py 三个核心源文件剖析检测器 跟踪器共享骨干的检测-跟踪融合流水线、Hotstart 去重机制与设备管理策略。一、概述Promptable Concept SegmentationPCSSAM3 Video 对应的模型为Sam3VideoModel与Sam3VideoProcessor模型文档见 docs/source/en/model_doc/sam3_video.md。它执行的核心任务是论文《SAM 3: Segment Anything with Concepts》定义的Promptable Concept SegmentationPCS可提示概念分割输入短名词短语如 yellow school bus、图像示例或二者组合输出所有匹配目标实例的分割掩码与唯一身份 ID跨帧保持。从源码结构看模型正是把识别detection与跟踪tracking拆成两个子模型组合而成Sam3VideoModel.__init__modeling_sam3_video.py#L507-L543中self.detector_model AutoModel.from_config(config.detector_config) # SAM3 检测器 self.tracker_model AutoModel.from_config(config.tracker_config, remove_vision_encoderTrue) # SAM2 风格跟踪器tracker_config的视觉编码器被显式移除remove_vision_encoderTrue后续由检测器的骨干特征经 FPN 颈部Sam3VisionNeck供给跟踪器见get_vision_features_for_trackermodeling_sam3_video.py#L545-L565——这正是论文摘要中图像级检测器与基于记忆的图像跟踪器共享单一骨干share a single backbone的落地方式。同时Sam3VideoPreTrainedModel声明了_supports_sdpa True、_supports_flash_attn True、_supports_flex_attn Truemodeling_sam3_video.py#L490-L503对应文档中 SDPA 与 FlashAttention 徽章。论文摘要引自文档We present Segment Anything Model (SAM) 3, a unified model that detects, segments, and tracks objects in images and videos based on concept prompts… Recognition and localization are decoupled with a presence head, which boosts detection accuracy. We open source SAM 3 along with our new Segment Anything with Concepts (SA-Co) benchmark for promptable concept segmentation.其中presence head在实现中体现为检测阶段的得分计算pred_probs pred_logits.sigmoid() * presence_logits.sigmoid()modeling_sam3_video.py#L607-L612即定位分支的预测概率与概念是否在场分支的得分相乘把识别什么与定位在哪解耦。二、快速上手预加载视频推理适用前提模型以facebook/sam3为 Hub checkpoint需要已安装 torch、torchvision处理器依赖requires(backends(torch, torchvision))见 processing_sam3_video.py#L37-L39以及 transformers 中的视频工具load_video。官方文档给出的完整示例Pre-loaded Video Inference如下对已加载全部帧的视频使用文本提示做检测跟踪from transformers import Sam3VideoModel, Sam3VideoProcessor import torch model Sam3VideoModel.from_pretrained(facebook/sam3, device_mapauto) processor Sam3VideoProcessor.from_pretrained(facebook/sam3) # 补充文档示例中的 device 变量需要由用户自行指定 device torch.device(cuda if torch.cuda.is_available() else cpu) # Load video frames from transformers.video_utils import load_video video_url https://huggingface.co/datasets/hf-internal-testing/sam2-fixtures/resolve/main/bedroom.mp4 video_frames, _ load_video(video_url) # Initialize video inference session inference_session processor.init_video_session( videovideo_frames, inference_devicedevice, processing_devicecpu, video_storage_devicecpu, ) # Add text prompt to detect and track objects text person inference_session processor.add_text_prompt( inference_sessioninference_session, texttext, ) # Process all frames in the video outputs_per_frame {} # Pass show_progress_barTrue to display a tqdm progress bar. for model_outputs in model.propagate_in_video_iterator( inference_sessioninference_session, max_frame_num_to_track50 ): processed_outputs processor.postprocess_outputs(inference_session, model_outputs) outputs_per_frame[model_outputs.frame_idx] processed_outputs print(fProcessed {len(outputs_per_frame)} frames) # Processed 51 frames # Access results for a specific frame frame_0_outputs outputs_per_frame[0] print(fDetected {len(frame_0_outputs[object_ids])} objects) print(fObject IDs: {frame_0_outputs[object_ids].tolist()}) print(fScores: {frame_0_outputs[scores].tolist()}) print(fBoxes shape (XYXY format, absolute coordinates): {frame_0_outputs[boxes].shape}) print(fMasks shape: {frame_0_outputs[masks].shape})要点拆解结合源码init_video_sessionprocessing_sam3_video.py#L133-L184先经video_processor把视频统一预处理缩放、归一化拿到pixel_values_videos与原始尺寸original_sizes再封装进Sam3VideoInferenceSession。三个设备参数分工明确inference_device放计算张量、video_storage_device放视频帧、inference_state_device放推理状态默认等于 inference_device。add_text_promptprocessing_sam3_video.py#L101-L131把文本 tokenizepaddingmax_length, max_length32后存入 session重复文本会复用既有prompt_id不会重复编码。propagate_in_video_iteratormodeling_sam3_video.py#L1791-L1842生成器逐帧调用model(inference_session..., frame_idx...)支持start_frame_idx、max_frame_num_to_track不传则跟踪全片、reverse反向传播与show_progress_bar参数。postprocess_outputsprocessing_sam3_video.py#L247-L371把跟踪器输出的低分辨率掩码low_res_mask_size默认 288×288双线性插值到视频原始分辨率并二值化用masks_to_boxes得到 XYXY 绝对坐标框再过滤零面积掩码与 hotstart 阶段被移除的对象最后做按提示分组的非重叠约束同一提示内每个像素只归属一个物体。返回字典含五个键object_ids、scores、boxes、masks、prompt_to_obj_ids。多提示并行一次前向处理多个概念文档指出可以一次性给出多个提示或直接向add_text_prompt传列表模型会跨所有提示复用视觉特征——从源码看_det_track_one_frame先对每帧计算一次get_vision_featuresmodeling_sam3_video.py#L1613-L1621run_detection内所有提示共享这一份视觉嵌入仅重复轻量级的文本匹配分支因此多提示的边际成本很低# Add multiple text prompts (or use a list in add_text_prompt) multi_prompt_session processor.init_video_session( videovideo_frames, inference_devicedevice, processing_devicecpu, video_storage_devicecpu, ) prompts [person, bed, lamp] processor.add_text_prompt(multi_prompt_session, prompts) # Process video - detects objects from ALL prompts in a single pass multi_outputs_per_frame {} # Pass show_progress_barTrue to display a tqdm progress bar. for model_outputs in model.propagate_in_video_iterator( inference_sessionmulti_prompt_session, max_frame_num_to_track50 ): processed_outputs processor.postprocess_outputs(multi_prompt_session, model_outputs) multi_outputs_per_frame[model_outputs.frame_idx] processed_outputs # Check which objects were detected by each prompt frame_0_outputs multi_outputs_per_frame[0] prompt_to_obj_ids frame_0_outputs[prompt_to_obj_ids] for prompt, obj_ids in prompt_to_obj_ids.items(): print(f{prompt}: {len(obj_ids)} objects) # person: 2 objects # bed: 1 objects # lamp: 1 objects多提示的隔离机制值得注意检测关联函数_associate_det_trk通过det_prompt_ids/trk_prompt_ids把跨提示的 IoU 置零防止person的检测去匹配lamp的已有轨迹modeling_sam3_video.py#L682-L714而各提示的检测输出会先由_merge_detections_from_prompts合并成一帧统一检测集再交给跟踪器modeling_sam3_video.py#L1551-L1596。三、流式视频推理Streaming Inference官方文档在流式小节中有一段重要的质量警告必须完整继承⚠️关于流式推理质量流式推理会禁用 hotstart 启发式移除未匹配与重复对象的逻辑因为这些逻辑需要访问未来帧才能做出可靠判断。相比预加载视频推理流式模式可能出现更多误检与重复轨迹。若所有帧均可用优先使用预加载视频推理。在源码中可验证这一点_det_track_one_frame携带streaming标志forward以streamingframe is not None判定是否处于流式模式modeling_sam3_video.py#L1695-L1730该标志传入run_tracker_update_planning_phase影响更新计划例如不执行需要未来帧的去重判断。适用于实时/边到达边处理的应用场景完整示例# Initialize session for streaming不提供 video 参数 streaming_inference_session processor.init_video_session( inference_devicedevice, processing_devicecpu, video_storage_devicecpu, ) # Add text prompt text person streaming_inference_session processor.add_text_prompt( inference_sessionstreaming_inference_session, texttext, ) # Process frames one by one (streaming mode) streaming_outputs_per_frame {} for frame_idx, frame in enumerate(video_frames[:50]): # Process first 50 frames # First, process the frame using the processor inputs processor(imagesframe, devicedevice, return_tensorspt).to(model.device) # Process frame using streaming inference - pass the processed pixel_values model_outputs model( inference_sessionstreaming_inference_session, frameinputs.pixel_values[0], # Provide processed frame - this enables streaming mode reverseFalse, ) # Post-process outputs with original_sizes for proper resolution handling processed_outputs processor.postprocess_outputs( streaming_inference_session, model_outputs, original_sizesinputs.original_sizes, # Required for streaming inference ) streaming_outputs_per_frame[frame_idx] processed_outputs if (frame_idx 1) % 10 0: print(fProcessed {frame_idx 1} frames...) print(f✓ Streaming inference complete! Processed {len(streaming_outputs_per_frame)} frames) # ✓ Streaming inference complete! Processed 50 frames # Access results frame_0_outputs streaming_outputs_per_frame[0] print(fDetected {len(frame_0_outputs[object_ids])} objects in first frame) print(fBoxes are in XYXY format (absolute pixel coordinates): {frame_0_outputs[boxes].shape}) print(fMasks are at original video resolution: {frame_0_outputs[masks].shape})流式与预加载模式的关键差异对照源码维度预加载视频流式会话初始化init_video_session(video...)整段视频存入 session不传video逐帧model(frame...)时由add_new_framemodeling_sam3_video.py#L384-L398动态追加帧坐标直接给frame_idx传frame张量即进入流式模式forward中frame is not None后处理尺寸session 已有video_height/video_width必须显式传original_sizesinputs.original_sizes否则postprocess_outputs抛ValueErrorprocessing_sam3_video.py#L280-L294Hotstart 去重启用缓冲hotstart_delay帧输出后剔除未匹配/重复轨迹禁用需要未来帧四、自定义分辨率推理⚠️性能提示原文档自定义分辨率可能降低精度。模型设计工作分辨率为1008px。在需要更快推理或更低显存占用时可以按官方示例同时修改 config 与 processorconfig Sam3VideoConfig.from_pretrained(facebook/sam3) config.image_size 560 model Sam3VideoModel.from_pretrained(facebook/sam3, configconfig, device_mapauto) processor Sam3VideoProcessor.from_pretrained(facebook/sam3, size{height: 560, width: 560})从 configuration_sam3_video.py#L165-L174 看image_size是一个代理属性property def image_size(self): return self.detector_config.image_size image_size.setter def image_size(self, value): Recursively propagate the image size to detector and tracker configs. self.detector_config.image_size value self.tracker_config.image_size value即设置config.image_size 560会自动级联到嵌套的检测器与跟踪器两个子配置这正是文档示例只需改一行的原因Sam3VideoConfig的is_composition Truesub_configs声明了detector_config与tracker_config两个AutoConfig槽位configuration_sam3_video.py#L105-L113。而 processor 侧的size决定视频预处理时的缩放目标二者必须保持一致。五、Sam3VideoConfig全部跟踪参数详解Sam3VideoConfigconfiguration_sam3_video.py是组合配置 跟踪超参二合一除两个子配置外集中定义了检测-跟踪融合流水线的所有阈值。默认值均来自源码字段声明configuration_sam3_video.py#L112-L135参数默认值说明detector_config默认Sam3Config检测器配置为None时自动按CONFIG_MAPPING[sam3]()初始化tracker_config默认Sam3TrackerVideoConfig跟踪器配置为None时自动按CONFIG_MAPPING[sam3_tracker_video]()初始化initializer_range0.02截断正态初始化权重矩阵的标准差low_res_mask_size288跟踪器低分辨率掩码边长后续上采样到视频分辨率score_threshold_detection0.5检测得分保留阈值低于该值的检出被丢弃det_nms_thresh0.1检测 NMS 的 IoU 阈值≤0 时关闭 NMS见run_detection中run_nms self.det_nms_thresh 0.0assoc_iou_thresh0.1检测-轨迹匹配的宽松 IoU 阈值匹配上的判定trk_assoc_iou_thresh0.5判定 masklet 未匹配任何检测的严格 IoU 阈值new_det_thresh0.7检出要成为新对象需要的最低得分recondition_on_trk_masksTrue重条件化用跟踪掩码True还是检测掩码False跟踪掩码质量更高时检测器作为校验信号强化记忆、抑制漂移hotstart_delay15hotstart 阶段缓冲输出的帧数hotstart_unmatch_thresh8hotstart 期间轨迹连续未匹配达到该帧数即被移除hotstart_dup_thresh8hotstart 期间重复轨迹重叠达到该帧数即被去重suppress_unmatched_only_within_hotstartTrue是否只在 hotstart 期内抑制未匹配掩码init_trk_keep_alive30新轨迹的初始 keep-alive 计数max_trk_keep_alive30持续匹配的轨迹 keep-alive 计数上限min_trk_keep_alive-1未匹配时 keep-alive 计数的下限suppress_overlapping_based_on_recent_occlusion_threshold0.7基于最近一次被遮挡抑制重叠掩码的 IoU 阈值decrease_trk_keep_alive_for_empty_maskletsFalse是否因 SAM2 预测出零面积掩码而扣减 keep-alivefill_hole_area16掩码填洞/去碎屑的最小连通域面积像素max_num_objects10000最大跟踪对象数默认值实际相当于关闭限制recondition_every_nth_frame16掩码重条件化频率帧设为 0 禁用high_conf_thresh0.8重条件化的检测得分门槛high_iou_thresh0.8重条件化的 IoU 门槛配置类还带strict校验validate_architecture要求hotstart_unmatch_thresh ≤ hotstart_delay且hotstart_dup_thresh ≤ hotstart_delayconfiguration_sam3_video.py#L153-L163否则抛ValueError——即缓冲窗口必须不短于判定窗口。自定义分辨率的最小示例来自 config docstringfrom transformers import Sam3VideoConfig, Sam3VideoModel # Initializing a SAM3 Video configuration with default detector and tracker configuration Sam3VideoConfig() # Changing image size for custom resolution inference (automatically propagates to all nested configs) configuration.image_size 560 # Initializing a model from the configuration model Sam3VideoModel(configuration) # Accessing the model configuration configuration model.config detector_config configuration.detector_config tracker_config configuration.tracker_config六、逐帧流水线检测-跟踪融合的源码剖析Sam3VideoModel.forwardmodeling_sam3_video.py#L1693-L1761是单帧推理入口其核心委托给_det_track_one_framemodeling_sam3_video.py#L1598-L1691该函数以注释明确列出五个阶段视觉特征提取detector_model.get_vision_features(pixel_values)每帧只算一次供所有提示的检测与跟踪器复用Step 1 检测run_detectionmodeling_sam3_video.py#L567-L637对每个提示用缓存的视觉嵌入 该提示文本嵌入跑检测器得分 定位概率 × presence 得分det_nms_thresh 0时对低分辨率掩码做 NMSnms_masksmodeling_sam3_video.py#L1886-L1925再按score_threshold_detection过滤输出bbox / mask / scoresStep 2 跟踪传播run_tracker_propagationmodeling_sam3_video.py#L639-L680已有轨迹跑 SAM2 传播得到 masklet 与对象分并对掩码执行fill_holes_in_mask_scores填洞 去碎屑阈值即fill_hole_area注意此步只做传播、不编码记忆Step 3 更新计划run_tracker_update_planning_phase基于检测与 masklet 的匹配_associate_det_trk含 IoU 双阈值assoc_iou_thresh/trk_assoc_iou_thresh、hotstart 启发式_process_hotstart等制定增删改轨迹的计划并在此阶段运行记忆编码器解决非重叠约束Step 4 执行更新run_tracker_update_execution_phase按计划增删 masklet新增对象需得分超过new_det_threshStep 5 构建输出build_outputs汇总成obj_id - mask/score字典由forward打包为Sam3VideoSegmentationOutput。Sam3VideoSegmentationOutput字段modeling_sam3_video.py#L462-L487object_ids当前帧在跟踪的 ID 列表、obj_id_to_mask低分辨率掩码(1, H_low, W_low)、obj_id_to_score检测分、obj_id_to_tracker_score当前帧跟踪分、removed_obj_ids、suppressed_obj_ids、frame_idx。Hotstart 缓冲为什么前 15 帧要延迟输出propagate_in_video_iterator中有一段与hotstart_delay配套的缓冲逻辑modeling_sam3_video.py#L1818-L1842hotstart_buffer [] for frame_idx in tqdm(processing_order, descpropagate in video, disablenot show_progress_bar): out self(inference_sessioninference_session, frame_idxframe_idx, reversereverse) if self.hotstart_delay 0: # accumulate the outputs for the first hotstart_delay frames hotstart_buffer.append(out) # update the object IDs removed by hotstart so that we dont output them inference_session.hotstart_removed_obj_ids.update(out.removed_obj_ids) if frame_idx end_frame_idx: # we reached the end of propagation -- yield all frames in the buffer yield_list hotstart_buffer hotstart_buffer [] elif len(hotstart_buffer) self.hotstart_delay: # we have enough frames -- yield and remove the first (oldest) frame from the buffer yield_list hotstart_buffer[:1] hotstart_buffer hotstart_buffer[1:] else: # not enough frames yet -- skip yielding yield_list [] else: yield_list [out] # output the current frame yield from yield_list含义是视频开头的检出不可靠同一目标可能被反复检出为多个轨迹模型先缓冲hotstart_delay默认 15帧的输出等 hotstart 启发式用未来帧信息判定哪些轨迹是未匹配噪声、哪些是重复轨迹后再开始放行帧输出且被 hotstart 移除的obj_id会被postprocess_outputs一并隐藏obj_ids_to_hide逻辑processing_sam3_video.py#L320-L328。这也解释了文档示例中处理 50 帧却得到 51 帧结果的细节缓冲在到达end_frame_idx时会一次性倾泻yield剩余缓冲帧。而流式模式因为拿不到未来帧此机制整体失效质量警告由此而来。推理会话与设备管理Sam3VideoInferenceSessionmodeling_sam3_video.py#L119-L457是贯穿全程的有状态对象承担了四类职责帧存储processed_frames以字典而非拼接张量存帧避免逐帧torch.cat的重复内存分配modeling_sam3_video.py#L153-L156多提示状态prompts、prompt_input_ids、prompt_embeddings文本嵌入缓存避免每帧重复编码文本、obj_id_to_prompt_id检出时绑定对象与提示对象与轨迹状态mask_inputs_per_obj、point_inputs_per_obj、output_dict_per_obj条件帧/非条件帧输出、trk_keep_alive、unmatched_frame_inds、overlap_pair_to_frame_inds等 hotstart 元数据智能设备调度store_output把小张量如object_pointer、object_score_logits留在推理设备把大张量掩码、特征搬到inference_state_device读取时再移回modeling_sam3_video.py#L320-L381视觉特征缓存Sam3VideoInferenceCache默认只保留最近 1 帧max_vision_features_cache_size1实现 GPU/显存与延迟的折中。此外 session 还提供remove_object(obj_id, strict...)把某对象从全片移除并重建索引映射modeling_sam3_video.py#L265-L317以及reset_tracking_data/reset_inference_session/reset_state三级重置保留视频帧、清缓存、全清。可选加速依赖kernels 库检测 NMS 与掩码连通域分析依赖kernels-community/cv-utilskernel。若未安装pip install kernels_load_cv_utils_kernel_once会发出一次性警告并优雅降级NMS 与填洞/去碎屑被跳过但推理仍可运行modeling_sam3_video.py#L43-L64、#L1972-L1978。追求最佳掩码质量时建议安装该可选依赖。七、生态周边测试、权重转换与相关文档单元测试tests/models/sam3_video/test_modeling_sam3_video.py 覆盖该模型的通用建模测试权重转换脚本src/transformers/models/sam3_video/convert_sam3_video_to_hf.py 将原始 SAM3 权重转换为 HF 格式本模型于 2025-11-19 由社区贡献进库组成部件文档detector_config指向 SAM3 检测器tracker_config指向 SAM2 风格视频跟踪器二者各自有独立模型文档docs/source/en/model_doc/sam3.md、docs/source/en/model_doc/sam2_video.md等阅读 SAM3 Video 文档时可交叉参考注意力后端Sam3VideoPreTrainedModel同时支持 SDPA、FlashAttention 与 flex-attentionmodeling_sam3_video.py#L500-L503可按部署环境选择以降低注意力开销。八、实践小结帧都在手上就用预加载模式init_video_session(video...)propagate_in_video_iterator享受 hotstart 去重带来的干净轨迹只有边到达边处理的实时场景才用流式模式并务必在postprocess_outputs传original_sizes。多提示几乎免费视觉特征跨提示复用检测-跟踪按提示分组隔离prompt_to_obj_ids直接给出哪个提示检出了哪些对象。分辨率取舍默认按 1008px 设计改到 560px 换取速度/显存时config.image_size与processor的size要同步修改且应预期精度下降。调参顺序先动score_threshold_detection/new_det_thresh检出灵敏度与新对象门槛再动assoc_iou_thresh/trk_assoc_iou_thresh匹配松紧最后考虑recondition_every_nth_frame等长期漂移抑制参数改 hotstart 系列阈值时注意validate_architecture的约束。可选依赖安装kernels可启用 NMS 与连通域后处理掩码更干净不安装功能不缺失但质量打折。核心文件一览模型文档 docs/source/en/model_doc/sam3_video.md、配置 configuration_sam3_video.py、模型 modeling_sam3_video.py、处理器 processing_sam3_video.py、测试 test_modeling_sam3_video.py。【免费下载链接】transformers Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.项目地址: https://gitcode.com/GitHub_Trending/tra/transformers创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考