verl 中的 Sandbox Fusion 工具集成：基于 sglang 多轮 rollout 的安全代码执行与分布式速率限制设计

verl 中的 Sandbox Fusion 工具集成基于 sglang 多轮 rollout 的安全代码执行与分布式速率限制设计【免费下载链接】verlverl/HybridFlow: A Flexible and Efficient RL Post-Training Framework项目地址: https://gitcode.com/GitHub_Trending/ve/verl本文围绕 verl/HybridFlow 仓库中的 Sandbox Fusion Tool Integration 设计文档 展开系统讲解如何在 verl 的 Actor 多轮 rolloutsglang 引擎阶段接入 sandbox-fusion 远程代码沙箱让模型在推理过程中调用code_interpreter工具执行生成的代码、并把执行结果与奖励纳入训练闭环。读者将掌握工具 Schema 如何定义、六个核心配置参数的含义与默认值、基于 Ray Global Actor 信号量的分布式令牌桶限流设计、BaseTool生命周期create/execute/calc_reward/release的实现要点以及如何通过rollout_data_dir与 RolloutViewer 调试多轮工具调用的完整轨迹。背景与动机verl 的多轮 rollout 支持见 多轮 Rollout 文档允许模型在生成过程中反复与外部环境交互把中间结果重新喂回推理引擎继续生成。Sandbox Fusion 集成正是这一能力在代码执行方向上的落地其核心动机包括作为 verl 用户我们希望允许模型在 Actor rollout 阶段调用某些工具并把工具返回结果合并进训练流程希望通过代码执行工具增强模型能力该方向最初由 ByteDance 同事提出的论文思路驱动目标是让推理引擎借助sandbox-fusion作为代码执行系统获得工具调用能力为社区提供retools的一个 reimplementation。在奖励计算一侧类似工作已有先例例如 Prime 使用本地子进程作为 runner 来执行模型生成的代码以计算奖励在此基础上verl 进一步把 FaaS函数即服务作为奖励计算的 runner实现远程、并发的代码验证。仓库中的 Sandbox Fusion 示例文档 对此有独立说明。设计边界目标与非目标该集成明确定义了范围Goals目标适配sglang的工具调用协议为 sandbox fusion 定义工具与async-rollout流程集成保证 sandbox fusion 工具遵循 asyncIO 约定设计并实现基础的速率限制器避免 429 等限流错误。Non-Goals明确不做训练效果评估不在范围内可观测性指标暂不考虑分布式故障转移与组件容错暂不处理。值得注意文档开头有一则重要提示——本文档讨论的 in-tree 参考实现verl.tools.sandbox_fusion_tools.SandboxFusionTool已从当前源码树中移除可通过 git 历史 commitf5e21df6及更早版本查看旧实现。但集成模式本身依然成立用户可基于 BaseTool 自行实现子类在沙箱侧仓库仍保留了完整的奖励计算实现verl/utils/reward_score/sandbox_fusion/供奖励阶段直接使用。工具 Schema 定义code_interpreter当前设计只考虑代码执行因此模型返回的 JSON 中只需包含code字段且现阶段仅支持 Python因此不定义language参数。工具采用 OpenAI Function Call 格式的 SchemaOpenAIFunctionToolSchema( typefunction, functionOpenAIFunctionSchema( namecode_interpreter, descriptionA tool for executing code., parametersOpenAIFunctionParametersSchema( typeobject, properties{ code: OpenAIFunctionPropertySchema( typestring, descriptionThe code to execute., enumNone, ) }, required[code], ), strictFalse, ) )对应的 Pydantic 模型定义位于 verl/tools/schemas.pyOpenAIFunctionToolSchema/OpenAIFunctionSchema/OpenAIFunctionParametersSchema/OpenAIFunctionPropertySchema。在模型侧系统提示词中会把该 Schema 以tools/toolsXML 标签形式注入并要求模型用tool_call/tool_callXML 标签返回 JSON 格式的工具调用例如tool_call {name: code_interpreter, arguments: {code: total_pay_this_year 200000\n...}} /tool_call这个格式与multiturn.rst中描述的 sglang 多轮工具调用约定一致hermes等 chat template 支持。配置参数参数名说明num_workers每个 DP数据并行请求 runner 的工作线程/进程数量rate_limit全局并发代码执行数上限。默认10default_timeout单次代码执行的超时时间秒。默认30default_language默认编程语言。默认pythonenable_global_rate_limit是否启用全局速率限制。默认Truesandbox_fusion_urlveFaas 沙箱执行服务的 URL其中sandbox_fusion_url的端点格式为https://ip-address-or-domain-name/run_code来自 Sandbox Fusion 示例文档。而在奖励计算侧的配置中见 verl/trainer/config/reward/reward.yaml还有一组对应参数# Cloud/local sandbox fusion configuration for custom reward logic sandbox_fusion: _target_: verl.workers.config.SandboxFusionConfig # Cloud /local function URL for sandbox execution url: null # Max concurrent requests allowed to sandbox max_concurrent: 64 # Max memory limit for each sandbox process in MB memory_limit_mb: 1024即reward_model.sandbox_fusion.url沙箱 API 端点、reward_model.sandbox_fusion.max_concurrent并发请求数上限示例文档推荐如 256、reward_model.sandbox_fusion.memory_limit_mb每个沙箱实例内存上限默认 1024MB。速率限制设计分布式令牌桶限流的目标是用令牌桶模型限制 inflight 请求数量保证提交给代码 runner 的请求有序避免因 backoff 导致饥饿。设计要点如下使用 Ray Global Actor 作为集群级别的单例分布式计数器namerate-limiterget_if_existsTrue保证全集群只有一个实例用信号量计数acquire与release放在不同的线程池中以保持提交顺序用 Ray 的 cloud-pickle 序列化函数为解耦的ExecutionWorker传递执行体。核心代码如下文档原设计ray.remote(concurrency_groups{acquire: 1,release: 10}) class TokenBucketWorker: def __init__(self, rate_limit: int): self.rate_limit rate_limit self.current_count 0 self._semaphore threading.Semaphore(rate_limit) ray.method(concurrency_groupacquire) def acquire(self): self._semaphore.acquire() self.current_count 1 ray.method(concurrency_grouprelease) def release(self): self._semaphore.release() self.current_count - 1 def get_current_count(self): return self.current_count class ExecutionWorker: def __init__(self, enable_global_rate_limitTrue, rate_limit10): self.rate_limit_worker self._init_rate_limit(rate_limit) if enable_global_rate_limit else None def _init_rate_limit(self, rate_limit): return TokenBucketWorker.options(namerate-limiter, get_if_existsTrue).remote(rate_limit) def execute(self, fn: Callable[..., T], *fn_args, **fn_kwargs) - T: with ExitStack() as stack: stack.callback(self.rate_limit_worker.release.remote) ray.get(self.rate_limit_worker.acquire.remote()) try: return fn(*fn_args, **fn_kwargs) except Exception as e: logger.warning(fError when executing code: {e}) def init_execution_pool(num_workers: int, enable_global_rate_limitTrue, rate_limit10, mode: PoolModePoolMode.ThreadMode): if mode PoolMode.ThreadMode: return ray.remote(ExecutionWorker).options(max_concurrencynum_workers).remote( enable_global_rate_limitenable_global_rate_limit, rate_limitrate_limit ) else: raise NotImplementedError(Process mode is not implemented yet)要点分析TokenBucketWorker是一个 Ray remote actoracquire与release被分配到不同 concurrency groupacquire 串行 1 并发、release 10 并发从而保证先获取后释放的顺序性ExecutionWorker.execute使用ExitStack注册 release 回调即使fn抛异常也会释放令牌避免限流器被卡死对应测试中的test_rotten_executionray.get(acquire.remote())会阻塞等待令牌天然把并发控制与 async 执行池解耦。工具实现SandboxFusionTool 生命周期工具类实现遵循 verl 的 BaseTool 接口该接口定义了五个核心方法get_openai_tool_schema返回 OpenAI 格式工具 Schemacreate为一条 trajectory轨迹创建工具实例默认生成uuid4()作为instance_idexecute执行工具返回(ToolResponse, reward, metrics)三元组calc_reward基于工具状态计算奖励release释放工具实例。SandboxFusionTool的参考实现要点class SandboxFusionTool(BaseTool): def __init__(self, config: dict, tool_schema: OpenAIFunctionToolSchema): ... self.execution_pool init_execution_pool(...) ... async def create(self, instance_id: Optional[str] None, ...): ... async def execute(self, instance_id: str, parameters: dict[str, Any], **kwargs) - Tuple[str, float, dict]: code parameters.get(code, ) timeout parameters.get(timeout, self.default_timeout) language parameters.get(language, self.default_language) if not isinstance(code, str): code str(code) result await self.execution_pool.execute.remote(self.execute_code,instance_id,code,timeout,language) self._instance_dict[instance_id][reward].append(result.strip()) return result, result, {} def execute_code(self,instance_id,code,timeout30,languagepython): result_status, metadata _process_single_case(0, None, None,self.sandbox_fusion_url, code, timeout, language) # we should always expect this since we dont have correct answer if metadata[run_status] Finished: actual_output metadata[stdout] if metadata[stdout] is not None else return actual_output else: return no stdout here async def calc_reward(self, instance_id: str, ...): ... async def release(self, instance_id: str, ...): ...设计语义instance_id用于跨多轮对话识别请求同一轮 rollout 中的多次工具调用共享实例工具内部状态如_instance_dict[instance_id]按实例累积execution_pool实现异步调用await self.execution_pool.execute.remote(...)把同步的代码执行放到 Ray actor 的并发池中不阻塞事件循环符合 async-rollout 的 asyncIO 约定rollout 完成后清理状态release负责回收实例状态例如清理_instance_dict中该实例的条目避免跨 trajectory 串扰。注意execute_code调用的_process_single_case位于 verl/utils/reward_score/sandbox_fusion/utils.py由于工具调用场景没有标准答案expected_outputNone因此只要沙箱run_status Finished就把stdout返回给模型失败时返回no stdout here占位文本。工具注册与 sglang 多轮 rollout 集成要让工具真正参与 rollout需要两步配置详见 多轮 Rollout 文档开启多轮 rolloutactor_rollout_ref: rollout: multi_turn: True name: sglang指定工具配置文件实现BaseTool子类后在 YAML 中声明并挂到 rollout 配置上tools: - class_name: # 例如你的 SandboxFusionTool 全限定类名 config: type: native tool_schema: actor_rollout_ref: rollout: tool_kwargs: tools_config_file: path_to_tool_yaml_file从源码看verl/tools/tool_registry.py 的initialize_tools_from_config会读取该 YAML通过get_tool_class按模块路径动态导入工具类config.type必须为native对应ToolType.NATIVEtool_schema会被解析为OpenAIFunctionToolSchema后与config一起传入构造函数。若同时配置了function_tool_pathload_all_tools会合并两类工具并检查名称冲突tool_registry.py#L83-L101。对于无状态工具也可直接用function_tool装饰器注册普通 Python 函数由transformers.get_json_schema自动推断 Schema无需完整生命周期而 Sandbox Fusion 这类需要跨轮状态沙箱实例、执行池与资源清理的工具则应坚持使用BaseTool路径。奖励计算Sandbox Fusion FaaS 集成除 rollout 工具调用外Sandbox Fusion 也被用于奖励计算阶段。整体思路是不再用本地子进程逐个执行生成的代码而是把代码提交到远程沙箱FaaS利用沙箱更充裕的 CPU 资源并发验证可缩短奖励阶段耗时 10%–30%取决于生成代码质量见 Sandbox Fusion 示例文档。对应配置reward_model.sandbox_fusion.urlAPI-endpoint # 必须结尾为 /run_code reward_model.sandbox_fusion.max_concurrent256 # 并发请求数 reward_model.sandbox_fusion.memory_limit_mb1024 # 单沙箱内存上限(MB) reward_model.reward_managerprime # Prime 奖励管理器多子进程并发验证底层实现 verl/utils/reward_score/sandbox_fusion/utils.py 包含三层关键逻辑call_sandbox_api向sandbox_fusion_url发起 HTTP POSTpayload 包含compile_timeout、run_timeout、code、stdin、memory_limit_MB、language、files、fetch_files字段对 504 Gateway Timeout 采用线性退避重试最多 3 次延迟 1s/2s/3s其余 HTTP/JSON 错误直接返回错误信息_process_single_case处理单个测试用例解析 API 返回的compile_result/run_result把api_status分类为Success/Failed/SandboxError并映射为细粒度结果码True通过、False答案错误、-1API/沙箱错误、-2运行时错误、-3超时、-4编译错误若language python且提供fn_name还会生成一段 wrapper 代码预导入math、itertools、collections等常用库支持全局函数或Solution类方法的定位与调用check_correctness并发调度多个用例ThreadPoolExecutormax_workers 取max(32, os.cpu_count() * 5)并通过concurrent_semaphore把并发数限制在sandbox_fusion_max_concurrent以内若某用例编译失败其后的用例会被标记为compile_error_skipped。SUPPORTED_LANGUAGES列表覆盖 29 种语言python、cpp、nodejs、go、java、rust、bash、sql、cuda 等见 utils.py#L34-L64但 rollout 工具阶段当前只支持 Python。相关测试位于 tests/utils/reward_score/test_sandbox_on_cpu.pytest_prime_code_sandbox_fusion与test_continuous_score_consistency通过SANDBOX_FUSION_URL环境变量指向真实沙箱服务未设置时自动 skip验证 sandbox_fusion 与 Prime 在连续分数计算上的一致性。测试计划单元测试设计文档列出的单元测试覆盖如下维度test_tools_registration工具注册与初始化test_rollout_req_creation验证AsyncRolloutReq构造正确test_over_size_case超过max_seq_len时 rollout 提前终止test_tool_call_basic_caseMock sglang 输出验证工具调用与结果test_tool_call_batch_case工具调用的批处理test_basic_multi_process_init验证 Ray Global Actor 的全局单例性TestSingleNodeRateLimiterCase单机模式下限流器正确工作test_rotten_execution函数出错时限流器能恢复令牌被正确释放TestMultiNodeRateLimiterCase多机环境下的限流行为。e2e 测试仓库提供 e2e 脚本历史上位于tests/special_e2e目录名为tests/special_e2e/run_gsm8k_fsdp_sgl_multiturn_sf_tool.sh当前源码树中该文件已移除可参考 git 历史。通过设置trainer.rollout_data_dir可以把 rollout 数据 dump 到本地磁盘。下面是取自 rollout 数据的一条 GSM8K 多轮工具调用样本已整理为可读格式[system] You are a math expert. You are given a question and you need to solve it step by step. Reasoning step by step before any tool call. You should use the calc_gsm8k_reward tool after step by step solving the question, before generate final answer at least once and refine your answer if necessary. Put your final answer in the format of #### answer. # Tools You may call one or more functions to assist with the user query. You are provided with function signatures within tools/tools XML tags: tools {type: function, function: {name: code_interpreter, description: A tool for executing code., parameters: {type: object, properties: {code: {type: string, description: The code to execute., enum: null}}, required: [code]}, strict: false}} /tools For each function call, return a json object with function name and arguments within tool_call/tool_call XML tags: tool_call {name: function-name, arguments: args-json-object} /tool_call [user] John gets a bonus thats the same percentage every year. Last year he made $100,000 and got a $10,000 bonus. This year he makes $200,000. How much will John make this year when adding both his total pay and bonus together? Lets think step by step and output the final answer after ####. [assistant] think Okay, lets see. John gets a bonus thats the same percentage every year. Last year, he made $100,000 and got a $10,000 bonus. This year, he makes $200,000. The question is asking how much hell make this year when adding both his total pay and bonus together. First, I need to figure out the percentage bonus he gets. Since the bonus is the same percentage each year, if I can find that percentage based on last years details, I can apply it to this years earnings to find this years bonus. Then, Ill add that bonus to his total pay this year. Last year, he earned $100,000 and got a $10,000 bonus. To find the percentage, I can divide the bonus by his total earnings. So, 10,000 divided by 100,000. Let me calculate that. 10,000 / 100,000 equals 0.1, which is 10% as a percentage. So, his bonus is 10% of his pay each year. This year, he makes $200,000. So, the bonus this year would be 10% of 200,000. Let me compute that using the code interpreter to ensure accuracy. tool_call {name: code_interpreter, arguments: {code: total_pay_this_year 200000\nbonus_percentage 10 / 100\nbonus_this_year total_pay_this_year * bonus_percentage\ntotal_income_this_year total_pay_this_year bonus_this_year\nprint(total_income_this_year), executes: True}} /tool_call [tool] 220000.0 [assistant] think So the code shows that Johns total income this year would be $220,000. This is consistent with the 10% bonus on last years pay of $100,000, which was $10,000. Therefore, this years bonus of $20,000 added to his total pay of $200,000 results in a total income of $220,000. I need to verify that there are no other factors at play. The problem states its a constant percentage bonus each year, so theres no mention of changing percentages or other variables. Therefore, applying a straightforward percentage of 10% on this years partial salary amount would result in $20,000. Adding that to his total pay gives $220,000. The calculation is correct and aligns with the questions parameters. /think #### 220000.0该样本完整展示了多轮工具调用的闭环模型先思考 → 调用code_interpreter执行计算 → 沙箱返回220000.0→ 模型基于工具结果推理并给出最终答案#### 220000.0。rollout 数据以 JSON 形式记录input含系统提示与完整工具 Schema、output多轮消息流、score与step字段。使用 RolloutViewer 调试dump 出的 rollout 数据可用仓库自带的 TUI 工具 RolloutViewer 查看python scripts/rollout_viewer.py ${trainer.rollout_data_dir}该脚本位于 scripts/rollout_viewer.py以交互式终端界面逐条回放 rollout 轨迹便于排查多轮工具调用中的格式、超时或奖励问题。实践建议与注意事项工具 Schema 保持精简目前仅暴露code字段且默认 Python如需多语言可在execute中通过parameters.get(language, self.default_language)透传沙箱侧SUPPORTED_LANGUAGES已支持 29 种语言限流参数与沙箱容量匹配rate_limitrollout 工具侧默认 10与max_concurrent奖励侧默认 64示例推荐 256应根据沙箱服务能力调整过小会拖慢训练、过大会触发 429令牌释放是正确性关键务必使用类似ExitStack的机制保证release在任何路径含异常下都被调用否则全局限流器会被耗尽参考test_rotten_execution状态清理不可省略多轮工具实例按instance_id累积状态release阶段必须清理防止跨轨迹泄漏参考实现的移除当前源码树不含verl.tools.sandbox_fusion_tools需要自行基于 BaseTool 实现可参考 git 历史 commitf5e21df6及更早版本奖励计算侧的verl/utils/reward_score/sandbox_fusion/仍完整可用是理解沙箱 API 协议的最佳源码入口。延伸阅读Sandbox Fusion 示例文档奖励阶段的端到端配置与 Eurus-2-RL-Data 数据集示例多轮 Rollout 支持文档多轮 tokenization、工具配置与多模态输入处理工具注册实现initialize_tools_from_config/load_all_tools的加载与冲突检查逻辑BaseTool 接口自定义工具必须实现的五个生命周期方法沙箱 API 客户端call_sandbox_api/_process_single_case/check_correctness沙箱奖励配置sandbox_fusion配置块url / max_concurrent / memory_limit_mb沙箱奖励测试基于SANDBOX_FUSION_URL环境变量的正确性验证RolloutViewer 脚本多轮 rollout 数据的 TUI 可视化【免费下载链接】verlverl/HybridFlow: A Flexible and Efficient RL Post-Training Framework项目地址: https://gitcode.com/GitHub_Trending/ve/verl创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考