一、按模型的默认值
BenchmarkConfig.for_model() 会根据内置表设置 max_tokens、timeout_seconds 和 context_window：
config = BenchmarkConfig.for_model( "Qwen/Qwen3.6-35B-A3B", save_dir="evals/qwen3.6",)# max_tokens=65536, context_window=65536, timeout_seconds=1800
二、多轮基准测试
Agent 类基准测试（swe_bench、terminal_bench、tau2_bench）需要上下文管理：
config = BenchmarkConfig( timeout_seconds=1800, max_trajectory_tokens=60000, # Total tokens across all turns max_generation_tokens=8192, # Tokens per generation step agent_concurrency=4,)
三、沙箱基准测试
需要执行代码的基准测试（mbpp、livecodebench、swe_bench）依赖沙箱：
from tinker_cookbook.sandbox.modal_sandbox import ModalSandboxconfig = BenchmarkConfig( sandbox_factory=ModalSandbox.create,)
四、裁判模型基准测试
arena_hard 这类基准测试需要额外的 LLM 裁判。配置细节与示例见 裁判模型。
五、保存与查看结果
设置 save_dir 以持久化轨迹和结果：
evals/run_01/├── summary.json # Combined scores (run_benchmarks only)├── gsm8k/│ ├── result.json # BenchmarkResult│ └── trajectories.jsonl # One StoredTrajectory per line└── ifeval/ ├── result.json └── trajectories.jsonl
六、加载结果
from tinker_cookbook.eval.benchmarks import ( load_result, load_trajectories, load_summary, print_trajectory,)result = load_result("evals/run_01", "gsm8k")print(f"Score: {result.score:.1%}, Completed: {result.score_completed:.1%}")# Filter trajectorieswrong = load_trajectories("evals/run_01", "gsm8k", incorrect_only=True)errors = load_trajectories("evals/run_01", "gsm8k", errors_only=True)print_trajectory(wrong[0])# Combined summary across benchmarkssummary = load_summary("evals/run_01")
七、可续跑
使用相同的 save_dir 重新运行时会跳过已完成的样本。去重基于 example_id（内容哈希），因此即使数据集被打乱也能稳定工作。
八、理解分数
·score — num_correct / num_examples。被截断和出错的样本计为 0。
·score_completed — num_correct / num_completed。分母中排除被截断和出错的样本。
对于经常触及 max_tokens 的思考型模型，用 score_completed 与公开分数对比更合适。
result = await run_benchmark("gsm8k", client, renderer, config)print(f"Raw: {result.score:.1%}") # 81.7%print(f"Completed: {result.score_completed:.1%}") # 95.6%print(f"{result.num_truncated} truncated, {result.num_errors} errors")
九、对比检查点
checkpoints = { "base": "tinker://run-abc/sampler_weights/step0", "step500": "tinker://run-abc/sampler_weights/step500", "final": "tinker://run-abc/sampler_weights/final",}for name, path in checkpoints.items(): client = await sc.create_sampling_client_async(model_path=path) results = await run_benchmarks( ["gsm8k", "ifeval"], client, renderer, BenchmarkConfig(save_dir=f"evals/{name}"), ) for bench, r in results.items(): print(f"{name}/{bench}: {r.score:.1%}")
十、用 EvalStore 追踪多次运行
EvalStore 通过云端兼容的存储管理跨检查点的评测运行：
from tinker_cookbook.stores.eval_store import EvalStorestore = EvalStore("~/experiments/evals")run_id = store.create_run( model_name="Qwen/Qwen3.6-35B-A3B", checkpoint_path="tinker://run-123/weights/step500", checkpoint_name="step500", benchmarks=["gsm8k", "ifeval"],)config = BenchmarkConfig(save_dir=store.run_dir(run_id))await run_benchmarks(["gsm8k", "ifeval"], client, renderer, config)metadata = store.finalize_run(run_id)print(metadata.scores) # {"gsm8k": 0.847, "ifeval": 0.923}
十一、Pass@k 评测
设置 num_samples > 1 可计算无偏的 pass@k 估计（Codex 公式）：
config = BenchmarkConfig(num_samples=4, save_dir="evals/pass_at_k")result = await run_benchmark("gsm8k", sampling_client, renderer, config)print(result.pass_at_k) # {1: 0.45, 2: 0.58, 4: 0.72}
十二、自定义
关于系统提示词、自定义评分、答案解析和裁判模型配置，见 自定义基准测试。
十三、新增基准测试
基准测试复用与 RL 训练相同的 Env 协议——单个 MessageEnv 即可同时驱动训练和评测。概念性说明见 共享 Env 抽象。
要新增基准测试，实现一个 BenchmarkBuilder 并注册：
from tinker_cookbook.eval.benchmarks import BenchmarkBuilder, BenchmarkConfig, registerfrom tinker_cookbook.eval.benchmarks._common import load_benchmark_dataset, make_example_idfrom tinker_cookbook.rl.message_env import MessageEnv, MessageStepResult, EnvFromMessageEnvclass MyMessageEnv(MessageEnv): """Single-turn env for one example.""" def __init__(self, question: str, expected: str, example_id: str): self.question = question self.expected = expected self._example_id = example_id @property def example_id(self) -> str: return self._example_id async def initial_observation(self) -> list[Message]: return [{"role": "user", "content": self.question}] async def step(self, message: Message) -> MessageStepResult: response = get_text_content(message).lower() correct = self.expected.lower() in response return MessageStepResult( reward=1.0 if correct else 0.0, episode_done=True, next_messages=[], metrics={"correct": float(correct)}, logs={"expected": self.expected}, )class MyBenchmark(BenchmarkBuilder): name = "my_benchmark" recommended_system_prompt = "Answer concisely." def make_envs(self, renderer, config): ds = load_benchmark_dataset("my/dataset", split="test") envs = [] for row in ds: env = EnvFromMessageEnv( renderer=renderer, message_env=MyMessageEnv( question=row["question"], expected=row["answer"], example_id=make_example_id("my_benchmark", row["question"]), ), ) envs.append(env) return envsregister(MyBenchmark())
要点：
·单轮基准测试用 MessageEnv——它负责提示词构建与响应解析
·通过 make_example_id() 设置 example_id，保证多次运行间可稳定续跑
·logs 存放每条轨迹的诊断数据（期望答案、抽取出的答案）；metrics 存放用于聚合的数值
·多轮基准测试请直接实现 Env，并在 builder 上设置 multi_turn = True
十四、延伸阅读
·评测概览：可用基准测试与架构
·自定义基准测试：系统提示词、评分、裁判模型、共享 Env 设计
·评测教程：交互式演练
·RL 环境：基准测试使用的 Env 协议