云点播 智能成片/问答 Agent 接入

二、智能成片 Agent

智能成片 Agent 支持原声成片和旁白成片两种成片模式，通过自然语言输入可覆盖从原始素材到可直接发布成片的全流程。所有成片均支持字幕、背景音乐、画幅等通用配置。

三、成片模式

四、通用视频配置

·字幕：不压制字幕 / 压制字幕（支持配置字幕大小、描边等）。

·背景音乐：无需 BGM / 上传 BGM / AI 生成 BGM。

·画幅：横屏 / 竖屏，支持横转竖能力。

五、智能问答 Agent

支持对智能媒资库内的媒体资源进行自然语言问答，实现便捷的内容检索与交互。

注意：说明：

注意：使用此能力，需要创建视频知识库，并进行入库。

六、计费说明

1. 智能成片 Agent 计费：使用媒体 AI Agent 智能成片能力将产生计费，统一按照 Token 计费，计费详情见 媒体 AI Agent 计费。

2. 智能问答 Agent 计费：

2.1 视频大模型解析费用：入库理解按照启用的理解维度进行计费：

·音频级理解：收取视频摘要费用，具体可参考 智能媒资 > 大模型视频摘要计费。

·视频级/画面摘要理解费用：收取大模型视频理解费用，具体可参考 智能媒资 > 大模型理解计费。

·文本转录：收取 ASR 语音识别费用，具体可参考 智能媒资 > ASR 语音识别计费。

2.2 视频入库将产生产品向量化存储费用，具体可参考 智能媒资 > 向量化存储计费说明。

2.3 媒体 AI Agent 视频问答所产生的问答费用，具体可参考 媒体 AI Agent > Agent LLM 计费说明。

七、控制台使用

八、API 接入

此能力使用 AG-UI 协议接入。

九、1. 协议概述

AG-UI（Agent-User Interaction Protocol）是基于 HTTPS + SSE 的流式 Agent 交互协议，提供丰富的能力：

·完整的事件体系（文本消息、推理过程、工具调用生命周期、步骤追踪）。

·原生 HITL 中断/恢复机制（通过 RunFinished outcome 实现）。

·每个工具独立中断与恢复（每个待确认工具生成独立 Interrupt，可分别批准/拒绝）。

·消息快照查询（/history 端点）。

十一、创建 API Token

在进行调用前，需要通过 创建 AIGC API Token 接口 创建 API Token。

十二、启动 Agent 对话

VOD Agent 采用 AG-UI 协议，你也可以参考 AG-UI 产品文档 了解各个字段的含义。

请求地址：https://smartmedia.vod-qcloud.com/agui/chat

POST /agui/chatHost: smartmedia.vod-qcloud.comContent-Type: application/jsonAuthorization: Bearer <TOKEN>{ "threadId": "my-thread-001", "runId": "run-001", "messages": [ {"role": "user", "content": "帮我根据提供的素材剪一条抖音竖屏带货短视频，时长 30 秒左右，主推产品是XXX，突出 XXX 卖点。你来写口播文案、配 AI 配音-1.2倍速、加背景音乐、去掉素材里的原声、加字幕-字号50px，描边3px，节奏明快一点，结尾引导下单。素材URL：XXXX "} ], "forwardedProps": { "approval_mode": "level:high", "scenario_name": "video-mixcut" }}

Message 字段

ContentPart 字段

forwardedProps 字段

十三、查询消息历史

请求地址：https://smartmedia.vod-qcloud.com/agui/history

POST /agui/historyHost: smartmedia.vod-qcloud.comContent-Type: application/jsonAuthorization: Bearer <TOKEN>{ "threadId": "my-thread-001", "forwardedProps": { "scenario_name": "video-mixcut" }}

返回该线程的完整消息快照。

十四、中断进行中的会话

请求地址：https://smartmedia.vod-qcloud.com/agui/cancel

POST /agui/cancelHost: smartmedia.vod-qcloud.comContent-Type: application/jsonAuthorization: Bearer <TOKEN>{ "threadId": "my-thread-001", "forwardedProps": { "scenario_name": "video-mixcut" }}

在发起 Agent 对话后，即使断开连接，会话也会继续在后台运行。如果你需要中断 Agent 执行，需要调用本接口进行取消。

十五、3. 响应（SSE 事件流）

id: <ID>data: {"type":"<EventType>","timestamp":<timestamp>,…}

十六、事件类型

生命周期事件

消息事件

推理事件

工具调用事件

十七、正常对话事件流

以下是一次完整对话的实际 SSE 输出（简化）：

id: RUN_STARTED_1784102011155data: {"type":"RUN_STARTED","timestamp":1784102011155,"threadId":"thread-001","runId":"run-001"}id: REASONING_START_1784102012264data: {"type":"REASONING_START","timestamp":1784102012264,"messageId":"msg-001"}id: REASONING_MESSAGE_START_1784102012264data: {"type":"REASONING_MESSAGE_START","timestamp":1784102012264,"messageId":"msg-001","role":"reasoning"}id: REASONING_MESSAGE_CONTENT_1784102012264data: {"type":"REASONING_MESSAGE_CONTENT","timestamp":1784102012264,"messageId":"msg-001","delta":"用户想要搜索海浪视频…"}id: REASONING_MESSAGE_END_1784102013400data: {"type":"REASONING_MESSAGE_END","timestamp":1784102013400,"messageId":"msg-001"}id: REASONING_END_1784102013401data: {"type":"REASONING_END","timestamp":1784102013401,"messageId":"msg-001"}id: TEXT_MESSAGE_START_1784102019000data: {"type":"TEXT_MESSAGE_START","timestamp":1784102019000,"messageId":"msg-002"}id: TEXT_MESSAGE_CONTENT_1784102019050data: {"type":"TEXT_MESSAGE_CONTENT","timestamp":1784102019050,"messageId":"msg-002","delta":"我来帮你搜索。"}id: TEXT_MESSAGE_END_1784102024858data: {"type":"TEXT_MESSAGE_END","timestamp":1784102024858,"messageId":"msg-002"}id: TOOL_CALL_START_1784102126631data: {"type":"TOOL_CALL_START","timestamp":1784102126631,"toolCallId":"call-001","toolCallName":"search_media_by_semantics","parentMessageId":"msg-002"}id: TOOL_CALL_ARGS_1784102126631data: {"type":"TOOL_CALL_ARGS","timestamp":1784102126631,"toolCallId":"call-001","delta":"{\\"query\\":\\"海浪\\"}"}id: TOOL_CALL_END_1784102126631data: {"type":"TOOL_CALL_END","timestamp":1784102126631,"toolCallId":"call-001"}id: TOOL_CALL_RESULT_1784102126886data: {"type":"TOOL_CALL_RESULT","timestamp":1784102126886,"messageId":"msg-003","toolCallId":"call-001","content":"{\\"Recall\\":[]}","role":"tool"}id: RUN_FINISHED_1784102025004data: {"type":"RUN_FINISHED","timestamp":1784102025004,"threadId":"thread-001","runId":"run-001"}

十八、HITL 中断事件流

当工具需要确认时，RUN_FINISHED 的 data 中携带 outcome.type = "interrupt"。每个待确认工具生成一个独立的 Interrupt：

id: TOOL_CALL_START_1784102316642data: {"type":"TOOL_CALL_START","timestamp":1784102316642,"toolCallId":"call-001","toolCallName":"search_media_by_semantics","parentMessageId":"msg-001"}id: TOOL_CALL_END_1784102316642data: {"type":"TOOL_CALL_END","timestamp":1784102316642,"toolCallId":"call-001"}id: RUN_FINISHED_1784102316731data: {"type":"RUN_FINISHED","timestamp":1784102316731,"threadId":"thread-001","runId":"run-001","outcome":{"type":"interrupt","interrupts":[{"id":"interruptId-001","reason":"tool_call","message":"在视频库中语义检索与海浪相关的视频片段","toolCallId":"call-001","responseSchema":{"properties":{"feedback":{"description":"Optional user feedback for the approval decision.","type":"string"}},"type":"object"}}]}}

其中，outcome.interrupts[N].message 为使用自然语义描述的工具调用说明，客户端可以把它展示给用户，并请求用户审批。

Interrupt 字段

每个待确认工具都有独立的 Interrupt ID，客户端可据此对每个工具分别恢复。

20、审批模式

取值不为 never 时，可能在 tool 调用前触发 HITL 中断。

21、恢复请求

用户做出决策后，发送新请求，在 resume 数组中为每个 Interrupt 传入一个恢复条目。

全部批准

{ "threadId": "my-thread-001", "runId": "run-002", "messages": [{"role": "user", "content": ""}], "forwardedProps": {"approval_mode": "level:low", "scenario_name": "video-mixcut"}, "resume": [ {"interruptId": "interruptId-001", "status": "resolved"}, {"interruptId": "interruptId-002", "status": "resolved"} ]}

全部拒绝（附带反馈）

{ "threadId": "my-thread-001", "runId": "run-002", "messages": [{"role": "user", "content": ""}], "forwardedProps": {"approval_mode": "level:low", "scenario_name": "video-mixcut"}, "resume": [ {"interruptId": "interruptId-001", "status": "cancelled", "payload": {"feedback": "不需要裁剪"}}, {"interruptId": "interruptId-002", "status": "cancelled", "payload": {"feedback": "不需要水印"}} ]}

每工具独立决策（部分批准、部分拒绝）

{ "threadId": "my-thread-001", "runId": "run-002", "messages": [{"role": "user", "content": ""}], "forwardedProps": {"approval_mode": "level:low", "scenario_name": "video-mixcut"}, "resume": [ {"interruptId": "interruptId-001", "status": "resolved"}, {"interruptId": "interruptId-002", "status": "cancelled", "payload": {"feedback": "不要添加水印"}} ]}

Resume 条目字段

注意：说明：

22、5. Scenario 场景配置

通过 forwardedProps.scenario_name 引用服务端预设场景：

{ "threadId": "thread-001", "runId": "run-001", "messages": [{"role": "user", "content": "搜索海浪视频"}], "forwardedProps": {"scenario_name": "video-mixcut"}}

当前支持的场景为：

·video-qa

·video-mixcut

23、video-qa（视频问答）

视频问答场景支持在 forwardedProps.database 中指定 agent 可见的智能媒资知识库，你可以在 腾讯云控制台 中查看和管理已有知识库。

24、video-mixcut（智能成片）

·视频混剪场景支持在用户消息中带上混剪素材，素材使用自定义的 ContentPart，类型为 mixcut_assets 。

·附件列表放在 metadata.attachments 中，每个附件只填写一个来源字段：url 或 fileId。

完整请求示例：

{ "threadId": "thread-001", "runId": "run-001", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "帮我混剪这些素材" }, { "type": "mixcut_assets", "metadata": { "attachments": [ {"url": " http://example.com/video-a.mp4"}, {"fileId": "vod-file-001"} ] } } ] } ]}

25、6. 外部工具

外部工具是 Agent 调用，但不由服务端直接执行的工具。LLM 决策调用后，服务端挂起执行将工具调用信息返回给客户端，由用户在客户端执行工具并回填执行结果，然后继续 Agent 流程。这适用于需要人工介入或客户端侧执行的场景（如内部知识库查询、审批操作等）。

26、工具 Schema 定义

每个外部工具通过标准 JSON Schema 定义，由客户端传入到请求的 tools 字段进行注册：

{ "name": "<工具名>", "description": "<工具用途描述，LLM 据此决策何时调用>", "parameters": <工具参数的JSON schema定义>}

示例—：clarify 工具 Schema：

{ "name": "clarify", "description": "Ask the user a question when you need clarification…", "parameters": { "type": "object", "properties": { "question": { "type": "string", "description": "The question itself. Do NOT embed options here." }, "choices": { "type": "array", "items": { "type": "string" }, "maxItems": 4, "description": "Selectable options (up to 4). Omit for free-text." } }, "required": ["question"] }}

27、交互流程

外部工具的完整交互模型：

请求 → LLM 决策调用外部工具 ↓ SSE 事件流 TOOL_CALL_START { toolCallId, toolCallName } TOOL_CALL_ARGS { delta (JSON fragment) } TOOL_CALL_END { toolCallId } ↓ 工具执行被挂起，Agent结束当前轮次 RUN_FINISHED[客户端执行外部工具，填入结果] ↓ 发送新请求（/agui/chat） { "threadId": "…", "runId": "…", "messages": [ { "role": "tool", "content": "<工具执行结果>", "toolCallId": "<id>" } ], "tools": [ … ] } → Agent 继续执行

28、SSE 事件示例

id: TOOL_CALL_START_xxxdata: {"type":"TOOL_CALL_START","timestamp":…,"toolCallId":"call-ext-001","toolCallName":"external_search","parentMessageId":"msg-001"}id: TOOL_CALL_ARGS_xxxdata: {"type":"TOOL_CALL_ARGS","timestamp":…,"toolCallId":"call-ext-001","delta":"{\\"query\\":\\"VOD 架构文档\\"}"}id: TOOL_CALL_END_xxxdata: {"type":"TOOL_CALL_END","timestamp":…,"toolCallId":"call-ext-001"}id: RUN_FINISHED_xxxdata: {"type":"RUN_FINISHED","timestamp":…,"threadId":"thread-001","runId":"run-001"}

RUN_FINISHED 为普通结束事件，不携带 outcome 或 interrupts。

29、恢复请求

外部工具执行完毕后，客户端发起新请求，在 messages 中以 role: "tool" 回传执行结果：

{ "threadId": "my-thread-001", "runId": "run-002", "messages": [ { "role": "tool", "content": "<工具执行结果（字符串，LLM 会解读）>", "toolCallId": "call-ext-001" } ], "tools": [ { "name": "clarify", "description": "…", "parameters": {"type":"object","properties":{}} } ]}

tools、forwardedProps 参数请与前一次请求保持一致。

31、Python（AG-UI Python SDK）

安装：pip install ag-ui-protocol

from ag_ui.client import HttpAgentfrom ag_ui.core import RunAgentInput, Message, ResumeEntryBASE_URL = "https://smartmedia.vod-qcloud.com/agui"API_KEY = "<your-token>"def run_agent(thread_id, run_id, message, resume=None, forwarded_props=None): agent = HttpAgent( base_url=BASE_URL, api_key=API_KEY, ) input_params = RunAgentInput( thread_id=thread_id, run_id=run_id, messages=[Message(role="user", content=message)], ) if forwarded_props: input_params.forwarded_props = forwarded_props if resume: input_params.resume = resume interrupts = [] for event in agent.run(input_params): etype = event.type if etype == "TEXT_MESSAGE_CONTENT": print(event.delta, end="", flush=True) elif etype == "TOOL_CALL_START": print(f"\\n[工具] {event.tool_call_name}") elif etype == "RUN_FINISHED": outcome = getattr(event, "outcome", None) if outcome and outcome.type == "interrupt": interrupts = outcome.interrupts for i, intr in enumerate(interrupts): print(f"\\n[中断 {i + 1}] {intr.message} (id: {intr.id})") elif etype == "RUN_ERROR": print(f"\\n[错误] {event.message}") return interrupts# 1. 首次对话interrupts = run_agent( thread_id="thread-001", run_id="run-001", message="搜索海浪视频并裁剪前10秒", forwarded_props={ "approval_mode": "level:low", "scenario_name": "media_management", },)# 2. 每工具独立决策 — 批准第一个，拒绝第二个if interrupts: print(f"\\n共 {len(interrupts)} 个工具需要确认") resume = [] for i, intr in enumerate(interrupts): if i == 0: resume.append(ResumeEntry( interrupt_id=intr.id, status="resolved", )) else: resume.append(ResumeEntry( interrupt_id=intr.id, status="cancelled", payload={"feedback": "不需要此操作"}, )) # 3. 恢复（需携带相同的 approval_mode） run_agent( "thread-001", "run-002", "", resume=resume, forwarded_props={"approval_mode": "level:low"}, )

32、Node.js / TypeScript（AG-UI TypeScript SDK）

安装：npm install @ag-ui/client @ag-ui/core

import { HttpAgent } from "@ag-ui/client";import type { RunAgentInput, Interrupt } from "@ag-ui/core";const BASE_URL = "https://smartmedia.vod-qcloud.com/agui";const API_KEY = "<your-token>";async function runAgent(params: { threadId: string; runId: string; message: string; forwardedProps?: Record<string, unknown>; resume?: Array<{ interruptId: string; status: "resolved" | "cancelled"; payload?: Record<string, unknown>; }>;}): Promise<Interrupt[]> { const agent = new HttpAgent({ serverUrl: BASE_URL, apiKey: API_KEY, }); const input: RunAgentInput = { threadId: params.threadId, runId: params.runId, messages: [{ role: "user", content: params.message }], forwardedProps: params.forwardedProps, resume: params.resume, }; const interrupts: Interrupt[] = []; // 订阅事件流 const unsubscribe = agent.subscribe((event: any) => { switch (event.type) { case "TEXT_MESSAGE_CONTENT": process.stdout.write(event.delta || ""); break; case "TOOL_CALL_START": console.log(`\\n[Tool] ${event.toolCallName}`); break; case "RUN_FINISHED": if (event.outcome?.type === "interrupt") { interrupts.push(…event.outcome.interrupts); interrupts.forEach((intr, i) => console.log(`\\n[Interrupt ${i + 1}] ${intr.message} (id: ${intr.id})`) ); } break; case "RUN_ERROR": console.error(`\\n[Error] ${event.message}`); break; } }); // 运行 Agent await agent.run(input); unsubscribe(); return interrupts;}// 使用示例：每工具独立审批async function main() { // 1. 首次对话 const interrupts = await runAgent({ threadId: "thread-001", runId: "run-001", message: "搜索海浪视频并裁剪前10秒", forwardedProps: { approval_mode: "level:low", scenario_name: "media_management", }, }); // 2. 每工具独立决策 — 批准第一个，拒绝第二个 if (interrupts.length > 0) { console.log(`\\n共 ${interrupts.length} 个工具需要确认`); await runAgent({ threadId: "thread-001", runId: "run-002", message: "", forwardedProps: { approval_mode: "level:low" }, resume: interrupts.map((intr, idx) => ({ interruptId: intr.id, status: idx === 0 ? "resolved" as const : "cancelled" as const, payload: idx === 0 ? undefined : { feedback: "不需要此操作" }, })), }); }}main();

33、cURL

# 启动对话（可能触发多工具中断）curl -N -X POST https://smartmedia.vod-qcloud.com/agui/chat \\ -H "Content-Type: application/json" \\ -H "Authorization: Bearer <token>" \\ -d '{ "threadId": "thread-001", "runId": "run-001", "messages": [{"role": "user", "content": "搜索海浪视频并裁剪前10秒"}], "forwardedProps": { "approval_mode": "level:low", "scenario_name": "media_management" } }'# 恢复 — 每工具独立决策curl -N -X POST https://smartmedia.vod-qcloud.com/agui/chat \\ -H "Content-Type: application/json" \\ -H "Authorization: Bearer <token>" \\ -d '{ "threadId": "thread-001", "runId": "run-002", "messages": [{"role": "user", "content": ""}], "forwardedProps": {"approval_mode": "level:low"}, "resume": [ {"interruptId": "lineage-uuid:ckpt-uuid:call-001", "status": "resolved"}, {"interruptId": "lineage-uuid:ckpt-uuid:call-002", "status": "cancelled", "payload": {"feedback": "不需要裁剪"}} ] }'# 查询历史curl -X POST https://smartmedia.vod-qcloud.com/agui/history \\ -H "Content-Type: application/json" \\ -H "Authorization: Bearer <token>" \\ -d '{"threadId": "thread-001"}'

34、8. 事件处理建议

RUN_STARTED → UI: 显示"正在思考…"REASONING_START → UI: 开始推理区域REASONING_MESSAGE_* → UI: 展示推理过程（可折叠）REASONING_END → UI: 结束推理区域TEXT_MESSAGE_* → UI: 流式渲染回复TOOL_CALL_START/END → UI: 显示工具调用卡片TOOL_CALL_RESULT → UI: 展示工具结果RUN_FINISHED → 检查 outcome: 无 outcome → 正常结束 type=interrupt → 进入 HITL 确认流程RUN_ERROR → UI: 显示错误

35、附录：智能成片 Prompt 指南

以下仅为示例 Prompt，请根据实际场景需要撰写提示词。