一、为什么 AI 平台需要专门的可观测性方案
传统 Web 服务的可观测性已经相对成熟：监控 QPS、Latency、Error Rate 三大指标，配合 ELK 日志和分布式链路追踪，一套经典方案可以覆盖 90% 的问题排查场景。
但 AI 平台不一样。AI 平台的"请求"是 Token 序列，"响应"也是 Token 序列，整个过程发生在 GPU 的黑箱里。传统的可观测性工具对大模型场景几乎是"睁眼瞎"：
指标层缺失：GPU 利用率、显存占用、KV Cache 命中率、Token 吞吐……这些是 AI 平台独有的关键指标
日志层不匹配：LLM 的推理日志包含 Prompt、Completion、Token 统计，这些信息量巨大且格式特殊
链路层更复杂：一次用户请求可能触发 Prompt Enrichment → RAG Retrieval → Model Inference → Response Formatting 多个环节
本文将介绍如何构建一套面向 AI 平台的可观测性方案，覆盖指标（Metrics）、日志（Logs）、链路（Traces） 三大支柱。
二、AI 平台 vs 传统可观测性
┌─────────────────────────────────────────────────────────────────────────────────┐
│ AI 平台可观测性 vs 传统可观测性 │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ 传统 Web 服务 │ │
│ │ │ │
│ │ 请求 ──▶ 网关 ──▶ 服务 A ──▶ 服务 B ──▶ 数据库 │ │
│ │ │ │ │ │
│ │ ▼ ▼ │ │
│ │ 指标: QPS, Latency, Error Rate 指标: CPU, Memory, 连接数 │ │
│ │ 日志: HTTP Access Log, Error Log 日志: SQL Log, Slow Query │ │
│ │ 链路: HTTP Header Trace ID 链路: Dubbo/gRPC Trace │ │
│ │ │ │
│ │ 特点: 结构清晰，依赖明确，单次请求耗时可控 │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ AI 平台 │ │
│ │ │ │
│ │ 用户请求 ──▶ Prompt Enrich ──▶ RAG ──▶ LLM ──▶ 后处理 ──▶ 响应 │ │
│ │ │ │ │ │ │ │
│ │ ▼ ▼ ▼ ▼ │ │
│ │ 指标: TTFT, TPOT, 指标: 指标: GPU利用率 指标: │ │
│ │ Token吞吐 召回率 命中率 KV Cache 格式正确率 │ │
│ │ │ │ │ │ │
│ │ ▼ ▼ ▼ │ │
│ │ 日志: Prompt文 日志: 日志: 日志: 模型权重 │ │
│ │ Completion 检索结果 向量距离 梯度日志 │ │
│ │ Token统计 │ │
│ │ │ │ │ │ │
│ │ ▼ ▼ ▼ │ │
│ │ 链路: 用户请求 ──▶ 检索请求 ──▶ 模型调用 ──▶ 采样请求 │ │
│ │ (span 跨多服务，包含 tokenize/detokenize/forward 等子操作) │ │
│ │ │ │
│ │ 特点: GPU 是黑箱，Token 粒度追踪，KV Cache 动态变化，批量调度复杂 │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────────┘三、三大支柱在 AI 场景的具体化
3.1 指标（Metrics）
AI 平台的核心指标与传统服务有显著不同：
3.2 日志（Logs）
AI 平台的日志有其独特结构：
{
"timestamp": "2025-12-01T10:30:00.000Z",
"log_type": "llm_inference",
"request_id": "req-uuid-12345",
"trace_id": "trace-uuid-67890",
"request": {
"model": "Qwen2.5-72B-Instruct",
"prompt": "请介绍一下北京的历史...",
"prompt_tokens": 28,
"max_tokens": 1024,
"temperature": 0.7,
"top_p": 0.9
},
"response": {
"completion": "北京是中华人民共和国的首都...",
"completion_tokens": 156,
"finish_reason": "stop",
"generated_tokens": [234, 567, 890, ...]
},
"metrics": {
"ttft_ms": 45.2,
"tpot_ms": 12.3,
"e2e_latency_ms": 2018.6,
"prompt_tokens": 28,
"completion_tokens": 156,
"total_tokens": 184
},
"system": {
"gpu_ids": [0, 1, 2, 3],
"gpu_memory_used_gb": 280.5,
"batch_size": 32,
"kv_cache_hit": false
}
}3.3 链路（Traces）
AI 平台的分布式链路需要追踪从用户请求到模型推理的完整路径：
┌─────────────────────────────────────────────────────────────────────────────────┐
│ LLM 请求完整链路 │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ trace_id: abc123 │
│ │
│ ├─ span: api_gateway │
│ │ └─ 用户请求进来，记录 metadata │
│ │ │ │
│ │ ▼ │
│ │ ├─ span: prompt_enrichment │
│ │ │ └─ 添加 system prompt、历史对话、few-shot examples │
│ │ │ │ │
│ │ │ ▼ │
│ │ │ ├─ span: template_rendering │
│ │ │ │ └─ 模板变量替换 │
│ │ │ │ │
│ │ │ └─ span: conversation_history │
│ │ │ └─ 从存储加载对话历史 │
│ │ │ │
│ │ └─ span: rag_retrieval │
│ │ │ │
│ │ ├─ span: embedding │
│ │ │ └─ 将 prompt 向量化 │
│ │ │ │
│ │ ├─ span: vector_search │
│ │ │ └─ 向量数据库检索 │
│ │ │ │
│ │ └─ span: rerank │
│ │ └─ 重排序 │
│ │ │
│ │ └─ span: model_inference │
│ │ │ │
│ │ ├─ span: tokenize │
│ │ │ └─ 分词 │
│ │ │ │
│ │ ├─ span: prefill │
│ │ │ └─ Prefill 阶段（计算密集） │
│ │ │ │
│ │ ├─ span: decode (×N tokens) │
│ │ │ └─ Decode 阶段（显存带宽密集） │
│ │ │ │
│ │ └─ span: detokenize │
│ │ └─ 反分词 │
│ │ │
│ └─ span: response_postprocess │
│ └─ 后处理：安全检查、格式转换、敏感词过滤 │
│ │
└─────────────────────────────────────────────────────────────────────────────────┘四、监控架构图
┌─────────────────────────────────────────────────────────────────────────────────┐
│ AI 平台可观测性架构图 │
├─────────────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ 数据采集层 │ │
│ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ vLLM/TGI │ │ DCGM Exporter│ │ FastAPI/uvicorn│ │ │
│ │ │ 内置指标 │ │ GPU 指标 │ │ 应用日志 │ │ │
│ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │ │
│ │ │ │ │ │ │
│ │ └──────────────────┴──────────────────┘ │ │
│ │ │ │ │
│ └────────────────────────────┼─────────────────────────────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────────────┐ │
│ │ 指标存储层 │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Prometheus / Mimir │ │ │
│ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │
│ │ │ │ GPU指标 │ │ 推理指标 │ │ 业务指标 │ │ 系统指标 │ │ │ │
│ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │
│ │ └─────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ └────────────────────────────────────┬────────────────────────────────────┘ │
│ │ │
│ ┌────────────────────────────────────┴────────────────────────────────────┐ │
│ │ 日志存储层 │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Loki / Elasticsearch │ │ │
│ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │
│ │ │ │ 推理日志 │ │ 训练日志 │ │ 访问日志 │ │ 错误日志 │ │ │ │
│ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │
│ │ └─────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ └────────────────────────────────────┬────────────────────────────────────┘ │
│ │ │
│ ┌────────────────────────────────────┴────────────────────────────────────┐ │
│ │ 链路存储层 │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Jaeger / Tempo │ │ │
│ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │
│ │ │ │ API层Span│ │ RAG链路 │ │ 模型Span │ │ 后处理Span│ │ │ │
│ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │
│ │ └─────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ └────────────────────────────────────┬────────────────────────────────────┘ │
│ │ │
│ ┌────────────────────────────────────┴────────────────────────────────────┐ │
│ │ 可视化层 │ │
│ │ │ │
│ │ ┌─────────────────────────────────────────────────────────────────┐ │ │
│ │ │ Grafana 仪表盘 │ │ │
│ │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │
│ │ │ │ GPU监控 │ │ 推理监控 │ │ 日志查询 │ │ 链路追踪 │ │ │ │
│ │ │ └──────────┘ └──────────┘ └──────────┘ └──────────┘ │ │ │
│ │ └─────────────────────────────────────────────────────────────────┘ │ │
│ │ │ │
│ └─────────────────────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────────────┘五、GPU 指标采集
5.1 DCGM Exporter 部署
DCGM (Data Center GPU Manager) 是 NVIDIA 提供的 GPU 监控工具，DCGM Exporter 可以将 GPU 指标暴露给 Prometheus：
# dcgm-exporter.yaml
# DCGM Exporter Kubernetes 部署配置
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: dcgm-exporter
namespace: monitoring
spec:
selector:
matchLabels:
app: dcgm-exporter
template:
metadata:
labels:
app: dcgm-exporter
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9400"
prometheus.io/path: "/metrics"
spec:
containers:
- name: exporter
image: nvcr.io/nvidia/k8s/dcgm-exporter:3.3.7-3.1.5-ubuntu22.04
securityContext:
privileged: true
env:
- name: DCGM_EXPORTER_INTERVAL
value: "15" # 采集间隔（秒）
- name: DCGM_EXPORTER_COLLECTORS
value: "/etc/dcgm-exporter/dcgm-fields.csv"
ports:
- name: metrics
containerPort: 9400
protocol: TCP
resources:
requests:
memory: "128Mi"
cpu: "100m"
limits:
memory: "256Mi"
cpu: "500m"
volumeMounts:
- name: dcgm-socket
mountPath: /var/lib/dcgm
- name: dcgm-config
mountPath: /etc/dcgm-exporter
volumes:
- name: dcgm-socket
hostPath:
path: /var/lib/dcgm
- name: dcgm-config
configMap:
name: dcgm-exporter-config
tolerations:
- key: "nvidia.com/gpu"
operator: "Exists"
effect: "NoSchedule"
---
# DCGM Exporter Service
apiVersion: v1
kind: Service
metadata:
name: dcgm-exporter
namespace: monitoring
spec:
type: ClusterIP
selector:
app: dcgm-exporter
ports:
- name: metrics
port: 9400
targetPort: metrics5.2 Prometheus 配置
# prometheus-config.yaml
# Prometheus 抓取配置
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: monitoring
data:
prometheus.yml: |
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
# DCGM GPU 指标
- job_name: 'dcgm-exporter'
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- monitoring
relabel_configs:
- source_labels: [__meta_kubernetes_endpoint_port_name]
action: keep
regex: metrics
- source_labels: [__meta_kubernetes_pod_name]
target_label: pod
# vLLM 推理指标
- job_name: 'vllm'
static_configs:
- targets: ['vllm-service:8000']
metrics_path: '/metrics'
relabel_configs:
- source_labels: [__address__]
regex: '([^:]+):(\d+)'
target_label: __address__
replacement: '${1}:8000'
# API Server 指标
- job_name: 'api-server'
static_configs:
- targets: ['api-server:8000']
metrics_path: '/metrics'
# 自定义 LLM 指标
- job_name: 'llm-custom-metrics'
static_configs:
- targets: ['llm-metrics-sidecar:9090']六、自定义 LLM 指标采集
6.1 Prometheus 指标 SDK 封装
#!/usr/bin/env python3
"""
自定义 LLM 指标采集模块
为 LLM 推理服务添加 Prometheus 指标暴露
"""
from prometheus_client import Counter, Histogram, Gauge, Info, CollectorRegistry, REGISTRY
from prometheus_client.multiprocess import MultiProcessCollector
from prometheus_client.exposition import generate_latest, CONTENT_TYPE_LATEST
from typing import Optional, Dict, Any, List
import time
import logging
from functools import wraps
from contextlib import contextmanager
# 配置日志
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class LLMMetrics:
"""LLM 推理指标收集器"""
def __init__(self, namespace: str = "llm", service_name: str = "inference"):
"""
初始化指标收集器
Args:
namespace: 指标命名空间
service_name: 服务名称
"""
self.namespace = namespace
self.service_name = service_name
# 1. 请求相关指标
self.request_total = Counter(
f"{namespace}_requests_total",
"LLM 请求总数",
["model", "status", "finish_reason"]
)
self.request_in_progress = Gauge(
f"{namespace}_requests_in_progress",
"正在处理的请求数",
["model"]
)
# 2. Token 相关指标
self.prompt_tokens_total = Counter(
f"{namespace}_prompt_tokens_total",
"Prompt Token 总数",
["model"]
)
self.completion_tokens_total = Counter(
f"{namespace}_completion_tokens_total",
"Completion Token 总数",
["model"]
)
self.tokens_per_request = Histogram(
f"{namespace}_tokens_per_request",
"每个请求的 Token 数量分布",
["model", "type"], # type: prompt/completion/total
buckets=[16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192]
)
# 3. 延迟相关指标
self.latency_seconds = Histogram(
f"{namespace}_latency_seconds",
"请求延迟分布",
["model", "stage"], # stage: total/ttft/tpot/prefill/decode
buckets=[0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0, 30.0, 60.0]
)
# 4. 吞吐量指标
self.throughput_tokens = Gauge(
f"{namespace}_throughput_tokens_per_second",
"Token 吞吐率（tokens/s）",
["model"]
)
self.throughput_requests = Gauge(
f"{namespace}_throughput_requests_per_second",
"请求吞吐率（requests/s）",
["model"]
)
# 5. 错误相关指标
self.error_total = Counter(
f"{namespace}_errors_total",
"错误总数",
["model", "error_type"] # error_type: timeout/validation/oom/timeout
)
# 6. GPU 相关指标
self.gpu_memory_used = Gauge(
f"{namespace}_gpu_memory_used_bytes",
"GPU 显存使用量（字节）",
["model", "gpu_id"]
)
self.gpu_utilization = Gauge(
f"{namespace}_gpu_utilization_ratio",
"GPU 利用率",
["model", "gpu_id"]
)
# 7. KV Cache 指标（vLLM 特有）
self.kv_cache_usage = Gauge(
f"{namespace}_kv_cache_usage_ratio",
"KV Cache 利用率",
["model"]
)
self.kv_cache_hits = Counter(
f"{namespace}_kv_cache_hits_total",
"KV Cache 命中次数",
["model"]
)
# 8. 模型信息
self.model_info = Info(
f"{namespace}_model_info",
"模型信息"
)
# 内部状态
self._total_tokens_last_check = 0
self._last_check_time = time.time()
def record_request_start(
self,
model: str,
request_id: str,
prompt_tokens: int
):
"""
记录请求开始
Args:
model: 模型名称
request_id: 请求 ID
prompt_tokens: Prompt 的 Token 数
"""
self.request_in_progress.labels(model=model).inc()
self.prompt_tokens_total.labels(model=model).inc(prompt_tokens)
self.tokens_per_request.labels(model=model, type="prompt").observe(prompt_tokens)
def record_request_end(
self,
model: str,
request_id: str,
status: str,
finish_reason: str,
prompt_tokens: int,
completion_tokens: int,
latency_seconds: float,
ttft_seconds: Optional[float] = None,
tpot_seconds: Optional[float] = None
):
"""
记录请求结束
Args:
model: 模型名称
request_id: 请求 ID
status: 请求状态 (success/error)
finish_reason: 结束原因 (stop/length/error)
prompt_tokens: Prompt Token 数
completion_tokens: Completion Token 数
latency_seconds: 总延迟（秒）
ttft_seconds: 首 Token 延迟（秒）
tpot_seconds: 每 Token 延迟（秒）
"""
# 更新请求状态
self.request_in_progress.labels(model=model).dec()
self.request_total.labels(
model=model,
status=status,
finish_reason=finish_reason
).inc()
# 记录 Token 统计
self.completion_tokens_total.labels(model=model).inc(completion_tokens)
self.tokens_per_request.labels(model=model, type="completion").observe(completion_tokens)
self.tokens_per_request.labels(model=model, type="total").observe(
prompt_tokens + completion_tokens
)
# 记录延迟
self.latency_seconds.labels(model=model, stage="total").observe(latency_seconds)
if ttft_seconds is not None:
self.latency_seconds.labels(model=model, stage="ttft").observe(ttft_seconds)
if tpot_seconds is not None:
self.latency_seconds.labels(model=model, stage="tpot").observe(tpot_seconds)
def record_error(
self,
model: str,
error_type: str,
error_message: Optional[str] = None
):
"""
记录错误
Args:
model: 模型名称
error_type: 错误类型
error_message: 错误信息（用于调试，不用于指标）
"""
self.error_total.labels(model=model, error_type=error_type).inc()
logger.warning(f"LLM 错误: model={model}, type={error_type}, msg={error_message}")
def update_gpu_metrics(
self,
model: str,
gpu_metrics: Dict[str, Dict[str, float]]
):
"""
更新 GPU 指标
Args:
model: 模型名称
gpu_metrics: GPU 指标字典 {gpu_id: {"memory_used": bytes, "utilization": ratio}}
"""
for gpu_id, metrics in gpu_metrics.items():
if "memory_used" in metrics:
self.gpu_memory_used.labels(model=model, gpu_id=gpu_id).set(
metrics["memory_used"]
)
if "utilization" in metrics:
self.gpu_utilization.labels(model=model, gpu_id=gpu_id).set(
metrics["utilization"]
)
def update_kv_cache_metrics(
self,
model: str,
usage_ratio: float,
hit: bool = False
):
"""
更新 KV Cache 指标
Args:
model: 模型名称
usage_ratio: Cache 使用率
hit: 是否命中
"""
self.kv_cache_usage.labels(model=model).set(usage_ratio)
if hit:
self.kv_cache_hits.labels(model=model).inc()
def update_throughput(self, model: str, tokens_per_second: float, requests_per_second: float):
"""
更新吞吐量指标
Args:
model: 模型名称
tokens_per_second: Token 吞吐率
requests_per_second: 请求吞吐率
"""
self.throughput_tokens.labels(model=model).set(tokens_per_second)
self.throughput_requests.labels(model=model).set(requests_per_second)
def set_model_info(self, model: str, info: Dict[str, str]):
"""
设置模型信息
Args:
model: 模型名称
info: 模型信息字典
"""
self.model_info.info({**info, "model": model})
@contextmanager
def track_latency(self, model: str, stage: str):
"""
上下文管理器：追踪延迟
Usage:
with metrics.track_latency("qwen", "prefill"):
# 执行 prefill 操作
prefill()
"""
start_time = time.time()
try:
yield
finally:
latency = time.time() - start_time
self.latency_seconds.labels(model=model, stage=stage).observe(latency)
# ============ FastAPI 集成示例 ============
from fastapi import FastAPI, Request, Response
from fastapi.responses import PlainTextResponse
import uuid
import time
# 创建指标收集器
metrics = LLMMetrics(namespace="llm", service_name="inference")
# 创建 FastAPI 应用
app = FastAPI(title="LLM Inference API with Metrics")
@app.middleware("http")
async def track_request_metrics(request: Request, call_next):
"""请求指标追踪中间件"""
request_id = str(uuid.uuid4())
request.state.request_id = request_id
# 获取模型名称（从路径或参数）
model = request.query_params.get("model", "default")
# 记录请求开始
# 注意：这里只是简化示例，实际需要解析 request body 获取 prompt_tokens
prompt_tokens = 0 # 需要从请求中提取
metrics.record_request_start(model, request_id, prompt_tokens)
start_time = time.time()
try:
response = await call_next(request)
status = "success"
finish_reason = "stop"
except Exception as e:
status = "error"
finish_reason = "error"
metrics.record_error(model, type(e).__name__, str(e))
raise
finally:
latency = time.time() - start_time
# 模拟记录（实际需要从响应中获取）
metrics.record_request_end(
model=model,
request_id=request_id,
status=status,
finish_reason=finish_reason,
prompt_tokens=prompt_tokens,
completion_tokens=0, # 从响应获取
latency_seconds=latency
)
return response
@app.get("/metrics")
async def get_metrics():
"""Prometheus 抓取端点"""
return Response(
content=generate_latest(REGISTRY),
media_type=CONTENT_TYPE_LATEST
)
@app.get("/health")
async def health():
"""健康检查端点"""
return {"status": "healthy"}
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)6.2 自定义指标采集服务
#!/usr/bin/env python3
"""
LLM 自定义指标采集服务
从 vLLM API 获取模型特定指标并暴露给 Prometheus
"""
import requests
import time
import logging
from typing import Dict, List, Optional
from prometheus_client import start_http_server, Gauge, Counter
from threading import Thread
import argparse
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
class LLMCustomMetricsCollector:
"""LLM 自定义指标采集器"""
def __init__(
self,
vllm_url: str,
collect_interval: int = 10,
port: int = 9090
):
self.vllm_url = vllm_url.rstrip("/")
self.collect_interval = collect_interval
self.port = port
# 初始化 Prometheus 指标
self._init_metrics()
# 运行状态
self.running = False
def _init_metrics(self):
"""初始化 Prometheus 指标"""
# KV Cache 利用率
self.kv_cache_usage = Gauge(
"vllm_kv_cache_usage_ratio",
"KV Cache 利用率",
["model"]
)
# 前缀缓存命中率
self.prefix_cache_hit_rate = Gauge(
"vllm_prefix_cache_hit_rate",
"前缀缓存命中率",
["model"]
)
# 正在处理的请求数
self.num_requests_running = Gauge(
"vllm_num_requests_running",
"正在处理的请求数",
["model"]
)
# 等待中的请求数
self.num_requests_waiting = Gauge(
"vllm_num_requests_waiting",
"等待中的请求数",
["model"]
)
# GPU 显存使用
self.gpu_memory_used = Gauge(
"vllm_gpu_memory_used_bytes",
"GPU 显存使用量（字节）",
["model", "gpu_id"]
)
# GPU 预估显存（模型加载所需）
self.gpu_memory_reserved = Gauge(
"vllm_gpu_memory_reserved_bytes",
"GPU 显存预留量（字节）",
["model", "gpu_id"]
)
# 分块 Prefill 指标
self.chunked_prefill_enabled = Gauge(
"vllm_chunked_prefill_enabled",
"是否启用分块 Prefill",
["model"]
)
# 自定义计数器
self.custom_counter = Counter(
"vllm_custom_requests_total",
"自定义请求计数",
["model", "type"]
)
def collect(self):
"""从 vLLM API 采集指标"""
try:
# 获取模型列表
models_url = f"{self.vllm_url}/v1/models"
models_response = requests.get(models_url, timeout=5)
models_response.raise_for_status()
models = models_response.json().get("data", [])
for model_info in models:
model_name = model_info.get("id", "unknown")
# 获取详细统计
stats_url = f"{self.vllm_url}/stats"
stats_response = requests.get(
stats_url,
params={"model": model_name},
timeout=5
)
if stats_response.status_code == 200:
stats = stats_response.json()
self._update_metrics(model_name, stats)
else:
logger.warning(
f"获取模型 {model_name} 统计失败: {stats_response.status_code}"
)
except requests.RequestException as e:
logger.error(f"采集指标失败: {e}")
def _update_metrics(self, model: str, stats: Dict):
"""更新 Prometheus 指标"""
# KV Cache 使用率
kv_cache_usage = stats.get("kv_cache_usage_ratio", 0)
self.kv_cache_usage.labels(model=model).set(kv_cache_usage)
# 前缀缓存命中率
prefix_hit_rate = stats.get("prefix_cache_hit_rate", 0)
self.prefix_cache_hit_rate.labels(model=model).set(prefix_hit_rate)
# 请求数
num_running = stats.get("num_running", 0)
num_waiting = stats.get("num_waiting", 0)
self.num_requests_running.labels(model=model).set(num_running)
self.num_requests_waiting.labels(model=model).set(num_waiting)
# GPU 显存
gpu_memory = stats.get("gpu_memory", {})
for gpu_id, memory_info in gpu_memory.items():
if isinstance(memory_info, dict):
self.gpu_memory_used.labels(
model=model,
gpu_id=gpu_id
).set(memory_info.get("used", 0))
self.gpu_memory_reserved.labels(
model=model,
gpu_id=gpu_id
).set(memory_info.get("reserved", 0))
# 分块 Prefill
chunked = 1 if stats.get("chunked_prefill", False) else 0
self.chunked_prefill_enabled.labels(model=model).set(chunked)
logger.debug(f"更新指标完成: model={model}")
def start(self):
"""启动采集服务"""
# 启动 Prometheus HTTP 服务器
start_http_server(self.port)
logger.info(f"Prometheus 指标端点已启动: :{self.port}/metrics")
self.running = True
# 定期采集
while self.running:
self.collect()
time.sleep(self.collect_interval)
def stop(self):
"""停止采集服务"""
self.running = False
logger.info("指标采集服务已停止")
def main():
parser = argparse.ArgumentParser(description="LLM 自定义指标采集服务")
parser.add_argument(
"--vllm-url",
type=str,
default="http://localhost:8000",
help="vLLM 服务地址"
)
parser.add_argument(
"--port",
type=int,
default=9090,
help="Prometheus 指标端点端口"
)
parser.add_argument(
"--interval",
type=int,
default=10,
help="采集间隔（秒）"
)
args = parser.parse_args()
collector = LLMCustomMetricsCollector(
vllm_url=args.vllm_url,
collect_interval=args.interval,
port=args.port
)
try:
collector.start()
except KeyboardInterrupt:
collector.stop()
if __name__ == "__main__":
main()七、Grafana Dashboard 配置
7.1 LLM 推理监控 Dashboard
{
"annotations": {
"list": [
{
"builtIn": 1,
"datasource": "-- Grafana --",
"enable": true,
"hide": true,
"iconColor": "rgba(0, 211, 255, 1)",
"name": "Annotations & Alerts",
"type": "dashboard"
}
]
},
"editable": true,
"fiscalYearStartMonth": 0,
"graphTooltip": 0,
"id": null,
"links": [],
"liveNow": false,
"panels": [
{
"title": "请求延迟 (TTFT)",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
"targets": [
{
"expr": "histogram_quantile(0.50, sum(rate(llm_latency_seconds_bucket{stage=\"ttft\"}[5m])) by (le, model))",
"legendFormat": "p50 - {{model}}",
"refId": "A"
},
{
"expr": "histogram_quantile(0.95, sum(rate(llm_latency_seconds_bucket{stage=\"ttft\"}[5m])) by (le, model))",
"legendFormat": "p95 - {{model}}",
"refId": "B"
},
{
"expr": "histogram_quantile(0.99, sum(rate(llm_latency_seconds_bucket{stage=\"ttft\"}[5m])) by (le, model))",
"legendFormat": "p99 - {{model}}",
"refId": "C"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"drawStyle": "Line",
"lineInterpolation": "Smooth",
"showPoints": "Never"
}
}
}
},
{
"title": "Token 吞吐率",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 0},
"targets": [
{
"expr": "llm_throughput_tokens_per_second",
"legendFormat": "{{model}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "tokens/s",
"custom": {
"drawStyle": "Area",
"lineInterpolation": "Smooth"
}
}
}
},
{
"title": "GPU 显存使用",
"type": "gauge",
"gridPos": {"h": 8, "w": 8, "x": 0, "y": 8},
"targets": [
{
"expr": "vllm_gpu_memory_used_bytes / vllm_gpu_memory_reserved_bytes",
"legendFormat": "{{model}}-GPU{{gpu_id}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"max": 1,
"thresholds": {
"mode": "absolute",
"steps": [
{"value": 0, "color": "green"},
{"value": 0.7, "color": "yellow"},
{"value": 0.85, "color": "red"}
]
}
}
}
},
{
"title": "KV Cache 利用率",
"type": "timeseries",
"gridPos": {"h": 8, "w": 8, "x": 8, "y": 8},
"targets": [
{
"expr": "vllm_kv_cache_usage_ratio",
"legendFormat": "{{model}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"max": 1,
"custom": {
"drawStyle": "Line",
"lineInterpolation": "Smooth"
}
}
}
},
{
"title": "请求队列深度",
"type": "timeseries",
"gridPos": {"h": 8, "w": 8, "x": 16, "y": 8},
"targets": [
{
"expr": "vllm_num_requests_running",
"legendFormat": "Running - {{model}}",
"refId": "A"
},
{
"expr": "vllm_num_requests_waiting",
"legendFormat": "Waiting - {{model}}",
"refId": "B"
}
],
"fieldConfig": {
"defaults": {
"custom": {
"drawStyle": "Line",
"lineInterpolation": "Smooth"
}
}
}
},
{
"title": "错误率",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 16},
"targets": [
{
"expr": "sum(rate(llm_errors_total[5m])) by (model, error_type) / sum(rate(llm_requests_total[5m])) by (model)",
"legendFormat": "{{model}} - {{error_type}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"custom": {
"drawStyle": "Line"
}
}
}
},
{
"title": "Token 分布",
"type": "histogram",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 16},
"targets": [
{
"expr": "sum(rate(llm_tokens_per_request_bucket[5m])) by (le, model)",
"legendFormat": "{{model}}",
"refId": "A"
}
]
}
],
"refresh": "10s",
"schemaVersion": 38,
"tags": ["llm", "inference", "gpu"],
"templating": {
"list": []
},
"time": {
"from": "now-1h",
"to": "now"
},
"timepicker": {},
"timezone": "browser",
"title": "LLM 推理监控",
"uid": "llm-inference-dashboard",
"version": 1,
"weekStart": ""
}7.2 GPU 监控 Dashboard
{
"dashboard": {
"id": null,
"uid": "gpu-monitoring",
"title": "GPU 集群监控",
"tags": ["gpu", "dcgm", "infrastructure"],
"timezone": "browser",
"panels": [
{
"title": "GPU 利用率",
"type": "stat",
"gridPos": {"h": 6, "w": 6, "x": 0, "y": 0},
"targets": [
{
"expr": "avg(DCGM_FI_DEV_GPU_UTIL)",
"legendFormat": "平均利用率",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{"value": 0, "color": "red"},
{"value": 50, "color": "yellow"},
{"value": 80, "color": "green"}
]
}
}
}
},
{
"title": "GPU 显存使用",
"type": "stat",
"gridPos": {"h": 6, "w": 6, "x": 6, "y": 0},
"targets": [
{
"expr": "avg(DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE)",
"legendFormat": "显存使用率",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"thresholds": {
"mode": "absolute",
"steps": [
{"value": 0, "color": "green"},
{"value": 70, "color": "yellow"},
{"value": 90, "color": "red"}
]
}
}
}
},
{
"title": "GPU 温度",
"type": "gauge",
"gridPos": {"h": 6, "w": 6, "x": 12, "y": 0},
"targets": [
{
"expr": "avg(DCGM_FI_DEV_GPU_TEMP)",
"legendFormat": "",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "celsius",
"max": 100,
"thresholds": {
"mode": "absolute",
"steps": [
{"value": 0, "color": "blue"},
{"value": 70, "color": "green"},
{"value": 85, "color": "yellow"},
{"value": 95, "color": "red"}
]
}
}
}
},
{
"title": "功耗",
"type": "stat",
"gridPos": {"h": 6, "w": 6, "x": 18, "y": 0},
"targets": [
{
"expr": "sum(DCGM_FI_DEV_POWER_USAGE)",
"legendFormat": "",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "watt"
}
}
},
{
"title": "GPU 利用率详情",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 6},
"targets": [
{
"expr": "DCGM_FI_DEV_GPU_UTIL",
"legendFormat": "GPU {{gpu}}",
"refId": "A"
}
],
"fieldConfig": {
"defaults": {
"unit": "percent",
"custom": {
"drawStyle": "Line",
"lineInterpolation": "Smooth"
}
}
}
},
{
"title": "显存使用详情",
"type": "timeseries",
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 6},
"targets": [
{
"expr": "DCGM_FI_DEV_FB_USED",
"legendFormat": "已用 - GPU {{gpu}}",
"refId": "A"
},
{
"expr": "DCGM_FI_DEV_FB_FREE",
"legendFormat": "剩余 - GPU {{gpu}}",
"refId": "B"
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes",
"custom": {
"drawStyle": "Area",
"stacking": {"mode": "normal"}
}
}
}
}
],
"refresh": "5s",
"schemaVersion": 38,
"version": 1
}
}八、告警规则配置
8.1 Prometheus 告警规则
# alermanager-rules.yaml
# Prometheus 告警规则
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-alerts
namespace: monitoring
data:
llm-alerts.yml: |
groups:
- name: llm_inference_alerts
rules:
# GPU 显存告警
- alert: GPU_Memory_High
expr: DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE > 0.90
for: 5m
labels:
severity: warning
component: gpu
annotations:
summary: "GPU 显存使用率过高"
description: "GPU {{ $labels.gpu }} 显存使用率超过 90%，当前值: {{ $value | humanizePercentage }}"
- alert: GPU_Memory_Critical
expr: DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_USED + DCGM_FI_DEV_FB_FREE > 0.95
for: 1m
labels:
severity: critical
component: gpu
annotations:
summary: "GPU 显存即将耗尽"
description: "GPU {{ $labels.gpu }} 显存使用率超过 95%，即将 OOM"
# GPU 利用率告警
- alert: GPU_Utilization_Low
expr: DCGM_FI_DEV_GPU_UTIL < 10
for: 10m
labels:
severity: info
component: gpu
annotations:
summary: "GPU 利用率过低"
description: "GPU {{ $labels.gpu }} 利用率低于 10%，可能存在资源浪费"
- alert: GPU_Utilization_High
expr: DCGM_FI_DEV_GPU_UTIL > 95
for: 5m
labels:
severity: warning
component: gpu
annotations:
summary: "GPU 利用率过高"
description: "GPU {{ $labels.gpu }} 利用率超过 95%"
# GPU 温度告警
- alert: GPU_Temperature_High
expr: DCGM_FI_DEV_GPU_TEMP > 85
for: 5m
labels:
severity: warning
component: gpu
annotations:
summary: "GPU 温度过高"
description: "GPU {{ $labels.gpu }} 温度超过 85°C，当前值: {{ $value }}°C"
- alert: GPU_Temperature_Critical
expr: DCGM_FI_DEV_GPU_TEMP > 95
for: 1m
labels:
severity: critical
component: gpu
annotations:
summary: "GPU 温度危险"
description: "GPU {{ $labels.gpu }} 温度超过 95°C，即将触发降频或关机"
# LLM 延迟告警
- alert: LLM_TTFT_High
expr: histogram_quantile(0.95, sum(rate(llm_latency_seconds_bucket{stage="ttft"}[5m])) by (le, model)) > 5
for: 5m
labels:
severity: warning
component: inference
annotations:
summary: "LLM 首 Token 延迟过高"
description: "模型 {{ $labels.model }} 的 p95 TTFT 超过 5 秒"
- alert: LLM_TTFT_Critical
expr: histogram_quantile(0.99, sum(rate(llm_latency_seconds_bucket{stage="ttft"}[5m])) by (le, model)) > 10
for: 2m
labels:
severity: critical
component: inference
annotations:
summary: "LLM 首 Token 延迟严重"
description: "模型 {{ $labels.model }} 的 p99 TTFT 超过 10 秒"
# 错误率告警
- alert: LLM_Error_Rate_High
expr: sum(rate(llm_errors_total[5m])) by (model) / sum(rate(llm_requests_total[5m])) by (model) > 0.01
for: 5m
labels:
severity: warning
component: inference
annotations:
summary: "LLM 错误率过高"
description: "模型 {{ $labels.model }} 错误率超过 1%"
- alert: LLM_Error_Rate_Critical
expr: sum(rate(llm_errors_total[5m])) by (model) / sum(rate(llm_requests_total[5m])) by (model) > 0.05
for: 1m
labels:
severity: critical
component: inference
annotations:
summary: "LLM 错误率严重"
description: "模型 {{ $labels.model }} 错误率超过 5%"
# KV Cache 告警
- alert: KV_Cache_Full
expr: vllm_kv_cache_usage_ratio > 0.95
for: 5m
labels:
severity: warning
component: inference
annotations:
summary: "KV Cache 接近满载"
description: "模型 {{ $labels.model }} 的 KV Cache 使用率超过 95%"
# 请求积压告警
- alert: Request_Backlog_High
expr: vllm_num_requests_waiting > 100
for: 5m
labels:
severity: warning
component: inference
annotations:
summary: "请求积压过多"
description: "模型 {{ $labels.model }} 等待中的请求超过 100 个"
- alert: Request_Backlog_Critical
expr: vllm_num_requests_waiting > 500
for: 2m
labels:
severity: critical
component: inference
annotations:
summary: "请求积压严重"
description: "模型 {{ $labels.model }} 等待中的请求超过 500 个，可能需要扩容"
gpu-health-alerts.yml: |
groups:
- name: gpu_health_alerts
rules:
# GPU Xid 错误告警（NVIDIA 驱动错误）
- alert: GPU_Xid_Error
expr: DCGM_FI_DEV_XID_ERRORS_COUNT > 0
for: 0s
labels:
severity: critical
component: gpu
annotations:
summary: "GPU Xid 错误"
description: "GPU {{ $labels.gpu }} 发生 Xid 错误，可能需要重启"
# ECC 错误告警
- alert: GPU_ECC_Error
expr: DCGM_FI_DEV_ECC_SBE_VOL_TOTAL + DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 10
for: 5m
labels:
severity: warning
component: gpu
annotations:
summary: "GPU ECC 错误"
description: "GPU {{ $labels.gpu }} ECC 错误数过多"
# NVLink 状态告警
- alert: NVLink_Down
expr: DCGM_FI_DEV_NVLINK_STATUS == 0
for: 1m
labels:
severity: warning
component: gpu
annotations:
summary: "NVLink 连接断开"
description: "GPU {{ $labels.gpu }} 的 NVLink 连接可能存在问题"8.2 AlertManager 配置
# alertmanager-config.yaml
# AlertManager 通知配置
apiVersion: v1
kind: ConfigMap
metadata:
name: alertmanager-config
namespace: monitoring
data:
alertmanager.yml: |
global:
resolve_timeout: 5m
route:
group_by: ['alertname', 'cluster', 'service']
group_wait: 30s
group_interval: 5m
repeat_interval: 4h
receiver: 'default-receiver'
routes:
# 严重告警直接通知
- match:
severity: critical
receiver: 'critical-receiver'
group_wait: 0s
repeat_interval: 1h
# GPU 告警单独分组
- match:
component: gpu
receiver: 'gpu-oncall'
group_by: ['gpu']
# LLM 推理告警
- match:
component: inference
receiver: 'llm-oncall'
group_by: ['model']
receivers:
- name: 'default-receiver'
webhook_configs:
- url: 'http://notification-service:8080/webhook'
send_resolved: true
- name: 'critical-receiver'
webhook_configs:
- url: 'http://notification-service:8080/webhook/critical'
send_resolved: true
pagerduty_configs:
- service_key: '<PAGERDUTY_SERVICE_KEY>'
severity: critical
- name: 'gpu-oncall'
webhook_configs:
- url: 'http://notification-service:8080/webhook/gpu'
- name: 'llm-oncall'
webhook_configs:
- url: 'http://notification-service:8080/webhook/llm'九、总结
本文介绍了 AI 平台可观测性的完整方案：
核心要点回顾：
AI 平台 vs 传统可观测性：AI 平台有其独特的指标体系（TTFT、TPOT、KV Cache），传统的 APM 工具无法直接覆盖这些需求。
三大支柱具体化：
指标层：GPU 利用率/显存、推理延迟、Token 吞吐、KV Cache 利用率
日志层：LLM 推理日志（Prompt/Completion/Token 统计）
链路层：从用户请求到模型推理的完整调用链
监控架构：Prometheus + Loki + Jaeger + Grafana 是当前的主流组合，分别负责指标、日志、链路和可视化。
GPU 监控：DCGM Exporter 是采集 NVIDIA GPU 指标的标准方案，暴露 DCGM_FI_* 系列指标。
自定义 LLM 指标：通过 Prometheus Client SDK 封装 LLM 特有指标（TTFT、TPOT、Token 吞吐等）。
Grafana Dashboard：预置 LLM 推理监控和 GPU 监控 Dashboard，开箱即用。
告警规则：分层告警设计（warning → critical），按组件分组通知，确保告警精准触达。
在实际落地时，建议先建立基础监控（GPU 利用率 + 推理延迟），再逐步扩展到完整指标体系和链路追踪。可观测性建设的优先级应该是：能看到问题 → 能定位问题 → 能预防问题。
相关技术栈：Prometheus、Grafana、DCGM Exporter、vLLM、Loki、Jaeger、Grafana
评论区