Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX
Quick Answer
This study benchmarks confidential GPU inference on NVIDIA H100 under Intel TDX, revealing a 21.8% and 27.8% increase in time to first token for Mistral-7B and Qwen3-30B-A3B, respectively, while global token throughput decreases by 17.7% and 21.1%.
Quick Take
The findings indicate that while confidential execution maintains usable throughput, it incurs a performance penalty that must be considered in capacity planning.
Key Points
- Confidential mode increases average time to first token by 21.8% for Mistral-7B.
- Qwen3-30B-A3B shows a 27.8% increase in time to first token under confidential mode.
- Global token throughput drops by 17.7% for Mistral-7B and 21.1% for Qwen3-30B-A3B.
- Throughput gaps in closed-loop concurrency experiments range from 11.5% to 20.2%.
- Larger models reach saturation knee earlier in confidential execution mode.
DeepSignal Analysis
What happened
A benchmark study evaluated the performance of confidential GPU inference on the NVIDIA H100 using Intel TDX. The study found that enabling confidential execution increased the time to first token for two language models while decreasing global token throughput.
Key evidence
- The study reported a 21.8% increase in time to first token for the Mistral-7B model and a 27.8% increase for the Qwen3-30B-A3B model under confidential execution.
- Global token throughput decreased by 17.7% for Mistral-7B and 21.1% for Qwen3-30B-A3B when using confidential mode.
- In closed-loop concurrency tests, throughput gaps remained between 11.5% and 20.2%, with larger models reaching saturation earlier in confidential mode.
Why it matters
The findings highlight the trade-offs involved in implementing confidential computing for AI workloads. While it can maintain usable throughput, the performance penalties observed necessitate careful capacity planning, especially for larger models that exhibit earlier saturation.
What to watch
Paper Resources
Source Excerpt
Confidential computing is becoming a practical deployment requirement for AI inference workloads that process sensitive inputs or protect proprietary model assets. However, the performance cost of enabling confidential execution for GPU-accelerated serving remains workload dependent and operationally important. This paper presents a benchmark study comparing standard non-confidential execution with confidential computing mode on a single NVIDIA H100 80GB GPU hosted in an Int
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →RAIL Guard: Closing the Evaluation-to-Remediation Gap in Responsible AI for Agents
RAIL Guard introduces a closed-loop AI pipeline for large language models (LLMs) that evaluates outputs across eight dimensions and iteratively remediates failures, achieving 96.9% convergence compared to 49.1% for traditional block-and-retry methods. The system reduces unsafe agent executions by 33% without impacting task completion and is available as open-source SDKs.
