The Hidden Cost: GPU Electricity Expense Per Training Run

GPU Power Draw: The Numbers

Every GPU has a Thermal Design Power (TDP) — the maximum sustained power draw. But real power consumption depends on workload intensity, memory access patterns, and thermal conditions. Here are the actual TDP ratings for popular AI GPUs:

Key insight: an idle GPU still draws 15% of TDP. An idle H100 consumes ~105W — costing $0.013/hr in electricity even when doing nothing. Across an 8-GPU cluster sitting idle for a month, that's $73 in electricity for zero compute.

Electricity Cost Per Training Run

We modeled energy consumption for common ML workloads using real TDP data and typical utilization profiles. All costs use the US average electricity rate of $0.12/kWh (European rates at $0.25/kWh shown for comparison).

The Idle Cost Problem

The last row in the table above tells the real story. Industry research consistently shows average GPU utilization of just 5%. An 8-GPU H100 cluster sitting idle for a month wastes over $73 in electricity alone — and over $23,000/month in cloud compute costs for GPUs doing essentially nothing.

The electricity cost of idling is small compared to the cloud rental cost, but it's a useful signal: if you can measure power draw, you can measure utilization, and if you can measure utilization, you can eliminate waste.

The Formula

GPU energy cost per hour follows a simple formula:

Electricity vs. Cloud Compute: The Ratio

For cloud deployments, electricity is already baked into the instance price — you pay it indirectly. But for on-premise or colocation deployments, electricity is a direct, measurable cost:

• H100 SXM: Electricity is ~2% of total cloud cost, ~20% of on-prem TCO
• RTX 4090: Electricity is ~6% of total cloud cost (lower cloud price, high TDP)
• T4: Electricity is ~2% of total cloud cost (low TDP, low price)

For cloud users, the bigger win is eliminating idle compute time. For on-prem teams, both idle compute and direct electricity savings matter.

Carbon Footprint

Every kWh of electricity produces CO₂ emissions depending on the local grid mix. Using the US average of 385g CO₂/kWh:

• Fine-tuning Llama 3 8B: ~1 kg CO₂ (equivalent to driving 2.5 miles)
• Pre-training a 7B model: ~1,700 kg CO₂ (equivalent to a cross-country flight)
• Pre-training a 70B model: ~100,000+ kg CO₂ (equivalent to 15 cars for a year)

The EU AI Act (Article 13) requires GPAI model providers to report energy consumption during training. NemulAI provides automated energy metering and compliance-ready reports. Read our EU AI Act compliance guide →

How to Measure Real Energy Costs

The estimates above use TDP as a proxy, but real power draw varies with workload. The only way to know your actual energy cost is to measure it:

Try the Calculator

Model your own GPU electricity costs with our free GPU Cost Calculator. Includes compute costs, electricity, and CO₂ footprint across 9 providers.