I care about the environment, and I build AI systems — so the energy cost of what we ship is something I think about a lot. The good news is that the greenest choices are almost always the cheapest ones too. Efficiency is where sustainability and good engineering happen to point in the same direction.
Green AI is the practice of building systems that hit the quality bar you need while consuming as little compute, energy, and money as possible. Not by cutting corners — by not reaching for the biggest, most expensive tool out of habit.
Why AI consumes so much energy
Large models are large because they have billions of parameters, and every one of those parameters does arithmetic on every request. That takes powerful hardware, which draws real electricity and generates real heat that data centres then spend more energy cooling. The bigger the model and the more often you call it, the bigger the footprint.
Training vs inference: where the cost really lives
Training a large model is enormously expensive — but it usually happens once, far away, done by someone else. For most teams building applications, the footprint you actually control is inference: the millions of small calls your product makes every single day, forever. That is the number that scales with your success, and the one worth optimising.
The mindset shift
Stop asking 'which is the most powerful model?' and start asking 'what is the smallest, cheapest thing that clears my quality bar?' The answer is often far lighter than the default you would have reached for.
Right-sizing your model
A frontier model on every request is like taking a freight truck to buy groceries. Many real tasks — classification, extraction, routing, simple Q&A — are handled beautifully by a small model at a fraction of the cost and energy. Use the big model where its reasoning genuinely earns its keep, and a smaller one everywhere else.
Always use the largest model
- Highest cost and energy per call
- Overkill for simple tasks
- Slower responses
- Footprint scales with traffic
Match model to task
- Small models for routine work
- Large models only where needed
- Faster, cheaper, greener
- Footprint stays under control
The efficiency toolkit
There is a well-established set of techniques for getting the same quality for less compute. You do not need all of them — but knowing the toolbox lets you pick the right lever.
- Quantization: run a model at lower numerical precision (for example 8-bit or 4-bit) to cut memory and energy with little quality loss.
- Parameter-efficient fine-tuning (LoRA): adapt a model by training a tiny set of extra weights instead of the whole thing — far cheaper than full fine-tuning.
- Distillation: train a small model to mimic a large one for your specific task.
- Caching: store answers to repeated or similar questions so you never pay to compute them twice.
- Batching: group requests together to use the hardware more efficiently.
RAG instead of unnecessary fine-tuning
One of the most common wasteful patterns I see is fine-tuning a model to teach it facts. If the goal is to give the model knowledge, retrieval (RAG) is usually the greener and more flexible answer — you keep the knowledge in a searchable store and fetch it at query time, instead of burning compute to bake it into weights that go stale the moment the facts change. Fine-tune to change behaviour or style; retrieve to supply knowledge.
A simple rule of thumb
Need the model to know something? Reach for RAG. Need the model to behave differently? Consider fine-tuning — ideally the parameter-efficient kind.
Optimising the inference pipeline
The single biggest win is often architectural: not sending every request to a model at all. Answer repeats from cache, handle the obvious cases with simple logic, route to a small model next, and only escalate to a large model when nothing lighter will do. Most requests never need to reach the expensive end of that chain.
Cache
Repeats cost nothing.
Rules
Simple logic, instant.
Small model
Handles most requests.
Large model
Only when it earns it.
Measuring what matters
You cannot improve what you do not measure. Track cost per request, latency, and — where you can — energy or the share of traffic that reaches your largest model. These numbers turn sustainability from a vague good intention into an engineering target you can actually move.
Chasing the last 1%
- Huge compute for tiny gains
- Users rarely notice the difference
- Cost and carbon climb fast
Good enough, sustainably
- Hit the quality bar that matters
- Spend compute where users feel it
- Keep headroom in budget and footprint
“Efficiency is not a compromise on quality. It is a form of respect — for your budget, for the team that maintains the system, and for the resources it quietly consumes every day.”
What I recommend to teams
- Default to the smallest model that clears your quality bar; escalate only when needed.
- Cache aggressively — it is the cheapest performance and sustainability win there is.
- Prefer RAG over fine-tuning for supplying knowledge.
- Use quantization and LoRA to shrink the compute you do need.
- Measure cost, latency, and energy so sustainability becomes a number you can improve.
Green AI rarely looks dramatic. It looks like a cache, a smaller model, a sensible fallback, and a habit of asking whether you really need the biggest tool. But those unglamorous choices are what keep a system affordable, fast, and light on the planet a year after launch — which is the only kind of success worth building for.