Rust AI Engine Resilience
For async AI engines in Rust that need retry, caching, rate limiting, and streaming.
The recurring failure is a blocking, fragile engine that fails on transient errors,
wastes tokens on repeated prompts, and provides no streaming feedback.
When to Use
- Adding retry with exponential backoff to an AI engine
- Adding response caching to avoid duplicate API calls
- Adding rate limiting to avoid 429 errors
- Adding streaming support for token-by-token display
- Debugging AI engine resilience issues
- Implementing AI provider resilience in any Rust async codebase
Always-on rules
- Retry only transient errors. 429, 503, 529, and "rate limit" messages are worth
retrying. 400, 401, 403, 404 are not. Check the error message, not just the
status code. A 404 means the model doesn't exist — retrying won't help.
- Use
tokio::sync::Semaphore for concurrent request limiting. Acquire a permit
before each request, drop it after. The semaphore should be a field on the
engine struct, wrapped in Arc. Default to 4 concurrent requests.
- Use
lru::LruCache for response caching. Key by a hash of (prompt, model,
temperature, max_tokens, tier, history). Set a TTL (5 minutes is reasonable
for chat responses). Only cache non-streaming requests.
- Hash
Option<f32> via to_bits(). f32 doesn't implement Hash, so
Option<f32> doesn't either. Use req.temperature.map(|t| t.to_bits()).hash(&mut hasher).
- Hash enums via
format!("{:?}", value). If an enum doesn't derive Hash,
convert it to a string first.
- Stream bytes to String via
from_utf8_lossy. reqwest bytes streams yield
Bytes, not String. Convert with String::from_utf8_lossy(&bytes).to_string().
- Wrap Win32 API calls in
unsafe. InvalidateRect, SetWindowPos, etc. are
unsafe in Rust. unsafe { InvalidateRect(hwnd, ptr::null_mut(), 0); }
- Don't use
self in window procs. Get the state pointer from
GetWindowLongPtrW first. See win32-gui-rendering skill.
- Integration tests hang when external services aren't running. Check if the
service is running before running integration tests. Use
curl or a health check.
A hanging integration test is an environment issue, not a code issue.
- Build locks on Windows. If
cargo build fails with "failed to remove file",
kill the running process first: taskkill -f -im app.exe.
Patterns
Retry with exponential backoff
async fn retry<F, Fut, T>(&self, operation: F) -> Result<T>
where
F: Fn() -> Fut,
Fut: std::future::Future<Output = Result<T>>,
{
let mut attempts = 0;
let max_attempts = 3;
let mut delay = Duration::from_millis(500);
loop {
match operation().await {
Ok(result) => return Ok(result),
Err(e) if attempts < max_attempts && Self::is_retryable(&e) => {
attempts += 1;
tokio::time::sleep(delay).await;
delay *= 2;
}
Err(e) => return Err(e),
}
}
}
fn is_retryable(error: &Error) -> bool {
let text = error.to_string().to_ascii_lowercase();
text.contains("429") || text.contains("503") || text.contains("529")
|| text.contains("rate limit") || text.contains("too many requests")
}
Response caching
use lru::LruCache;
use std::num::NonZeroUsize;
// Field on engine:
response_cache: Arc<Mutex<LruCache<u64, (AiResponse, Instant)>>>,
// In constructor:
response_cache: Arc::new(Mutex::new(LruCache::new(
NonZeroUsize::new(128).unwrap(),
))),
// Check cache before request:
let cache_key = Self::cache_key(&req);
{
let mut cache = self.response_cache.lock().unwrap();
if let Some((response, timestamp)) = cache.get(&cache_key) {
if timestamp.elapsed() < Duration::from_secs(300) {
return Ok(response.clone());
}
}
}
// Store after success:
if let Ok(ref response) = result {
let mut cache = self.response_cache.lock().unwrap();
cache.put(cache_key, (response.clone(), Instant::now()));
}
Rate limiting
use tokio::sync::Semaphore;
// Field on engine:
request_semaphore: Arc<Semaphore>,
// In constructor:
request_semaphore: Arc::new(Semaphore::new(4)),
// Acquire before request:
let _permit = self
.request_semaphore
.acquire()
.await
.map_err(|e| Error::Network(e.to_string()))?;
Streaming
use futures::StreamExt;
pub async fn generate_stream(
&self,
req: AiRequest,
) -> Result<impl futures::Stream<Item = Result<String>>> {
// ... build request ...
let response = self.client.post(&url).json(&body).send().await?;
let byte_stream = response.bytes_stream();
let stream = byte_stream.map(|chunk| {
chunk
.map(|bytes| String::from_utf8_lossy(&bytes).to_string())
.map_err(|e| Error::Network(e.to_string()))
});
Ok(stream)
}
Cache key hashing
fn cache_key(req: &AiRequest) -> u64 {
use std::hash::{Hash, Hasher};
let mut hasher = std::collections::hash_map::DefaultHasher::new();
req.prompt.hash(&mut hasher);
req.force_model.hash(&mut hasher);
req.temperature.map(|t| t.to_bits()).hash(&mut hasher);
req.max_tokens.hash(&mut hasher);
req.force_tier.hash(&mut hasher);
if let Some(ref history) = req.history {
for msg in history {
format!("{:?}", msg.role).hash(&mut hasher);
msg.content.hash(&mut hasher);
}
}
hasher.finish()
}
Dependencies
reqwest = { version = "0.12", features = ["json", "stream"] }
lru = "0.12"
futures = "0.3"
Pitfalls
resolve_local_model() returns String, not ModelInfo. Wrap with
self.resolved_local(...) when the caller expects a ModelInfo.
Option<f32> doesn't implement Hash. Use .map(|t| t.to_bits()) before hashing.
- Enums without
Hash derive. Use format!("{:?}", value).hash(&mut hasher).
Bytes vs String in streams. reqwest yields Bytes, convert with
from_utf8_lossy.
InvalidateRect is unsafe. Wrap in unsafe { } blocks.
self doesn't exist in window procs. Get pointer from GetWindowLongPtrW.
- Integration tests hang when external services aren't running. Check if the
service is running before running integration tests.
- Build locks on Windows. Kill the running process before rebuilding.
label_w variable removed by patch but still referenced. When a patch removes
a variable declaration, check all references to that variable in the surrounding
scope. A removed declaration with live references is a compile error.
C_BORDER and other constants appear multiple times. Use replace_all=true
when patching a constant that appears in multiple locations. A non-unique match
causes the patch to fail.
1---2name: rust-ai-resilience3description: Add resilience patterns to Rust async AI engines.4version: 1.0.05author: Hermes Agent6license: MIT7metadata:8 source: https://skillmd.com/skills/loopyluci/rust-ai-resilience9 hermes:10 tags: [rust, async, ai, resilience, retry, caching, rate-limiting, streaming]11 related_skills: [software-development/systematic-debugging]12---1314# Rust AI Engine Resilience1516For async AI engines in Rust that need retry, caching, rate limiting, and streaming.17The recurring failure is a blocking, fragile engine that fails on transient errors,18wastes tokens on repeated prompts, and provides no streaming feedback.1920## When to Use2122- Adding retry with exponential backoff to an AI engine23- Adding response caching to avoid duplicate API calls24- Adding rate limiting to avoid 429 errors25- Adding streaming support for token-by-token display26- Debugging AI engine resilience issues27- Implementing AI provider resilience in any Rust async codebase2829## Always-on rules3031- **Retry only transient errors.** 429, 503, 529, and "rate limit" messages are worth32 retrying. 400, 401, 403, 404 are not. Check the error message, not just the33 status code. A 404 means the model doesn't exist — retrying won't help.34- **Use `tokio::sync::Semaphore` for concurrent request limiting.** Acquire a permit35 before each request, drop it after. The semaphore should be a field on the36 engine struct, wrapped in `Arc`. Default to 4 concurrent requests.37- **Use `lru::LruCache` for response caching.** Key by a hash of (prompt, model,38 temperature, max_tokens, tier, history). Set a TTL (5 minutes is reasonable39 for chat responses). Only cache non-streaming requests.40- **Hash `Option<f32>` via `to_bits()`.** `f32` doesn't implement `Hash`, so41 `Option<f32>` doesn't either. Use `req.temperature.map(|t| t.to_bits()).hash(&mut hasher)`.42- **Hash enums via `format!("{:?}", value)`.** If an enum doesn't derive `Hash`,43 convert it to a string first.44- **Stream bytes to String via `from_utf8_lossy`.** `reqwest` bytes streams yield45 `Bytes`, not `String`. Convert with `String::from_utf8_lossy(&bytes).to_string()`.46- **Wrap Win32 API calls in `unsafe`.** `InvalidateRect`, `SetWindowPos`, etc. are47 unsafe in Rust. `unsafe { InvalidateRect(hwnd, ptr::null_mut(), 0); }`48- **Don't use `self` in window procs.** Get the state pointer from49 `GetWindowLongPtrW` first. See win32-gui-rendering skill.50- **Integration tests hang when external services aren't running.** Check if the51 service is running before running integration tests. Use `curl` or a health check.52 A hanging integration test is an environment issue, not a code issue.53- **Build locks on Windows.** If `cargo build` fails with "failed to remove file",54 kill the running process first: `taskkill -f -im app.exe`.5556## Patterns5758### Retry with exponential backoff5960```rust61async fn retry<F, Fut, T>(&self, operation: F) -> Result<T>62where63 F: Fn() -> Fut,64 Fut: std::future::Future<Output = Result<T>>,65{66 let mut attempts = 0;67 let max_attempts = 3;68 let mut delay = Duration::from_millis(500);69 loop {70 match operation().await {71 Ok(result) => return Ok(result),72 Err(e) if attempts < max_attempts && Self::is_retryable(&e) => {73 attempts += 1;74 tokio::time::sleep(delay).await;75 delay *= 2;76 }77 Err(e) => return Err(e),78 }79 }80}8182fn is_retryable(error: &Error) -> bool {83 let text = error.to_string().to_ascii_lowercase();84 text.contains("429") || text.contains("503") || text.contains("529")85 || text.contains("rate limit") || text.contains("too many requests")86}87```8889### Response caching9091```rust92use lru::LruCache;93use std::num::NonZeroUsize;9495// Field on engine:96response_cache: Arc<Mutex<LruCache<u64, (AiResponse, Instant)>>>,9798// In constructor:99response_cache: Arc::new(Mutex::new(LruCache::new(100 NonZeroUsize::new(128).unwrap(),101))),102103// Check cache before request:104let cache_key = Self::cache_key(&req);105{106 let mut cache = self.response_cache.lock().unwrap();107 if let Some((response, timestamp)) = cache.get(&cache_key) {108 if timestamp.elapsed() < Duration::from_secs(300) {109 return Ok(response.clone());110 }111 }112}113114// Store after success:115if let Ok(ref response) = result {116 let mut cache = self.response_cache.lock().unwrap();117 cache.put(cache_key, (response.clone(), Instant::now()));118}119```120121### Rate limiting122123```rust124use tokio::sync::Semaphore;125126// Field on engine:127request_semaphore: Arc<Semaphore>,128129// In constructor:130request_semaphore: Arc::new(Semaphore::new(4)),131132// Acquire before request:133let _permit = self134 .request_semaphore135 .acquire()136 .await137 .map_err(|e| Error::Network(e.to_string()))?;138```139140### Streaming141142```rust143use futures::StreamExt;144145pub async fn generate_stream(146 &self,147 req: AiRequest,148) -> Result<impl futures::Stream<Item = Result<String>>> {149 // ... build request ...150 let response = self.client.post(&url).json(&body).send().await?;151 let byte_stream = response.bytes_stream();152 let stream = byte_stream.map(|chunk| {153 chunk154 .map(|bytes| String::from_utf8_lossy(&bytes).to_string())155 .map_err(|e| Error::Network(e.to_string()))156 });157 Ok(stream)158}159```160161### Cache key hashing162163```rust164fn cache_key(req: &AiRequest) -> u64 {165 use std::hash::{Hash, Hasher};166 let mut hasher = std::collections::hash_map::DefaultHasher::new();167 req.prompt.hash(&mut hasher);168 req.force_model.hash(&mut hasher);169 req.temperature.map(|t| t.to_bits()).hash(&mut hasher);170 req.max_tokens.hash(&mut hasher);171 req.force_tier.hash(&mut hasher);172 if let Some(ref history) = req.history {173 for msg in history {174 format!("{:?}", msg.role).hash(&mut hasher);175 msg.content.hash(&mut hasher);176 }177 }178 hasher.finish()179}180```181182## Dependencies183184```toml185reqwest = { version = "0.12", features = ["json", "stream"] }186lru = "0.12"187futures = "0.3"188```189190## Pitfalls191192- **`resolve_local_model()` returns `String`, not `ModelInfo`.** Wrap with193 `self.resolved_local(...)` when the caller expects a `ModelInfo`.194- **`Option<f32>` doesn't implement `Hash`.** Use `.map(|t| t.to_bits())` before hashing.195- **Enums without `Hash` derive.** Use `format!("{:?}", value).hash(&mut hasher)`.196- **`Bytes` vs `String` in streams.** `reqwest` yields `Bytes`, convert with197 `from_utf8_lossy`.198- **`InvalidateRect` is unsafe.** Wrap in `unsafe { }` blocks.199- **`self` doesn't exist in window procs.** Get pointer from `GetWindowLongPtrW`.200- **Integration tests hang when external services aren't running.** Check if the201 service is running before running integration tests.202- **Build locks on Windows.** Kill the running process before rebuilding.203- **`label_w` variable removed by patch but still referenced.** When a patch removes204 a variable declaration, check all references to that variable in the surrounding205 scope. A removed declaration with live references is a compile error.206- **`C_BORDER` and other constants appear multiple times.** Use `replace_all=true`207 when patching a constant that appears in multiple locations. A non-unique match208 causes the patch to fail.