One post tagged with "AI applications using async"

Async processing decouples request acceptance from long-running AI work so your API tier stays responsive while model calls, agent tool chains, and multi-step workflows complete in the background. The role of async processing in AI apps is not a nice architectural flourish — it is the difference between a system that survives production and one that times out, drops jobs, and frustrates users. Here are the four concrete things it delivers:

1. Responsiveness — the API returns a job ID immediately; the client never blocks on long LLM calls.
2. Horizontal scalability — independent worker processes can be added without touching the API tier, preventing callback-stack explosion under load.
3. Resilience — jobs survive worker crashes, deploys, and transient API failures because state lives in a durable store, not in process memory.
4. Operational control — you can throttle, prioritize, replay, and inspect jobs independently of the request path.

Tools like Redis, Celery, RabbitMQ, Apache Kafka, AWS Step Functions, and Azure Durable Functions each address a slice of this problem. Mlflow ties the observability layer together, giving you end-to-end tracing across agentic reasoning steps and job lifecycle events.

Table of Contents​

• Why AI workloads are asynchronous by nature
• Short-lived async vs. durable long-running jobs
• Why holding HTTP streams open is an anti-pattern
• What breaks when you treat the cloud as one box
• Core building blocks: queues, workers, orchestrators, and storage
• How to build an async AI workflow step by step
• Coordinating agents, state management, and checkpointing
• Retries, exponential backoff, dead-letter queues, and rate-limiting
• Observability and lifecycle management: where Mlflow fits async agentic AI
• When should you choose async over synchronous?
• Common patterns and anti-patterns: a quick reference
• Key Takeaways
• Real-world pitfalls we've seen
• Mlflow brings production-grade visibility to your async AI pipeline
• Useful sources and further reading

Why AI workloads are asynchronous by nature​

Synchronous request/response works when the server can answer in milliseconds. LLM inference rarely qualifies. A single GPT-4-class generation can take anywhere from a few seconds to several minutes depending on context length, sampling parameters, and provider load. Image synthesis, video transcoding, large-file ingestion, and multi-step agent tool calls compound that unpredictability further.

The deeper problem is coupling compute lifetime to a single TCP connection. When a client disconnects mid-generation, or a load balancer enforces a 30-second timeout, or a rolling deploy restarts the pod, any in-flight work tied to that HTTP request is simply gone. There is no recovery path because the state was never written anywhere. Persisting the prompt and job metadata to durable storage before invoking the model is the architectural fix: any node can pick up and resume the job regardless of what happened to the original request.

Statistic callout: Polling intervals of a few seconds are commonly used for AI job status updates — short enough to feel responsive, long enough to avoid hammering your infrastructure. Polling too frequently overloads servers; polling too slowly feels broken to users.

Multi-step agentic flows add another dimension. An agent that calls a search tool, waits for a human approval, invokes a code interpreter, and then synthesizes a final answer may span minutes or hours across multiple external services. No single HTTP request can hold that lifetime. The decoupling of task submission from execution is what makes these flows survivable and independently scalable.

Short-lived async vs. durable long-running jobs​

Not every async pattern is the same, and conflating them is one of the most common architectural mistakes we see. There are two fundamentally different models.

Short-lived async uses await, coroutines, and streaming responses. Python's asyncio event loop and JavaScript's Promise/async-await both fall here. The work is bounded by the lifetime of the process and the connection. If the process dies, the work dies.

Durable long-running async uses an external job queue, a persistent worker, and checkpointed state. The work survives process restarts because its state lives outside the process. Durable workflow runtimes like AWS Step Functions and Azure Durable Functions journal each step, enabling pause, resume, configurable retries with backoff, and long pauses awaiting human input or external events.

A short TypeScript example using an await-based call:

// Short-lived: fine for a quick completion, fragile for long jobs
const result = await openai.chat.completions.create({
  model: "gpt-4o",
  messages: [{ role: "user", content: prompt }],
});

return result.choices[0].message.content;

The Python equivalent using Celery to enqueue a durable job:

# Durable: persist first, then enqueue
job = Job(id=uuid4(), prompt=prompt, status="pending")
db.session.add(job)
db.session.commit()          # state is safe before any model call
run_llm_task.delay(job.id)   # Celery enqueues; worker picks up independently
return {"job_id": str(job.id)}, 202

The decision to escalate from short-lived to durable comes down to a few signals: expected runtime beyond a short threshold, the need for checkpoints, human-in-the-loop pauses, or any requirement that the job survive a deploy. If any of those apply, reach for a queue.

Why holding HTTP streams open is an anti-pattern​

Streaming tokens over a long-lived HTTP connection feels elegant in a demo. In production, it breaks in ways that are hard to debug and harder to recover from.

• Proxy and gateway timeouts — most reverse proxies (nginx, AWS ALB, Cloudflare) enforce idle or total connection timeouts that will silently kill a stream mid-generation.
• Client disconnects — a mobile user switching networks, a browser tab closing, or a flaky Wi-Fi connection drops the stream. The model keeps running on the server, burning tokens, with no way to deliver the result.
• Rolling deploys — a Kubernetes rolling update will terminate pods mid-stream. Any in-flight generation is lost.
• No recovery path — because the result was never persisted, there is nothing to replay or resume.

Safer alternatives for each failure mode:

• Persist-first + job ID — write the prompt to durable storage, return a 202 Accepted with a job ID, and let the client poll or subscribe. This is the production pattern for long-running AI tasks.
• Reactive DB subscriptions (Convex-style) — the worker writes partial outputs to a reactive datastore; the client subscribes to that record and receives updates as the worker checkpoints. No HTTP stream required.
• SSE or WebSocket for near-real-time UI — use Server-Sent Events or WebSockets to push incremental updates from the job store to the client. The transport is separate from the model execution, so a dropped connection does not kill the job.
• Email or webhook delivery for very long jobs — for jobs that run for minutes or hours, notify the user asynchronously via email, Slack webhook, or push notification when the result is ready.

The pattern to internalize: the HTTP request path accepts work and returns a handle. It never holds the work.

What breaks when you treat the cloud as one box​

Engineers new to distributed AI systems often make the same set of mistakes. Here are the failure modes we see most often, and the mitigations that actually work.

Worker crash mid-job. A worker process dies while running a 3-minute LLM chain. If the job state was only in memory, it is gone. Mitigation: write job status and partial outputs to a durable store (Postgres, Redis with persistence, or an object store) at every meaningful checkpoint.

HTTP timeout. The API gateway kills the connection after 30 seconds. The model is still running on the backend, but the client has no job ID to poll. Mitigation: accept the request, persist the job record, enqueue the work, and return a job ID immediately — before any model call starts.

Deployment restart. A rolling deploy terminates all running workers. Any in-process work using asyncio.create_task() or FastAPI BackgroundTasks is silently dropped. Mitigation: use an external queue (Celery + Redis, RabbitMQ, or Kafka) so the job survives the restart and a new worker picks it up.

Transient external API failures. The LLM provider returns a 429 or 503. Without retry logic, the job fails permanently. Mitigation: configure exponential backoff with jitter on the worker, and route exhausted jobs to a dead-letter queue (DLQ) for inspection.

Duplicate webhooks. A webhook fires twice because the upstream provider retried. Without idempotency checks, you process the same job twice. Mitigation: use the webhook's event ID as the job's idempotency key; check for an existing job record before creating a new one.

Pro Tip: Never rely on sticky sessions or local process memory for job state. If your system cannot answer "where is this job's state if this pod disappears right now?" with a specific external store, the architecture has a gap.

Core building blocks: queues, workers, orchestrators, and storage​

Every production async AI system is assembled from the same set of components. Understanding the trade-offs between them is what separates a system that scales from one that collapses under load.

Producers are the API endpoints or event sources that create jobs. They validate the request, persist the job record, and enqueue a message. They never run the model.

Brokers transport messages from producers to workers. Redis is the simplest option: low operational overhead, fast, and adequate for most AI workloads. RabbitMQ adds routing, priority queues, and more sophisticated DLQ behavior. Apache Kafka is the right choice when you need durable, replayable event streams at high throughput — particularly useful for audit trails and event sourcing in agentic pipelines.

Worker consumers pull jobs from the broker and execute them. Celery is the most widely used Python worker framework; it integrates natively with Redis and RabbitMQ, supports task routing by queue name, and provides built-in retry and backoff configuration.

Orchestrators coordinate multi-step workflows. AWS Step Functions and Azure Durable Functions both journal each step, support parallel branches, and handle long pauses for human approval or external events. Durable functions enable pause/resume and configurable retries with backoff, making them the right tool for complex agentic flows rather than single-task workers.

Durable storage holds job records, partial outputs, and checkpoints. Postgres is the default for structured job metadata. Object stores (S3, Azure Blob Storage) handle large intermediate artifacts like generated images or transcripts.

Convex's reactive database model is worth noting here: rather than polling a REST endpoint, clients subscribe to a document in the reactive store, and the worker writes partial outputs directly to that document. The UI updates automatically without a separate push channel. It is a clean pattern for streaming-style UX without the fragility of a long HTTP stream.

Mlflow sits at the observability layer across all of these components, providing LLM and agent tracing, prompt registry, and automated evaluation hooks that tie job lifecycle events to model behavior.

How to build an async AI workflow step by step​

This is the recipe we recommend for any AI job expected to run longer than 10 seconds.

7. Orchestrator resumes multi-step flows — For agentic workflows with branching or human approvals, delegate to AWS Step Functions or Azure Durable Functions. Pass the job_id and trace_id as execution context.

TypeScript example (Convex-style persist-first):

// API handler
export const submitJob = mutation(
  async ({ db }, { prompt }: { prompt: string }) => {
    const jobId = await db.insert("jobs", {
      prompt,
      status: "pending",
      createdAt: Date.now(),
      output: null,
    });
    await scheduler.runAfter(0, internal.workers.runLlmJob, { jobId });
    return { jobId };
  },
);

// Worker
export const runLlmJob = internalAction(async ({ runMutation }, { jobId }) => {
  await runMutation(internal.jobs.updateStatus, { jobId, status: "running" });
  const result = await callLlm(/* ... */);
  await runMutation(internal.jobs.finalize, { jobId, output: result });
});

Python example (FastAPI + Celery + Redis):

# FastAPI route
@app.post("/jobs", status_code=202)
async def create_job(request: JobRequest, db: Session = Depends(get_db)):
    job = Job(id=uuid4(), prompt=request.prompt, status="pending")
    db.add(job)
    db.commit()
    run_llm_job.delay(str(job.id))   # Celery task; Redis broker
    return {"job_id": str(job.id)}

# Celery worker
@celery_app.task(bind=True, max_retries=5)
def run_llm_job(self, job_id: str):
    db = SessionLocal()
    job = db.query(Job).get(job_id)
    job.status = "running"
    db.commit()
    try:
        result = call_llm(job.prompt)
        job.output = result
        job.status = "complete"
        db.commit()
    except TransientError as exc:
        raise self.retry(exc=exc, countdown=2 ** self.request.retries)

Pro Tip: Separate your worker pools by latency class. Fast, latency-sensitive completions (under 5 seconds) should run on a dedicated queue with high concurrency. Batch or long-running jobs belong on a separate queue with lower concurrency and higher memory limits. Mixing them on the same pool lets slow jobs starve fast ones.

For AI service load management at scale, batching long jobs into manifests rather than issuing thousands of individual calls can dramatically reduce both runtime and provider cost.

Coordinating agents, state management, and checkpointing​

Multi-agent systems introduce a new class of state management problems. When three agents are collaborating on a task, each writing partial outputs and reading from shared context, you need a clear answer to: "What is the authoritative state of this job at any point in time?"

The answer is a single durable job record that every agent reads from and writes to. Store the following in that record for every job:

• job_id and trace_id (for observability correlation)
• status (pending, running, paused, complete, failed)
• attempts (retry count)
• last_checkpoint (timestamp and step name of the last successful write)
• partial_outputs (a JSON array or object-store reference for intermediate results)
• tool_results (keyed by tool call ID for safe replay)
• session_id (for multi-turn agent conversations)

Checkpointing at the section level means writing a partial output to the store after each discrete step, not just at the end. If the worker crashes after step 3 of a 7-step chain, the next worker picks up from step 3 rather than step 1. Durable workflow runtimes like AWS Step Functions handle this journaling automatically; with Celery you implement it explicitly in the task body.

Idempotency is non-negotiable for safe replay. Every worker action that has external side effects (sending an email, calling a paid API, writing to a third-party system) must check whether it has already been performed for this job_id before executing. Use the job_id plus the step name as a composite idempotency key.

Asynchronous authorization patterns are particularly useful here: when an agent needs human consent for a sensitive action, it pauses the workflow, records the pending approval in the job record, and resumes only after the approval event arrives. The background processing continues for non-sensitive steps; only the gated action waits.

Retries, exponential backoff, dead-letter queues, and rate-limiting​

Production resilience for async AI systems comes from three interlocking mechanisms: retry policies, DLQs, and rate-limiting on external APIs.

Distinguishing failure types:

• Transient failures (429 rate limit, 503 service unavailable, network timeout) are safe to retry. The job should back off and try again.
• Permanent failures (400 bad request, 401 unauthorized, malformed prompt) will never succeed on retry. Route them to the DLQ immediately.

Exponential backoff recipe:

• Base delay: 2 seconds
• Multiplier: 2x per attempt
• Maximum interval: 120 seconds
• Add jitter (±20%) to prevent thundering herd when many workers retry simultaneously
• Maximum attempts: 5 before routing to DLQ

countdown = min(2 ** self.request.retries + random.uniform(-0.5, 0.5), 120)
raise self.retry(exc=exc, countdown=countdown)

Dead-letter queues hold jobs that have exhausted their retry budget. DLQ depth is a leading indicator of systemic issues — a growing DLQ means something upstream is broken, not just noisy. Alert on DLQ depth, not just on individual job failures.

Operational checklist for resilience:

• Set visibility timeouts longer than your longest expected job runtime to prevent duplicate processing.
• Monitor queue depth, DLQ depth, and worker saturation as your three primary async health signals.
• Implement a heartbeat mechanism for very long jobs: the worker updates last_heartbeat every 30 seconds; a separate monitor marks jobs as stalled if the heartbeat is absent for more than 2x the expected interval.
• Rate-limit outbound LLM API calls at the worker level using a token bucket or leaky bucket algorithm to stay within provider rate limits without failing jobs.
• For AI load balancing techniques across multiple LLM providers, route overflow traffic to a secondary provider rather than failing the job.

Observability and lifecycle management: where Mlflow fits async agentic AI​

Async systems are harder to debug than synchronous ones because a single logical operation spans multiple processes, queues, and time windows. Without end-to-end tracing, a failed job is a black box: you know it failed, but not which step, which tool call, or which model response caused the failure.

Mlflow addresses this directly. Its LLM and agent tracing captures the full reasoning trace of an agentic workflow, including individual tool calls, sub-agent invocations, token counts, and step durations. By attaching the trace_id to every job record at creation time, you can correlate a DLQ event back to the exact model call that caused it.

Integration recipe for async workflows:

1. Generate a trace_id when the job record is created and store it in the job document.
2. Pass the trace_id as a span context when the worker starts the Mlflow trace.
3. Instrument each step (prompt construction, model call, tool invocation, output parsing) as a child span within that trace.
4. On job completion or failure, emit the final span with status, retry count, and checkpoint metadata.
5. Use Mlflow's automated evaluation hooks to run LLM-as-a-Judge quality checks on completed outputs, flagging regressions before they reach users.

Mlflow's prompt registry gives you versioned, auditable prompt templates that workers load by name and version rather than hardcoding strings. When a prompt change causes a quality regression in async jobs, you can roll back to the previous version without a code deploy.

The AI Gateway handles cross-provider governance: workers call the Gateway rather than individual provider SDKs, and the Gateway enforces rate limits, logs every call, and routes overflow to secondary providers. This is particularly valuable in async systems where workers may be calling multiple providers in parallel.

Pro Tip: Capture these four telemetry signals for every async job: prompt hash (for deduplication and cache hit analysis), partial-output checkpoint timestamps (for step duration profiling), retry count (for provider health monitoring), and final token count (for cost attribution). These four fields answer 80% of production debugging questions.

Mlflow's AI observability layer ties all of this together, giving engineering teams a single pane of glass for agentic reasoning traces, job lifecycle events, and automated evaluation results.

When should you choose async over synchronous?​

Apply this checklist before committing to an async architecture for a given workload.

Choose async when:

• Expected runtime exceeds 10–15 seconds under normal conditions.
• The job must survive a process restart or deploy.
• You need checkpoints, partial output delivery, or human-in-the-loop pauses.
• The workload is I/O-bound (external model calls, database writes, network requests). Async is an optimization strategy best suited to I/O-bound workloads; the overhead of async state management pays off when I/O dominates.
• Cost sensitivity requires batching or rate-limiting that cannot be done in a single request.
• Client UX tolerates a polling or notification model (background processing, batch reports).

Choose synchronous when:

• Expected runtime is under 5 seconds and the client needs an immediate response.
• The operation is CPU-bound and compute-intensive in a way that async state management cannot help (consider dedicated compute workers instead).
• Simplicity and debuggability outweigh the resilience benefits for the specific use case.

For mixed workloads, the right answer is usually both: synchronous for short, latency-sensitive completions (autocomplete, quick chat turns), async for everything else (document analysis, multi-step agents, batch evaluation runs). Route by expected duration at the API boundary, not by job type.

A note on CPU-bound work: Python's GIL means that asyncio does not parallelize CPU-bound computation. For CPU-intensive preprocessing or postprocessing, use concurrent.futures.ProcessPoolExecutor or dedicated worker processes rather than coroutines.

Common patterns and anti-patterns: a quick reference​

Patterns that work:

• Job queues for survivability — use an external broker (Redis, RabbitMQ, Kafka) so jobs survive restarts. Never use in-process background tasks for work that must complete.
• Durable workflows for multi-step agents — use AWS Step Functions or Azure Durable Functions when workflows span multiple steps, external events, or human approvals.

Anti-patterns to avoid:

For testing async flows, run replay tests against your DLQ: take a failed job, fix the underlying issue, and replay it through the worker. If the worker is correctly idempotent, the replay produces the same output without side effects.

Key Takeaways​

Async processing in AI apps works because it separates job acceptance from execution, letting durable queues and workers handle long-running model calls while the API tier stays responsive and resilient.

Real-world pitfalls we've seen​

The most expensive async mistakes are not architectural — they are operational. Teams spend weeks building a queue-backed system and then skip the monitoring. DLQs fill silently. Workers saturate. Users see failures with no alert firing.

The second most common mistake is treating await as a durability primitive. We have seen production systems where FastAPI BackgroundTasks were used for 20-minute document processing jobs. The first rolling deploy wiped out every in-flight job. The fix was straightforward — Celery + Redis — but the rework cost two sprints that a correct initial design would have avoided.

Insufficient checkpointing is the third pattern. A worker that checkpoints only at the end of a 15-step agent chain will restart from step 1 on any failure. Checkpointing at every step costs a few extra database writes; not checkpointing costs the entire job on every failure.

Dos:

• Instrument every job with a trace_id from creation to completion.
• Alert on DLQ depth, queue latency, and worker saturation as primary health signals.
• Run replay tests against your DLQ regularly — they are the best integration test for your worker's idempotency.
• Separate worker pools by latency class from day one; retrofitting this later is painful.

Don'ts:

• Don't use in-process background tasks for work that must survive a deploy.
• Don't mix slow batch jobs and fast interactive jobs on the same worker pool.
• Don't skip idempotency on any step that has external side effects.
• Don't treat a quiet DLQ as a sign that everything is fine — check that messages are actually reaching it.

The teams that get async right treat their queue infrastructure with the same rigor they apply to their database: schema migrations, monitoring, capacity planning, and regular failure drills.

Mlflow brings production-grade visibility to your async AI pipeline​

Building a correct async architecture is half the work. Knowing what is happening inside it at runtime is the other half. Mlflow gives async AI teams production-grade observability that maps directly onto the patterns described here: deep agentic reasoning traces tied to job IDs, automated LLM-as-a-Judge evaluations on completed runs, a versioned prompt registry for reproducible worker behavior, and an AI Gateway that centralizes cross-provider governance so workers never call provider SDKs directly.

The integration is lightweight: generate a trace_id at job creation, pass it as span context in your worker, and instrument each step as a child span. Mlflow handles the rest, from capturing token counts and step durations to surfacing evaluation regressions before they reach users. If you are building or scaling an async AI pipeline and want end-to-end visibility from queue event to model output, start with Mlflow's LLM tracing quickstart and instrument your first worker in under an hour.

Useful sources and further reading​

• Async programming for AI apps (Convex) — covers the persist-first pattern, reactive DB subscriptions, and durable workflow concepts with concrete implementation guidance.
• How to Build an Async AI Pipeline That Runs for Hours Without Timing Out — FastAPI + Celery + Redis implementation walkthrough with clear separation of the API and worker tiers.
• Production-Ready LLM Apps: Batch Processing, Async Patterns and Scaling — practical guidance on queue separation by latency class and batching strategies for cost and throughput control.
• Async Processing Hides Latency (The New Stack) — systems-architecture perspective on decoupling task submission from execution and horizontal worker scaling.
• Asynchronous Processing in System Design (GeeksforGeeks) — broad conceptual overview of async patterns, benefits, and implementation strategies in distributed systems.
• A Conceptual Overview of asyncio (Python docs) — authoritative reference for Python's event loop model and the correct scope of asyncio for I/O-bound concurrency.
• Asynchronous Authorization for AI Agents (Auth0) — covers human-in-the-loop authorization patterns for agentic workflows that need to pause for user consent mid-execution.
• Apache Kafka Documentation — official reference for Kafka's log-based architecture, replication, and consumer group patterns relevant to high-throughput async AI pipelines.
• Mlflow AI Observability — production observability features for LLM and agent workloads, including tracing, evaluation, and the AI Gateway.

Recommended​

• One post tagged with "AI in app development" | MLflow
• One post tagged with "AI technology in apps" | MLflow
• One post tagged with "guide to AI-powered applications" | MLflow
• One post tagged with "benefits of AI in apps" | MLflow