Introduction

In Parts 1 and 2, we built the foundation: the Upload Layer that securely ingests files at scale, while the Processing Layer uses SQS queues and worker services to decouple uploads from processing. These layers handle the “plumbing”, moving files reliably from users to processing pipelines.

Now we reach the heart of the system which is the AI Service that transforms raw files into actionable insights. This is where architectural complexity peaks. Unlike the Upload and Processing layers, which follow well-established patterns (pre-signed URLs, message queues, workers), the AI Service must navigate murky territory: multimodal inputs, unpredictable AI outputs, rate limits, and cost optimization.

This part explores how to architect an AI service that’s production-ready and not just a wrapper around OpenAI’s API, but a resilient, cost-effective intelligence layer that handles the chaos of real-world AI integration.

High-Level Architecture

The AI Service follows an agent-based architecture where the Worker Service invokes an orchestrated pipeline of seven specialized components before reaching the core intelligence layer. This architecture wraps Google’s Agent Development Kit (ADK) with production-grade infrastructure for caching, validation, observability, and memory management.

Components:

1. API Entry Point - Orchestrates request lifecycle and coordinates all services

2. Prompt Management - Version-controlled, A/B-tested prompts from database

3. Guardrails - Validates inputs for injection attacks and outputs for hallucinations

4. Google ADK - Core intelligence layer with integrated context caching, reasoning, and tool orchestration

5. Tool & Artifact Service - File operations (read_file, extract_pdf_text, analyze_image)

6. Memory & Observability - Context persistence and cost/latency tracking

Component Deep Dive

This section examines each component in the agent-based architecture, detailing their specific responsibilities, implementation patterns, and interaction flows. Each component includes a focused sequence diagram showing its role in the processing pipeline.

1. API Entry Point

Central orchestration hub that coordinates all downstream services and manages the request lifecycle

The API Entry Point serves as the single interface between the Worker Service and the entire AI intelligence pipeline. It orchestrates the flow through prompt management, guardrails, and the agent framework while handling errors gracefully and tracking request context throughout the pipeline.

Key Functions:

• Request Coordination: Routes requests through the pipeline in the correct order (prompts → guardrails → agent → guardrails → response)

• Context Enrichment: Transforms file paths into file URLs and adds extra context (metadata, processing options) before validation

• Error Handling: Catches and categorizes errors from downstream services, decides whether to retry or fail fast

• Context Management: Maintains request context (file_id, user_id, trace_id) as it flows through components

• Circuit Breaking: Protects downstream services by failing fast when they’re unhealthy

2. Guardrails

Two-way validation layer that checks inputs for prompt injection and malware, and validates outputs for hallucinations and schema compliance

Guardrails act as the security and quality gates for the AI pipeline. Input guardrails prevent malicious content from reaching expensive AI services and protect against prompt injection attacks. Output guardrails validate AI responses for correctness, hallucinations, and schema compliance before returning results to users.

Key Functions:

• Input Validation: Detects prompt injection, malware, PII leakage attempts

• Output Validation: Checks for hallucinations, schema compliance, confidence thresholds

• Policy Enforcement: Blocks requests that violate safety or compliance policies

• Fail-Fast Pattern: Rejects invalid requests before consuming AI credits

3. Google Agent Framework (ADK)

Responsibility: The core intelligence orchestrator that reasons about tasks, invokes tools, manages context caching, and coordinates with LLMs through multi-step workflows

The Google Agent Development Kit (ADK) is the heart of the intelligence layer. Unlike simple API wrappers, ADK treats AI as an agentic system that can plan multi-step workflows, invoke tools dynamically, and maintain context across reasoning steps. Critically, ADK now includes integrated context caching that automatically caches prompts, system instructions, and file artifacts-eliminating 30-40% of redundant token processing costs.

Key Capabilities:

• Context Caching: Automatically caches static content (system prompts, file artifacts) for reuse across requests, reducing token costs by 30-40%

• Multi-Step Reasoning: Breaks complex tasks into subtasks (”extract text from PDF, then analyze sentiment”)

• Tool Orchestration: Dynamically invokes file readers, PDF extractors, image analyzers based on file type

• Context Management: Maintains conversation history and intermediate results across tool calls

• Error Recovery: Retries failed tool invocations with adjusted parameters

How Context Caching Works:

1. First Request: ADK sends full prompt + system instructions + file artifact to LLM. Google caches this context.

2. Subsequent Requests: For identical system prompts/files, ADK references the cached context, sending only the new user query.

3. Cost Savings: Cached tokens cost 10x less than fresh tokens ($0.01 vs $0.10 per 1M tokens).

4. Automatic Management: ADK handles cache invalidation when prompts or tools change.

Tool & Artifact Service :

Provide file reading and processing capabilities that the agent can invoke (read_file, extract_pdf_text, analyze_image)

The Tool & Artifact Service is the agent’s “hands”-it performs the actual file manipulation that the agent needs but cannot do directly. Tools are registered with the agent framework and invoked dynamically based on the agent’s reasoning. Each tool is designed to be idempotent and atomic.

Memory :

Memory maintains context across sessions using vector databases and Redis, while Observability tracks costs, latency, and tool usage. ADK has the support for in memory MemoryService as well as external Memory service such as Vertext AI and postgres.

You can literally build your own memory layer.

Observability :

For observability and evals, one can decorate the google adk calls with Opik.

🚀 Implementation

The architecture in this series is now available as an open source framework. It implements all three layers-secure uploads, async processing, and AI integration with guardrails-ready to deploy. Check it out on Github.

Built something with it? Share your story in comments -I’d love to hear about it.

Subscribe now

Introduction

In Part 1, we built the foundation for a secure, scalable upload layer that handles 10,000 concurrent file uploads with pre-signed URLs, encrypted storage, and proper rate limiting. Files are safely sitting in S3, metadata is tracked in PostgreSQL, and users have received their upload confirmations. 👏

But now comes the hard part : turning those raw files into actionable AI insights.

Uploading a file is synchronous and straightforward but processing the files and fetching insights through AI is not. It’s variable (5-30 seconds), expensive, rate-limited, and dependent on external services that can fail. You can’t block your API server waiting for AI responses as your connection pool would exhaust under load.

This is where Layer 2: Processing Layer comes in. This layer transforms the system into an intelligent processing pipeline that queues thousands of jobs, scales workers independently, retries failed requests, and handles failures gracefully.

In this part, we’ll architect the asynchronous processing pipeline. We’ll explore message queues, worker deployment strategies, AI service integration, and error handling.

By the end, you’ll have a complete blueprint for Layer 2, the asynchronous processing engine that reliably transforms uploaded files into AI-generated insights at scale.

Let’s dive in. 🚀

High Level Overview

The Processing Layer is the asynchronous engine that transforms uploaded files into AI-powered insights. Unlike the synchronous upload Layer, this layer handles variable processing times (5-30 seconds), manages expensive AI API calls, and gracefully handles failures at scale.

Core Components:

• Backend Layer (Service) : When a file is uploaded, the backend enqueues a job with metadata ({file_id, user_id, s3_key}) to the message queue for preparing the file insights.

• Message Queue (SQS/Kafka): Acts as a buffer between the upload layer and workers. . The queue decouples traffic spikes from processing capacity, allowing the system to absorb bursts of 10,000 uploads while workers process them at their own pace.

• Dead-Letter Queue (DLQ): Captures failed jobs after exhausting retries (typically 3 attempts). Failed jobs are isolated for manual investigation, preventing them from blocking the main queue. DLQ messages include error details, stack traces, and the original job payload for debugging.

• Worker Service (Lambda): The brain of the processing layer. Workers poll the queue, fetch job metadata, generates the pre signed url and handles rate limiting, retries with exponential backoff, and timeout management. Workers auto-scale based on queue depth thereby spinning up when jobs pile up and scaling down during idle periods.

• AI Service Integration: The worker sends the S3 file reference (pre-signed URL) to the AI service, which reads the file directly from S3 and processes it as an artifact.

• Results Storage (PostgreSQL): Stores AI-generated insights in a structured schema linked to the original file record. Results include extracted text, metadata, confidence scores, and processing timestamps. This enables fast retrieval, search, and audit trails.

Low level overview for each component

In this section we will dissect mainly the Message Queue and the workers logic to understand how they fit in the bigger picture to generate AI insights. Ai Service will be covered in part 3.

Message Queue

When a file successfully uploads to S3, the backend immediately enqueues a processing job to the message queue. This job contains all necessary metadata for workers to process the file independently without querying the API server.

{
  “job_id”: “uuid-v4”,
  “file_id”: “123e4567-e89b-12d3-a456-426614174000”,
  “user_id”: “user-789”,
  “s3_bucket”: “ai-uploads-prod”,
  “s3_key”: “uploads/user-789/file-123/document.pdf”,
  “file_type”: “application/pdf”,
  “file_size_bytes”: 2457600,
  “uploaded_at”: “2025-11-22T13:29:00Z”,
  “processing_options”: {
    “extract_text”: true,
    “analyze_sentiment”: false,
    “language”: “en”
  }
}

Why Use a Queue :

• Traffic spike absorption: Queue buffers 10,000+ uploads during peak times while workers process at their own pace

• Independent scaling: Workers scale based on queue depth, not API server capacity

• Reliability: Jobs persist in the queue until successfully processed, with automatic retries on failure

• Observability: Queue depth metrics help identify bottlenecks and trigger auto-scalingbs persist in the queue until successfully processed, with automatic retries on failure

• Improve observability: Queue depth metrics help identify bottlenecks and trigger auto-scaling

Worker Process:

The Worker Service orchestrates the complex flow between message queues, file storage, AI services, and databases. Here’s the complete processing pipeline:

The 9-Step Processing Pipeline :

1. Poll SQS queue (long polling, 20s waits)
2. Download file from S3 to /tmp
3. Deep validation (magic bytes, malware scan, integrity check)
4. Preprocessing (resize images, convert PDFs, chunk large docs)
5. Call AI service with retry logic (exponential backoff)
6. Validate AI response schema
7. Store results in database
8. Cleanup (delete /tmp files, ACK message, update status)
9. Publish status update (WebSocket/polling)

Polling and Visibility: Workers use long polling (20s) to reduce costs. SQS sets a visibility timeout (5 min) to prevent duplicate processing. If a worker crashes, the message automatically returns to the queue.

Deep Validation: Beyond upload-time checks, workers verify magic bytes (true file type), scan for malware (ClamAV), and validate integrity. This prevents malicious executables renamed as PDFs.

AI Integration Patterns: Three options exist: (A) Pass pre-signed S3 URL to AI service (recommended - reduces memory), (B) Upload file bytes directly (more secure), (C) Use streaming SDK for large files.

Retry Strategy: Exponential backoff with jitter for rate limits and timeouts. First retry after 1s, second after 2s, third after 4s. Jitter prevents thundering herd when many workers hit limits simultaneously.

Error Handling: After max retries, SQS moves messages to Dead-Letter Queue (DLQ). Workers log errors with full context, update database status to failed, and trigger alerts for investigation.

Cleanup Process: Workers delete local `/tmp` files, ACK the SQS message, update database status to completed, and publish status updates. The original S3 file remains for audit trails and reprocessing.

This pipeline handles the complexity of reliable AI processing at scale including validation, preprocessing, retry logic, and cleanup, while maintaining observability and fault tolerance.

Conclusion

The Processing Layer is the backbone of reliable AI file processing, transforming chaotic concurrent uploads into an ordered, scalable pipeline. Three architectural decisions define its success:

• Queue-based decoupling: SQS buffers traffic spikes, enables independent scaling, and provides built-in retry logic—essential beyond 100 requests/minute.

• Worker reliability: The 9-step pipeline (poll → validate → preprocess → AI call → cleanup) handles the unglamorous but critical work that separates production systems from demos.

• Start serverless, scale smart: Lambda’s zero idle cost and instant scaling work perfectly until you hit 15-minute timeouts or 10GB files, then migrate to containers.

This architecture proves that great systems aren’t built on perfect components, but on reliable communication between imperfect ones.

Coming Up in Part 3: AI Service Integration Deep Dive

In the next part, we’ll explore the intelligence layer that transforms raw files into structured insights:

• Prompt engineering strategies: How to construct prompts that consistently return valid, structured JSON

• Response validation patterns: Using Pydantic schemas and JSON validators to catch AI hallucinations before they reach users

• Cost optimization techniques: Token tracking, model selection strategies, and caching patterns that cut API costs by 60%

• Managed vs. self-hosted decision: The $3000/month tipping point where self-hosting becomes economical

• Rate limiting and backoff strategies: Handling OpenAI’s 3,500 req/min limits without thundering herd failures

Stay tuned for Part 3 : Building the AI Service for preparing file insights, where we turn files into intelligence. 🚀

Subscribe now

Subscribe now

Introduction

Imagine building a system where users upload medical reports and instantly get diagnosis summaries, or financial teams drop invoices and automatically extract line items, or recruiters upload resumes and get candidate insights in seconds. AI has made extracting insights from unstructured files trivial but the hard part is building the production system around it.

AI introduces a new architectural component with unique operational characteristics such as variable latency (5-30 seconds), external API dependencies, rate limits, per-request costs, and data residency concerns.

This isn’t just about calling google_adk.analyze(file). It’s a system design problem that forces you to rethink traditional patterns:

• How do you handle 10,000 concurrent uploads without overwhelming AI rate limits ?

• Should files go through your backend or directly to S3? What about validation and security ?

• Synchronous processing or async queues? What happens when the AI service is down ?

• How do you perform the backfills for missing data with all the GPU scarcity in place ?

• Do you delete files after processing, or retain them for compliance and reprocessing ?

• Should you use managed AI services or self-host models on GPUs ?

This post architectures a production-grade system for AI-powered file insights, treating AI services as first-class distributed components and examining the design decisions that balance security, cost, scalability, and reliability.

Problem Statement

Design a system that allows users to securely upload files (documents, images, PDFs) and have them processed by an AI service to generate the insights, with the following requirements:

Functional Requirements

1. Upload: Users can upload files up to 50MB.

2. Validation: System validates file type, size, and content before processing

3. AI Processing: Files are sent to an AI service ( Wrappers on ADK, GPT-4 Vision, Claude) for analysis.

4. Result Storage: AI-generated results are stored and retrievable.

5. File Retention: Files are retained for audit and compliance purposes with configurable retention policies.

6. Multi-user: Support concurrent uploads from thousands of users.

Non-Functional Requirements

1. Security: Files must be encrypted in transit and at rest, with no unauthorized access.

2. Privacy: Comply with GDPR/HIPAA, no data leakage to unauthorized parties.

3. Performance: File upload and processing should complete within 30 seconds for 95th percentile.

4. Scalability: Handle 10,000 concurrent uploads.

5. Cost: Minimize storage and compute costs.

6. Reliability: 99.9% uptime, with proper error handling and retries.

Subscribe now

System Architecture - Layer View

The system can be organized into three architectural layers, each responsible for a distinct phase of the file processing lifecycle.

This layered architecture improves clarity and maintainability by separating concerns: upload handles secure file acceptance, processing manages AI analysis asynchronously, and the data layer handles storage and lifecycle management.

Below is a breakdown of each layer’s purpose.

Layer 1: Upload Layer

This layer acts as the entry point to the system, ensuring that only authenticated users can upload files, that files meet security and format requirements, and that they are stored in a secure, encrypted manner before being queued for AI processing. The upload layer is designed to handle high concurrency while minimizing backend load through direct client-to-storage uploads.

The table below compares two implementation approaches for this layer. We’ll use the pre-signed URL approach in further section.

High Level Design Overview:

Before diving into implementation details, let’s understand the key components and their interactions in the upload layer:

The upload layer consists of seven key components that work together to securely accept and store files:

1. The Frontend Client (built with React, Vue, or as a mobile app) initiates the upload process by requesting pre-signed URLs, uploading files directly to S3, and notifying the backend when uploads complete.

2. The Load Balancer (AWS ALB or NGINX) sits at the entry point, distributing incoming requests across multiple backend instances while handling SSL termination and performing health checks to ensure system availability.

3. The API Gateway (Kong or AWS API Gateway) acts as the security layer, authenticating users via JWT tokens, enforcing rate limits (10 uploads per minute per user), and logging all requests for audit purposes.

4. The Backend API (FastAPI or Node.js) serves as the orchestration hub, generating time-limited pre-signed URLs with encryption requirements, validating file metadata, and storing upload records in the database.

5. The Database (DynamoDB/PostgresSQL) handles metadata storage, maintaining records of file information, tracking upload status throughout the lifecycle, and providing a complete audit trail of all operations.

6. Finally, Object Storage (AWS S3) provides the actual file storage, keeping uploaded files encrypted at rest and organized into separate buckets for different processing stages.

Component Deep-Dive

This section examines each component in the upload layer, detailing their specific responsibilities, implementation patterns, and security considerations. We’ll explore how these components work together to create a robust, scalable file upload system.

A. Frontend Client (Web/Mobile Application)

The frontend client serves as the user-facing interface where file uploads begin. It handles initial validation, requests secure upload URLs from the backend, and manages the complete upload lifecycle from file selection to processing confirmation.

Some Key Points:

• File Selection & Pre-Upload Validation: Validates file size (max 50MB) and type before requesting upload URLs to prevent unnecessary backend calls

• Pre-Signed URL Request: Sends file metadata (name, size, type) to backend via authenticated API call to receive temporary S3 upload URL

• Direct S3 Upload: Uploads file directly to S3 using pre-signed URL, bypassing backend to reduce server load and bandwidth costs

• Upload Completion Notification: Notifies backend after successful S3 upload to trigger AI processing pipeline and update file status

// Validation and upload flow
function validateFile(file) {
  const MAX_SIZE = 50 * 1024 * 1024; // 50MB
  const ALLOWED_TYPES = [’application/pdf’, ‘image/jpeg’, ‘image/png’];
  
  if (file.size > MAX_SIZE || !ALLOWED_TYPES.includes(file.type)) {
    throw new Error(’Invalid file’);
  }
}

async function uploadFile(file) {
  // Get pre-signed URL
  const { upload_url, file_id } = await fetch(’/api/v1/files/upload-url’, {
    method: ‘POST’,
    headers: { ‘Authorization’: `Bearer ${jwtToken}` },
    body: JSON.stringify({ file_name: file.name, file_size: file.size, file_type: file.type })
  }).then(r => r.json());
  
  // Upload to S3
  await fetch(upload_url, { method: ‘PUT’, body: file });
  
  // Trigger processing
  await fetch(`/api/v1/files/${file_id}/process`, { 
    method: ‘POST’, 
    headers: { ‘Authorization’: `Bearer ${jwtToken}` } 
  });
  
  return file_id;
}

B. Load Balancer (Application Load Balancer)

Load balancer helps to distribute incoming traffic across backend servers, ensure high availability through health monitoring, and handle SSL/TLS termination for secure communication

Some Key Points:

• Traffic Distribution: Routes requests across multiple backend instances using intelligent algorithms to prevent server overload

• Health Monitoring: Continuously checks backend server health and automatically removes unhealthy instances from rotation

• SSL/TLS Termination: Handles encryption/decryption at the edge, reducing computational burden on backend servers

• Connection Management: Maintains persistent connections and optimizes request routing for better performance

Algorithm Selection for Load balancer:

Why are we selecting Least Outstanding Requests? This is because our system handles both fast requests (pre-signed URL generation ~50ms) and slower requests (processing triggers ~200ms). This algorithm prevents slow requests from blocking fast ones.

C. API Gateway + Authentication Layer

This is the first line of defense which is responsible to authenticate users, enforce rate limits, and protect backend infrastructure from unauthorized access.

Some Key Points:

• JWT Authentication: Validates bearer tokens to verify user identity and permissions before allowing any file operations

• Rate Limiting: Implements token bucket algorithm via Redis to prevent abuse by enforcing 10 uploads per user per minute

• DDoS Protection: Blocks malicious traffic and unauthorized requests before they consume backend resources

• Token Expiry Validation: Ensures expired JWTs are rejected to maintain security posture

Why This Matters: Prevents DDoS attacks and unauthorized uploads before they reach your backend, reducing costs and protecting infrastructure.

D. Backend API Server

The Backend API Server acts as the central orchestration layer that validates requests, generates pre-signed URLs and triggers processing for preparing AI insights.

Some Key Points:

• Request Validation: Ensures all incoming file metadata (name, size, type) meets system requirements before proceeding with upload flow

• Pre-Signed URL Generation: Creates cryptographically signed, time-bound URLs (15-minute expiry) that grant temporary S3 access with enforced encryption and content-type restrictions

• Metadata Management: Stores comprehensive file records in PostgreSQL including upload status, S3 location, timestamps, and user associations for complete lifecycle tracking

• Processing Orchestration: Triggers asynchronous AI processing workflows after upload completion and provides status endpoints for monitoring

• Security Enforcement: Implements multiple security layers including encryption requirements, size limits, type restrictions, and user-scoped object keys to prevent unauthorized access

Key APIs:

POST /api/v1/files/upload-url
- Input: {file_name, file_size, file_type, user_id}
- Output: {upload_url, file_id, expires_in}

POST /api/v1/files/{file_id}/process
- Input: {file_id}
- Output: {job_id, status}

GET /api/v1/files/{file_id}/status
- Input: {file_id}
- Output: {status, result, error}

E. Pre-Signed URL Generation - Deep Dive

Pre-signed URLs are time-limited, cryptographically signed URLs that grant temporary access to S3 objects.

A simple service logic could look like this :

class FileUploadService:
    def __init__(self):
        self.s3_client = boto3.client(’s3’)
        self.bucket_name = ‘ai-uploads-prod’
    
    async def generate_upload_url(self, user_id: str, file_metadata: dict):
        “”“
        Generate pre-signed URL with multiple security layers:
        1. Time-bound: 15-minute expiry
        2. Scoped: Only PUT to specific object key
        3. Type-restricted: Enforces Content-Type
        4. Size-restricted: Max 50MB
        5. Encryption-enforced: Must use AES256
        “”“
        
        file_id = str(uuid.uuid4())
        object_key = f”uploads/{user_id}/{file_id}/{file_metadata[’file_name’]}”
        
        # Security conditions
        presigned_url = self.s3_client.generate_presigned_url(
            ClientMethod=’put_object’,
            Params={
                ‘Bucket’: self.bucket_name,
                ‘Key’: object_key,
                ‘ContentType’: file_metadata[’file_type’],
                ‘ServerSideEncryption’: ‘AES256’,  # Enforce encryption
                ‘Metadata’: {
                    ‘user-id’: user_id,
                    ‘upload-timestamp’: datetime.utcnow().isoformat()
                }
            },
            ExpiresIn=900,  # 15 minutes
            HttpMethod=’PUT’
        )
        
        # Store metadata in database
        await self.store_file_metadata(
            file_id=file_id,
            user_id=user_id,
            s3_key=object_key,
            status=’pending_upload’,
            url_expires_at=datetime.utcnow() + timedelta(seconds=900)
        )
        
        return {
            “upload_url”: presigned_url,
            “file_id”: file_id,
            “expires_in”: 900
        }

Why 15-Minute TTL for URL Expiry?

• Security: Minimizes risk if URL is leaked (attacker has limited time)

• User Experience: Sufficient for 50MB uploads even on slow connections (~2-5 minutes typical)

• Cost Optimization: Prevents stale URLs from uploading unwanted files

Encryption at Multiple Layers:

• In Transit: HTTPS/TLS 1.3 during upload (enforced by pre-signed URL)

• At Rest: Server-Side Encryption (SSE-S3 or SSE-KMS for HIPAA compliance)

• Metadata: Database encryption for sensitive fields

F. Database (PostgreSQL)

PostgreSQL serves as the central metadata store that tracks the complete lifecycle of every file upload, that is, from initial URL generation through processing completion. It enables audit trails, supports concurrent access through connection pooling, and provides indexed queries for fast lookups by user, status, and timestamp.

Some Key Points:

• File Metadata Storage: Maintains comprehensive records including file details (name, size, type), S3 location, and user associations for complete file tracking

• Lifecycle Tracking: Monitors upload status progression through states (pending_upload → uploaded → queued → processing → completed → failed) with timestamps at each stage

• Audit Trail: Records creation and update timestamps for compliance and debugging, enabling full traceability of file operations

• High Concurrency Support: Uses connection pooling with 20 base connections and 10 overflow connections to handle concurrent requests efficiently

• Optimized Queries: Implements indexes on user_id, status, and created_at fields to enable fast lookups and filtering operations

Schema Diagram:

Indexes: We will need few indices to speed up the query performance on the postgres. Some of these could be as below, if needed can be composite index too :

• idx_user_id on user_id - for filtering by user

• idx_status on status - for filtering by upload state

• idx_created_at on created_at (DESC) - for chronological queries

Connection Pooling : Connection pooling maintains a pool of reusable database connections, avoiding the overhead of creating new connections for each request.

• Prevents Connection Exhaustion: Without pooling, each request creates a new database connection, which is expensive (handshake overhead ~10-50ms). Pooling reuses existing connections, reducing latency.

• Handles Concurrent Requests: During traffic spikes (e.g., 100 simultaneous uploads), pooling ensures connections are efficiently shared rather than creating 100+ database connections that could overwhelm PostgreSQL.

• Database Protection: PostgreSQL has a maximum connection limit (default 100). Connection pooling enforces a cap per application instance, preventing one service from consuming all database connections.

• Resource Efficiency: Idle connections are kept warm and reused, avoiding the overhead of repeatedly opening/closing connections for each request.

• Graceful Degradation: When all connections are busy, new requests wait in queue (pool_timeout=30s) rather than failing immediately, improving reliability during load spikes.

G. Object Storage (S3)

S3 serves as the secure, encrypted storage layer with strict access controls and compliance-ready configuration

Some Key Points:

• Public Access Prevention: All public access blocked at bucket level

• Encryption at Rest: SSE-S3 or SSE-KMS for HIPAA compliance

• Version Control: Versioning enabled for complete audit trail

• Access Control: IAM roles restrict read access to Worker Service only

Design Decision: Bucket Organization Strategy

There are multiple ways to organize the bucket while storing the files. On a high level there could be two approaches here:

As we can see, we need to setup buckets for different stages and store the uploads data for the user in a upload directory. Also security benefits outweigh the operational complexity.

Conclusion

In this first part of our AI-Powered File Insights series, we’ve explored the foundational Layer 1: Handling Uploads, the critical infrastructure that handles secure file uploads at scale. We’ve covered:

• Client-side validation and chunking for large files

• Backend architecture with pre-signed URLs for direct-to-S3 uploads

• Database schema design with proper indexing and connection pooling

• S3 bucket security with encryption, versioning, and least-privilege IAM policies

• Rate limiting and abuse prevention to protect against malicious actors

But uploading files is only half the story. The real magic happens in Layer 2: Processing Layer, where we transform raw files into actionable AI-powered insights.

In the next post, we’ll dive deep into:

• Message queue architecture : why async processing beats synchronous, and how to choose between SQS, Kafka, and RabbitMQ

• Worker service design : Lambda vs. Kubernetes, handling long-running AI calls, and implementing robust retry mechanisms

• AI service integration patterns : managed services (OpenAI, Claude) vs. self-hosted models, cost optimization, and rate limit handling

• Results storage and retrieval : schema design for AI outputs, search capabilities, and real-time status updates

• Error handling and observability : dead-letter queues, monitoring queue depth, and debugging failed processing jobs

Stay tuned for Part 2: Building the AI Processing Pipeline, where we’ll complete the journey from uploaded file to extracted insights! 🚀

👉 Follow along as we build production-ready AI systems. Next up: asynchronous processing, worker orchestration, and turning files into intelligence.

Subscribe now

LLM streaming is a technique that enables incremental reception of data as it's generated by a large language model, rather than waiting for the entire response to be completed before sending it to the client.

Think of it like watching a live broadcast versus waiting for a pre-recorded message - you see the content as it's being created.

Subscribe now

Real-World Example: ChatGPT

ChatGPT is a prime example of LLM streaming in action, where users can watch responses being typed out word by word, creating a more interactive and engaging experience.

Why is Streaming Important?

Streaming significantly enhances user experience by:

• Reducing perceived wait times for next tokens from LLM.

• Allowing users to see responses being generated in real-time.

• Enabling early interruption if the AI isn't heading in the desired direction.

• Optimizing costs by preventing unnecessary token generations.

The Dissection using the first principles

To understand the LLM streaming, we need to dissect it at three levels:

• Frontend Perspective: How does ChatGPT UI stream content to user.

• Backend Perspective: How does backend supports LLM streaming.

• AI Perspective: How does LLM supports token generation for LLM Streaming.

Let’s dive in…

How does backend support LLM streaming 🤔

When it comes to backend, everything boils down to the client-server architecture and we need to understand how content can be streamed at the the request response level.

There are three main approaches to implement streaming in a client-server architecture:

1. Chunk Encoding

2. Server-Sent Events (SSE)

3. WebSockets

To explore each approach in detail, we will try to understand the networking protocol, how FAST API server implements that protocol and how data flows back to react.

Understanding Chunk Encoding 🚀

Imagine you're sending a very long letter, but instead of waiting to write the entire letter, you send it page by page as you write it. That's essentially what chunk encoding does in the digital world. It's a method introduced in HTTP/1.1 that allows data to be sent in pieces (chunks) rather than all at once.

Let's break down the process step by step:

Initial Setup : The server (In the above case FAST API) first sends the below response headers to tell the client "Hey, I'm going to send you data in chunks!". This helps make sure request from client does not gets marked complete on first chunk.

HTTP/1.1 200 OK
content-type: text/plain; charset=utf-8
Transfer-Encoding: chunked 

Chunk Format : Each chunk follows this structure:

[Size of chunk in hexadecimal]\r\n
[Actual chunk data]\r\n

Example: 
11\r\n
Developer Network\r\n
7\r\n 
Mozilla\r\n

Fast API Implementation:

Fast API has a response type that takes care of setting up the above response header. This response type is called as StreamingResponse.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import time

app = FastAPI()

def number_generator():
    for i in range(10):
        time.sleep(1)  # Simulate processing time
        yield f"Number {i}\n"

@app.get("/stream")
def stream_numbers():
    return StreamingResponse(number_generator(), media_type="text/plain")

Curl response :

curl http://localhost:8000/stream

Number 0
Number 1
Number 2
...
Number 9

Do you know that, Sagemaker uses chunk encoding to expose streaming responses for ML endpoints. Check out this Blog: https://aws.amazon.com/blogs/machine-learning/elevating-the-generative-ai-experience-introducing-streaming-support-in-amazon-sagemaker-hosting/

Understanding Server Side Events (SSE) 🚀

Server-Sent Events (SSE) is a technology that enables servers to push real-time updates to clients over a single, long-lived HTTP connection. Think of it as a one-way communication channel where the server can continuously broadcast information to the client.

Let's break down the process step by step:

Client Request: Client first send the request saying that it is looking for the events at `/events` and is ready to accept the data as event-streams.

// Client Request
GET /events HTTP/1.1
Accept: text/event-stream

Server Response : The server (In the above case FAST API) first sends the below response headers to tell the client "Hey, I'll send you data as a stream of events". This helps make sure request from client does not gets marked complete on first event and makes it a long lived connection.

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive

Fast API implementation:

from fastapi import FastAPI
from sse_starlette.sse import EventSourceResponse
import asyncio

app = FastAPI()

@app.get("/stream")
async def stream():
    async def event_generator():
        while True:
            yield {"data": "Hello! This is a server update"}
            await asyncio.sleep(2)  # Send update every 2 seconds
    return EventSourceResponse(event_generator())

Curl Response:

curl -N -H "Accept: text/event-stream" http://localhost:8000/stream

data: Hello! This is a server update

data: Hello! This is a server update

data: Hello! This is a server update

Understanding Websockets 🚀

WebSocket is like having a phone conversation where both parties can speak at any time. It's more complex but offers greater flexibility. This is usually used in Chat Applications.

Comparison of 3 approaches:

Dance of tokens: How LLMs stream their thoughts 🤔

When you see an AI chatbot responding to you word by word, what's happening behind the scenes is a fascinating interplay between the LLM's token generation and the backend's streaming capabilities. We have already covered how backend supports streaming capabilities above.

Let's understand this dance of tokens generated by LLM that makes real-time AI interactions possible.

The Token Generation Process

At its heart, an LLM thinks in tokens, not words. When you send a prompt, the model begins what's called autoregressive generation – a fancy term for predicting one token at a time, each prediction influenced by what came before it. Imagine someone writing a story, but instead of thinking about the whole sentence at once, they're deciding each word based on what they've written so far.

The process is sequential and methodical. The model looks at your input, generates a probability distribution over its entire vocabulary, and selects the most likely next token. This could be a full word like "hello" or just part of a word like "ing" or "pre". Each token is a building block that contributes to the final response.

These tokens are what streamed using APIs implemented by OpenAI, Anthropic AI etc. Players like OpenAI, Anthropic AI uses one of the 3 approaches shared above for their backend implementations to stream content.

Subscribe now

References: 📖

Diving deep into the LLM

Open AI Streaming completion in Python

Possible Llama Opensource implementation for handling streaming of tokens