Document Management & Secure File Processing in Web Applications

Handling user-uploaded files is one of the most common requirements in B2B software—and one of the easiest ways to introduce security vulnerabilities.

Whether handling PDF contracts, medical scans, tax returns, or CSV spreadsheets, storing files directly on application web servers is a recipe for server exhaustion and security breaches.

Here is how modern engineering teams architect secure, high-throughput document management pipelines.

The shape of the problem is always the same: files are large, untrusted, and arrive unpredictably, while application servers are small, trusted and expected to answer requests in milliseconds. Every rule below follows from separating those two worlds.

4 Rules of Enterprise File Handling

1. Direct-to-Cloud Uploads (Client ──► S3 via Pre-Signed POST)
2. Asynchronous Security & Antivirus Scanning (ClamAV / Lambda)
3. Zero Public Buckets (All downloads require Pre-Signed URLs)
4. Automated Encryption at Rest (AWS KMS / SSE-S3)

1. Direct-to-S3 Uploads (Bypassing the App Server)

Uploading large 50MB files directly to your Python/Django backend ties up web workers and consumes bandwidth.

Instead, implement Pre-Signed Uploads: 1. Frontend requests a signed upload URL from Django: GET /api/documents/upload-url/?filename=contract.pdf 2. Django generates an Amazon S3 pre-signed POST policy valid for 10 minutes. 3. Vue 3 uploads the file directly to S3 with real-time progress bars. 4. On upload completion, Vue notifies Django to record the metadata in PostgreSQL.

The Details That Make This Work in Production

The flow above is the right shape, but it only works reliably if you handle the states between those four steps:

Constrain the presigned policy. A presigned URL that accepts anything is a public bucket with extra steps. Your policy should lock down the exact key prefix, a maximum content length, the allowed content types, and a short expiry. Sign for uploads/<user-id>/<uuid>.pdf, not for a free-form key.

Track the document in two phases. Create the database record before upload with status pending, then flip it to scanning when the client confirms, and available only after the scan passes. Every read path in your application must filter on status='available'. Files in pending or quarantined should never appear in a listing, a search result, or a download link.

Use multipart uploads for large files. Anything above roughly 100MB should use S3 multipart upload with retryable chunks, so a dropped connection on a 2GB upload does not restart from zero. For mobile clients, use a resumable protocol and store progress server-side.

Configure CORS deliberately. The bucket needs to accept uploads from your application origin only, with the exact methods and headers required. Copying a wildcard CORS configuration from a tutorial is how buckets end up writable from anywhere.

Do not trust the client's filename. Never build the S3 key from user-supplied text. Generate a UUID, keep the original filename as metadata for display, and sanitise it when it appears in a Content-Disposition header.

Set a hard size ceiling at the policy level and in the interface, and give the user a clear error when they exceed it. Silent failures on large uploads are a recurring support burden.

Handle the orphan case. Uploads that are started and never confirmed leave objects in the bucket. A nightly task that deletes unreferenced objects older than 24 hours keeps storage costs and clutter down.

2. Background File Processing with Celery

Once the file lands in S3, asynchronous Celery workers handle background enrichment: * PDF Thumbnail Generation: Create lightweight preview images for fast dashboard rendering. * OCR Text Extraction: Extract text using Tesseract or AWS Textract for full-text search. * Malware & Virus Scanning: Quarantine infected files automatically before other users download them.

Practical notes on each, because the naive version of all three causes trouble:

Scan before you serve, not after. Trigger the scan from an S3 event notification rather than polling. Files that fail scanning move to a quarantined prefix with a different access policy, and the uploader sees a neutral message. Never tell an attacker precisely what your scanner detected.

Validate the actual file type. Extensions lie. Check magic bytes (python-magic) against an allowlist of types you genuinely support, and reject anything that does not match the declared type. This blocks the classic attack of uploading a script with a .pdf extension.

Watch out for malicious documents that are not malware. A PDF can be a decompression bomb; a DOCX can contain external references that call out to an attacker's server when previewed; an image can carry EXIF metadata with personal data you would rather not store. Strip metadata and pixel limits on images, and render previews in a sandboxed process.

Generate previews outside the web process. Convert Office documents with LibreOffice headless on a worker, render PDFs to images with pdftoppm, and cap the pages you render. A 400-page PDF should produce a preview of page one, not 400 images.

Choose OCR by volume. Tesseract is free and adequate for clean printed text; AWS Textract or Google Document AI is dramatically better on scanned forms, handwriting and tables, at roughly $1.50 per 1,000 pages for basic text detection and more for form or table extraction. For a product processing a few thousand documents a month, the managed service usually wins on accuracy alone.

Make processing idempotent. Workers retry. Key every step on the document ID and version, so a retry regenerates a preview rather than appending a second set of pages.

3. Locking Down Downloads

Rule three is the one that gets compromised most often, because the convenient version of a download link is a public URL.

• Signed URLs, short-lived. 5–15 minutes for sensitive documents, generated per request after an authorisation check. Not 24 hours, and never a permanent bucket URL.
• Authorise before you sign. The signing endpoint is an API endpoint like any other: verify the requesting user has access to that document, in that organisation, before generating anything.
• Do not log signed URLs. They are credentials for their lifetime. Support tooling, error trackers and analytics should all redact them.
• Stream, do not proxy large downloads through Django. For documents over ~25MB, redirect the client to the signed URL rather than reading the file into your application's memory.
• Set Content-Disposition: attachment for anything a browser might try to render inline, and add X-Content-Type-Options: nosniff.

4. Encryption, Retention and Compliance

The fourth rule — encryption at rest — is necessary but not sufficient on its own:

• At rest: SSE-S3 or SSE-KMS on the bucket, plus database-level encryption for PostgreSQL, plus application-level encryption for the columns that would be catastrophic if leaked (tax IDs, identity documents, medical notes).
• In transit: TLS everywhere, including the internal hop from workers to S3.
• Versioning on: it protects against accidental overwrite and ransomware-style deletion, and it gives you a defensible history when a client disputes which version of a contract was signed.
• Lifecycle rules: move to infrequent-access storage after 90 days and to archive after a year if retention requires it. Storage costs are small ($0.023/GB/month on standard S3, roughly a tenth of that in infrequent-access tiers) but egress and duplicated backups add up at scale.
• Retention and legal hold: some documents must be kept (contracts, tax records), some must be destroyed on a schedule, and some must be frozen because of litigation. Model retention as data, per document class, so deleting is a policy decision rather than a manual chore.
• Deletion must be real. A GDPR erasure request that removes the database row but leaves the S3 object is not compliance — and neither is leaving the file in a backup forever. Plan for the object, the versions, the thumbnails, the OCR text index and the backup lifecycle.
• Audit the reads, not just the writes. Who downloaded which document, when, and from where. For regulated clients this log is frequently the first thing an auditor asks to see.

A Word on Search and Scale

Once a client has 20,000 documents, "just list them" stops working. Two decisions follow:

Metadata search versus full-text search. Structured filters (client, date, type, status, tag) belong in PostgreSQL with proper indexes and will handle millions of rows comfortably. Full-text search over OCR'd content is a different problem — PostgreSQL's built-in full-text search is surprisingly capable up to a few million documents; beyond that, or when you need fuzzy matching and relevance tuning, a dedicated engine earns its place.

Upload concurrency. A single large upload should not block others. Keep document processing on a separate Celery queue from transactional email, export generation and other time-sensitive jobs, or a backlog of 200 scanned PDFs will delay password-reset emails.

A Storage Cost Sanity Check

Teams are often surprised in both directions — by how cheap storage is, and by how expensive the wrong decisions are:

The lesson is that storage is rarely the cost problem — egress, duplicated thumbnails, and un-lifecycled old versions are. Instrument the bucket early: how much is in each prefix, how many versions exist per object, and what share of traffic is downloads versus uploads. A quarterly look at those three numbers will catch the runaway cases long before the invoice does.

The Pipeline at a Glance

Get that table right and document handling becomes a boring, reliable part of the product — which is exactly what it should be.

Related reading: The client portal guide is where most of these pipelines end up; PDF report generation covers the outgoing side; and compliance covers retention and deletion obligations. Need custom document processing or secure cloud storage integration? Partner with KEHEM IT.

Drowning in files, versions and access requests? Talk to us about a document system that holds up to an audit.