Aderra

Why the search runs on our own machines

A company has thousands of contracts and meeting notes, and people want to ask plain questions about them and see which document the answer came from. Sending that content to a hosted AI service wasn't an option for us. It had to run on our own machines.

Aderra pulls documents from systems like Slack or Google Drive, splits them into passages, and converts each passage into a vector so questions match on meaning rather than keywords. That search, the web app, and the production deploy are Aderra's.

What using it looks like

Someone drops a file into a project in the web app. The upload returns immediately. In the background the file gets its text pulled out, chopped into passages, and turned into vectors stored in a search index.

Later they type a question. The system finds the most relevant passages, hands them to a language model as context, and the answer appears a few words at a time with links back to the file it came from.

How the pieces fit

Four processes, each owning one job. A Next.js web app owns the interface and no logic beyond it. A FastAPI service on port 8080 owns requests, and sends chat answers back as server-sent events, a one-way stream of JSON chunks over a single HTTP connection. A worker pool owns anything slow, so a ten minute transcription never happens inside a web request. Embedding models run in a fourth process with their own container image.

Storage splits three ways. Postgres for metadata about users and files. Vespa for the search index holding the vectors. MinIO, an S3-compatible file store, for the original uploaded bytes.

What I worked on

Getting it running in production was most of it. Three GitHub Actions workflows that build container images on self-hosted ARM machines and push them to Azure's registry. Four Kubernetes config files, including a deliberate split of the public routing rules so that a URL-rewriting rule meant for the API stopped breaking pages in the web app.

On the backend I added the audio path. That's a new `audio_transcription.py` module, a database column to hold the resulting text, and changes to the background worker and indexer so the text actually reaches the search index.

Turning audio into searchable text

A recording is useless to a text search engine until someone transcribes it. Azure's speech service offers two ways to do that, and they don't behave the same. The quick endpoint returns a result in one request but handles only about 15 seconds of audio. The batch endpoint handles long recordings, and it insists on fetching the audio from a URL reachable over the public internet. Our files sit in storage inside a private cluster.

So the code branches on file size at 256 KB, a conservative stand-in for 15 seconds that holds for both uncompressed WAV and 128 kbps MP3. Larger files get written to a temporary key and handed to Azure as a link that expires after 3600 seconds. The job is then polled every 5 seconds, with a hard stop at 600 seconds, because a job that never finishes would otherwise occupy a worker indefinitely. Cleanup of the temp file runs on every exit path, including failure.

Keeping answers streaming

Answers are meant to appear word by word, and every layer between the model and the browser would rather collect the whole response and send it as one lump. That turns a live answer into a blank screen followed by a wall of text. Three separate layers did this to us.

Fixing it meant turning buffering off explicitly at the cluster's entry point, raising the read timeout to 300 seconds so long answers don't get cut off mid-sentence, and tagging streaming responses with headers that tell proxies not to hold them. The web app also stopped routing chat through its own server and now calls the API directly. Getting the CDN to cooperate took five commits.

Where it's weak

The deploy pushes an image with no version tag and tells Kubernetes to restart with it. Nothing records which build is currently running, so there's no previous version to fall back to if one breaks. We took that shortcut while the cluster was still changing shape every week. Tagging each image with its commit hash is the fix, and it isn't done.

The database change for audio text also went in by hand rather than through the project's migration tool, and it still carries a placeholder version id. It works, but the database can now be in a state the migration history doesn't know about. There's a small script that checks the column really exists, which is a patch over the problem rather than a solution.