Psyche is the AI gateway in the SnoutData stack: one OpenAI-compatible endpoint in front of your own provider keys, with every call counted and logged to your own Postgres. Here is why we built it, what it does, and what happened when its providers failed on purpose.
The short version
- Psyche is a small Rust server that speaks OpenAI's API and sends each call to the provider a model belongs to: any OpenAI-compatible server, Anthropic, or Amazon Bedrock. Answers stream back as they arrive; a caller that goes away cancels the provider's call.
- It is built to run once per project, next to that project's database, so it has to be tiny: on a 2-vCPU arm64 machine it sat at 0.3 MB of memory idle and held 1,000 open streams in 61 MB, from a 4.5 MB image.
- With its request log on, it added 0.28 ms at the median to a call (0.35 ms at p99) and served 5,235 requests a second on one CPU.
- Against stand-in providers that failed, stalled and rate-limited at random, none of 10,769,726 requests failed while a model in its route was healthy throughout.
- No call waits for the database: the log is written in batches by a task of its own. A clean stop writes every call; a crash loses at most what that task held, which we measured.
- It is open source (snoutdata/snout-psyche). Every SnoutData Cloud project can switch its own on from the dashboard or the
snoutdataCLI, the self-hosted stack carries it, and SnoutData AI, the assistant in Studio and on the dashboard, runs on it.
When each result was measured
Psyche went from 0.0.1 to 0.2.1 in under four weeks, and we do not rerun every bench for each release. So every number here is a record: what one build did on one day. What works today describes Psyche 0.2.1.
| Result | Measured | On |
|---|---|---|
| Speed and memory, request log on | Oct 18 | Psyche 0.0.1, AWS t4g.small |
| Request log, clean stop and kill -9 | Oct 18 | Psyche 0.0.1, a laptop |
| A budget under 300 calls at once | Oct 18 | Psyche 0.0.1 test build, a laptop |
| Speed with health and a route, log off | Oct 21 | Psyche 0.1.6, AWS t4g.small |
| Providers failing at random | Oct 21 | Psyche 0.1.6, a laptop |
Why we built it
Three things pushed us to a gateway of our own:
- One per project, not one for everyone. Every SnoutData project is a Postgres database with its own servers in front of it: auth, storage, realtime, functions. We want AI to be one more of them, so an app's users can call a model with the project's own keys, budgets and log. That means one gateway per project, mostly idle, which makes its size when it is doing nothing the number that matters.
- The state belongs in the project's own database. Keys, budgets and the request log should be rows the project owns, readable with the same row-level security as the rest of its data, not a separate service to run beside it. Psyche keeps them in a
psycheschema in the project's Postgres and needs nothing else: no cache server, no second database. - Some details we wanted to own end to end. Which models take a prompt-caching breakpoint, how each model takes a reasoning effort, how a Bedrock model id with a colon in it is signed: these decide whether a call works at all, and they change as providers ship new models. Testing against Bedrock, we found that two models we sell take reasoning in two different forms, neither of them the one older models use. In Psyche each model's catalogue entry states what it takes, and nothing is guessed from its name.
What works today
- OpenAI's API:
/ai/v1/chat/completions,/completions,/embeddingsand/models, whole or streamed, so the official OpenAI SDKs work unchanged. And/responses, translated to chat for every provider, so Codex runs on a project key. - Anthropic's API:
/ai/v1/messages, passed through to Anthropic and translated for every other provider, so a tool built for Anthropic can use a project. - Anthropic and Amazon Bedrock, translated both ways, tools, images and streaming included. Bedrock is signed with AWS Signature Version 4, our own implementation, checked against signatures the AWS SDK makes for the same requests, or called with a Bedrock API key. Tested against Bedrock itself with Claude Sonnet 5, OpenAI's gpt-5.6-luna and Qwen3 Coder for chat, and Amazon Titan and Cohere v3 and v4 for embeddings.
- Reasoning, in the form each model takes, with the model's thinking returned and sent back unchanged on the next turn, which a tool loop with thinking on needs.
- Prompt-caching breakpoints reach only a model whose catalogue entry says it takes them; a model that does not can refuse the whole request otherwise.
- A model on your own machine: any server that speaks OpenAI's API, an Ollama included, also when it sits behind a tunnel and a firewall rule that checks a header of its own. The provider's key is sent in that header instead of as a Bearer token.
- Routes and fallbacks: one name for several models, tried in turn, in the order written, cheapest first or drawn by weight, when one fails for a reason that is not the caller's. OpenRouter's
modelsfallback list works too. A call is retried once on the same model, before any of the answer has been sent, after a server error or a short rate limit, and every try is its own row in the request log, so the log is the explanation of the route a call took. A route that has tried every model goes round once more while the call has time, asking again only those whose failure a moment may fix. - Health, measured rather than configured: each model's time to its first byte, tokens a second and recent failures, read from its own calls. A model that fails three times in a row for a reason that was its own is cooled down, so a route asks the others first, and a probe call brings it back once its provider answers again; a cooldown never fails a call by itself. Routes can also be sorted by measured latency or throughput.
- Spreading and sticking: a weighted route draws the order of its models in proportion to their weights, their health and the share of their quota left, so traffic moves off a model as it fills, before the 429s start. A conversation that hit a provider's prompt cache stays on that model while it is healthy and has room.
auto, by rules: a model says what it can take (its window, tools, images, a JSON schema), and a request forautoleaves out every model that cannot take it, then tries the rest cheapest first or in the order the project sets. No model decides where a call goes, and its log row names every model left out and why. Every route makes the same check, so a model that takes no tools is skipped rather than tried and failed.
- How far each model reaches, set per model:
nonesends its calls only to a private network address, which suits a model on your own network;provider, the default, reaches its provider and no web tool;webadds that provider's web search and fetch tools.
- SnoutData AI inside a project: the models in your SnoutData AI plan can be added to a project's gateway beside your own keys, and sit in the same routes. Their calls are paid from the plan of whoever added them.
- Errors that say whose fault they are: the provider refusing Psyche's own credential is never reported as if the caller's key were wrong, a rate limit keeps its
retry-after, a provider account with no credit left is not offered as something to retry, and a provider's error that echoes a key never reaches the caller. - The request log: one row per call in the project's Postgres (model, provider, outcome, tokens, timings, never a prompt or a reply), daily totals kept after the rows expire, and partitions by day, so an expired day is dropped whole.
- What each call cost, on its row, exact to a millionth of a millionth of a dollar: from a price the project sets, or from the Amazon Bedrock prices Psyche ships, matched on region and inference profile, since both change the price. A model with no price is logged with no cost, never a cost of zero.
- Keys of its own, for a person, an app or an agent, limited to some models if you like. Only a hash is stored, and a key revoked anywhere, plain SQL included, stops at once.
- Budgets that are kept before a call is sent: a call's worst case is reserved against every budget on it, and settled to what the provider reported when it ends, so calls in flight together cannot spend past a budget. A budget counts a day, the last few days, from a date, or a month at a time from a day of your choosing, and a refusal says whose budget it was (the project's, a key's, a user's or a group's), so an app can tell its user the right thing. And rate limits in requests and tokens a minute, with OpenAI's
x-ratelimitheaders. - Calls from the browser: an app's signed-in users call with their own token and the project's app key, and are taken only once the project has a hard budget of its own and one for every user. Each user has a budget and limits of their own, a group shares one, and each user reads their own usage through row-level security.
On the dashboard and in Studio
A project's models, routes, keys, budgets and limits are set from its AI page on the dashboard. The drawings below use a made-up project and made-up models.
Studio manages the same gateway from the project's AI section, and moves the provider keys you use in Studio into a project.
What we measured
On an AWS t4g.small (2 arm64 vCPUs, 2 GB), with Psyche limited to one CPU and writing its request log to Postgres 17 on the same machine, against a stand-in provider that answers at once, so what is measured is Psyche and not a model. "Added" is over the same load calling the stand-in directly.
| Measure | Result |
|---|---|
| Image | 4.5 MB |
| Memory, idle | 0.3 MB |
| Ready after start | 389 ms |
| Latency added, one call at a time (p50 / p99) | 0.28 ms / 0.35 ms |
| First byte added, streamed (p50 / p99) | 0.28 ms / 0.53 ms |
| Requests a second, 64 in flight, one CPU | 5,235 (p99 21.5 ms, no errors) |
| 1,000 streams held at once | 61 MB at the peak, no errors |
| Rows in the request log at the end | 112,080, one per call |
The request log is written by a task of its own, one COPY every half second or every thousand rows, so a call never waits for the database. On a laptop, a clean stop after 426,632 calls left 426,632 rows. Killed with kill -9 while 64 calls were in flight, it lost 1,268 rows of 159,550: what the task held since its last write. That is the bound: about a thousand rows under load, at most half a second of calls when traffic is light.
Budgets were tested the same way, against the stand-in provider and a made-up price of $1 per million tokens in and $2.50 out: 300 calls sent at once on a key with a $0.001 budget for the day. 18 were answered and 282 refused before anything was sent, and the answered calls cost $0.000342 together, so the budget was never overspent. The same numbers show the price of that guarantee: the room a call reserves is counted from the size of its prompt, which for a short prompt is about ten times its real tokens, so near its end a budget turns away some calls that would have fit.
Again, with health and routing
Three days later we ran the same bench on another t4g.small, at Psyche 0.1.6, which tracks each model's health and routes by it, with the request log off. One CPU served 4,053 requests a second (p99 26.2 ms, no errors), and 4,060 when each call was asked for through a route of three models sorted by latency rather than by a model's name. So choosing among models costs next to nothing: 0.001 ms at the median, 0.2 ms at p99, and no throughput. A thousand streams held at once peaked at 90 MB, again with no errors. The throughput is below the first run's 5,235. Run side by side on one laptop, the two versions were within each other's run-to-run spread, the newer one ahead, so we read the gap as the machine and report both: between about 4,000 and 5,200 requests a second on one CPU, adding about 0.3 ms to each call.
Under failure
Speed under a stand-in that always answers says nothing about the case a gateway exists for, so we also ran Psyche against providers that misbehave on purpose. A route over three stand-in models (two in one run), and every 0.2 to 1.2 seconds one of them changed at random: answering, answering slowly, failing with a 500 or a 503, rate-limiting with a retry-after of one to four seconds, or hanging. At every instant at least one was healthy. Callers sent requests back to back, half of them streamed, and a stream counted as answered only if it reached its end with no error. The rule was strict: a request fails the test if it failed while some model was healthy from the moment it was sent to the moment it ended.
| Measure | Result |
|---|---|
| Runs | 6, one to two minutes each, 2 to 16 callers, six random seeds |
| Requests | 10,769,726 |
| Failed while a model was healthy throughout | 0 |
| Attempts that reached a model while it was failing | 1,889 of 10,771,587, each moved to the next model and answered |
| Failed with no single model healthy throughout | 28 |
Health moved a route off a failing model within a few calls of its change, which is why so few attempts ever met one. The 28 are worth reading. All 16 in one run were in flight at the same moment: each model was failing when Psyche reached it, and recovered only after its turn had passed. In 0.1.6 a call tried each of its route's models once, in the order fixed when it started, so a model that recovered behind it was not asked again. A second pass while time remained would have answered them, and Psyche 0.2.0 makes one. This test is about correctness, not speed, so it ran on a laptop; the program that drives it is in the repository, and each run is reproduced from its seed.