A SnoutData Cloud project's API can run as more than one instance, on more than one server. Metis is what runs them: it places the instances and keeps them running, moves them, asks for a server when none has room, and empties servers so they can be retired. Here is how it works, and what happened when we stopped an instance, silenced a server and packed two servers into one.
The short version
- Calls go to running instances. Through one router, 40 of 40 calls answered, spread evenly over two servers.
- A failure cost no call. 40 of 40 answered while an instance stopped, and 20 of 20 while a server went silent to the router.
- The router's hop is free at the median. 2 ms through the router, 2 ms straight to the server.
- A move never leaves a project short. The new instance must run and answer for 30 seconds before the old one goes. In the drill both ran at once, and the emptied server was retired at once.
- It works on the live service. A project scaled to 2 had its second instance running 7 minutes later and answered 40 of 40 calls from the internet.
- It is open source (snoutdata/snout-metis), and you use it through Scale, on the dashboard or the
snoutdataCLI.
When each result was measured
| Result | Measured | On |
|---|---|---|
| Calls through a router, an instance stopped, a server silent, the hop | Oct 30 | A throwaway project, one router, two servers |
| A container through the router, a move, two servers retired | Oct 30 | Two throwaway projects, two servers |
| A project scaled to 2, calls from the internet | Oct 30 | The live service, both of its routers |
Why we built it
Every Cloud project starts with its API on one server, beside its database. Scale gives it more instances on other servers, and that takes three things:
- Something that decides where each instance runs, and keeps deciding: a server fills up, a server is lost, an instance stops.
- Servers that come and go with the instances. A new one when none has room, and none left running half empty.
- Routing that does not wait on the decider. A call must never depend on Metis being up.
How it works
The pieces
- Instances. Scale gives a project's app API (auth, the data API and its database router) more instances: 1 by default, up to 2 on Pro and 3 on Business, in the project's region. Functions, realtime and storage stay beside the database.
- Servers. Instance 1 runs beside the database, on the project's own server. Each instance after it is an API Stack Copy: instance 2, instance 3. Metis never touches the database, which is a layer below it.
- Members. Metis runs as three members in three separate locations. One of them leads and decides, and a decision stands once a majority of them hold it.
- Agents. Every server has one. It runs the instances Metis places there, restarts what stops, and reports what runs.
- Routers. Every call to a project's API address reaches one of the region's routers, which holds a table of where every instance runs and chooses an instance for each call.
Where a call goes
When an instance refuses a call, the router tries another instance of the same project. A call already sent to a server that goes silent is sent again only when that is safe; a POST is answered 503 and never sent twice.
Metis pushes the table to the routers, and a router never asks Metis for it, so Metis is never in a call's path. With every Metis member down, the routers keep routing from the last table they hold.
A server when none has room
When no server has room for the next instance, Metis places what fits and its placement shows the project short. That starts a new server, which our fleet launches in about ten minutes, and the project keeps answering on the instances it has meanwhile.
Moving an instance
We move instances to fill servers rather than leave them half empty. A move is make-before-break: the new instance must run and answer for 30 seconds before the old one is taken off, so the project never has fewer instances than it was scaled to. A server emptied this way is retired automatically, and every move appears on the project's audit log.
What we measured
Calls through the router
- Spread. 40 calls through one router, to a project with two instances on two servers: the servers took 90 and 91 packets from the router.
- An instance stopped while 40 calls ran: every one answered, and the router's log shows it found the stopped instance refusing and sent the call to the other.
- Every kind of call went through the router and answered: the data API, auth, storage (64 MB uploaded and downloaded, byte for byte), a function, and a realtime websocket held open for 20 seconds.
A server gone silent
We made one server drop everything from the router, open connections included, with no reset and no refusal, and kept calling.
The router probes each server once a second, and two misses take it out. The one call caught on the silent server was closed then and sent to the other instance, so it answered in 2.7 s. The 19 after it went straight to the instance that answers.
The router's hop
50 calls each way within the region, once through the router and once straight to the server.
Packing two servers into one
Two projects, each scaled to 2, their second instances on two half-full servers, left alone. After an hour one of the two servers was drained, and Metis moved its instance.
- Both instances ran at once during the move, and never fewer than one.
- The emptied server was retired at once.
- A container on one of the projects answered 200 through the router.
- Unscaling both projects then left the other server empty, and it was retired on its own about an hour later.
On the live service
A project on a Business plan, scaled to 2 through the same call the dashboard makes, had its second instance, an API Stack Copy, running 7 minutes later. Both live routers held both instances in their tables, and 40 of 40 calls from the internet answered 200.
How we measured
- The router drill, Oct 30, 01:28 to 02:14 UTC: a throwaway project with two instances on two servers, reached through one router. Which server served each call was read from the servers themselves, counting the packets each received from the router while the calls ran. The hop was timed on the router and on the project's own server, 50 calls each.
- The packing drill, Oct 30, 17:25 to 20:00 UTC: two throwaway projects through one router, watched every 30 seconds, then unscaled and left for the empty server's retirement.
- The live check, Oct 30, 21:36 to 21:54 UTC: a throwaway project on the live service, called from the internet through the live routers, then unscaled and deleted.
How to scale a project is in the Servers and instances guide, the announcement is the blog post, and Metis's source is public. If you think we got something wrong, write to research@snoutdata.com and we will re-run it and publish the correction.