Reverse proxy

proxy <backend>... {
    lb least_conn|round_robin|first
    health </path>
    health_interval <duration>
    health_timeout <duration>
    fail_duration <duration>
    host keep|upstream|<name>
    header_up <Name> <value>
    header_up -<Name>
    timeout <duration>
    limit <n> [wait]
    limit off
    max_body <size>
}

proxy forwards a request to one of its backends and streams the answer back. The block is optional. Each option goes on its own line.

site app.example.com {
    proxy 127.0.0.1:3001 127.0.0.1:3002 127.0.0.1:3003 {
        health /api/health
    }
}

The backends of one proxy line form a pool. Each proxy line is its own pool, with its own health state and limits, even if two of them list the same addresses. In logs and on the dashboard a pool is named after its site and route, like example.com /api/*, or just example.com for a proxy directly in the site block.

Backend addresses

Backends are spoken to in plain HTTP/1.1. https:// backends are refused when the config loads (HTTPS backends aren't supported, point at the plain HTTP port). A path after the port is not allowed.

Options

lb

A request counts as in flight from the moment it is sent until the response body has been fully passed on. An open WebSocket counts for as long as it is open.

Backends that failed their last health check, or failed a request within fail_duration, are skipped. If that leaves none, all of them are tried anyway, since a health check can be wrong but a request that is never sent is certainly lost.

first gives you a primary and a standby:

proxy 127.0.0.1:3000 127.0.0.1:3100 {
    lb first
    health /health
}

health, health_interval, health_timeout

proxy 127.0.0.1:3001 127.0.0.1:3002 {
    health /health
    health_interval 5s
    health_timeout 2s
}

With health, every backend gets a GET to that path every health_interval, starting at once. The check has Host set to the backend's own address (for example 127.0.0.1:3001, or localhost for a Unix socket) and User-Agent: xtq-web health check, whatever the host option says. A 2xx answer within health_timeout is healthy. Anything else is unhealthy, redirects included. Backends start out healthy.

State changes are logged:

xtq-web: app.example.com: 127.0.0.1:3002 failed its health check
xtq-web: app.example.com: 127.0.0.1:3002 passes its health check again

health_interval and health_timeout do nothing without health.

fail_duration

Passive health checks are always on. When a backend can't be connected to, or the connection breaks while a request is on it, it is taken out of rotation for fail_duration:

xtq-web: app.example.com: 127.0.0.1:3102 failed (Connection refused (os error 111)), out of rotation for 30s

A backend that is merely slow, hitting timeout, is not taken out. Neither is one whose request failed because of the client: a broken request body gets 400 Bad Request and a body over max_body gets 413 Payload Too Large, and the backend stays in rotation.

host

proxy 127.0.0.1:8080 {
    host upstream
}

header_up

proxy 127.0.0.1:3000 {
    header_up X-Request-Source edge
    header_up -Cookie
}

Sets a request header for the backend, replacing any value it had, or removes it with a leading -. header_up lines are applied last, after the forwarding headers below, so they can also override Host or X-Forwarded-Proto.

timeout

How long the backend may take to send its response headers, counted from when the request starts going out. If the backend reads a whole upload before it answers, the upload time counts too. Once the headers arrive there is no time limit on the body, so streams and long downloads are not cut. Opening a connection has its own limit of 5 seconds, and a backend that can't be reached in that time counts as unreachable.

proxy 127.0.0.1:3000 {
    timeout 30s
}

limit

limit <n> [wait]
limit off

At most n requests are in flight across the whole pool. When all slots are taken, a new request waits up to wait (default 3s) for one to free up. If none does, it gets 503 Service Unavailable with the body Server busy, retry in a moment and Retry-After: 2. The default is limit 1024 3s.

The cap exists because every request in flight can hold its own connection to the backend. Without one, a burst opens connections without end, and past roughly 28,000 at once the machine runs out of source ports toward that backend.

A request holds its slot until its response body has been fully sent. Server-sent events, long polls and other long-lived responses each hold a slot for as long as they stay open, so a pool that serves many of them needs a higher limit:

proxy 127.0.0.1:3131 {
    health /health
    limit 4096 3s
}

limit off removes the cap. WebSockets don't use the pool's slots, see WebSockets.

The same in-flight count decides when challenge treats the pool as under heavy load, see Protection.

max_body

max_body <size>

The largest request body the pool passes on. Without it there is no limit. Sizes are a whole number with an optional K, M or G, in powers of 1024, upper or lower case, and a B after the unit is allowed. A bare number is bytes.

Fractions such as 1.5M are not accepted. Write 1536K.

proxy 127.0.0.1:3000 {
    max_body 10M
}

A request that declares a Content-Length over the limit gets 413 Payload Too Large before anything is sent to the backend. A body without a Content-Length, such as a chunked upload, is streamed until it passes the limit. Then it is cut off and the client gets 413 Payload Too Large. In both cases the backend is not marked as failed and the request is not tried on another backend.

WebSockets are not affected: once the backend accepts the upgrade, bytes flow without a limit.

What the backend receives

The method, and the path and query string as they are after strip and rewrite. The request is sent over HTTP/1.1, whatever the client used.

Headers are the client's, with these changes:

• Hop-by-hop headers are removed: Connection, Keep-Alive, Proxy-Connection, Te, Trailer, Transfer-Encoding, Upgrade, Proxy-Authenticate, Proxy-Authorization, Expect, and any header the client named in Connection. TE: trailers is passed on when the client sent it.
• Headers starting with X-Xtq- are removed.
• X-Forwarded-For, X-Forwarded-Proto, X-Forwarded-Host, Forwarded, CF-Connecting-IP, X-Real-IP and True-Client-IP are removed unless the client connection comes from an address in trust. From a trusted proxy, Forwarded, CF-Connecting-IP, X-Real-IP and True-Client-IP are passed on as they are.
• Host is set as the host option says.

Then the forwarding headers are set:

Finally the header_up lines are applied.

No X-Real-IP is added. A backend that needs the client address should read X-Forwarded-For from the right and take the first address that is not one of its own proxies.

The response comes back with its hop-by-hop headers removed and everything else as the backend sent it, apart from compression (below), the route's header lines, and the Server and X-Powered-By headers (see Operations).

Streaming

Nothing is buffered. The request body flows to the backend as the client sends it, and the response flows back as the backend writes it. Server-sent events, long polls and streamed HTML arrive when they are produced.

By default the proxy puts no limit on request body size. Set one per pool with max_body. XTQW_MAX_BODY applies only to php sites.

Compression

Proxied responses are compressed on the way out unless the route says compress off. A response is left alone when:

• the request was HEAD, or the status is 1xx, 204, 206 or 304,
• it already has a Content-Encoding,
• its content type is not text: compressed types are text/* and anything containing javascript, json, xml or svg,
• its Cache-Control contains no-transform,
• it declares a Content-Length under 1024 bytes,
• the client accepts none of zstd, gzip or br.

The encoding is chosen in the order zstd, gzip, br, the first the client accepts. A compressed response loses its Content-Length and Accept-Ranges, a strong ETag becomes weak (W/"..."), and trailers are dropped. Every compressible response gets Vary: Accept-Encoding.

Compressed output is flushed whenever the backend pauses, and at least every 16 KB, so a live event is never held back waiting to fill a buffer.

WebSockets

An HTTP/1.1 request with an Upgrade header and Connection: upgrade is a WebSocket (or other upgrade) request. xTQ Web sends the handshake to one backend on a new connection. If the backend answers 101 Switching Protocols, the two connections are joined and bytes flow both ways until either side closes. If the backend answers anything else, for example 401, that answer goes back to the client as it is.

Nothing extra is needed in the config:

site chat.example.com {
    proxy 127.0.0.1:4000 127.0.0.1:4001
}

Things to know:

• Open tunnels are capped by XTQW_MAX_TUNNELS (default 16384) for the whole server. Past that, new ones get 503 Server busy, retry in a moment.
• Tunnels don't take slots from the pool's limit, but they count as in flight for least_conn.
• The handshake must get its answer within timeout. An open tunnel has no idle timeout.
• A handshake that can't reach its backend is not retried on another one. It gets 502 Bad Gateway.
• WebSockets over HTTP/2 are not supported. Browsers then use HTTP/1.1 for the WebSocket, which works.

Failover and retries

A request is sent to one other backend only when the first could not be reached at all: the connection failed, or it closed before the request went out. The request body is intact in that case, so even a POST is safe to send again. Once a request has reached a backend it is never repeated:

Answers the backend gives itself, a 500 for instance, are passed on unchanged.

Local port exhaustion

Every connection to a TCP backend uses a source port on this machine. Under extreme load the machine can run out of them toward one address (EADDRNOTAVAIL), or out of file descriptors (EMFILE, ENFILE). That is not the backend's fault, so xTQ Web does not take the backend out of rotation and does not log it as failed. It answers 503 with Edge out of connection capacity toward this site, retry in a moment and Retry-After: 1, and counts it as out_of_ports on the dashboard. Health checks that can't run for the same reason don't change a backend's state.

The default limit keeps a pool well away from this. If you see it, lower limit or use a Unix socket backend, which needs no ports.

Connection reuse

Connections to backends are kept open and reused, HTTP/1.1 keep-alive. Each backend keeps up to 2048 idle connections, and the most recently used one is picked first. An idle connection is closed after 4 seconds, just under the 5 seconds after which Node.js closes idle connections itself, so a request never lands on a connection the backend is about to drop. If one closes anyway before the request goes out, the request goes out on a fresh connection.

A response the client stops reading, because it went away, closes its backend connection instead of returning it to the pool.

Connections are not shared between pools, and a reload starts every pool fresh: new idle lists, new health state, new in-flight counts.