Troubleshooting

Start with the log. Under systemd it is journalctl -u xtq-web -n 100, in Docker docker logs xtq-web. Most problems are named there. The answers xTQ Web gives itself have short plain-text bodies, so curl -i usually tells you which part answered.

The backend answers 404 for everything under a prefix

What you see. route /api/* { proxy ... } reaches the backend, but every request gets the backend's own 404.

Why. The path the backend gets doesn't match its routes. With strip, /api/auth/me arrives as /auth/me. Without it, as /api/auth/me. If your backend defines /api/auth/me, strip breaks it. If it defines /auth/me, leaving strip out breaks it.

Fix. Use strip only when the backend's routes do not include the prefix. Check what the backend expects by calling it directly, for example curl -i http://127.0.0.1:3000/api/auth/me and curl -i http://127.0.0.1:3000/auth/me. See strip.

Also check that the route matches at all. /api/* does not match /api without a trailing slash. Use /api*, or add route /api { redirect /api/ 301 }.

502 Bad Gateway

What you see. 502 with the body Bad Gateway, and in the log something like xtq-web: example.com: 127.0.0.1:3000 failed (Connection refused (os error 111)), out of rotation for 30s.

Why. No backend of the pool could be reached, or the connection broke after the request was sent. Common causes: the app is not running, it listens on another port or only on another address, or it crashed during the request.

A client that sends a broken request body, for example bad chunked encoding, or hangs up during an upload, gets 400 Bad Request instead, and the backend stays in rotation.

Fix. Call the backend from the server with the exact address in the config, like curl -i http://127.0.0.1:3000/. For unix: backends, check the socket path and that the xtq-web user may connect to it. Under the packaged unit, sockets in /tmp and under /home are not visible, see Install.

504 Gateway Timeout

What you see. 504 with the body Gateway Timeout.

Why. The backend accepted the request but did not send response headers within the pool's timeout, 120 seconds by default. For php sites, XTQW_UPSTREAM_TIMEOUT.

Fix. Find out why the app is slow. If it is slow on purpose, for example a report that takes minutes, raise timeout for that route. Responses that send their headers early and then stream are not affected by timeout.

413 Payload Too Large

What you see. An upload fails with 413 and the body Payload Too Large.

Why. The request body is larger than the limit for that route. For proxy, that is the pool's max_body. There is no limit unless you set one. For php sites, it is XTQW_MAX_BODY, 64 MB by default. A body without a Content-Length is cut off once it passes the limit, so the client may have sent part of it already. The backend is not marked as failed.

Fix. Raise max_body in the proxy block, for example max_body 100M, and reload. For php sites, raise XTQW_MAX_BODY (in bytes) and restart. See max_body.

503 "Server busy, retry in a moment"

What you see. 503 with Server busy, retry in a moment and Retry-After: 2.

Why. One of three limits is full:

• a proxy pool's in-flight cap (limit, 1024 by default): all slots stayed taken for the whole wait,
• the server-wide WebSocket cap, XTQW_MAX_TUNNELS,
• for php sites, the waiting room. Then the response also has X-Xtq-Cache: EDGE-FULL.

A pool that serves many server-sent event streams or long polls fills up even at low traffic, because each open stream holds a slot.

Fix. If the backend can take more, raise limit, for example limit 4096 3s. If the backend is really saturated, the 503 is doing its job: add instances, or let challenge keep automated traffic out.

503 "Edge out of connection capacity toward this site"

What you see. 503 with Edge out of connection capacity toward this site, retry in a moment, and out_of_ports rising on the dashboard.

Why. This machine ran out of source ports toward the backend, or of file descriptors. The backend never saw these requests and is not marked as failed.

Fix. Lower the pool's limit, which bounds the connections it opens. Use a Unix socket backend, which needs no ports. Check that the open-file limit is high, the packaged unit sets LimitNOFILE=1048576.

503 "Service temporarily unavailable, retry shortly"

What you see. On a php site, 503 with Service temporarily unavailable, retry shortly.

Why. xTQ Web could not reach xTQ PHP on its socket, or the reply broke off.

Fix. Check that xTQ PHP is running and that the socket path in php <socket> is right and reachable by the xtq-web user.

503 "Too many concurrent requests from your address"

What you see. On a php site, some visitors get 503 with Too many concurrent requests from your address, retry shortly and X-Xtq-Cache: FAIR-SHARE.

Why. One client address had more than XTQW_PER_IP_SLOTS requests (default 4) in PHP at once. If every visitor seems to share one address, trust is wrong, see wrong client address.

Fix. Fix trust first. If many real users share one address, an office behind NAT for instance, raise XTQW_PER_IP_SLOTS.

404 "Not Found" for a hostname you configured

What you see. Every path of a site answers 404 with the body Not Found, and the backend sees nothing.

Why. No site matched the request's hostname on the listener it arrived on. Typical causes: the site has its own bind and the request came in on another address, the hostname is spelled differently, or the request goes to the server's IP address with no site *.

Fix. Run xtq-web check and look at the listeners and sites lines. Test with the hostname set explicitly, curl -i -H 'Host: example.com' http://127.0.0.1/. Add a site * block to see whether requests arrive at all.

A site that has routes but no handler of its own also answers 404 Not Found for paths no route matches.

A real path gets a joke 404

What you see. A path like /actuator/health, /cgi-bin/... or /api/.env answers 404 with a short note instead of reaching the app.

Why. It looks like a scanner probe, and those are answered before any route of the site is looked at. See Scanner probes for the list.

Fix. Put probes off directly in that site's block, then reload. The site's routes then get these paths like any other. XTQW_PROBE_MEMES=0 only makes the answer a plain 404, it still blocks the path.

Paths under /.xtq/ answer 404

What you see. Your app has routes under /.xtq/, and they answer 404.

Why. The /.xtq/ prefix belongs to xTQ Web on every site.

Fix. Move those routes elsewhere in your app.

A reload is refused

What you see. systemctl reload xtq-web fails, or the log says reload refused, still serving the previous config: ....

Why. The new config has a mistake. Under systemd, reload runs xtq-web check first and does not send the signal at all. With a manual SIGHUP, the running process refuses the new file and keeps the old one.

Fix. Run xtq-web check /etc/xtq-web/xtq-web.conf. It names the file and line. Fix it and reload again. Nothing was interrupted in the meantime.

A config can also be refused for things outside the file, such as a php docroot that does not exist.

"can't listen on ... yet"

What you see. xtq-web: can't listen on 0.0.0.0:80 yet (...), retrying every 100 ms, and that listener does not answer.

Why. xTQ Web retries a listen address until it works, whatever the error, and logs only the first failure. The error in brackets says why:

• Address already in use: another program has the port. Find it with sudo ss -ltnp 'sport = :80'.
• Permission denied: ports below 1024 need root or CAP_NET_BIND_SERVICE. The packaged unit grants it. When you run the binary by hand as a normal user, use sudo, or sudo setcap cap_net_bind_service=+ep /usr/local/bin/xtq-web.
• Cannot assign requested address: the IP in bind is not on this machine.

On most Linux systems a [::] listener also takes IPv4, so bind 0.0.0.0:80 [::]:80 makes the second one wait forever. Use [::]:80 alone for both, or name specific addresses.

Fix. Remove the cause. xTQ Web picks the address up within 100 ms and logs now listening on ..., no restart needed.

Certificates are not issued

What you see. Browsers get a certificate for localhost, or a warning, and the log shows xtq-web: acme: <host>: ... errors, or no acme lines at all.

Why. One of:

• XTQW_ACME is not set to 1. There is then no acme: managing certificates line at startup.
• The hostname is not in the list. Only exact names with a dot are used, not * or *.example.com sites, and not IP addresses.
• Let's Encrypt could not fetch the challenge. The DNS does not point at this server, port 80 is closed in a firewall, or a proxy in front does not pass /.well-known/acme-challenge/ through.
• Let's Encrypt's rate limits were hit after many failed tries.
• XTQW_ACME_DIR is not writable by the service user.

Fix. Check DNS with dig +short example.com, and port 80 from outside the server. While testing, use XTQW_ACME_STAGING=1 to avoid production rate limits. After a failure the next attempt is 12 hours later, so reload or restart once the cause is fixed to retry at once.

Browsers warn about the certificate

What you see. A certificate warning, and the certificate is for localhost or is self-signed.

Why. No issued certificate matches the name the browser asked for, so the fallback was served. Also the case for clients that connect by IP address, without a name, and for certificates from the staging service.

Fix. See the section above, or set XTQW_CERT and XTQW_KEY to a real certificate. Turn XTQW_ACME_STAGING off for production.

The app sees the wrong client address

What you see. Logs in your app, ip rules or per-address limits all see the same address, usually 127.0.0.1 or your CDN's.

Why. The proxy or CDN in front of xTQ Web is not in trust, so its X-Forwarded-For is ignored. Or you wrote a trust line without loopback, which replaces the default 127.0.0.1 ::1.

Fix. List every proxy in front, for example trust 127.0.0.1 ::1 cloudflare. In the app, read X-Forwarded-For from the right and skip your own proxies. xTQ Web appends the address it was connected from to the chain it received. See The real client address.

WebSockets don't connect

What you see. The browser reports the WebSocket closed or failed during the handshake.

Why. Usual causes:

• The backend answered the handshake with something other than 101. That answer is passed to the client as it is. Check the status in the browser's network panel.
• A proxy in front of xTQ Web drops the Upgrade and Connection headers.
• The WebSocket cap, XTQW_MAX_TUNNELS, is full: the client gets 503 Server busy, retry in a moment.
• The backend took longer than the pool's timeout to answer the handshake.
• The app runs on several instances and its library falls back to long polling, which needs sticky sessions that xTQ Web does not offer.

Fix. Call the backend's WebSocket endpoint directly to rule it out. Make sure the proxy in front forwards upgrades. For several instances, configure the client to use WebSockets only, or run a single instance.

The browser check keeps coming back

What you see. "Verifying your browser" appears, finishes, and appears again.

Why. The browser did not send back a valid xtq_pass cookie. Usually XTQW_SECRET is not set and the server was restarted or reloaded, which invalidates every pass. With several xTQ Web servers behind one load balancer, each one has its own key unless they share XTQW_SECRET, so a pass from one is refused by the others. A browser that blocks cookies for the site never keeps a pass at all.

Fix. Set XTQW_SECRET to a long random string in /etc/xtq-web/env, the same on every server, and restart. Check that the browser accepts cookies for the site.

Every page shows "Verifying your browser" or "Service temporarily unavailable"

What you see. Right after starting, every hostname shows the browser check page or 503 Service temporarily unavailable, retry shortly, and the startup line says config=environment.

Why. XTQW_CONFIG is not set, so xTQ Web runs in environment-only mode. It listens on ports 80 and 443 and sends everything to xTQ PHP at /run/xtq/php.sock, which isn't there.

Fix. Set XTQW_CONFIG to your config file. The packaged unit does this.

A backend is marked down but the app works

What you see. failed its health check in the log, but the app answers in the browser.

Why. The health check sends GET <path> with Host set to the backend's own address, like 127.0.0.1:3000, whatever host says. Only 2xx counts as healthy. An app that redirects the health path, needs a particular hostname, or answers slower than health_timeout (3 seconds by default) fails.

Fix. Try what the check does: curl -i -H 'Host: 127.0.0.1:3000' http://127.0.0.1:3000/health. Point health at a path that answers 200 for any host, or raise health_timeout.

Files don't change after a deploy

What you see. You deployed new files, but the old ones are still served.

Why. If the files root is a symlink you switched to a new folder, it was resolved when it was first used and still points at the old folder.

Fix. Reload after switching the symlink: systemctl reload xtq-web. Files changed in place are picked up at once, since the file cache follows modification times.

Files answer 404 under systemd but exist

What you see. files answers 404 for files that are there, though it works when you run the binary by hand.

Why. The xtq-web user can't read them, or they are under /home or /root, which the packaged unit hides with ProtectHome=true. In Docker, the installer mounts only /srv and /var/www from the host, so a folder elsewhere is not visible in the container.

Fix. Move the files to a place like /var/www or /srv, make them readable, or mount them into the container at the same path.

The dashboard answers 404

What you see. /.xtq/dashboard?token=... answers 404.

Why. XTQW_ADMIN_TOKEN is not set in the service's environment, is shorter than 16 characters, or the token in the URL does not match exactly. A token with characters such as +, /, = or & can break when it is put in a URL.

Fix. Use a token of letters and digits, at least 16 long, such as the output of openssl rand -hex 32. Put it in /etc/xtq-web/env and restart.

An environment variable has no effect

What you see. You changed XTQW_..., but nothing changed.

Why. Environment variables are read only at startup, so a reload does not pick them up. Values that can't be read are silently replaced by the default, so XTQW_ACME=true or XTQW_CACHE_MB=1G do nothing. In Docker, the installer passes /etc/xtq-web/env when it creates the container, and Docker reads that file only then.

Fix. Use 1 and 0 for switches and plain numbers elsewhere, then restart. In Docker, put the variable in /etc/xtq-web/env and run the installer again with --docker, which recreates the container. A docker restart alone does not read the file again, see Install.