# Troubleshooting failed deploys: errors and fixes Fix a failed deploy: health check failed, build failed, domain not working, disk full, out of memory, server offline. The exact message, cause and fix. When an ox deploy fails, the run's log names the failed step, its cause and the fix, then the last lines of output, and the previous release keeps serving. Find the message you see below: each entry gives the symptom, the cause and the fix. In the log, a failure always has this shape: Four common failures, each with what the log or the domain row says, what to look at, and the fix. While you fix a failed deploy, the previous release is still serving. How a failure reads: ```text : fix: --- last output --- ``` Open the run in the dashboard, or run `ox runs ` and then `ox logs --run `. A failed `ox deploy --wait` exits with status 1. ## Health check failed: the app crashed or did not answer within 120s ### Symptom The log shows `waiting for … (up to 120s)`, then a step named `health ()` with one of these causes. The health causes: ```text the app exited while starting (systemd restarted it 3 times) the app exited while starting (failed) … did not answer within 120s: … ``` ### Cause The new release started but never answered. `the app exited while starting` means the unit crashed: systemd restarts it after every crash, and ox stops waiting at the third restart (within about 15 seconds) instead of the full 120, then shows the output, which ends with the error that stopped it. A worker or a service says `the worker exited` or `the service exited`. `did not answer within 120s` means the unit stayed up and never answered, with one exception: a server whose ox agent is older than this change still waits the full 120 seconds on a crash loop and says `did not answer within 120s`, so read the output above the message there (`ox servers update ` brings the agent up to date). ox polls every 2 seconds for up to 120 seconds: an HTTP GET on `[app] health` that must return a 2xx or 3xx status, or, when no health path is set, any answer to `GET /` but 502, 503 or 504 within 5 seconds (an agent from before 2026-10-14 only connects to the port; `ox servers update ` updates it). `did not answer GET / within 5 seconds` means the home page is slow: set `health` in `[app]` to a path that answers fast. The usual reasons are a start command that does not listen on `$PORT`, a missing variable that makes the app crash, or a health path that returns 404. The app may write its home folder while it runs, but a PHP app may not: when a PHP app tried to write in its home folder outside the project's own folders, the cause says so: `the app wrote to in its home folder, which is read-only while it runs`. The fix line says to point that path under `$OX_DATA_DIR` with the app's own setting, or to set `[app] sandbox = "relaxed"`. The app may write its own release (a PHP release excepted), but a file it writes there is gone on the next deploy: the deploy's run log says what the app wrote in the live release, so you can add that folder to `[storage] keep`. A database file (`.db`, `.sqlite`, `.sqlite3`, `.db3`) written there stops the deploy before anything changes, and the live version keeps serving. The message names the file and gives the `[storage] keep` line to set in `ox.toml`. Set it and deploy again: that deploy stops the app for a moment, carries the database into the project's data folder, and starts the new version on the same rows. When the app refused the host name it was asked for, the cause names the framework and the name: `Django refused the host name shop.k7mq2xw4ab.preview.getox.in (status 400): ALLOWED_HOSTS does not list it` (Rails' `config.hosts` and Vite's `allowedHosts` read the same way). A staging copy or preview that copied production's list as plain text fails like this. ### Fix Nothing changed for your visitors: the fix line ends with `the previous release is still serving`. Read the app's own log above the failure, then fix it and push. - For `the app exited while starting`, the fix line says to read the output above it for the error, and to run the start command on your own computer to see it fail. A missing variable, a wrong path to the built file and a missing module are the usual ones. Two causes have their own fix line: `the port is already in use` (listen on `$PORT`, not a fixed number, since another project may hold that one) and `killed by the out-of-memory killer as it started` (raise its `memory` in `ox.toml`, or use a bigger server; for a preview that hit the previews' share, remove another preview; when the whole server ran out, stop a project, staging or preview you do not need; see [Not enough memory](https://console.t51.sorv.dev/docs/troubleshooting#out-of-memory)). - Bind to `127.0.0.1:$PORT`, as in [[app] start](https://console.t51.sorv.dev/docs/config#app). - Point `[app] health` at a path that answers without a login. - For a refused host name, read the name from `PUBLIC_HOST`, which is each environment's own: `ALLOWED_HOSTS=${PUBLIC_HOST}` on the [Variables](https://console.t51.sorv.dev/docs/operate/variables#reference) screen, in production too so its copies inherit the reference rather than production's name. - For a write in the home folder, follow the fix line. For data the app writes in its release, add the folder to [`[storage] keep`](https://console.t51.sorv.dev/docs/config#tools), which links it to a writable folder that survives deploys. ## Build failed: exit status 1, or set these before deploying ### Symptom The run stops at an install, build or migrate step, or before anything starts. A failed build step: ```text build 1/2: npm run build: exit status 1 fix: the command's own output is above; fix it in the repo and push ``` A deploy that is missing variables: ```text preflight: 1 problem(s); nothing on the host changed fix: fix the listed problems and deploy again ``` ### Cause A command from [[build]](https://console.t51.sorv.dev/docs/config#build) exited non-zero, or preflight found a problem, such as `set these before deploying: SECRET_KEY` for a variable with no value. A failed build or migrate never touches the live release. ### Fix Run `ox check` in the repo first: it prints `Ready to deploy.`, or a list of problems with the field each one is about. Then reproduce the failing command locally, since the log shows each one as it ran. Set missing variables on the dashboard or with `ox vars set KEY`; [Variables](https://console.t51.sorv.dev/docs/operate/variables#set) shows how. ## Domain not working: waiting for DNS, points elsewhere, no certificate ### Symptom The domain's row on the project's Settings tab says **waiting for DNS**, **points elsewhere** or **No certificate** instead of **HTTPS on**. ### Cause The name does not resolve to the server yet, or it resolves somewhere else, or a Cloudflare proxy in front of it has no certificate for it. The row says which one, for example `It resolves to 203.0.113.9, not this server. Change the record to:` followed by the A record to create. ### Fix Create or change the record the row shows, at your DNS provider ([how to point a domain](https://console.t51.sorv.dev/docs/operate/domains#a-record)). From a terminal, `ox domains ` lists each domain with its DNS state and prints `Point each domain's A record at `. The row turns green by itself once the record resolves. For **No certificate** behind Cloudflare, check the certificate settings for the name in Cloudflare, or set its record to DNS only so the server makes its own certificate. ## another operation is already running ### Symptom A refused deploy: ```text another operation is already running for shop: the daily backup, run Qd8xT2LmA0c (ox logs shop --run Qd8xT2LmA0c --follow); wait for it to finish, then retry ``` ### Cause ox runs one thing at a time per project: a deploy, a rollback, a backup, a restore or a delete. A push, the daily backup or another person may have started one a moment before you. ### Fix Wait for it to end, then run your command again. The message names the run that holds the project, and the `ox logs` command in it prints that run's log until it ends. `ox runs shop` lists the project's other runs. ## interrupted by ox, not the app ### Symptom An interrupted deploy: ```text deploy interrupted: interrupted by ox, not the app: the ox agent stopped mid-run (an ox update or a reboot); ox is deploying it again as run Qd8xT2LmA0c (ox logs shop --run Qd8xT2LmA0c --follow) ``` ### Cause ox itself cut the run short: the ox agent on your server restarted (an update or a reboot), or the server did not reconnect within an hour. Your app did nothing wrong, and a step the deploy did not finish leaves the previous release serving. ### Fix Usually nothing: ox deploys an interrupted deploy again once, at once or when the server is back, and the message names that run. Follow it with the `ox logs` command it gives. When the message ends with `deploy again` or `run it again`, ox did not retry (a rollback, a retry that was interrupted too, or a newer run replaced it), so run your command again. ## Disk full: a deploy needs at least 2 GB ### Symptom A deploy stopped by preflight: ```text disk: only 1400 MB free on /srv/ox; a deploy needs at least 2 GB fix: delete old projects or grow the disk ``` `sudo ox doctor` on the server says `only 4% of the disk is free (…); deploys and backups fail soon` once less than 5% is free. ### Cause The server's disk is close to full: releases, backups, logs and your app's own files share it. ox deletes older releases as the disk runs low, and keeps one project's backups under 20% of the disk, but it always keeps the newest release you can roll back to and the newest backup of each service. ### Fix Delete projects you no longer need, or resize the disk at your provider. Backups move off the server when you add an S3 bucket, so fewer stay on the disk. See [how many backups ox keeps](https://console.t51.sorv.dev/docs/operate/backups#retention). ## Not enough memory: killed by the out-of-memory killer ### Symptom Preflight refuses the deploy, a build step dies, or an app, worker or service is stopped as it starts: Seven memory failures: ```text capacity: the host would need 2304 MB but has 1987 MB: … capacity: previews on this server share 3968 MB, half of it, so production keeps the rest; they would need 4096 MB: … killed by the out-of-memory killer (the build needs more memory than the project or host allows) killed by the out-of-memory killer: the whole server ran out of memory while it ran (the apps and services running on it held the rest) killed by the out-of-memory killer as it started: it needs more memory than its limit allows killed by the out-of-memory killer as it started: the previews on this server ran out of the memory they share killed by the out-of-memory killer as it started: the whole server ran out of memory (what else runs on it held the rest) ``` ### Cause The apps, their services and their staging copies add up to more memory than the server has (a stopped project gives back its `[limits] memory`, but its services stay reserved), the previews add up to more than their half of it, or one build step needs more than its limit. A build step gets at most three quarters of the server's memory, or the project's `[limits] memory` when that is lower. The last line means the step stayed under its own limit, but the whole server ran out: the apps and services running on it held the rest. Build and migrate steps run one at a time on a server, so another build is not the cause: a step that waits names the one it waits for (`waiting for the build of shop (run ...) to finish`), and ox stops a step whose run was cancelled or whose agent restarted. A `[models]` download is not a build step and may run beside one. An app, worker or service has no memory limit unless `ox.toml` sets one (its own `memory`, or the project's `[limits] memory`; a preview also shares half the server with the other previews), so one stopped as it starts went past that limit, a preview's went past the previews' share, or the whole server ran out. ### Fix For the capacity check, the fix line says it: `stop a project whose [limits] memory is listed, delete one you do not need, drop a service or make it shared, lower a [limits] memory, or use a bigger server`, or for previews `stop a preview whose [limits] memory is listed, remove one you do not need, lower this preview's [limits] memory, or use a bigger server`. `ox stop` frees a project's `[limits] memory` at once, but not its services' memory: delete the project, or make a service shared, to free that too. A refused `ox start` gives the same message and leaves the project stopped: stop a project whose `[limits] memory` is listed or delete one you do not need first, or lower the project's `[limits] memory` and deploy. A refused `ox rollback` to a release with a larger `[limits] memory` says the same and changes nothing. For a build step, raise `memory` in `ox.toml` if it sets one, or use a bigger server. For an app, worker or service stopped as it starts, raise its `memory` in `ox.toml` (or `[limits] memory` if the project sets one), or use a bigger server. When the whole server ran out, use a bigger server, or stop a project, staging or preview you do not need. Fewer [previews at once](https://console.t51.sorv.dev/docs/operate/previews#limits) also frees memory. A preview that ran out says so: all previews share half the server's memory, so production keeps the rest, and the fix is to remove another preview. ## A Node build stops: ran out of memory at its heap limit ### Symptom A build step ends with exit code 134 and the build output shows `JavaScript heap out of memory`. The failure names the heap Node reached, read from its last trace: The heap failure: ```text Node ran out of memory at its heap limit, about 2082 MB ``` ### Cause Node limits its heap: to about half the memory it may use, and never past about 2.2 GB. A build step that may use more than 5 GB gets half its memory from ox instead; a running app keeps Node's own limit. A large front-end build (a big Vite or webpack bundle in a monorepo) can need more. ### Fix Set the variable `NODE_OPTIONS` to the size the failure suggests, half again above the heap it reached and at least `--max-old-space-size=4096` (megabytes), on the Variables screen. When the variable already carries other flags, add this one beside them: the variable replaces ox's. When the build command sets `--max-old-space-size` itself, that flag wins over the variable, so the failure says to raise it in the command instead. The server needs that much free memory while the build runs, and a `[limits] memory` cap must be higher than the number you set. ## A compiled module needs a system library this server does not have ### Symptom A build, a migration or the app's start fails, and the output shows `cannot open shared object file`. The failure names the library: The library failure: ```text a compiled module needs the system library libcairo.so.2, which this server does not have ``` ### Cause A package with compiled code (a Node addon, a Python extension) loads a library from the operating system when it starts. A Docker image often has it already; a fresh Ubuntu server has only what ox and your `packages` install. ### Fix Add the Ubuntu package that ships the file to `packages` in `ox.toml` (for `libcairo.so.2`, `packages = ["libcairo2"]`). [packages.ubuntu.com](https://packages.ubuntu.com) finds the package by the file's name. When this Ubuntu ships only another version of the library (Ubuntu 26.04 has `libxml2.so.16`, not `libxml2.so.2`), the module needs a build made for it, or a version of the module that does not use it. ## The app keeps crashing and restarting ### Symptom The deploy passed, but later an alert says: The crash alert: ```text web of shop on my-server keeps crashing: it restarted 5 times in the last 10 minutes. The Logs tab shows why. ``` When the server stopped it for running out of memory, the alert says that instead, with the fix: The crash alert for memory: ```text web of shop on my-server keeps crashing: it restarted 5 times in the last 10 minutes, and it ran out of memory: it needs more than its limit allows. Raise its memory in ox.toml (the memory of its own entry, or [limits] memory if the project sets one), or use a bigger server. ``` ### Cause The process starts, then exits, and the server restarts it again and again: after 2 seconds, then waiting longer each time up to once a minute, so a broken process never floods the server. It never gives up, so it comes back by itself once what it waits for (a database, a network) is back. A missing variable, a lost database connection, a full disk or too little memory are common reasons. A deploy whose service or HTTP worker does not come up puts back the version that ran before, or stops it, so a failed deploy never leaves one restarting. ### Fix Open the **Logs** tab, pick **Errors**, and read the lines just before each restart, or run `ox logs shop`. See [Logs](https://console.t51.sorv.dev/docs/operate/logs). If the last deploy caused it, [roll back](https://console.t51.sorv.dev/docs/operate/rollback) while you fix it. An alert that says it ran out of memory carries its fix; [Not enough memory](https://console.t51.sorv.dev/docs/troubleshooting#out-of-memory) says more. ## Server offline: the server has not connected yet ### Symptom A deploy or a command is refused with one of these: An unreachable server: ```text server my-server has not connected yet server my-server went offline 12 minutes ago ``` ### Cause The ox agent on the server is not talking to ox. The server may be off or off the network, or the agent may have stopped. Your sites keep running unless the server itself is down, but ox cannot deploy to it until it reconnects. ### Fix Check that the server is on at your provider. Then, on the server: Check the agent: ```sh sudo systemctl status ox-agent sudo journalctl -u ox-agent -n 50 ``` `sudo systemctl restart ox-agent` starts it again. If the server was rebuilt, add it again from the dashboard. ## Install refused: this server can't take ox yet ### Symptom The Add server command stops before it changes anything: A server that cannot take ox: ```text this server can't take ox yet; nothing was changed: - port 80 is in use by nginx; ox's proxy (Caddy) needs it: stop that program, or use a fresh server - the server has 512 MB of RAM and ox needs at least 1 GB: resize it - the disk has 6.2 GB free on / and ox needs at least 10 GB: free space or resize the disk ``` ### Cause ox needs ports 80 and 443 free for Caddy, at least 1 GB of memory and at least 10 GB of free disk. ### Fix Do what each line says, then press Add server for a new command and run it: each command works once. `sudo ox install --check` runs only these checks. A fresh Ubuntu LTS server is the easy way. ## repository access removed: GitHub gives ox no access ### Symptom A deploy that cannot read the repo: ```text repository access removed: GitHub gives ox no access to acme/shop (Not Found); connect GitHub again and pick the repository ``` ### Cause The ox GitHub App was removed from the repository or the account on GitHub, or the repository was renamed or moved. ### Fix Connect GitHub again from the dashboard and pick the repository, then deploy. ## not signed in; run ox login ### Symptom CLI sign-in errors: ```text not signed in; run ox login the plane refused the token (expired or revoked); sign in again with ox login ``` ### Cause This computer has no token yet, or its token expired or was revoked in the dashboard. The command exits with status 3. ### Fix Run `ox login` and approve the code in the browser. A `could not reach …` message with status 4 is the network, not the sign-in. See [exit codes](https://console.t51.sorv.dev/docs/cli#exit-codes). Last updated 2026-10-08. The page as HTML: https://console.t51.sorv.dev/docs/troubleshooting