# Updating a host with several Palaver servers

- prerequisite: host set up per [INSTALL.md](INSTALL.md)
- one `/opt/palaver/bin/palaverd` for all servers; further servers: old version until restarted, restart right
  after the main server

Placeholders in the commands:

| Placeholder | Replace with |
|---|---|
| `0.3.1` | new Palaver release |
| `https://git.internal.incudex.de/lasse-tech/palaver/releases/download` | release download location; Palaver's internal Forgejo, reachable from your own machine only |
| `palaver.social` | the main server's domain; as `palaver.social:` the host for `scp`; in `/etc/letsencrypt/live/palaver.social/`: the certificate's name |
| `guild` | a further server's name; a `-` in the name becomes `_` in the database name (e.g. `palaver_my_guild`); commands marked "for each further server": once for each |
| `guild.palaver.social` | that further server's domain |
| `8091` | that further server's port |
| `2026-10-10` | today's date (backup file names) |
| `1.13.7` | LiveKit version from the release notes |
| `vi` | editor of your choice |

## Contents

1. [Release notes](#1-release-notes)
2. [Download Palaver](#2-download-palaver)
3. [Back up the main server](#3-back-up-the-main-server)
4. [Back up the further servers](#4-back-up-the-further-servers)
5. [palaverd](#5-palaverd)
6. [Web client](#6-web-client)
7. [nginx](#7-nginx)
8. [Restart](#8-restart)
9. [LiveKit](#9-livekit)
10. [Check](#10-check)
- [Back to the previous version](#back-to-the-previous-version)
- [Once: hosts set up with palaver-host](#once-hosts-set-up-with-palaver-host)
- [Once: the machine palaver-ops ran on](#once-the-machine-palaver-ops-ran-on)

## 1. Release notes

- read: release notes of every version after the installed one
- apply: changes listed there (configuration files, units, nginx, LiveKit version)

Show the installed version.

```sh
/opt/palaver/bin/palaverd version
```

## 2. Download Palaver

On your own machine: download the server and the web client.

```sh
curl -fLO -H "Authorization: token $FORGEJO_TOKEN" https://git.internal.incudex.de/lasse-tech/palaver/releases/download/v0.3.1/palaverd-0.3.1-linux-amd64
curl -fLO -H "Authorization: token $FORGEJO_TOKEN" https://git.internal.incudex.de/lasse-tech/palaver/releases/download/v0.3.1/palaver-web-0.3.1.tar.gz
```

Copy both into your home directory on the host.

```sh
scp palaverd-0.3.1-linux-amd64 palaver-web-0.3.1.tar.gz palaver.social:
```

- from here on: on the host, in your home directory

## 3. Back up the main server

Dump the database.

```sh
sudo -u palaver pg_dump -Fc palaver > palaver-2026-10-10.dump
```

Save the configuration.

```sh
sudo tar -czf palaver-etc-2026-10-10.tar.gz -C /opt/palaver etc
```

Keep the current palaverd for the way back.

```sh
sudo cp /opt/palaver/bin/palaverd /opt/palaver/bin/palaverd.old
```

## 4. Back up the further servers

Show the further servers.

```sh
ls /opt/palaver/instances/
```

For each further server: dump its database.

```sh
sudo -u palaver-guild pg_dump -Fc palaver_guild > palaver-guild-2026-10-10.dump
```

Save their configuration.

```sh
sudo tar -czf palaver-instances-2026-10-10.tar.gz -C /opt/palaver instances
```

Check the dumps; expected: one per server, none empty.

```sh
ls -l palaver-*.dump
```

## 5. palaverd

Install the new palaverd.

```sh
sudo install -m 755 palaverd-0.3.1-linux-amd64 /opt/palaver/bin/palaverd
```

## 6. Web client

Unpack the new web client next to the old one.

```sh
sudo install -d -m 755 /opt/palaver/web.new
sudo tar -xzf palaver-web-0.3.1.tar.gz --no-same-owner -C /opt/palaver/web.new
sudo chmod -R u=rwX,go=rX /opt/palaver/web.new
```

Swap them.

```sh
sudo mv /opt/palaver/web /opt/palaver/web.old
sudo mv /opt/palaver/web.new /opt/palaver/web
```

## 7. nginx

Only if the release notes change the web snippet or the site. Open the main server's snippet in an editor.

```sh
sudo vi /etc/nginx/snippets/palaver-web.conf
```

Open the main server's site in an editor.

```sh
sudo vi /etc/nginx/sites-available/palaver
```

For each further server: open its snippet in an editor; keep `wss://guild.palaver.social`.

```sh
sudo vi /etc/nginx/snippets/palaver-web-guild.conf
```

For each further server: open its site in an editor; keep its port, logs and snippet.

```sh
sudo vi /etc/nginx/sites-available/palaver-guild
```

Change:

- what the release notes say, in every file above

Check and load them.

```sh
sudo nginx -t && sudo systemctl reload nginx
```

## 8. Restart

Restart the main server.

```sh
sudo systemctl restart palaverd
```

Restart every further server right after it.

```sh
sudo systemctl restart 'palaverd@*'
```

Show their state; expected: `active` each.

```sh
systemctl list-units --all 'palaverd*'
```

## 9. LiveKit

Only for a newer LiveKit in the release notes. Show the installed version.

```sh
/opt/palaver/bin/livekit-server --version
```

Download it from GitHub.

```sh
curl -fLO https://github.com/livekit/livekit/releases/download/v1.13.7/livekit_1.13.7_linux_amd64.tar.gz
```

Unpack and install the binary.

```sh
tar -xzf livekit_1.13.7_linux_amd64.tar.gz livekit-server
sudo install -m 755 livekit-server /opt/palaver/bin/livekit-server
```

Restart LiveKit; calls on every server drop.

```sh
sudo systemctl restart livekit
```

## 10. Check

Run doctor for the main server; expected: no findings.

```sh
sudo -u palaver /opt/palaver/bin/palaverd -env /opt/palaver/etc/palaverd.env doctor
```

For each further server: run doctor; expected: no findings.

```sh
sudo -u palaver-guild /opt/palaver/bin/palaverd -env /opt/palaver/instances/guild/palaverd.env doctor
```

Show the version every server runs.

```sh
/opt/palaver/bin/palaverd version
```

Open the main server's and one further server's web client and log in.

**Destructive:** remove the previous web client (`/opt/palaver/web.old`, release files only).

```sh
sudo rm -r /opt/palaver/web.old
```

**Destructive:** remove the previous palaverd (`/opt/palaver/bin/palaverd.old`).

```sh
sudo rm /opt/palaver/bin/palaverd.old
```

Remove the downloaded files.

```sh
rm palaverd-0.3.1-linux-amd64 palaver-web-0.3.1.tar.gz
```

## Back to the previous version

- for: new version does not work
- database changes are not undone; if needed: [Restore a dump](#restore-a-dump)

Put the previous palaverd back.

```sh
sudo cp /opt/palaver/bin/palaverd.old /opt/palaver/bin/palaverd
```

Put the previous web client back.

```sh
sudo mv /opt/palaver/web /opt/palaver/web.failed
sudo mv /opt/palaver/web.old /opt/palaver/web
```

Restart the main server.

```sh
sudo systemctl restart palaverd
```

Restart every further server on the previous palaverd.

```sh
sudo systemctl restart 'palaverd@*'
```

### Restore a dump

Main server: stop it.

```sh
sudo systemctl stop palaverd
```

**Destructive:** delete the main server's database (`palaver`, all accounts and messages since the dump).

```sh
sudo -u postgres dropdb palaver
```

Create it again, load the dump from [step 3](#3-back-up-the-main-server) and start it.

```sh
sudo -u postgres createdb -O palaver palaver
sudo -u palaver pg_restore -d palaver palaver-2026-10-10.dump
sudo systemctl start palaverd
```

For each further server: stop it.

```sh
sudo systemctl stop palaverd@guild
```

**Destructive:** delete its database (`palaver_guild`, all accounts and messages since the dump).

```sh
sudo -u postgres dropdb palaver_guild
```

Create it again, load the dump from [step 4](#4-back-up-the-further-servers) and start it.

```sh
sudo -u postgres createdb -O palaver-guild palaver_guild
sudo -u palaver-guild pg_restore -d palaver_guild palaver-guild-2026-10-10.dump
sudo systemctl start palaverd@guild
```

## Once: hosts set up with palaver-host

- for: hosts set up with `palaver-host` and the open bundle's `install.sh`
- once, after [step 10](#10-check)
- leftovers: tools, saved templates, logs, nginx sites with the web client inline

### Main server

- palaver.social: landing page at `/`, snippet and site as in palaver-social's `docs/server.md`
- Ubuntu 26.04: sudo-rs without `sudoedit`, edit as root

Show the certificate names.

```sh
sudo ls /etc/letsencrypt/live/
```

Keep the old nginx site outside nginx's directories.

```sh
sudo cp /etc/nginx/sites-available/palaver /root/nginx-palaver-2026-10-10.conf
```

Open `/etc/nginx/snippets/palaver-web.conf` in an editor.

```sh
sudo vi /etc/nginx/snippets/palaver-web.conf
```

Content:

```nginx
root /opt/palaver/web;
index index.html;

location / {
    try_files $uri /index.html;
    add_header Cache-Control "no-cache" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self'; worker-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:; media-src 'self' blob: mediastream:; font-src 'self'; connect-src 'self' wss://palaver.social; object-src 'none'; base-uri 'none'; form-action 'self'; frame-ancestors 'none'" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer" always;
    add_header Permissions-Policy "camera=(), geolocation=(), payment=()" always;
    add_header Strict-Transport-Security "max-age=31536000" always;
}

location /assets/ {
    try_files $uri =404;
    add_header Cache-Control "public, max-age=31536000, immutable" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self'; worker-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:; media-src 'self' blob: mediastream:; font-src 'self'; connect-src 'self' wss://palaver.social; object-src 'none'; base-uri 'none'; form-action 'self'; frame-ancestors 'none'" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Strict-Transport-Security "max-age=31536000" always;
}
```

Open `/etc/nginx/sites-available/palaver` in an editor.

```sh
sudo vi /etc/nginx/sites-available/palaver
```

Content; certificate name from above in both `ssl_certificate` lines:

```nginx
server {
    listen 80;
    listen [::]:80;
    server_name palaver.social;

    access_log /var/log/nginx/palaver/access.log;
    error_log /var/log/nginx/palaver/error.log;

    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }
    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name palaver.social;

    ssl_certificate /etc/letsencrypt/live/palaver.social/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/palaver.social/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    access_log /var/log/nginx/palaver/access.log;
    error_log /var/log/nginx/palaver/error.log;

    client_max_body_size 64k;

    # server push, one WebSocket per client
    location = /api/events {
        proxy_pass http://127.0.0.1:8090;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 1h;
        proxy_send_timeout 1h;
    }

    # profile pictures, server icon: 5 MiB
    location ~ ^/api/(me/avatar|server/icon)$ {
        client_max_body_size 5m;
        proxy_pass http://127.0.0.1:8090;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # attachments: 25 MiB
    location ~ ^/api/channels/[0-9]+/attachments$ {
        client_max_body_size 25m;
        proxy_pass http://127.0.0.1:8090;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /api/ {
        proxy_pass http://127.0.0.1:8090;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # LiveKit signalling; the trailing slash strips /livekit
    location /livekit/ {
        proxy_pass http://127.0.0.1:7880/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_read_timeout 1h;
        proxy_send_timeout 1h;
    }

    include snippets/palaver-web.conf;
}
```

Check and load it.

```sh
sudo nginx -t && sudo systemctl reload nginx
```

**Destructive:** remove the helper script (`/usr/local/sbin/palaverctl`).

```sh
sudo rm /usr/local/sbin/palaverctl
```

**Destructive:** remove the install log (`/var/log/palaver-install.log`).

```sh
sudo rm /var/log/palaver-install.log
```

**Destructive:** remove install.sh's certificate note (`/opt/palaver/etc/cert`, one line).

```sh
sudo cat /opt/palaver/etc/cert
sudo rm /opt/palaver/etc/cert
```

### Every further server

For each further server: show its domain, port and certificate name.

```sh
sudo grep -E '^PALAVER_(PUBLIC_URL|LISTEN)=' /opt/palaver/instances/guild/palaverd.env
sudo cat /opt/palaver/instances/guild/cert
```

Keep the old nginx site outside nginx's directories.

```sh
sudo cp /etc/nginx/sites-available/palaver-guild /root/nginx-palaver-guild-2026-10-10.conf
```

Open `/etc/nginx/snippets/palaver-web-guild.conf` in an editor.

```sh
sudo vi /etc/nginx/snippets/palaver-web-guild.conf
```

Content:

```nginx
root /opt/palaver/web;
index index.html;

location / {
    try_files $uri /index.html;
    add_header Cache-Control "no-cache" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self'; worker-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:; media-src 'self' blob: mediastream:; font-src 'self'; connect-src 'self' wss://guild.palaver.social; object-src 'none'; base-uri 'none'; form-action 'self'; frame-ancestors 'none'" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer" always;
    add_header Permissions-Policy "camera=(), geolocation=(), payment=()" always;
    add_header Strict-Transport-Security "max-age=31536000" always;
}

location /assets/ {
    try_files $uri =404;
    add_header Cache-Control "public, max-age=31536000, immutable" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self'; worker-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:; media-src 'self' blob: mediastream:; font-src 'self'; connect-src 'self' wss://guild.palaver.social; object-src 'none'; base-uri 'none'; form-action 'self'; frame-ancestors 'none'" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Strict-Transport-Security "max-age=31536000" always;
}
```

Open `/etc/nginx/sites-available/palaver-guild` in an editor.

```sh
sudo vi /etc/nginx/sites-available/palaver-guild
```

Content; its port in every `proxy_pass` to palaverd, its certificate name in both `ssl_certificate` lines:

```nginx
server {
    listen 80;
    listen [::]:80;
    server_name guild.palaver.social;

    access_log /var/log/nginx/palaver/guild.access.log;
    error_log /var/log/nginx/palaver/guild.error.log;

    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }
    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name guild.palaver.social;

    ssl_certificate /etc/letsencrypt/live/palaver.social/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/palaver.social/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    access_log /var/log/nginx/palaver/guild.access.log;
    error_log /var/log/nginx/palaver/guild.error.log;

    client_max_body_size 64k;

    # server push, one WebSocket per client
    location = /api/events {
        proxy_pass http://127.0.0.1:8091;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 1h;
        proxy_send_timeout 1h;
    }

    # profile pictures, server icon: 5 MiB
    location ~ ^/api/(me/avatar|server/icon)$ {
        client_max_body_size 5m;
        proxy_pass http://127.0.0.1:8091;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # attachments: 25 MiB
    location ~ ^/api/channels/[0-9]+/attachments$ {
        client_max_body_size 25m;
        proxy_pass http://127.0.0.1:8091;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /api/ {
        proxy_pass http://127.0.0.1:8091;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # LiveKit signalling, shared; the trailing slash strips /livekit
    location /livekit/ {
        proxy_pass http://127.0.0.1:7880/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_read_timeout 1h;
        proxy_send_timeout 1h;
    }

    include snippets/palaver-web-guild.conf;
}
```

Check and load them.

```sh
sudo nginx -t && sudo systemctl reload nginx
```

Open `/opt/palaver/instances/guild/palaverd.env` in an editor.

```sh
sudo vi /opt/palaver/instances/guild/palaverd.env
```

Change:

- remove palaver-host's comment lines (`#`)
- add at the end, for doctor's hints:

```ini
PALAVER_CTL="sudo -u palaver-guild /opt/palaver/bin/palaverd -env /opt/palaver/instances/guild/palaverd.env"
PALAVER_UNIT=palaverd@guild
PALAVER_ENV_FILE=/opt/palaver/instances/guild/palaverd.env
```

Check it.

```sh
sudo -u palaver-guild /opt/palaver/bin/palaverd -env /opt/palaver/instances/guild/palaverd.env doctor
```

**Destructive:** remove palaver-host's certificate note (`/opt/palaver/instances/guild/cert`, one line).

```sh
sudo rm /opt/palaver/instances/guild/cert
```

### The host

Unit template stays. Show it; expected: `User=palaver-%i`, `EnvironmentFile=/opt/palaver/instances/%i/palaverd.env`,
`ExecStart=/opt/palaver/bin/palaverd serve`, `LogNamespace=palaver`.

```sh
grep -v '^#' /etc/systemd/system/palaverd@.service
```

Show the enabled sites.

```sh
ls /etc/nginx/sites-enabled/
```

With `palaver-catch-all`: open it in an editor.

```sh
sudo vi /etc/nginx/sites-available/palaver-catch-all
```

Content:

```nginx
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    location /.well-known/acme-challenge/ { root /var/www/html; }
    location / { return 301 https://palaver.social/; }
}

server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    server_name _;
    ssl_certificate /etc/letsencrypt/live/palaver.social/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/palaver.social/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    return 301 https://palaver.social/;
}
```

Check and load it.

```sh
sudo nginx -t && sudo systemctl reload nginx
```

**Destructive:** remove the tool (`/usr/local/sbin/palaver-host`).

```sh
sudo rm /usr/local/sbin/palaver-host
```

**Destructive:** remove the saved nginx templates (`/opt/palaver/hosting/templates/`, two files).

```sh
sudo rm -r /opt/palaver/hosting
```

**Destructive:** remove palaver-host's log (`/var/log/palaver-hosting.log`).

```sh
sudo rm /var/log/palaver-hosting.log
```

**Destructive:** remove unpacked bundles and tarballs in your home directory (`palaver-server-*`,
`palaver-hosting-*`); list, then remove with confirmation.

```sh
ls -d ~/palaver-server-* ~/palaver-hosting-*
rm -ri ~/palaver-server-* ~/palaver-hosting-*
```

## Once: the machine palaver-ops ran on

**Destructive:** remove `palaver-ops` (`~/.local/bin/palaver-ops`).

```sh
rm ~/.local/bin/palaver-ops
```

**Destructive:** remove its list of servers (`~/.config/palaver-ops/servers.json`).

```sh
cat ~/.config/palaver-ops/servers.json
rm -r ~/.config/palaver-ops
```