# Further Palaver servers on one host

- **main server** plus any number of **further servers** on one host
- shared with the main server: programs (`/opt/palaver/bin`), web client (`/opt/palaver/web`), LiveKit
- updates: [UPDATE.md](UPDATE.md)

For each further server (example: `guild`):

| Part | Value |
|---|---|
| name | `guild` |
| domain | `guild.palaver.social` |
| system user, database role | `palaver-guild` |
| database | `palaver_guild` |
| configuration | `/opt/palaver/instances/guild/palaverd.env` |
| systemd unit | `palaverd@guild` |
| port | `8091` ([step 3](#3-port)) |
| LiveKit room prefix | `guild-` (`PALAVER_LIVEKIT_ROOM_PREFIX`) |
| LiveKit key | own key in `/opt/palaver/etc/livekit.yaml` |
| nginx | site `palaver-guild`, snippet `palaver-web-guild.conf` |
| nginx logs | `/var/log/nginx/palaver/guild.access.log`, `guild.error.log` |

Placeholders in the commands:

| Placeholder | Replace with |
|---|---|
| `guild` | the server's name ([step 2](#2-name-and-domain)), in every name of the table above; a `-` in the name becomes `_` in the database name (e.g. `palaver_my_guild`) |
| `guild.palaver.social` | the server's domain |
| `8091` | the server's port ([step 3](#3-port)) |
| `palaver.social` | in `/etc/letsencrypt/live/palaver.social/`: the certificate's name (own certificate: the server's domain); elsewhere: the main server's domain |
| `you@example.org` | e-mail address for Let's Encrypt; own certificate only |
| `<livekit-key>`, `<livekit-secret>` | output of [step 7](#7-livekit-key), "Generate the LiveKit API …" |
| `'Guild'`, `'Guild subtitle'` | the server's name and subtitle in the app ([step 12](#12-first-admin)); no subtitle: `''` |
| `2026-10-10` | today's date (backup file names) |
| `vi` | editor of your choice |

## Contents

1. [Requirements](#1-requirements)
2. [Name and domain](#2-name-and-domain)
3. [Port](#3-port)
4. [Unit template](#4-unit-template)
5. [User and directory](#5-user-and-directory)
6. [PostgreSQL](#6-postgresql)
7. [LiveKit key](#7-livekit-key)
8. [palaverd](#8-palaverd)
9. [Certificate](#9-certificate)
10. [nginx](#10-nginx)
11. [Start](#11-start)
12. [First admin](#12-first-admin)
13. [Check](#13-check)
- [List servers](#list-servers)
- [Backup](#backup)
- [Remove a server](#remove-a-server)
- [Optional: unknown names to the main server](#optional-unknown-names-to-the-main-server)
- [Optional: wildcard certificate](#optional-wildcard-certificate)
- [Files](#files)
- [Troubleshooting](#troubleshooting)

## 1. Requirements

- main server per [Palaver's `INSTALL.md`](https://palaver.social/install-server.html) (`palaverd.service`, user and database `palaver`, `/opt/palaver/etc`,
  LiveKit, snippet `palaver-web.conf`), port 8090
- shell access with `sudo`
- no further ports: 443 and the main server's LiveKit ports

Check that the main server is there.

```sh
sudo ls /opt/palaver/etc/palaverd.env /opt/palaver/etc/livekit.yaml
```

## 2. Name and domain

- name: `a-z`, `0-9`, `-` (not first or last), at most 24 characters
- reserved: `palaver`, `www`, `account`, `konto`, `support`, `api`, `admin`, `dogan`, `livekit`, `mail`, `status`
- domain: host name, lower case
- certificate: the wildcard certificate `palaver.social`, or an own one ([step 9](#9-certificate))
- name and subtitle in the app: up to 64 characters each
- an order from account.palaver.social shows these values ready to copy

Check that the name is free; expected: both commands fail.

```sh
ls -d /opt/palaver/instances/guild
id palaver-guild
```

## 3. Port

- one localhost port per server; main server: 8090
- new server: highest port in use + 1

Show the ports in use; take the highest + 1.

```sh
sudo sh -c 'grep -H "^PALAVER_LISTEN=" /opt/palaver/etc/palaverd.env /opt/palaver/instances/*/palaverd.env 2>/dev/null'
```

## 4. Unit template

Once per host; skip if `/etc/systemd/system/palaverd@.service` exists.

```sh
ls /etc/systemd/system/palaverd@.service
```

Open `/etc/systemd/system/palaverd@.service` in an editor.

```sh
sudo vi /etc/systemd/system/palaverd@.service
```

Content, unchanged:

```ini
[Unit]
Description=Palaver app server %i
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
User=palaver-%i
Group=palaver-%i
EnvironmentFile=/opt/palaver/instances/%i/palaverd.env
ExecStart=/opt/palaver/bin/palaverd serve
Restart=on-failure
RestartSec=2
LogNamespace=palaver

NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
LockPersonality=yes
MemoryDenyWriteExecute=yes

[Install]
WantedBy=multi-user.target
```

Load it.

```sh
sudo systemctl daemon-reload
```

## 5. User and directory

Create the directory for all further servers.

```sh
sudo install -d -o root -g root -m 755 /opt/palaver/instances
```

Create the system user.

```sh
sudo useradd --system --home-dir /opt/palaver/instances/guild --no-create-home --shell /usr/sbin/nologin palaver-guild
```

Create the configuration directory.

```sh
sudo install -d -o palaver-guild -g palaver-guild -m 700 /opt/palaver/instances/guild
```

## 6. PostgreSQL

Create the database role and the database.

```sh
sudo -u postgres createuser palaver-guild
sudo -u postgres createdb -O palaver-guild palaver_guild
```

## 7. LiveKit key

Generate the LiveKit API key (`<livekit-key>`).

```sh
openssl rand -hex 8 | sed 's/^/API/'
```

Generate the LiveKit API secret (`<livekit-secret>`).

```sh
openssl rand -base64 36 | tr -d '/+='
```

Open `/opt/palaver/etc/livekit.yaml` in an editor.

```sh
sudo vi /opt/palaver/etc/livekit.yaml
```

Add a line under `keys:`, indented by two spaces:

```yaml
  <livekit-key>: <livekit-secret>
```

Check the keys; expected: one line per server.

```sh
sudo sed -n '/^keys:/,/^[a-z]/p' /opt/palaver/etc/livekit.yaml
```

Restart LiveKit; calls on every server drop.

```sh
sudo systemctl restart livekit
```

- LiveKit keys are not limited to rooms: one server's configuration gives access to all servers' calls

## 8. palaverd

Open `/opt/palaver/instances/guild/palaverd.env` in an editor.

```sh
sudo vi /opt/palaver/instances/guild/palaverd.env
```

Content:

```ini
PALAVER_DATABASE_URL=postgres:///palaver_guild?host=/var/run/postgresql
PALAVER_LISTEN=127.0.0.1:8091
PALAVER_PUBLIC_URL=https://guild.palaver.social
PALAVER_LIVEKIT_URL=wss://guild.palaver.social/livekit
PALAVER_LIVEKIT_API_URL=http://127.0.0.1:7880
PALAVER_LIVEKIT_API_KEY=<livekit-key>
PALAVER_LIVEKIT_API_SECRET=<livekit-secret>
PALAVER_LIVEKIT_ROOM_PREFIX=guild-
PALAVER_OPEN_REGISTRATION=false
PALAVER_CTL="sudo -u palaver-guild /opt/palaver/bin/palaverd -env /opt/palaver/instances/guild/palaverd.env"
PALAVER_UNIT=palaverd@guild
PALAVER_ENV_FILE=/opt/palaver/instances/guild/palaverd.env
```

Make it readable by `palaver-guild` only.

```sh
sudo chown palaver-guild:palaver-guild /opt/palaver/instances/guild/palaverd.env
sudo chmod 600 /opt/palaver/instances/guild/palaverd.env
```

- other settings: [Palaver's `INSTALL.md`, Configuration reference](https://palaver.social/install-server.html#configuration-reference)

## 9. Certificate

### Under a wildcard certificate

Check that it covers the domain; expected: `DNS:*.palaver.social`.

```sh
sudo openssl x509 -noout -ext subjectAltName -in /etc/letsencrypt/live/palaver.social/fullchain.pem
```

Next: [step 10](#10-nginx).

### Own certificate

Open `/etc/nginx/sites-available/palaver-guild` in an editor.

```sh
sudo vi /etc/nginx/sites-available/palaver-guild
```

Content:

```nginx
server {
    listen 80;
    listen [::]:80;
    server_name guild.palaver.social;
    location /.well-known/acme-challenge/ { root /var/www/html; }
    location / { return 404; }
}
```

Enable the site, check and load it.

```sh
sudo ln -sf /etc/nginx/sites-available/palaver-guild /etc/nginx/sites-enabled/palaver-guild
sudo nginx -t && sudo systemctl reload nginx
```

Get the certificate.

```sh
sudo certbot certonly --webroot -w /var/www/html -d guild.palaver.social -m you@example.org \
    --agree-tos --no-eff-email --non-interactive \
    --deploy-hook "systemctl reload nginx"
```

- step 10's site: `/etc/letsencrypt/live/guild.palaver.social/` instead of `/etc/letsencrypt/live/palaver.social/`

## 10. nginx

Open `/etc/nginx/snippets/palaver-web-guild.conf` in an editor.

```sh
sudo vi /etc/nginx/snippets/palaver-web-guild.conf
```

Content:

```nginx
root /opt/palaver/web;
index index.html;

location / {
    try_files $uri /index.html;
    add_header Cache-Control "no-cache" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self'; worker-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:; media-src 'self' blob: mediastream:; font-src 'self'; connect-src 'self' wss://guild.palaver.social; object-src 'none'; base-uri 'none'; form-action 'self'; frame-ancestors 'none'" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Referrer-Policy "no-referrer" always;
    add_header Permissions-Policy "camera=(), geolocation=(), payment=()" always;
    add_header Strict-Transport-Security "max-age=31536000" always;
}

location /assets/ {
    try_files $uri =404;
    add_header Cache-Control "public, max-age=31536000, immutable" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self'; worker-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:; media-src 'self' blob: mediastream:; font-src 'self'; connect-src 'self' wss://guild.palaver.social; object-src 'none'; base-uri 'none'; form-action 'self'; frame-ancestors 'none'" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Strict-Transport-Security "max-age=31536000" always;
}
```

Open `/etc/nginx/sites-available/palaver-guild` in an editor.

```sh
sudo vi /etc/nginx/sites-available/palaver-guild
```

Content:

```nginx
server {
    listen 80;
    listen [::]:80;
    server_name guild.palaver.social;

    access_log /var/log/nginx/palaver/guild.access.log;
    error_log /var/log/nginx/palaver/guild.error.log;

    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }
    location / {
        return 301 https://$host$request_uri;
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name guild.palaver.social;

    ssl_certificate /etc/letsencrypt/live/palaver.social/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/palaver.social/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;

    access_log /var/log/nginx/palaver/guild.access.log;
    error_log /var/log/nginx/palaver/guild.error.log;

    client_max_body_size 64k;

    # server push, one WebSocket per client
    location = /api/events {
        proxy_pass http://127.0.0.1:8091;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_read_timeout 1h;
        proxy_send_timeout 1h;
    }

    # profile pictures, server icon: 5 MiB
    location ~ ^/api/(me/avatar|server/icon)$ {
        client_max_body_size 5m;
        proxy_pass http://127.0.0.1:8091;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # attachments: 25 MiB
    location ~ ^/api/channels/[0-9]+/attachments$ {
        client_max_body_size 25m;
        proxy_pass http://127.0.0.1:8091;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location /api/ {
        proxy_pass http://127.0.0.1:8091;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    # LiveKit signalling, shared; the trailing slash strips /livekit
    location /livekit/ {
        proxy_pass http://127.0.0.1:7880/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $host;
        proxy_read_timeout 1h;
        proxy_send_timeout 1h;
    }

    include snippets/palaver-web-guild.conf;
}
```

Enable, check and load it.

```sh
sudo ln -sf /etc/nginx/sites-available/palaver-guild /etc/nginx/sites-enabled/palaver-guild
sudo nginx -t && sudo systemctl reload nginx
```

## 11. Start

Start the server.

```sh
sudo systemctl enable --now palaverd@guild
```

Check that it answers; expected: JSON.

```sh
curl -fsS http://127.0.0.1:8091/api/registration
```

## 12. First admin

Set the server's name and subtitle.

```sh
sudo -u palaver-guild /opt/palaver/bin/palaverd -env /opt/palaver/instances/guild/palaverd.env profile -name 'Guild' -subtitle 'Guild subtitle'
```

Create an invite code for the first admin; single use, valid 7 days.

```sh
sudo -u palaver-guild /opt/palaver/bin/palaverd -env /opt/palaver/instances/guild/palaverd.env invite -role Admin -valid 168h
```

- register: Palaver app → server `https://guild.palaver.social` → **Register** → code
- first admin = server **owner**
- account registered without admin (code without `-role Admin`): give it the role

```sh
sudo -u palaver-guild /opt/palaver/bin/palaverd -env /opt/palaver/instances/guild/palaverd.env grant <username> Admin
```

## 13. Check

Run doctor; expected: `ok` on every line.

```sh
sudo -u palaver-guild /opt/palaver/bin/palaverd -env /opt/palaver/instances/guild/palaverd.env doctor
```

Check that the main server still answers.

```sh
sudo -u palaver /opt/palaver/bin/palaverd -env /opt/palaver/etc/palaverd.env doctor
```

## List servers

Names of the further servers.

```sh
ls /opt/palaver/instances/
```

Domain and port of every server.

```sh
sudo sh -c 'grep -H -E "^PALAVER_(PUBLIC_URL|LISTEN)=" /opt/palaver/etc/palaverd.env /opt/palaver/instances/*/palaverd.env'
```

Name and subtitle of one server.

```sh
sudo -u palaver-guild /opt/palaver/bin/palaverd -env /opt/palaver/instances/guild/palaverd.env profile
```

State of every server.

```sh
systemctl list-units --all 'palaverd*'
```

## Backup

- database: all data of the server
- `/opt/palaver/instances/guild`: configuration, LiveKit key

Dump the database.

```sh
sudo -u palaver-guild pg_dump -Fc palaver_guild > palaver-guild-2026-10-10.dump
```

Save the configuration.

```sh
sudo tar -czf palaver-guild-etc-2026-10-10.tar.gz -C /opt/palaver/instances guild
```

### Restore

Prerequisite: server with the same name and domain, steps 1–12.

Stop it.

```sh
sudo systemctl stop palaverd@guild
```

**Destructive:** delete the database of the new server (`palaver_guild`, all its accounts and messages).

```sh
sudo -u postgres dropdb palaver_guild
```

Create it again and load the dump.

```sh
sudo -u postgres createdb -O palaver-guild palaver_guild
sudo -u palaver-guild pg_restore -d palaver_guild palaver-guild-2026-10-10.dump
```

Start it.

```sh
sudo systemctl start palaverd@guild
```

## Remove a server

Show the domain and the LiveKit key (`<livekit-key>`).

```sh
sudo grep -E '^PALAVER_(PUBLIC_URL|LIVEKIT_API_KEY)=' /opt/palaver/instances/guild/palaverd.env
```

Show the site's certificate; own certificate: name `guild.palaver.social`.

```sh
grep ssl_certificate /etc/nginx/sites-available/palaver-guild
```

Back up first: [Backup](#backup).

Stop and disable the service.

```sh
sudo systemctl disable --now palaverd@guild
```

Remove the nginx site and snippet and reload nginx.

```sh
sudo rm /etc/nginx/sites-enabled/palaver-guild /etc/nginx/sites-available/palaver-guild /etc/nginx/snippets/palaver-web-guild.conf
sudo nginx -t && sudo systemctl reload nginx
```

**Destructive:** delete the database `palaver_guild` (all accounts, messages, attachments) and its role `palaver-guild`.

```sh
sudo -u postgres dropdb palaver_guild
sudo -u postgres dropuser palaver-guild
```

**Destructive:** delete the server's LiveKit key; open `/opt/palaver/etc/livekit.yaml` in an editor.

```sh
sudo vi /opt/palaver/etc/livekit.yaml
```

Remove:

- under `keys:` the line `  <livekit-key>: …`

Restart LiveKit; calls on the other servers drop briefly.

```sh
sudo systemctl restart livekit
```

**Destructive:** delete the configuration `/opt/palaver/instances/guild` and the user `palaver-guild`.

```sh
sudo rm -r /opt/palaver/instances/guild
sudo userdel palaver-guild
```

**Destructive:** delete the certificate; only an own one (`guild.palaver.social`), never the wildcard certificate.

```sh
sudo certbot delete --cert-name guild.palaver.social
```

**Destructive:** delete nginx's logs of the server (`/var/log/nginx/palaver/guild.*`).

```sh
sudo sh -c 'rm /var/log/nginx/palaver/guild.access.log* /var/log/nginx/palaver/guild.error.log*'
```

### After the last further server

Remove the unit template.

```sh
sudo rm /etc/systemd/system/palaverd@.service
sudo systemctl daemon-reload
```

Remove the empty directory.

```sh
sudo rmdir /opt/palaver/instances
```

- before uninstalling the main server ([Palaver's `INSTALL.md`, Uninstall](https://palaver.social/install-server.html#uninstall)): remove every further server

## Optional: unknown names to the main server

- for: wildcard DNS record and certificate for the main server's domain
- unknown names (typos): redirect to the main server instead of nginx's first site

Check that the certificate covers `*.palaver.social`.

```sh
sudo openssl x509 -noout -ext subjectAltName -in /etc/letsencrypt/live/palaver.social/fullchain.pem
```

Check whether the distribution's placeholder site claims `default_server`.

```sh
grep default_server /etc/nginx/sites-enabled/default
```

If so: disable it.

```sh
sudo rm /etc/nginx/sites-enabled/default
```

Open `/etc/nginx/sites-available/palaver-catch-all` in an editor.

```sh
sudo vi /etc/nginx/sites-available/palaver-catch-all
```

Content:

```nginx
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;
    location /.well-known/acme-challenge/ { root /var/www/html; }
    location / { return 301 https://palaver.social/; }
}

server {
    listen 443 ssl default_server;
    listen [::]:443 ssl default_server;
    server_name _;
    ssl_certificate /etc/letsencrypt/live/palaver.social/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/palaver.social/privkey.pem;
    ssl_protocols TLSv1.2 TLSv1.3;
    return 301 https://palaver.social/;
}
```

Enable, check and load it.

```sh
sudo ln -sf /etc/nginx/sites-available/palaver-catch-all /etc/nginx/sites-enabled/palaver-catch-all
sudo nginx -t && sudo systemctl reload nginx
```

## Optional: wildcard certificate

- one certificate for the main domain and every name under it (`palaver.social`, `*.palaver.social`)
- certbot DNS challenge
- palaver.social: certbot snap, plugin `certbot-dns-hetzner-cloud`, credentials `/root/.secrets/hetzner.ini`,
  deploy hook `systemctl reload nginx`
- one certbot per host only

List the certificates and the names they cover.

```sh
sudo certbot certificates
```

Check the deploy hook of the wildcard certificate.

```sh
sudo grep -E 'authenticator|renew_hook|deploy_hook' /etc/letsencrypt/renewal/palaver.social.conf
```

## Files

| Path | Content |
|---|---|
| `/etc/systemd/system/palaverd@.service` | unit template |
| `/opt/palaver/instances/guild/palaverd.env` | configuration (user `palaver-guild` only) |
| `/opt/palaver/etc/livekit.yaml` | one key per server under `keys:` |
| `/etc/nginx/sites-available/palaver-guild` | nginx site |
| `/etc/nginx/snippets/palaver-web-guild.conf` | web client, CSP for `guild.palaver.social` |
| `/etc/nginx/sites-available/palaver-catch-all` | unknown names (optional) |
| `/var/log/nginx/palaver/guild.access.log`, `guild.error.log` | nginx logs (rotated by Palaver's `/etc/logrotate.d/palaver`) |

## Troubleshooting

Doctor of one server.

```sh
sudo -u palaver-guild /opt/palaver/bin/palaverd -env /opt/palaver/instances/guild/palaverd.env doctor
```

What palaverd says.

```sh
sudo journalctl --namespace=palaver -u palaverd@guild -n 100
```

What nginx says.

```sh
sudo tail -n 50 /var/log/nginx/palaver/guild.error.log
```

| Symptom | Cause |
|---|---|
| `useradd: invalid user name` | name too long (at most 24 characters) |
| voice fails on this server only | key missing in `livekit.yaml` or LiveKit not restarted after step 7 |
| voice fails on every server | `livekit.yaml` broken by an edit: `sudo journalctl --namespace=palaver -u livekit -n 50` |
| another server's site answers | `server_name` or the symlink in `sites-enabled` missing |
| web client without voice in the browser | CSP: `wss://guild.palaver.social` missing in `palaver-web-guild.conf` |