Skip to content

Troubleshooting

Unreachable Dashboard

If the Cloudron dashboard is unreachable, the first step is to SSH into the server and run the troubleshooting tool. The output should give you a good idea of why it's not working. If the checks fail, the tool will give further suggestions.

# cloudron-support --troubleshoot
Vendor: Hetzner Product: vServer
Linux: 6.8.0-38-generic
Ubuntu: noble 24.04
Processor: AMD EPYC Processor
BIOS NotSpecified  CPU @ 2.0GHz x 3
RAM: 3911380KB
Disk: /dev/sda1        39G
[OK]    node version is correct
[OK]    docker is running
[OK]    MySQL is running
[OK]    nginx is running
[OK]    dashboard cert is valid
[OK]    dashboard is reachable via loopback
[OK]    box v8.0.3 is running
[OK]    netplan is good
[OK]    DNS is resolving via systemd-resolved
[OK]    Dashboard is reachable via domain name
[OK]    Domain smartserver.io is valid and has not expired
[OK]    unbound is running

Expired domain

Check with your registrar if the domain has expired. In most cases, you can check this quickly using whois example.com.

If the domain has expired and you renew it, you have to wait a bit for the DNS records to come back. This is because most registrars replace your DNS with parking pages until you renew. Some registrars, won't revert back your old DNS records. In such situations, add a DNS A record for my.example.com to point to the server's IP. Then, wait and a bit for the DNS to propagate and login to the dashboard. Go to Domains view and click the Sync DNS button to re-add Cloudron related DNS records.

If the domain has expired and you do not intend to renew it, switch to another domain as follows:

  • Edit your PC/Mac's /etc/hosts and add an entry (the old domain and not the new one) to point to the server.
1.2.3.4 my.example.com

  • With the above entry in place, you can visit https://my.example.com in your browser. This works because the browser always looks up the /etc/hosts file first before looking up the DNS. You may have to purge the site from your browser's history to get over HSTS certificate issues.

  • Login and go to Domains view. Add the new domain.

  • Change the dashboard domain to this new domain.

  • Move all the apps one by one from the old domain to the new domain by changing the location.

  • Delete the old domain and remove the entry we added earlier from /etc/hosts.

Invalid Certificate

Here are some of the common reasons why Cloudron might fail to get certificates via Let's Encrypt.

  • The Cloudron administrator email is not valid. Let's Encrypt requires a valid email id for issuing certificates. Please check the email id in the Account page.

  • Let's Encrypt requires incoming port 80 to be accepted from all IPs. Note that Cloudron enforces port 443/HTTPS for all communication and any request on port 80 is redirected to HTTPS. For this reason, it is safe to keep port 80 completely open. Port 433/HTTPS can be restricted to specific IPs safely.

  • Let's Encrypt rate limit was reached.

  • Make sure that the DNS credentials for the domain are still valid. You can check by this by clicking 'Edit domain' in the Domains view and saving it without making any changes.

  • If all looks good, click the 'Renew All' button in Domains view to renew all the certs. See the logs for more information on why certificate renewal might be failing.

Failed upgrade

If the domain looks ok and nginx is running fine, check the box logs at /home/yellowtent/platformdata/logs/box.log.

To make sure, all migrations are up to date:

  • Run /home/yellowtent/box/setup/start.sh on the server.
  • Check systemctl status box. If it's not running, try systemctl restart box

If it still has errors, please contact support.

Hairpin NAT

Cloudron configures the DNS to be the public IP address. If the router does not support Hairpin NAT, then you will have trouble reaching the dashboard and the apps from inside your network.

There are many ways to fix this and the solution depends on your use case. Here are some suggestions:

  • If you intend to use Cloudron and the apps from only inside your network, change the network settings to use the local IP address. When doing so, Cloudron will use the local IP in the DNS instead of the public IP address.

  • If you intend to use Cloudron from outside your network as well, the easiest fix is to purchase a new router that supports hairpin NAT.

  • If purchasing a new router is not an option:

    • Configure your network's DNS server to return the Local VM IP for all the subdomain in use. This way when your PC/Laptop accesses a domain, it starts using the Local VM IP instead of the public IP to connect to Cloudron. Devices outside the network will continue to use the public IP address as expected.

    • Some apps use the domain name to connect with each other (For example, collabora app connects to nextcloud app with domain name). For such cases, configure Cloudron's DNS server unbound to use your network's DNS.

MySQL (host) Corruption

There are two instances of MySQL on Cloudron. One instance runs on the host and is used by the platform. Another instance is the MySQL addon which runs in a container named mysql and is shared by apps. The instructions below are for the MySQL running on the host.

First, check if it is running using:

systemctl status mysql

Check for errors and warnings in /var/log/mysql/error.log.

If MySQL is running but generating many warnings/errors, check if the Restore steps below are possible. If MySQL is not running at all, attempt the Recovery steps.

Restore MySQL (host)

If MySQL is still running, you can connect to it and create a database dump. Then, drop the database and re-create it from the previously created dump.

mysqldump -uroot -ppassword --single-transaction --routines --triggers box > box.mysql
mysql -uroot -ppassword -e "DROP DATABASE box"
mysql -uroot -ppassword -e "CREATE DATABASE IF NOT EXISTS box"
mysql -uroot -ppassword box < box.mysql

If one or more steps above get "stuck", read the recovery section below.

Recover MySQL (host)

We recommend taking a snapshot of your server before performing the operations below.

  • Stop box code: systemctl stop box

  • Edit /etc/mysql/my.cnf to have the below. See the recovery docs for details.

[mysqld]
innodb_force_recovery = 1

  • Keep increasing the above value till MySQL starts with systemctl start mysql

  • Once it starts, we have to take a dump of the database:

# mysqldump -uroot -ppassword --skip-lock-tables -A > /root/alldb.sql

  • Now that we created the dump, stop MySQL - systemctl stop mysql

  • Remove the innodb_force_recovery in my.cnf

  • Recreate the existing MySQL installation:

# mv /var/lib/mysql /var/lib/mysql.old
# mkdir /var/lib/mysql
# chown -R mysql:mysql /var/lib/mysql
# mysqld --initialize   # This will dump the MySQL root password in /var/log/mysql/error.log

  • Start MySQL - systemctl start mysql

  • Change the root password to password (sic) -

# mysql -uroot -p<password from /var/log/mysql/error.log>  # there is no space between p and the password. e.g -pAS23kdI
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6365
Server version: 5.7.25-0ubuntu0.18.04.2 (Ubuntu)

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'password';

  • Import the database - mysql -uroot -ppassword < /root/alldb.sql

  • Start the platform code again - systemctl restart box

Unresponsive App

As a first step, check the app logs to look for errors. If the issue is related to a mis-configuration or an error in some plugin, theme or extension, use the following steps to recover the app:

  • Enable recovery mode from the the Repair section. In recovery mode, the app is started in a "paused" state. In this state, the app's container is accessible via the Web Terminal but the app itself is not running (after all, if it runs, it will crash).

  • Edit the filesystem and/or fix plugins using the Web Terminal. The approach to fix depends on the app and the tooling the app provides. For example, WordPress plugins can be disabled using WP CLI and Nextcloud can be disabled using occ. Please look into the Cloudron's app docs for hints (Look under Apps in the side bar).

  • To test if the app starts, run the start.sh script using the Web Terminal. This script is usually located under /app/pkg or /app/code (this is a Cloudron packaging convention).

  • Once fixed, disable recovery mode. This will start the app.

Unresponsive Service

Check if all the services are running in the Services view. If they are not green, the first step is to restart them via the UI and to try increasing the memory limit.

Check the service logs for hints of why it's not starting up. If that doesn't work, please contact Cloudron support.

Corrupt Addon

The instructions here apply to the following database addons:

Database Container name Logs Data Directory
MySQL mysql /home/yellowtent/platformdata/logs/mysql/app.log /home/yellowtent/platformdata/mysql
PostgreSQL postgresql /home/yellowtent/platformdata/logs/postgresql/app.log /home/yellowtent/platformdata/postgresql
MongoDB mongodb /home/yellowtent/platformdata/logs/mongodb/app.log /home/yellowtent/platformdata/mongodb
Redis redis- /home/yellowtent/platformdata/logs/redis-/app.log /home/yellowtent/platformdata/redis/

In case of disk corruption, it's best to just start afresh and restore the data from backups.

To restore a specific database:

docker stop <containername>
mv /home/yellowtent/platformdata/<database> /home/yellowtent/platformdata/<database>-copy # stash the database data directory. see table above
mkdir /home/yellowtent/platformdata/<database> # create a fresh database directory
docker restart <containername> # start the addon. this will re-initialize the addon with no data

Now, restore each app that uses the addon in the Cloudron dashboard.