Skip to content

Troubleshooting

Unreachable Dashboard

The Cloudron dashboard can become unreachable for a variety of reasons. Please go through the items below in order to identify where the issue might be.

Expired domain

Check with your registrar if the domain has expired. In most cases, you can check this quickly using whois example.com.

If the domain has expired and you renew it, you have to wait a bit for the DNS records to come back. This is because most registrars replace your DNS with parking pages until you renew. Some registrars, won't revert back your old DNS records. In such situations, add a DNS A record for my.example.com to point to the server's IP. Then, wait and a bit for the DNS to propagate and login to the dashboard. Go to Domains view and click the Sync DNS button to re-add Cloudron related DNS records.

If the domain has expired and you do not intend to renew it, switch to another domain as follows:

  • Edit your PC/Mac's /etc/hosts and add an entry (the old domain and not the new one) to point to the server.
1.2.3.4 my.example.com

  • With the above entry in place, you can visit https://my.example.com in your browser. This works because the browser always looks up the /etc/hosts file first before looking up the DNS. You may have to purge the site from your browser's history to get over HSTS certificate issues.

  • Login and go to Domains view. Add the new domain.

  • Change the dashboard domain to this new domain.

  • Move all the apps one by one from the old domain to the new domain by changing the location.

  • Delete the old domain and remove the entry we added earlier from /etc/hosts.

Nginx error

If the domain looks ok (see above), check if Nginx is running using systemctl status nginx. If it is not running, check the nginx logs using journactl -u nginx -fa. If the error is related to bad config files, delete the bad config files (this is safe, cloudron can always re-generate nginx files). Once deleted, run systemctl restart nginx. Sometimes, you might have to delete one or two config files. This can happen if your server ran out of disk space and the config files are corrupt.

Failed upgrade

If the domain looks ok and nginx is running fine, check the box logs at /home/yellowtent/platformdata/logs/box.log.

To make sure, all migrations are up to date:

  • Run /home/yellowtent/box/setup/start.sh on the server.
  • Check systemctl status box. If it's not running, try systemctl restart box

If it still has errors, please contact support.

Unresponsive App

As a first step, check the app logs to look for errors. If the issue is related to a mis-configuration or an error in some plugin, theme or extension, use the following steps to recover the app:

  • Enable recovery mode from the the Repair section. In recovery mode, the app is started in a "paused" state. In this state, the app's container is accessible via the Web Terminal but the app itself is not running (after all, if it runs, it will crash).

  • Edit the filesystem and/or fix plugins using the Web Terminal. The approach to fix depends on the app and the tooling the app provides. For example, WordPress plugins can be disabled using WP CLI and Nextcloud can be disabled using occ. Please look into the Cloudron's app docs for hints (Look under Apps in the side bar).

  • To test if the app starts, run the start.sh script using the Web Terminal. This script is usually located under /app/pkg or /app/code (this is a Cloudron packaging convention).

  • Once fixed, disable recovery mode. This will start the app.

Unresponsive Service

Check if all the services are running in the Services view. If they are not responding, try increasing the memory limit.

If that doesn't work, please contact Cloudron support.

Recovery after disk full

One or more system services may go down if the disk becomes full. Once some space has been freed up, check all the services below.

The System view displays the current status of the internal services on the Cloudron. Make sure everything is green.

/boot is full

On some systems /boot is on a separate partition where the linux kernel and init system are installed. If that partition is filled up, the system is unable to apply security updates to the kernel.

Run the following to uninstall all linux-image packages currently not used:

apt-get remove `dpkg --list 'linux-image*' |grep ^ii | awk '{print $2}'\ | grep -v \`uname -r\``

Unbound

Cloudron uses an internal DNS server called unbound. This server stops working if it is unable to save the trust anchor file. To get it running again, one has to re-download the root key and restart the unbound service.

First, check the status of unbound using:

systemctl status unbound

It must say active (running). If not, run the following commands:

unbound-anchor -a /var/lib/unbound/root.key
systemctl restart unbound

If DNS queries are failing with SERVFAIL, then you can try forwarding all DNS requests to other nameservers to see if it fixes the problems. Add the following file /etc/unbound/unbound.conf.d/forward-everything.conf and restart unbound:

forward-zone:
    name: "."
    forward-addr: 1.1.1.1
    forward-addr: 8.8.8.8

Nginx

Check the status of nginx:

systemctl status nginx

If nginx is not running:

systemctl restart nginx

Docker

Docker can be restarted using the following command:

systemctl restart docker

Note that the above command will restart all the apps and addon services.

MySQL (host)

There are two instances of MySQL on Cloudron. One instance run on the host and is used by the platform. Another instance is the MySQL addon which runs in a container named mysql and is shared by apps. The instructions below are for the MySQL running on the host.

First, check if it is running using:

systemctl status mysql

If it is not, check the contents of the file /var/log/mysql/error.log.

Sometimes, the only way is to recreate the database from a dump. For this, re-create a database dump like so:

mysqldump -uroot -ppassword --single-transaction --routines --triggers box > box.mysql
mysql -uroot -ppassword -e "DROP DATABASE box"
mysql -uroot -ppassword -e "CREATE DATABASE IF NOT EXISTS box"
mysql -uroot -ppassword box < box.mysql

It can happen at times that the creation of the mysqldump is 'stuck'. This can happen if one or more tables is corrupt. In this case, read the recovery section below.

Recover MySQL

MySQL might sometimes refuse to start with INVALIDARGUMENT or mysqldump gets stuck and refuses to create a dump or the connection to the database keep dropping causing the box code and backup code to crash. This means that the database is corrupt and to fix this, we have to recreate the whole database.

We recommend taking a snapshot of your server before performing the operations below.

  • Stop box code: systemctl stop box

  • Edit /etc/mysql/my.cnf to have the below. See the recovery docs for details.

[mysqld]
innodb_force_recovery = 1

  • Keep increasing the above value till mysql start with systemctl start mysql

  • Once it starts, we have to take a dump of all the database:

# mysqldump -uroot -ppassword --skip-lock-tables -A > /root/alldb.sql

  • Now that we created the dump, stop MySQL - systemctl stop mysql

  • Remove the innodb_force_recovery in my.cnf

  • Recreate the existing MySQL installation:

# mv /var/lib/mysql /var/lib/mysql.old
# mkdir /var/lib/mysql
# chown -R mysql:mysql /var/lib/mysql
# mysqld --initialize   # This will dump the MySQL root password in /var/log/mysql/error.log

  • Start MySQL - systemctl start mysql

  • Change the root password to password (sic) -

# mysql -uroot -p<password from /var/log/mysql/error.log>  # there is no space between p and the password. e.g -pAS23kdI
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6365
Server version: 5.7.25-0ubuntu0.18.04.2 (Ubuntu)

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'password';

  • Import the database - mysql -uroot -ppassword < /root/alldb.sql

  • Start the platform code again - systemctl restart box

MySQL (addon)

There are two instances of MySQL on Cloudron. One instance run on the host and is used by the platform. Another instance is the MySQL addon which runs in a container named mysql and is shared by apps. The instructions below are for the MySQL addon running in the container.

First, check if it is running using:

docker ps | grep mysql

The logs are located inside the container:

docker exec -ti mysql /bin/bash
tail -f /tmp/mysqld.err

Depending on the error, the config has to be changed in /run/mysqld/my.cnf. After making config changes, restart the container using docker restart mysql.

In case of disk corruption errors, it's best to just start afresh if you have uptodate backups.

docker stop mysql
mv /home/yellowtent/platformdata/mysql /home/yellowtent/platformdata/mysql-copy
mkdir /home/yellowtent/platformdata/mysql
docker restart mysql

Then, just restore the apps one by one from the latest backup.

Certificates

Here are some of the common reasons why the Cloudron might fail to get certificates via Let's Encrypt.

  • The Cloudron administrator email is not valid. Let's Encrypt requires a valid email id for issuing certificates. Please check the email id in the Account page.

  • Let's Encrypt requires incoming port 80 to be accepted from all IPs. Note that Cloudron enforces port 443/HTTPS for all communication and any request on port 80 is redirected to HTTPS. For this reason, it is safe to keep port 80 completely open. Port 433/HTTPS can be restricted to specific IPs safely.

  • Let's Encrypt rate limit was reached.

  • Make sure that the DNS credentials for the domain are still valid. You can check by this by clicking 'Edit domain' in the Domains view and saving it without making any changes.

  • If all looks good, click the 'Renew All' button in Domains view to renew all the certs. See the logs for more information on why certificate renewal might be failing.

DNS

If your network uses an internal DNS server, you might find apps erroring with ECANCELLED or EREFUSED errors. Other symptoms include apps like NextCloud being unable to install apps and internal mail relay not working.

To remedy, configure Cloudron's DNS server unbound to forward the queries to the internal DNS server. Create a file named /etc/unbound/unbound.conf.d/private-dns.conf:

# this disables DNSSEC
server:
  val-permissive-mode: yes

# forward all queries to the internal DNS
forward-zone:
  name: "."
  forward-addr: 1.2.3.4

Then restart unbound:

systemctl restart unbound
host www.cloudron.io 127.0.0.1 # this command should work

Hairpin NAT

Cloudron configures the DNS to be the public IP address. If the router does not support Hairpin NAT, then you will have trouble reaching the dashboard and the apps from inside your network.

There are many ways to fix this and the solution depends on your use case. Here are some suggestions:

  • If you intend to use Cloudron and the apps from only inside your network, change the network settings to use the local IP address. When doing so, Cloudron will use the local IP in the DNS instead of the public IP address.

  • If you intend to use Cloudron from outside your network as well, the easiest fix is to purchase a new router that supports hairpin NAT.

  • If purchasing a new router is not an option:

    • Configure your network's DNS server to return the Local VM IP for all the subdomain in use. This way when your PC/Laptop accesses a domain, it starts using the Local VM IP instead of the public IP to connect to Cloudron. Devices outside the network will continue to use the public IP address as expected.

    • Some apps use the domain name to connect with each other (For example, collabora app connects to nextcloud app with domain name). For such cases, configure Cloudron's DNS server unbound to use your network's DNS.