Skip to content

Troubleshooting

Unresponsive App

As a first step, check the app logs to indentify the source of the problem. If the issue is related to a mis-configuration or an error in some plugin, theme or extension, use the following steps to recover the app:

  • Enable recovery mode from the the Repair section. In recovery mode, the app is started in a "paused" state. In this state, the app's container is accessible via the Web Terminal but the app itself is not running (after all, if it runs, it will crash).

  • Edit the filesystem and/or fix plugins using the Web Terminal. The approach to fix depends on the app and the tooling the app provides. For example, for WordPress, plugins can be disabled using WP CLI. Please look into the Cloudron's app docs for hints (Click the documentation dropdown in the above screenshot).

  • To test if the app starts, run the start.sh script. This script is usually located under /app/code or /app/pkg (this is a Cloudron packaging convention).

  • Once fixed, disable recovery mode. This will start the app.

Unresponsive Service

Check if all the services are running in the Services view. If they are not responding, try increasing the memory limit.

If that doesn't work, please contact Cloudron support.

Unreachable Dashboard

If the Cloudron dashboard is down/not reachable, try the following steps:

  • Make sure all migrations are run on the server - Run /home/yellowtent/box/setup/start.sh on the server.
  • Try systemctl status box on the server. If it's not running, try systemctl restart box

Check, if this a networking issue on your PC

  • Check the output of host my.<domain> on your PC and verify that the IP address is pointing to your server.

If the dashboard was using a domain that you no longer control and you want to switch it to a new domain, change it via the dashboard as mentioned here or if the dashboard is not reachable anymore do the following:

  • Run those commands to update the database records manually, make sure to change example.com in both:
mysql -uroot -ppassword box -e "UPDATE settings SET value='example.com' WHERE name='admin_domain'"
mysql -uroot -ppassword box -e "UPDATE settings SET value='my.example.com' WHERE name='admin_fqdn'"
  • Add an A record manually in your DNS provider for my.example.com to your server's IP

  • Run the command systemctl restart box

In a few minutes, you should be able to reach https://my.example.com. You may have to purge the site from your browser's history to get over HSTS certificate issues.

  • Add the domain now in the Domains view.

Logs

Logs for each component are located in /home/yellowtent/platformdata/logs/. Many of these logs are viewable directly using the Cloudron dashboard in the support, mail or system view.

Recovery after disk full

One or more system services may go down if the disk becomes full. Once some space has been freed up, check all the services below.

The System view displays the current status of the internal services on the Cloudron. Make sure everything is green.

/boot is full

On some systems /boot is on a separate partition where the linux kernel and init system are installed. If that partition is filled up, the system is unable to apply security updates to the kernel.

Run the following to uninstall all linux-image packages currently not used:

apt-get remove `dpkg --list 'linux-image*' |grep ^ii | awk '{print $2}'\ | grep -v \`uname -r\``

Unbound

Cloudron uses an internal DNS server called unbound. This server stops working if it is unable to save the trust anchor file. To get it running again, one has to re-download the root key and restart the unbound service.

First, check the status of unbound using:

systemctl status unbound

It must say active (running). If not, run the following commands:

unbound-anchor -a /var/lib/unbound/root.key
systemctl restart unbound

Nginx

Check the status of nginx:

systemctl status nginx

If nginx is not running:

systemctl restart nginx

Docker

Docker can be restarted using the following command:

systemctl restart docker

Note that the above command will restart all the apps and addon services.

MySQL (host)

There are two instances of MySQL on Cloudron. One instance run on the host and is used by the platform. Another instance is the MySQL addon which runs in a container named mysql and is shared by apps. The instructions below are for the MySQL running on the host.

First, check if it is running using:

systemctl status mysql

If it is not, check the contents of the file /var/log/mysql/error.log.

Sometimes, the only way is to recreate the database from a dump. For this, re-create a database dump like so:

mysqldump -uroot -ppassword --single-transaction --routines --triggers box > box.mysql
mysql -uroot -ppassword -e "DROP DATABASE box"
mysql -uroot -ppassword -e "CREATE DATABASE IF NOT EXISTS box"
mysql -uroot -ppassword box < box.mysql

It can happen at times that the creation of the mysqldump is 'stuck'. This can happen if one or more tables is corrupt. In this case, read the recovery section below.

Recover MySQL

MySQL might sometimes refuse to start with INVALIDARGUMENT or mysqldump gets stuck and refuses to create a dump or the connection to the database keep dropping causing the box code and backup code to crash. This means that the database is corrupt and to fix this, we have to recreate the whole database.

We recommend taking a snapshot of your server before performing the operations below.

  • Stop box code: systemctl stop box

  • Edit /etc/mysql/my.cnf to have the below. See the recovery docs for details.

[mysqld]
innodb_force_recovery = 1
  • Keep increasing the above value till mysql start with systemctl start mysql

  • Once it starts, we have to take a dump of all the database:

# mysqldump -uroot -ppassword --skip-lock-tables -A > /root/alldb.sql
  • Now that we created the dump, stop MySQL - systemctl stop mysql

  • Remove the innodb_force_recovery in my.cnf

  • Recreate the existing MySQL installation:

# mv /var/lib/mysql /var/lib/mysql.old
# mkdir /var/lib/mysql
# chown -R mysql:mysql /var/lib/mysql
# mysqld --initialize   # This will dump the MySQL root password in /var/log/mysql/error.log
  • Start MySQL - systemctl start mysql

  • Change the root password to password (sic) -

# mysql -uroot -p<password from /var/log/mysql/error.log>  # there is no space between p and the password. e.g -pAS23kdI
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 6365
Server version: 5.7.25-0ubuntu0.18.04.2 (Ubuntu)

mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY 'password';
  • Import the database - mysql -uroot -ppassword < /root/alldb.sql

  • Start the platform code again - systemctl restart box

MySQL (addon)

There are two instances of MySQL on Cloudron. One instance run on the host and is used by the platform. Another instance is the MySQL addon which runs in a container named mysql and is shared by apps. The instructions below are for the MySQL addon running in the container.

First, check if it is running using:

docker ps | grep mysql

The logs are located inside the container:

docker exec -ti mysql /bin/bash
tail -f /tmp/mysqld.err

Depending on the error, the config has to be changed in /run/mysqld/my.cnf. After making config changes, restart the container using docker restart mysql.

In case of disk corruption errors, it's best to just start afresh if you have uptodate backups.

docker stop mysql
mv /home/yellowtent/platformdata/mysql /home/yellowtent/platformdata/mysql-copy
mkdir /home/yellowtent/platformdata/mysql
docker restart mysql

Then, just restore the apps one by one from the latest backup.

Certificates

Here are some of the common reasons why the Cloudron might fail to get certificates via Let's Encrypt.

  • The Cloudron administrator email is not valid. Let's Encrypt requires a valid email id for issuing certificates. Please check the email id in the Account page.

  • Let's Encrypt requires incoming port 80 to be accepted from all IPs. Note that Cloudron enforces port 443/HTTPS for all communication and any request on port 80 is redirected to HTTPS. For this reason, it is safe to keep port 80 completely open. Port 433/HTTPS can be restricted to specific IPs safely.

  • Let's Encrypt rate limit was reached.

  • Make sure that the DNS credentials for the domain are still valid. You can check by this by clicking 'Edit domain' in the Domains view and saving it without making any changes.

  • If all looks good, click the 'Renew All' button in Domains view to renew all the certs. See the logs for more information on why certificate renewal might be failing.

DNS

If your network uses an internal DNS server, you might find apps erroring with ECANCELLED or EREFUSED errors. Other symptoms include apps like NextCloud being unable to install apps and internal mail relay not working.

To remedy, configure Cloudron's DNS server unbound to forward the queries to the internal DNS server. Create a file named /etc/unbound/unbound.conf.d/private-dns.conf:

# this disables DNSSEC
server:
  val-permissive-mode: yes

# forward all queries to the internal DNS
forward-zone:
  name: "."
  forward-addr: 1.2.3.4

Then restart unbound:

systemctl restart unbound
host www.cloudron.io 127.0.0.1 # this command should work