Upgrade guide

This guide helps HoyoShip administrators perform upgrades safely and provides guidance to troubleshoot problems.

Before you upgrade

• Read the release notes of every release between your current version and the target version (see Releases).
• Take a backup, as described below.
• Review your interface customizations (custom templates and CSS) against the new release.

Backup

To be safe, make sure to perform a full backup before upgrading. It is a requirement when upgrading to a new minor release (going from v0.16.1 to v0.17.0 for instance) but is also a good precaution when installing a patch release (going from v0.16.1 to v0.16.2 for instance).

The reliable way to perform a backup is with a synchronized point-in-time snapshot of all the storage used by HoyoShip: the database, the data volume (/var/lib/gitea in the HoyoShip image, which holds the repositories and app.ini) and any object storage.

For instance, if everything (SQLite database + repositories etc.) is on a single QCOW2 disk attached to a virtual machine, a qemu snapshot guarantees the backup is consistent and can be done while the HoyoShip server is running.

However, if HoyoShip uses a S3 storage for attachments with a PostgreSQL database, stores Git repositories on a remote file system and queues in a Redis server, the only way to ensure a consistent state is to shut down HoyoShip during the backup. On Kubernetes, scale the deployment to zero replicas, snapshot the volume and back up the database, then scale it back up.

In the simplest case where everything is on a single file system and if the instance is not busy (no mirrors, no users), the backup can be done with:

• hoyoship dump to collect everything into one zip file
• psql/mysql dump. Although the zip file created by hoyoship dump contains a copy of the database, re-injecting that SQL dump into a new database is unreliable, so dump the database with its own tools. Note that there is no need to dump SQLite because the database itself is included in the zip file already.

Preparing the upgrade

• Do not use hoyoship help to figure out the location of CustomPath, look at the configuration tab of the Site administration panel when logged in as an admin.
• Run hoyoship manager flush-queues against the running server. If it times out, run it again with a more generous --timeout argument. It is important because the queues contain serialized data that is not guaranteed to be backward compatible between versions. On Kubernetes, run it in the pod, for example kubectl exec -n hoyoship deploy/hoyoship-forgejo -- hoyoship manager flush-queues.

Note: the HoyoShip image requires docker >= 20.10.6; with an older version the container refuses to start.

Performing the upgrade

Change the image to the new release, pinned by tag and digest. The database migrations run automatically when the new version starts.

• On Kubernetes, set image.tag and image.digest in your values and run helm upgrade, or commit the change to the repository your GitOps tool syncs (see Installation on Kubernetes).
• With Docker, change the image: of the HoyoShip service in your compose file to the new tag and digest, then run docker compose pull and docker compose up -d (see Installation with Docker).

Verify HoyoShip works

It is critical to verify that HoyoShip works very carefully. Restoring the backup done before the upgrade is easy and does not lose any information. But if a problem is discovered days or weeks after the upgrade, it will not be an option and fixing it on a live HoyoShip instance will be much more challenging.

• Run hoyoship doctor check --all --log-file /tmp/doctor.log and make sure it does not report any problem.
• Manually verify via the web interface. Making a checklist of a typical use case is a good way to not miss anything.
• If there is a problem of any kind, increase the log level and take a look at the logs. If you cannot figure out what the problem is, contact support@doricod.com before trying to fix it.

Troubleshooting

• Increase the log level. By default, the logs are outputted to console. If you need to collect logs from files, you could copy the following config into your app.ini file (remove all other [log] sections), then you can find the *.log files in HoyoShip’s log directory.

   ; To show all SQL logs, you can also set LOG_SQL=true in the [database] section
   [log]
   LEVEL=debug
   MODE=console,file
   ROUTER=console,file
   XORM=console,file
   ENABLE_XORM_LOG=true
   FILE_NAME=hoyoship.log
   [log.file.router]
   FILE_NAME=router.log
   [log.file.xorm]
   FILE_NAME=xorm.log

• If an upgrade across several minor releases fails and there is a need to narrow down the problem, upgrade through the latest patch release of each minor release in turn, verifying HoyoShip works after each one. It shows which release causes the issue and helps debug the problem.

Unexpected database version

The database version is stored in the database and used to prevent an accidental downgrade. If a HoyoShip instance is started with an older release than the one that last migrated its database, it refuses to start because it would damage the content of the database. To go back to an older release, restore the backup taken before the upgrade.