Differences between revisions 17 and 18
Revision 17 as of 2018-09-12 15:23:52
Size: 7186
Editor: adam-collard
Comment:
Revision 18 as of 2018-10-02 10:57:12
Size: 7236
Editor: waveform
Comment: Re-write backup instructions to include/require PITR for PostgreSQL
Deletions are marked like this. Additions are marked like this.
Line 3: Line 3:
== Overview ==
Line 4: Line 5:
These are general guidelines that you can follow to backup and restore Landscape Dedicates Server. Please be aware that depending on your specific deployment, additional files and setting may be needed to be backed up and restored. When restoring files from the backup, make sure you restore the original file permission and ownership. On-Prem Landscape consists (at a high level) of several stateful components, all of which need to be roughly in sync to guarantee correct functioning of the system as a whole. Specifically:
Line 6: Line 7:
= BACKUP =  * The Landscape server, at least 6 PostgreSQL databases, a cache of hash databases, and a variety of configuration files.
 * The Landscape clients, comprising several SQLite databases for tracking package states.
Line 8: Line 10:
== On the Application server == Given the wide variety of clients (from physical hardware, to VMs, to containers, some of which may be permanent and others merely temporary), backup of clients (if required at all) is beyond the scope of this document.
Line 10: Line 12:
'''1)''' Stop the Landscape services on the application server(s): Backup of the Landscape server must be performed in such a fashion as to permit restoration to its latest possible state. Hence, the only supported backup option is one that permits point-in-time recovery (PITR hereafter).
Line 12: Line 14:
{{{
$ sudo lsctl stop
}}}
We strongly recommend that administrators of an On-Prem Landscape instance first familiarize themselves with PostgreSQL's archived logging and PITR facilities. Some syntactic configuration changes have occurred across PostgreSQL versions, hence you should select the documentation for your particular PostgreSQL server version:
Line 16: Line 16:
Please note that LDS can be deployed using several servers, so when you are taking the offline backup route, remember to disable all the Landscape services on all server machines!

'''2)''' Take a backup of important files:
Remember that upon restoration, you will need to adjust the original file permissions and ownership of these files accordingly.

 * `/etc/landscape:` configuration files and the LDS license
 * `/etc/default/landscape-server`: file to configure which services will start on this machine
 * `/var/lib/landscape/hash-id-databases`: these files are recreated by a weekly cron job, which can take several minutes to run, so backing them up can save time
 * `/etc/apache2/sites-available/`: the Landscape Apache vhost configuration file, usually named after the FQDN of the server
 * `/etc/ssl/certs/`: the Landscape server X509 certificate is in there
 * `/etc/ssl/private/`: the Landscape server X509 key file
 * `/etc/ssl/certs/landscape_server_ca.crt`: if in use, this is the CA file for the internal CA used to issue the Landscape server certificates. Clients need to have this file in that case.
 * `/etc/postgresql/9.3/main/`: postgresql configuration files, in particular postgresql.conf for the tuning and pg_hba.conf for the access rules. These files may be in a separate host, dedicated to the database.
 * `/var/log/landscape/`: all the LDS log files (useful - but not mandatory in the restore process)

You may backup any other files that you think are important to your particular case. Files that may not necessarily be part of Landscape or PostgreSQL but that are a part of your custom setup (firewall rules, proxy settings,…etc).

== On the Database server ==

'''1)''' Backup the PostgreSQL database as described in the official documentation:
The PostgreSQL documentation is quite extensive on backup procedures, we recommend that administrators tasked with database backups read and understand it:
http://www.postgresql.org/docs/9.3/static/backup-dump.html
A quick way of backing up the database is using `pg_dumpall`:
http://www.postgresql.org/docs/9.3/static/backup-dump.html#BACKUP-DUMP-ALL
 * [[https://www.postgresql.org/docs/9.3/static/continuous-archiving.html|PostgreSQL 9.3]] (trusty)
 * [[https://www.postgresql.org/docs/9.5/static/continuous-archiving.html|PostgreSQL 9.5]] (xenial)
 * [[https://www.postgresql.org/docs/10/static/continuous-archiving.html|PostgreSQL 10]] (bionic)
Line 42: Line 21:
= RESTORE = == Backup & Retention Policy ==
Line 44: Line 23:
== On the Database server == Before configuring your PostgreSQL instance for continuous archiving and PITR, it is important to decide on a backup policy:
Line 46: Line 25:
When restoring files from the backup, make sure you restore the original file permission and ownership accordingly.  1. When should base backups be taken? (recommendation: daily or weekly depending on volume of WAL logs, and desired recovery speed)
 1. How many base backups should be retained at any given time? (this will dictate the earliest point in time to which you can initially restore)
 1. Where should WAL logs be archived to? (recommendation: a separate machine or some form of networked, but secure storage)
 1. Where should base backups be stored (recommendation: the same machine as the WAL archive so that all materials necessary for restoration are available in one place)
Line 48: Line 30:
'''1)''' Install PostgreSQL and required libraries:

{{{
$ sudo apt-get install postgresql-9.3 python-apt postgresql-plpython-9.3 postgresql-contrib-9.3 postgresql-9.3-debversion
}}}

'''2)''' Create a superuser Landscape can use:

{{{
$ sudo -u postgres createuser --createdb --createrole --superuser --pwprompt landscape_superuser (remember this password)
}}}

'''3)''' Configure PostgreSQL:

Restore from the backup the following files and adjust permissions and ownership accordingly:

{{{
/etc/postgresql/9.3/main/pg_hba.conf
/etc/postgresql/9.3/main/postgresql.conf
}}}
Although it is possible to backup and archive on the same machine as the PostgreSQL server, we recommend a separate machine is used for base backup and archived log storage to permit restoration in the case the server becomes inaccessible for whatever reason. We further recommend that other files required to restore the Landscape application server (configuration files which will be listed below) are also copied to this location to permit recovery of the entire service from one place.
Line 70: Line 33:
'''4)''' Restart the service: == PostgreSQL Configuration ==
Line 72: Line 35:
{{{
$ sudo /etc/init.d/postgresql restart
}}}
In the `postgresql.conf` configuration file, set `wal_level`, `archive_mode`, and `archive_command` according to the PostgreSQL documentation for your server's version. As recommended by the PostgreSQL documentation, ''test'' your `archive_command` operates correctly in all circumstances, including returning the correct exit codes.
Line 76: Line 37:
'''5)''' Restoring the Database from the backup as detailed in the official documentation:
http://www.postgresql.org/docs/9.3/static/backup-dump.html
Once you are confident the configuration is correct, restart the PostgreSQL service to activate archived logging. Monitor the archive destination to ensure logs begin to appear there (if your server is very low traffic, you may wish to use the `archive_timeout` setting to force archiving of partial logs after a timeout).

When WAL logs are being archived successfully, construct a script that executes `pg_basebackup` and stores the result in your base backup storage destination. Test that this operates correctly as the cluster owner (typically `postgres`), then add a cronjob to periodically execute this script (as the cluster owner).

Note that there is no need to take Landscape offline to perform these backups. In fact, `pg_basebackup` can only execute when the cluster is up anyway. There is no need to worry about inconsistency between Landscape's various databases either: a base backup represents the state of the cluster (across all databases within it) at the instant the backup starts.
Line 80: Line 44:
== On the Application server == == Server Configuration Files ==
Line 82: Line 46:
'''1)''' Adding the Landscape package archive and installing the package: The following files should also be copied from your Landscape server(s) to your backup destination to ensure restoration of the Landscape application server is also possible:
Line 84: Line 48:
As part of your LDS purchase, you should have gotten a URL pointing to a private archive which hosts the LDS packages.
To access that archive, please follow the instructions on https://landscape.canonical.com/~/how-to-get-dedicated, replacing <youraccount> with your Landscape account name.
 * `/etc/landscape/*` - Landscape configuration files, and the On-Prem Landscape license)
 * `/etc/default/landscape-server` - Specifies which Landscape application services start on a given machine
 * `/etc/apache2/sites-available/<server-name>` - the Landscape Apache vhost configuration file, usually named after the FQDN of the server
 * `/etc/ssl/certs/<landscape-cert>` - the Landscape server X.509 certificate
 * `/etc/ssl/private/<landscape-cert>` - the Landscape server X.509 key file
 * `/etc/postgresql/<pg-version>/main/*` - various PostgreSQL configuration files
Line 87: Line 55:
'''2)''' Add the sources for LDS:
Please add the line corresponding to your version of Ubuntu to a new file, named: /etc/apt/sources.list.d/landscape-dedicated-server.list
Optionally, you may also wish to backup the following files. They are not required for normal operation of Landscape server, but may provide additional information in the case of service outages:
Line 90: Line 57:
Adjust permissions on the sources file:  * `/var/log/landscape-server/*` - the Landscape server log files
Line 92: Line 59:
{{{
$ sudo chmod 0640 /etc/apt/sources.list.d/landscape-dedicated-server.list
$ sudo chown root:landscape /etc/apt/sources.list.d/landscape-dedicated-server.list
}}}

'''3)''' Import the signing key for this archive:

{{{
$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-key 4652B4E6
}}}

'''4)''' Install the package.
Note that it will complain about a missing license file:

{{{
$ sudo apt-get update
$ sudo apt-get install landscape-server rabbitmq-server apache2-mpm-worker
}}}

'''5)''' Install the license file by copying it to /etc/landscape:

{{{
$ sudo cp license.txt /etc/landscape
}}}

Make sure it's readable by everybody, or at least the landscape user:

{{{
$ sudo chown root:landscape /etc/landscape/license.txt
}}}

'''6)''' Configure rabbitmq:
Just run the following commands, replacing <password> with the old password. The password can be found in the [broker] section of the `/etc/landscape/service.conf` file.

{{{
$ sudo rabbitmqctl add_user landscape <password>
$ sudo rabbitmqctl add_vhost landscape
$ sudo rabbitmqctl set_permissions -p landscape landscape ".*" ".*" ".*"
}}}

'''7)''' Restore from the backup the `/etc/rabbitmq/rabbitmq.conf` file and adjust permissions and ownership accordingly.

'''8)''' Restart the rabbitmq-server service:

{{{
$ sudo /etc/init.d/rabbitmq-server restart
}}}

'''9)''' Restore from the backup the `/etc/landscape/service.conf` file and adjust permissions and ownership accordingly.

'''10)''' Run:

{{{
$ sudo setup-landscape-server
}}}

'''11)''' Restore from the backup the `/etc/default/landscape-server` file and adjust permissions and ownership accordingly.

=== Webserver configuration ===

'''12)''' Restore from the backup the `/etc/apache2/sites-available/<YOUR_SITE>` file and adjust permissions and ownership accordingly.

'''13)''' Restore from the backup the Landscape Server certificate, the CA's certificate and the Landscape Server certificate private key and adjust permissions and ownership accordingly.

'''14)''' Enable the following modules:

{{{
$ for module in rewrite proxy_http ssl headers expires; do sudo a2enmod $module; done
}}}

'''15)''' Enable the new site:

{{{
$ sudo a2ensite landscape
$ sudo service apache2 restart
}}}

'''16)''' Start Landscape services:

{{{
$ sudo lsctl start
}}}
If any of these files change periodically (e.g. the SSL certificates), you may also wish to set up a cronjob to handle backing-up these files regularly too.
Line 176: Line 62:
'''17)''' Access your server at `https://<servername>` == Restoration ==
Line 178: Line 64:
We recommend that after configuring their Landscape server(s) for archived logging and PITR, administrators of On-Prem Landscape test their recovery procedures, partially to ensure that backups are valid and restorable, and partially to ensure familiarity with such procedures.
Line 179: Line 66:
= Documentation links =

[[https://help.landscape.canonical.com/LDS/ManualInstallation15.11|OPL (LDS) Recommended Deployment 15.11 ]]

[[https://help.landscape.canonical.com/LDS/BackupNotes|LDS Backup Notes ]]

[[http://www.postgresql.org/docs/9.3/static/backup.html|PostreSQL Backup and Restore ]]
 1. Provision a spare server (or servers) and install Landscape server as you have on your production machine(s)
 1. Stop the Landscape application server, and the PostgreSQL cluster on the spare
 1. Copy configuration files (see prior section) to the spare; you may wish to keep a script handy to perform this task in your backup location
 1. If your spare isn't installed from scratch (e.g. if it is installed from an image), remove everything under `/var/lib/landscape/hash-id-databases`
 1. Restore a recent PostgreSQL base-backup onto the spare; this usually involves (re)moving the existing PostgreSQL cluster's data directory (e.g. /var/lib/postgresql/9.5/main) and replacing it with the contents of the base-backup (or un-tarring the base-backup into it, if tar format was chosen); ensure file ownership and modes are preserved!
 1. Construct an appropriate `recovery.conf` file in the new PostgreSQL cluster's data directory; a template for this can be found in `/usr/share/postgresql/<pg-version>/main/recovery.conf.sample`
 1. Start the PostgreSQL cluster on the spare and watch the PostgreSQL logs to ensure recovery proceeds by retrieving and replaying WAL logs
 1. Once recovery is complete, run `/opt/canonical/landscape/scripts/hash-id-databases.sh` to regenerate the hash databases cache
 1. Finally, start the Landscape application server on the spare and test it to verify correct operation


Overview

On-Prem Landscape consists (at a high level) of several stateful components, all of which need to be roughly in sync to guarantee correct functioning of the system as a whole. Specifically:

  • The Landscape server, at least 6 PostgreSQL databases, a cache of hash databases, and a variety of configuration files.
  • The Landscape clients, comprising several SQLite databases for tracking package states.

Given the wide variety of clients (from physical hardware, to VMs, to containers, some of which may be permanent and others merely temporary), backup of clients (if required at all) is beyond the scope of this document.

Backup of the Landscape server must be performed in such a fashion as to permit restoration to its latest possible state. Hence, the only supported backup option is one that permits point-in-time recovery (PITR hereafter).

We strongly recommend that administrators of an On-Prem Landscape instance first familiarize themselves with PostgreSQL's archived logging and PITR facilities. Some syntactic configuration changes have occurred across PostgreSQL versions, hence you should select the documentation for your particular PostgreSQL server version:

Backup & Retention Policy

Before configuring your PostgreSQL instance for continuous archiving and PITR, it is important to decide on a backup policy:

  1. When should base backups be taken? (recommendation: daily or weekly depending on volume of WAL logs, and desired recovery speed)
  2. How many base backups should be retained at any given time? (this will dictate the earliest point in time to which you can initially restore)
  3. Where should WAL logs be archived to? (recommendation: a separate machine or some form of networked, but secure storage)
  4. Where should base backups be stored (recommendation: the same machine as the WAL archive so that all materials necessary for restoration are available in one place)

Although it is possible to backup and archive on the same machine as the PostgreSQL server, we recommend a separate machine is used for base backup and archived log storage to permit restoration in the case the server becomes inaccessible for whatever reason. We further recommend that other files required to restore the Landscape application server (configuration files which will be listed below) are also copied to this location to permit recovery of the entire service from one place.

PostgreSQL Configuration

In the postgresql.conf configuration file, set wal_level, archive_mode, and archive_command according to the PostgreSQL documentation for your server's version. As recommended by the PostgreSQL documentation, test your archive_command operates correctly in all circumstances, including returning the correct exit codes.

Once you are confident the configuration is correct, restart the PostgreSQL service to activate archived logging. Monitor the archive destination to ensure logs begin to appear there (if your server is very low traffic, you may wish to use the archive_timeout setting to force archiving of partial logs after a timeout).

When WAL logs are being archived successfully, construct a script that executes pg_basebackup and stores the result in your base backup storage destination. Test that this operates correctly as the cluster owner (typically postgres), then add a cronjob to periodically execute this script (as the cluster owner).

Note that there is no need to take Landscape offline to perform these backups. In fact, pg_basebackup can only execute when the cluster is up anyway. There is no need to worry about inconsistency between Landscape's various databases either: a base backup represents the state of the cluster (across all databases within it) at the instant the backup starts.

Server Configuration Files

The following files should also be copied from your Landscape server(s) to your backup destination to ensure restoration of the Landscape application server is also possible:

  • /etc/landscape/* - Landscape configuration files, and the On-Prem Landscape license)

  • /etc/default/landscape-server - Specifies which Landscape application services start on a given machine

  • /etc/apache2/sites-available/<server-name> - the Landscape Apache vhost configuration file, usually named after the FQDN of the server

  • /etc/ssl/certs/<landscape-cert> - the Landscape server X.509 certificate

  • /etc/ssl/private/<landscape-cert> - the Landscape server X.509 key file

  • /etc/postgresql/<pg-version>/main/* - various PostgreSQL configuration files

Optionally, you may also wish to backup the following files. They are not required for normal operation of Landscape server, but may provide additional information in the case of service outages:

  • /var/log/landscape-server/* - the Landscape server log files

If any of these files change periodically (e.g. the SSL certificates), you may also wish to set up a cronjob to handle backing-up these files regularly too.

Restoration

We recommend that after configuring their Landscape server(s) for archived logging and PITR, administrators of On-Prem Landscape test their recovery procedures, partially to ensure that backups are valid and restorable, and partially to ensure familiarity with such procedures.

  1. Provision a spare server (or servers) and install Landscape server as you have on your production machine(s)
  2. Stop the Landscape application server, and the PostgreSQL cluster on the spare
  3. Copy configuration files (see prior section) to the spare; you may wish to keep a script handy to perform this task in your backup location
  4. If your spare isn't installed from scratch (e.g. if it is installed from an image), remove everything under /var/lib/landscape/hash-id-databases

  5. Restore a recent PostgreSQL base-backup onto the spare; this usually involves (re)moving the existing PostgreSQL cluster's data directory (e.g. /var/lib/postgresql/9.5/main) and replacing it with the contents of the base-backup (or un-tarring the base-backup into it, if tar format was chosen); ensure file ownership and modes are preserved!
  6. Construct an appropriate recovery.conf file in the new PostgreSQL cluster's data directory; a template for this can be found in /usr/share/postgresql/<pg-version>/main/recovery.conf.sample

  7. Start the PostgreSQL cluster on the spare and watch the PostgreSQL logs to ensure recovery proceeds by retrieving and replaying WAL logs
  8. Once recovery is complete, run /opt/canonical/landscape/scripts/hash-id-databases.sh to regenerate the hash databases cache

  9. Finally, start the Landscape application server on the spare and test it to verify correct operation

LDS/Backup-Restore-Notes (last edited 2018-10-02 19:29:24 by waveform)