Complete Backup and Recovery under Linux¶

A repository is required for both, creating and restoring backups. You have to set up the repository in advance. The procedure for a complete backup divides into the following steps:

Source system: Creating backups
1. Register a repository for the backups or determine information about an existing backup reposistory.
2. Set up a snapshot policy for a complete backup, if not already existing.
3. Create a manual snapshot/backup independent of the configured schedule.
Target system: Full Recovery of the database from a complete backup
1. Remove the existing database and create a new "empty" database.
2. Register the repository.
3. Restore the configuration from the latest snapshot/backup.
4. Activate the security, if required.
5. Restore the data.
6. Grant access to Elasticsearch from outside again (if you blocked it for this update).

Source System: Creating Complete Snapshots/Backups¶

Registering a Repository¶

Setting up a Snapshot Policy¶

Set up a snapshot policy for a full backup.

In a browser, open the DevTools Console of Kibana:
```
http://localhost:5601/app/dev_tools#/console
```

Set up a snapshot policy for a full backup.

With this snapshot policy, you back up all data and associated database configurations (global state). The snapshots created are suitable for completely restoring the last backed up state.

PUT /_slm/policy/nightly-global-snapshots
{
  "schedule": "0 30 1 * * ?",
  "name": "<nightly-global-snap-{now/d}>",
  "repository": "my_repository",
  "config": {
    "indices": "*",
    "include_global_state": true
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}

Or: Determining the Repository Information¶

If repository and snapshot policies have been set up some time ago, you can find out the repository name and path.

Determine the name of the repository:

Determine the configuration of the nightly-global-snapshots snapshot policy:
```
GET /_slm/policy/nightly-global-snapshots
```
Or determine all snapshot policies:
```
GET /_slm/policy
```

In the following sample response, you find the name of the associated repository in the policy.repository key:

{
  "nightly-global-snapshots" : {
    "version" : 1,
    "modified_date_millis" : 1668622663378,
    "policy" : {
      "name" : "<nightly-global-snap-{now/d}>",
      "schedule" : "0 30 1 * * ?",
      "repository" : "my_repository",
    ...
    }
  }
}

Determine the path of the repository in the file system:

Use the the repository name determined above to request the associated configuration:
```
GET /_snapshot/my_repository
```
In the following sample response, you find the name of the associated repository path in the settings.location key:
```
{
  "my_repository" : {
    "type" : "fs",
    "uuid" : "d0Wq2tr6Ssm1Mv_wDC8dNg",
    "settings" : {
      "location" : "my_repository_location"
    }
  }
}
```

Creating an Unscheduled Snapshot/Backup¶

You can create snapshots/backups independent of the configured schedule.

In the DevTools Console of Kibana:
1. In a browser, open the DevTools Console of Kibana:
```
http://localhost:5601/app/dev_tools#/console
```
2. Configure an unscheduled snapshot/backup:
```
POST /_slm/policy/nightly-global-snapshots/_execute
```
In the Kibana user interface:
1. In a Browser, open the the Kibana user interface:
```
http://localhost:5601/app/management/data/snapshot_restore/policies
```
2. Click the arrow on the right in the line with the snapshot policy set up. Its tooltip shows run now.
Check the result:
1. In a Browser, open the the Kibana user interface:
```
http://localhost:5601/app/management/data/snapshot_restore/policies
```
2. Switch to the view of available snapshots:
```
http://localhost:5601/app/management/data/snapshot_restore/snapshots
```
  You will find the newly created snapshot in the list.
3. Click the snapshot.
  
  The information about the snapshot is displayed on the right, and the list of saved indices underneath.

Target system: Complete Recovery from a Full Backup¶

Caution - new, fresh database required

The existing Elasticsearch database needs to be discarded!

You do not need to merge any data, you have to restore the Elasticsearch database from the latest backed up state. Ideally, the database's state is as fresh as possible.

Removing the Existing and Creating a New, Empty Database¶

On the PLOSSYS server, stop Filebeat:
```
sudo systemctl stop seal-filebeat
```
On the management server, stop Elasticsearch and Kibana::
- Kibana:
```
sudo systemctl stop seal-kibana
```
- Elasticsearch:
  - Version 7.17.0:
```
sudo systemctl stop seal-elasticsearch
```
  - Previous versions:
```
sudo systemctl stop elasticsearch
```
Delete or rename the previous database directory "nodes":
- As "root" user, change to the Elasticsearch data directory:
  - SEAL Elastick Stack 7.17.0:
```
cd /opt/seal/data/seal-elasticsearch
```
  - Previous versions:
```
cd /var/lib/elasticsearch
```
- Delete or rename the previous (corrupted) database directory:
  - Delete:
```
rm -rf nodes
```
  - Rename:
```
mv nodes nodes.off
```
Adjust the Elasticsearch configuration:

Create a new, empty database, into which you can import our backup.
1. In and editor, open the Elasticsearch configuration file:
  - Elasticsearch:
    - Version 7.17.0:
      /opt/seal/etc/seal-elasticsearch/elasticsearch.yml
    - Previous versions:
      /etc/elasticsearch/elasticsearch.yml
2. Deactivate the security mode:
  
  Initially, a newly created database does not contain any user information. So you cannot log on in a browser. Therefore you need to deactivate the security mode:
```
xpack.security.enabled: false
```
3. Enter the path of the repository for the backups, e.g. backups.
  
  The first two lines belong to the default configuration and are used for orientation.
  - Version 7.17.0.
```
path.data: /opt/seal/data/seal-elasticsearch
path.logs: /var/log/seal/seal-elasticsearch
path.repo: /opt/seal/data/seal-elasticsearch/backups
```
  - Previous versions:
```
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
path.repo: /var/lib/elasticsearch/backups
```
4. Deactivate the access from outside:
  
  Any data Filebeat tries to upload to the Elasticsearch database during the recovery process will be lost.
  
  Instead of stopping every single Filebeat you have installed on remote systems, you can dectivate the access to the database from outside:
  
  Replace
```
network.host: 0.0.0.0
```
  by
```
network.host: localhost
```
  Caution - later reactivation
  
  Keep in mind that you must reactivate the access from outside after finishing the restauration!
5. Save the file and exit.
On the management server, restart Elasticsearch and Kibana:
- Elasticsearch:
  - Version 7.17.0:
```
sudo systemctl start seal-elasticsearch
```
  - Previous versions:
```
sudo systemctl start elasticsearch
```
- Kibana:
```
sudo systemctl start seal-kibana
```

Registering the Backup Repository¶

Register the repository that contains the snapshots to be restored:
1. In a browser, open the DevTools Console of Kibana:
```
http://localhost:5601/app/dev_tools#/console
```
2. Register the repository:
  
  The following command registers a repository named my_repository.
  
  This is a subdirectory of my_repository_location in the backups repository path you have defined in the path.repo key in the elasticsearch.yml file before.
  
  Alternatively, you can specify an absolute path here, which has to start with the repository path.
```
PUT /_snapshot/my_repository
{
  "type": "fs",
  "settings": {
    "location": "my_repository_location"
  }
}
```
3. Verify the repository to grant access to the linked file system:
  
  This verifies access to the linked file system.
```
POST /_snapshot/my_repository/_verify
```

Restoring the Configuration¶

Restore the system configuration, i. e. users, index patterns, index templates, ... from the latest snapshot/backup

In a browser, open the DevTools Console of Kibana:
```
http://localhost:5601/app/dev_tools#/console
```
Hint - copy the commmands
1. Copy the following sample commands one by one.
  
  You can use the browser cache to copy, modify and save the commands.
2. Paste the commands in the DevTools Console.
3. Replace the sample names by your own names.
4. Execute the commands by clicking the green arrow on the right in the first line of the command.

Find the latest snapshot:

GET /_snapshot/my_repository/nightly-global-snap*?size=1&sort=start_time&order=desc

Or search for the 10 most recent backups of various snapshot policies in descending order:

GET /_snapshot/my_repository/*?size=10&sort=start_time&order=desc

Select the latest snapshot from the list, e. g. nightly-global-snap-2022.1-16-8fn7qrfsrguogphyhciynw from snapshots[0].snapshot:

{
  "snapshots" : [
    {
      "snapshot" : "nightly-global-snap-2022.1-16-8fn7qrfsrguogphyhciynw",
      "uuid" : "2765_TIYRvywZ-UUT1yHUg",
      "repository" : "my_repository",
      ...
    }
  ]
}

Start the restauration of the configuration excluding the SEAL indices:

Caution - configuration only

You must disable the restauration of the aliases here. The system aliases have already been created when the new, empty Elasticsearch database has been built. These aliases must not be overwritten.
```
POST /_snapshot/my_repository/nightly-global-snap-2022.1-16-8fn7qrfsrguogphyhciynw/_restore
{
  "indices": "-*",
  "include_global_state": true,
  "include_aliases": false
}
```

Enabling the Security¶

Hint - only if ...

The following steps are only necessary, if the security had been activated in the previous system and you want to activate it again.

Alternatively, you can isolate the server by means of a firewall and reactivate the security after the complete recovery of all data.

On the management server, stop Elasticsearch and Kibana:
- Kibana:
```
sudo systemctl stop seal-kibana
```
- Elasticsearch:
  - Version 7.17.0:
```
sudo systemctl stop seal-elasticsearch
```
  - Previous versions:
```
sudo systemctl stop elasticsearch
```
```
In and editor, open the Elasticsearch configuration file:
- Elasticsearch:
  - Version 7.17.0:
```
/opt/seal/etc/seal-elasticsearch/elasticsearch.yml
```
  - Previous versions:
```
/etc/elasticsearch/elasticsearch.yml
```
Activate the security mode:

With the previous restauration of the system configuration, the users, etc. have been restored as well. Therefore you can activate the security mode:
```
xpack.security.enabled: true
```
On the management server, restart Elasticsearch and Kibana:
- Elasticsearch:
  - Version 7.17.0:
```
sudo systemctl start seal-elasticsearch
```
  - Previous versions:
```
sudo systemctl start elasticsearch
```
- Kibana:
```
sudo systemctl start seal-kibana
```
In a browser, open the DevTools Console of Kibana:
```
http://localhost:5601/app/dev_tools#/console
```
Now you need to enter user and password.

Restoring the Data¶

Restore the SEAL indices with the same snapshot as the configuration.

In a browser, open the DevTools Console of Kibana:
```
http://localhost:5601/app/dev_tools#/console
```
Start the restauration of the data including the aliases:

Caution - with aliases

You must enable the restauration of the aliases here. Otherwise the index lifecycle policies can no longer work properly and indices can become too large because they are not rolled over anymore.
```
POST /_snapshot/my_repository/nightly-global-snap-2022.1-16-8fn7qrfsrguogphyhciynw/_restore
{
  "indices": "seal-*",
  "include_global_state": false,
  "include_aliases": true
}
```

Enabling Access to Elasticsearch¶

On the management server, stop Elasticsearch and Kibana:
- Kibana:
```
sudo systemctl stop seal-kibana
```
- Elasticsearch:
  - Version 7.17.0:
```
sudo systemctl stop seal-elasticsearch
```
  - Previous versions:
```
sudo systemctl stop elasticsearch
```
In and editor, open the Elasticsearch configuration file:
- Elasticsearch:
  - Version 7.17.0:
```
/opt/seal/etc/seal-elasticsearch/elasticsearch.yml
```
  - Previous versions:
```
/etc/elasticsearch/elasticsearch.yml
```

Reactivate the access from outside:

Replace

network.host: localhost

by

network.host: 0.0.0.0

Save the file and exit.
On the management server, restart Elasticsearch and Kibana:
- Elasticsearch:
  - Version 7.17.0:
```
sudo systemctl start seal-elasticsearch
```
  - Previous versions:
```
sudo systemctl start elasticsearch
```
- Kibana:
```
sudo systemctl start seal-kibana
```
On the PLOSSYS server, restart Filebeat:
```
sudo systemctl start seal-filebeat
```