Skip to content

Complete Backup and Recovery under Linux


A repository is required for both, creating and restoring backups. You have to set up the repository in advance. The procedure for a complete backup divides into the following steps:

  1. Source system: Creating backups

    1. Register a repository for the backups or determine information about an existing backup reposistory.
    2. Set up a snapshot policy for a complete backup, if not already existing.
    3. Create a manual snapshot/backup independent of the configured schedule.
  2. Target system: Full Recovery of the database from a complete backup

    1. Remove the existing database and create a new "empty" database.
    2. Register the repository.
    3. Restore the configuration from the latest snapshot/backup.
    4. Activate the security, if required.
    5. Restore the data.
    6. Grant access to Elasticsearch from outside again (if you blocked it for this update).

Source System: Creating Complete Snapshots/Backups

Registering a Repository

See also Snapshot Repositories

  1. On the management server, stop Elasticsearch and Kibana:

    • Kibana:

      sudo systemctl stop seal-kibana
      
    • Elasticsearch:

      • Version 7.17.0:

        sudo systemctl stop seal-elasticsearch
        
      • Previous versions:

        sudo systemctl stop elasticsearch
        
  2. Customize the Elasticsearch configuration:

    1. In and editor, open the Elasticsearch configuration file:

      • Elasticsearch:

        • Version 7.17.0:

          /opt/seal/etc/seal-elasticsearch/elasticsearch.yml
          
        • Previous versions:

          /etc/elasticsearch/elasticsearch.yml
          
    2. Enter the path of the repository for the backups, e.g. backups.

      The first two lines belong to the default configuration and are used for orientation.

      • Version 7.17.0.

        path.data: /opt/seal/data/seal-elasticsearch
        path.logs: /var/log/seal/seal-elasticsearch
        path.repo: /opt/seal/data/seal-elasticsearch/backups
        
      • Previous versions:

        path.data: /var/lib/elasticsearch
        path.logs: /var/log/elasticsearch
        path.repo: /var/lib/elasticsearch/backups
        
    3. Save the file and exit.

  3. On the management server, restart Elasticsearch and Kibana:

    • Elasticsearch:

      • Version 7.17.0:

        sudo systemctl start seal-elasticsearch
        
      • Previous versions:

        sudo systemctl start elasticsearch
        
    • Kibana:

      sudo systemctl start seal-kibana
      
  4. Register a repository for storing snapshots:

    1. In a browser, open the DevTools Console of Kibana:

      http://localhost:5601/app/dev_tools#/console
      

    Hint - copy the commmands

    1. Copy the following sample commands one by one.

      You can use the browser cache to copy, modify and save the commands.

    2. Paste the commands in the DevTools Console.

    3. Replace the sample names by your own names.

    4. Execute the commands by clicking the green arrow on the right in the first line of the command.

    1. Register the repository:

      The following command registers a repository named my_repository.

      This is a subdirectory of my_repository_location in the backups repository path you have defined in the path.repo key in the elasticsearch.yml file before.

      Alternatively, you can specify an absolute path here, which has to start with the repository path.

      PUT /_snapshot/my_repository
      {
        "type": "fs",
        "settings": {
          "location": "my_repository_location"
        }
      }
      
    2. Verify the repository to grant access to the linked file system:

      This verifies access to the linked file system.

      POST /_snapshot/my_repository/_verify
      

Setting up a Snapshot Policy

Set up a snapshot policy for a full backup.

  1. In a browser, open the DevTools Console of Kibana:

    http://localhost:5601/app/dev_tools#/console
    
  2. Set up a snapshot policy for a full backup.

    With this snapshot policy, you back up all data and associated database configurations (global state). The snapshots created are suitable for completely restoring the last backed up state.

    PUT /_slm/policy/nightly-global-snapshots
    {
      "schedule": "0 30 1 * * ?",
      "name": "<nightly-global-snap-{now/d}>",
      "repository": "my_repository",
      "config": {
        "indices": "*",
        "include_global_state": true
      },
      "retention": {
        "expire_after": "30d",
        "min_count": 5,
        "max_count": 50
      }
    }
    

Or: Determining the Repository Information

If repository and snapshot policies have been set up some time ago, you can find out the repository name and path.

  1. Determine the name of the repository:

    • Determine the configuration of the nightly-global-snapshots snapshot policy:

      GET /_slm/policy/nightly-global-snapshots
      
    • Or determine all snapshot policies:

      GET /_slm/policy
      

    In the following sample response, you find the name of the associated repository in the policy.repository key:

    {
      "nightly-global-snapshots" : {
        "version" : 1,
        "modified_date_millis" : 1668622663378,
        "policy" : {
          "name" : "<nightly-global-snap-{now/d}>",
          "schedule" : "0 30 1 * * ?",
          "repository" : "my_repository",
        ...
        }
      }
    }
    
  2. Determine the path of the repository in the file system:

    Use the the repository name determined above to request the associated configuration:

    GET /_snapshot/my_repository
    

    In the following sample response, you find the name of the associated repository path in the settings.location key:

    {
      "my_repository" : {
        "type" : "fs",
        "uuid" : "d0Wq2tr6Ssm1Mv_wDC8dNg",
        "settings" : {
          "location" : "my_repository_location"
        }
      }
    }
    

Creating an Unscheduled Snapshot/Backup

You can create snapshots/backups independent of the configured schedule.

  1. In the DevTools Console of Kibana:

    1. In a browser, open the DevTools Console of Kibana:

      http://localhost:5601/app/dev_tools#/console
      
    2. Configure an unscheduled snapshot/backup:

      POST /_slm/policy/nightly-global-snapshots/_execute
      
  2. In the Kibana user interface:

    1. In a Browser, open the the Kibana user interface:

      http://localhost:5601/app/management/data/snapshot_restore/policies
      
    2. Click the arrow on the right in the line with the snapshot policy set up. Its tooltip shows run now.

  3. Check the result:

    1. In a Browser, open the the Kibana user interface:

      http://localhost:5601/app/management/data/snapshot_restore/policies
      
    2. Switch to the view of available snapshots:

      http://localhost:5601/app/management/data/snapshot_restore/snapshots
      

      You will find the newly created snapshot in the list.

    3. Click the snapshot.

      The information about the snapshot is displayed on the right, and the list of saved indices underneath.


Target system: Complete Recovery from a Full Backup

Caution - new, fresh database required

The existing Elasticsearch database needs to be discarded!

You do not need to merge any data, you have to restore the Elasticsearch database from the latest backed up state. Ideally, the database's state is as fresh as possible.


Removing the Existing and Creating a New, Empty Database

  1. On the PLOSSYS server, stop Filebeat:

    sudo systemctl stop seal-filebeat
    
  2. On the management server, stop Elasticsearch and Kibana::

    • Kibana:

      sudo systemctl stop seal-kibana
      
    • Elasticsearch:

      • Version 7.17.0:

        sudo systemctl stop seal-elasticsearch
        
      • Previous versions:

        sudo systemctl stop elasticsearch
        
  3. Delete or rename the previous database directory "nodes":

    • As "root" user, change to the Elasticsearch data directory:

      • SEAL Elasticstack 7.17.0:

        cd /opt/seal/data/seal-elasticsearch
        
      • Previous versions:

        cd /var/lib/elasticsearch
        
    • Delete or rename the previous (corrupted) database directory:

      • Delete:

        rm -rf nodes
        
      • Rename:

        mv nodes nodes.off
        
  4. Adjust the Elasticsearch configuration:

    Create a new, empty database, into which you can import our backup.

    1. In and editor, open the Elasticsearch configuration file:

      • Elasticsearch:

        • Version 7.17.0:

          /opt/seal/etc/seal-elasticsearch/elasticsearch.yml
          
        • Previous versions:

          /etc/elasticsearch/elasticsearch.yml
          
    2. Deactivate the security mode:

      Initially, a newly created database does not contain any user information. So you cannot log on in a browser. Therefore you need to deactivate the security mode:

      xpack.security.enabled: false
      
    3. Enter the path of the repository for the backups, e.g. backups.

      The first two lines belong to the default configuration and are used for orientation.

      • Version 7.17.0.

        path.data: /opt/seal/data/seal-elasticsearch
        path.logs: /var/log/seal/seal-elasticsearch
        path.repo: /opt/seal/data/seal-elasticsearch/backups
        
      • Previous versions:

        path.data: /var/lib/elasticsearch
        path.logs: /var/log/elasticsearch
        path.repo: /var/lib/elasticsearch/backups
        
    4. Deactivate the access from outside:

      Any data Filebeat tries to upload to the Elasticsearch database during the recovery process will be lost.

      Instead of stopping every single Filebeat you have installed on remote systems, you can dectivate the access to the database from outside:

      Replace

      network.host: 0.0.0.0
      

      by

      network.host: localhost
      

      Caution - later reactivation

      Keep in mind that you must reactivate the access from outside after finishing the restauration!

    5. Save the file and exit.

  5. On the management server, restart Elasticsearch and Kibana:

    • Elasticsearch:

      • Version 7.17.0:

        sudo systemctl start seal-elasticsearch
        
      • Previous versions:

        sudo systemctl start elasticsearch
        
    • Kibana:

      sudo systemctl start seal-kibana
      
  6. On the PLOSSYS server, restart Filebeat:

    sudo systemctl start seal-filebeat
    

Registering the Backup Repository

  1. Register the repository that contains the snapshots to be restored:

    1. In a browser, open the DevTools Console of Kibana:

      http://localhost:5601/app/dev_tools#/console
      
    2. Register the repository:

      The following command registers a repository named my_repository.

      This is a subdirectory of my_repository_location in the backups repository path you have defined in the path.repo key in the elasticsearch.yml file before.

      Alternatively, you can specify an absolute path here, which has to start with the repository path.

      PUT /_snapshot/my_repository
      {
        "type": "fs",
        "settings": {
          "location": "my_repository_location"
        }
      }
      
    3. Verify the repository to grant access to the linked file system:

      This verifies access to the linked file system.

      POST /_snapshot/my_repository/_verify
      

Restoring the Configuration

Restore the system configuration, i. e. users, index patterns, index templates, ... from the latest snapshot/backup

  1. In a browser, open the DevTools Console of Kibana:

    http://localhost:5601/app/dev_tools#/console
    

    Hint - copy the commmands

    1. Copy the following sample commands one by one.

      You can use the browser cache to copy, modify and save the commands.

    2. Paste the commands in the DevTools Console.

    3. Replace the sample names by your own names.

    4. Execute the commands by clicking the green arrow on the right in the first line of the command.

  2. Find the latest snapshot:

    GET /_snapshot/my_repository/nightly-global-snap*?size=1&sort=start_time&order=desc
    

    Or search for the 10 most recent backups of various snapshot policies in descending order:

    GET /_snapshot/my_repository/*?size=10&sort=start_time&order=desc
    
  3. Select the latest snapshot from the list, e. g. nightly-global-snap-2022.1-16-8fn7qrfsrguogphyhciynw from snapshots[0].snapshot:

    {
      "snapshots" : [
        {
          "snapshot" : "nightly-global-snap-2022.1-16-8fn7qrfsrguogphyhciynw",
          "uuid" : "2765_TIYRvywZ-UUT1yHUg",
          "repository" : "my_repository",
          ...
        }
      ]
    }
    
  4. Start the restauration of the configuration excluding the SEAL indices:

    Caution - configuration only

    You must disable the restauration of the aliases here. The system aliases have already been created when the new, empty Elasticsearch database has been built. These aliases must not be overwritten.

    POST /_snapshot/my_repository/nightly-global-snap-2022.1-16-8fn7qrfsrguogphyhciynw/_restore
    {
      "indices": "-*",
      "include_global_state": true,
      "include_aliases": false
    }
    

Enabling the Security

Hint - only if ...

The following steps are only necessary, if the security had been activated in the previous system and you want to activate it again.

Alternatively, you can isolate the server by means of a firewall and reactivate the security after the complete recovery of all data.

  1. On the management server, stop Elasticsearch and Kibana:

    • Kibana:

      sudo systemctl stop seal-kibana
      
    • Elasticsearch:

      • Version 7.17.0:

        sudo systemctl stop seal-elasticsearch
        
      • Previous versions:

        sudo systemctl stop elasticsearch
        

    ```

  2. In and editor, open the Elasticsearch configuration file:

    • Elasticsearch:

      • Version 7.17.0:

        /opt/seal/etc/seal-elasticsearch/elasticsearch.yml
        
      • Previous versions:

        /etc/elasticsearch/elasticsearch.yml
        
  3. Activate the security mode:

    With the previous restauration of the system configuration, the users, etc. have been restored as well. Therefore you can activate the security mode:

    xpack.security.enabled: true
    
  4. On the management server, restart Elasticsearch and Kibana:

    • Elasticsearch:

      • Version 7.17.0:

        sudo systemctl start seal-elasticsearch
        
      • Previous versions:

        sudo systemctl start elasticsearch
        
    • Kibana:

      sudo systemctl start seal-kibana
      
  5. In a browser, open the DevTools Console of Kibana:

    http://localhost:5601/app/dev_tools#/console
    

    Now you need to enter user and password.


Restoring the Data

Restore the SEAL indices with the same snapshot as the configuration.

  1. In a browser, open the DevTools Console of Kibana:

    http://localhost:5601/app/dev_tools#/console
    
  2. Start the restauration of the data including the aliases:

    Caution - with aliases

    You must enable the restauration of the aliases here. Otherwise the index lifecycle policies can no longer work properly and indices can become too large because they are not rolled over anymore.

    POST /_snapshot/my_repository/nightly-global-snap-2022.1-16-8fn7qrfsrguogphyhciynw/_restore
    {
      "indices": "seal-*",
      "include_global_state": false,
      "include_aliases": true
    }
    

Enabling Access to Elasticsearch

  1. On the management server, stop Elasticsearch and Kibana:

    • Kibana:

      sudo systemctl stop seal-kibana
      
    • Elasticsearch:

      • Version 7.17.0:

        sudo systemctl stop seal-elasticsearch
        
      • Previous versions:

        sudo systemctl stop elasticsearch
        
  2. In and editor, open the Elasticsearch configuration file:

    • Elasticsearch:

      • Version 7.17.0:

        /opt/seal/etc/seal-elasticsearch/elasticsearch.yml
        
      • Previous versions:

        /etc/elasticsearch/elasticsearch.yml
        
  3. Reactivate the access from outside:

    Replace

    network.host: localhost
    

    by

    network.host: 0.0.0.0
    
  4. Save the file and exit.

  5. On the management server, restart Elasticsearch and Kibana:

    • Elasticsearch:

      • Version 7.17.0:

        sudo systemctl start seal-elasticsearch
        
      • Previous versions:

        sudo systemctl start elasticsearch
        
    • Kibana:

      sudo systemctl start seal-kibana
      

Back to top