Skip to content

Elasticsearch Database Backup

Elasticsearch Documentation

You will find a more detailed description of how to back up and restore the database in the original Elasticsearch documentation: Restore a Snapshot.

The following chapter describes how to back up and restore specific indices in the Elasticsearch database. In addition, we recommend you also back up the Elasticsearch configuration directory on a daily basis at the file system level: Back up Configuration Files.

Backup and snapshot are used interchangeably here.

Outlining the Relationships

  1. In the elasticsearch.yml configuration file, set the repository path:

    path.repo

  2. Register a snapshot repository. This is required for defining a snapshot lifecycle policy.

  3. Define a snapshot lifecycle policy.

  4. Create a snapshot.

    The snapshots are created according to the schedule of the snapshot lifecycle policy and stored in the linked snapshot repository.

  5. Restore a snapshot, if necessary, via the Kibana user interface.

Snapshot Repositories

For storing backups, you can specify one or more repositories in Elasticsearch. These are used for defining snapshot policies.

Literature

For details on the definition of repositories, refer to the original documentation of the software producer:

Register a snapshot repository.

Repository Path

In the elasticsearch.yml configuration file, you have to define the allowed locations for repositories in advance.

Example - allowed locations

Elasticsearch is allowed to store repositories in the /mount/backups and /mount/long_term_backups directories. You can specify the 2 directories

  • in separate lines:

    path.repo:
    - /mount/backups
    - /mount/long_term_backups

  • in one single line:

    path.repo: ["/mount/backups","/mount/long_term_backups"]

Caution - existing directories with writing access

These directories must already be existing and Elasticsearch must be granted writing access.

Under Linux you can grant wrinting access with the following command:

sudo chown -R elasticsearch:elasticsearch /mount/backups

Registering a Snapshot Repository

You can register snapshot repositories

  • interactively via the Kibana user interface:

    http://localhost:5601/app/management/data/snapshot_restore/repositories

  • with the appropriate command via the DevTools Console in the Kibana user interface:

    http://localhost:5601/app/dev_tools#/console

Example - registering a repository

The my_repository repository ist to be registered with the path /mount/backups/my_repository_location:

PUT _snapshot/my_repository
{
    "type": "fs",
    "settings": {
    "location": "/mount/backups/my_repository_location"
    }
}

Use the my_repository name of the snapshot repository in the snapshot policies below.

Caution - database failure

In case of a complete failure of the database contents, also the information about the registered repositories is lost. You have to reregister the snapshot directory with the above command first, before you can restore backups.

Verifying a Repository

We recommend verifying the connection to a newly registered snapshot repository. You can co this

  • in the Kibana user interface by clicking the Verify repository button:

    http://localhost:5601/app/management/data/snapshot_restore/repositories/my_repository

  • in the DevTools Console with the following command:

    POST /_snapshot/my_repository/_verify

Creating Backups with Snapshot Policies

Hint - repositories before policies

You need snapshot repositories to define a storage location, before you define snapshot policies.

You can define any snapshot policy to back up specific data either together or separately. You can create snapshot policies

  • in the Kibana user interface:

    http://localhost:5601/app/management/data/snapshot_restore/policies

  • in the DevTools Console:

    http://localhost:5601/app/dev_tools#/console

Hint - incremental snapshots

In Elasticsearch, the snapshots created based on a certain policy are incremental. This means that only the differences to the previous snapshot are saved in the new snapshot.

... for Security Settings

In the Kibana DevTools Console, you can define a snapshot lifecycle policy to back up users, roles and role mappings.

Example of a backup definition for security settings

Indices belonging to the security feature are to be backed up every night at 1:00. The created snapshots are to be stored in the already registered my_repository snapshot repository. Independent of the allowed maximum age of 30 days, a minimum of 5 and a maximum of 50 snapshots are stored.

PUT _slm/policy/nightly-security-snapshots
{
  "schedule": "0 0 1 * * ?",
  "name": "<nightly-security-snap-{now/d}>",
  "repository": "my_repository",
  "config": {
    "indices": "-.*",
    "include_global_state": false,
    "feature_states": [
      "security"
    ]
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}

... for Kibana Settings

In the Kibana DevTools Console, you can define a snapshot lifecycle policy to back up Kibana settings, e. g. searches and dashboards.

Example of a backup definition for Kibana settings

Indices belonging to the kibana feature are to be backed up every night at 0:30. The created snapshots are to be stored in the my_repository snapshot repository. Independent of the allowed maximum age of 30 days, a minimum of 5 and a maximum of 50 snapshots are stored.

PUT _slm/policy/nightly-kibana-snapshots
{
  "schedule": "0 30 0 * * ?",
  "name": "<nightly-kibana-snap-{now/d}>",
  "repository": "my_repository",
  "config": {
    "indices": "-.*",
    "include_global_state": false,
    "feature_states": [
      "kibana"
    ]
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 5,
    "max_count": 50
  }
}

... for SEAL Log, Accounting and Audit Data

Standard log messages of a system and accounting and audit data need to be stored for different periods of time.

Below you will find several examples for snapshot lifecycle policies. You can copy them to use them as templates and modify them according to your needs.

Example 1 - hourly backups

This snapshot lifecycle policy takes up to 24 hourly backups, which are stored for one day:

PUT _slm/policy/hourly-audit-snapshots
{
  "name": "<hourly-audit-snap-{now/d}>",
  "schedule": "0 0 * * * ?",
  "repository": "my_repository",
  "config": {
    "indices": "seal-*-audit",
    "include_global_state": false
  },
  "retention": {
    "expire_after": "1d",
    "min_count": 1,
    "max_count": 24
  }
}

Example 2 - daily backups

This snapshot lifecycle policy takes a backup every night at 23:45, which is stored for one month:

PUT _slm/policy/daily-audit-snapshots
{
  "name": "<daily-audit-snap-{now/d}>",
  "schedule": "0 45 23 * * ?",          
  "repository": "my_repository",
  "config": {
    "indices": "seal-*-audit",
    "include_global_state": false
  },
  "retention": {
    "expire_after": "30d",
    "min_count": 1,
    "max_count": 31
  }
}

Example 3 - monthly backups

This snapshot lifecycle policy takes a backup at the first day of every month, which are stored for one year:

PUT _slm/policy/monthly-audit-snapshots
{
  "name": "<monthly-audit-snap-{now/d}>",
  "schedule": "0 56 23 1 * ?",            
  "repository": "my_repository",
  "config": {
    "indices": "seal-*-audit",
    "include_global_state": false
  },
  "retention": {
    "expire_after": "366d",
    "min_count": 1,
    "max_count": 12
  }
}

Hint- long term data storage

We recommend setting up several coordinated snapshot lifecycle policies for data that have to be stored for a longer period and must not get lost.

In the examples above all snapshots are stored in the directory that is linked to the my_repository snapshot repository. You can differentiate between the snapshots of the different Elasticsearch indices by their names.

If, additionally, you wish to backup the snapshot directory itself by means of your operating system, we recommend you use different snapshot repositories.

Literature

For details on backing up snapshot repositories, refer to the original documentation of the software producer:

Backup a repository.

... for Elasticsearch Cluster Configurations

Literature

For details on cluster configurations refer to the the original documentation of the software producer:

Back to top