mattintosh note

Hello Raspberry Pi!

Amazon Elasticsearch Service 6.x 用に curator をインストールする

容量削減のためインデックス自動削除に curator を導入することにした。

Amazon Elasticsearch Service のバージョンは「6.2.3」。EC2 インスタンスの AMI は「Ubuntu Server 18.04 LTS (HVM), SSD Volume Type - ami-07ad4b1c3af1ea214」。

$ curl -sq 'https://search-XXXX-XXXX-XXXXXXXXXXXXXXXXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com'
{
  "name" : "_XXXXXX",
  "cluster_name" : "XXXXXXXXXXXX:XXXX-XXXX",
  "cluster_uuid" : "XXXXXXXXXXXXXXXXXXXXXX",
  "version" : {
    "number" : "6.2.3",
    "build_hash" : "XXXXXXX",
    "build_date" : "2018-07-16T10:43:54.041989Z",
    "build_snapshot" : false,
    "lucene_version" : "7.2.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
$ cat /etc/ec2_version 
Ubuntu 18.04.1 LTS (Bionic Beaver)

標準の APT でインストール出来る elasticsearch-curator はバージョンが 5.2.0 で、Amazon Elasticsearch Service 6.2.3 とは互換性エラーが発生するため 5.6.0 をインストールする必要があった。

www.elastic.co

DEB パッケージからインストールする場合

$ wget 'https://packages.elastic.co/curator/5/debian9/pool/main/e/elasticsearch-curator/elasticsearch-curator_5.6.0_amd64.deb'
$ sudo dpkg -i elasticsearch-curator_5.6.0_amd64.deb

リポジトリを登録してインストールする場合

$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo 'deb [arch=amd64] https://packages.elastic.co/curator/5/debian stable main' | sudo tee /etc/apt/sources.list.d/curator.list
deb [arch=amd64] https://packages.elastic.co/curator/5/debian stable main
$ sudo apt-get update
$ sudo apt-get install elasticsearch-curator

pip3 でインストールする場合

pip3 でも 5.6.0 がインストール可能だったので試してみたところ特に問題なかった。

$ pip3 install elasticsearch-curator

curator_cli でのテスト

curator_cli でひとまずインデックスが拾えるかどうかテストする。

Usage: curator_cli [OPTIONS] COMMAND [ARGS]...

Options:
  --config PATH       Path to configuration file. Default:
                      ~/.curator/curator.yml
  --host TEXT         Elasticsearch host.
  --url_prefix TEXT   Elasticsearch http url prefix.
  --port TEXT         Elasticsearch port.
  --use_ssl           Connect to Elasticsearch through SSL.
  --certificate TEXT  Path to certificate to use for SSL validation.
  --client-cert TEXT  Path to file containing SSL certificate for client auth.
  --client-key TEXT   Path to file containing SSL key for client auth.
  --ssl-no-validate   Do not validate SSL certificate
  --http_auth TEXT    Use Basic Authentication ex: user:pass
  --timeout INTEGER   Connection timeout in seconds.
  --master-only       Only operate on elected master node.
  --dry-run           Do not perform any changes.
  --loglevel TEXT     Log level
  --logfile TEXT      log file
  --logformat TEXT    Log output format [default|logstash|json].
  --version           Show the version and exit.
  --help              Show this message and exit.

Commands:
  alias             Add/Remove Indices to/from Alias
  allocation        Shard Routing Allocation
  close             Close Indices
  delete_indices    Delete Indices
  delete_snapshots  Delete Snapshots
  forcemerge        forceMerge Indices (reduce segment count)
  open              Open Indices
  replicas          Change Replica Count
  restore           Restore Indices
  rollover          Rollover Index associated with Alias
  show_indices      Show Indices
  show_snapshots    Show Snapshots
  shrink            Shrink Indices to --number_of_shards
  snapshot          Snapshot Indices

Amazon Elasticsearch Service は 9200 番ポートではなく 443 番ポートだが --host https:// であれば --port 443--use_ssl の指定が無くてもいいらしい。

$ curator_cli --host https://search-XXXX-XXXX-XXXXXXXXXXXXXXXXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com show_indices
.kibana
 :
 :
$ curator_cli --host search-XXXX-XXXX-XXXXXXXXXXXXXXXXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com --port 443 --use_ssl show_indices
.kibana
 :
 :

curator でテスト

~/.curator/curator.yml を作成。テストなので項目は最低限で済ませる。

client:
  hosts:
    - search-XXXX-XXXX-XXXXXXXXXXXXXXXXXXXXXXXXXX.ap-northeast-1.es.amazonaws.com:443
  use_ssl: True

~/.curator/delete_indices.yml を作成。logstash- で始まる日付ベースのインデックスを1日分残して削除する。

actions:
  1:
    action: delete_indices
    description: delete old logstash indexes.
    options:
      disable_action: False
      ignore_empty_list: True
    filters:
    - filtertype: pattern
      kind: prefix
      value: logstash-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 1

テストなので --dry-run で実行。今日は 2018-11-21 なので 2018-11-20 のインデックスが削除される。

$ curator --dry-run ~/.curator/delete_indices.yml 
2018-11-21 22:20:39,436 INFO      Preparing Action ID: 1, "delete_indices"
2018-11-21 22:20:39,494 INFO      Trying Action ID: 1, "delete_indices": delete old logstash indexes.
2018-11-21 22:20:39,600 INFO      DRY-RUN MODE.  No changes will be made.
2018-11-21 22:20:39,600 INFO      (CLOSED) indices may be shown that may not be acted on by action "delete_indices".
2018-11-21 22:20:39,600 INFO      DRY-RUN: delete_indices: logstash-2018.11.20 with arguments: {}
2018-11-21 22:20:39,600 INFO      Action ID: 1, "delete_indices" completed.
2018-11-21 22:20:39,600 INFO      Job completed.

あとは保持期間を本来の運用に合わせて cron に登録しておく。


以下、elasticsearch-curator 5.2.0 での検証結果。

標準のリポジトリから elasticsearch-curator パッケージをインストール。

$ sudo apt-get install elasticsearch-curator
$ curator_cli --version
curator_cli, version 5.2.0

互換性エラーが発生した。

$ curator_cli show_indices
2018-11-21 20:33:05,764 ERROR     Elasticsearch version 6.2.3 incompatible with this version of Curator (5.2.0)
Traceback (most recent call last):
  File "/usr/bin/curator_cli", line 11, in <module>
    load_entry_point('elasticsearch-curator==5.2.0', 'console_scripts', 'curator_cli')()
  File "/usr/lib/python2.7/dist-packages/curator/curator_cli.py", line 5, in main
    cli(obj={})
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/dist-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/curator/singletons.py", line 531, in show_indices_singleton
    client = get_client(**c_args)
  File "/usr/lib/python2.7/dist-packages/curator/utils.py", line 800, in get_client
    'Error: {0}'.format(e)
elasticsearch.exceptions.ElasticsearchException: Unable to create client connection to Elasticsearch.  Error: Elasticsearch version 6.2.3 incompatible with this version of Curator (5.2.0)

Version Compatibility | Curator Reference [5.2] | Elastic を確認してみると Curator 5.2 は ES 5.x 系までの記載しかなかった。

f:id:mattintosh4:20181121203926p:plain

2018-11-21 時点の現行バージョンは 5.6.0 なのでそっちならいけるかと思い確認してみと注釈に AWS ES (which is different from installing Elasticsearch on your own EC2 instances) version 5.3 officially supports Curator. とは書いてあるが AWS ES 6.2 に関する記載は無かった。

f:id:mattintosh4:20181121204905p:plain