Upgrade Elasticsearch from 2.x to 5.x

  1. 安裝 plugin
  2. 連線到 localhost:2812/_plugin/elasticsearch-migration
  1. 確定有用 snapshot 備份資料
>curl -XGET http://140.119.163.70:2812/_snapshot
{"_backup":{"type":"fs","settings":{"compress":"true","location":"/mnt/cluster/floodfire-backup/elasticsearch/snapshot/"}}}

> ls /mnt/cluster/floodfire-backup/elasticsearch/snapshot/
index                     meta-20170130-050001.dat  meta-20170501-050001.dat  snap-20170102-050001.dat  snap-20170320-050001.dat
indices                   meta-20170206-050001.dat  meta-20170508-050001.dat  snap-20170109-050001.dat  snap-20170403-050001.dat
meta-20161209-163329.dat  meta-20170213-050001.dat  meta-20170515-050001.dat  snap-20170115-223403.dat  snap-20170410-050001.dat
meta-2016120916.dat       meta-20170220-050001.dat  meta-20170522-050001.dat  snap-20170115-230141.dat  snap-20170417-050001.dat
meta-20161212-050001.dat  meta-20170227-050001.dat  meta-20170529-050001.dat  snap-20170116-050001.dat  snap-20170424-050001.dat
meta-20161219-050001.dat  meta-20170306-050001.dat  meta-20170605-050001.dat  snap-20170130-050001.dat  snap-20170501-050001.dat
meta-20161226-050001.dat  meta-20170313-050001.dat  meta-20170612-050001.dat  snap-20170206-050001.dat  snap-20170508-050001.dat
meta-20170102-050001.dat  meta-20170320-050001.dat  snap-20161209-163329.dat  snap-20170213-050001.dat  snap-20170515-050001.dat
meta-20170109-050001.dat  meta-20170403-050001.dat  snap-2016120916.dat       snap-20170220-050001.dat  snap-20170522-050001.dat
meta-20170115-223403.dat  meta-20170410-050001.dat  snap-20161212-050001.dat  snap-20170227-050001.dat  snap-20170529-050001.dat
meta-20170115-230141.dat  meta-20170417-050001.dat  snap-20161219-050001.dat  snap-20170306-050001.dat  snap-20170605-050001.dat
meta-20170116-050001.dat  meta-20170424-050001.dat  snap-20161226-050001.dat  snap-20170313-050001.dat  snap-20170612-050001.dat
  1. clusters migration check

檢查的結果看起來是有一個動作需要處理才能進行升級,是 X-pack 的 license 中 marvel-enable 這個設定的名稱被重新命名為 xpack.monitoring.enable,這會導致升級後無法使用 marvel (marvel-agent 已經 是 X-pack 中的一部分),所以才需要處理,這個設定是在 config/elasticseach.yml 中

修改完以後,重新啟動 elasticsearch 再檢查一次就沒問題了

因為 kopf 和 sql 這兩個 plugins 被標記維不會再支援(其實是有新的版本),因此也順便移除了

因為 delete-by-query 這個外掛的功能被加入到核心模組中,所以也可以移除的

因為 license 也是 x-pack 中的一部分,所以也移除了

  1. 做 reindexing 檢查
  1. 啟用或關閉 deprecation loggin

看說明沒有說到底需不需要啟用或關閉,但是首頁有說這個功能 only available in 2.x,推測 5.x 應該沒有這個功能,等同於 disabled 吧?!

那既然原本就是關閉,就不開啟試試看

開始升級

查到一個網頁說:

Elasticsearch 2.3.3 提供压缩包和软件包两种安装方式。压缩包只需解压,然后进入 bin 目录运行 Elasticsearch 脚本

該不會其實前面的工作都是為了要以軟件包方式安裝才需要的吧…😂

總之:

  1. download elasticsearch 5.4.1
  2. unzip
    比較一下解壓縮後的檔案和目錄,只有 data 和 logs 不同,logs 應該是系統運作產生的,data 還不確定,du -sh data/ 有 46 G,可能是匯入 elasticsearch 後的文件檔案(所以我想的應該沒錯,ES 和 lucene 依樣不需要資料庫)
  3. 2.4.1 只有幾個地方的 elasticsearch 有開啟設定
    1. network.host: 0.0.0.0
    2. http.port: 2812
    3. http.host: 0.0.0.0
    4. path.repo: ["/mnt/cluster/floodfire-backup/elasticsearch/snapshot"]
      這是 snapshot 的 path
    5. marvel.enable: true
      這個在 5.x 應該是可以不用動,因為用 migration 檢查的時候說 marvel-agent 已經是 x-pack 的一部分了

所以除了 marvel.enable 之外的部分我都加到了 5.4.1 的 config/elasticsearch.yml 中

到這邊使用 ./bin/elasticsearch 已經可以連線了

但是索引檢索會出現:

> {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"platform","index_uuid":"_na_","index":"platform"}],"type":"index_not_found_exception","reason":"no such index","resource.type":"index_or_alias","resource.id":"platform","index_uuid":"_na_","index":"platform"},"status":404}

所我試著 copy data/… 到 5.4.1 (如果以後確定是要有這包東西,其實就不用複製資料檔案,直接用 5.4.1 的 elasticsearch 替換掉 2.4.1 中的檔案就好了),在做一次索引檢所:

DONE!

Elasticsearch Monitoring

Marvel-Agent in Kibana

這個是官方出的監控系統

  1. 在 elasticsearch-2.4.1 中安裝 marvel-agent plugin
  2. restart elasticsearch 測試安裝: http://host:2812/.marvel-es-data-1/_search?pretty
  3. 到 kibana 中安裝 bin/kibana plugin –install elasticsearch/marvel/2.4.5
  4. 確認 kibana/config/kibana.yml 中的 elasticsearch-url 有正確設定
  5. 啟動 kibana
  6. 連線到 port 5601

目前出現這樣的狀況

有可能是:

一般是人为删除(比如在sense插件里执行删除命令)marvel数据,导致marvel采集出错(删除了半天数据,另外半天数据将无法正常采集),不能统计;对于这种情况,等第二天marvel就可以正常使用了。
也有可能是9300端口被占用,marvel默认使用9300端口;对于这种情况,找到9300端口占用进程,kill掉,重启kibana即可。
10 Bad Request, you must reconsidered your request. <400>

正常的情況應該馬上就會出現 elasticsearch 的項目,有一個可能是因為 license expired 了

測試 marvel-agent 的輸出

"license" : {
  "status" : "expired",
  "uid" : "aaaca7d3-ee84-448a-92ab-f57d426f5406",
  "type" : "trial",
  "issue_date" : "2017-02-13T10:12:06.302Z",
  "issue_date_in_millis" : 1486980726302,
  "expiry_date" : "2017-03-15T10:12:06.302Z",
  "expiry_date_in_millis" : 1489572726302,
  "max_nodes" : 1000,
  "issued_to" : "elasticsearch",
  "issuer" : "elasticsearch",
  "hkey" : "3d510ac4c9b5954a11e607d64a62d569468b1283f2e332523f6cfa51db06fa87"
}

這邊有說

At the end of the trial period, you can purchase a subscription to keep using the full functionality of Shield along with Marvel and Watcher.

When your license expires, Shield operates in a degraded mode where access to the Elasticsearch cluster health, cluster stats, and index stats APIs is blocked. Shield keeps on protecting your cluster, but you won’t be able to monitor its operation until you update your license.

不過只要到那邊的這邊,選擇 BASIC 的 Free 方案,註冊一個新的 product 後就可以下載新的 license 了,時間一樣是一年

然後我更新了 license

floodfire@r420:~/install/elasticsearch-2.4.1$ curl -XPUT -u admin 'http://localhost:2812/_license?pretty&acknowledgement=true' -d @veck-hsiao-641d2d44-175d-41c8-84c2-e1b330d1d38f-v2.json
Enter host password for user 'admin':
{
  "acknowledged" : false,
  "license_status" : "valid",
  "acknowledge" : {
    "message" : "This license update requires acknowledgement. To acknowledge the license, please read the following messages and update the license again, this time with the \"acknowledge=true\" parameter:",
    "marvel" : [ "basic", "Automatic index cleanup is locked to 7 days for clusters with [{}] license.basic" ]
  }
}
floodfire@r420:~/install/elasticsearch-2.4.1$ curl -XPUT -u admin 'http://localhost:2812/_license?pretty&acknowledge=true' -d @veck-hsiao-641d2d44-175d-41c8-84c2-e1b330d1d38f-v2.json
Enter host password for user 'admin':
{
  "acknowledged" : true,
  "license_status" : "valid"
}

測試 marvel-agent 的輸出中就更新了

"license" : {
  "status" : "active",
  "uid" : "641d2d44-175d-41c8-84c2-e1b330d1d38f",
  "type" : "basic",
  "issue_date" : "2017-05-25T00:00:00.000Z",
  "issue_date_in_millis" : 1495670400000,
  "expiry_date" : "2018-05-25T23:59:59.999Z",
  "expiry_date_in_millis" : 1527292799999,
  "max_nodes" : 100,
  "issued_to" : "Veck Hsiao (National Chengchi University)",
  "issuer" : "Web Form",
  "hkey" : "e69f88819c65320cd6b62c16e9eee7ffa72957205e48265f967f3a9b08fe8e41"
},

但是剛更新完不會馬上能夠讓 marvel 正常運作,所以不要啟動了 kibana 看到還是 No date 就罵幹,讓子彈飛一會,就會看到鬆了一口氣的畫面: