Appreciate life and record it

ElasticSearch

 Roams  1179  3 分钟   
ElasticSearch

ELK准备

添加源

具体可以参考https://www.elastic.co/guide/en/logstash/7.16/installing-logstash.html#_yum

debian系 

1
2
3

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
sudo sh -c 'echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" > /etc/apt/sources.list.d/elastic-7.x.list'

centos系 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
cat <<EOF > /etc/yum.repos.d/elastic.repo
[logstash-7.x]
name=Elastic repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

Logstash

安装

debian系 
1

sudo apt-get install logstash
centos系 
1

sudo yum -y install logstash

ElasticSearch

介绍

参考https://www.elastic.co/guide/cn/elasticsearch/guide/current/getting-started.html

elasticsearch是一个基于lucene库的实时的分布式搜索分析引擎，主要用作全文检索，结构化搜索，分析以及这三种的组合

常见的应用场景有系统日志分析、应用数据分析、安全审计、关键词搜索等

es是面向文档的，对于复杂关系，比如地理信息日期等对象都可以保存，这是相比较于关系型数据库优势的地方

安装 

1
2
3
4

  # if centos
  yum -y install elasticsearch
  # if debian
  apt-get install elasticsearch

启动 

1
2
3
4
5

systemctl start elasticsearch
systemctl enable elasticsearch
lsof -i:9200
COMMAND   PID          USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
java    32481 elasticsearch  280u  IPv4 29805357      0t0  TCP localhost:wap-wsp (LISTEN)

可能报错：

启动报错了 failed; error='Not enough space' (errno=12) 修改下es的启动参数

1
2
3
4
5
6

liuliancao@liuliancao-dev:~/projects/lion$ sudo cat /etc/elasticsearch/jvm.options|grep Xm
## -Xms4g
## -Xmx4g
-Xms200m
-Xmx200m
启动时间较长，我的虚拟机大概20s..

测试 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

cat logstash-first.conf
input { stdin { } }
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}
# logstash -f logstash-first.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
hello, world!
[INFO ] 2021-07-07 11:09:32.287 [Ruby-0-Thread-10: :1] elasticsearch - Installing ILM policy {"policy"=>{"phases"=>{"hot"=>{"actions"=>{"rollover"=>{"max_size"=>"50gb", "max_age"=>"30d"}}}}}} {:name=>"logstash-policy"}
{
      "@version" => "1",
       "message" => "hello, world!",
          "host" => "xxx",
    "@timestamp" => 2021-07-07T03:09:32.187Z
}

代表es数据成功写入

集群搭建

参考集群搭建，

三台服务器

RESTful API with JSON over http

通过9200交互

liuliancao@liuliancao-dev:~/projects/lion$ sudo lsof -i:9200 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 50762 elasticsearch 284u IPv6 141518 0t0 TCP localhost:9200 (LISTEN) java 50762 elasticsearch 285u IPv6 141519 0t0 TCP localhost:9200 (LISTEN)

Curl, Groovy, Javascript, .NET, PHP, Perl, Python, Ruby (https://www.elastic.co/guide/en/elasticsearch/client/index.html)
Curl

curl -X<VERB> '<PROTOCOL>://<HOST>:<PORT>/<PATH>?<QUERY_STRING>' -d '<BODY>'

查询集群中文档数量

curl -XGET 'http://localhost:9200/_count?pretty' -d ' { "query": { "match_all": {} } } ' 实际执行结果是 liuliancao@liuliancao-dev:~/projects/lion$ curl -XGET 'http://localhost:9200/_count?pretty' -d ' { "query": { "match_all": {} } } ' { "error" : "Content-Type header [application/x-www-form-urlencoded] is not supported", "status" : 406 } ..., 需要调整下header， 这个结果代表我们没有分片和文档存在 liuliancao@liuliancao-dev:~/projects/lion$ curl -XGET -H 'Content-Type: application/json' 'http://localhost:9200/_count?pretty' -d ' { "query": { "match_all": {} } } ' { "count" : 0, "_shards" : { "total" : 0, "successful" : 0, "skipped" : 0, "failed" : 0 } }

JSON形式保存对象
一些es中的概念
索引
类型

Q***** 属性

集群状态查看 
1
2

# curl -XGET 'http://localhost:9200/_cluster/health'
{"cluster_name":"web","status":"red","timed_out":false,"number_of_nodes":6,"number_of_data_nodes":3,"active_primary_shards":4416,"active_shards":4416,"relocating_shards":0,"initializing_shards":12,"unassigned_shards":34046,"delayed_unassigned_shards":0,"number_of_pending_tasks":66,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":907745,"active_shards_percent_as_number":11.477881166502053}
列出所有index 
1

curl -X GET "localhost:9200/_cat/indices?v"
模糊删除index 
1

DELETE /your-index-pattern*

DSL

Query查询

一个典型的查询 https://www.elastic.co/guide/en/elasticsearch/reference/current/query-filter-context.html

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

  GET /_search
  {
    "query": {
      "bool": {
        "must": [
          { "match": { "title":   "Search"        }},
          { "match": { "content": "Elasticsearch" }}
        ],
        "filter": [
          { "term":  { "status": "published" }},
          { "range": { "publish_date": { "gte": "2015-01-01" }}}
        ]
      }
    }
  }
指定正则匹配

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-regexp-query.html

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

  GET /_search
  {
      "query": {
            "regexp": {
                    "user.id": {
                              "value": "k.*y",
                              "flags": "ALL",
                              "case_insensitive": true,
                              "max_determinized_states": 10000,
                              "rewrite": "constant_score"
                    }
            }
      }
  }

聚合查询

聚合里面进行count排序 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

"aggs": {
       "hours data": {
          "date_histogram": {
            "field": "@timestamp",
            "calendar_interval": "1m",
            "time_zone": "Asia/Shanghai",
            "min_doc_count": 100,
            "order": {
              "_count": "desc"
            }
          }
       }
  }

kibana

测试使用

浏览器访问服务器地址:5601端口 建议通过nginx+ssl配置，会比较安全