Elasticsearch启动异常处理

最近在使用zipkin, 遇到如下问题，于是就网上搜了一下解决方案并记录了下来

• 使用tar.gz安装包解压安装的方式

• 使用./bin/elasticsearch方式启动elasticsearch

• 使用java -DSTORAGE_TYPE=elasticsearch -DES_HOSTS=http://ip:9200 -jar zipkin.jar命令启动zipkin

• Zipkin启动报错如下：

  java.util.concurrent.RejectedExecutionException: finishConnect(..) failed: Connection refused: 10.13.0.92/10.13.0.92:9200
          at zipkin2.elasticsearch.internal.client.HttpCall.lambda$sendRequest$3(HttpCall.java:235) ~[zipkin-storage-elasticsearch-2.19.1.jar!/:?]
          at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) ~[?:1.8.0_212]
          at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852) ~[?:1.8.0_212]
          at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_212]
          at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_212]
          at com.linecorp.armeria.common.HttpMessageAggregator.fail(HttpMessageAggregator.java:146) ~[armeria-0.95.0.jar!/:?]
          at com.linecorp.armeria.common.HttpMessageAggregator.onError(HttpMessageAggregator.java:60) ~[armeria-0.95.0.jar!/:?]
          at com.linecorp.armeria.common.stream.FilteredStreamMessage$FilteringSubscriber.onError(FilteredStreamMessage.java:277) ~[armeria-0.95.0.jar!/:?]
          at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:370) ~[armeria-0.95.0.jar!/:?]
          at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:195) ~[armeria-0.95.0.jar!/:?]
          at com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:365) ~[armeria-0.95.0.jar!/:?]
          at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:308) ~[armeria-0.95.0.jar!/:?]
          at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.43.Final.jar!/:4.1.43.Final]
          at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:510) ~[netty-common-4.1.43.Final.jar!/:4.1.43.Final]
          at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:413) ~[netty-transport-native-epoll-4.1.43.Final-linux-x86_64.jar!/:4.1.43.Final]
          at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) ~[netty-common-4.1.43.Final.jar!/:4.1.43.Final]
          at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.43.Final.jar!/:4.1.43.Final]
          at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.43.Final.jar!/:4.1.43.Final]
          at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]
  Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..) failed: Connection refused: 10.13.0.92/10.13.0.92:9200
  Caused by: java.net.ConnectException: finishConnect(..) failed: Connection refused
          at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124) ~[netty-transport-native-unix-common-4.1.43.Final.jar!/:4.1.43.Final]
          at io.netty.channel.unix.Socket.finishConnect(Socket.java:243) ~[netty-transport-native-unix-common-4.1.43.Final.jar!/:4.1.43.Final]
          at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:660) ~[netty-transport-native-epoll-4.1.43.Final-linux-x86_64.jar!/:4.1.43.Final]
          at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:637) ~[netty-transport-native-epoll-4.1.43.Final-linux-x86_64.jar!/:4.1.43.Final]
          at io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:524) ~[netty-transport-native-epoll-4.1.43.Final-linux-x86_64.jar!/:4.1.43.Final]
          at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:492) ~[netty-transport-native-epoll-4.1.43.Final-linux-x86_64.jar!/:4.1.43.Final]
          at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) ~[netty-transport-native-epoll-4.1.43.Final-linux-x86_64.jar!/:4.1.43.Final]
          ... 4 more

• 解决方法：修改config/elasticsearch.yml配置项network.host: 0.0.0.0

• 重启ElasticSearch，ElasticSearch启动报错如下：

  [2019-11-19T14:51:10,811][ERROR][o.e.b.Bootstrap          ] [test] node validation exception
  [5] bootstrap checks failed
  [1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
  [2]: max number of threads [1024] for user [elsearch] is too low, increase to at least [4096]
  [3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
  [4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
  [5]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
  [2019-11-19T14:51:10,814][INFO ][o.e.n.Node               ] [test] stopping ...
  [2019-11-19T14:51:10,833][INFO ][o.e.n.Node               ] [test] stopped
  [2019-11-19T14:51:10,833][INFO ][o.e.n.Node               ] [test] closing ...
  [2019-11-19T14:51:10,852][INFO ][o.e.n.Node               ] [test] closed

[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]

1. 切换root用户
2. vi /etc/security/limits.conf
3. 文件末尾添加如下内容

   *  soft  nofile  65536
   *  hard  nofile  65536

4. wq保存

[2]: max number of threads [1024] for user [elsearch] is too low, increase to at least [4096]

1. vi /etc/security/limits.d/90-nproc.conf
2. 修改* soft nproc 1024为* soft nproc 4096
3. wq保存

[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

1. vi /etc/sysctl.conf
2. 末尾添加vm.max_map_count=262144
3. wq保存
4. sysctl -p 重新加载系统参数

[4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk

1. vi ELS_HOME/config/elasticsearch.yml
2. 修改配置项为bootstrap.system_call_filter: false

[5]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured

1. vi ELS_HOME/config/elasticsearch.yml
2. 修改配置项为cluster.initial_master_nodes: ["node-1"]
3. wq保存
4. 重启elasticsearch