One annoyance I’ve hit whilst running ELK on Docker is, after rebooting my system, the same error keeps returning:
* Starting periodic command scheduler cron
...done.
* Starting Elasticsearch Server
...done.
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
waiting for Elasticsearch to be up (9/30)
waiting for Elasticsearch to be up (10/30)
waiting for Elasticsearch to be up (11/30)
waiting for Elasticsearch to be up (12/30)
waiting for Elasticsearch to be up (13/30)
waiting for Elasticsearch to be up (14/30)
waiting for Elasticsearch to be up (15/30)
waiting for Elasticsearch to be up (16/30)
waiting for Elasticsearch to be up (17/30)
waiting for Elasticsearch to be up (18/30)
waiting for Elasticsearch to be up (19/30)
waiting for Elasticsearch to be up (20/30)
waiting for Elasticsearch to be up (21/30)
waiting for Elasticsearch to be up (22/30)
waiting for Elasticsearch to be up (23/30)
waiting for Elasticsearch to be up (24/30)
waiting for Elasticsearch to be up (25/30)
waiting for Elasticsearch to be up (26/30)
waiting for Elasticsearch to be up (27/30)
waiting for Elasticsearch to be up (28/30)
waiting for Elasticsearch to be up (29/30)
waiting for Elasticsearch to be up (30/30)
Couln't start Elasticsearch. Exiting.
Elasticsearch log follows below.
[2017-07-14T08:36:42,337][INFO ][o.e.n.Node ] [] initializing ...
[2017-07-14T08:36:42,437][INFO ][o.e.e.NodeEnvironment ] [71cahpZ] using [1] data paths, mounts [[/var/lib/elasticsearch (/dev/sde2)]], net usable_space [51.1gb], net total_space [146.6gb], spins? [possibly], types [ext4]
[2017-07-14T08:36:42,438][INFO ][o.e.e.NodeEnvironment ] [71cahpZ] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-07-14T08:36:42,463][INFO ][o.e.n.Node ] node name [71cahpZ] derived from node ID [71cahpZ4SjeKKAoH8X5dYg]; set [node.name] to override
[2017-07-14T08:36:42,463][INFO ][o.e.n.Node ] version[5.3.0], pid[64], build[3adb13b/2017-03-23T03:31:50.652Z], OS[Linux/4.4.0-45-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_121/25.121-b13]
[2017-07-14T08:36:43,910][INFO ][o.e.p.PluginsService ] [71cahpZ] loaded module [aggs-matrix-stats]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService ] [71cahpZ] loaded module [ingest-common]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService ] [71cahpZ] loaded module [lang-expression]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService ] [71cahpZ] loaded module [lang-groovy]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService ] [71cahpZ] loaded module [lang-mustache]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService ] [71cahpZ] loaded module [lang-painless]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService ] [71cahpZ] loaded module [percolator]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService ] [71cahpZ] loaded module [reindex]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService ] [71cahpZ] loaded module [transport-netty3]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService ] [71cahpZ] loaded module [transport-netty4]
[2017-07-14T08:36:43,911][INFO ][o.e.p.PluginsService ] [71cahpZ] no plugins loaded
[2017-07-14T08:36:45,703][INFO ][o.e.n.Node ] initialized
[2017-07-14T08:36:45,703][INFO ][o.e.n.Node ] [71cahpZ] starting ...
[2017-07-14T08:36:45,783][WARN ][i.n.u.i.MacAddressUtil ] Failed to find a usable hardware address from the network interfaces; using random bytes: 58:01:4e:51:11:f3:c9:da
[2017-07-14T08:36:45,835][INFO ][o.e.t.TransportService ] [71cahpZ] publish_address {172.20.0.3:9300}, bound_addresses {0.0.0.0:9300}
[2017-07-14T08:36:45,839][INFO ][o.e.b.BootstrapChecks ] [71cahpZ] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-07-14T08:36:45,840][ERROR][o.e.b.Bootstrap ] [71cahpZ] node validation exception
bootstrap checks failed
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-07-14T08:36:45,842][INFO ][o.e.n.Node ] [71cahpZ] stopping ...
[2017-07-14T08:36:45,909][INFO ][o.e.n.Node ] [71cahpZ] stopped
[2017-07-14T08:36:45,909][INFO ][o.e.n.Node ] [71cahpZ] closing ...
[2017-07-14T08:36:45,914][INFO ][o.e.n.Node ] [71cahpZ] closed
The core of the fix to this problem is helpfully included in the output:
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
And on Ubuntu fixing this is a one-liner:
sudo sysctl -w vm.max_map_count=262144
Only, this is not persisted across a reboot.
To fix this permenantly, I needed to do:
sudo vim /etc/sysctl.d/60-elasticsearch.conf
And add in the following line:
vm.max_map_count=262144
I am not claiming credit for this fix. I found it here. I’m just sharing here because I know in future I will need to do this again, and it’s easiest if I know where to start looking 🙂
Another important step if persisting data is to ensure the folder is owned by 991:991 .
