loading...

SonarQube: solving “Unrecoverable indexation failures” and Elasticsearch “Disk watermark exceeded” errors

setevoy profile image Arseny Zinchenko Originally published at rtfm.co.ua on ・4 min read

We are using SonarQube started from Jenkins jobs.

See more at SonarQube: running tests from Jenkins Pipeline in Docker.

So, the SonarQube is started from the Jenkins – all good here, but inside SonarQube we can see errors:

java.lang.IllegalStateException: Unrecoverable indexation failures: 1 errors among 1 requests

at org.sonar.server.es.IndexingListener$1.onFinish(IndexingListener.java:39)

at org.sonar.server.es.BulkIndexer.stop(BulkIndexer.java:122)

And host logs for SonarQube:

Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1 | 2019.08.29 09:30:26 ERROR web[][o.s.s.es.BulkIndexer] index [components], type [auth], id [AWzcfdpYaLFDDA6l6km3], message [ClusterBlockException[blocked by: [FORBIDD

EN/12/index read-only / allow delete (api)];]]

Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1 | 2019.08.29 09:30:26 ERROR web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery – too many failures [167/167 documents], waiting for next run

Aug 29 12:30:26 jenkins-production docker-compose[12591]: sonarqube_1 | 2019.08.29 09:30:26 INFO web[][o.s.s.e.RecoveryIndexer] Elasticsearch recovery – 167 documents processed [167 failures] | time=51ms

Aug 29 12:30:48 jenkins-production docker-compose[12591]: sonarqube_1 | 2019.08.29 09:30:48 INFO es[][o.e.c.r.a.DiskThresholdMonitor] low disk watermark [85%] exceeded on [mjJvGbfOSZyVqbKWKCGwxw][sonarqube][/opt/sonarqube/data/es6/nodes

/0] free: 11.6gb[11.8%], replicas will not be assigned to this node

Check the documentation about Disk-based shard allocation – need to set cluster.routing.allocation.disk.watermark.low or disable those checks at all by setting the "cluster.routing.allocation.disk.threshold_enabled" : false.

Although we have 12 GB free from the 100 GB total on the Jenkins host – but ElasticSearch by default count in percents so it’s not enough for him.

Because Elasticsearch is running inside (!) of the SonarQube’s container – let’s find where is the ElasticSearch config is stored there:

root@jenkins-production:/opt/jenkins# docker exec -ti jenkins_sonarqube_1 find / -name elasticsearch.yml
find: ‘/proc/tty/driver’: Permission denied
/opt/sonarqube/temp/conf/es/elasticsearch.yml
/opt/sonarqube/elasticsearch/config/elasticsearch.yml

Here we are – /opt/sonarqube/elasticsearch/config/elasticsearch.yml.

Sonar is started from a Docker Compose file with some files/dirs mapping:

version: '3.5'
networks:
  sonar:
    external:
      name: jenkins
services:
  sonarqube:
    user: 1004:1004
    image: sonarqube:7.9.1-community
    ports:
      - "9000:9000"
    networks:
      - sonar
    environment:
      - sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
    volumes:
      - /data/sonarqube/conf:/opt/sonarqube/conf
      - /data/sonarqube/logs:/opt/sonarqube/logs
      - /data/sonarqube/temp:/opt/sonarqube/temp
      - /data/sonarqube/data:/opt/sonarqube/data
      - /data/sonarqube/extensions:/opt/sonarqube/extensions
      - /data/sonarqube/bundled_plugins:/opt/sonarqube/lib/bundled-plugins
    logging:
      driver: "journald"
...

Available options for the disk management are:

  • cluster.routing.allocation.disk.threshold_enabled: будет ли Elasticsearch вообще проверять состояние диска
  • cluster.routing.allocation.disk.watermark.low: default 85%, Elasticsearch не будет размещать шарды на этой ноде
  • cluster.routing.allocation.disk.watermark.high: default 90%, Elasticsearch попробует перенести шарды с этой ноды на другие
  • cluster.routing.allocation.disk.watermark.flood_stage: default 95%, запрещает обновление индексов, шарды которых на этой ноде

Create a new config on the host – /data/sonarqube/conf/elasticsearch.yml:

cluster.routing.allocation.disk.watermark.flood_stage: 95%
cluster.routing.allocation.disk.watermark.high: 90%

Map it to the SonarQube’s container SonarQube:

...
    volumes:
      - /data/sonarqube/conf:/opt/sonarqube/conf
      - /data/sonarqube/conf/elasticsearch.yml:/opt/sonarqube/elasticsearch/config/elasticsearch.yml
...

To access the Elasticsearch – need to set sonar.search.httpPort for the SonarQube, see the How to Monitor ElasticSearch.

Enable this connector 9100:

...
    environment:
      - sonar.jdbc.url=jdbc:postgresql://db:5432/sonar
      - sonar.search.httpPort=9100
...

Restart the service:

root@jenkins-production:/home/admin# systemctl restart sonarqube

And a bit logs says this option will be deprecated soon:

2019.08.29 10:56:15 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001

2019.08.29 10:56:15 WARN app[][o.s.a.es.EsSettings] Elasticsearch HTTP connector is enabled on port 9100. MUST NOT BE USED FOR PRODUCTION

2019.08.29 10:55:10 WARN es[][o.e.d.c.s.Settings] [http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.

Check parameters after the restart to make sure Elasticsearch sees them.

As Elasticsearch listen on the localhost only inside (again!) the SonarQube’s container – use curl to check from the SonarQube itself:

root@jenkins-production:/home/admin# docker exec -ti jenkins_sonarqube_1 curl localhost:9100/_cluster/settings?include_defaults=true | jq '.[].cluster.routing.allocation.disk'
null
null
{
  "threshold_enabled": "true",
  "watermark": {
  "low": "85%",
  "flood_stage": "95%",
  "high": "90%"
},
  "include_relocations": "true",
  "reroute_interval": "60s"
}

Okay – new settings applied, can proceed with builds now.

Used disc space per node can be checked by calling the next API:

root@jenkins-production:/home/admin# docker exec -ti jenkins_sonarqube_1 curl 'http://localhost:9100/_cat/allocation?v&pretty'

shards disk.indices disk.used disk.avail disk.total disk.percent host      ip        node

24      613.4kb    86.3gb     11.5gb     97.9gb           88 127.0.0.1 127.0.0.1 sonarqube

All available parameters for the Elasticsearch – https://nickcanzoneri.com/elasticsearch-settings.

Done.

Similar posts

Posted on Mar 3 '19 by:

setevoy profile

Arseny Zinchenko

@setevoy

DevOps, cloud and infrastructure engineer. Love Linux, OpenSource, and AWS.

Discussion

markdown guide