Make Elasticsearch 6.x Cluster Writable from Readonly

Sometimes you might get

Error: disk usage exceeded flood-stage watermark, index has read-only-allow-delete block

or

ClusterBlockException[blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]

This error indicates a data node is critically low on disk space and has reached the flood-stage disk usage watermark. To prevent a full disk, when a node reaches this watermark, Elasticsearch blocks writes to any index with a shard on the node. If the block affects related system indices, Kibana and other Elastic Stack features may become unavailable.

This means you'll only be able to read the Indices, Delete the Indices but cannot write new Indices.

Get the Cluster Settings

curl elasticsearch-sc:9200/_settings?pretty

You'll see the read_only_allow_delete following as true for the Indices that means the cluster is in read-only mode

"blocks" : {
"read_only_allow_delete" : "true"
},

After the Disk Cleanup, When you have deleted the non-required Indices or after adding more space to the cluster, The Cluster needs to be put back to the writable mode

Putting Cluster back to Writable Mode

curl -XPUT -H "Content-Type: application/json" \
http://elasticsearch-sc:9200/_all/_settings \
-d '{"index.blocks.read_only_allow_delete": null}'

You can check the cluster settings again to confirm the change