As an Elasticsearch user, I subscribed to elasticsearch groups and channels in many places (Twitter, Reddit, Facebook, StackOverflow) and I keep seeing people struggling to start with elasticsearch, especially loading bunch of CSV files into it.
Hey does anyone know how can i upload multiple CSV files to Kibana?
The first mistake as I see it is to use Logstash in order to load CSV files to elasticsearch.
I see Logstash as a streaming service to elasticsearch rather than a tool to load data to elasticsearch from a static source (like CSV).
I wrote a wrapper on top of Elasticsearch's bulk API in order to reduce the barrier in loading files into Elasticsearch.
The usage simple as running a command from your terminal.
pip install elasticsearch-loader
There are several plugins like Parquet, Redis, and S3 as input sources, read more about them in the readme
Load CSVs into elasticsearch running on localhost
elasticsearch_loader --index incidents csv file1.csv file2.csv
Load CSV from URL and specify id field
elasticsearch_loader --index data --id-field state csv https://raw.githubusercontent.com/jakevdp/data-USstates/master/state-abbrevs.csv
Read CSV data from stdin
generate_data | elasticsearch_loader --index data csv -
And much more, for more examples and documantaion please follow elasticsearch-loader's README file