Read and Write Files From Elasticsearch With Python

How to read and write files from Elasticsearch with Python.

  1. Install the following packages:

    import pandas as pd
from elasticsearch import Elasticsearch
from elasticsearch.helpers import bulk
import os

  2. Connect to Elasticsearch by running the following lines of code:

    # ====== Connecting to Elasticsearch ====== #
# To connect to Elasticsearch.
es = Elasticsearch(['http://'+os.environ['ELASTICSEARCH_IP'] +':'+ os.environ['ELASTICSEARCH_PORT']],timeout=600)
# To create a simple index without any particular mapping.
es.indices.create(index='helloworld',body={})
    The default port is 9200.

  3. You can now insert and search documents in Elasticsearch by running the following lines of code:

    • Insert Documents

    • Search Documents

    # ====== Inserting Documents ====== #
# To create a simple pandas DataFrame.
liste_hello = ['hello1','hello2']
liste_world = ['world1','world2']
df = pd.DataFrame(data = {'hello' : liste_hello, 'world': liste_world})

# Bulk insertion of documents. Each row in the DataFrame will be a document in ElasticSearch.
documents = df.to_dict(orient='records')
bulk(es, documents, index='helloworld',doc_type='foo', raise_on_error=True)
    # ====== Searching Documents ====== #
# To retrieve all documents in the index (no query given).
documents = es.search(index='helloworld',body={})['hits']['hits']
df = pd.DataFrame(documents)

# To retrieve documents from the index that match a query.
documents2 = es.search(index='helloworld',body={"query":{"term":{"hello" : "hello1" }}})['hits']['hits']
df2 = pd.DataFrame(documents2)
See also