Read and Write Files From Elasticsearch With Python

How to read and write files from Elasticsearch with Python.

  1. Install the following packages:

    import pandas as pd
    from elasticsearch import Elasticsearch
    from elasticsearch.helpers import bulk
    import os
  2. Connect to Elasticsearch by running the following lines of code:

    # ====== Connecting to Elasticsearch ====== #
    # To connect to Elasticsearch.
    es = Elasticsearch(['http://'+os.environ['ELASTICSEARCH_IP'] +':'+ os.environ['ELASTICSEARCH_PORT']],timeout=600)
    # To create a simple index without any particular mapping.
    The default port is 9200.
  3. You can now insert and search documents in Elasticsearch by running the following lines of code:

    • Insert Documents

    • Search Documents

    # ====== Inserting Documents ====== #
    # To create a simple pandas DataFrame.
    liste_hello = ['hello1','hello2']
    liste_world = ['world1','world2']
    df = pd.DataFrame(data = {'hello' : liste_hello, 'world': liste_world})
    # Bulk insertion of documents. Each row in the DataFrame will be a document in ElasticSearch.
    documents = df.to_dict(orient='records')
    bulk(es, documents, index='helloworld',doc_type='foo', raise_on_error=True)
    # ====== Searching Documents ====== #
    # To retrieve all documents in the index (no query given).
    documents ='helloworld',body={})['hits']['hits']
    df = pd.DataFrame(documents)
    # To retrieve documents from the index that match a query.
    documents2 ='helloworld',body={"query":{"term":{"hello" : "hello1" }}})['hits']['hits']
    df2 = pd.DataFrame(documents2)