Read and Write Files From Amazon S3 Bucket With Talend

How to read and write files from Amazon S3 Bucket with Talend.

Before you begin:

Make sure you that you meet all the following requirements:

  • You have an S3 bucket with a file on it.

  • You have an AWS access key and secret key.

  • You have Talend Open Studio version 7.1.1 or higher.

Read Files

  1. Create a new job in Talend.

  2. Add the following components:

    • tS3Connection, to establish a connection to Amazon S3 to store and retrieve data.

    • tS3Get, to retrieve a file from Amazon S3.

    • tS3Close, to shut down a connection to Amazon S3, and releasing the network resources.

  3. Link these components with the OnSubjobOk connection.

    S3 read file with talend

  4. Double-click each component and configure their settings as follows:

    • tS3Connection

    • tS3Get

    • tS3Close

    In the Basic settings tab:

    • In the Access Key field, enter the access key ID.

    • In the Secret Key field, enter the secret access.

    For more information, you can refer to Talend’s documentation on the tS3Connection component.

    In the Basic settings tab:

    1. Select the Use an existing connection option.

    2. From the Component List list, select the tS3Connection connection component to reuse the connection details already defined.

    3. In the Bucket field, specify the name of the bucket.

    4. In the Key field, specify the path to the file saved on the S3 server.

    5. In the File field, specify the name of the file to be saved locally.

    For more information, you can refer to Talend’s documentation on the tS3Get component.

    In the Basic settings tab:

    1. From the Component List list, select the tS3Connection connection component to close the connection.

    For more information, you can refer to Talend’s documentation on the tS3Close component.

Write Files

  1. Create a new job in Talend.

  2. Add the following components:

    • tS3Connection, to establish a connection to Amazon S3 to store and retrieve data.

    • tS3Put, to upload data onto Amazon S3 from a local file or from cache memory via the streaming mode.

    • tS3Close, to shut down a connection to Amazon S3, and releasing the network resources.

  3. Link these components with the OnSubjobOk connection.

    S3 write file with talend

  4. Double-click each component and configure their settings as follows:

    • tS3Connection

    • tS3Put

    • tS3Close

    In the Basic settings tab:

    • In the Access Key field, enter the access key ID.

    • In the Secret Key field, enter the secret access.

    For more information, you can refer to Talend’s documentation on the tS3Connection component.

    In the Basic settings tab:

    1. Select the Use an existing connection option.

    2. From the Component List list, select the tS3Connection connection component to reuse the connection details already defined.

    3. In the Bucket field, specify the name of the bucket.

    4. In the Key field, specify the path to the file that will hold the uploaded data on the S3 server.

    5. In the File or Stream field, enter the path to the local file to be uploaded or the input stream from which data will be uploaded.

    For more information, you can refer to Talend’s documentation on the tS3Put component.

    In the Basic settings tab:

    1. From the Component List, select the tS3Connection connection component to close the connection.

    For more information, you can refer to Talend’s documentation on the tS3Close component.