Amazon Simple Storage Service (S3) is an object storage service that offers high availability, durability, scalability, and security. It is a popular choice for storing a variety of data, including images, videos, documents, and backups.

In this article, you will learn how to write data or a file to an AWS S3 bucket using Boto3 Python. Boto3 is the official AWS SDK for Python. It provides a high-level abstraction for interacting with AWS services.

Setting Up Boto3

• Install Boto3 using pip install boto3
• Configure AWS security credentials in your system using the command aws configure. The security credentials can be generated from the Profile -> Security Credentials and by clicking the Create access key option available under the Access keys section

Writing Data or Stream to AWS S3 Bucket

When you have the data in a variable instead of a file, you can write the data to the AWS S3 using the Boto3 Client or the Boto3 resource methods.

The Boto3 client creates a low-level service client using the default session.

The Boto3 Resource represents an object-oriented interface to AWS services.

Using Boto3 Client put_object Method

To write data into an AWS S3 bucket,

• Create a Boto3 client object for the S3 service using the boto3.client('s3')
• Invoke the put_object() method and pass the bucket name, object name and the data stream using the body parameter

Use this method when you already have data in memory, don’t need to stream it, and want to deal with small objects.

Code

import boto3

s3 = boto3.client('s3')

bucket_name = 'mrcloudgurudemo'

object_key = 'test_file_object.txt'

data = b'Hello, S3!'

s3.put_object(Bucket=bucket_name, Key=object_key, Body=data)

print('Data was written to the S3 bucket successfully.')

Output

  Data was written to the S3 bucket successfully.

Using Boto3 Resource Put() method

You can use the Boto3 resource put() method to write data into the AWS S3 bucket.

The AWS Python SDK team does not intend to add new features to the resources interface in boto3. Existing interfaces will continue to operate during boto3’s lifecycle. You can use the client interface to interact with the service.

• Create a resource object to represent the S3 service
• Using the resource, create an object representation using the bucket name and the object name
• Invoke the object.put() method and pass the data using the Body parameter
• A response will be returned, and you can check the HTTPStatusCode of the ResponseMetadata to check whether the data upload is successful.
• If HTTPStatusCode is 200, then the upload is successful. Else not.

Code

import boto3

s3 = boto3.resource('s3')

bucket_name = 'mrcloudgurudemo'
object_key = 'test_file_object.txt'

object = s3.Object(bucket_name, object_key)

txt_data = b'This is the content of the file uploaded from python boto3'

result = object.put(Body=txt_data)

res = result.get('ResponseMetadata')

if res.get('HTTPStatusCode') == 200:
    print('File Uploaded to Boto3 Successfully')
else:
    print('File Not Uploaded')

Output

    File Uploaded to Boto3 Successfully

Writing File to AWS S3 Bucket

This section teaches you the different methods to write a File or file-like objects to the AWS S3 bucket using the Boto3 client or Boto3 Resource.

Using Boto3 Client Upload_file Method

To upload a file to the AWS S3 bucket, you can use the upload-file() method.

You can use the upload_file()method when you need the following:

• Local File Upload: You have a local file on your system that you want to upload to an S3 bucket. The upload_file method simplifies this process as it directly handles file uploads.
• Simplified Error Handling: upload_file provides built-in error handling for common issues like file not found or insufficient permissions. It’s a more straightforward choice if you want the method to manage these issues for you.
• No Need for a File-Like Object: You don’t have a file-like object, and your data is already in a local file.

To upload a file using the upload_file() method,

• Create a Boto3 client representation
• Invoke the upload_file() method and pass the path of the file to be uploaded along with the target bucket name and the target object name
• If an object with the same name already exists in the bucket, it will be replaced.

Code

import boto3

s3 = boto3.client('s3')

bucket_name = 'mrcloudgurudemo'
object_key = 'test_file_object.txt'

local_file_path = 'boto3_test_file.txt'

s3.upload_file(local_file_path, bucket_name, object_key)

print('File uploaded to S3 bucket successfully.')

Output

    File uploaded to S3 bucket successfully.

Using Boto3 Client Upload_fileobj (For File-Like Object)

Another method to upload files to the AWS S3 bucket is to use the upload_fileobj() method.

You can use the upload_ fileobj()method when you need the following:

• Streaming Data: You want to upload data from a source that doesn’t exist as a local file but is instead available as a file-like object (e.g., BytesIO, StringIO, or data being generated on the fly). upload_fileobj allows you to stream data directly from such objects.
• Optimized Memory Usage: If you are working with large files or large datasets that may not fit entirely in memory, upload_fileobj() allows you to stream the data in smaller chunks, minimizing memory consumption.
• Custom Error Handling: You prefer to handle errors related to the data source yourself and want more control over the upload process.

To upload a file using the upload_fileobj() method,

• Create a Boto3 client representation for the S3 service
• Invoke the upload_fileobj() method and pass the data, target bucket name and the target object name
• This method will replace any existing object with the same name; hence, try to use a unique name for the target object

Code

import boto3

s3 = boto3.client('s3')

bucket_name = 'mrcloudgurudemo'
object_key = 'test_file_object.txt'

local_file_path = 'boto3_test_file.txt'

with open(local_file_path, 'rb') as data:

    s3.upload_fileobj(data, bucket_name, object_key)

print('File uploaded to S3 bucket successfully.')

Output

    File uploaded to S3 bucket successfully.

Handling Exceptions

During the file write operation, there is a possibility of three exceptions.

NoSuchBucket Error:

The NoSuchBucket error occurs when you attempt to perform an operation on an S3 bucket that doesn’t exist in your AWS account or is misspelt.

To avoid this exception,

• Double-check the bucket name you use in your code to ensure it matches the exact name of the S3 bucket in your AWS account.
• You can also use the head_bucket() method to check if the bucket exists before you try to upload data to it.
• Verify that the bucket is created in the correct AWS region if your code specifies a region.

No Credentials Error:

The No Credentials error occurs when your code lacks the necessary AWS credentials (access key and secret key) to authenticate with AWS services.

To avoid this exception,

• Always provide valid AWS credentials by configuring them in your AWS CLI/SDK or using environment variables, IAM roles, or AWS configuration files.
• Double-check that your AWS credentials are correctly set up and have the required permissions to access S3.

No Region Specified Error:

The No Region Specified error occurs when you attempt to interact with AWS services without specifying the AWS region in your code. AWS services require you to specify a region to determine the resources’ location.

To avoid this exception,

• Always specify the AWS region when creating a client or resource for AWS services, including S3. You can do this by passing the region_name parameter when initialising the client or resource.

Conclusion

In conclusion, you learned how to write data or a file to an AWS S3 bucket using Boto3 Python. We covered the three main methods for writing data to an S3 bucket: the put_object() method, the upload_file() method, and the upload_fileobj() method. You also learned how to handle the three most common exceptions that can occur when writing data to an S3 bucket: the NoSuchBucketError, the NoCredentialsError, and the NoRegionSpecifiedError.

I hope this article was helpful. If you have any questions, please let me know.

Additional Resources

• Reading File Content From AWS S3 Bucket Using Boto3