Interacting with your cloud storage buckets and objects can be a pain at some point, especially if you need the right tools. But why struggle when there is the Google Cloud Storage Python Client?
The Google Cloud Storage Python Client lets you securely store, organize and access data from anywhere in the world. And in this tutorial, you will learn to install the Google Cloud Storage Python Client and use it to interact with your Google Cloud Storage resources.
Ready? Stay tuned to improve your cloud storage management efficiently!
Prerequisites
This tutorial will be a hands-on demonstration. To follow along, be sure you have the following:
- A Linux machine – This tutorial uses Ubuntu 20.04, but any recent Linux distribution works.
- A Google Cloud Platform (GCP) account with active billing and admin privileges – A free trial account will suffice.
- Both gcloud CLI and Python are installed on your Linux machine.
Setting up a Local Development Environment
Before taking advantage of the Google Cloud Storage Python Client, you must set up a local development environment. A local development environment is where you can write and test your code without affecting your production environment. Testing out new things should not have to break anything.
You must first install the PIP package manager and activate a virtual environment to set up a local development environment. A virtual environment allows you to work on your project in an isolated environment.
- Open a terminal and run the below commands to perform the following:
- Update your system package index
- Install
wget
on your machine.
- Download, and install the PIP package manager.
sudo apt update -y && sudo apt install wget -y && wget https://bootstrap.pypa.io/get-pip.py && sudo python3 get-pip.py
![Installing wget and PIP](png/image-3125.png)
2. Once installed, run the following pip3
command to check the installed PIP version (-V
).
pip3 -V
![Checking the installed PIP version](png/image-3126.png)
3. Now, run the following pip
command to install
virtualenv
, a tool for creating isolated Python environments.
pip install virtualenv
![Installing virtualenv](png/image-3127.png)
4. After installing virtualenv
, run the below commands, which do not provide output but create your project directory and move into it.
This directory is where you will write and test your code.
mkdir my-project && cd my-project
5. Next, run the following virtualenv
command to start a new virtual environment. You can name your virtual environment differently, but this tutorial’s choice is my-project.
virtualenv my-project
![Starting a new virtual environment](png/image-3128.png)
6. Lastly, run the below source
command to activate your virtual environment, so you can start working on it.
With your virtual environment activated, any changes you make to your project’s dependencies (e.g., installing new libraries) only affect the virtual environment. Moreover, you can have different versions of the same library or dependency installed on your machine. You never have to worry about conflicts or compatibility issues.
source my-project/bin/activate
You will see your prompt change (my-project), as shown below, to indicate you are working in the virtual environment.
Now sharing your project with others becomes a breeze since you can include the virtual environment. Anyone who clones your project gets the same dependencies and libraries installed. This feature helps ensure your project works the same way for everyone who uses it.
![Activating the virtual environment](png/image-3129.png)
Installing the Google Cloud Storage Python Client
Now you have a local development environment and can start installing the Google Cloud Storage Python Client. This client allows you to access and manage Google Cloud Storage from within your Python code.
Run the below pip
command to download and install
the latest version of the google-cloud-storage
Python Client library from PyPI to your system.
The google-cloud-storage
package is a Python library that provides a user-friendly interface for interacting with the Google Cloud Storage API.
pip install google-cloud-storage
Once the library is installed, you can perform various tasks with your Google Cloud Storage data. For example, you can create new buckets, list buckets, upload objects, and more.
![Installing the Google Cloud Storage Python Client](png/image-3130.png)
Authenticating Environment with Google Cloud
You have successfully installed the Google Cloud Storage Python Client library, but the Google Cloud Storage API still requires authentication. Why? Doing so lets the Google Cloud Storage API access and manage your cloud storage data.
There are several ways to authenticate with Google Cloud. But in this tutorial, you use the default application method provided by Google.
1. Run the below gcloud auth
command to authenticate your environment with Google Cloud.
gcloud auth application-default login
Note down the URL in the output, as shown below. You will need it to access the Google Cloud Platform authentication page.
![Authenticating environment with Google Cloud](png/image-3131.png)
2. Next, navigate to the URL you noted in step one to a new browser tab, and click Allow, as shown below, to confirm the authentication.
![Confirming the authentication](png/image-3132.png)
3. Copy the generated code when authentication is successful.
![Copying the generated code](png/image-3133.png)
4. Now, switch back to your terminal, paste the code into your terminal, and press Enter to complete the authentication process.
![Finalizing the authentication](png/image-3134.png)
Creating a New Cloud Storage Bucket
You are now ready to use the Google Cloud Storage Python Client library with your environment authenticated. But first, you need a place to organize and store data — a bucket. A bucket (like a folder) is a logical container for storing objects (like files), and each object’s name is unique.
Buckets allow you to group objects and control access to them. For example, you can create a bucket for a specific project and grant access to only the team members working on that project. This feature helps prevent unauthorized access to your data.
To create a new bucket, follow the steps below:
1. Create a Python (create_bucket.py
), which you can name as you like, with your preferred editor. This tutorial uses the nano
text editor.
nano create_bucket.py
2. Next, populate the code below into the create_bucket.py file. The code below uses google.cloud.storage
to create a new bucket in your Google Cloud Storage.
Notes to keep in mind when using this code |
The bucket’s name must be unique within your Google Cloud Storage project and comply with buckets’ naming conventions. The name must be between three and 63 characters long and can only contain lowercase letters, numbers, and hyphens. |
This code does not verify a specified bucket, which results in an error if the bucket you are trying to create already exists. To avoid getting an error, check all existing buckets before calling the create_bucket() method in your script. |
# Imports the 'storage' module from the google.cloud package
# to allow interactions with the Google Cloud Storage.
from google.cloud import storage
# Creates a Client object that allows the script to communicate
# with Google Cloud Storage and perform operations on it (like creating a bucket).
client = storage.Client()
# Creates a new bucket with a specified name
bucket = client.create_bucket("my-first-bucket14755286")
# Prints a message indicating the bucket was successfully created.
print("Bucket {} created.".format(bucket.name))
3. Save and close the create_buckets.py file, and execute the below command to run the script.
python create_bucket.py
If all goes well, you will see a message that says Bucket <bucket-name> created., as shown below.
![Creating a new bucket](png/image-3135.png)
Listing Available Buckets in the Google Cloud Storage
Now you have created a new bucket, and you can use the Google Cloud Storage Client library to list all existing buckets in your project. Why? Listing all available buckets in your project is good practice before performing operations like uploading objects.
To list all the buckets in your project, follow these steps:
1. Create a Python file (named as you desire) with your editor.
nano list_buckets.py
2. Next, add the code below to the list_buckets.py file.
This code lists all the buckets in your specified project and checks if the created bucket is in the list. If found, a message prints saying Bucket found!
along with information about the bucket.
from google.cloud import storage
# Create a client object
client = storage.Client()
# List all the buckets in your project
buckets = list(client.list_buckets())
# Check if the bucket you created is in the list of buckets
for bucket in buckets:
if bucket.name == "my-first-bucket14755286":
print("Bucket found!")
print(buckets)
3. Save and close the list_buckets.py file and run it.
python list_buckets.py
If the bucket you created is found, you will see a similar output to the one below.
![Listing all buckets](png/image-3136.png)
Uploading Objects to a Bucket
Now that you have created an empty bucket, the first step in making your bucket useful is uploading objects to the bucket.
Which type of objects can you upload? Any file type will do, such as an image, video, text file, etc. But before uploading, you need an object first.
1. Create a new text file named my-file.txt, a test file you will upload to your bucket.
2. Next, create a Python file called upload_object.py (or any other name), and populate the code below into the file.
The code below uses the upload_from_filename
method to upload your text file to your bucket.
from google.cloud import storage
# Create a client object
client = storage.Client()
# Get a reference to the bucket you want to upload to
bucket = client.bucket("my-first-bucket14755286")
# Create a new blob object
blob = bucket.blob("my-file.txt")
# Upload the file to the bucket
if blob.upload_from_filename("./my-file.txt"):
print("File successfully uploaded to bucket.")
else:
print("Error uploading file to bucket.")
3. Save the file, and execute the below command to run the script (upload_object.py
).
python upload_object.py
If all goes well, you will see a message that says File successfully uploaded to bucket, as shown below. At this point, the file has been uploaded to your bucket.
![Uploading an object to the bucket](png/image-3137.png)
4. Now, navigate the Storage page in the Google Cloud Console to verify the uploaded object exists.
You will see your bucket (my-first-bucket14755286) with the uploaded object (my-file.txt) listed, as shown below.
![Verifying the uploaded object exists in the bucket](png/image-3138.png)
5. Finally, run the following deactivate
command to leave or exit your virtual environment.
deactivate
Successfully deactivating your virtual environment switches you back to your default prompt, as in the output below.
![Deactivating the virtual environment](png/image-3139.png)
Conclusion
This tutorial taught you how to create buckets and manage objects with the Google Cloud Storage Python Client. Whether you are a beginner or an experienced developer, the Google Cloud Storage Python Client is a valuable addition to your toolkit for working with data in the cloud.
Now, why not explore the advanced features? Indulge in object versioning, lifecycle management, and permissions for deeper control over your data!