Learning the gsutil Command Through Examples

Published:27 December 2022 - 7 min. read

Nicholas Xuan Nguyen Image

Nicholas Xuan Nguyen

Read more tutorials by Nicholas Xuan Nguyen!

Azure Cloud Labs: these FREE, on‑demand Azure Cloud Labs will get you into a real‑world environment and account, walking you through step‑by‑step how to best protect, secure, and recover Azure data.

Accessing and managing your Cloud Storage buckets can be a complex endeavor. Fortunately, Google Cloud Platform (GCP) provides an incredible tool to make the task easier — gsutil.

gsutil is a powerful command line utility that provides you fast access to hundreds of features that GCP’s Cloud Storage service offers. And in this tutorial, you will learn how to use gsutil through a series of examples demonstrating its many capabilities.

Read on and confidently manage your Cloud Storage buckets with ease!

Prerequisites

This tutorial will be a hands-on demonstration. To follow along, be sure you have a Google Cloud Platform (GCP) account — a free trial will suffice.

Creating a Google Cloud Project

Google Cloud projects allow you to organize resources, including storage buckets. But before creating a project, you must ensure that Cloud Shell is activated. Cloud Shell is a sandboxed command line environment pre-installed with the gsutil command line tool.

To create a Google Cloud Project:

1. Open your favorite web browser, and log in to the Google Cloud Console.

2. Next, click the Activate Cloud Shell icon (top-right) to activate the Cloud Shell.

You will see a pane at the bottom of your browser, as shown below, where you can run commands to interact with GCP services.

Activating the Cloud Shell
Activating the Cloud Shell

3. Once Cloud Shell is activated, run the following gcloud projects command in the Cloud Shell to create your project. You can name your projects differently. but this tutorial’s choice is gsutil-project-demo

gcloud projects create gsutil-project-demo

4. Now, click AUTHORIZE when prompted to allow the required permissions.

Authorizing the Cloud Shell
Authorizing the Cloud Shell

If successful, you will see a message like the one below.

Creating a Google Cloud project
Creating a Google Cloud project

5. Lastly, run the below gcloud config set command to set your newly created project (gsutil-project-demo) as the default project. As a result, all subsequent commands you issue in the Cloud Shell are executed against the default project. gcloud config set project gsutil-project-demo

gcloud config set project gsutil-project-demo
Setting the newly-created project as the default project
Setting the newly-created project as the default project

Creating a Bucket with gsutil

Your default project is set, and you are ready to work with the gsutil tool. What is the most basic operation that gsutil lets you perform? Create a bucket in the cloud to store and access your data.

The traditional storage options, like block storage and file storage, work fine when you have fewer data. But when your data starts to pile up, like Terabytes or Petabytes, you need a more robust and cost-effective storage option. Google Cloud Storage offers that option.

Run the gsutil command below to make a bucket (mb), but ensure you replace *gs://my-first-bucket54621* bucket name with your own that is globally unique.

💡 You can create as many buckets as you need, depending on the data you wish to store. But note that the bucket’s name must be in lowercase and should not contain any spaces or special characters.

Notice below that the bucket’s name is in URL form (*gs://my-first-bucket54621*). This URL helps you locate and access the data stored in your bucket faster. Unlike other storage solutions, you can even use an endpoint to access the bucket’s content.

gsutil mb gs://my-first-bucket54621

As you can see below, the bucket has been successfully created.

Creating a bucket
Creating a bucket

But if the command fails, the reason might be the name of your bucket is already taken, and you will see an error message like the one below.

Getting an error when the bucket already exists
Getting an error when the bucket already exists

Uploading an Object to a Bucket

You have created a bucket in the Google Cloud Storage with gsutil, but that bucket is not serving any purpose if it is empty.

In this example, you will upload an object to your bucket using the gsutil tool. An object in Cloud Storage is like a file but contains much more extensive metadata. Each metadata is in key-value pairs that describe the object in more detail.

To upload an object to your bucket, follow these steps:

1. Run the curl command below to download an image of a cat from its URL and store it in Cloud Shell.

curl https://cloud.google.com/storage/images/kitten.png -o kitten.png
Downloading an image
Downloading an image

2. Next, run the following command to list (ls) the image (kitten.png) to ensure the file exists.

ls -la kitten.png
Ensuring the downloaded file exists
Ensuring the downloaded file exists

3. Now, run the below gsutil command to upload the file (kitten.png) into your Cloud Storage bucket (gs://my-first-bucket54621).

gsutil cp kitten.png gs://my-first-bucket54621
Uploading an object to the bucket
Uploading an object to the bucket

Rewriting an Object’s Metadata

Google Cloud Storage automatically generates a system of metadata when you upload the object. These default metadata can be the uploaded date, content type, storage class, and so on, which Cloud Storage uses as part of its object management.

In this example, you will rewrite existing metadata on the file you uploaded to your bucket. But first, you must view the metadata already set to the file.

1. Run the gsutil stat command below to list the metadata already set to your image file.

gsutil stat gs://my-first-bucket54621/kitten.png

As you can see below, many of the default metadata are set, like the Creation time, Update time, Storage class, and more.

Pick one metadata you want to set or rewrite, but this tutorial’s choice is the Storage class:STANDARD.

Viewing a file’s metadata
Viewing a file’s metadata

2. Next, run the below gsutil rewrite command to set (-s) the storage class of the uploaded object (kitten.png) from STANDARD to nearline.

gsutil rewrite -s nearline gs://my-first-bucket54621/kitten.png
Rewriting the Storage class metadata value
Rewriting the Storage class metadata value

3. Rerun the gsutil stat command below to verify the changes to your file’s (*kitten.png*) metadata.

gsutil stat gs://my-first-bucket54621/kitten.png

Below, you can see that this time the Storage class is set to NEARLINE as expected.

Verifying the metadata changes
Verifying the metadata changes

Setting a Custom Metadata to an Object

Rewriting an object’s metadata comes in handy for object management. But can you also set your own metadata instead of rewriting the existing ones? Yes! These custom metadata are anything that helps describe or identify your objects.

Run the below gsutil command to set custom metadata (setmeta). Ensure you replace the following with your own:

KEY:VALUEA metadata key and value pair for your object. This example uses animal_type as the KEY and cat as the VALUE.
gs://OBJECT_PATHYour object’s path (URL) in your bucket (i.e., gs://my-first-bucket54621/kitten.png)

💡 Do not start your key with goog, as it is reserved for Google Cloud Storage. Additionally, do not use non-ASCII characters in your metadata key and value. Why? You might use an HTTP endpoint to access your objects, and non-ASCII characters will be rejected, which leads to an error.

gsutil setmeta -h "x-goog-meta-KEY:VALUE" gs://OBJECT_PATH
Setting a custom metadata
Setting a custom metadata

Now, run the following command to view your object’s (kitten.png) custom metadata.

gsutil stat gs://my-first-bucket54621/kitten.png

If successful, you will see your newly-set custom metadata, as shown below.

Viewing the custom metadata
Viewing the custom metadata

Copying an Object to Another Bucket

As an admin, making copies of objects for backup is crucial. But how exactly do you copy objects? Worry not. The gsutil cp command allows you to copy objects from one another to another quickly.

To see the cp command in action, follow these steps:

1. Create a second bucket as you did in the “Creating a Bucket with gsutil” section, and name the bucket as you like. But this tutorial uses my-second-bucket1234 as the name for this second bucket.

2. Next, run the below command to copy (cp) the object kitten.png from your first bucket ( my-first-bucket54621) to your second bucket (my-second-bucket1234).

gsutil cp gs://my-first-bucket54621/kitten.png gs://my-second-bucket1234/kitten.png

The object is copied to the second bucket, including its metadata.

Copying an object to another bucket
Copying an object to another bucket

3. Now, run the following command to verify you have successfully copied the object (kitten.png) to your second bucket (my-second-bucket1234).

gsutil stat gs://my-second-bucket1234/kitten.png

If successful, you will see the object in your second bucket with all its metadata intact, including custom ones, as shown below.

Verifying the object was successfully copied to another bucket
Verifying the object was successfully copied to another bucket

Deleting Objects and Buckets

Being charged for unnecessary storage consumption seem frustrating, especially when you set custom metadata. Why? Custom metadata is subject to storage charges aside from the object.

To avoid these charges, delete objects or buckets you no longer need with the following steps:

1. Run the following gsutil command to remove (rm) or delete an object (kitten.png) from your first bucket (my-first-bucket54621).

gsutil rm gs://my-first-bucket54621/kitten.png

You have deleted the kitten.png object in your first bucket, but the same object still exists in your second bucket.

Deleting objects from buckets
Deleting objects from buckets

Next, run each command below to list the objects (kitten.png) in your first (my-first-bucket54621) and second (my-second-bucket1234) buckets.

gsutil ls -l gs://my-first-bucket54621/kitten.png
gsutil ls -l gs://my-second-bucket1234/kitten.png

The output below confirms that the kitten.png object no longer exists in your first bucket but is still in your second bucket.

Verifying an object has been deleted
Verifying an object has been deleted

3. Run the below commands to delete your two buckets recursively. These commands delete all the objects in each bucket and their associated custom metadata.

gsutil rm -r gs://my-first-bucket54621
gsutil rm -r gs://my-second-bucket1234

Notice below that the first command immediately removed your first bucket, while the second command removed the object before removing your second bucket.

Deleting buckets
Deleting buckets

Finally, run the below gcloud projects command to delete your Google Cloud project (gsutil-project-demo). This command deletes your project and all its associated resources.

gcloud projects delete gsutil-project-demo

Input Y and press Enter when prompted to confirm deleting the project.

💡 After deleting a project, you will have a 30-day grace period if you ever change your mind and want to recover your project. Once the 30-day period has passed, the project and its resources are deleted and cannot be recovered.

Deleting a Google Cloud project
Deleting a Google Cloud project

Conclusion

As an alternative to the Google Cloud Console, gsutil lets you manage your Cloud Storage projects via Cloud Shell. And in this tutorial, you have learned how to use gsutil to create buckets, upload and copy objects, and delete unused ones to avoid recurring charges.

At this point, you can easily manage your Google Cloud projects from the Cloud Shell. But this tutorial can only cover the tip of the iceberg regarding what gsutil can offer.

With this newfound knowledge, why not use gsutil with Boto to automate your AWS cloud operations?

Hate ads? Want to support the writer? Get many of our tutorials packaged as an ATA Guidebook.

Explore ATA Guidebooks

Looks like you're offline!