Lab - Cloud Storage

In the last lesson, you learned how to create and upload files to the GCS bucket using UI.

In this lab, you will use Gcloud CLI to carry out the different operations using gsutil component of the SDK. You can use your cloud shell, laptop, or the terminal provided at the end of the lesson to complete this lab.

• Run gcloud init in case you are using your laptop or terminal provided at the end of the lesson. This will configure the gcloud SDK with your GCP account.

• Run gcloud config list and check whether your account and project are set correctly.

root@educative:/# gcloud config list
[core]
account = gcpheadstart@gmail.com #Your email
disable_usage_reporting = True
project = gcp-headstart-educative #your project

Your active configuration is: [default]

Creating a bucket

1. Type gsutil ls to list existing buckets. If you get a list of buckets then the configurations are right for the project you have chosen.

2. Type gsutil mb gs://<bucketname>. The bucket name needs to be a globally unique name.

3. Type gsutil ls to see your newly created bucket.

Now we will upload files to the bucket.

Uploading files

There 2 ways in which you can upload a file to the bucket from your system. Using copy or move command. Both are self-explanatory. To copy, use gsutil cp [source_location] [destination] and to move use gsutil mv [source] [destination] command. We will move the files for now.

• Type echo "demo file" > demo.txt to create a file in the current directory.

• gsutil mv ./demo.txt gs://[bucketname]. This will move the file into the bucket.

• Type gsutil ls gs://[bucketname] to cross check.

Multithreaded upload

If you need to upload so many files at once, then using multithreaded upload will save a lot of time.

Adding -m option to gsutil command runs it in a multithreaded way.

• gsutil -m mv [largedirectory] gs://[bucketname]

Example of moving current directory files to bucket.

$ gsutil -m mv . gs://cloudstoragelab/

Parallel composite upload

In contrast, if you have a large file say >1GB. You can use parallel composite upload which will upload the files in chunks. Use -o option to use this feature.

• Type gsutil -o GSUTIL:parallel_composite_upload_component_size=10M mv [bigfile] gs://[bucketname]

The parallel_composite_upload_component_size parameter defines the size of the chunk while uploading the big file.

Creating a signed URL

Signed URL is the feature of cloud storage where you can share any file from the bucket with an expiration time for the link and access.

This is useful when you have to share some files to an outsider or any third party which does not have an active Google Cloud account.

The signurl command uses the private key for a service account to generate the cryptographic signature for the generated URL. So, we need to create a service account key file. Follow the steps to do that.

• Open main menu> Identity > service accounts

• Click on “create a service account”.

• Fill the form.

• Assign the storage admin role.

• Download the key file in JSON format.

If you are using the cloud shell, upload the file to the current directory.

• Type gcloud auth activate-service-account --key-file [path/to/key_file.json]

• We will generate the URL to access the demo file we pushed earlier.

You might get an error that will ask you to install pyopenssl so before creating the signed URL, install the pyopenssl using pip3 install pyopenssl.

• Now, type gsutil signurl -d 10m -u gs://[bucket_name]/demo.txt

This URL will be valid for 10m means 10 minutes.

Example:

gcpheadstart@cloudshell:~ (gcp-headstart-educative)$ gcloud auth activate-service-account --key-file gcp-headstart-educative-d2b5d3bfeb13.json
Activated service account credentials for: [signurl@gcp-headstart-educative.iam.gserviceaccount.com]


gcpheadstart@cloudshell:~ (gcp-headstart-educative)$ gsutil signurl -d 10m -u gs://cloudstoragelab/demo/demo.txt
URL     HTTP Method     Expiration      Signed URL
gs://cloudstoragelab/demo/demo.txt      GET     2020-12-19 13:43:56     https://storage.googleapis.com/cloudstoragelab/demo/demo.txt?x-goog-signature=08a38d07f1f144031c3cc12754a4ef119c40199d6ff6e36177e47b1b613af142dd34a61492588b2ac93c1e809e3d678346500356a5a43ff9f87bb6abec9e22cbfc2f912441e1c5bf8f0ac5f9a790d718763bedcb72f02437cb7f0714d644e0c604881a55189dbf9b4841c3b942d41ad26716418962176ea671cea17ca9515a3948344610e8b734fd574379acbcb629fde3773dafa9665cdbfc4a90c93f6f16ac0555553a4ed5c3e3fb3303ba190059f02368d3a59b6bdc0267b7bf90f0048ff3cd850cca265bbaee6d321c65dd4d5ab643ec38f852ad284e13fe64da214f77c4bc50c6dbf3b4c9780dcafe79354fa983c0ef99c47ad2b17745c898f5efb83903&x-goog-algorithm=GOOG4-RSA-SHA256&x-goog-credential=signurl%40gcp-headstart-educative.iam.gserviceaccount.com%2F20201219%2Fus%2Fstorage%2Fgoog4_request&x-goog-date=20201219T133356Z&x-goog-expires=600&x-goog-signedheaders=host
gcpheadstart@cloudshell:~ (gcp-headstart-educative)$

You can find more about signed URLs here.