Cloud Storage and Signed URLs

Signed URLs are a generic concept widely adopted by many products and services especially which requires to provide temporary access to a certain user. These URLs are generated along with a expiry attached. So, the user will be able to access the resource only for the specific duration and then it becomes invalid. This writeup is all about how it is implemented in Google Cloud Storage(GCS) and how to test it from cli with cURL command.

GCS' Signed URLs provide a time-limited URL that users can use to upload or download a file without getting authenticated explicitly.

Signed URLs contain the security control information in the URL itself as query parameters. This means you can give someone very precise access to a single object without having to add any credentials to your account or open up the whole bucket.

Anyone with a valid signed URL can interact with the objects as specified during creation. For example, if a signed URL is created for the GET method, it can only be used to download the object without getting modified or deleted.

When to use? (sample use cases)

Assume you have a e-commerce application, where you want to send your users an authenticated link to download receipts for a specific duration. Your application can send an email embedding the Signed URL for the users to access and download the receipts(files) within the duration.

Assume you have an image gallery application, where you want to allow users from the browser to directly upload the images to their respective storage location rather than proxying with a web server in between. Your backend application creates a signed URL for uploading objects(files) and the front end(client-side) application can upload directly to the Google Cloud Storage bucket.

Things to remember:

• Like the signed URL has to be shared securely, as the URL can provide access to anyone who holds it. This is where validity plays an important role.
• Keep the validity as minimal as possible, based on the product needs.
• Allow only the specific methods when creating the URL.
• Be sure to omit the authorization header from any requests that use a signed URL. If both are used, Cloud Storage may authenticate against the credentials provided in the header, rather than the signed URL. Doing so could allow more access to resources than you intended

How to upload an object using a signed URL with cURL?

Step 1: Generate a signed URL with gsutil using following command:

gsutil signurl -m PUT -d 1h \
-c image/jpg \
<service account keys(.json)> \
gs://<bucket name>/<object name>

The above command generates signed URL with options,
‘m’ - Specifies the HTTP method (“PUT”) to be authorized for use with the signed URL, default is GET.
‘d’ - Specifies the duration that the signed URL should be valid for, default duration is 1 hour.
‘c’ - Specifies the content type for which the signed URL is valid for.
For more options, you can refer here.

The options in the above command are followed by service account keys and the path where you want to locate your object/file in the storage bucket.

The generated signed URL looks like,

https://storage.googleapis.com/<bucketname>/<objectname>?x-goog-signature=99be060fa0e8089e66cf4279c2b5aeeb65d821771a644803899ca7a6bcdbb6ee95d143c5f1f644fa3c2219f539b396647213f662004674eba0b0cdd1b081896825c1ba7bb95265b3e21599ceb1f22183ce4e5c426557176761997b8843b6011fe517614e12bdb842d0d238523ce2229a9739ba8ddd2c6f32f0df7e342d898ac739fc9a5404a049124810c590736afeb85f33fcf070b2c46bc5575fbab33b27d1773b5f24c4cd7dd0fc97f9c189469b4a9799a3fd736a9e7ff9de0d11cc8fc13b04395083e9e194554a90ce8b9234b0482c9d5f463c659a9852ec431c060f5870b302b61a4d764ea4b7e459faf08e86e37ba848e10697ac5ae4cc5937ae068308&x-goog-algorithm=GOOG4-RSA-SHA256&x-goog-credential=<serviceaccount>%40targetcka.iam.gserviceaccount.com%2F20220126%2Fus%2Fstorage%2Fgoog4_request&x-goog-date=20220126T072642Z&x-goog-expires=3600&x-goog-signedheaders=content-type%3Bhost

Step 2: Use cURL with required options and signed URL to upload an object by following command:

curl -X PUT \
-H "Content-TYpe:image/jpeg" \
-d "Content-Length:<size of the object>" \
--upload-file <object name> \
"<signed URL>"

Make sure that the content length should be in bytes.

The above command uploads the file with options,

‘X’ - Specifies a custom request method to use when communicating with the HTTP server (which defaults to GET).
‘H’ - Extra header to include in the request when sending HTTP to a server. You may specify any number of extra headers.

If your request is missing a required header, it will raise an error.

How to download an object using a signed URL with cURL?

With similar commands, you can download objects from the gcs bucket.

Step 1: Generate a signed  url with gsutil using following command:

gsutil signurl -d 1h \
-c image/jpg \
<service account keys(.json)> \
gs://<bucket name>/<object name>

Step 2: Use cURL with required options and signed URL to download an object by following command:

curl -X GET \
-H "Content-Type:image/jpeg" \
--output<object name> \
"<signed URL>"

Resumable uploads

Resumable upload allows you to resume data transfer operations to Cloud Storage after a communication failure has interrupted the flow of data. It works by sending multiple requests, each of which contains a portion of the object you’re uploading.

Step 1: Generate a signed  url with gsutil using following command:

gsutil signurl -m Resumable \
-d 1h \
-c image/jpg \
<service account key(.json)> \
gs://<bucket name>/<object name>

Step 2: Use cURL with required options and signed URL to get resumable url:

Curl -v -X POST \
-H "Content-Type:image/jpg" \
-H "x-goog-resumable:start" \
-d '' \
"<signed url>"

This command will generate a Location URI to upload the files. Location URI looks something like below,

https://storage.googleapis.com/<bucket name>/<object name>?x-goog-signature=2a13294517c8f2f209b990295da6b09a809394ea818953012a0298728124b253fcab38d0d40d7d9bd834d6e94c6a5c594cef0284d13d600552602a336c53a3c0a86ff99728be3d9a8b76eae6dad1fce507f1bd116feea902e5d112cc899535fc0e31f51b1092e9b3c47fd5efabd136c6c61c06889b494efeba8d72f345767a4b6f4aa890c08eb3883c35983932c7ac0f639d4ba77f00c4a88a7c55b7b21a0332b4fe62035a8882b0ad3613ca3a45fb36a4a37e1f92915518a98aad5f7c18c98d5f0e12ed91653778037acb1150cb8075d30f2f78413887257f74d27199eb648fd8374b7d307653ca3fd93a83eb449217741a49f4d4e93cc4797d641d25f7611c&x-goog-algorithm=GOOG4-RSA-SHA256&x-goog-credential=photo-album-runner%40targetcka.iam.gserviceaccount.com%2F20220202%2Fus%2Fstorage%2Fgoog4_request&x-goog-date=20220202T054541Z&x-goog-expires=3600&x-goog-signedheaders=content-type%3Bhost%3Bx-goog-resumable&upload_id=ADPycdvS4hW9BQmxI8fLMQmvPszRtBbbF1_0Ghay6GvbY4HX-Esab4y-B7FXeI6_PF3nNuOq6fZ_bzf9BhdPRD6vW0-TBvdxZg

Step 3: Now, use the URI with cURL to upload the file in the following way,

curl -v -X PUT \
-H "Content-Type:image/jpg" \
--upload-file <object name> \
<Location URI>

Simple, isn’t it!

References:
https://cloud.google.com/storage/docs/access-control/signed-urls