22

How can I upload a file to Google Cloud Storage from Python 3? Eventually Python 2, if it's infeasible from Python 3.

I've looked and looked, but haven't found a solution that actually works. I tried boto, but when I try to generate the necessary .boto file through gsutil config -e, it keeps saying that I need to configure authentication through gcloud auth login. However, I have done the latter a number of times, without it helping.

  • 1
    The gcloud bundle uses its own auth mechanism that shares credentials amongst all its bundled CLIs. To configure a .boto file like this, you'll need to use the standalone install of gsutil. That being said, I'm not confident that the gcs_oauth2_boto_plugin module supports Python 3 (yet). So you may need to do your own port of that module, or the pieces of it that you need, to Python 3. – Travis Hobrla May 3 '16 at 16:11
41

Use the standard gcloud library, which supports both Python 2 and Python 3.

Example of Uploading File to Cloud Storage

from gcloud import storage
from oauth2client.service_account import ServiceAccountCredentials
import os


credentials_dict = {
    'type': 'service_account',
    'client_id': os.environ['BACKUP_CLIENT_ID'],
    'client_email': os.environ['BACKUP_CLIENT_EMAIL'],
    'private_key_id': os.environ['BACKUP_PRIVATE_KEY_ID'],
    'private_key': os.environ['BACKUP_PRIVATE_KEY'],
}
credentials = ServiceAccountCredentials.from_json_keyfile_dict(
    credentials_dict
)
client = storage.Client(credentials=credentials, project='myproject')
bucket = client.get_bucket('mybucket')
blob = bucket.blob('myfile')
blob.upload_from_filename('myfile')
  • 3
    oh my lord, thank you so much. I've been looking for something simple to describe how to connect to the system. there's no "authentication" teachings around. This is great! Thanks – arcee123 Jul 3 '17 at 13:25
  • 1
    Please note oauth2client is deprecated. github.com/google/oauth2client. Also, in the newer versions of google cloud python packages, use from google.cloud import storage instead of from gcloud... – Shiva Sep 3 '18 at 7:10
12

A simple function to upload files to a gcloud bucket.

from google.cloud import storage

def upload_to_bucket(blob_name, path_to_file, bucket_name):
    """ Upload data to a bucket"""

    # Explicitly use service account credentials by specifying the private key
    # file.
    storage_client = storage.Client.from_service_account_json(
        'creds.json')

    #print(buckets = list(storage_client.list_buckets())

    bucket = storage_client.get_bucket(bucket_name)
    blob = bucket.blob(blob_name)
    blob.upload_from_filename(path_to_file)

    #returns a public url
    return blob.public_url

You can generate a credential file using this link: https://cloud.google.com/storage/docs/reference/libraries?authuser=1#client-libraries-install-python

Asynchronous Example:

import asyncio
import aiohttp
# pip install aiofile
from aiofile import AIOFile
# pip install gcloud-aio-storage
from gcloud.aio.storage import Storage 

BUCKET_NAME = '<bucket_name>'
FILE_NAME  = 'requirements.txt'
async def async_upload_to_bucket(blob_name, file_obj, folder='uploads'):
    """ Upload csv files to bucket. """
    async with aiohttp.ClientSession() as session:
        storage = Storage(service_file='./creds.json', session=session) 
        status = await storage.upload(BUCKET_NAME, f'{folder}/{blob_name}', file_obj)
        #info of the uploaded file
        # print(status)
        return status['selfLink']


async def main():
    async with AIOFile(FILE_NAME, mode='r') as afp:
        f = await afp.read()
        url = await async_upload_to_bucket(FILE_NAME, f)
        print(url)


# Python 3.6
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

# Python 3.7+
# asyncio.run(main()) 
  • Is this method synchronous? – DJ319 Jun 21 at 9:45
  • @DJ319 I have also added asynchronous example – adam shamsudeen Aug 21 at 16:13
4

Imports the Google Cloud client library (need credentials)

from google.cloud import storage
import os
os.environ["GOOGLE_APPLICATION_CREDENTIALS"]="C:/Users/siva/Downloads/My First Project-e2d95d910f92.json" 

Instantiates a client

storage_client = storage.Client()

buckets = list(storage_client.list_buckets())

bucket = storage_client.get_bucket("ad_documents")//your bucket name

blob = bucket.blob('/chosen-path-to-object/{name-of-object}')
blob.upload_from_filename('D:/Download/02-06-53.pdf')
print(buckets)
1

When installing Google Cloud Storage API:

pip install google-cloud

will throw a ModuleNotFoundError:

    from google.cloud import storage
ModuleNotFoundError: No module named 'google'

Make sure you install as in Cloud Storage Client Libraries Docs:

pip install --upgrade google-cloud-storage

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.