Python - List all the files and blob inside an Azure Storage Container

2022-05-06 11:44 Car_mine imported from Stackoverflow

This is my first post here on StackOverflow, hope it respects the guideline of this community.

I'm trying to accomplish a simple task in Python because even though I'm really new to it, I found it very easy to use. I have a storage account on Azure, with a lot of containers inside. Each container contains some random files and/or blobs.

What I'm trying to do, is to get the name of all these files and/or blob and put it on a file.

For now, I got here:

import os, uuid
import sys
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, __version__
connection_string = "my_connection_string"
blob_svc = BlobServiceClient.from_connection_string(conn_str=connection_string)


try:

    print("Azure Blob Storage v" + __version__ + " - Python quickstart sample")
    print("\nListing blobs...")
    containers = blob_svc.list_containers()
    list_of_blobs = []


    for c in containers:
      container_client = blob_svc.get_container_client(c)
      blob_list = container_client.list_blobs()
      for blob in blob_list:
        list_of_blobs.append(blob.name)
      file_path = 'C:/my/path/to/file/randomfile.txt'
      sys.stdout = open(file_path, "w")
      print(list_of_blobs)

except Exception as ex:
    print('Exception:')
    print(ex)

But I'm having 3 problems:

I'm getting the <name_of_ the_blob>/<name_of_the_file_inside>: I would like to have just the name of the file inside the blob
If in a container there is a blob (or more than 1 blob) + a random file, this script prints only the name of the blob + the name of the file inside, skipping the other files outside the blobs.
I would like to put all the names of the blobs/files in a .csv file.

But I'm not sure how to do point 3, and how to resolve points 1 and 2.

Cloud some maybe help on this?

Thanks!

Edit:

I'm adding an image here just to clarify a little what I mean when I talk about blob/files

1 answer

answered 2022-05-06 13:07 SwethaKandikonda-MT
Just to clarify that there are no 2 things such as files or blobs in the Blob Storage the files inside Blob Storage are called blobs. Below is the hierarchy that you can observe in blob storage.

Blob Storage > Containers > Directories/Virtual Folders > Blobs

I'm getting the <name_of_ the_blob>/<name_of_the_file_inside>: I would like to have just the name of the file inside the blob

for this, you can iterate through your container using list_blobs(<Container_Name>) taking only the names of the blobs i.e., blob.name. Here is how the code goes when you are trying to list all the blobs names inside a container.
```
generator = blob_service.list_blobs(CONTAINER_NAME)
for blob in generator:
    print("\t Blob name: "+c.name+'/'+  blob.name)
```
If in a container there is a blob (or more than 1 blob) + a random file, this script prints only the name of the blob + the name of the file inside, skipping the other files outside the blobs.

you can use iterate for containers using list_containers() and then use list_blobs(<Container_Name>) for iterating over the blob names and then finally write the blob names to a local file.

I would like to put all the names of the blobs/files in a .csv file.

A simple with open('<filename>.csv', 'w') as f write. Below is the sample code
```
with open('BlobsNames.csv', 'w') as f:
     f.write(<statements>)
```
Here is the complete sample code that worked for us where each blob from every folder will be listed.
```
import os
from azure.storage.blob import BlockBlobService

ACCOUNT_NAME = "<ACCOUNT_NAME>"
SAS_TOKEN='<YOUR_SAS_TOKEN>'

blob_service = BlockBlobService(account_name=ACCOUNT_NAME,account_key=None,sas_token=SAS_TOKEN)

print("\nList blobs in the container")
with open('BlobsNames.txt', 'w') as f:
    containers = blob_service.list_containers()
    for c in containers:
        generator = blob_service.list_blobs(c.name)
        for blob in generator:
            print("\t Blob name: "+c.name+'/'+  blob.name)
            f.write(c.name+'/'+blob.name)
            f.write('\n')    
```
This works even when there are folders in containers.

RESULT:

NOTE: You can just remove c.name while printing the blob to file if your requirement is to just pull out the blob names.

See also questions close to this topic

Python File Tagging System does not retrieve nested dictionaries in dictionary

I am building a file tagging system using Python. The idea is simple. Given a directory of files (and files within subdirectories), I want to filter them out using a filter input and tag those files with a word or a phrase.

If I got the following contents in my current directory:

data/
   budget.xls
   world_building_budget.txt
a.txt
b.exe
hello_world.dat
world_builder.spec

and I execute the following command in the shell:

py -3 tag_tool.py -filter=world -tag="World-Building Tool"

My output will be:

These files were tagged with "World-Building Tool":
data/ world_building_budget.txt
hello_world.dat
world_builder.spec

My current output isn't exactly like this but basically, I am converting all files and files within subdirectories into a single dictionary like this:

def fs_tree_to_dict(path_):
    file_token = ''
    for root, dirs, files in os.walk(path_):
        tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
        tree.update({f: file_token for f in files})
        return tree

Right now, my dictionary looks like this: key:''.

In the following function, I am turning the empty values '' into empty lists (to hold my tags):

def empty_str_to_list(d):
    for k,v in d.items():
        if v == '':
            d[k] = []
        elif isinstance(v, dict):
            empty_str_to_list(v)

When I run my entire code, this is my output:

hello_world.dat ['World-Building Tool']
world_builder.spec ['World-Building Tool']

But it does not see data/world_building_budget.txt. This is the full dictionary:

{'data': {'world_building_budget.txt': []},  
'a.txt': [],
'hello_world.dat': [], 
'b.exe': [], 
'world_builder.spec': []}

This is my full code:

import os, argparse


def fs_tree_to_dict(path_):
    file_token = ''
    for root, dirs, files in os.walk(path_):
        tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
        tree.update({f: file_token for f in files})
        return tree


def empty_str_to_list(d):
    for k, v in d.items():
        if v == '':
            d[k] = []
        elif isinstance(v, dict):
            empty_str_to_list(v)


parser = argparse.ArgumentParser(description="Just an example", formatter_class=argparse.ArgumentDefaultsHelpFormatter)

parser.add_argument("--filter", action="store", help="keyword to filter files")
parser.add_argument("--tag", action="store", help="a tag phrase to attach to a file")
parser.add_argument("--get_tagged", action="store", help="retrieve files matching an existing tag")

args = parser.parse_args()

filter = args.filter
tag = args.tag
get_tagged = args.get_tagged
current_dir = os.getcwd()

files_dict = fs_tree_to_dict(current_dir)
empty_str_to_list(files_dict)

for k, v in files_dict.items():
    if filter in k:
        if v == []:
            v.append(tag)
            print(k, v)

        elif isinstance(v, dict):
            empty_str_to_list(v)

    if get_tagged in v:
        print(k, v)

Actaully i am working on a project and in it, it is showing no module name pip_internal plz help me for the same. I am using pycharm(conda interpreter

 File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\Scripts\pip.exe\__main__.py", line 4, in <module>
  File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\__init__.py", line 4, in <module>
    from pip_internal.utils import _log

I am using pycharm with conda interpreter.

Looping the function if the input is not string

I'm new to python (first of all) I have a homework to do a function about checking if an item exists in a dictionary or not.

inventory = {"apple" : 50, "orange" : 50, "pineapple" : 70, "strawberry" : 30}

def check_item():
    x = input("Enter the fruit's name: ")
    if not x.isalpha():
        print("Error! You need to type the name of the fruit")
    elif x in inventory:
        print("Fruit found:", x)
        print("Inventory available:", inventory[x],"KG")
    else:
        print("Fruit not found")
check_item()

I want the function to loop again only if the input written is not string. I've tried to type return Under print("Error! You need to type the name of the fruit") but didn't work. Help

How to get blob URL using Logic App Standard

How to get the blob URL from Azure Blob Storage?

The trigger is when a blob is added or modified. The blob content could be read and I was very happy as I could insert the content into an Azure Table property, so using PK and RK the file content would be quickly retrieved.

However with larger blobs it did not work due to 32k max size for a property. The approach to overcome this limitation seems to be to save the URL blob into the property but for my life I cannot figure it out how (tried almost all actions, even those that did not make sense, perhaps the url was somewhere in the output)

Azure Blob Starage - Rest API - 400 One of the request inputs is not valid

I am trying to delete a blob from an Azure Blob Storage with a batch request. At the moment I am using postman to verify my request before I use it in my application. I cannot manage to get a different output or error. For authorization I use SAS.

Here is my request:
Url: https://APPLICATIONNAME.blob.core.windows.net/MYCONTAINER?restype=container&comp=batch&sig=SIGNATURE&sp=racwdl&sip=92.60.9.18&sv=2020-08-04&st=2022-04-27T08:52:29Z&spr=https&se=2022-05-30T16:52:29Z&sr=c

Headers:

Content-Type: multipart/mixed; boundary=batch_357de4f7-6d0b-4e02-8cd2-6361411a9525
x-ms-version: 2020-08-04

Body:

--batch_357de4f7-6d0b-4e02-8cd2-6361411a9525
Content-Type: application/http
Content-Transfer-Encoding: binary
Content-ID: 0

DELETE /MYCONTAINER/NAVIGATIONS.sql?sp=racwdl&sig=SIGNATURE&sip=92.60.9.18&sv=2020-08-04&st=2022-04-27T08:52:29Z&spr=https&se=2022-05-30T16:52:29Z&sr=c HTTP/1.1
Content-Length: 0

--batch_357de4f7-6d0b-4e02-8cd2-6361411a9525--

Response:

--batchresponse_cdb19852-af76-4719-b9d2-8d61b5d3e6da
Content-Type: application/http

HTTP/1.1 400 One of the request inputs is not valid.
x-ms-error-code: InvalidInput
x-ms-request-id: 4dd49405-d01e-0035-12c4-601f12000000
x-ms-version: 2020-08-04
Content-Length: 221
Content-Type: application/xml
Server: Windows-Azure-Blob/1.0

<?xml version="1.0" encoding="utf-8"?>
<Error><Code>InvalidInput</Code><Message>One of the request inputs is not valid.
RequestId:4dd49405-d01e-0035-12c4-601f12000000
Time:2022-05-05T21:07:45.5349988Z</Message></Error>
--batchresponse_cdb19852-af76-4719-b9d2-8d61b5d3e6da--

I hope someone can give me a hint or solution.

Azure storage account container name format is incorrect

I'm trying to read a json file from storage account from a container name, the container name is (relative path )

var-storage/business/web/

file name is :

test.json

when I try my code :

blobService.getBlobToText(container, blobname, function(error, text){

if(error){

    console.error(error);

    res.status(500).send('Fail to download blob');

} else {

   var data = JSON.parse(text);

   res.status(200).send('Filtered Data you want to send back');

}

});

it gives me this error :

container name format is incorrect

can you help please ?

Azure file storage downloading file stream, Stream.Length throws NotSupportedException

I am downloading a file from Azure file storage and the Stream.Length throws NotSupportedException. I am using sharefileclient.DownloadAsync() method. I have to use the Stream object along with it's length so that I can upload the file in other directory of the Azure file storage. How to solve this?
how to connect or acquire a connection to a blob storage via outbound proxy
```
curl -v -X GET 'https://outboundproxy.xx.net:8443/<Container_name>/<SAS>Token' -H 
'Host: Azurestorage.blob.core.windows.net' -H "Rest of the curl 
headers"
```
We have a outbound proxy inplace and all of the external connection has to go via an 'https://outboundproxy.xx.net:8443'. Hence, In the above curl command i have used host header to pass the actual storage account base url.

Now, I want to replicate it using the python Azure blob SDK module but I am not sure how to structure the account url parameter. Any pointer or hint will be helpful
Azure blob storage - findBlobsByTags always fails with error Error parsing query at or near character position 1: unexpected '1'
I am using Azure Blob storage. While persisting blobs, I have persisted index tags. I am able to see index tags when I download the blobs through Microsoft Azure storage explorer. Programatically, I am trying to filter the tags(using azure-storage-blob version 12.15.0)

I am using this api to get details from DB - findBlobsByTags

This is the code I am trying
```
    FindBlobsOptions findByTags = new FindBlobsOptions(searchExpression));
    findByTags.setMaxResultsPerPage(100);
    PagedIterable<TaggedBlobItem> blobsByTags = blobContainerClient.findBlobsByTags(findByTags, Duration.ofSeconds(30), Context.NONE);
```
I have tried following queries
```
 1.   String searchExpression = "where=@container = 'container-name' AND \"field\" = \'value\'";
 2. String searchExpression = "where=field=value";
 3. String searchExpression = "where=\"field\" =\'value\'";
     String escapeJava = StringEscapeUtils.escapeJava(searchExpression);
 4. String searchExpression = "&where=\"trackingId\"=\'TA00965650\'";
```
and I am getting this exception.

[Request processing failed; nested exception is com.azure.storage.blob.models.BlobStorageException: Status code 400, "

InvalidQueryParameterValueError parsing query at or near character position 1: unexpected '1' RequestId:78043a6d-901e-00de-0797-510acf000000 Time:2022-04-16T13:42:00.7475672Zwherewhere="field"='value'This query parameter value is invalid."] with root cause com.azure.storage.blob.models.BlobStorageException: Status code 400, "

InvalidQueryParameterValueError parsing query at or near character position 1: unexpected '1' RequestId:78043a6d-901e-00de-0797-510acf000000 Time:2022-04-16T13:42:00.7475672Zwherewhere="trackingId"='TA00965650'This query parameter value is invalid." at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.azure.core.http.rest.RestProxy.instantiateUnexpectedException(RestProxy.java:390) at

From the exception log it looks like the query I am passing to the findBlobsByTags method has issues in parsing. Please let me know what I am doing wrong. If some escaping/encoding needs to be done

Snowflake and Data Factory Error because of regions

Im getting an error when running Data Factory to load data into Snowflake:

ErrorCode=SnowflakeUnsupportedCouldPlatformForImport,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Snowflake only support the account hosted in azure platform while as sink connector, please make sure your account is hosted in azure, current region and platform is '',Source=Microsoft.DataTransfer.ClientLibrary,'

Does the azure blob storage (staging area for copy command) needs to be in the same region as where snowflake was provision?

Related questions

Python - List all the files and blob inside an Azure Storage Container

1 answer

See also questions close to this topic