Memory size exceeded while sending data in Azure Eventhub using python code

2022-01-19 16:42 Saswat Ray imported from Stackoverflow

ValueError: EventDataBatch has reached its size limit: 1048576

I am using below code

1 answer

answered 2022-01-20 05:37 UtkarshPal-MT

Use the add method to add events until the maximum batch size limit in bytes has been reached - at which point a ValueError will be raised.

As per the above statement from official document, the events you have added in batch has exceeded the maximum size limit which is 1 MB.

Make sure that the batch must contain the events' size upto 1MB only.

See also questions close to this topic

Python File Tagging System does not retrieve nested dictionaries in dictionary

I am building a file tagging system using Python. The idea is simple. Given a directory of files (and files within subdirectories), I want to filter them out using a filter input and tag those files with a word or a phrase.

If I got the following contents in my current directory:

data/
   budget.xls
   world_building_budget.txt
a.txt
b.exe
hello_world.dat
world_builder.spec

and I execute the following command in the shell:

py -3 tag_tool.py -filter=world -tag="World-Building Tool"

My output will be:

These files were tagged with "World-Building Tool":
data/ world_building_budget.txt
hello_world.dat
world_builder.spec

My current output isn't exactly like this but basically, I am converting all files and files within subdirectories into a single dictionary like this:

def fs_tree_to_dict(path_):
    file_token = ''
    for root, dirs, files in os.walk(path_):
        tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
        tree.update({f: file_token for f in files})
        return tree

Right now, my dictionary looks like this: key:''.

In the following function, I am turning the empty values '' into empty lists (to hold my tags):

def empty_str_to_list(d):
    for k,v in d.items():
        if v == '':
            d[k] = []
        elif isinstance(v, dict):
            empty_str_to_list(v)

When I run my entire code, this is my output:

hello_world.dat ['World-Building Tool']
world_builder.spec ['World-Building Tool']

But it does not see data/world_building_budget.txt. This is the full dictionary:

{'data': {'world_building_budget.txt': []},  
'a.txt': [],
'hello_world.dat': [], 
'b.exe': [], 
'world_builder.spec': []}

This is my full code:

import os, argparse


def fs_tree_to_dict(path_):
    file_token = ''
    for root, dirs, files in os.walk(path_):
        tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
        tree.update({f: file_token for f in files})
        return tree


def empty_str_to_list(d):
    for k, v in d.items():
        if v == '':
            d[k] = []
        elif isinstance(v, dict):
            empty_str_to_list(v)


parser = argparse.ArgumentParser(description="Just an example", formatter_class=argparse.ArgumentDefaultsHelpFormatter)

parser.add_argument("--filter", action="store", help="keyword to filter files")
parser.add_argument("--tag", action="store", help="a tag phrase to attach to a file")
parser.add_argument("--get_tagged", action="store", help="retrieve files matching an existing tag")

args = parser.parse_args()

filter = args.filter
tag = args.tag
get_tagged = args.get_tagged
current_dir = os.getcwd()

files_dict = fs_tree_to_dict(current_dir)
empty_str_to_list(files_dict)

for k, v in files_dict.items():
    if filter in k:
        if v == []:
            v.append(tag)
            print(k, v)

        elif isinstance(v, dict):
            empty_str_to_list(v)

    if get_tagged in v:
        print(k, v)

Actaully i am working on a project and in it, it is showing no module name pip_internal plz help me for the same. I am using pycharm(conda interpreter

 File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\Scripts\pip.exe\__main__.py", line 4, in <module>
  File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\__init__.py", line 4, in <module>
    from pip_internal.utils import _log

I am using pycharm with conda interpreter.

Looping the function if the input is not string

I'm new to python (first of all) I have a homework to do a function about checking if an item exists in a dictionary or not.

inventory = {"apple" : 50, "orange" : 50, "pineapple" : 70, "strawberry" : 30}

def check_item():
    x = input("Enter the fruit's name: ")
    if not x.isalpha():
        print("Error! You need to type the name of the fruit")
    elif x in inventory:
        print("Fruit found:", x)
        print("Inventory available:", inventory[x],"KG")
    else:
        print("Fruit not found")
check_item()

I want the function to loop again only if the input written is not string. I've tried to type return Under print("Error! You need to type the name of the fruit") but didn't work. Help

Deploy VueJS + API app to Azure Static Web App with Gitlab doesn't create functions

I've started creating a small application that will use VueJS as a frontend with Azure Functions as the backend. I was looking at using Azure Static Web Apps to host both components for the application and Gitlab to store / deploy the app.

Everything but the creation of the Azure functions works. Following https://docs.microsoft.com/en-us/azure/static-web-apps/gitlab?tabs=vue

The output from the deploy step, listed below is:

App Directory Location: '/builds/*/valhalla/valhalla-client/dist/spa' was found.
Api Directory Location: '/builds/*/valhalla/valhalla-api/dist' was found.
Looking for event info
Could not get event info. Proceeding
Starting to build app with Oryx
Azure Static Web Apps utilizes Oryx to build both static applications and Azure Functions. You can find more details on Oryx here: https://github.com/microsoft/Oryx
---Oryx build logs---
Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
You can report issues at https://github.com/Microsoft/Oryx/issues
Oryx Version: 0.2.20220131.3, Commit: ec344c058843461525ff03b46031553b6e15a47a, ReleaseTagName: 20220131.3
Build Operation ID: |qAffRWArEg8=.deee9498_
Repository Commit : 7cdd5b61f956e6cb8459b13a42af363c4440a97b
Detecting platforms...
Could not detect any platform in the source directory.
Error: Could not detect the language from repo.
---End of Oryx build logs---
Oryx was unable to determine the build steps. Continuing assuming the assets in this folder are already built. If this is an unexpected behavior please contact support.
Finished building app with Oryx
Starting to build function app with Oryx
---Oryx build logs---
Operation performed by Microsoft Oryx, https://github.com/Microsoft/Oryx
You can report issues at https://github.com/Microsoft/Oryx/issues
Oryx Version: 0.2.20220131.3, Commit: ec344c058843461525ff03b46031553b6e15a47a, ReleaseTagName: 20220131.3
Build Operation ID: |NGXLP5bVBRk=.705477f6_
Repository Commit : 7cdd5b61f956e6cb8459b13a42af363c4440a97b
Detecting platforms...
Could not detect any platform in the source directory.
Error: Could not detect the language from repo.
---End of Oryx build logs---
Oryx was unable to determine the build steps. Continuing assuming the assets in this folder are already built. If this is an unexpected behavior please contact support.
[WARNING] The function language could not be detected. The language will be defaulted to node.
Function Runtime Information. OS: linux, Functions Runtime: ~3, node version: 12
Finished building function app with Oryx
Zipping Api Artifacts
Done Zipping Api Artifacts
Zipping App Artifacts
Done Zipping App Artifacts
Uploading build artifacts.
Finished Upload. Polling on deployment.
Status: InProgress. Time: 0.1762737(s)
Status: InProgress. Time: 15.3950401(s)
Status: Succeeded. Time: 30.5043965(s)
Deployment Complete :)
Visit your site at: https://polite-pebble-0dc00000f.1.azurestaticapps.net
Thanks for using Azure Static Web Apps!
Exiting
Cleaning up project directory and file based variables 00:00
Job succeeded

The deploy step appears to have succeeded, and the frontend is deployed, but there are no Azure Functions showing up in this Static Web App. Is something missed here? So far, the Azure Functions I have are the boiler-plate from instantiating a new Azure Function folder.

image: node:latest

variables:
  API_TOKEN: $DEPLOYMENT_TOKEN
  APP_PATH: '$CI_PROJECT_DIR/valhalla-client/dist/spa'
  API_PATH: '$CI_PROJECT_DIR/valhalla-api/dist'

stages:
  - install_api
  - build_api
  - install_client
  - build_client
  - deploy

install_api:
  stage: install_api
  script:
    - cd valhalla-api
    - npm ci
  artifacts:
    paths:
      - valhalla-api/node_modules/
  cache:
    key: node
    paths:
      - valhalla-api/node_modules/
  only:
    - master
    
install_client:
  stage: install_client
  script:
    - cd valhalla-client
    - npm ci
  artifacts:
    paths:
      - valhalla-client/node_modules/
  cache:
    key: node
    paths:
      - valhalla-client/node_modules/
  only:
    - master

build_api:
  stage: build_api
  dependencies:
    - install_api
  script:
    - cd valhalla-api
    - npm install -g azure-functions-core-tools@3 --unsafe-perm true
    - npm run build
  artifacts:
    paths:
      - valhalla-api/dist
  cache:
    key: build_api
    paths:
      - valhalla-api/dist
  only:
    - master
  needs:
    - job: install_api
      artifacts: true
      optional: true

build_client:
  stage: build_client
  dependencies:
    - install_client
  script:
    - cd valhalla-client
    - npm i -g @quasar/cli
    - quasar build
  artifacts:
    paths:
      - valhalla-client/dist/spa
  cache:
    key: build_client
    paths:
      - valhalla-client/dist/spa
  only:
    - master
  needs:
    - job: install_client
      artifacts: true
      optional: true

deploy:
  stage: deploy
  dependencies:
    - build_api
    - build_client
  image: registry.gitlab.com/static-web-apps/azure-static-web-apps-deploy
  script:
    - echo "App deployed successfully."
  only:
    - master

Azure Synapse Notebooks Vs Azure Databricks notebooks
I was going through the features of Azure Synapse Notebooks Vs Azure Databricks notebooks.
1. Are there any major differences between these apart from the component they belong to ?
2. Are there any scenarios where one is more appropriate over other?

How to authorize azure container registry requests from .NET CORE C#

I have a web application which creates ContainerInstances, I have specific container registry images I want to use. As a result, I use this code to get my azure container registry

IAzure azure = Azure.Authenticate($"{applicationDirectory}/Resources/my.azureauth").WithDefaultSubscription();
IRegistry azureRegistry = azure.ContainerRegistries.GetByResourceGroup("testResourceGroup", "testContainerRegistryName");

I get this error when the second line of code is hit

The client 'bc8fd78c-2b1b-4596-827e-6a3c918b7c17' with object id 'bc8fd78c-2b1b-4596-827e-6a3c918b7c17' does not have authorization to perform action 'Microsoft.ContainerRegistry/registries/read' over scope '/subscriptions/506b787d-83ef-426a-b7b8-7bfcdd475855/resourceGroups/testapp-live/providers/Microsoft.ContainerRegistry/registries/testapp' or the scope is invalid. If access was recently granted, please refresh your credentials.

I literally have no idea what to do about this. I have seen so many articles talking about Azure AD and giving user roles and stuff. Can someone please walk me step by step how to fix this? I REALLY appreciate the help. Thanks.

I cannot find any client under that object ID so perfectly fine starting from scratch again with a better understanding of what I am doing.

Event Hub Get Status with REST API - 401 Unauthorized
I am attempting to use the REST API to get the Status of my event hub but it is returning "401 Claim is empty or token is invalid".

Using the C# example here: https://docs.microsoft.com/en-us/rest/api/eventhub/generate-sas-token#c I can build a SAS token for a POST
```
POST https://my-hubnamespace.servicebus.windows.net/my-eventHub/messages

And this works - 201 returned
```
Using the same SAS token I call:
```
GET https://my-hubnamespace.servicebus.windows.net/my-eventHub

401- Claim is empty or Token is invalid
```
What Claim is it referring to? I can't see any errors in my Portal.
429 Errors during Azure Deployment for EventHub
I am deploying an Bicep template which contains a number of IPFilterRules and other network resources.

During the Deployment each individual resource is getting a 429 error, it then retries a minute later and some more succeed and some more fail and then eventually after 5-10 minutes they all succeed.

Here is an example of how I'm creating my ip filter rules:
```
resource EventHubNamespace 'Microsoft.EventHub/namespaces@2021-06-01-preview' = {
  // ...
  resource IpFilterRules 'ipfilterrules@2018-01-01-preview' = [for ip in companyIps: {
    name: uniqueString(resourceGroup().name, ip)
    properties: {
      action: 'Accept'
      ipMask: ip
    }
  }]
}
```
Here is what the Deployment panel looks like while erroring:

How can I prevent these 429 errors from happening? Can I increase some limit to allow for a more resources? Or is there a way to alter my bicep template so these resources are created 1 at a time instead of all at once? Or is there any other technique I could use to speed up this process?
Stream Analytics to Azure SQL (duplicate data)

My use case:
IoT Hub -> Event Hub -> ASA -> Azure SQL (staging table)

The problem is that i get duplicate data in my staging table.
For testing purpose i sent exactly 10k json messages to IoT-Hub but my staging table then contains much more data like over 40k.

Is there something i have to adjust in event hub or in ASA? Is this normal for ASA to process duplicate messages?

Related questions

Memory size exceeded while sending data in Azure Eventhub using python code

1 answer

See also questions close to this topic