See also questions close to this topic

• Python File Tagging System does not retrieve nested dictionaries in dictionary

  I am building a file tagging system using Python. The idea is simple. Given a directory of files (and files within subdirectories), I want to filter them out using a filter input and tag those files with a word or a phrase.

  If I got the following contents in my current directory:

  data/
     budget.xls
     world_building_budget.txt
  a.txt
  b.exe
  hello_world.dat
  world_builder.spec
  

  and I execute the following command in the shell:

  py -3 tag_tool.py -filter=world -tag="World-Building Tool"
  

  My output will be:

  These files were tagged with "World-Building Tool":
  data/ world_building_budget.txt
  hello_world.dat
  world_builder.spec
  

  My current output isn't exactly like this but basically, I am converting all files and files within subdirectories into a single dictionary like this:

  def fs_tree_to_dict(path_):
      file_token = ''
      for root, dirs, files in os.walk(path_):
          tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
          tree.update({f: file_token for f in files})
          return tree
  

  Right now, my dictionary looks like this: key:''.

  In the following function, I am turning the empty values '' into empty lists (to hold my tags):

  def empty_str_to_list(d):
      for k,v in d.items():
          if v == '':
              d[k] = []
          elif isinstance(v, dict):
              empty_str_to_list(v)
  

  When I run my entire code, this is my output:

  hello_world.dat ['World-Building Tool']
  world_builder.spec ['World-Building Tool']
  

  But it does not see data/world_building_budget.txt. This is the full dictionary:

  {'data': {'world_building_budget.txt': []},  
  'a.txt': [],
  'hello_world.dat': [], 
  'b.exe': [], 
  'world_builder.spec': []}
  
  

  This is my full code:

  import os, argparse
  
  
  def fs_tree_to_dict(path_):
      file_token = ''
      for root, dirs, files in os.walk(path_):
          tree = {d: fs_tree_to_dict(os.path.join(root, d)) for d in dirs}
          tree.update({f: file_token for f in files})
          return tree
  
  
  def empty_str_to_list(d):
      for k, v in d.items():
          if v == '':
              d[k] = []
          elif isinstance(v, dict):
              empty_str_to_list(v)
  
  
  parser = argparse.ArgumentParser(description="Just an example", formatter_class=argparse.ArgumentDefaultsHelpFormatter)
  
  parser.add_argument("--filter", action="store", help="keyword to filter files")
  parser.add_argument("--tag", action="store", help="a tag phrase to attach to a file")
  parser.add_argument("--get_tagged", action="store", help="retrieve files matching an existing tag")
  
  args = parser.parse_args()
  
  filter = args.filter
  tag = args.tag
  get_tagged = args.get_tagged
  current_dir = os.getcwd()
  
  files_dict = fs_tree_to_dict(current_dir)
  empty_str_to_list(files_dict)
  
  for k, v in files_dict.items():
      if filter in k:
          if v == []:
              v.append(tag)
              print(k, v)
  
          elif isinstance(v, dict):
              empty_str_to_list(v)
  
      if get_tagged in v:
          print(k, v)
  
• Actaully i am working on a project and in it, it is showing no module name pip_internal plz help me for the same. I am using pycharm(conda interpreter

   File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\Scripts\pip.exe\__main__.py", line 4, in <module>
    File "C:\Users\pjain\AppData\Local\Programs\Python\Python310\lib\site-packages\pip\_internal\__init__.py", line 4, in <module>
      from pip_internal.utils import _log
  

  I am using pycharm with conda interpreter.

  
• Looping the function if the input is not string

  I'm new to python (first of all) I have a homework to do a function about checking if an item exists in a dictionary or not.

  inventory = {"apple" : 50, "orange" : 50, "pineapple" : 70, "strawberry" : 30}
  
  def check_item():
      x = input("Enter the fruit's name: ")
      if not x.isalpha():
          print("Error! You need to type the name of the fruit")
      elif x in inventory:
          print("Fruit found:", x)
          print("Inventory available:", inventory[x],"KG")
      else:
          print("Fruit not found")
  check_item()
  

  I want the function to loop again only if the input written is not string. I've tried to type return Under print("Error! You need to type the name of the fruit") but didn't work. Help
• Images with Firebase and Python Flask API

  I am currently developing an API using Firebase from google and Python's Flask libraries. It is a proyect where I am in need of saving images to the DB and then adress them in the API. I would also like to know how to relate the image to an item in the database, say posting an image of Juan, and that is linked with ALL the information from Juan inside the DB. Thanks!
• Pandas 1.4.2 upgrade casuing FFlask: array(0.78015261) (0d array) is not JSON serializable at the moment

  We upgraded to a newer version of python with new pandas, numpy, etc. The versions are now:

  - python=3.10.4
  - pandas=1.4.2
  - numpy=1.22.3
  

  In previous versions, the error never occurred. When I do some debugging, it's not that the .to_json() is wrong or fails, it's that the pd.DataFrame([my_result]) doesn't return correctly.

  The code is this:

  // Used in debugging and doesn't return properly
  dataframe_test = pd.DataFrame([my_result])
  
  // Error gets thrown here
  return pd.DataFrame([my_result]).to_json()
  

  The dataframe_test looks like this when I go to view it in Data Viewer within VS Code (it also keeps processing as the data seems to still be running in VS Code - the bar above the viewer is indicating that its still trying to run/process):

  

  The my_result variable looks like this upon entering the dataframe:

  

  

  I am not sure what exactly is causing the issue and not sure how to debug in Pandas to see what is happening. Any ideas?
• Flask select which form to POST by button click

  I'm trying to have only one of two forms POST depending on which button from a btn-group is selected. Currently all of the forms POST with no issue, but there are two forms where only one or the other value is needed. I've unsuccessfully tried to parse the not needed value out at the app.py, so I decided to try and make this so only one or the other value gets posted.

  Here is the code from the .html where I'm having trouble, it's a fieldset from a larger form the rest of which is working for now.

   <fieldset class="row mb-3, container" id="program_value_form_id">
          <legend for="value_range" class="col-sm-2 col-form-label">Value Range:</legend>
              <p>
                  <div class="btn-group, com-sm-1" role="group" aria-label="Basic radio toggle button group">
                      <input type="radio" onchange="swapConfig(this)" class="btn-check" name="btnradio_valuer" id="btnradio_value1" autocomplete="off" value="valuerange1" checked>
                      <label class="btn btn-outline-primary" for="btnradio_value1">100-200</label>
  
                      <input type="radio" onchange="swapConfig(this)" class="btn-check" name="btnradio_valuer" id="btnradio_valuer2" autocomplete="off" value="valuerange2">
                      <label class="btn btn-outline-primary" for="btnradio_valuer2">400-500mhz</label>
                  </div>
              </p>
              <div id="btnradio_valuer1Swap">
                  <label for="value" class="col-sm-2 col-form-label">Value:</label>
                  <p>
                      <div class="col-sm-4">
                          <input id="value1" type="number" class="form-control" placeholder="xxx.xxx 100-200" name="value1" step="0.001" min="100" max="200">
                          <span class="validity"></span>
                      </div>
                  </p>
              </div>
              <div id="btnradio_valuer2Swap" style="display:none">
                  <label for="value" class="col-sm-2 col-form-label">Value:</label>
                  <p>
                      <div class="col-sm-4">
                          <input id="value2" type="number" class="form-control" placeholder="xxx.xxx 400-500" name="value2" step="0.001" min="400" max="500">
                          <span class="validity"></span>
                      </div>
                  </p>
              </div>                  
      </fieldset>
  

  The forms swap depending on button click. Here is the js for that I got from on here to swap them.

  <script>
      function swapConfig(x) {
        var radioName = document.getElementsByName(x.name);
        for(i = 0 ; i < radioName.length; i++){
          document.getElementById(radioName[i].id.concat("Swap")).style.display="none";
        }
        document.getElementById(x.id.concat("Swap")).style.display="initial";
      }
  </script>
  

  I have tried if statements and if's inside of for's, none have worked. In frustration I've deleted them, but I could try and rewrite them again if they are needed though I wouldn't expect much from them since my html experience is limited. Please let me know if there needs to be any corrections to what I've written or if there is a better way or place to do what I am trying to do.
• how can I solve this circular dependent imports in order to execute many to one insert

  I have the following two models in several files:

  routes/camera/model.py

  from services.db import db
  from dataclasses import dataclass
  from routes.project.models import Project
  
  @dataclass
  class Camera(db.Model):
      __tablename__ = 'cameras'
      id: int
      name: str
      project_id: int
      url: str
  
      id = db.Column(db.Integer, primary_key=True)
      name = db.Column(db.String(80), unique=True, nullable=False)
      url = db.Column(db.String(1000), unique=True, nullable=False)
      project_id = db.Column(db.Integer, db.ForeignKey('projects.id'), nullable=False)
      
  db.create_all()
  

  routes/project/model.py

  from services.db import db
  from dataclasses import dataclass
  from routes.camera.models import Camera
  
  @dataclass
  class Project(db.Model):
      __tablename__ = 'projects'
      id: int
      name: str
  
      id = db.Column(db.Integer, primary_key=True)
      name = db.Column(db.String(80), unique=True, nullable=False)
      cameras = db.relationship(Camera, backref='projects')
      
  db.create_all()
  

  The problem is that I need make a insert by project name in camera in order to do this I have the following approach:

  def create_camera(name: str, url:str, project_name:str):
      project = Project(name=project_name)
      camera = Camera(name=name, project_id=project, url=url)
      db.session.add(camera)
      db.session.commit()
  

  but I get a circular dependant and I cannot make this insert

  So, how can I solve this problem (circular dependent import) or the insert in order to avoid to instantiate the Project class?

  Thanks
• How to get rid of sqlalchemy.exc.NoReferencedTableError?

  Here's some code related to the problem:

  db_session.py

  import sqlalchemy as sa
  import sqlalchemy.orm as orm
  import sqlalchemy.ext.declarative as dec
  
  SqlAlchemyBase = dec.declarative_base()
  
  __Session = None
  
  
  def create_session():
      global __Session
      global_init()
      return __Session()
  
  
  def global_init():
      global __Session
  
      if __Session:
          return
  
      conn_str = f'sqlite:///sqlite.db'
  
      engine = sa.create_engine(conn_str, echo=True, future=True)
      __Session = orm.sessionmaker(bind=engine)
  
      SqlAlchemyBase.metadata.create_all(engine)
  

  User.py (error comes from User.department_id column)

  import sqlalchemy
  from db_session import SqlAlchemyBase
  
  
  class User(SqlAlchemyBase):
      __tablename__ = 'users'
  
      id = sqlalchemy.Column(sqlalchemy.Integer,
                             primary_key=True, autoincrement=True)
      department_id = sqlalchemy.Column(sqlalchemy.Integer, 
                                        sqlalchemy.ForeignKey('departments.id'), nullable=True)
  

  Departments.py

  import sqlalchemy
  from db_session import SqlAlchemyBase
  
  
  class Departments(SqlAlchemyBase):
      __tablename__ = 'departments'
  
      id = sqlalchemy.Column(sqlalchemy.Integer, 
                             primary_key=True, autoincrement=True)
      chief = sqlalchemy.Column(sqlalchemy.Integer, sqlalchemy.ForeignKey("users.id"))
      members = sqlalchemy.orm.relationship("User", foreign_keys="User.department_id")
  

  The error occurs whenever I try to access the 'users' table. Here's the error text:

  sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'users.department_id' could not find table 'departments' with which to generate a foreign key to target column 'id'
  

  I've checked the sqlite.db file with database management software and there definitely is 'departments' table. I've printed the output of

  SqlAlchemyBase.metadata.tables.keys()
  

  and the 'departments' table is there, too, and even has the right name. I've tried to add something to the 'departments' table, but that also didn't help. All tables' names are set explicitly with tablename property, so sqlalchemy's automatic table naming isn't the problem. How do I get rid of this error?
• Setting Data Frame Column Names with Data Frame includes extra characters: ('ColumnName',)

  I've got a python script set to pull data and column names from a Pervasive PSQL database, and it then creates the table and records in MS SQL. I'm creating data frames for the data and for the column names, then renaming the the data's column names from the column name date frame.

  However, when the table is created in MS SQL the column names come in as = ('ColumnName',)

  

  The desired column names would not have ('',) and should read as = ColumnName

  Below is the code i'm using to get here. Any help on formatting the column names to not include those extra characters would be very helpful!

  '''
  Start - Pull Table Data
  '''
  conn = pyodbc.connect(conn_str, autocommit=True)
  cursor = conn.cursor()
  statement = 'select * from '+db_tbl_name
  stRows = cursor.execute(statement)
  df_data = pandas.DataFrame((tuple(t) for t in stRows))
  df_data = df_data.applymap(str)
  '''
  End - Pull Table Data
  '''
  
  '''
  Start - Pull Column Names
  '''
  conn = pyodbc.connect(conn_str, autocommit=True)
  cursor_col = conn.cursor()
  statement_col = "select CustomColumnName from "+db_col_tbl_name+" where CustomTableName = '"+db_tbl_name+"' and ODBCOptions > 0 order by FieldNumber"
  stRows_col = cursor_col.execute(statement_col)
  df_col = pandas.DataFrame((tuple(t) for t in stRows_col))
  
  '''
  End - Pull Column Names
  '''
  
  '''
  Start - Add Column Names to Table data (df_data)
  '''
  
  df_data.columns = df_col
  
  '''
  End - Add Column Names to Table data (df_data)
  '''
  
  '''
  Start - Create a sqlalchemy engine
  '''
  
  params = urllib.parse.quote_plus("DRIVER={SQL Server Native Client 11.0};"
                                   "SERVER=Server;"
                                   "DATABASE=DB;"
                                   "UID=UID;"
                                   "PWD=PWD;")
  
  engine = sqlalchemy.create_engine("mssql+pyodbc:///?odbc_connect={}".format(params))
  
  '''
  End - Create a sqlalchemy engine
  '''
  
  
  
  '''
  Start - Create sql table and rows from sqlalchemy engine
  '''
  
  df_data.to_sql(name=db_tbl_name, con=engine, if_exists='replace')
  
  '''
  End - Create sql table and rows from sqlalchemy engine
  '''
  
• Test a decorated function in Python

  I have a python function which is decorated.

      @retry_if_access_token_expired(app_id)
      def add_something(
          self,
          *,
          argument1,
          argument1 = None,
      ):
  
            """adding something"""
  

  I've written tests for the given as below.

      @patch("other inside function to mock")
      @patch("other insdie function to mock 2")
      def test_add_something(
          self, mock_1, mock_2
      ):
          """ some logic to test the add something method """
  

  But I am getting this error that says add_somehthing takes 1 positional argument.

  TypeError: add_something() takes 1 positional argument but 3 were given
• Pytest does not find my tests in Poetry project (VSCode finds)

  I've just created my first Python package using Poetry using the usual poetry new mypackage command. My problem is that pytest does not execute any test when I run it. I'm developing using VSCode and the weird behavior is that VSCode successfully finds and executes my tests.

  Poetry created a subdir called mypackage and another called tests. My test file is called tests/test_mypackage.py.

  VSCode autodiscoverd the tests, and display them in the test tab. The .vscode/settings.json file has this configuration:

  "python.testing.pytestArgs": [
          "tests"
      ],
  

  I've tried the following commands to execute pytest:

  1. With my venv manually activated:
     1. pytest
     2. pytest tests
     3. pytest tests/test_mypackage.py
     4. cd tests;pytest test_mypackage.py
  2. without my venv activated:
     1. poetry run pytest
     2. poetry run pytest tests

  The behavior is always the same: nothing happens, as if pytest couldn't detect anything to run.

  I've been using VSCode to run the tests, but now I want to put the code under Continuous Integration. How do I run pytest to validate my package?
• Query difference between db.session.query and Model.query in flask-SQLAlchemy

  The database is near 5 millions rows. I declare a model like below:

  class Amodel(db.Model):
       id = db.Column(db.Integer, primary_key=True)
       date = db.Column(db.String)
       money = db.Column(db.String)
  

  • I made a index of money column and it doesn't affect result.

  Way 1 - session.query：

  from flask import Flask
  from flask_sqlalchemy import SQLAlchemy
  
  app = Flask(__name__)
  db = SQLAlchemy(app)
  s1 = db.session.query(Amodel).filter(Amodel.money=='1000').all()
  
  

  Way 2 - Model.query：

  s2 = Amodel.query.filter(Amodel.money=='1000').all()
  
  

  Time consumption

  test1:
  s1:0.06102442741394043
  s2:0.6709990501403809
  
  test2:
  s1:0.0010263919830322266
  s2:0.6235842704772949
  
  test3:
  s1:0.0029985904693603516
  s2:0.5942485332489014
  

  They got the same result but time consumption is so different. I usually use way2 for query because I think it's more readble. Could someone explain what's happen inside and how to optimize?