Replace data in json file with a dataframe

See also questions close to this topic

• AzureML Machine Learning Tasks

  Please email me for an easy AzureML task. Regression and Classification is needed in graphics. I will provide you details and pay you right away after we agree on the budget. email: burakkdedeoglu gmail.com

  Thanks for your helps. I am looking forward to start.
• Methods to remove Time Limit exceeded error from my code?

  https://www.codechef.com/MARCH21C/problems/COLGLF4

  This is the question, for which I have written the solution.

  It shows Time Limit Exceeded(TLE) error.

  Pls suggest how do I optimise it further.

  I have optimised it to some extent. How do I Optimise it further. Pls Help!

  Here in my code: e-> egg, h-> choclate

  #define fastio() ios_base::sync_with_stdio(false);cin.tie(NULL);cout.tie(NULL)
  #include <bits/stdc++.h>
  using namespace std;
  
  // help returns the min cost for n items
  int help(int n, int e, int h, int& a, int& b, int& c, int *** dp){
      
      if(n==0) return 0;
      if(dp[h][e][n]!=-1) return dp[h][e][n];
      
      int p1=INT_MAX,p2=INT_MAX,p3=INT_MAX;
      
      if(e>=2){
          int ans = help(n-1,e-2,h,a,b,c,dp);
          // dp[h][e-2] = ans;
          if(ans != INT_MAX) p1=ans + a;
      }
      if(h>=3){
          int ans = help(n-1,e,h-3,a,b,c,dp);
          // dp[h-3][e] = ans;
          if(ans != INT_MAX) p2 = ans + b;
      }
      if(e>=1&&h>=1){
          int ans = help(n-1, e-1,h-1,a,b,c,dp);
          // dp[h-1][e-1] = ans;
          if(ans != INT_MAX) p3= ans + c;
      }
      
      int f = min(p1,min(p2,p3));
      dp[h][e][n] = f;
      return f;
  }
  
  int main() {
      // your code goes here
      fastio();
      int T;
      cin>>T;
      for(int t=0;t<T;t++){
          int n,e,h,a,b,c;
          cin>>n>>e>>h>>a>>b>>c;
         // scanf("%d",&n);
         // scanf("%d",&e);
         // scanf("%d",&h);
         // scanf("%d",&a);
         // scanf("%d",&b);
         // scanf("%d",&c);
          
          
          int*** dp = new int**[h+1];
          for(int i=0;i<=h;i++){
              dp[i] = new int*[e+1];  
              for(int j=0;j<=e;j++){
                  dp[i][j] = new int[n+1];
                  for(int k=1;k<=n;k++) dp[i][j][k]=-1;
              }
          }
          for(int i=0;i<=h;i++){
              for(int j=0;j<=e;j++){
                  dp[i][j][0]=0;
              }
          }
          
          
          int ans = help(n,e,h,a,b,c,dp);
          if(ans == INT_MAX) cout<<-1<<'\n';
          else cout<<ans<<'\n';
          
         // for(int i=0;i<=h;i++){
         //     for(int j=0;j<=e;j++){
         //         cout<<dp[i][j]<<" ";
         //     }cout<<endl;
         // }
          
          for(int i=0;i<=h;i++){
              for(int j=0;j<=e;j++){
                  delete [] dp[i][j];
              }
              delete dp[i];
          }
          delete [] dp;
          
      }
      return 0;
  }
  
• Python Custom Config Files for Modules

  I have below Directory Structure:
  |-- masterFile.py
  |-- firstFolder
  |-- |-- first.py
  |-- |-- configFile
  |-- SecondFolder
  |-- |-- second.py
  |-- |-- configFile
  

  I have many folders like (firstFolder, secondFolder...) and all their python files have same type of code only difference is on processing the variables from config file. I have put all the common code in masterFile.py file and the variable processing has been kept in first.py,m second.py files as a function.

  What I want to do is: I will run the masterFile.py file and depending on some input/signal I will call the fucntion from first.py, second.py .. file and I want them to be run with their own config file variables.

  I am using below code format:
  

  In first.py, second.py

  import os
  import configparser
  config = configparser.ConfigParser()
  config.read(os.path.join(os.path.dirname(__file__),'config'))
  var1 = config['xyz']['abc']
  var2 = config['jkl']['mno']
  
  def doSomeMagic(INPUT_VAR):
      someProcessing_on var1, var2 & INPUT_VAR
  

  Importing the modules in masterFile.py

  from firstFolder import first
  from secondFolder import second
  
  
  initialization_of INPUT_VAR
  
  if SIGNAl == 1:
     first.doSomeMagic(INPUT_VAR)
  

  Depending on some input/signal, I will call doSomeMagic function from first.py, second.py ... files. I want these function to use their config file but Config files are not accessible when I am following this format/structure. < br>

  Any help on how to Achieve this and any advice to change these formats/stucture to make more Pythonic way.

  I have some folder with name starting with a digit like: 7Yes

  from 7Yes import 7yes
  

  How can I import the python file from this directory?
• How to upgrade Python to latest version in Windows 10?

  I'm currently running Python version 3.8.5 on windows 10 and the latest version of python is 3.9.2. How can I upgrade Python to the latest version 3.9.2 . Is it possible through Command Prompt?
• 'psycopg2.extensions.connection' object has no attribute 'get_dsn_parameters'

  I've just upgraded postgresql-client from 9.1 to 9.5 on Ubuntu 16.04. (I plan to upgrade everything further, but I want to get this working first.) I have psycopg2 2.6.1 installed, but when I try to execute

  print(conn.get_dsn_parameters())
  

  I get the error message

  'psycopg2.extensions.connection' object has no attribute 'get_dsn_parameters'
  

  even though the psycopg2 docs say get_dsn_parameters requires libpq >= 9.3. From this I infer psycopg2 has linked against libpq 9.1; would this be correct? pip refuses to uninstall psycopg2, so I can't do uninstall-reinstall. How do I get psycopg2 working with the correct version of libpq?

  Thanks.
• How to compare files in two zip file are totally same or not?

  I have two zip files. I want to see if everything (file name and each file's contents in the zip) are the same.

  There is a similar question. But the answer does not support zip files.

  Anyone has a good idea?
• Expand Pandas Dataframe Column with JSON

  I'm looking for a clean, fast way to expand a pandas dataframe column which contains a json object (essentially a dict of nested dicts), so I could have one column for each element in the json column in json normalized form; however, this needs to retain all of the original dataframe columns as well. In some instances, this dict might have a common identifier I could use to merge with the original dataframe, but not always. For example:

  import pandas as pd
  import numpy as np
  df = pd.DataFrame([
      {
          'col1': 'a',
          'col2': {'col2.1': 'a1', 'col2.2': {'col2.2.1': 'a2.1', 'col2.2.2': 'a2.2'}}
      },
      {
          'col1': 'b',
          'col2': {'col2.1': 'b1', 'col2.2': np.nan}
      },
      {
          'col1': 'c',
          'col2': {'col2.1': 'c1', 'col2.2': {'col2.2.1': 'c2.1', 'col2.2.2': 'c2.2'}}
      }
  ])
  

  Here is a sample dataframe. As you can see, col2 is a dict in all of these cases which has another nested dict inside of it, or could be a null value, containing nested elements I would like to be able to access. In this case, they have no ID that could link up to the original dataframe. My end goal would be essentially to have this:

  final = pd.DataFrame([
      {
          'col1': 'a',
          'col2.1': 'a1',
          'col2.2.col2.2.1': 'a2.1',
          'col2.2.col2.2.2': 'a2.2'
      },
      {
          'col1': 'b',
          'col2.1': 'b1',
          'col2.2.col2.2.1': np.nan,
          'col2.2.col2.2.2': np.nan
      },
      {
          'col1': 'c',
          'col2.1': 'c1',
          'col2.2.col2.2.1': 'c2.1',
          'col2.2.col2.2.2': 'c2.2'
      }
  ])
  

  In my instance, the dict could have up to 50 nested key-value pairs, and I might only need to access a few of them. So I suppose there might be two methods I'd be looking for--getting a column for each value in the dict, or getting a column for a select few. The former option I haven't yet found a great workaround for; I've looked at some prior answers but found them to be rather confusing, and most threw errors. This seems especially difficult when there are dicts nested inside of the column. To attempt the second solution, I tried the following code:

  def get_val_from_dict(row, col, label):
      if pd.isnull(row[col]):
          return np.nan
      
      norm = pd.json_normalize(row[col])
      
      try:
          return norm[label]
      except:
          return np.nan
  
  
  needed_cols = ['col2.1', 'col2.2.col2.2.1', 'col2.2.col2.2.2']
  
  
  for label in needed_cols:
      df[label] = df.apply(get_val_from_dict, args = ('col2', label), axis = 1)
  

  This seemed to work for this example, but for my actual dataframe which had substantially more data, this was quite slow--and, I would imagine, is not a great or scalable solution. Would anyone be able to offer an alternative to this sluggish approach to resolving the issue I'm having?

  (Also, apologies also about the massive amounts of nesting in my naming here. If helpful, I am adding in several images of the dataframes below--the original, then the target, and then the current output.)

  
• Label groups of columns in Pandas

  I am producing a data frame with many columns, and I would like to have groups of columns.

  I believe this and this questions ares related, but I'm not sure how to apply that to my case.

  In summary, I want to have a super label for sets of columns.

  My data is getting generated similarly to:

  def inspect_pages(cases, checks=[check_one, check_two]):
      columns = list(chain.from_iterable(
          [[(check.name, column) for column in check.reportable] for check in checks]
      ))
      column_index = MultiIndex.from_tuples(columns, names=["Check Name", "Check"])
      print(column_index)
      results = DataFrame(
          index=[case.label for case in cases],
          columns=column_index
      )
      for n, case in enumerate(cases):
          for (name, result) in map(lambda c: (c.name, c(**case._asdict())), checks):
              for (key, value) in result.items():
                  results.loc[case.label, key] = value
      return results
  
  

  So what I would like is to have a merged column that spans the same number of columns I'm getting per test labeled with the content test.name, and have regular columns with the key and value values per case.

  Any idea or pointer to the right direction?
• logarithm of the odds using np.where()

  Define a function logit to compute the log-odds (i.e., the logarithm of the odds) associated with a given probability p Provide your function definition in the cell below associated with the identifier logit. Recall the logit function can be defined mathematically as logit(p)= −∞ : p=0, log(p1−p): 0<p<1, +∞ : p=1

  I want the function to return -np.inf or np.inf when p is 0 or 1 respectively.And return np.nan if p<0 or p>1 . Be sure to use the natural logarithm (i.e., base e ); this is implemented in Numpy with numpy.log (usually aliased as np.log). Be sure that the function is vectorized, i.e., that it can accept a Numpy array, a Pandas Series, or a Pandas DataFrame as input and still return the corresponding result with logit applied element-wise. HINT: the function np.where is very useful for this.

  import numpy as np
  def logit(p):
      '''Computes logit function, i.e., log-odds from probability
      INPUT:
      p - probability (value expected to be stictly between 0 & 1)
      '''
  
  p = np.array([0, 0.1, 0.2, 0.8, 0.9, 1, 5])
  print('logit(p) = \n{}'.format(logit(p))) # Notice symmetry in values
  

  If anyone could help me complete the code.