See also questions close to this topic

• Improving performance of XML to dataframe parsing

  I have a long xml with the following structure:

  <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
  <SoccerFeed timestamp="20180212T2228+0000">
   <Game id="3323" home_team_id="83" home_team_name="CFC" away_team_id="335" away_team_name="WCFC" game_date="22180212T2000+0000" competition_id="18" competition_name="DCF" season_id="2011" season_name="Season 2017/2018" matchday="27" period_1_start="22180212T2000+0000" period_2_start="22180212T2105+0000" Status="FullTime">
      <Event id="1182774006" event_id="10223" type_id="37" period_id="14" min="0" sec="0" player_id="1212" team_id="83" outcome="1" x="45.7" y="59.0" timestamp="2018-02-12 21:52:45" last_modified="2018-02-12 21:52:48" version="15184723628801">
     <Q id="16517621865" qualifier_id="302"/>
    </Event>
    <Event id="120438223891" event_id="8726" type_id="30" period_id="14" min="0" sec="0" player_id="" team_id="335" outcome="0" x="" y="" timestamp="2018-02-12 21:52:46" last_modified="2018-02-12 21:52:46" version="15128472366490">
     <Q id="125850245236" qualifier_id="57" value="1.0"/>
     <Q id="1775059222" qualifier_id="209"/>
    </Event>
   </Game>
  </SoccerFeed>
  

  The file normally has 1500+ Event nodes.

  My current code parses the file in 20-21 seconds but I feel like there are some obvious performance improvements that I am missing. Any tips would be appreciated.

  I am pretty sure there are better ways to access all of the attributes and their values without having to define each one individually.

  Current Code

  import xml.etree.ElementTree as ET
  import pandas as pd
  
  # define the location of your file
  file_name = "possessions.xml"
  
  # parse the xml and conver to a tree and root
  tree = ET.parse(file_name)
  root = tree.getroot()
  
  ## get the main game info from the single 'Game' node
  gameinfo = root.findall('Game')
  gameinfo = gameinfo[0]
  game_id = gameinfo.get('id')
  home_team_id = gameinfo.get('home_team_id')
  home_team_name = gameinfo.get('home_team_name')
  away_team_id = gameinfo.get('away_team_id')
  away_team_name = gameinfo.get('away_team_name')
  competition_id = gameinfo.get('competition_id')
  competition_name = gameinfo.get('competition_name')
  season_id = gameinfo.get('season_id')
  
  ## create an empty dataframe to store the per event information
  catcher = pd.DataFrame()
  
  # loop through each event node and grab the information
  for i in root.iter('Event'):
  
      # get the info from the Event node main chunk
      id = i.get('id')
      event_id = i.get('event_id')
      type_id = i.get('type_id')
      period_id = i.get('period_id')
      min = i.get('min')
      sec = i.get('sec')
      player_id = i.get('player_id')
      team_id = i.get('team_id')
      x = i.get('x')
      y = i.get('y')
      possession_id = i.get('possession_id')
      sequence_id = i.get('sequence_id')
  
      # combine the event main node data into a dataframe
      Edata = pd.DataFrame(
          {'id': id,
           'event_id': event_id,
           'type_id': type_id,
           'period_id': period_id,
           'min': min,
           'sec': sec,
           'player_id': player_id,
           'team_id': team_id,
           'x': x,
           'y': y,
           'sequence_id': sequence_id,
           'possession_id':possession_id
          }, index = [0])
  
      # find all of the Q information for that file
      Qs = i.findall("./Q")
  
      # create some empty lists to append the results to
      qualifier_id = []
      Q_value = []
  
      # loop through all of the Qs and grab the info
      for child in Qs:
          qualifier_id.append(child.get('qualifier_id'))
          Q_value.append(child.get('value'))
  
      # create a dataframe from the two lists
      Qdata = pd.DataFrame(
      {'qualifier_id': qualifier_id,
       'value': Q_value
      })
  
      # transform the dataframe, 1st row to the headers,
      # drop the extra row, reset the index
      Qdata = Qdata.T
      Qdata.columns = Qdata.iloc[0]
      Qdata = Qdata.reindex(Qdata.index.drop('qualifier_id'))
      Qdata = Qdata.reset_index(drop=True)
  
      # combine the event info and qualifer info
      result = pd.concat([Qdata, Edata], axis=1)
  
      # bind to the catcher dataframe
      catcher = catcher.append(result)
  
  # Add all of the the game information to each row
  catcher['competition_id'] = competition_id
  catcher['game_id'] = game_id
  catcher['home_team_id'] = home_team_id
  catcher['home_team_name'] = home_team_name
  catcher['away_team_id'] = away_team_id
  catcher['away_team_name'] = away_team_name
  catcher['competition_id'] = competition_id
  catcher['competition_name'] = competition_name
  catcher['season_id'] = season_id
  catcher['competition_id'] = competition_id
  
• OpenCV. Find contour in an image

  How can I find contours in this image?

  

  After threshold I get this

  

  and I get two contours in the whole image

  

  This is my code

      image = cv2.imread(captcha_image_file)
      gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
      thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV | cv2.THRESH_OTSU)[1]
      contours = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
  
• How to add new cookies for xmlrpc.client in Python 3?

  I want to add a cookie name test=xxxxxxxxxxxxxx; to the xmlrpc-wordpress request.

  Is there any way to do that? I found How to retain cookies for xmlrpc.client in Python 3? but this is for retain cookies and I need a code to add new cookie.
• ValueError: No way to determine width or height from video. Need `-s` in `inputdict`. Consult documentation on I/O

  the following below is my code,while i run the code , got an error like "ValueError: No way to determine width or height from video. Need -s in inputdict. Consult documentation on I/O."

  vid_path = '/home/dataset/MSVD/YouTubeClips/'
  vid = skvideo.io.vread(my_dir)
  # Select 16 frames from video
  vid = vid[40:56]
  x = preprocess_input(vid) 
  
• Preparing images to feed into tensorflow as datasets

  AskTensorFlow

  I have used the tensorflow packaged datasets like MNIST, IMDB to study the working of tensorflow. However, in practical applications we have to preprocess and prepare the dataset on our own. Suppose I am working with image dataset, so I want to preprocess them to a format that can be fed into a tensorflow model. How can I preprocess image dataset to a tensorflow format?
• How can one set VLOG to specific level?

  Getting the same error as Tensorflow keep printing something related to FusedBatchNorm. One of the answers on that post says "With the fix, the output should only get printed out if the VLOG level is set to 2 or higher."

  I can't figure out how to set the VLOG level to a specific value.

  How does one do that?
• Operands could not broadcast together

  I'm trying to train a model using mini batches but I'm having a .... error.

  I'm using the same function I've already used (and it worked) with other models, but this time is crashing.

  def random_mini_batches(X, Y, mini_batch_size = 64):
  """
  Creates a list of random minibatches from (X, Y)
  
  Arguments:
  X -- input data, of shape (input size, number of examples)
  Y -- true "label" vector (1, number of examples)
  mini_batch_size - size of the mini-batches, integer
  
  Returns:
  mini_batches -- list of synchronous (mini_batch_X, mini_batch_Y)
  """
  
  m = X.shape[1]                  # number of training examples
  mini_batches = []
  
  # Step 1: Shuffle (X, Y)
  permutation = list(np.random.permutation(m))
  shuffled_X = X.iloc[:, permutation]
  shuffled_Y = Y[:, permutation].reshape((Y.shape[0],m))
  
  # Step 2: Partition (shuffled_X, shuffled_Y). Minus the end case.
  num_complete_minibatches = math.floor(m/mini_batch_size) # number of mini batches of size mini_batch_size in your partitionning
  for k in range(0, num_complete_minibatches):
      mini_batch_X = shuffled_X.iloc[:, k * mini_batch_size : k * mini_batch_size + mini_batch_size]
      mini_batch_Y = shuffled_Y[:, k * mini_batch_size : k * mini_batch_size + mini_batch_size]
      mini_batch = (mini_batch_X, mini_batch_Y)
      mini_batches.append(mini_batch)
  
  # Handling the end case (last mini-batch < mini_batch_size)
  if m % mini_batch_size != 0:
      mini_batch_X = shuffled_X.iloc[:, num_complete_minibatches * mini_batch_size : m]
      mini_batch_Y = shuffled_Y[:, num_complete_minibatches * mini_batch_size : m]
      mini_batch = (mini_batch_X, mini_batch_Y)
      mini_batches.append(mini_batch)
  
  return mini_batches
  

  I've used this function in a NN with 20 layers and this X and Y: And now I'm trying use it again with a 5 layer NN and shapes

  However, I'm getting this error at this part of the code epoch_cost += minibatch_cost/num_minibatches.

  The full code would be this:

          for epoch in range(num_epochs):
  
          epoch_cost = 0
          num_minibatches = int(m / minibatch_size) # number of minibatches of size minibatch_size in the train set
  
          minibatches = random_mini_batches(X_train, Y_train, minibatch_size)
  
          for minibatch in minibatches:
  
              #Select a minibatch
              (minibatch_X, minibatch_Y) = minibatch
  
              _, minibatch_cost = sess.run([optimizer, cost], feed_dict = {X: minibatch_X, Y: minibatch_Y})
  
              epoch_cost += minibatch_cost/num_minibatches
  
      # Print the cost every epoch
          if print_cost == True and epoch % 100 == 0:
              print("Cost after epoch %i: %f" % (epoch, epoch_cost))
          if print_cost == True and epoch % 5 == 0:
              costs.append(epoch_cost)
  

  Thanks in advance
• How to use deep learning to get my scene?

  there is some train data like :

  [30, 10,20,...] lable [scene 1]

  [31, 11,30,...] lable [scene 1]

  ...

  [25, 4,40,...] lable [scene 2]

  [26, 5,50,...] lable [scene 2]

  ...

  [30, 4,180,...] lable [scene 3]

  [28, 4,170,...] lable [scene 3]

  ...

  Then i give another new test data i hope it can give the answer which scene is belonged to.

  i should choose which algorithm of deep learning ? OR Is there a good example to show me?