How to configure Keras model so training array can be larger than prediction arrays?

See also questions close to this topic

• Spark, from training/testing loop to model, working to score in production

  Suppose we trained/tested the model, found it good and we saved the trained and good model on a file system.

  All that, using spark and spark ML lib. (python)

  1) How do we start to use this model in production to process actual requests and predict? Can we use the same spark cluster(I mean load the model in another spark app and process online requests with this model)?

  2) Should we in parallel run the training/testing on the more modern data and once in a while "refresh" the model we use in production? Is that acceptable solution?

  3) I'm afraid online/production performance of python might be low, so is there way to speedup the execution in production? I mean trained model, could it be transferred to C or in other way "speed-improved"?

  Thanks
• How can scroll a side scroll bar of web-page with selenium in python

  I've tried to scroll a side bar (not full page) of a web-page but couldn't proceed.

  element = driver.find_elements_by_css_selector(".sidebar-content > div:nth-child(3) > div:nth-child(1)")[-1]
  actions = ActionChains(driver)
  actions.move_to_element(element).perform()
  
• Update dataframe value conditionaly with its own value

  I have DF with float street numbers, sometime it is "NaN" or "x-y" (ex: 30-32) but often x.y (ex: 32.0 instead of 30) I need to change this to int (if there is no "-" in the number of course). Ive tried

  chunk.loc["-" not in chunk["Street Number"] & chunk["Street Number"].notna(), 'Street Number'] = chunk["Street Number"].astype(int)
  

  I know there is an issue after my "=" sign. How to update dataframe value conditionaly with its own value please ? Ive also tried with

  chunk[["Street Number"]].astype(int)
  

  There is no error

  Sample of DF :

  0 | NaN

  1 | 1.0

  2 | 6.0

  3 | 170.0

  4 | 61.0

  5 | 51-52

  I tried to force dtype "Street Number": np.uint16 but I got ValueError: Integer column has NA values in column 12
• Is Liblinear SVM dependent on feature-order?

  I am using the java-transpilation of Liblinear (in my own code, not from the CL) for a text-classification task using binary bag of words vectors. I just recently stumbled upon something that I didn't expect. 2 models trained on the same data, evaluated on the same data yield different evaluation results and the only thing different between the 2 experiments I could find was the ordering of the used vocabularies (while describing the exact same set of words), so we get a different ordering of coordinates in the text-vectorizations.

  Theoretically SVMs (and LR) shouldn't depend on feature order or am I mistaken; aren't we just renaming axises when we change the feature order in this case?

  Can someone explain this?
• How to extract tabular content from image into python dataframe?

  I am trying to extract tabular content from images into python dataframes, I tried tabula and camelot packages but no use.
• Memory issue with data splitting CNN

  im facing memory error, trying to run a code for data splitting into train validation and test sets, I have like 50000 images to use in CNN, with less images it works, but I would like to keep the 50000 images, is there any way of doing that? I have reed that with batches, but I dont know how to implement that, im kind of new, thank you so much

  N_CLASSES = 2
  
  # Generates training, validation and testing files
  def gen_data():
      X_train = []
      Y_train = []            
      X_valid = []
      Y_valid = []
      X_test = []
      Y_test = []
      count=0
      for i in species:
  
          # Samples Location
          train_samples = join(datapath, 'traind/'+i)
          valid_samples = join(datapath, 'valid/' +i)
          test_samples = join(datapath, 'test/'+i)
  
          # Samples Files
          train_files = listdir(train_samples)
          valid_files = listdir(valid_samples)
          test_files = listdir(test_samples)
  
          # Sorting according to the number
          train_files.sort(key=lambda f: int(''.join(filter(str.isdigit, f))))
          valid_files.sort(key=lambda f: int(''.join(filter(str.isdigit, f))))
          test_files.sort(key=lambda f: int(''.join(filter(str.isdigit, f))))
  
          for j in train_files:
  
              im = join(train_samples, j)
              img = cv2.imread(im,1)
              img = cv2.resize(img, (416,416))
              X_train.append(img)
              Y_train+=[count]
  
  
          for k in test_files:
              im = join(test_samples, k)
              img = cv2.imread(im,1)
              img = cv2.resize(img, (416, 416))
              X_test.append(img)
              Y_test+=[count]
  
          for l in valid_files:
              im = join(valid_samples, l)
              img = cv2.imread(im,1)
              img = cv2.resize(img, (416, 416))
              X_valid.append(img)
              Y_valid+=[count]            
  
          count+=1
  
      X_train = np.asarray(X_train).astype('float32')
      X_train/= 255
      Y_train = np.asarray(Y_train)
  
      X_valid = np.asarray(X_valid).astype('float32')
      X_valid/= 255
      Y_valid = np.asarray(Y_valid)
  
      X_test = np.asarray(X_test).astype('float32')
      X_test /= 255
      Y_test = np.asarray(Y_test)
      return X_train, Y_train, X_valid, Y_valid, X_test, Y_test
  
  
  if __name__ == '__main__':
  
  #   makedirs(final_1)
  #   makedirs(final_2)    
  
  #   for i in species:
  
  #       makedirs(join(final_1, i))
  #       makedirs(join(final_2, i))        
  
  #   create_validation()
  #   create_test() 
  
      x_train, y_train, x_valid, y_valid, x_test, y_test = gen_data()
  
      y_train = np_utils.to_categorical(y_train, N_CLASSES)
      y_valid = np_utils.to_categorical(y_train, N_CLASSES)
      y_test = np_utils.to_categorical(y_test, N_CLASSES)
  
      np.save('X_train.npy', x_train)
      np.save('Y_train.npy', y_train)
      np.save('X_valid.npy', x_valid)
      np.save('Y_valid.npy', y_valid)
      np.save('X_test.npy', x_test)
      np.save('Y_test.npy', y_test)
  
• TensorFlow Probability: How to get accuracy of a prediction?

  I'm coding a TensorFlow model in this way:

  model = Sequential()
  model.add(Dense(512, input_shape=(max_words,)))
  model.add(Activation('relu'))
  model.add(Dropout(0.5))
  model.add(Dense(num_classes))
  model.add(Activation('softmax'))
  
  model.compile(loss='categorical_crossentropy',
                optimizer='adam',
                metrics=['accuracy'])
  
  history = model.fit(x_train, y_train,
                      batch_size=batch_size,
                      epochs=epochs,
                      verbose=1,
                      validation_split=0.1)
  

  I have this method that returns the prediction class of input:

  def predict(input):
      try:
          q = model.predict(np.array([x_train[int(dict[input])],]))
          predicted_label = text_labels[np.argmax(q)]
          print("Classe Predetta: " + predicted_label + "\n")
  
          return predicted_label
  
      except:
          return "error"
  

  Now I use softmax to obtain level of accuracy but this isn't the better way. I know that I can use probabilistic model with the new TensorFlow Probability. I'd like to see code to know to get model and accuracy Thanks
• How to pass epoch and batch size when using label powerset in keras

  I have a multi-label problem and with some research, I was able to use Label powerset in conjunction with ML algorithms. Now I want to use the Label powerset with neural network and as per the official website I can use Label powerset. But I am not able to understand how to modify my existing code to be able to use Label Powerset.

  I want to know how can we pass epoch or batch_size or any other parameter passed in the fit function of the model.

  Since I have a multi-label problem I have used MultiLabelBinarizer of sklearn so my each target row looks like this [1,0,0,1,0,0,0,0,0,0,0,0].

  and lastly, if someone could explain to me what is KERAS_PARAMS and Keras() in the below line:

  def create_model_multiclass(input_dim, output_dim):
      # create model
      model = Sequential()
      model.add(Dense(8, input_dim=input_dim, activation='relu'))
      model.add(Dense(output_dim, activation='softmax'))
      # Compile model
      model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
      return model
  clf = LabelPowerset(classifier=Keras(create_model_multiclass, True, KERAS_PARAMS), require_dense=[True,True])
  clf.fit(X_train,y_train)
  y_pred = clf.predict(X_test)
  

  Below is my existing neural network code

  cnn_model = Sequential()
  cnn_model.add(Dropout(0.5))
  cnn_model.add(Conv1D(25,7,activation='relu'))
  cnn_model.add(MaxPool1D(2))
  cnn_model.add(Dropout(0.2))
  cnn_model.add(Conv1D(25,7,activation='relu'))
  cnn_model.add(MaxPool1D(2))
  cnn_model.add(Flatten())
  cnn_model.add(Dense(25,activation='relu'))
  cnn_model.add(Dense(12,activation='softmax'))
  cnn_model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['acc'])
  history = cnn_model.fit(X_train, y_train, validation_data=(X_test,y_test), batch_size=32, epochs=180,verbose=1)
  plot_history(history)
  predictions = cnn_model.predict(X_test)
  

  I want my output row to look like this only [1,0,0,1,0,0,0,0,0,0,0,0] as later I will use my MultiLabelBinarizer for the inverse transform of this.