Why my training model(K-Fold) return Nan Value?

See also questions close to this topic

• Tabulate Spherical Harmonic Functions

  Hey I am trying to express the spherical harmonics in a compact way. I can not figure out how to express this in a table I got the spherical Harmonics for l = 0,1,2 and m = -l, ...,+l in a list. But wanted a Table in the form:

   m -2 -1 0 1 2
  l        
  0        x
  1      x x x
  2    x x x x x
  

  Does anyone know how to do this since I only worked out to create simpler tables with tabulate using with normal headers. I have got the functions in the list sph_harmonics[].

  from sympy import Ynm
  from tabulate import tabulate
  
  sph_harmonics = []
  for l in range (0, 3):
              for m in range(-l, l+1):
                  print('l =', l)
                  print('m =',m)
                  print(sp.simplify(Ynm(l, m, theta, phi).expand(func=True)))
                  sph_harmonics.append(sp.simplify(Ynm(l, m, theta, phi).expand(func=True)))
  
• How to convert object string to int?

  i have a dataframe with object type on it, i need to make it type be an integer, so i can convert it to float. I already try to loop it and change with if else condition, but it makes another problem, what i must to do? thank's before.

  for x in df_merge3["type"]:
  if x == 'course':
      df_merge3[["type"][x]] = 1
  elif x == exercise:
      df_merge3[["type"][x]] = 2
  elif x == challange:
      df_merge3[["type"][x]] = 3
  elif x == achievement:
      df_merge3[["type"][x]] = 4
  print(x)
  
• adding an auth check to github webhooks using flask + python3

  I've been trying to add auth to github but for some reason my hashes don't match,

  here is my code:

  @app.route('/update', methods=["POST"])
  def update():
      assert request.method == "POST"
      signature = request.headers.get("X-Hub-Signature")
      if not signature or not signature.startswith("sha1="):
          abort(400, "X-Hub-Signature required")
  
      sha_name, sinature = signature.split("=")
      if sha_name != "sha1":
          abort(501)
  
      # Create local hash of payload
      digest = hmac.new(github_secret.encode(), msg=request.data, digestmod="sha1").hexdigest()
  
  
      # Verify signature
      if not hmac.compare_digest(signature, digest):
          abort(400, "Invalid signature")
  
  
• Q-learning for classification

  please I do not know how to write the program for using a q-learning algorithm for classification, all the examples I have seen are about using q-learning in the games, please any code for using a q-learning algorithm for classification?
• How to balance images in folder by doing augmentation such that number of images in this folder are equal to number of images in other folder?

  I have 5 folders named as class_i each folder has the i class images. the images are with .jpg format. How can I balance the images in each folder by doing augmentation such that number of images in this folder will be equal to the number of images in the folder with highest number of images? Also, could you please help in plotting a curve shows number of images in each folder before and after balancing?
• How to decide on the right target size of an input image for inception_v3? (predictions differ)

  I am running an Inception V3 model to make predictions. I am finding out that different target sizes for the input image lead to different predictions.

  For the same image, I got different predictions. For (416, 416): tiger_cat (6.228156387805939), for (299, 299): Egyptian_cat (55.72159290313721), for (224, 224): lynx (48.244667053222656).

  I thought that the input size should be divided by 32. Which is not the case for (299, 299). But, for another input size I get the error message:

  WARNING:tensorflow:Model was constructed with shape (None, 299, 299, 3) for input Tensor("input_13:0", shape=(None, 299, 299, 3), dtype=float32), but it was called on an input with incompatible shape (None, 224, 224, 3).
  

  How do I decide on the best target size?
• How to improve Confusion Matrix scores for 3 Class - Covid19 Detection chest xrays images of 3 classes (covid , normal, viral pneumonia)

  #I am using the below code and the report is being generated but the scores are not good (approx 0.31). #However, during training and testing the performance of model is above 90%.

  #Note: "eval_batches" is the dataset used during the training phase to check the val_accuracy.

  #(4/4 - 19s - loss: 0.2456 - accuracy: 0.9000 - val_loss: 0.2301 - val_accuracy: 0.9500)

  #Confution Matrix and Classification Report

  from sklearn.metrics import classification_report
  
  batch_size = 10
  num_of_train_samples = 2721
  num_of_eval_samples = 583
  
  
  Y_pred = model.predict_generator(eval_batches, num_of_eval_samples // batch_size+1)
  y_pred = np.argmax(Y_pred, axis=1)
  print('Confusion Matrix')
  print(confusion_matrix(eval_batches.classes, y_pred))
  print('Classification Report')
  target_names = ['COVID-19','NORMAL','Viral_Pneumonia']
  print(classification_report(eval_batches.classes, y_pred, target_names=target_names))
  

  #Classification ReportBelow is the Output:

  ""

  Confusion Matrix
  [[58 58 64]
   [66 56 79]
   [59 69 74]]
  Classification Report
  
                   precision    recall  f1-score   support
  
         COVID-19       0.32      0.32      0.32       180
           NORMAL       0.31      0.28      0.29       201
  Viral_Pneumonia       0.34      0.37      0.35       202
  
         accuracy                           0.32       583
        macro avg       0.32      0.32      0.32       583
     weighted avg       0.32      0.32      0.32       583
  

  "" Please help me to get better scores in the confusion matrix.

  Any help will be highly appreciated !!!
• Classification Problem - Every record is a unique class

  I have thousands of records where each is a unique class. I think what this means is that, which ever ML algorithm I choose, I will not be able to execute a train/test split because the test split cannot be modelled. This is because there is only one record for each class, therefore there is no way the model could have trained for anything that is in the test split.

  I can't justify introducing unseen records for prediction if my model cannot be validated.

  Each record is similar to a generic accounting record with about 20 features. I thought about using KNN, but KNN would not be able to do a train/test split neither. How do I go about mitigating this? Is KNN the wrong model to use here?
• Unable to calculate different metrics for multi class classification

  My confusion matrix for binary classification has the following structure:

                  (Predicted)
  
  C=   ( actual)    [TN FP
                   FN TP]
  

  In my case, the first row represents class 0 and second row as class 1. I have denoted class 0 as the negative class and class 1 as the positive class.
  How can I calculate the Mathews Correlation Coefficient (MCC) value for multi-class expressed as MCC = (TP .* TN - FP .* FN) ./ ... sqrt( (TP + FP) .* (TP + FN) .* (TN + FP) .* (TN + FN) );

  In order to calculate the MCC value, I need TP, FP etc for multi classes. I have tried to do so in the following way, but I am unsure if this is correct as well. Can somebody please help in calculating MCC value for multi class classification problem? Please correct me where wrong.

  for i=1:nClasses
     TN(i)=C(i,i);
      FP(i)=sum(C(i,:))-C(i,i);
      FN(i)=sum(C(:,i))-C(i,i);
      TP(i)=sum(C(:))-TP(i)-FP(i)-FN(i);
  end
  
• Powershell and yaml files

  We're trying to write a script in which we need to fill several yaml template files. The situation is as follows: We want to use the heartbeat monitoring for our applications, all for the uptime stuff. We found out that Heartbeat does not like it when there are more then 300 applications being monitored via 1 heartbeat service. That is why we have decided to split it up in several services, this unfortunately requires us to change our Powershell script. Our current script is as follows:

  $appservers = 
  $servers = $appservers 
  $templatelocation = 
  $ymllocation = 
  $Directory = 
  
  
  #Copy the template file to the correct folder
  try {
      Copy-Item -Path $templatelocation -Destination $ymllocation -Recurse -Force -ErrorAction Stop
      echo "Date: $((Get-Date).ToString()). Status: Template Successfully copied!" 
  }
  catch {
      $ErrorMessage = $Error[0].Exception.Message
      echo "Date: $((Get-Date).ToString()). Status: Copy Failure - $ErrorMessage"
  }
  
  ForEach ($server in $servers) {
      Invoke-Command -ComputerName $server -ArgumentList $server, $ymllocation, $templatelocation, $Directory -ScriptBlock {
          param($server, $ymllocation, $templatelocation, $Directory)
          Import-module powershell-yaml
          Write-Host "Connecting to server: $server in order to load the websites."
          $Sites = Get-Website | Where-Object { $_.Name -notlike '' }
          Write-Host "Loading websites of server: $server succesfull."
          if ($server -like '*') {
              ForEach ($site in $sites) {
                  $SiteName = $Site.Name
                  $Pattern = "heartbeat.monitors:"  
                  $Filebeat = $ymllocation  
                  $FileOriginal = Get-Content $Filebeat       
                  #Import the hearbeat file as RAW yml file
                  $yml = convertfrom-yaml $(Get-content $ymllocation -raw)    
                  #Prepare the file for the count
                  $monitorcount = $yml.'heartbeat.monitors'                                         
                  [String[]] $FileModified = @() 
                  Foreach ($Line in $FileOriginal) {    
                      $FileModified += $Line
                      if ($Line -match $pattern) {
                          $FileModified += ""
                          $FileModified += "- type: http"
                          $FileModified += "  id: $sitename"
                          $FileModified += "  name: $sitename"
                          $FileModified += "  urls:"
                          $FileModified += "  - https://$sitename"
                          $FileModified += ""
                          $FileModified += "  #Configure task schedule"
                          $FileModified += "  schedule: '@every 120s'"
                          $FileModified += "  # Total test connection and data exchange timeout"
                          $FileModified += "  timeout: 30s"
                          $fileModified += "  # Name of corresponding APM service, if Elastic APM is in use for the monitored service."
                          $FileModified += "  #service_name: my-apm-service-name"
                          $FileModified += ""
                      }                    
                  }
                  Set-Content $Filebeat $FileModified 
              }        Write-host "Added the websites of server: $server succesfully."
  
          }
      }
  }
  

  I've cleared out a couple of sections in order to protect the names of the environment. The above script currently generates one big file but what we would like to have is to separate it into several files consisting of 50 monitors each. The problem is now that we can get a count on the added monitors but we have no clue on how to go further then than and how to, from that point, create a new file and proceed filling the file from where it stopped (so from 51 and so on).

  Does anyone have an advice perhaps?

  Thank you for your time!

  Kind regards.
• Effeciently detect if heartbeat not recieved (and fire an event)

  I have devices which send heartbeats every 15 seconds. I am wondering what are good approaches to being able to detect every 15 seconds.

  One naive approach I can think of is, we know all our machine and their machineIds. We create a set and every time we receive a heartbeat (containing the machine Id) and we remove it from the set. After 30 seconds (nyquist) we check the set and any remaining machine Ids tell us what machines didn't send a heartbeat. Any better approaches than this?

  Thanks
• How to launch a process at boot time and restart it when it has crashed or quit unexpectedly on OS X and windows?

  What is the best way to launch a background application when an at boot time and manage its lifecycle in the sense that when it crashes or quits unexpectedly, it is possible to relaunch it. My application is a bit heavy .. so I was thinking of two options :

  1. My_Application itself as a daemon process ( I am not willing to opt this method as My_APllicationis a process with a large footprint)

  2. A manager daemon process which will in turn spawn My_Application process as a child process.

  This is my opinion. Can anyone suggest me any other efficient way or comment on above approach.

  Also, how to better implement the Heartbeat mechanism to periodically check if the background process is running or not and force launch if it is not running

  Thanks!
• k-fold accuracy does not return to 0

  I have a problem where the accuracy of my k-fold does not return to 0, but continues the accuracy of the last epoch on the previous fold, please find a solution

  Below is the code for the image generator

  IMAGE_SHAPE = (224, 224)
  TRAINING_DATA_DIR = str(directory_root)
    datagen_kwargs = dict(
    rescale = 1. / 255,
    rotation_range=25,
    shear_range=0.2,
    horizontal_flip=True,
    validation_split=.20)
  
  # Make Validation Datagen
  valid_datagen = tf.keras.preprocessing.image.ImageDataGenerator(**datagen_kwargs)
    valid_generator = valid_datagen.flow_from_directory(
    TRAINING_DATA_DIR,
    subset='validation',
    shuffle=True,
    target_size=IMAGE_SHAPE,
    batch_size=100)
  
  # Make Train Datagen
   train_datagen = tf.keras.preprocessing.image.ImageDataGenerator(**datagen_kwargs)
   train_generator = train_datagen.flow_from_directory(
     TRAINING_DATA_DIR,
     subset='training',
     shuffle=True,
     target_size=IMAGE_SHAPE,
     batch_size=100)
  

  Below is the code for k-fold

  for train, test in kfold.split(inputs, targets):
  # Define callbacks
  checkpoint_path = f'/content/drive/MyDrive/Colab Notebooks/saveModel/Model 1/{fold_no}'
  os.mkdir(checkpoint_path)
  keras_callbacks = [
  ModelCheckpoint(checkpoint_path, monitor='val_loss', save_best_only=True, mode='min')
  ]
  
  x_t, x_ts  = inputs[train] , targets[test]
  y_t, y_ts  = inputs[train] , targets[test]
  
  model_history = model.fit(
    train_generator, 
    epochs=EPOCHS,
    verbose=1,
    steps_per_epoch=steps_per_epoch,
    validation_data=valid_generator,
    validation_steps=val_steps_per_epoch,
    callbacks=keras_callbacks).history
  
• Google Colab crashing when Storing multiple images from a Folder

  When i'm trying to store a set of images from a dataset in a numpy array for processing them using stratified k folder for splitting dataset it ends up crashing before it finishes storing them. I'm currently using Google Collaboratory for this Project but i think is some short of overloading on the RAM of the virtual machine. Can someone tell me another way to store images in an array or try to fix this problem?

  This is the way that i'm storing images:

  jpgs = "/content/pathTofolder"
  images = []
  for nimg in os.listdir(jpgs):
    img = cv2.imread(os.path.join(jpgs,nimg))
    if img is not None:
      #La añadimos a la lista de imagenes
      images.append(img)
  print(images)
  

  I also tryed this way but same crash occurs:

  images = []
  for filename in os.listdir(jpgs):
    img = Image.open(jpgs+filename)
    data = np.asarray(img)
    images.append(data)
  print(images)
  

  Can someone help?
• I am trying to run stratifiedkfold with randomizedsearchcv but want to know whether the tried method is correct or not

  I am specifically doing this way since i need X_train, X_test, y_train, y_test else i could have easily fit with X,y. But i am not sure whether below mentioned code is the right way to do this.

  fold = 3
  skf = StratifiedKFold(n_splits=fold, shuffle=True, random_state=100) 
  
  dtc = DecisionTreeClassifier(random_state=100)
  
  params = {
      'max_depth': [2, 3, 5, 10, 20],
      'min_samples_leaf': [5, 10, 20, 50, 100],
      'criterion': ["gini", "entropy"]
  }
  
  for train_index, test_index in skf.split(X, y):
      X_train, X_test = X.iloc[train_index], X.iloc[test_index]
      y_train, y_test = y.iloc[train_index], y.iloc[test_index]
  
  rcv = RandomizedSearchCV(estimator=dtc, param_distributions=params, n_iter=10, cv=skf, n_jobs=-1, verbose=1, scoring='accuracy', random_state=100) #skf.split(X_train, y_train)
  rcv.fit(X_train, y_train)