See also questions close to this topic

• Hierarquical clustering with big vector

  I'm working in MATLAB and I have a vector of 4 million values. I've tried to use the linkage function but I get this error:

  Error using linkage (line 240) Requested 1x12072863584470 (40017.6GB) array exceeds maximum array size preference. Creation of arrays greater than this limit may take a long time and cause MATLAB to become unresponsive. See array size limit or preference panel for more information.

  I've found that some people avoid this error using the kmeans functions however I would like to know if there is a way to avoid this error and still use the linkage function.
• I have two tall arrays and i want to find the intersection of them matlab

  In common tables the intersection function would have worked great, but for tall tables this function has not been implemented yet. So is there an other way to find the common elements of these two???

  I use MatLab2017a.
• Unable to run Java code on MATLAB Runtime in Unix environment

  After exporting a matlab function to java package. I have tried to run the following code unix environment which contains MATLAB runtime.

  import com.mathworks.toolbox.javabuilder.MWArray;
  import com.mathworks.toolbox.javabuilder.MWCharArray;
  import calibrationAPINew.Calibration;
  
  public class CHIndexCalculation {
          public static void main(String[] args) {
  
                MWCharArray number1 = null;
                Object[] result = null;
                Calibration calibration = null;
                String data = "{\r\n" +
                          "\"tsId\" : \"121213\",\r\n" +
                          "\"datapoints\" : [0.1,0.2,0.3,0.4]\r\n" +
                          "}";
  
                try
                {
                   System.out.println("Input data: " + data);
                   number1 = new MWCharArray(data); // Array of Input to be sent to MATLAB Runtime
                   System.out.println("MWCharArray: " + number1);
                   calibration = new Calibration();
                   result = calibration.calibrationAPI(1,number1);
  
                   System.out.println("Output : " +  result[0]);
                }
                catch (Exception e)
                {
                  System.out.println("Exception Occurred : " + e);
                   e.printStackTrace();
                }
                finally
                {
                   //MWArray.disposeArray(number1);
                   //MWArray.disposeArray(result);
                   //calibration.dispose();
                }
  
          }
  }
  

  I have installed the matlabruntime on unix box and set the environment variables using export statement

  export LD_LIBRARY_PATH=matlabrt/v92/runtime/glnxa64:matlabrt/v92/bin/glnxa64:matlabrt/v92/sys/os/glnxa64:matlabrt/v92/sys/opengl/lib/glnxa64
  

  Now when I try to run the command ( calibrationAPINew.jar is the jar file exported from MATLAB)

  java -cp javabuilder.jar:calibrationAPINew.jar:. CHIndexCalculation
  

  I get the following exception

  com.mathworks.toolbox.javabuilder.MWException: Undefined function 'pdist2mex' for input arguments of type 'double'.
          at com.mathworks.toolbox.javabuilder.internal.MWMCR.mclFeval(Native Method)
          at com.mathworks.toolbox.javabuilder.internal.MWMCR.access$600(MWMCR.java:31)
          at com.mathworks.toolbox.javabuilder.internal.MWMCR$6.mclFeval(MWMCR.java:882)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:498)
          at com.mathworks.toolbox.javabuilder.internal.MWMCR$5.invoke(MWMCR.java:769)
          at com.sun.proxy.$Proxy0.mclFeval(Unknown Source)
          at com.mathworks.toolbox.javabuilder.internal.MWMCR.invoke(MWMCR.java:443)
          at calibrationAPINew.Calibration.calibrationAPI(Calibration.java:223)
          at CHIndexCalculation.main(CHIndexCalculation.java:24)
  

  I know this has something to do with MATLAB runtime but I am unable to figure out what ? The same set up works fine on windows environment.
• What is the best neural network configuration to solve the XOR dataset on Tensorflow Playground?

  

  Hi All,

  I have toyed with 2 network configurations on Tensorflow playground

  1. 1 hidden layer with 2 nodes
  2. 2 hidden layers with 2 nodes each

  I find that the second option (2 hidden layers) gives better classification accuracy. However, I find that the network does not converge to a solution all the time.

  Thanks, Sau
• How do epochs work in this tensorflow example

  The following is a snippet of code used in training a simple neural network.

  for epoch in range(hm_epochs):
              epoch_loss = 0
              for _ in range(int(mnist.train.num_examples/batch_size)):
                  epoch_x, epoch_y = mnist.train.next_batch(batch_size)
                  _, c = sess.run([optimizer, cost], feed_dict={x: epoch_x, y: epoch_y})
                  epoch_loss += c
  
              print('Epoch', epoch, 'completed out of',hm_epochs,'loss:',epoch_loss)
  

  This isn't the full code, but from what I see, the inner loop trains using all the test data (split into batches) and optimizes using an optimizing algorithm. With the inner loop running one the accuracy of the algorithm is 90% but when it runs 10 times (hm_epochs=10) it's correct 95% of the time. That doesn't make any sense to me, how does training it with the same data multiple times (which is what happens when the outer loop runs), make it any more accurate.

  I am new to tensorflow.

  This is not my code, it comes from this series: https://pythonprogramming.net/tensorflow-neural-network-session-machine-learning-tutorial/
• Why Does My Neural Network Always Predicts The Same Output? (synaptic.js)

  Hello this question is related to Synaptic.js.

  I'm trying to use Neural Network to read handwritten digits from images. Image size is 120x90 and it only contains black and white pixels. I'm using html5 canvas to transform images to Matrix before feeding it into a NN with 10800 inputs, 30 hidden layers and 3 outputs.

  var myNetwork = new synaptic.Architect.Perceptron(10800,30,3);
  
  var learningRate = .3; 
  

  The code to transform Image to Matrix :

  var matrix = [];                
  for(i = 0; i < img.height; i++) {
          for(j = 0; j < img.width; j++) {
                  var rgba = ctx.getImageData(j, i, 1, 1);
                  var avg = ((rgba["data"][0] + rgba["data"][1] + rgba["data"][2]) / 3);
  
                  var indice;
  
                  if(avg < 150) indice = 1;
                  else indice = 0;
  
                  matrix.push(indice);
              }
          }
  

  The code above gives a result I expected, basically it transforms 'white' pixels into 0 and 'black' pixels into 1. Attached is a matrix example for an image containing '1' (I resized the actual Matrix because it's too big) : matrix illustration

  When I feed the NN two images, each containing '1' or '2' handwritten digits it categorize them just fine. Same thing when I try to feed it five more images containing '1' or '2' handwritten digits, the NN is still able to categorize them well.

  The code I use to train NN :

  myNetwork.propagate(learningRate, [1, 0]); //Train NN to learn '1'
  myNetwork.propagate(learningRate, [0, 1]); //Train NN to learn '2'
  

  However, when I start to feed '3' into the network, it starts to categorize every image inputs (including '1' and '2') into the third category. Basically when the output size is more than 2, the NN will always predict the same output which is the last category I train it to learn.

  myNetwork.propagate(learningRate, [0, 0, 1]); //Train NN to learn '3'
  

  I have more than 75 images per number and I have tried to feed them all but the NN always categorize them into '3'. What do I do wrong here? any help is appreciated, Thanks!
• Can change network architecture during training in tensorflow?

  I want to change channel dimensions of convolution layer during training, but in testing, always keeping same dimensions. I tried to implement this in tensorflow, and i failed. If the output dimension changes during training, an error occurs because the tensor flow does not recognize the changed graph. Does not the tensor flow implement these features? If not, is there any other way to implement this feature?
• How to get train accuracy in SSD for caffe version?

  I am testing SSD caffe version for object detection. This caffe version doesn't log into the file for accuracy in training. Just log detection evaluation in training. So I am trying to put similar layers in train.prototxt file as follow. But it produces a lot in each iterations, how can I have only one for each iteration.

  add in layers to train.prototxt file

  layer {
    name: "mbox_conf_reshape"
    type: "Reshape"
    bottom: "mbox_conf"
    top: "mbox_conf_reshape"
    reshape_param {
      shape {
        dim: 0
        dim: -1
        dim: 2
      }
    }
    propagate_down: false
  }
  layer {
    name: "mbox_conf_softmax"
    type: "Softmax"
    bottom: "mbox_conf_reshape"
    top: "mbox_conf_softmax"
    softmax_param {
      axis: 2
    }
    propagate_down: false
  }
  layer {
    name: "mbox_conf_flatten"
    type: "Flatten"
    bottom: "mbox_conf_softmax"
    top: "mbox_conf_flatten"
    flatten_param {
      axis: 1
    }
    propagate_down: false
  }
  layer {
    name: "detection_out"
    type: "DetectionOutput"
    bottom: "mbox_loc"
    bottom: "mbox_conf_flatten"
    bottom: "mbox_priorbox"
    top: "detection_out"
    include {
      phase: TRAIN
    }
    propagate_down: false
    propagate_down: false
    propagate_down: false
    detection_output_param {
      num_classes: 2
      share_location: true
      background_label_id: 1
      nms_param {
        nms_threshold: 0.449999988079
        top_k: 400
      }    
      code_type: CENTER_SIZE
      keep_top_k: 200
      confidence_threshold: 0.00999999977648
    }
  }
  layer {
    name: "detection_eval"
    type: "DetectionEvaluate"
    bottom: "detection_out"
    bottom: "label"
    top: "detection_eval"
    include {
      phase: TRAIN
    }
    propagate_down: false
    propagate_down: false
    detection_evaluate_param {
      num_classes: 2
      background_label_id: 1
      overlap_threshold: 0.3
      evaluate_difficult_gt: false
      name_size_file: "/home/coie/data/NumberPlate/test_name_size.txt"
      resize_param {
        prob: 1.0
        resize_mode: WARP
        height: 480
        width: 480
        interp_mode: LINEAR
        height_scale: 480
        width_scale: 480
      }
    }
  }
  
• Validation accuracy in training CNN

  I am confused between test accuracy and validation accuracy in CNN training. What I knew before is validation set is totally not used in training. During training, only testing set is used and calculate accuracy.

  Validation set is used only after training finished, test the model using validation set.

  This lecture has the following plot.

  

  They said validation accuracy, is that test accuracy using test set in training?
• Hovering over Mat from opencv 3.4.1 in VS 15.7.5 still freezes

  Is anybody else still getting freezes in VS 15.7.5 while hovering over the Mat keyword in openCV 3.4.1? (note: I have included openCV 3.4.1 and the contrib modules into VS using cmake.)

  Reference link to the same question but for an older version of Visual Studios: Link for old question. Apparently 15.7.4 was supposed to be a fix? Upvoted answer saying that the fix happened.

  Andrew's solution works for 15.7.5 too. Link for temp solution. Please take the time to give him credit.

  Screenshot for Andrew's solution
• OpenCV Read Mat from Memory returns null

  I create a MAT from a given hWnd

  cv::Mat hwnd2mat(HWND hwnd) {
  
  HDC hwindowDC, hwindowCompatibleDC;
  
  int height, width, srcheight, srcwidth;
  HBITMAP hbwindow;
  cv::Mat src;
  BITMAPINFOHEADER  bi;
  
  hwindowDC = GetDC(hwnd);
  hwindowCompatibleDC = CreateCompatibleDC(hwindowDC);
  SetStretchBltMode(hwindowCompatibleDC, COLORONCOLOR);
  
  RECT windowsize;    // get the height and width of the screen
  GetClientRect(hwnd, &windowsize);
  
  srcheight = windowsize.bottom;
  srcwidth = windowsize.right;
  height = windowsize.bottom;  
  width = windowsize.right;
  
  src.create(height, width, CV_8UC4);
  
  // create a bitmap
  hbwindow = CreateCompatibleBitmap(hwindowDC, width, height);
  bi.biSize = sizeof(BITMAPINFOHEADER);    //http://msdn.microsoft.com/en-us/library/windows/window/dd183402%28v=vs.85%29.aspx
  bi.biWidth = width;
  bi.biHeight = -height;  //this is the line that makes it draw upside down or not
  bi.biPlanes = 1;
  bi.biBitCount = 32;
  bi.biCompression = BI_RGB;
  bi.biSizeImage = 0;
  bi.biXPelsPerMeter = 0;
  bi.biYPelsPerMeter = 0;
  bi.biClrUsed = 0;
  bi.biClrImportant = 0;
  
  // use the previously created device context with the bitmap
  SelectObject(hwindowCompatibleDC, hbwindow);
  // copy from the window device context to the bitmap device context
  StretchBlt(hwindowCompatibleDC, 0, 0, width, height, hwindowDC, 0, 0, srcwidth, srcheight, SRCCOPY); //change SRCCOPY to NOTSRCCOPY for wacky colors !
  GetDIBits(hwindowCompatibleDC, hbwindow, 0, height, src.data, (BITMAPINFO *)&bi, DIB_RGB_COLORS);  //copy from hwindowCompatibleDC to hbwindow
  
                                                                                                     // avoid memory leak
  DeleteObject(hbwindow); DeleteDC(hwindowCompatibleDC); ReleaseDC(hwnd, hwindowDC);
  
  return src;
  }
  

  And I try to load the images to run the template matching algorithm. However, when I try to imgdecode I get img = NULL. If I save the file to a png and read it afterwards, it's working.

  I even tried to use the different flags to read the image.

  cv::Mat resMat = hwnd2mat(blueStackshWnd);
  cv::imwrite("foo.png", resMat);
  img = cv::imdecode(resMat, cv::IMREAD_GRAYSCALE);
  //img = cv::imread("fuck.png", cv::IMREAD_COLOR);
  templ = cv::imread("images\\giant.png", cv::IMREAD_GRAYSCALE);
  

  Here is a screenshot while debugging

  EDIT:

  When assigning the ret value of hwnd2mat to img I get the following error

  error: (-215) (depth == 0 || depth == 5) && type == _templ.type() && _img.dims() <= 2 in function cv... ...} ...}   const cv::Exception &
  
• OpenCV Java Making Mat object bigger inside function

  I want to make my own function with in-out argument.

  static void maxDistance(Mat input, Mat output)
  

  input is a grayscale image and output is in-out argument. I want to call this function like this:

  Mat distance = new Mat();
  maxDistance(gray, distance); 
  

  Default constructor creates distance object without any rows and columns and I need to expand this object inside my function. I want it to have the exact same size as input object and I know that in c++ API there is function called resize(). There is no resize function in Java though and I don't know how to make distance bigger.
• Reshape cannot infer the missing input size for an empty tensor unless all specified input sizes are non-zero

  I am having some issues with my code. I want to build a multi-class (4 Classes) Image classifier using Keras with Tensorflow as a backend, I have written some code, but I keep getting the following error : "Reshape cannot infer the missing input size for an empty tensor unless all specified input sizes are non-zero" at the line below: model.fit_generator(train_generator, samples_per_epoch=samples_per_epoch, epochs=epochs, validation_data=validation_generator, callbacks=cbks, validation_steps=validation_steps)

  below is my code base

  from __future__ import print_function
  from keras.engine.input_layer import InputLayer
  import numpy as np
  
  import sys
  import os
  import argparse  
  
  import keras
  from keras import optimizers
  from keras import callbacks
  from keras import backend as K
  
  from keras.models import Model
  from keras.models import Sequential
  from keras.layers import Input
  from keras.layers import Lambda
  from keras.layers import Layer
  from keras.layers import Dense
  from keras.layers import Dropout
  from keras.layers import Activation
  from keras.layers import Flatten
  from keras.layers import Conv2D
  from keras.layers import MaxPooling2D
  from keras.optimizers import SGD
  from keras.optimizers import RMSprop
  from keras.preprocessing.image import ImageDataGenerator
  
  epochs = 20
  train_data_path = os.path.join(os.getcwd(),"Scans","Train")
  validation_data_path = train_data_path
  
  ##### Parameters #####
  img_width, img_height = 150, 150
  batch_size = 32
  samples_per_epoch = 1000
  validation_steps = 300
  nb_filters1 = 8 #16 #8 #4
  nb_filters2 = 16 #32 #16 #8
  conv1_size = 8
  conv2_size = 8
  pool_size = 2
  classes_num = 4
  lr = 0.0004
  strides = (2,2)
  
  ################# Main ##################
  model = Sequential()
  inputshape = (img_width,img_height,3)
  model.add(InputLayer(input_shape=inputshape))
  
  model.add(Conv2D(filters=nb_filters1,   kernel_size=conv1_size,strides=1,padding="same"))
  model.add(Activation("relu"))
  
  model.add(MaxPooling2D(pool_size=(pool_size,pool_size), strides=(2,2),padding="same"))
  
  
  model.add(Conv2D(filters=nb_filters2, kernel_size=conv2_size,padding="same"))
  model.add(Activation("relu"))
  
  model.add(MaxPooling2D(pool_size=(pool_size,pool_size), strides=(2,2),padding="same"))
  
  model.add(Dropout(0.25))
  model.add(Flatten())
  
  model.add(Dense(512))
  
  model.add(Activation("relu"))
  
  model.add(Dropout(0.5))
  
  model.add(Dense(classes_num))
  model.add(Activation("softmax"))
  
  model.compile(loss='categorical_crossentropy', optimizer=optimizers.RMSprop(lr=lr), metrics=['accuracy'])
  
  
  
  train_datagen = ImageDataGenerator(rescale=1. /255, shear_range=0.2, zoom_range=0.2, horizontal_flip=True)
  test_datagen = ImageDataGenerator(rescale=1. /255)
  
  train_generator = train_datagen.flow_from_directory(train_data_path, target_size=(img_width,img_height), batch_size=batch_size, class_mode='categorical')
  
  validation_generator = test_datagen.flow_from_directory(validation_data_path, target_size=(img_width,img_height), batch_size=batch_size, class_mode='categorical')
  
  ######### TensorBoard Log ###############
  log_dir = os.path.join(os.getcwd(), "tensorflow_log")
  tb_cb = callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)
  cbks = [tb_cb]
  
  model.fit_generator(train_generator, samples_per_epoch=samples_per_epoch, epochs=epochs, validation_data=validation_generator, callbacks=cbks, validation_steps=validation_steps)
  
  model.summary()
  
  
  target_dir = os.path.join(os.getcwd(),"models")
  if not os.path.exists(target_dir):
       os.mkdir(target_dir)
  
  model.save(os.path.join(target_dir,"model.h5"))
  model.save_weights(os.path.join(target_dir, "weights.h5"))
  
• How to use Gabor filters for feature extraction from image?

  I want to apply Gabor filter for feature extraction from image then on the trained data I will be applying NN or SVM.I didn't applied batch processing though but it will be done or if you can help me with the machine learning part it will be great for me.Thank you. Here is my code:

  import cv2
  import numpy as np
  import glob
  
  img=glob.glob("C://Users//USER//Pictures//Saved Pictures//tuhin.jpg")
  
  img = cv2.threshold(img, 127, 255, cv2.THRESH_BINARY)[1]  
  ret, labels = cv2.connectedComponents(img)
  label_hue = np.uint8(179*labels/np.max(labels))
  blank_ch = 255*np.ones_like(label_hue)
  labeled_img = cv2.merge([label_hue, blank_ch, blank_ch])
  labeled_img = cv2.cvtColor(labeled_img, cv2.COLOR_HSV2BGR)
  labeled_img[label_hue==0] = 0
  cv2.imshow('labeled.png', labeled_img)
  cv2.waitKey()
  
  def build_filters():
  filters = []
  ksize = 31
  for theta in np.arange(0, np.pi, np.pi / 16):
      kern = cv2.getGaborKernel((ksize, ksize), 4.0, theta, 10.0, 0.5, 0, 
                ktype=cv2.CV_32F)
      kern /= 1.5*kern.sum()
      filters.append(kern)
      return filters
  
  def process(img, filters):
      accum = np.zeros_like(img)
      for kern in filters:
          fimg = cv2.filter2D(img, cv2.CV_8UC3, kern)
          np.maximum(accum, fimg, accum)
          return accum
  
  filters=build_filters()
  res1=process(img,filters)
  cv2.imshow('result',res1)
  cv2.waitKey(0)
  cv2.destroyAllWindows()