See also questions close to this topic

• Building a model on Keras correctly

  I'm new to neural networks and Keras, and I want to build a CNN that predicts certain values of an image. (the three values predict the size, length, and width of a blur put on top of the image). All 3 values can range from 0 to 1, and I have a large data set.

  I am not exactly sure how to build the CNN to do this though, as all the prototype codes that I have built so far give me predictions of the format [1.,0.,0.] instead of ranges between 0 and 1 for each value. On top of that, despite changing the number of epochs and decay value in the SGD optimizer, I don't get any change in my loss function at all. Can you please tell me where I am going wrong? Here is what I have so far:

  images, labels = load_dataset("images")   # function that loads images
  images = np.asarray(images) # images are flattened 424*424 arrays (grayscale)
  labels = np.asarray(labels) # Lables are 3-arrays, each value is float from 0-1
  
  # I won't write this part but here I split into train_imgs and test_imgs
  
  model = keras.Sequential()
  # explicitly define SGD so that I can change the decay rate
  sgd = keras.optimizers.SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True)
  
  model.add(keras.layers.Dense(32, input_shape=(424*424,) ))
  model.add(keras.layers.Activation('relu'))
  model.add(keras.layers.Dense(3, activation='softmax'))
  
  model.compile(loss='mean_squared_error',optimizer=sgd)
  # note: I also tried defining a weighted binary crossentropy but it changed nothing
  
  checkpoint_name = 'Weights-{epoch:03d}--{val_loss:.5f}.hdf5' 
  checkpoint = ModelCheckpoint(checkpoint_name, monitor='val_loss', verbose = 0, save_best_only = True, mode ='auto')
  callbacks_list = [checkpoint]
  
  model.fit(train_imgs, train_labls, epochs=20, batch_size=32, validation_split = 0.2, callbacks=callbacks_list)
  
  predictions = model.predict(test_imgs) # make predictions on same test set!
  

  Now I know that I am leaving out dropout layers, but I WANT the CNN to overfit my data, at this point I just want it to do anything. When I predict on the same set of images, I would hopefully get exact results, no? I'm not quite sure what I'm missing. Thanks for the help!
• Switching from Keras to tf.keras spams my screen with #010

  I've built a simple Keras model for experimentation within Amazon SageMaker. I'm using Python 3.5 TensorFlow 1.12.0. Recently I switched my model to use TensorFlow.keras, but doing so has resulted in the printing of #010 repeatedly followed by #015, while downloading image net weights and displaying batch accuracy during the fit call.

  For example, with verbose=1 in model.fit:

  Epoch 1/1

  015 1/1563 [..............................] - ETA: 5:50:36 - loss: 2.2798 - acc: 0.1875#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015

  3/1563 [..............................] - ETA: 1:57:18 - loss: 2.3002 - acc: 0.1146#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 5/1563 [..............................] - ETA: 1:10:36 - loss: 2.3088 - acc: 0.1062#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010

  Does anyone know why this might be happening or how I can prevent this? Reproducing with a minimal example may require running through SageMaker, but the code I switched from Keras to tf.keras is from this example, and specifically the trainer/start.py file:

  # Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.
  #
  # Licensed under the Apache License, Version 2.0 (the "License"). You
  # may not use this file except in compliance with the License. A copy of
  # the License is located at
  #
  #     http://aws.amazon.com/apache2.0/
  #
  # or in the "license" file accompanying this file. This file is
  # distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF
  # ANY KIND, either express or implied. See the License for the specific
  # language governing permissions and limitations under the License.
  from __future__ import absolute_import
  from __future__ import print_function
  
  import keras
  from keras.preprocessing.image import ImageDataGenerator
  from keras.models import Sequential
  from keras.layers import Dense, Dropout, Activation, Flatten
  from keras.layers import Conv2D, MaxPooling2D
  import os
  import numpy as np
  
  from trainer.environment import create_trainer_environment
  
  NUM_CLASSES = 10
  EPOCHS = 10
  NUM_PREDICTIONS = 20
  MODEL_NAME = 'keras_cifar10_trained_model.h5'
  
  # the trainer environment contains useful information about
  env = create_trainer_environment()
  print('creating SageMaker trainer environment:\n%s' % str(env))
  
  # getting the hyperparameters
  batch_size = env.hyperparameters.get('batch_size', object_type=int)
  data_augmentation = env.hyperparameters.get('data_augmentation', default=True, object_type=bool)
  learning_rate = env.hyperparameters.get('learning_rate', default=.0001, object_type=float)
  width_shift_range = env.hyperparameters.get('width_shift_range', object_type=float)
  height_shift_range = env.hyperparameters.get('height_shift_range', object_type=float)
  EPOCHS = env.hyperparameters.get('epochs', default=10, object_type=int)
  
  # reading data from train and test channels
  train_data = np.load(os.path.join(env.channel_dirs['train'], 'cifar-10-npz-compressed.npz'))
  (x_train, y_train) = train_data['x'], train_data['y']
  
  test_data = np.load(os.path.join(env.channel_dirs['test'], 'cifar-10-npz-compressed.npz'))
  (x_test, y_test) = test_data['x'], test_data['y']
  
  
  model = Sequential()
  model.add(Conv2D(32, (3, 3), padding='same', input_shape=x_train.shape[1:]))
  model.add(Activation('relu'))
  model.add(Conv2D(32, (3, 3)))
  model.add(Activation('relu'))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  model.add(Dropout(0.25))
  
  model.add(Conv2D(64, (3, 3), padding='same'))
  model.add(Activation('relu'))
  model.add(Conv2D(64, (3, 3)))
  model.add(Activation('relu'))
  model.add(MaxPooling2D(pool_size=(2, 2)))
  model.add(Dropout(0.25))
  
  model.add(Flatten())
  model.add(Dense(512))
  model.add(Activation('relu'))
  model.add(Dropout(0.5))
  model.add(Dense(NUM_CLASSES))
  model.add(Activation('softmax'))
  
  # initiate RMSprop optimizer
  opt = keras.optimizers.rmsprop(lr=learning_rate, decay=1e-6)
  
  # Let's train the model using RMSprop
  model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])
  
  x_train = x_train.astype('float32')
  x_test = x_test.astype('float32')
  x_train /= 255
  x_test /= 255
  
  if not data_augmentation:
      print('Not using data augmentation.')
      model.fit(x_train, y_train, batch_size=batch_size, epochs=EPOCHS, validation_data=(x_test, y_test), shuffle=True)
  else:
      print('Using real-time data augmentation.')
      # This will do preprocessing and real time data augmentation:
      data_generator = ImageDataGenerator(
          featurewise_center=False,  # set input mean to 0 over the dataset
          samplewise_center=False,  # set each sample mean to 0
          featurewise_std_normalization=False,  # divide inputs by std of the dataset
          samplewise_std_normalization=False,  # divide each input by its std
          zca_whitening=False,  # apply ZCA whitening
          rotation_range=0,  # randomly rotate images in the range (degrees, 0 to 180)
          width_shift_range=width_shift_range,  # randomly shift images horizontally (fraction of total width)
          height_shift_range=height_shift_range,  # randomly shift images vertically (fraction of total height)
          horizontal_flip=True,  # randomly flip images
          vertical_flip=False)  # randomly flip images
  
      # Compute quantities required for feature-wise normalization
      # (std, mean, and principal components if ZCA whitening is applied).
      data_generator.fit(x_train)
  
      # Fit the model on the batches generated by data_generator.flow().
      data_generator_flow = data_generator.flow(x_train, y_train, batch_size=batch_size)
      model.fit_generator(data_generator_flow, epochs=EPOCHS, validation_data=(x_test, y_test), workers=4)
  
  # Save model and weights
  model_path = os.path.join(env.model_dir, MODEL_NAME)
  model.save(model_path)
  print('Saved trained model at %s ' % model_path)
  
  # Score trained model.
  scores = model.evaluate(x_test, y_test, verbose=1)
  print('Test loss:', scores[0])
  print('Test accuracy:', scores[1])
  
• Hardware Requirements for Pandas Dataframe, Tensorflow and LightGBM

  I ask for help with information on the minimum hardware requirements, such as HD space, memory, processor, video card, for the use of Pandas Dataframe, Tensorflow and LightGBM libraries.

  Thank you!
• Generate .net model using CNN Machine learning

  Can anyone please help me to generate(train) a .net model of CNN using images data, and make it usable for prediction, as we generate .h5 model. I am struggling to find resource that can help me, or if anyone have useful reference or tutorial that have guidance about this, please let me know.
• No overfitting by increasing the number of epochs

  I use a feed foreward neural network with one hidden layer for my thesis. Threby I have 600 training data and 104 input and output values. There I want to show now the properties of the neural network and also want to show the overfitting when I increase the number of epochs. To do so, I first wanted to find the optimum for learning rate and number of hidden nodes, where I got the following results:

  

  Based on that I decided to choose a learning rate of 0.0125 and 250 hidden nodes. But by using this set of parameters, I still have no overfitting when I increase the number of epochs, which can be seen here:

  

  In this plot I showed in blue my old set of parameters and in theory I wanted to show how it improve when I use the best set of parameters, but it's just varying a bit. I also tested it until epoch 1000 but the accuracy with this value was still 0.830.

  Does someone has an idea why this happen?

  Thanks a lot for your help!
• Serving multiple deep learning models from cluster

  I was thinking about how one should deploy multiple models for use. I am currently dealing with tensorflow. I was referring this and this article.

  But I am not able to find any article which targets need to serve several models distributed manner. Q.1. Does tensorflow serving serve models off from single machine? Is there any way to set up a cluster of machines running tensorflow serving? So that multiple machines serve same model somewhat working as master and slave or say load balance between them while serving different models.

  Q.2. Does similar functionality exist for other deep learning frameworks, say keras, mxnet etc (not just restricting to tensorflow and serving models from different frameworks)?
• Automatic Differentiation with CoDiPack

  The following code:

  #include <codi.hpp>
  ...
  codi::RealForward Gcodi[l];
  for (int p = 0; p < l; p++) 
  {
       ...
       double a = Gcodi[p];
  }
  

  gives me the compilation error:

  nnBFAD.cpp: In function ‘void OptBF()’:

  nnBFAD.cpp:156:25: error: cannot convert ‘codi::RealForward {aka codi::ActiveReal >}’ to ‘double’ in initialization double

  a = Gcodi[p];
  

  Hints?
• How is the derivative calculated for tf.losses.cosine_distance?

  I am using tf.losses.cosine_distance as an added penalty to my cost function, tf.nn.nce_loss.

  If I understand correctly, the cosine distance measure is not differentiable for zero length vectors. And when the gradient does not exist, we must resort to the sub-gradient, so my question is: how is that implemented in libraries such as TensorFlow?

  Also, when is it advisable to use a function such as the L2 distance measure in its place, which is both convex and (everywhere) differentiable?
• How does tensorflow handle non differentiable nodes during gradient calculation?

  I understood the concept of automatic differentiation, but couldn't find any explanation how tensorflow calculates the error gradient for non differentiable functions as for example tf.where in my loss function or tf.cond in my graph. It works just fine, but I would like to understand how tensorflow backpropagates the error through such nodes, since there is no formula to calculate the gradient from them.
• Tensorflow tf.placeholder with shape = []

  I am looking at a Tensorflow code that has learning rate input to the graph using placeholder with shape = [], as below:

  self.lr_placeholder = tf.placeholder(dtype=tf.float32, shape=[])
  

  I looked at the official documentation page of Tensorflow (https://www.tensorflow.org/api_docs/python/tf/placeholder) to understand what would shape=[] mean, but could not get an explanation for the shape set to empty list. If someone can explain what does this mean.
• How can I replace a variable with another one in Tensorflow's computation graph?

  Problem: I have two pretrained models with variables W1,b1 and W2,b2 saved as numpy arrays.

  I want to set a mixture of these two pretrained models as the variables of my model, and only update the mixture weights alpha1 and alpha2 during training.

  In order to do that I create two variables alpha1 and alpha2 and load the numpy arrays and create the mixture nodes: W_new, b_new.

  I want to replace W and b in the computation graph with W_new and b_new and then only train alpha1 and alpha2 parameter by opt.minimize(loss, var_list= [alpha1, alpha2]).

  I don't know how to replace W_new and b_new in the computation graph. I tried assigning tf.trainable_variables()[0] = W_new but this doesn't work.

  I'd appreciate if anyone could give me some clues.

  Note 1: I don't want to assign values to W and b (this will disconnect the graph from alpha1 and alpha2), I want the mixture of parameters to be a part of the graph.

  Note 2: You might say that you could compute y using the new variables, but problem is, the code here is just a toy sample to simplify things. In reality instead of linear regression I have several bilstms with crf. So I can't manually compute the formula. I'll have to replace these variables in the graph.

  import tensorflow as tf
  import numpy as np
  np.random.seed(7)
  tf.set_random_seed(7)
  
  #define a linear regression model with 10 params and 1 bias
  with tf.variable_scope('main'):
      X = tf.placeholder(name='input', dtype=float)
      y_gold = tf.placeholder(name='output', dtype=float)
      W = tf.get_variable('W', shape=(10, 1))
      b = tf.get_variable('b', shape=(1,))
      y = tf.matmul(X, W) + b
      #loss = tf.losses.mean_squared_error(y_gold, y)
  
  
  #numpy matrices saved from two different trained models with the exact same architecture
  W1 = np.random.rand(10, 1)
  W2 = np.random.rand(10, 1)
  b1 = np.random.rand(1)
  b2 = np.random.rand(1)
  
  with tf.variable_scope('mixture'):
      alpha1 = tf.get_variable('alpha1', shape=(1,))
      alpha2 = tf.get_variable('alpha2', shape=(1,))
  
      W_new = alpha1 * W1 + alpha2 * W2
      b_new = alpha1 * b1 + alpha2 * b2
  
  all_trainable_vars = tf.trainable_variables()
  print(all_trainable_vars)
  
  
  #replace the original W and b with the new mixture variables in the computation graph (**doesn't do what I want**)
  all_trainable_vars[0] = W_new
  all_trainable_vars[1] = b_new
  #this doesn't work
  
  #note that I could just do the computation for y using the new variables as y = tf.matmul(X, W_new) + b_new
  #but the problem is this is just a toy example. In real world, my model has a big architecture with several
  #bilstms whose variables I want to replace with these new ones.
  
  #Now what I need is to replace W and b trainable parameters (items 0 and 1 in all_trainable vars)
  #with W_new and b_new in the computation graph.
  
  with tf.Session() as sess:
      sess.run(tf.global_variables_initializer())
      train_writer = tf.summary.FileWriter('./' + 'graph',
                                           sess.graph)
      #print(sess.run([W, b]))
      #give the model 3 samples and predict on them
      print(sess.run(y, feed_dict={X:np.random.rand(3, 10)}))
  

  Why do I want to do this?

  Assume you have several pretrained models (in different domains) but you don't have access to any of their data.

  Then you have a little bit of training data from another domain that doesn't give you that much performance, but if you could train the model jointly with the data that you don't have you could get a good performance.

  Assuming the data is somehow represented in the trained models, we want to learn a mixture of the pretrained models, by learning the mixing coefficients, using little labelled data that we have as supervision.

  We don't want to pretrain any parameters, we only want to learn a mix of pretrained models. What are the mixture weights? we need to learn that from the little supervision that we have.