keras neural network training and validation loss with regular spikes
I trained a simple neural network in keras. The training loss and validation loss shows regular spikes after certain epochs. The xaxis of the figure is epochs, and the yaxis is loss. Notice the large validation loss spike correspond with small training loss spike. I used fit_generator. I tried optimizer SGD and Adam, and the spikes still appear.
Interestingly, changing batch size doesn't influence the shape or period of the spike. Changing optimizer influences spike magnitude, but doesn't influence the repeating period.
Any thoughts on why this happens?
See also questions close to this topic

Tensorflow performance: Numpy matrix o TF matrix?
I have the following code:
with tf.Session() as sess: sess.run(init_vars) cols = sess.run(tf.shape(descriptors)[1]) descriptor_matrix = np.zeros((n_batches*batch_size, cols)) while True: batch_descriptor = sess.run(descriptors, feed_dict={dropout_prob: 1}) descriptor_matrix[i:i+elements_in_batch] = np.array(batch_descriptor)
I am mixing tensors and numpy vectors. Does this have an important impact on performance? Why is it? Should I just use tensors instead?

Return dataframe rows where the values in a column are not of type date
I have a dataframe
df
that looks like:Name Date of birth Bob Steve 22/07/1963 Jo pencil Karen 03/02/1953 Frank 29/09/1994
Is there a way to return rows where
Date of birth
is not a date?In the above example I would have returned:
Name Date of birth Bob Jo pencil
Where
Date of birth
is not a date.I can identify where there is a blank value for Date of birth using:
missingDoBError = df.loc[df['Date of birth'].isnull()]
I have tried to find Date of birth values where the value is not a date format at set to NaT by using:
if pd.to_datetime(df['Date of birth'], format='%d%b%Y', errors='coerce').notnull().all():
But I can't get this to work.

How to implement tSNE in a model?
I split my data to train/test. When i use PCA It is straight forward.
from sklearn.decomposition import PCA pca = PCA() X_train_pca = pca.fit_transform(X_train) X_test_pca = pca.transform(X_test)
From here i can use X_train_pca and X_test_pca in the next step and so on..
But when i use tSNE
from sklearn.manifold import TSNE X_train_tsne = TSNE(n_components=2, random_state=0).fit_transform(X_train)
I can't seem to transform the test set so that i can use the tSNE data for the next step e.g. SVM.
Any help?

Merge multiple CNNs
I am trying to perform
Conv1D
on multiple inputs in my model. So I have 15 inputs of size 1x1500 each, where each one is an input to a series of layers. So I have 15 convolutional models which I want to merge before Fully Connected Layer. I have defined the convolutional model in a function, but I cannot understand how to call the function and then merge them.def defineModel(nkernels, nstrides, dropout, input_shape): model = Sequential() model.add(Conv1D(nkernels, nstrides, activation='relu', input_shape=input_shape)) model.add(Conv1D(nkernels*2, nstrides, activation='relu')) model.add(BatchNormalization()) model.add(MaxPooling1D(nstrides)) model.add(Dropout(dropout)) return model models = {} for i in range(15): models[i] = defineModel(64,2,0.75,(64,1))
I have successfully concatenated 4 models as follows:
merged = Concatenate()([ model1.output, model2.output, model3.output, model4.output]) merged = Dense(512, activation='relu')(merged) merged = Dropout(0.75)(merged) merged = Dense(1024, activation='relu')(merged) merged = Dropout(0.75)(merged) merged = Dense(40, activation='softmax')(merged) model = Model(inputs=[model1.input, model2.input, model3.input, model4.input], outputs=merged)
How do I do it for 15 layers in the for loop as writing 15 layers separately isn't efficient?

Neural network input being a list
I'm trying to train a neural network by supervised training model using the R language, but I'm having problems with the data frame because RN does not accept a list as input. Code used to create the dataframe:
df < data.frame("dados"= numeric(69),"saida"= numeric(69))
I insert each line of the data frame manually with the following code:
df$dados[1] < list(resultante1) df$saida[1] < 1
Here what you have in the resultante1:resultante1
After the long work, I have the following data frame: data frame
Note: the list has a high number of information, some inputs reach 50 thousand numbers, these values represent a signal and need to train the neural network with several signals so that it identifies similar signals. So I used 1 to say when is the signal I want and 0 for a similar signal but it is a false positive.
I used the RSNNS library to make use of the mpl function, but when I enter the input data, the following error occurs:
Function mlp:
mlp(df$dados, df$saida, size=nNeuronios, maxit=maxEpocas, initFunc="Randomize_Weights", initFuncParams=c(0.3, 0.3), learnFunc="Std_Backpropagation", learnFuncParams=c(0.1), updateFunc="Topological_Order", updateFuncParams=c(0), hiddenActFunc="Act_Logistic", shufflePatterns=F, linOut=TRUE)
Error:
Error in checkInput(x, y) : 'x' has to be numeric, after a conversion to matrix
Does any way the mlp function accept my data frame, or is there any other way to create and train a network to accept?
Link dataframe.Rdata, the file has 9 MB.

Keras weight file load exception: loading 2 layers into a model with 0 layers
Exception happened when I add dropout to the input layer.
The exception was mentioned in other threads as well related to another issues and most common suggested solution is to downgrade the Keras version. Is there a workaround for this exception?
def baseline_model() : model = Sequential() model.add(Dropout(0.35)) #THIS LINE CAUSES THE EXCEPTION model.add(Dense(200, input_dim=1200, kernel_initializer='normal', activation='relu')) model.add(Dropout(0.8)) rms = RMSprop(lr = 0.00050) model.add(Dense(1, kernel_initializer='normal', activation='sigmoid')) model.compile(loss='binary_crossentropy', optimizer=rms, metrics=['accuracy']) return model
Model throws the following exception during weight file loading:
ValueError: You are trying to load a weight file containing 2 layers into a model with 0 layers.

What is the difference OpenCV and Keras + Tensorflow?
What is the difference OpenCV and Keras + Tensorflow? Is it possible to use both library for face recognition with python 3?

How to merge layers in Keras?
I implemented a model in Keras year ago. But it doesn't work.
It builds two LSTM models.
Using the results(the value of final nodes) of two LSTM models, dense model predict binart value.
At this time, we concatenate the result of two LSTM models.
I searched many answers on this topic, but I can't solve this problem..
Below is my code.
Thank you!
print("Build model..1") model1 = Sequential() model1.add(LSTM(5, input_shape=(max_sequence, input_dim), return_sequences=True)) model1.add(BatchNormalization()) model1.add(Activation('sigmoid')) model1.add(Flatten()) print("Build model..2") model2 = Sequential() model2.add(LSTM(3, input_shape=(max_sequence, input_dim), return_sequences=True)) model2.add(BatchNormalization()) model2.add(Activation('sigmoid')) model2.add(Flatten()) model = Sequential() model.add(concatenate([model1, model2])) model.add(Dense(3)) model.add(BatchNormalization()) model.add(Activation('sigmoid')) model.add(Dense(1)) model.add(Activation('tanh')) optimizer = RMSprop(lr=0.5) model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy']) history = model.fit([train_x_1, train_x_2], train_y_class, nb_epoch=300, batch_size=128, validation_data=([val_x_1, val_x_2], val_y_class), callbacks=[callback_log])

Are there better / similar services like google colab?
From a long time I have been doing my experiments on google colab and opted not to buy a gpu instead. But are there better or similar services like Google Colab that is used for deep learning?

Plot validation loss in Tensorflow Object Detection API
I'm using Tensorflow Object Detection API for detection and localization of one class object in images. For these purposes I use pretrained faster_rcnn_resnet50_coco_2018_01_28 model.
I want to detect under/overfitting after training of the model. I see training loss, but after evaluating Tensorboard only shows mAP and Precision metrics and no loss.
Is this possible to plot a validation loss on Tensorboard too?

why not solve hyper parameter λ of L2 regularization in DNN simultaneously
The regularation like weight decay in DNN is related to the Inequality Constraints optimisation. According to Lagrange Multipliers and the KarushKuhnTucker conditions, there exist a unique λ when subject to KKT condition. But when we solve the optimize problem, we just set a prior value instead of solve it, why?