keras neural network training and validation loss with regular spikes
I trained a simple neural network in keras. The training loss and validation loss shows regular spikes after certain epochs. The xaxis of the figure is epochs, and the yaxis is loss. Notice the large validation loss spike correspond with small training loss spike. I used fit_generator. I tried optimizer SGD and Adam, and the spikes still appear.
Interestingly, changing batch size doesn't influence the shape or period of the spike. Changing optimizer influences spike magnitude, but doesn't influence the repeating period.
Any thoughts on why this happens?
See also questions close to this topic

Aggregated instance list from all regions of AWS using Boto3 Python library
I want to get instances from all regions of AWS in a single call of Boto3 python library.
We have like below in GCP for aggregated instances list:
request = service.instances().aggregatedList(project=project)
Currently I'm using instance.all() method but problem with this is that, I've to create ec2resource for each region separately which takes lot of time for listing instances from each region. Currently my code is as below:
region='useast2' ec2resource = boto3.resource('ec2', region_name=region, aws_access_key_id=ac_k, aws_secret_access_key=sec_k) instances = ec2resource.instances.all()
Is there any alternate and time efficient solution for above problem so that I can get all instances from all regions in a single call and save time?

get wrong list use map function when geting the coresponding elements of a list from another list in python
I have two lists:
a = [1,2,3,4] b = [True,False,True,False]
and I want get the elements of
a
that coresponding to theTrue
inb
, don't usefor
loop. I use themap
function to solve it, but it went wrong:def f(x,y): if x: return y s = list(map(f,b,a))
s
is[1, None, 3, None]
， and i don't want theNone
, how can I do? 
Python multivolume zip extract
There are a Python library (not external application like 7z) to extract multipart Zip archive (*.z01, *z02....) ? Standard libs don't permit it and I don't find external lib taht make it.

Having Problems With Implementing the Perceptron Algorithm in Python
I'm having a problem debugging the following code, for some reason The perceptron stops updating itself after a couple of steps with random values as the weights. I have tried not using a class for my work and edited everything to the bare minimum, but still had the same problem. I have also checked the Perceptron.train(), and it works just fine. So, I'm guessing the main problem is with the train function itself. I am kind of new to python programming so any help would be apreciated guys. import random import Plot as plt import numpy as np
#Function Of the line that seperates the two different Data Types$ def f(x): return x #Activation Function# def act(x): if x >= 0: return 1.0 return 0.0 class Point: def __init__(self, x, y): self.X = x self.Y = y if y > f(x): self.Target = 1.0 else: self.Target = 0.0 class Perceptron: def __init__(self, n, actFunc = act, lr = 0.2): self.Weights = [0 for i in range(n)] self.ActFunc = actFunc self.LR = lr def guess(self, inputs): valSum = 0 for i in range(len(inputs)): valSum += self.Weights[i] * inputs[i] return self.ActFunc(valSum) def train(self, inputs, target): cal = self.guess(inputs) err = target  cal for i in range(0, len(self.Weights)): self.Weights[i] += self.LR * err * inputs[i] def printWeights(self): for i in range(len(self.Weights)): print("WEIGHT[" + str(i) + "] = " + str(self.Weights[i])) print("") def lineFunc(self): # y = w0 + w1x + w2y # (1  w2)y = w0 + w1x # y = w0/(1w2) + w1/(1  w2)x w0 = self.Weights[0] w1 = self.Weights[1] w2 = self.Weights[2] return (str(w0/(1  w2)) + " + " + str(w1/(1  w2)) + " * x") #INITIALISING DATA# brain = Perceptron(3) n = 20 points = [Point(random.uniform(10, 10), random.uniform(10, 10)) for x in range(n)] t = 1000 #Training# for i in range(t): point = points[random.randrange(0, n)] brain.train([1, point.X, point.Y], point.Target) brain.printWeights() print(brain.lineFunc())

Error in neural network loop
I have constructed a loop that builds separate neural network models with various numbers of nodes in the hidden layer and it works as planned. However, I am now trying to add in to the loop some code that uses ModelCheckpoint (https://keras.io/callbacks/) to store the model with the best weights and then refit a model using those weights. However, I receive the following error:
ValueError: You are trying to load a weight file containing 6 layers into a model with 9 layers.
The numbers (i.e., 6 and 9) change each time I receive this error. There must be something going wrong with the way the 'model_weights.hdf5' is being loaded into the loop.
Here is my code:
# Import libraries import math from keras.models import Sequential from keras.layers import Dense from keras.callbacks import EarlyStopping # Save objects model = Sequential() n_cols = len(X.columns) # Define early_stopping_monitor early_stopping_monitor = [EarlyStopping(patience=2)] # Set number of epochs epochs = 10 # Save model and weights with lowest loss val filepath = 'model_weights.hdf5' from keras.callbacks import ModelCheckpoint callbacks_list = [ModelCheckpoint(filepath, monitor='val_loss', verbose=True, save_best_only=True, mode='min')] # Initiate lists (3 models) nodes_list = [math.ceil((n_cols+2)/2), math.ceil((n_cols*(2/3))+2), n_cols] for nodes in nodes_list: # Add first layer model.add(Dense(units=n_cols, activation='relu', input_shape=(n_cols,))) # Add hidden layer model.add(Dense(units=nodes, activation='relu')) # Add output layer model.add(Dense(units=2, activation='softmax')) # Compile model model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) # Fit model model.fit(X, y, validation_split=0.33, epochs=epochs, callbacks=callbacks_list, verbose=True) # Load in the model with the best weights model.load_weights('model_weights.hdf5') # Compile model model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy']) # Refit the model model.fit(X, y, validation_split=0.33, epochs=epochs, callbacks=early_stopping_monitor, verbose=True)

Validation loss and accuracy fluctuates
I am training a simple classifier to classify ALL (Acute Lymphoid Leukemia) images. It is basically cancerous vs noncancerous classification problem. The overall training looks okay, all the accuracy and loss curves are behaving the way they should. I am getting decent results on test set as well.
I noticed that the Validation loss fluctuates more than usual and so does the validation accuracy. The fluctuation range for Val accuracy lies between 0  30 %, i.e. The accuracy of current epoch is in the following range:
previous epoch's accuracy ± 30
Same goes for the loss, the fluctuation range is slightly higher than expected. Is this something that I should be worried about?
This can referred from the training graph attached below.

Keras LSTM has different output for single input
I came to a weird problem when use Keras LSTM model. I build a single layer LSTM and try to play with it. I found the the output of model is different between one input and multiple inputs, as shown in the following code.
def lstmTest(training, latent_dim=10): _, time_dim, input_dim = training.shape # Define an input sequence and process it. encoder_inputs = Input(shape=(time_dim, input_dim), name='input') encoder = LSTM(latent_dim, return_state=False, name='lstm') encoder_outputs = encoder(encoder_inputs) model = Model(encoder_inputs, encoder_outputs) return model def trainingTest(model, training, nb_epoch=10, batch_size=300): model.compile(optimizer='adam', loss='mse', metrics=['acc']) history = model.fit(training, training[:, 1, :10], epochs=nb_epoch, batch_size=batch_size, shuffle=True, verbose=1, ).history return history myVector = [[[i]*20]*8 for i in range(100)] myVector = np.array(myVector) lstmTest = lstmTest(myVector) history = trainingTest(lstmTest, myVector) vector = myVector[:2] res1 = lstmTest.predict(vector) vector = myVector[:1] res2 = lstmTest.predict(vector) res2[0]  res1[0]
I got the following result
array([0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 5.8207661e11, 0.0000000e+00, 2.3283064e10, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00, 0.0000000e+00], dtype=float32)
But If I change res2 to the same as res1 I got the expected result
array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)
Anyone came to this problem before?

Second Loss Function in a Intermadiate Layer of Multilabel Classification CNN for Binary Feature Extraction
I tried to extract binary codes over a multilabel classification cnn and i did. But to get more efficient codes i applied a second loss function, maximizes square error between Dense_48's sigmoid and 0.5, to Out_1. ( codes are below ) There are 2000 images and all of them have the size of 256x256 rgb.
print(train_X.shape) (1980, 256, 256, 3) print(valid_X.shape) (20, 256, 256, 3) print(train_label.shape) (1980, 5) print(valid_label.shape) (20, 5)
When compile the code everything is OK but during training (model.fit .....) i got this error :
File "./mult_loss.py", line 132, in train_dropout = model.fit(np.array(train_X), np.array(train_label), validation_data=(np.array(valid_X), np.array(valid_label)))
File "/usr/local/lib/python2.7/distpackages/keras/engine/training.py", line 955, in fit batch_size=batch_size)
File "/usr/local/lib/python2.7/distpackages/keras/engine/training.py", line 792, in _standardize_user_data exception_prefix='target')
File "/usr/local/lib/python2.7/distpackages/keras/engine/training_utils.py", line 100, in standardize_input_data str(len(data)) + ' arrays: ' + str(data)[:200] + '...')
ValueError: Error when checking model target: the list of Numpy arrays that you are passing to your model is not the size the model expected. Expected to see 2 array(s), but instead got the following list of 1 arrays:
[array([[1, 0, 0, 0, 0], [0, 0, 1, 0, 0], [0, 1, 0, 0, 1], ..., [1, 0, 0, 0, 0], [1, 0, 0, 0, 1], [1, 0, 0, 0, 0]], dtype=uint8)]...
HERE IS THE CODE
batch_size = 64 epochs = 32 num_classes = train_Y.shape[1] hash_bits = 48 visible = Input(shape=(256,256,3),name = 'visible') conv_1 = Conv2D(32, kernel_size=4, activation='linear')(visible) lReLU_1 = LeakyReLU(alpha=0.1)(conv_1) MaxPool_1 = MaxPooling2D((2, 2),padding='same')(lReLU_1) Drop_1 = Dropout(0.25)(MaxPool_1) conv_2 = Conv2D(64, kernel_size=3, activation='linear')(Drop_1) lReLU_2 = LeakyReLU(alpha=0.1)(conv_2) MaxPool_2 = MaxPooling2D((2, 2),padding='same')(lReLU_2) Drop_2 = Dropout(0.25)(MaxPool_2) conv_3 = Conv2D(128, kernel_size=3, activation='linear')(Drop_2) lReLU_3 = LeakyReLU(alpha=0.1)(conv_3) MaxPool_3 = MaxPooling2D((2, 2),padding='same')(lReLU_3) Drop_3 = Dropout(0.4)(MaxPool_3) Flat_1 = Flatten()(Drop_3) Dense_12 = Dense(128, activation='linear')(Flat_1) lReLU_4 = LeakyReLU(alpha=0.1)(Dense_128) Drop_4 = Dropout(0.4)(lReLU_4) Dense_48 = Dense(48, activation='sigmoid')(Drop_4) Out_1 = Dense_48 Dense_5 = Dense(5, activation='sigmoid')(Dense_48) Out_2 = Dense_5 model = Model(inputs=visible, outputs=[Out_1, Out_2]) e=0.5 import keras.backend as K def custom_loss (Out_1, e): return (K.sum((Out_1e)**2) ) * (1/48) from keras.optimizers import SGD sgd = SGD(lr=0.01, decay=1e6, momentum=0.9, nesterov=True) model.compile(loss=[custom_loss, 'categorical_crossentropy'], loss_weights=[1., 1.], optimizer=sgd, metrics=['accuracy']) train_X = train_X.astype("float32") valid_X = valid_X.astype("float32") train_X /= 255 valid_X /= 255 train_dropout = model.fit(train_X,train_label, validation_data=(valid_X,valid_label))
Any recommendation or solution appreciated ...

RNN not learning Keras
I have been training RNNs in keras for some time now, but recently I was faced with a problem of RNN not learning anything. Therefore, I want to make sure that I am feeding in the data correctly. I know that keras expect your input shape to be a 3D tensor, so following is the code that I am using to reshape a 2D numpy array to a 3D array. Please check if this is the right approach or not? The idea here is to take the first 3 input values and add it to the time dimension. I have also included a small snippet of the training session. You can see that the loss is changing extremely slowly after each epoch and the accuracy not improving at all.
My question here is this, am I making any mistake in reshaping the input? Also what could be the reasons for my model not learning anything? I have tried a lot of hyperparameters, optimizers, different batch sizes, simplified model as much as possible (just used a single RNN cell). Still nothing, no learning at all. This a classification problem, by the way, I have 118 features and 17 classes. I am using categorical cross entropy loss and Adam optimizer.
X_train_seq = [] for i in range(3,len(X_train_scaled)): X_train_seq.append(X_train_scaled[i3:i,0:]) X_train_seq = np.array(X_train_seq) 3 is the time step
screenshot of the model getting stuck and not learning anything during training

NonLinear Delay Model for logic gates using Machine learning
I have to develop a non linear model for cell libraries (inverters, nand, etc) which will calculate the delay. This happens in simulator SPICE, but i have to do this using machine learning. I am new to machine learning so that's why i wanted to know how i can proceed with this.
I have ".lib" file which contains something like
cell_rise (delay_template) { index_1 ("0.001169, 0.005416, 0.01391, 0.03037, 0.06381, 0.1307, 0.2645, 0.532"); index_2 ("7.906e05, 0.007225, 0.02152, 0.04921, 0.1055, 0.218, 0.4431, 0.8933"); values ( \ "0.001217, 0.003174, 0.006568, 0.01305, 0.0262, 0.05249, 0.1051, 0.2101", \ "0.001635, 0.003949, 0.00758, 0.01413, 0.02727, 0.05354, 0.1062, 0.2112", \ "0.00197, 0.004875, 0.009089, 0.01613, 0.02952, 0.05581, 0.1085, 0.2133", \ "0.002252, 0.005958, 0.011, 0.01895, 0.03323, 0.06004, 0.1127, 0.2179", \ "0.002358, 0.007173, 0.01348, 0.02288, 0.03891, 0.06749, 0.1211, 0.2264", \ "0.002113, 0.008425, 0.0165, 0.02816, 0.04704, 0.07897, 0.1361, 0.2436", \ "0.0008438, 0.009034, 0.01957, 0.03442, 0.05773, 0.09518, 0.159, 0.2734", \ "0.002203, 0.008119, 0.02182, 0.04102, 0.07071, 0.1167, 0.1914, 0.3188" \ );
Now in my machine learning model i have inputs
index_1
,index_2
,(5,5) matrix
andgate_name
. I have to predict the whole8*8 matrix(values)
and whichgate_name
it belongs to.I know this is a regression and classfication problem as well, but how to go forward with it? Kindly share your instincts with me.

Voltile GPU utilization in caffe training
I use GPU for caffe training. My caffe algorithm has printed output as
I0814 12:27:37.772343 22016 caffe.cpp:217] Using GPUs 0 I0814 12:27:37.793123 22016 caffe.cpp:222] GPU 0: Tesla V100PCIE16GB I0814 12:27:38.285532 22016 solver.cpp:65] Initializing solver from parameters: train_net: "/home/user/Projects/caffessd/Numberplate/RregularNet/Net_19/train_0.prototxt" test_net: "/home/user/Projects/caffessd/Numberplate/RregularNet/Net_19/test_0.prototxt"
That means GPU is used in training and my
GPU is Tesla V100
.I checked my nvidiasmi and Voltile GPU Utilization is quite low. Maximum I have seen is 17% only in training Caffe.
Is that normal or something not right?
Nvidiasmi screen shot is shown below.

Building CNN with large images
as I am now building an 3d cnn for medical images with high resolution, which means that the size is large. So I am just wandering that is there any effective way to crop the images into pieces with a fixed size, e.g. 64 * 64 * 64 and then put it into the cnn?