scipy.optimize.minimize cost function of neural networks fails
I have written the cost function of a neural network for image classification problem. To optimize my cost function i have used scipy.optimize.minimize with solver ='TNC' though their isn't any run time error but after complication of optimization, result says success = 'false'
Here is my cost function
def costandGrad(nn_param,X,Y_matrix,lam,i,j):
Theta1,Theta2 = reshape(nn_param,i,j)
Theta1 = np.matrix(Theta1)
Theta2 = np.matrix(Theta2)
#part1  IMPLEMENTING FEEDFORWARD PROPOGATION ALGORTHIM
a1 = X
Z2 = X*np.transpose(Theta1)
a2 = sigmoid(Z2)
a2 = np.hstack((np.ones([len(a2),1]),a2))
Z3 = a2*np.transpose(Theta2)
a3 = sigmoid(Z3)
Hyp = a3
#PART2  IMPLEMENTING COST WITH REGULARISATION
first_term = np.trace((np.transpose(Y_matrix))* (np.log(Hyp)));
second_term = np.trace(((1(np.transpose(Y_matrix)))*(np.log(1Hyp))));
JU = first_term + second_term
sq1 = (np.square(Theta1[:,1:]))
sq2 = (np.square(Theta2[:,1:]))
regularization = (lam/(2*m)) * (sum(np.transpose(sum(sq1))) + sum(np.transpose(sum(sq2))))
J = JU + regularization
#PART3  BACKPROPOGATION ALGORTHIM
D3 = Hyp  Y_matrix
D2 = D3*Theta2[:,1:]
D2 = np.multiply(D2,sigmoidGrdadient(Z2))
Delta1 = np.transpose(D2)*X;
Delta2 = np.transpose(D3)*a2;
Theta1_Grad = 1/m*(Delta1)
Theta2_Grad = 1/m*Delta2
nn_parameters = np.concatenate((np.array(Theta1).ravel(), np.array(Theta2).ravel()))
return J,nn_parameters
here is the result
fim = sc.optimize.minimize(fun=costandGrad, x0=nn_params, args=(X_train, Y_matrix, 3,i,j), method='TNC', jac=True, options={'maxiter': 2000})
fun: matrix([[227085.02475254]])
jac: array([ 0.08245473, 0.00159231, 0.47998975, ..., 1.12524447,
1.30664152, 1.14908335])
message: 'Linear search failed'
nfev: 50
nit: 0
status: 4
success: False
x: array([ 0.08245473, 0.00159231, 0.47998975, ..., 1.12524447,
1.30664152, 1.14908335])enter code here
See also questions close to this topic

Selenium Python Unable to scroll down, while fetching google reviews
I am trying to fetch google reviews with the help of selenium in python. I have imported webdriver from selenium python module. Then I have initialized self.driver as follows:
self.driver = webdriver.Chrome(executable_path="./chromedriver.exe",chrome_options=webdriver.ChromeOptions())
After this I am using the following code to type the company name on google homepage whose reviews I need, for now I am trying to fetch reviews for "STANLEY BRIDGE CYCLES AND SPORTS LIMITED ":
company_name = self.driver.find_element_by_name("q") company_name.send_keys("STANLEY BRIDGE CYCLES AND SPORTS LIMITED ") time.sleep(2)
After this to click on the google search button, using the following code:
self.driver.find_element_by_name("btnK").click() time.sleep(2)
Then finally I am on the page where I can see results. Now I want to click on the View on google reviews button. For that using the following code:
self.driver.find_elements_by_link_text("View all Google reviews")[0].click() time.sleep(2)
Now I am able to get reviews, but only 10. I need at least 20 reviews for a company. For that I am trying to scroll the page down using the following code:
self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);") time.sleep(5)
Even while using the above code to scroll the down the page, I am still getting only 10 reviews. I am not getting any error though.
Need help on how to scroll down the page to get atleast 20 reviews. As of now I am able to get only 10 reviews. Based on my online search for this issue, people have mostly used: "driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")" to scroll the page down whenever required. But for me this is not working. I checked the the height of the page before and after ("driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")") is the same.

Create a new column from itereated rows of timedate data
I am attempting to create a downward velocity model for offshore drilling which uses the variables Depth (which increases every 1 foot) and DateTime data which is more intermittent and is only updated every foot of depth:
Dept DateTime 1141 5/24/2017 04:31 1142 5/24/2017 04:32 1143 5/24/2017 04:40 1144 5/24/2017 04:42 1145 5/25/2017 04:58
I am trying to get something like this:
Where Velocity iterated down dept/(DateTime gap)

one to one mapping in shell script
I am in process of migration. Migrating from old set of servers to new set of servers, where there is no logical relationship in the server names between the 2 sets. I have a script that runs on old server, takes all necessary backups and the run another script to copy the backups to new server and execute it.
I can combine both scripts(taking backup and copying to new server), if I can include a logic to map the old server to new server. Is there a way I can do this.
Old server New server King Queen Bat Ball water fire sand rock
What I am expecting is, if the script is run on server 'King', I want the script to identify that the corresponding new server is 'Queen' and copy the backups to Queen.
 How to plan path from point 1 to point 2 so that it remains in the specified region?

R  How to maximise based of sum of other column from data frame
So I have created a data frame, representing a fantasy team, where each position (a,b,c) have a given points (points) gained and a price(price):
library(reshape2) x<102 #num players replicate(x/3,paste(letters[1:3]))>l l<melt(l) l<l[,3] l<data.frame(l) # pnt< as.integer(runif(90,min=4, max=30)) pnt<pmax(round(as.numeric(rnorm(x,mean=6,sd=4)),digits=0),3) prc<pmax(sort(round(as.numeric(rnorm(x,mean=6,sd=3)),digits=1)),4.5) df1<as.data.frame(pnt) df2<as.data.frame(prc) df<data.frame(df1,df2,l) rownames(df) < seq(from=1,to=x) colnames(df)<c("points","price","pos") p<df[rev(order(df$pos)),] a<p[((2*x/3)+1):x,1:3] a<a[rev(order(a$points)),]
I am using just my data frame 'a' at the moment that looks like :
price points pos 1 7 14 a 2 8 12 a 3 3 8 a 4 10 7 a
I am trying to select the maximum number of points possible through combinations of 4 players (represented by each row). Normally that would be easy and just order the dataframe by points and select the top 4. However I want to impose a maximum price limit on the four players of 28. (this is an arbitrary number to show the problem) This would potentially rule out the first 4 players, and potentially allow for the maximum points under that price limit to not be consecutive (in points order).
Have you any suggestions on how to go about this? I tried the following but it only allows for consecutive points to be selected.
z<integer() y<integer() for(i in 1:31){ j<i+2 x<sum(a[i:j,]$point) xx<sum(a[i:j,]$price) y<c(y,x) z<c(z,xx) yz<data.frame(y,z) } yz #add points per price yz$c<with(yz,y/z) yz[which(match(yz$c,max(yz$c))==TRUE),]
I have a feeling this is an optimisation problem

Cost of using OPTION (RECOMPILE) vs Joining to tables
Using: SQL Server 2016+
I've been looking to see if there is any way to evaluate how long it takes SQL Server to recompile the execution plan for a query. We have several stored procedures where we select from table 1 based on a nullable parameter in table 2. E.g. If customer is not null return their sales else return all sales.
Sample data:
DROP TABLE IF EXISTS dbo.TestTable1; DROP TABLE IF EXISTS dbo.TestTable2; CREATE TABLE dbo.TestTable1 (ID INT NOT NULL PRIMARY KEY CLUSTERED , TextValue NVARCHAR(255) NULL); CREATE TABLE dbo.TestTable2 (ID INT NOT NULL PRIMARY KEY CLUSTERED , TextValue NVARCHAR(255) NULL); INSERT INTO TestTable1 (ID, TextValue) VALUES (1, N'Table 1  Text 1'), (2, N'Table 1  Text 2'), (3, N'Table 1  Text 3'), (4, N'Table 1  Text 4'), (5, N'Table 1  Text 5'), (6, N'Table 1  Text 6'), (7, N'Table 1  Text 7'), (8, N'Table 1  Text 8'), (9, N'Table 1  Text 9'), (10, N'Table 1  Text 10'); INSERT INTO TestTable2 (ID, TextValue) VALUES (1, N'Table 2  Text 1'), (2, N'Table 2  Text 2'), (3, N'Table 2  Text 3'), (4, N'Table 2  Text 4'), (5, N'Table 2  Text 5'), (6, N'Table 2  Text 6'), (7, N'Table 2  Text 7'), (8, N'Table 2  Text 8'), (9, N'Table 2  Text 9'), (10, N'Table 2  Text 10');
This is greatly over simplified as we would have multiple possible conditions, linking to multiple tables. We are currently looking at recompiling the query so that the joins to the secondary table are only made when required.
DECLARE @LookupValue NVARCHAR(50); SET @LookupValue = NULL; SELECT * FROM dbo.TestTable1 T1 WHERE @LookupValue IS NULL OR EXISTS ( SELECT TOP (1) 1 A FROM dbo.TestTable2 T2 WHERE T1.ID = T2.ID AND T2.TextValue = @LookupValue) OPTION (RECOMPILE) SET @LookupValue = N'Table 2  Text 1'; SELECT * FROM dbo.TestTable1 T1 WHERE @LookupValue IS NULL OR EXISTS ( SELECT TOP (1) 1 A FROM dbo.TestTable2 T2 WHERE T1.ID = T2.ID AND T2.TextValue = @LookupValue) OPTION (RECOMPILE);
As you can see from the query plan below with the recompile table 2 is effectively removed from the execution.
There is however a cost to recompiling, and I'm looking to trace this overhead so I can make sensible decisions as to which of our queries to format in this way. In general I am seeing that recompiling is consistently faster but there are lots of post stating that this means the execution plans can come out as being far less than optimal.
Any guidance on measuring these overheads or any issues I should investigate before implementing this more widely would be gratefully received.
Many thanks.

NaNs at the end of an array are creeping in when using interp1d
I have 2 2D arrays of x and y positions that express values of a contour plot. As I translate the contour plot multiple times, I need to track the original coordinates as they move too, so that I can express a final displacement at all points of the original image.
I am writing in python3 using scipy.interpolate.interp1d. After one iteration, I have no problems, and the coordinates have been successfully interpolated back onto the original grid. I print out the final 10 elements of one of the columns in the 2D array:
import numpy as np from scipy.interpolate import interp1d print(y_coordinate[10:, 20]) >>> [ 60. 61. 62. 63. 64. nan nan nan nan nan]
I have nans because the displacement was 5 in the y direction of an image 64 pixels wide, so the 5 pixels at the top have shifted down. The problem is, that in the next step I have zero displacement in the y direction, so the points do not move here, but they might move elsewhere in my large array, so need to interpolate over the entire array.
I interpolate column by column using scipy.interpolate.interp1d as so (and remove the indices of the array for clarity, but know they are the same as the previous lines above):
f = interp1d(y, y_coordinate, kind='linear', bounds_error = False) new_y_coordinate = f(y)
where y_coordinate is as printed out above, y is
>>>[55,56,57,58,59,60,61,62,63,64]
So essentially I want it to return the exact same values as they already lie on the points I want it to interpolate onto. Instead, I get this:
print(new_y_coordinate) >>> [ 60. 61. 62. 63. nan nan nan nan nan nan]
I've got the nan creeping in to good data. What's strange is first of all I tried this with complex numbers and thought it was an error to do with those. I then changed it to be two arrays of real numbers and found the error was solved on the left hand side of the array (for x direction translations) but is now occurring on the top for y displacement translations. A simple test I did with a similar situation didn't have this error at all.
Any ideas on what is happening and how I can solve it?

Scipy script not working as expected even after working with toy values
i'm trying to write a curve fit and chi squared script. i have some sample data, but for some reason the fit doesn't come out as expected, and i get chi squared values that don't make sense (i calculated it with matlab already so i know what to expect). I have no idea why the script doesn't work, since it already worked fine with toy values. Here is the script:
import matplotlib.pyplot as plt import matplotlib as matp import numpy as np import scipy.optimize as opt import math import pandas as pd from scipy import stats #Read the csv file to a DataFrame df = pd.read_csv('HarmonicData.csv') #Define data xdata = df.m dX = df.dm ydata = df.Tavg dY = df.dTavg def func(x, a, b): return a/x**2 + b #Title and axis title Variables title = 'Fit' ytitle = 'Tavg [sec]' xtitle = 'r [cm]' #Define plot and add errorbars fig, ax = plt.subplots() ax.errorbar(xdata, ydata, yerr=dY, xerr=dX, fmt='o', ms=2) #Axis Title Setting ax.set_title(title) ax.set_ylabel(ytitle) ax.set_xlabel(xtitle) #Curve parameter initial guess paraguess = ([0.230000, 0.300000]) #Curve Fit parafit, pcov = opt.curve_fit(func, xdata, ydata, p0=paraguess) #Reduced Chi squared chi_sq = np.sum(((func(xdata, *parafit)ydata)/dX)**2) red_chi_sq = (chi_sq)/(len(xdata)len(parafit)) #Print results print('initial guess for parameters: a= %.8f b= %.8f' % tuple(paraguess)) print ('The degrees of freedom for this test is', len(xdata)len(parafit)) print ('The chi squared value is: ',("%.2f" %chi_sq)) print ('The reduced chi squared value is: ',("%.2f" %red_chi_sq)) #Show and plot plt.plot(xdata, func(xdata, *parafit), 'r', label='fit: a=%.3f b=%.3f' % tuple(parafit)) plt.legend() plt.show()
I have no idea why this happened, and i can't find a reason to it. I would love some help since i'm not very experienced with scipy/python in general

seaborn distplot() shows negative values when display a gamma distribtuion
I'm trying to display a gamma distribution by using
sns.distplot()
to aviod plotting the pdf of it like the following:import seaborn as sns import numpy as np from scipy.stats import gamma # creating a population shape, scale, size = 2, 1, 200 pop = np.random.gamma(shape, scale, size) # plotting sns.distplot(pop, hist=False, kde=False, fit=gamma, color='r', fit_kws={'color': 'r', 'linewidth': 2.5})
For some reason, seaborn shows the following plot with a left tail, but I checked the
pop.min()
and there is no negative vlaue. I must useplt.xlim(0)
to avoid it. Why did this happen and is there any way to avoid it? 
Strange result Neural network Python
I followed an article here: TowardsDataScience.
I wrote math equations about the network, everything made sense.
However, after writing the code, results are pretty strange, like it is predicting always same class...
I spent a lot of time on it, changed many things, but I still cannot understand what I did wrong.
Here is the code:
# coding: utf8 from mnist import MNIST import numpy as np import math import os import pdb DATASETS_PREFIX = '../Datasets/MNIST' mndata = MNIST(DATASETS_PREFIX) TRAINING_IMAGES, TRAINING_LABELS = mndata.load_training() TESTING_IMAGES , TESTING_LABELS = mndata.load_testing() ### UTILS def sigmoid(x): return 1 / (1 + np.exp(x)) def d_sigmoid(x): return x.T * (1  x) #return np.dot(x.T, 1.0  x) def softmax(x): e_x = np.exp(x  np.max(x)) return e_x / e_x.sum() def d_softmax(x): #This function has not yet been tested. return x.T * (1  x) def tanh(x): return np.tanh(x) def d_tanh(x): return 1  x.T * x def normalize(image): return image / (255.0 * 0.99 + 0.01) ### !UTILS class NeuralNetwork(object): """ This is a 3layer neural network (1 hidden layer). @_input : input layer @_weights1: weights between input layer and hidden layer (matrix shape (input.shape[1], 4)) @_weights2: weights between hidden layer and output layer (matrix shape (4, 1)) @_y : output @_output : computed output @_alpha : learning rate """ def __init__(self, xshape, yshape): self._neurones_nb = 20 self._input = None self._weights1 = np.random.randn(xshape, self._neurones_nb) self._weights2 = np.random.randn(self._neurones_nb, yshape) self._y = np.mat(np.zeros(yshape)) self._output = np.mat(np.zeros(yshape)) self._alpha1 = 0.1 self._alpha2 = 0.1 self._function = sigmoid self._derivative = d_sigmoid self._epoch = 1 def Train(self, xs, ys): for j in range(self._epoch): for i in range(len(xs)): self._input = normalize(np.mat(xs[i])) self._y[0, ys[i]] = 1 self.feedforward() self.backpropagation() self._y[0, ys[i]] = 0 def Predict(self, image): self._input = normalize(image) out = self.feedforward() return out def feedforward(self): self._layer1 = self._function(np.dot(self._input, self._weights1)) self._output = self._function(np.dot(self._layer1, self._weights2)) return self._output def backpropagation(self): d_weights2 = np.dot( self._layer1.T, 2 * (self._y  self._output) * self._derivative(self._output) ) d_weights1 = np.dot( self._input.T, np.dot( 2 * (self._y  self._output) * self._derivative(self._output), self._weights2.T ) * self._derivative(self._layer1) ) self._weights1 += self._alpha1 * d_weights1 self._weights2 += self._alpha2 * d_weights2 if __name__ == '__main__': neural_network = NeuralNetwork(len(TRAINING_IMAGES[0]), 10) print('* training neural network') neural_network.Train(TRAINING_IMAGES, TRAINING_LABELS) print('* testing neural network') count = 0 for i in range(len(TESTING_IMAGES)): image = np.mat(TESTING_IMAGES[i]) expected = TESTING_LABELS[i] prediction = neural_network.Predict(image) if i % 100 == 0: print(expected, prediction) #print(f'* results: {count} / {len(TESTING_IMAGES)}')
Thank you for your help, really appreciated.
Julien

No overfitting by increasing the number of epochs
I use a feed foreward neural network with one hidden layer for my thesis. Threby I have 600 training data and 104 input and output values. There I want to show now the properties of the neural network and also want to show the overfitting when I increase the number of epochs. To do so, I first wanted to find the optimum for learning rate and number of hidden nodes, where I got the following results:
Based on that I decided to choose a learning rate of 0.0125 and 250 hidden nodes. But by using this set of parameters, I still have no overfitting when I increase the number of epochs, which can be seen here:
In this plot I showed in blue my old set of parameters and in theory I wanted to show how it improve when I use the best set of parameters, but it's just varying a bit. I also tested it until epoch 1000 but the accuracy with this value was still 0.830.
Does someone has an idea why this happen?
Thanks a lot for your help!

Perfrom Genetic Algorithm in Training a Neural Network
I want to train neural network using Genetic ALgorithm. Is there any library available in Python like Keras? Thanks. Any help will be very much appreciated.