Can Python optimize my function inputs to get a target value?
I have been trying to locate a method similar to Excel's Solver where I can target a specific value for a function to converge on. I do not want a minimum or maximum optimization.
For example, if my function is:
f(x) = A^2 + cos(B)  sqrt(C)
I want f(x) = 1.86, is there a Python method that can iterate a solution for A, B, and C to get as close to 1.86 as possible? (given an acceptable error to target value?)
1 answer

You need a root finding algorithm for your problem. Only a small transformation required. Find roots for g(x):
g(x) = A^2 + cos(B)  sqrt(C)  1.86
Using
scipy.optimize.root
, Refer documentation:import numpy as np from scipy import optimize # extra two 0's as dummy equations as root solves a system of equations # rather than single multivariate equation def func(x): # A,B,C represented by x ndarray return [np.square(x[0]) + np.cos(x[1])  np.sqrt(x[2])  1.86, 0, 0] result = optimize.root(func , x0 = [0.1,0.1,0.1]) x = result.x A, B, C = x x # array([ 1.09328544, 0.37977694, 0.06970678])
you can now check your solution:
np.square(x[0]) + np.cos(x[1])  np.sqrt(x[2]) # 1.8600000000000005
See also questions close to this topic

Improving performance of XML to dataframe parsing
I have a long xml with the following structure:
<?xml version="1.0" encoding="UTF8" standalone="yes"?> <SoccerFeed timestamp="20180212T2228+0000"> <Game id="3323" home_team_id="83" home_team_name="CFC" away_team_id="335" away_team_name="WCFC" game_date="22180212T2000+0000" competition_id="18" competition_name="DCF" season_id="2011" season_name="Season 2017/2018" matchday="27" period_1_start="22180212T2000+0000" period_2_start="22180212T2105+0000" Status="FullTime"> <Event id="1182774006" event_id="10223" type_id="37" period_id="14" min="0" sec="0" player_id="1212" team_id="83" outcome="1" x="45.7" y="59.0" timestamp="20180212 21:52:45" last_modified="20180212 21:52:48" version="15184723628801"> <Q id="16517621865" qualifier_id="302"/> </Event> <Event id="120438223891" event_id="8726" type_id="30" period_id="14" min="0" sec="0" player_id="" team_id="335" outcome="0" x="" y="" timestamp="20180212 21:52:46" last_modified="20180212 21:52:46" version="15128472366490"> <Q id="125850245236" qualifier_id="57" value="1.0"/> <Q id="1775059222" qualifier_id="209"/> </Event> </Game> </SoccerFeed>
The file normally has 1500+ Event nodes.
My current code parses the file in 2021 seconds but I feel like there are some obvious performance improvements that I am missing. Any tips would be appreciated.
I am pretty sure there are better ways to access all of the attributes and their values without having to define each one individually.
Current Code
import xml.etree.ElementTree as ET import pandas as pd # define the location of your file file_name = "possessions.xml" # parse the xml and conver to a tree and root tree = ET.parse(file_name) root = tree.getroot() ## get the main game info from the single 'Game' node gameinfo = root.findall('Game') gameinfo = gameinfo[0] game_id = gameinfo.get('id') home_team_id = gameinfo.get('home_team_id') home_team_name = gameinfo.get('home_team_name') away_team_id = gameinfo.get('away_team_id') away_team_name = gameinfo.get('away_team_name') competition_id = gameinfo.get('competition_id') competition_name = gameinfo.get('competition_name') season_id = gameinfo.get('season_id') ## create an empty dataframe to store the per event information catcher = pd.DataFrame() # loop through each event node and grab the information for i in root.iter('Event'): # get the info from the Event node main chunk id = i.get('id') event_id = i.get('event_id') type_id = i.get('type_id') period_id = i.get('period_id') min = i.get('min') sec = i.get('sec') player_id = i.get('player_id') team_id = i.get('team_id') x = i.get('x') y = i.get('y') possession_id = i.get('possession_id') sequence_id = i.get('sequence_id') # combine the event main node data into a dataframe Edata = pd.DataFrame( {'id': id, 'event_id': event_id, 'type_id': type_id, 'period_id': period_id, 'min': min, 'sec': sec, 'player_id': player_id, 'team_id': team_id, 'x': x, 'y': y, 'sequence_id': sequence_id, 'possession_id':possession_id }, index = [0]) # find all of the Q information for that file Qs = i.findall("./Q") # create some empty lists to append the results to qualifier_id = [] Q_value = [] # loop through all of the Qs and grab the info for child in Qs: qualifier_id.append(child.get('qualifier_id')) Q_value.append(child.get('value')) # create a dataframe from the two lists Qdata = pd.DataFrame( {'qualifier_id': qualifier_id, 'value': Q_value }) # transform the dataframe, 1st row to the headers, # drop the extra row, reset the index Qdata = Qdata.T Qdata.columns = Qdata.iloc[0] Qdata = Qdata.reindex(Qdata.index.drop('qualifier_id')) Qdata = Qdata.reset_index(drop=True) # combine the event info and qualifer info result = pd.concat([Qdata, Edata], axis=1) # bind to the catcher dataframe catcher = catcher.append(result) # Add all of the the game information to each row catcher['competition_id'] = competition_id catcher['game_id'] = game_id catcher['home_team_id'] = home_team_id catcher['home_team_name'] = home_team_name catcher['away_team_id'] = away_team_id catcher['away_team_name'] = away_team_name catcher['competition_id'] = competition_id catcher['competition_name'] = competition_name catcher['season_id'] = season_id catcher['competition_id'] = competition_id

OpenCV. Find contour in an image
How can I find contours in this image?
After threshold I get this
and I get two contours in the whole image
This is my code
image = cv2.imread(captcha_image_file) gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) thresh = cv2.threshold(gray, 0, 255, cv2.THRESH_BINARY_INV  cv2.THRESH_OTSU)[1] contours = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)

How to add new cookies for xmlrpc.client in Python 3?
I want to add a cookie name test=xxxxxxxxxxxxxx; to the xmlrpcwordpress request.
Is there any way to do that? I found How to retain cookies for xmlrpc.client in Python 3? but this is for retain cookies and I need a code to add new cookie.

Optimising a code for bar chart creation from different data frames
I am new to Python and I am learning to create charts from data frames. I have few data frames that look like this:
Company Case 1 Case 2 Case 3 Max value Min value A 15 25 28 28 15 B 20 26 21 26 21 C 18 22 10 22 22 Company Case 1 Case 2 Case 3 Max value Min value A 15 28 26 28 15 B 12 26 11 26 11 C 12 25 10 12 25
Where three first columns are just some values and the last two columns correspond to max and mix values for each company. I have created two codes for a bar chart for each data frame. Codes are the following:
graph_A = df1[['Case 1', 'Case', 'Case 3']].plot(kind = 'bar', width=.33, legend = True, color=['green', 'blue', 'red'], alpha = .7)
plt.suptitle("Scenario 1")
graph_C.yaxis.set_major_formatter(mpl.ticker.StrMethodFormatter('{x:,.0f}'))
plt.legend(frameon = False)
plt.xticks(rotation = 0)
plt.xlabel("Company name")
plt.ylabel("Value")
graph_B = df2[['Case 1', 'Case', 'Case 3']].plot(kind = 'bar', width=.33, legend = True, color=['green', 'blue', 'red'], alpha = .7)
plt.suptitle("Scenario 2")
graph_C.yaxis.set_major_formatter(mpl.ticker.StrMethodFormatter('{x:,.0f}'))
plt.legend(frameon = False
plt.xticks(rotation = 0)
plt.xlabel("Company name")
plt.ylabel("Value")
My question is how could I optimise my code? For example, imagine I had 100 similar looking tables and repeating the same code just with few changes would make no sense. Could I use a loop in this case or something?
Thank you.

How to reform the quadratic program objective function to fit matlab quadprog?
I have a quadratic problem question with the objective function
f=arg min(A*fb)^T*S*(A*fb)+alpha*f^T*W*f s.t. d_low < C*f < d_up
where
f
is the optimization variable,S
andW
are positivedefine weight matrices.A*fb
is a matrix functionA*f=b
my question is how to reform the quadratic objective function to fit the matlab solver
quadprog
, with the general formmin 0.5*x^T*H*x+f^T*x
could you please give me a tip or example, thanks.
/==========================================/
I asked a classmate, he told me the item
(A*fb)^T*S*(A*fb)
could be expanded as(A*fb)^T*S*(A*fb)=(f^T*A^Tb^T)(S*A*fS*b)=f^T*A^T*S*A*ff^T*A^T*S*bb^T*S*A*f+b^T*S*b=f^T*A^T*S*A*f2*b^T*S*A*f+b^T*S*b
is it right?

How does adding constraints to an Objective Function affects the optimization perfomance?
I'm currently modelling a large scale timetabling problem in ILOG Cplex O.S. I've been working with soft constraints added to the objective function, and it is noticeable, the process performance gets severely affected by the amount of soft constraints that I add parallel to the objective function.
I'd like to understand why does that happen and if there is some smarter way of dealing with this kind of situation.
Since it's a timetabling problem, that are situations that are allowed to happen at a certain level, causing penalties to the objective when they do. Adding constraints to the objective function is the only way to model a situation like this?

scipy.stats  get a lognormal's underlying mu and sigma
In Python's scipy.stats library, it has a very stylized set of random variable classes, methods, and attributes. Most relate to the distribution itself, e.g., "what is the mean?" or "what is the variance?"
The lognormal distribution is an oddball because the parameters that define it are not the usual parameters for the distribution, but the parameters for a normal distribution that it derives from. Simply put, if X is a normal distribution with mean mu and stdev sigma, Y=e^X is a lognormal that has its own means, mode, variance, stdev, etc.
Does anyone know of a quick or clever way to recover the underlying mu and sigma (of the normal distribution X) via methods or attributes of a 'frozen' RV in scipy.stats?
Since scipy.stats does give the mean, stdev, etc., of the actual lognormal, one could do a lot of algebra and recover it from the standard translations...but the code is likely unmaintainable.
For reference< see: https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.lognorm.html and https://en.wikipedia.org/wiki/Lognormal_distribution

Shape Error in Andrew NG Logistic Regression using Scipy.opt
I've been trying to write Andrew NG's Logistic Regression Problem Using python and Scipy.opt for optimizing the function. However, I get a VALUE ERROR that says I have mismatching dimensions. I've tried to flatten() my theta array as scipy.opt doesn't seem to work very well with single column/row vector, however the problem still persists.
Kindly point me in the right direction as to what is causing the problem and how to avoid it.
Thanks a million!
import pandas as pd import numpy as np import matplotlib.pyplot as plt import scipy.optimize as opt dataset = pd.read_csv("Students Exam Dataset.txt", names=["Exam 1", "Exam 2", "Admitted"]) print(dataset.head()) positive = dataset[dataset["Admitted"] == 1] negative = dataset[dataset["Admitted"] == 0] #Visualizing Dataset plt.scatter(positive["Exam 1"], positive["Exam 2"], color="blue", marker="o", label="Admitted") plt.scatter(negative["Exam 1"], negative["Exam 2"], color="red", marker="x", label="Not Admitted") plt.xlabel("Exam 1 Score") plt.ylabel("Exam 2 Score") plt.title("Admission Graph") plt.legend() #plt.show() #Preprocessing Data dataset.insert(0, "x0", 1) col = len(dataset.columns) x = dataset.iloc[:,0:col1].values y = dataset.iloc[:,col1:col].values b = np.zeros([1,col1]) m = len(y) print(f"X Shape: {x.shape} Y Shape: {y.shape} B Shape: {b.shape}") #Defining Functions def hypothesis(x, y, b): h = 1 / (1+np.exp(x @ b.T)) return h def cost(x, y, b): first = (y.T @ np.log(hypothesis(x, y, b))) second = (1y).T @ np.log(1  hypothesis(x, y, b)) j = (1/m) * np.sum(first+second) return j def gradient(x, y, b): grad_step = ((hypothesis(x, y, b)  y) @ x.T) / m return b #Output initial_cost = cost(x, y, b) print(f"\nInitial Cost = {initial_cost}") final_cost = opt.fmin_tnc(func=cost, x0=b.flatten() , fprime=gradient, args=(x,y)) print(f"Final Cost = {final_cost} \nTheta = {b}")
Dataset Used: ex2.txt
34.62365962451697,78.0246928153624,0 30.28671076822607,43.89499752400101,0 35.84740876993872,72.90219802708364,0 60.18259938620976,86.30855209546826,1 79.0327360507101,75.3443764369103,1 45.08327747668339,56.3163717815305,0 61.10666453684766,96.51142588489624,1 75.02474556738889,46.55401354116538,1 76.09878670226257,87.42056971926803,1 84.43281996120035,43.53339331072109,1 95.86155507093572,38.22527805795094,0 75.01365838958247,30.60326323428011,0 82.30705337399482,76.48196330235604,1 69.36458875970939,97.71869196188608,1 39.53833914367223,76.03681085115882,0 53.9710521485623,89.20735013750205,1 69.07014406283025,52.74046973016765,1 67.94685547711617,46.67857410673128,0 70.66150955499435,92.92713789364831,1 76.97878372747498,47.57596364975532,1 67.37202754570876,42.83843832029179,0 89.67677575072079,65.79936592745237,1 50.534788289883,48.85581152764205,0 34.21206097786789,44.20952859866288,0 77.9240914545704,68.9723599933059,1 62.27101367004632,69.95445795447587,1 80.1901807509566,44.82162893218353,1 93.114388797442,38.80067033713209,0 61.83020602312595,50.25610789244621,0 38.78580379679423,64.99568095539578,0 61.379289447425,72.80788731317097,1 85.40451939411645,57.05198397627122,1 52.10797973193984,63.12762376881715,0 52.04540476831827,69.43286012045222,1 40.23689373545111,71.16774802184875,0 54.63510555424817,52.21388588061123,0 33.91550010906887,98.86943574220611,0 64.17698887494485,80.90806058670817,1 74.78925295941542,41.57341522824434,0 34.1836400264419,75.2377203360134,0 83.90239366249155,56.30804621605327,1 51.54772026906181,46.85629026349976,0 94.44336776917852,65.56892160559052,1 82.36875375713919,40.61825515970618,0 51.04775177128865,45.82270145776001,0 62.22267576120188,52.06099194836679,0 77.19303492601364,70.45820000180959,1 97.77159928000232,86.7278223300282,1 62.07306379667647,96.76882412413983,1 91.56497449807442,88.69629254546599,1 79.94481794066932,74.16311935043758,1 99.2725269292572,60.99903099844988,1 90.54671411399852,43.39060180650027,1 34.52451385320009,60.39634245837173,0 50.2864961189907,49.80453881323059,0 49.58667721632031,59.80895099453265,0 97.64563396007767,68.86157272420604,1 32.57720016809309,95.59854761387875,0 74.24869136721598,69.82457122657193,1 71.79646205863379,78.45356224515052,1 75.3956114656803,85.75993667331619,1 35.28611281526193,47.02051394723416,0 56.25381749711624,39.26147251058019,0 30.05882244669796,49.59297386723685,0 44.66826172480893,66.45008614558913,0 66.56089447242954,41.09209807936973,0 40.45755098375164,97.53518548909936,1 49.07256321908844,51.88321182073966,0 80.27957401466998,92.11606081344084,1 66.74671856944039,60.99139402740988,1 32.72283304060323,43.30717306430063,0 64.0393204150601,78.03168802018232,1 72.34649422579923,96.22759296761404,1 60.45788573918959,73.09499809758037,1 58.84095621726802,75.85844831279042,1 99.82785779692128,72.36925193383885,1 47.26426910848174,88.47586499559782,1 50.45815980285988,75.80985952982456,1 60.45555629271532,42.50840943572217,0 82.22666157785568,42.71987853716458,0 88.9138964166533,69.80378889835472,1 94.83450672430196,45.69430680250754,1 67.31925746917527,66.58935317747915,1 57.23870631569862,59.51428198012956,1 80.36675600171273,90.96014789746954,1 68.46852178591112,85.59430710452014,1 42.0754545384731,78.84478600148043,0 75.47770200533905,90.42453899753964,1 78.63542434898018,96.64742716885644,1 52.34800398794107,60.76950525602592,0 94.09433112516793,77.15910509073893,1 90.44855097096364,87.50879176484702,1 55.48216114069585,35.57070347228866,0 74.49269241843041,84.84513684930135,1 89.84580670720979,45.35828361091658,1 83.48916274498238,48.38028579728175,1 42.2617008099817,87.10385094025457,1 99.31500880510394,68.77540947206617,1 55.34001756003703,64.9319380069486,1 74.77589300092767,89.52981289513276,1

Why is optimize curve fit in scipy creating a straight line for this non linear model?
I'm trying to fit some data that i have to a model that is inversely proportional to x to some power. I've looked through a few similar threads and I think the problem might have to do with scaling my data but I've experimented with that and had no success unfortunately.
import numpy as np import scipy.optimize as optimize import matplotlib.pyplot as plt x = np.array([367.18,411.42,443.76,565.22,689.31,778.94,867.38,963.98,1085.79,1112.01,1212.47,1299.21,1408.08,1458.8,1528.76]) y = np.array([17.21,13.38,9.41,11.40,6.40,6.62,6.11,5.50,5.03,4.52,2.34,3.62,3.84,5.97,1.97]) def f(a,n,x1): return (a*np.power(x1,n)); popt, pcov = optimize.curve_fit(f,x,y, diag=(1./x.mean(),1./y.mean())) x1 = np.linspace(0,1700,num =1700) plt.plot(x1, f(x1, *popt)) plt.plot(x,y,'ro') plt.show() print(popt, pcov)
The result is this: Plot

Different random weight initilization leading to different performances
I'm training a 3D UNet on an EM dataset of a brain. The objective is to segment neurons in it. During the experiments, I've noticed, different random initialization of the network leads to different performances. I evaluate the performance based on mean Intersection over Union, and I observe differences as large as 5%.
I use xavier initilization with a uniform distribution and use a constant learning rate of 1e4.
Edit: I'm asking how I can avoid a behavior like this?

Fuzzy QLearning convergance
I am having some troubles with FQL (Fuzzy QLearning). I have implemented my algorithm on Matlab. It is actually quite simple to code FQL. One just needs to go over the different steps (layers) from fuzzification to defuzzification then update.
However, my FQL code is not converging. In fact, I am replicating a work done by others who got a nice convergence.
I will summarize my model and the steps I did in the following:
 My state vector contains 4 values s(C,B,R,P)
 I map the above vector s into its corresponding fuzzy vector sbar
 I have 4 linguistic variables each is defined over a set as follows: C={Off,idle,LM,H}, B={VL,L,M,H,VH}, R={L,M,H} and P={L,M,H}.
 I calculate the truth values of every combination (node). Here I have 4x5x3x3=180 nodes which is nothing but the product of the four membership values for each node.
 For each node, I choose an action (based on the epsilongreedy algorithm). Then I compute the weighted action and weighted qvalue for each node.
 Defuzzification: I calculate the action A by summing over the weighted action. Similar to the Qvalue.
 I update my qvalues.
In all the reference I have read, these are the natural steps taken. And it has been proven that the system will converge. However, mine is not.
I understood that in computing the future qvalue, I have to first take the action A calculated from the defuzzification step above. Then use it to move to a new state. Afterward, I repeat and find the weights of the fuzzy nodes. Then for each node, I find the optimal qvalue to finally get the next state Qvalue.
Is there something I am missing? I couldn't think of a reason why it should not converge.
I can further expand the eplanation if needed.
Many thanks in advance.

Convergence of the Qlearning on the inverted pendulum
Hello I'm working on a total control of the cartpole problem (inverted pendulum). My aim is for the system to reach stability meaning all the states(x, xdot,theta and theta) should converge to zero. I am using qlearning with a reward function as defined below.
Q_table[pre_s + (a,)] += alpha * (R + gamma *(argmax(Q_table[s]))  Q_table[pre_s + (a,)]) R=1000*cos(theta)1000*(theta_dot**2)100*(x_dot**2)100*(x**2)
unfortunately, there is no convergence. By the qtable graph, I can see it increasing and stabilising at the maximum value, but the states just stay within a certain bound and do not go to zero. I feel like my agent is not learning fast enough and at some point i not learning anymore. Can anyone help me.