Image  CNN clarification
I want to clarify a few points in Convulutional Neural Network, I am implementing image recognition in CNN using Keras.
1  Reducing size of images can help the model to learn faster. I read some blogs, in which they reduce the image size from (150,150) to (32,32) .
2  Increasing no. of layers and nodes can help increases the accuracy or not. I started to train the model at one CNN layer. But the accuracy is low. Then I add two more CNN layer, it gives high accuracy about 74. After that, I add one more layer it gives accuracy in the same range of 70s.
3  Is there a way to look images after every layer in CNN using keras. It will help to study the image in CNN.
Thank you
2 answers

Let me attempt to answer your queries.
Reducing size of the image size helps model learn faster and reduce the memory requirement. 150 X 150 pixels require more number of nodes in a single layer of neural network and thereby more memory. I am not sure about the accuracy stats of Squashing versus Center Cropping.
Increasing number of nodes may not directly relate to improvement in accuracy. Bagging and Boosting can improve accuracy way better than increasing number of layers in many scenarios. Drop out is a problem in deep neural networks but with proper data normalization along with proper activation functions and sufficient training data should help improve accuracy with multiple layers.
Short answer is yes. You can do it by reshaping the feature vectors to the original image shape at the end of each layer for which you have to define your own neural network.

Here are my inputs regarding your questions:
1) In an ideal world, you should be able to take a photo/video of anything with a camera as big as it fits your needs. However, this approach is not practical, although can result in a quite good performance. Actually, the more features you provide sans noise the more accurate is your neural net. However, where will you use such a network with such a big camera as its input generating machine? For all practical purposes you use a camera with a lens of a size not bigger than a tennis ball. At the same time, it is extremely expensive to train images with sizes bigger than 64x64 pixels. You can hardly afford a batch of 2 with a single GPU with any descent neural net. At the same time, this will increase the training time and you may end up waiting days for it to finish.
2) The more layers do not produce better results and there are other ways to achieve that. One problem with this is that the more layers you have you face the exploding or vanishing gradient problem. Also make sure you are regularizing your data, providing enough of training data and that its distribution is similar to the valid/test set distribution. So as you can see there are many aspects to the accuracy of a neural net and these were just some.
3) Yes you can do it. It will take some time to write it thoroughly so I am leaving you a link from medium. It is more detailed and has what you need:
See also questions close to this topic

How to get a user's avatar with their id in discord.py?
I tried using the following code but it didn't work.
@bot.command() async def avatar(ctx,*, avamember): user = bot.get_user(avamember) await ctx.send(f"{user.avatar_url}")

Deep sort list of dict of list (deep nested) with duplicate keys in dict
I am trying to deep sort a list of dict of list with duplicate keys (can go recursively nested) without a specific key to sort on.
Input:
{"payload": [ { "a": { "aa": [ { "aa12": { "aaa23": 230, "aaa21": 210 } }, { "aa11": { "aaa12": 120, "aaa11": 110 } }, { "aa13": { "aaa35": 350, "aaa32": 320 } } ], "aa": [ { "aa12": { "aaa22": 22, "aaa21": 21 } }, { "aa10": { "aaa03": 3, "aaa01": 1 } }, { "aa13": { "aaa33": 33, "aaa32": 32 } }, { "aa1": "aab" } ], "ac": [ "ac3", "ac1", "ac2" ] } }, { "b": { "bb": [ "bb4", "bb2", "bb3", "bb1" ] } } ]}
Expected Output:
{"payload": [ { "a": { "aa": [ { "aa1": "aab" }, { "aa10": { "aaa01": 1, "aaa03": 3 } }, { "aa12": { "aaa21": 21, "aaa22": 22 } }, { "aa13": { "aaa32": 32, "aaa33": 33 } } ], "aa": [ { "aa11": { "aaa11": 110, "aaa12": 120 } }, { "aa12": { "aaa21": 210, "aaa23": 230 } }, { "aa13": { "aaa32": 320, "aaa35": 350 } } ], "ac": [ "ac1", "ac2", "ac3" ] } }, { "b": { "bb": [ "bb1", "bb2", "bb3", "bb4" ] } } ]}
I have tried using the below recursive method:
ls = {'payload': [{'a': {'aa': [{'aa12': {'aaa23': 230, 'aaa21': 210}}, {'aa11': {'aaa12': 120, 'aaa11': 110}}, {'aa13': {'aaa35': 350, 'aaa32': 320}}], 'ac': ['ac3', 'ac1', 'ac2'], 'aa': [{'aa12': {'aaa22': 22, 'aaa21': 21}}, {'aa10': {'aaa03': 3, 'aaa01': 1}}, {'aa13': {'aaa33': 33, 'aaa32': 32}, {"aa1": "aab"}}]}}, {'b': {'bb': ['bb4', 'bb2', 'bb3', 'bb1']}}]} output = sorted_deep(ls) print(output) def sorted_deep(d): if isinstance(d,list): return sorted(sorted_deep(v) for v in d) if isinstance(d,dict): return {k: sorted_deep(d[k]) for k in sorted(d)} return d
But this ain't working. It overrides the duplicate key's value with the last found value when it sorts. Since the duplicate key can be any string, we can't iterate by specifying the key name. I'm looking for a generic solution which sorts any given complex list of dicts with nested list/dicts.
My end goal is to deep match 2 such JSON's to find the differences.

Find the most recent data available and the year it corresponds to
I have a data set containing an index for different years and countries. It's a csv and I read it using Pandas.
There are a lot of missing values for the index, so I want to create new columns with the most recent data available and the year it corresponds to. I have no idea how to code that in Python.

Tensorflow, tf.multinomial, get the associated probabilities failed
I am trying to using
tf.multinomial
to sample, and I want to get the associated probability value of the sampled values. Here is my example code,In [1]: import tensorflow as tf In [2]: tf.enable_eager_execution() In [3]: probs = tf.constant([[0.5, 0.2, 0.1, 0.2], [0.6, 0.1, 0.1, 0.1]], dtype=tf.float32) In [4]: idx = tf.multinomial(probs, 1) In [5]: idx # print the indices Out[5]: <tf.Tensor: id=43, shape=(2, 1), dtype=int64, numpy= array([[3], [2]], dtype=int64)> In [6]: probs[tf.range(probs.get_shape()[0], tf.squeeze(idx)]  TypeError Traceback (most recent call last) <ipythoninput856ef51f84ca2> in <module> > 1 probs[tf.range(probs.get_shape()[0]), tf.squeeze(idx)] C:\ProgramData\Anaconda3\lib\sitepackages\tensorflow\python\ops\array_ops.py in _slice_helper(tensor, slice_spec, var) 616 new_axis_mask = (1 << index) 617 else: > 618 _check_index(s) 619 begin.append(s) 620 end.append(s + 1) C:\ProgramData\Anaconda3\lib\sitepackages\tensorflow\python\ops\array_ops.py in _check_index(idx) 514 # TODO(slebedev): IndexError seems more appropriate here, but it 515 # will break `_slice_helper` contract. > 516 raise TypeError(_SLICE_TYPE_ERROR + ", got {!r}".format(idx)) 517 518 TypeError: Only integers, slices (`:`), ellipsis (`...`), tf.newaxis (`None`) and scalar tf.int32/tf.int64 tensors are valid indices, got <tf.Tensor: id=7, shape=(2,), dtype=int32, numpy=array([3, 2])>
The expected result I want is
[0.2, 0.1]
as indicated byidx
.But in Numpy, this method works as answered in https://stackoverflow.com/a/23435869/5046896
How can I fix it?

Tensorflow  How to predict image using .pb file in Tensorflow.JS
Is there any way how to predict .pb file that generated by my python script (https://mnurdin.com/easyimageclassificationwithtensorflow) using Tensorflow.JS? I would like to use my .pb model deploy in my ionic app (attendance system) and predict the faces. But what I heard I need to convert the .pb file into .tflite file first then deploy.
Any idea so far? Please advise.

How to initialise a trainable variable with value of a placeholder in tensorflow V1?
I want to initialise a trainable variable with value of a placeholder in tensorflow V1, tried several times but it seems that the variable does not update itself in the training.
self.a = tf.get_variable("a", initializer=tf.zeros_like(self.a_static)) self.a = tf.assign(self.a, self.a_static)
while
self.a_static
is a placeholder. In this caseself.a
seems not updating during training.When I use
self.a = tf.get_variable("a", initializer=self.a_static)
the program reports error
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'a' with dtype float and shape xxx.
but I did feed the placeholder during the training.
Are there any better ways to initialise a trainable variable with placeholder?
Many thanks in advance!

Linear Regression does not yeild correct values data with multiple inputs
gitlab link : Script file and csv file are at https://gitlab.com/pbmbjoshi/lineargradient
I am writing a python script to solve housing prediction example.
Housing prices are calculated as 4000 Rs per square feet (say base price). Additional 1% on the additional bed rooms ( more than one ), additional 3% on each floor rise ( above first floor ), discount of 2% on age (in years) of the building and 6% discount on distance ( in kilo meters ) from the nearest railway station
When I apply linear gradient on the data, I get ONLY the first parameter value (base price) near to correct one, but other parameters are totally incorrect. Can any one help me.
def linear_regression_with_multiple_variables(): print("linear_regression_with_multiple_variables") filePath = r'LinearGradient_Example.csv' data = pd.read_csv(filePath) x1 = np.asarray(data['Total Area']) x2 = np.asarray(data['# of bed rooms']) x3 = np.asarray(data['Floor Count']) x4 = np.asarray(data['Age Of The Building']) x5 = np.asarray(data['Distance from nearest railway station']) y = np.asarray(data['Final Price']) x1_max = max(x1); x2_max = max(x2); x3_max = max(x3); x4_max = max(x4); x5_max = max(x5); x1 = x1 / x1_max; x2 = x2 / x2_max; x3 = x3 / x3_max; x4 = x4 / x4_max; x5 = x5 / x5_max; theta_1 = 0 theta_2 = 0 theta_3 = 0 theta_4 = 0 theta_5 = 0 number_of_training_sets = len(x1) max_number_of_iterations = 1000000 learning_rate = 0.001 iteration_counter = 1; precision = 0.001 while iteration_counter <= max_number_of_iterations: old_theta_1 = theta_1 old_theta_2 = theta_2 old_theta_3 = theta_3 old_theta_4 = theta_4 old_theta_5 = theta_5 der_x1 = 0; der_x2 = 0; der_x3 = 0; der_x4 = 0; der_x5 = 0; for i in range(0, number_of_training_sets): diff_val = theta_1 * x1[i] + theta_2 * x2[i] + theta_3 * x3[i] + theta_4 * x4[i] + theta_5 * x5[i]  y[i] der_x1 = der_x1 + diff_val * x1[i] der_x2 = der_x2 + diff_val * x2[i] der_x3 = der_x3 + diff_val * x3[i] der_x4 = der_x4 + diff_val * x4[i] der_x5 = der_x5 + diff_val * x5[i] avg_der_x1 = der_x1 / number_of_training_sets avg_der_x2 = der_x2 / number_of_training_sets avg_der_x3 = der_x3 / number_of_training_sets avg_der_x4 = der_x4 / number_of_training_sets avg_der_x5 = der_x5 / number_of_training_sets theta_1 = theta_1  learning_rate * avg_der_x1 theta_2 = theta_2  learning_rate * avg_der_x2 theta_3 = theta_3  learning_rate * avg_der_x3 theta_4 = theta_4  learning_rate * avg_der_x4 theta_5 = theta_5  learning_rate * avg_der_x5 print(iteration_counter, theta_1, theta_2, theta_3, theta_4, theta_5) iteration_counter = iteration_counter + 1 if (abs(theta_1  old_theta_1) <= precision and abs(theta_2  old_theta_2) <= precision and abs(theta_3  old_theta_3) <= precision and abs(theta_4  old_theta_4) <= precision and abs(theta_5  old_theta_5) <= precision): break theta_1 = theta_1 / x1_max theta_2 = theta_2 / x2_max theta_3 = theta_3 / x3_max theta_4 = theta_4 / x4_max theta_5 = theta_5 / x5_max print('Final values : ', theta_1, theta_2, theta_3, theta_4, theta_5)```

Regression problem on normal distribution data
I have data that resembles students' grade which is normal distribution from 50 to 100 with mean of 80. I would like to do regression to predict this. In order to do regression analysis, because my data is imbalanced (as it has normal distribution), will it be a problem? Or it doesn't matter? Thanks!

Accuracy "jumps" wildly
I have a (triplet loss for MNIST) model. It learn poorly: a plateau in val_accuracy is around 50%. However, val_accuracy also jumps up and down (loss going down, no jumping). So it can show 40%, 60%  one time out of, say, 10. Also, one time out of 100 it can jump up to 70% and down to 25%... If I reduce learning rate, it becomes more steady, but then I will never get 70% accuracy!
My questions: is it just LR, or can anything else contribute to "jumping"? And: is there a way to make it steadily go up, instead of occasionally jumping up and then down again?

Tensor's data type is not supported in function 'blobFromTensor" in python
I tried to train from keras for my own project. However, I encounter an error that Tensor's data type is not supported in function 'bloblFromTensor' from the use of cv2.readNetFromTensorflow or cv2.readNet.
I used K.set_learing_phase(0) befor I build model
my model build is
def build(width, height, depth, classes): model = K.models.Sequential() inputShape = (height, width, depth) print(" there is the inputShape for hidden layers : {} ".format(inputShape)) # todo if u delete set_learning, the result return it that below # raise ValueError("Tensor name '{0}' is invalid.".format(node.input[0])) # dk.set_learning_phase(0) # first hidden layers model.add(K.layers.Dense(16, input_shape=inputShape)) model.add(K.layers.BatchNormalization()) # first set of convv => relu => pool layers model.add(K.layers.Conv2D(32, (7, 7), padding="same", activation="relu")) model.add(K.layers.MaxPooling2D(pool_size=(2,2))) model.add(K.layers.BatchNormalization()) model.add(K.layers.Dropout(0.25)) # init dense layer for image resolution model.add(K.layers.Dense(64, activation="relu")) model.add(K.layers.Conv2D(64, (3, 3), padding="same", activation="relu")) model.add(K.layers.MaxPooling2D(pool_size=(2,2))) model.add(K.layers.BatchNormalization()) model.add(K.layers.Dropout(0.25)) model.add(K.layers.Conv2D(128, (3, 3), activation="relu", padding="same")) model.add(K.layers.MaxPooling2D(pool_size=(2,2))) model.add(K.layers.BatchNormalization()) model.add(K.layers.Dropout(0.25)) # (None, 7, 7, 50) > model.output_shape a, b, c, d = model.output_shape print("model ouput_shape giving ", a, b, c, d) a = b * c * d print("a > ", a) model.add(K.layers.Permute([1, 2, 3])) model.add(K.layers.Reshape((a,))) # first set of fc => relu layers model.add(K.layers.Dense(units=512, activation='relu')) model.add(K.layers.BatchNormalization()) model.add(K.layers.Dropout(0.5)) # model.add(K.layers.Flatten()) # sofmax classifier model.add(K.layers.Dense(units=classes, activation='softmax')) return model
So, I used fit_generation method for history of training then applied
# save file of model model.save(h5_file) # freeze the graph to pb file and pbtxt # ! not showing to console because that so take a long time frozen_graph = freeze_session(sess, output_names=[out.op.name for out in model.outputs]) # convert to keras file to pb files for using readNetTensorflow try: tf.io.write_graph(frozen_graph, learn_path, model_file , as_text=False) log_file.register(0,"graph file saving to pb is success ") except ValueError as er: os.system('clear') console_log.d("ERROR > NOT CONVERT TO PB FILE ",er) log_file.register(1, " ERROR that convert to pbfile : {} ".format(er)) console_log.d("FROZEN GRAPH TO PB FILE ") # pb file convert to pbtxt file create_pbtxt(model_file , pbtxt_file, learn_path )
finally, I converted pb file to pbtxt file with some node deleted
def create_pbtxt(pb_path, pbtxt_path, path): # clear console os.system("clear") console_log.d("CREATE_TXT FILE") console_log.d("there is the path file is this   > " , pb_path ) # check path file from the news found_path_file = path + "/" + pb_path console_log.d("original path that saving here:",found_path_file) with tf.Session() as sess: with tf.gfile.GFile(found_path_file, 'rb')as f: # tf.io.compat graph_def = tf.GraphDef() # graph_def.ParseFromString(f.read()) try: # text_format.Merge(f.read(), graph_def) graph_def.ParseFromString(f.read()) tf.import_graph_def(graph_def, name='') console_log.d("pbtxt file is saving ") except text_format.ParseError: console_log.d("error from merge file for text_format") # inp_node = 'MobilenetV2/MobilenetV2/input' # oup_node = 'logits/semantic/BiasAdd' # gragh_def = optimize_for_inference_lib.optimğize_for_inference() name_array =[ ] op_array=[] # print("graph_def_node is here: ", graph_def.node) # for i in reversed((range(len(graph_def.node) ))): for i in reversed(range(len(graph_def.node))): # if op == 'Switch' or op == 'Merge': # inp = graph_deadnode[i].input[0] # print("inp : {}".format(inp)) # graph_def node for addConstNodes error that log_file.register(0, " file is node for reversed : {} ".format(graph_def.node[i].op)) # cv2 test file return addConstNode error from Const node exist (maybe) if graph_def.node[i].op =='Const': console_log.d("found const variables!!!") # console clear os.system('clear') del graph_def.node[i] # if graph_def.op == 'Const': # os.system("clear") # console_log.d("node deleting Cons ") # console_log.d("graph_def.node[i]: ".format(graph_def.node[i]) ) # del graph_def.node[i] # ! extends adding for any error that return cv2 test file for attr in ['T', 'data_format', 'Tshape', 'N', 'Tidx', 'Tdim','use_cudnn_on_gpu', 'Index', 'Tperm', 'is_training','Tpaddings']: if attr in graph_def.node[i].attr: console_log.d(" THERE IS THE ATTR ") del graph_def.node[i].attr[attr] op = graph_def.node[i].op op_array.append(op) print("op: {}".format(op)) name = graph_def.node[i].name name_array.append(name) print("name : {} ".format(name)) # todo , try to op and name from nodes to logs file # convert to np array for values save to logs file op2_array = np.array(op_array) name2_array = np.array(name_array) # send values for logs log_file.register(0," op : {} , name:{} ".format(op2_array, name2_array)) # saving pbtxt file from pb file that convert tf.io.write_graph( graph_def, path, pbtxt_path, as_text=True)
This issue from testing.py. when debug cursor comes the cv2.readNetTensorflow or cv2.readNet, program returns " Tensor's data type is not supported in function 'blobFromTensor "
this is the testing.py
# check the pbtxt with or without processing if flags.text_file is None: net = cv2.dnn.readNet(pb_file) else: txt_file = os.getcwd() + "/" + flags.text_file print(txt_file) net = cv2.dnn.readNetFromTensorflow(pb_file,txt_file)
Why do I get the error? I have no idea. Please help me.

Is there a GitHub code for VQVAE 2 with Keras?
I am trying to find an opensource implementation of the "Generating Diverse HighFidelity Images with VQVAE2" paper to train it to generate RGB images, I see bunch of examples with Pytorch but none with Keras or Tensorflow2.0. Google papers usually have an implementation that comes with them but I can't seem to find the implementation for this one.

Why the accuracy of my RNN is different when facing two similar distribution?
everyone.
We are doing a project: to learn the distribution underlying datas produced by experiments using recurrent neural network (RNN).
However, if we generate two sets of datas using different experiment methods respectively, the number of samples needed to approach a same accuracy (0.99) is largely different (~3000 and ~12000).
Following are some trials we adapted already:
We plot the two distributions and observe that they are very similar (fidelity~0.982).
We repeat learning code several times, finding the phenomena does not change.
We using same number of samples (N=5000), and accuracy are 0.992 and 0.988 respectively.
Please give us any favor!!! Thanks for any potential clue!!!

Using a trained Keras model to make predictions on new csv data
so I'm making a project where basically i have to predict whether or not a house price is above or below its median price and to do that, I'm using this dataset from Kaggle(https://drive.google.com/file/d/1GfvKA0qznNVknghV4botnNxyHKvODOC/view). 1 means "Above Median" and 0 means "Below Median". I wrote this code to train a neural network and save it as a .h5 file:
import pandas as pd from sklearn import preprocessing from sklearn.model_selection import train_test_split from keras.models import Sequential from keras.layers import Dense import h5py df = pd.read_csv('housepricedata.csv') dataset = df.values X = dataset[:,0:10] Y = dataset[:,10] min_max_scaler = preprocessing.MinMaxScaler() X_scale = min_max_scaler.fit_transform(X) X_train, X_val_and_test, Y_train, Y_val_and_test = train_test_split(X_scale, Y, test_size=0.3) X_val, X_test, Y_val, Y_test = train_test_split(X_val_and_test, Y_val_and_test, test_size=0.5) model = Sequential([ Dense(32, activation='relu', input_shape=(10,)), Dense(32, activation='relu'), Dense(1, activation='sigmoid'), ]) model.compile(optimizer='sgd', loss='binary_crossentropy', metrics=['accuracy']) hist = model.fit(X_train, Y_train, batch_size=32, epochs=100, validation_data=(X_val, Y_val)) model.save("house_price.h5")
After running it, it successfully saves the .h5 file to my directory. What I want to do now is use my trained model to make predictions on a new .csv file and determine whether or not each of those are above or below median price. This is an image of the csv file in VSCode that i want it to make predictions on: csv file image As you can see, this file doesn't contain a 1(above median) or 0(below median) because that's what I want it to predict. This is the code I wrote to do that:
import pandas as pd from sklearn import preprocessing from sklearn.model_selection import train_test_split from keras.models import Sequential from keras.layers import Dense from keras.models import load_model import h5py df = pd.read_csv('data.csv') dataset = df.values X = dataset[:,0:10] Y = dataset[:,10] min_max_scaler = preprocessing.MinMaxScaler() X_scale = min_max_scaler.fit_transform(X) X_train, X_val_and_test, Y_train, Y_val_and_test = train_test_split(X_scale, Y, test_size=0.3) X_val, X_test, Y_val, Y_test = train_test_split(X_val_and_test, Y_val_and_test, test_size=0.5) model = load_model("house_price.h5") y_pred = model.predict(X_test) print(y_pred)
It's output is
[[0.00101464]]
I have no clue what that is and why it's only returning one value even though the csv file has 4 rows. Does anyone know how I can fix that and be able to predict either a 1 or a 0 for each row in the csv file? Thank You! 
Machine Learning in PreEmployment Personality Assessment
Briefly, I would like to know (From your experiences), if I can build automatedmachine learningSoftware, which is able to assess the personality of the candidates before the employment process happening (Before the real HR Interview).
Q1 from where should start?
Q2 which algorithms are the best in order to be availed?
Q3 would python be enough tool? Or we still have to call Java?
I think all these questions seem to be absolutely basic questions because I'm a beginner, and this project is actually my Ph.D. doctoral topic.
please, guys, any suggestions, criticisms would, Indeed, be HELPFUL.
Thanks in advance.