Tensorflow Predicting values from custom estimator
I've created a custom estimator using the tutorial here, and trained it. It's saved a bunch of files in the model_dir that I passed to the estimator. I'm now trying to load the created model in another file to predict values.
What I tried was to copy/paste the same model function I used to train the model into a new file, then do
variable_predictor = tf.estimator.Estimator(
model_fn=my_model_fn, model_dir="/tmp/modeltest1")
to load the model. Having done this I'm able to evaluate the model just fine, getting the same loss it reported when I evaluated it right after training. Giving it 82,283 entries (a numpy array that's 82283*10*25), I get a MSE of 5400ish for a batch size of 25.
When I try to predict with it I get nonsensical answers. I'm doing this:
predict_input_fn = tf.estimator.inputs.numpy_input_fn(
x=test_independent_variables,
num_epochs=1,
shuffle=False)
results = variable_predictor.predict(input_fn=predict_input_fn)
resultlist=[]
for result in results:
resultlist.append(result)
What I get back does not make sense. I'm feeding in 82,283 entries and I'm getting out 643 predictions that are not nearly as close to the actual values as evaluation says the model is.
82283/643 = 128ish, while as in training my batch size was 25  no relation there. What's causing it to not give back as many predictions as I ask it to make, and how can I fix this?
See also questions close to this topic

how to add non .py files into python egg
I have a flask app which looks like
myapp │ └── src │ └── python │ └── config │ └── app │── MANIFEST.in └── setup.py
The config folder is full of *.yaml files, I want to add all the static config files into my python egg after using
python setup.py install
My setup.py looks like
import os from setuptools import setup, find_packages path = os.path.dirname(os.path.abspath(__file__)) setup( name="app", version="1.0.0", author="Anna", description="", keywords=[], packages=find_packages(path + '/src/python'), package_dir={'': path + '/src/python'}, include_package_data=True )
I am trying the use the MANIFEST.in to add the config file However, it always give error
error: Error: setup script specifies an absolute path: /Users/Anna/Desktop/myapp/src/python/app setup() arguments must *always* be /separated paths relative to the setup.py directory, *never* absolute paths.
I have not used any absolute paths in my code, I've seen other posts trying to bypass this error, by removing
include_package_data=True
However, in my case, if i do this to avoid this error, all my yamls won't be added.
I was wondering if there are ways to fix this problem. Thanks

How to extract all functions and API calls used in a Python source code?
Let us consider the following Python source code;
def package_data(pkg, roots): data = [] for root in roots: for dirname, _, files in os.walk(os.path.join(pkg, root)): for fname in files: data.append(os.path.relpath(os.path.join(dirname, fname), pkg)) return {pkg: data}
From this source code, I want to extract all the functions and API calls. I found a similar question and solution. I ran the solution given here and it generates the output
[os.walk, data.append]
. But I am looking for the following output[os.walk, os.path.join, data.append, os.path.relpath, os.path.join]
.What I understood after analyzing the following solution code, this can visit the every node before the first bracket and drop rest of the things.
import ast class CallCollector(ast.NodeVisitor): def __init__(self): self.calls = [] self.current = None def visit_Call(self, node): # new call, trace the function expression self.current = '' self.visit(node.func) self.calls.append(self.current) self.current = None def generic_visit(self, node): if self.current is not None: print("warning: {} node in function expression not supported".format( node.__class__.__name__)) super(CallCollector, self).generic_visit(node) # record the func expression def visit_Name(self, node): if self.current is None: return self.current += node.id def visit_Attribute(self, node): if self.current is None: self.generic_visit(node) self.visit(node.value) self.current += '.' + node.attr tree = ast.parse(yoursource) cc = CallCollector() cc.visit(tree) print(cc.calls)
Can anyone please help me to modified this code so that this code can traverse the API calls inside the bracket?
N.B: This can be done using regex in python. But it requires a lot of manual labors to find out the appropriate API calls. So, I am looking something with help of Abstract Syntax Tree.

use correct version of 'pip' installed for your Python interpreter
I am using pycharm , i am having this error by adding any package , Click Here
i have tried lot of methods , but didn't succeed yet.
Info :
 python 3.6.2
 pip 10.0.1
 VirtualEnv

Jupyter does not show folder in my working directory
I am running docker Jupiter notebook on a MacBook Pro. When starting Jupyter home only shows some of the folders in the working directory. When I cd to a folder and use it as the working directory I get the message "Notebook list is empty." See examples below.
My directory:
LewIssMacBookPro:MyTensorFlow lewleib$ ls
Gorner_tensorflowmnist Tensor2018 models Gorner_tensorflowrnn Untitled.ipynb tensorflow MyDeepTest generate_hmb3.py tensorflowwithoutaphdmaster My_tensor1.html guided testgen NeuralNet1.ipynb install.sh README.md mnist
One level down:
LewIssMacBookPro:MyDeepTest lewleib$ ls README.md guided models generate_hmb3.py install.sh testgen
And one level more:
LewIssMacBookPro:guided lewleib$ ls
chauffeur_guided.py epoch_guided.py ncoverage.py rambo_guided.py
When I try and call Jupiter note book:
LewIssMacBookPro:guided lewleib$ docker run it p 8888:8888 p 6006:6006 v ~/lewleib/MyTensorFlow/MyDeepTest/guided:/notebooks tensorflow/tensorflow
I get the following:
guided Last Modified Name ..seconds ago The notebook list is empty.

TensorFlow 1.3 ROCm port: cannot open '_pywrap_tensorflow_internal'
In Ubuntu 16.04.4, I installed the TensorFlow 1.3 ROCm port (for an AMD Radeon RX Vega 64) according to the instructions starting at "Install required python packages" in
where I had previously installed ROCm from the AMD Debian repository according to the instructions in
https://github.com/RadeonOpenCompute/ROCm
Then, using pip to install the TF .whl package with no virtualization:
$ wget http://repo.radeon.com/rocm/misc/tensorflow/tensorflow1.3.0cp27cp27mumanylinux1_x86_64.whl $ sudo python m pip install tensorflow1.3.0cp27cp27mumanylinux1_x86_64.whl
When I try to verify the installation using
$ python c "import tensorflow as tf; print(tf.__version__)"
I get the following error:
Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib/python2.7/distpackages/tensorflow/__init__.py", line 24, in <module> from tensorflow.python import * File "/usr/local/lib/python2.7/distpackages/tensorflow/python/__init__.py", line 49, in <module> from tensorflow.python import pywrap_tensorflow File "/usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow.py", line 52, in <module> raise ImportError(msg) ImportError: Traceback (most recent call last): File "/usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow.py", line 41, in <module> from tensorflow.python.pywrap_tensorflow_internal import * File "/usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow_internal.py", line 28, in <module> _pywrap_tensorflow_internal = swig_import_helper() File "/usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow_internal.py", line 24, in swig_import_helper _mod = imp.load_module('_pywrap_tensorflow_internal', fp, pathname, description) ImportError: libCXLActivityLogger.so: cannot open shared object file: No such file or directory
I verified that _pywrap_tensorflow_internal.so is present:
$ find / name \*pywrap\* ls 2>/dev/null 27526810 4 rwrr 1 root staff 2558 Jul 20 11:41 /usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow.py 27526811 4 rwrr 1 root staff 1312 Jul 20 11:41 /usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow.pyc 27526813 92 rwrr 1 root staff 93912 Jul 20 11:41 /usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow_internal.pyc 27526815 227172 rwxrxrx 1 root staff 232620600 Jul 20 11:41 /usr/local/lib/python2.7/distpackages/tensorflow/python/_pywrap_tensorflow_internal.so 27526816 72 rwrr 1 root staff 70386 Jul 20 11:41 /usr/local/lib/python2.7/distpackages/tensorflow/python/pywrap_tensorflow_internal.py
Also checked my wheel and pip versions:
$ pip list  grep wheel wheel 0.29.0 $ pip V pip 10.0.1 from  python2.7/sitepackages/pip (python 2.7)
At first glance it looks as if some environmental variable is not set, so that _pywrap_tensorflow_internal.so is not being searched for on the correct path. Can anyone tell me if this is the case  or if the source of the problem is elsewhere? I did some searches and have come up essentially empty. Thanks in advance for any helpful responses.

Calculate the gradients of the last state with respect to the initial state with GRU and understanding gradient tensor sizes in Tensorflow
I have the following model in tensorflow:
def output_layer(input_layer, num_labels): ''' :param input_layer: 2D tensor :param num_labels: int. How many output labels in total? (10 for cifar10 and 100 for cifar100) :return: output layer Y = WX + B ''' input_dim = input_layer.get_shape().as_list()[1] fc_w = create_variables(name='fc_weights', shape=[input_dim, num_labels], initializer=tf.uniform_unit_scaling_initializer(factor=1.0)) fc_b = create_variables(name='fc_bias', shape=[num_labels], initializer=tf.zeros_initializer()) fc_h = tf.matmul(input_layer, fc_w) + fc_b return fc_h def model(input_features): with tf.variable_scope("GRU_Layer1"): cell1 = tf.nn.rnn_cell.GRUCell(gru1_cell_size) # shape=(?, 64) ... gru1_cell_size=64 initial_state1 = tf.placeholder(shape=[None, gru1_cell_size], dtype=tf.float32, name="initial_state1") output1, new_state1 = tf.nn.dynamic_rnn(cell1, input_features, dtype=tf.float32, initial_state=initial_state1) with tf.variable_scope("GRU_Layer2"): cell2 = tf.nn.rnn_cell.GRUCell(gru2_cell_size) # shape=(?, 32)...gru2_cell_size=32 initial_state2 = tf.placeholder(shape=[None, gru2_cell_size], dtype=tf.float32, name="initial_state2") output2, new_state2 = tf.nn.dynamic_rnn(cell2, output1, dtype=tf.float32, initial_state=initial_state2) with tf.variable_scope("output2_reshaped"): # before, shape: (34, 100, 32), after, shape: (34 * 100, 32) output2 = tf.reshape(output2, shape=[1, gru2_cell_size]) with tf.variable_scope("output_layer"): # shape: (34 * 100, 3), num_labels=3 predictions = output_layer(output2, num_labels) predictions = tf.reshape(predictions, shape=[1, 100, 3]) return predictions, initial_state1, initial_state2, new_state1, new_state2
So as we can see from the code that the cell size of the first GRU is 64, the cell size of the second GRU is 32. And the batch size is 34 (but this is not important for me now). And the size of input features is 200. I have tried computing the gradients of the loss with respect to the trainable variables through:
local_grads_and_vars = optimizer.compute_gradients(loss, tf.trainable_variables()) # only the gradients are taken to add them later with the back propagated gradients from previous batch. local_grads = [grad for grad, var in local_grads_and_vars] for v in local_grads: print("v", v)
After printing out the grads I got the following:
v Tensor("Optimizer/gradients/GRU_Layer1/rnn/while/gru_cell/MatMul/Enter_grad/b_acc_3:0", shape=(264, 128), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer1/rnn/while/gru_cell/BiasAdd/Enter_grad/b_acc_3:0", shape=(128,), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer1/rnn/while/gru_cell/MatMul_1/Enter_grad/b_acc_3:0", shape=(264, 64), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer1/rnn/while/gru_cell/BiasAdd_1/Enter_grad/b_acc_3:0", shape=(64,), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer2/rnn/while/gru_cell/MatMul/Enter_grad/b_acc_3:0", shape=(96, 64), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer2/rnn/while/gru_cell/BiasAdd/Enter_grad/b_acc_3:0", shape=(64,), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer2/rnn/while/gru_cell/MatMul_1/Enter_grad/b_acc_3:0", shape=(96, 32), dtype=float32) v Tensor("Optimizer/gradients/GRU_Layer2/rnn/while/gru_cell/BiasAdd_1/Enter_grad/b_acc_3:0", shape=(32,), dtype=float32) v Tensor("Optimizer/gradients/output_layer/MatMul_grad/tuple/control_dependency_1:0", shape=(32, 3), dtype=float32) v Tensor("Optimizer/gradients/output_layer/add_grad/tuple/control_dependency_1:0", shape=(3,), dtype=float32)
Here is the GRU cell from "Mastering TensorFlow 1.x: Advanced machine learning and deep learning concepts using TensorFlow 1.x and Keras" book. Here is the link as well: https://play.google.com/store/books/details?id=xtRJDwAAQBAJ&rdid=bookxtRJDwAAQBAJ&rdot=1&source=gbs_vpt_read&pcampaignid=books_booksearch_viewport
So I was trying to understand the shapes of the gradients after printing out the tensors as shown in the code
local_grads
.From the GRU cell shown above I assumed that:
1 Having a grad tensor with shape
(264, 128)
is used to calculate the activation before the input tor()
andu()
. If the output ofr()
andu()
is 64, then there is a tensor of shape(128)
.2 Since the output size of the GRU is
64
, therefore, I assumed that the input to the second layer GRU will of size 64 + 32(this is the cell size of the second GRU) which gives96
. Hence, similar to point 1, the gradient tensors will have shapes of(96, 64)
and(64)
.3 Given that we have a dense layer after the second GRU layer, since the output is of size 3, then there is a gradient tensor for the corresponding weight of size
(32, 3)
and(3)
My concern is why do we have tensors of shape
(264, 64)
,(64)
and(96, 32)
,(32)
.Second, Assume that I saved the gradients after training the model on the first batch, that is, after feeding a tensor of shape:
(34, 100, 200)
asinput_features
"In the model function argument", and output of shape(34 * 100, 3)
, how to back propagate these gradients on the second minibatch?I would like to fix the gap as in the following image from https://deepmind.com/blog/decoupledneuralnetworksusingsyntheticgradients/:
Where instead of having synthetic gradients, I would like to back propagate the gradients from the previous time step. So I was trying something like:
prev_grads_val__ = tf.gradients([new_state1, new_state2], [initial_state1, initial_state2], grad_ys=previous_gradients)
but this won't work giving the following error:
ValueError: Passed 10 grad_ys for 2 ys
And then
prev_grads_val__
should be added tolocal_grads
before performing the back propagation.Any help is much appreciated!!!

How to train a machine learning algorithm to find this pattern: x1 < x2 without generating a new feature (e.g. x1x2) first?
If I had 2 features x1 and x2 where I know that the pattern is:
if x1 < x2 then class1 else class2
How can I train a machine learning algorithm to find such a pattern?
I know that I could create a third feature x3 = x1x2. Then feature x3 can easily be used by some machine learning algorithms. For example a decision tree can solve the problem 100% using x3 and just 3 nodes (1 decision and 2 leaf nodes).
But, is it possible to solve this without creating new features?
I tried MLP and SVM with different kernels, including svg kernel and the results are not great. This seems like a problem that should be easily solved 100% if a machine learning algorithm could only find such a pattern.
As an example of what I tried, here is the scikitlearn code where the SVM could only get a score of 0.992:
import numpy as np from sklearn.svm import SVC # Generate 1000 samples with 2 features with random values X_train = np.random.rand(1000,2) # Label each sample. If feature "x1" is less than feature "x2" then label as 1, otherwise label is 0. y_train = X_train[:,0] < X_train[:,1] y_train = y_train.astype(int) # convert boolean to 0 and 1 svc = SVC(kernel = "rbf", C = 0.9) # tried all kernels and C values from 0.1 to 1.0 svc.fit(X_train, y_train) print("SVC score: %f" % svc.score(X_train, y_train))
Output running the code:
SVC score: 0.992000
This is an oversimplification of my problem. The real problem may have hundreds of features and different patterns, not just x1 < x2. However, to start with it would help a lot to know how to solve for this simple pattern.

Best Machine Learning Algorithm to Rank Top 10 Items Based on Different Attributes?
I am working on a project where I want to retrieve top n selections from a dataset based on different attributes. Let's say I am looking for the best store to buy a product. The algorithm will take in location, prices, closing/opening times, and return policy, and whether they have the product. It will return the top n (let's say 10) stores out of the dataset that it has found.
I want to know what the best machine learning algorithm is for this scenario.

How to consider word pairs/phrases for Word2Vec and other preprocessing
So it's my first time using Word2Vec and Im using a wikipedia dump with WikiCorpus to preprocess the file before training my Word2Vec model. I want to use the following preprocessing techniques:
 Convert all letters to lowercase (I think WikiCorpus does this already).
 Remove all punctuation (Done by WikiCorpus).
 Consider word pairs/phrases as a single word, for example 'Big Apple' > 'big_apple', not 'big', 'apple'.
 Convert all digits to their word forms, so '3' > 'three'
At the moment I have no idea how to do the last two. I know about num2text but not sure how to incorporate with WikiCorpus or Word2vec. Can anyone help?

Tensors not in same graph for custom estimator
I'm new to tensorflow, and trying to make a ConvLSTM run using custom estimator.
I defined the model_fn as such :
def model_fn(features,labels,mode,params = None): if params==None: batch_size = 50 time_steps = 150 dim =40 else: batch_size = params['batch_size'] time_steps = params['time_steps'] dim = params['dim'] #instantiate cell net = tf.contrib.rnn.ConvLSTMCell(conv_ndims = 2,input_shape = [dim,dim,1],output_channels = 1,kernel_shape = [3,3]) state = net.zero_state(batch_size,dtype = tf.float32) features = tf.cast(features,tf.float32) if mode != tf.estimator.ModeKeys.PREDICT: # Added in order to solve tf.cast problem if there is no labels labels = tf.cast(labels,tf.float32) state = net.zero_state(batch_size,dtype = tf.float32) # < inconsistent state size between training and predict, is it problematic ? else: state = net.zero_state(1,dtype = tf.float32) inputs = tf.split(features,time_steps,axis = 1) inputs_list = [tf.squeeze(input_,[1]) for input_ in inputs] outputs = [] with tf.variable_scope("convLSTM") as scope: for i, input_ in enumerate(inputs_list): if i>0: scope.reuse_variables() t_output ,state = net(input_,state) outputs.append(t_output) outputs = tf.stack(outputs,1) rmse = tf.Variable(tf.zeros([],dtype = np.float32)) if mode == tf.estimator.ModeKeys.PREDICT: return tf.estimator.EstimatorSpec(mode,predictions=outputs) elif mode == tf.estimator.ModeKeys.TRAIN: loss = tf.losses.absolute_difference(labels,outputs) optimizer= tf.train.AdagradOptimizer (learning_rate = 0.1) train_op = optimizer.minimize(loss,global_step = tf.train.get_global_step()) rmse = tf.metrics.root_mean_squared_error(labels,outputs)[0] tf.summary.scalar('RMSE loss',rmse) return tf.estimator.EstimatorSpec(mode,loss=loss,train_op = train_op) elif mode == tf.estimator.ModeKeys.EVAL: loss = tf.losses.absolute_difference(labels,outputs) rmse = tf.metrics.root_mean_squared_error(labels,outputs)[0] tf.summary.scalar('RMSE loss',rmse) return tf.estimator.EstimatorSpec(mode,loss=loss,eval_metric_ops = {'RMSE':rmse})
The input functions:
def input_fn_train(batch_size): dataset = tf.data.TFRecordDataset(['Data/train.tfrecords']) dataset = dataset.map(parse_) dataset = dataset.shuffle(buffer_size = 5) dataset = dataset.batch(batch_size) return dataset.prefetch(buffer_size = 5) def input_fn_eval(batch_size): dataset = tf.data.TFRecordDataset(['Data/eval.tfrecords']) dataset = dataset.map(parse_) dataset = dataset.shuffle(buffer_size = 5) dataset = dataset.batch(batch_size) return dataset.make_one_shot_iterator().get_next()
Which way is better, the iterator or the dataset output ?
And for the main :
def main(): batch_size = 5 data_pred = misc.input_fn_eval(1) rnn = tf.estimator.Estimator( model_fn = model.model_fn, model_dir = "logs/20_08/", params = {'batch_size':batch_size,'time_steps':150,'dim':40}) rnn.train(input_fn =lambda : misc.input_fn_train(batch_size),steps = 1) video = rnn.predict(input_fn = lambda:data_pred) print(next(video)) if __name__ == "__main__": main()
Now, the code seems to be fine, at least in syntax, for training. I wanted to predict a few frames, in order to check the evolution, but I keep getting errors :
ValueError: Tensor("ConvLSTMCellZeroState_1/zeros_1:0", shape=(1, 40, 40, 1), dtype=float32) must be from the same graph as Tensor("Squeeze:0", shape=(?, 40, 40, 1), dtype=float32).
I also had this for Iterator and Dataset ( I believe for the prediction input function, which used to be the same as the one for the training. Creating a different one seems to have solved it.).
Thanks a lot for the help ! I hope the question is clear enough, please let me know if it isn't.

Tensorflow estimator.DNNClassifier not repeating results
Each time I run the following code I get a different 'Loss for final step' when training my model. The subsequent evaluation accuracy also changes. I have checked that the input data from train_test_split is constant.I have set the value of tf.random_seed, turned off shuffling and set the value of num_threads. I am using Tensorflow 1.8. Can anyone advise me on what else I need to do?`
from __future__ import print_function import numpy as np import pandas as pd import tensorflow as tf from sklearn.model_selection import train_test_split np.random.seed(1) tf.set_random_seed(1) df = pd.read_csv('diabetes.csv') X = df.iloc[:,0:8] y = df['Outcome'] X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, stratify=None, random_state=1) def create_feature_cols(): return [ tf.feature_column.numeric_column('Pregnancies'), tf.feature_column.numeric_column('Glucose'), tf.feature_column.numeric_column('BloodPressure'), tf.feature_column.numeric_column('SkinThickness'), tf.feature_column.numeric_column('Insulin'), tf.feature_column.numeric_column('BMI'), tf.feature_column.numeric_column('DiabetesPedigreeFunction'), tf.feature_column.numeric_column('Age') ] input_func = tf.estimator.inputs.pandas_input_fn(x=X_train,y=y_train, batch_size=10,num_epochs=1000,shuffle=False,num_threads=1) model = tf.estimator.DNNClassifier(hidden_units=[20,20], feature_columns=create_feature_cols(),n_classes=2) model.train(input_fn=input_func,steps=1000) eval_input_func = tf.estimator.inputs.pandas_input_fn( x=X_test, y=y_test, batch_size=10, num_epochs=1, shuffle=False, num_threads=1) results = model.evaluate(eval_input_func)`

Serializing a `tf.Summary` object to `string` Tensor
I try to generate image summaries to be displayed in tensorboard. This worked in an eager execution environment.
Now, I try to use the
eval_metric_ops
returning a dict of operations to compute metrics during execution of the computation graph. For this, I rely ontf.py_func
to do my metrics computations and plots. This function signature istf.py_func( func, inp, Tout, stateful=True, name=None )
Where
Tout
is the returned type of the function. I managed to make it work for simple metrics (float values). As far as I understand, I need to define astring
returned type for my summaries which will be parsed after to rebuild my images.Here is the blocking point.
I build my Summary with:
summ = tf.Summary(value=[ tf.Summary.Value( tag=metric_name, image=tf.Summary.Image( encoded_image_string=encode_image_array_as_png_str( self._last_metrics[metric_name])))])
Returning it as is, I get:
W tensorflow/core/framework/op_kernel.cc:1306] Unimplemented: Unsupported object type Summary
Returning
str(summ)
gives:WARNING:tensorflow:Skipping summary for ..., cannot parse string to Summary
.I also tried to build it with:
tf.summary.image( name, tensor, max_outputs=3, collections=None, family=None )
But this gives:
W tensorflow/core/framework/op_kernel.cc:1306] Unimplemented: Unsupported object type Tensor
Do you know how to serialize a Summary to a string/bytes iterable/whatever can be interpreted as a string Tensor, in a way that it can be parsed back to an image Summary after that.
Thanks.