Tensorflow Predicting values from custom estimator
I've created a custom estimator using the tutorial here, and trained it. It's saved a bunch of files in the model_dir that I passed to the estimator. I'm now trying to load the created model in another file to predict values.
What I tried was to copy/paste the same model function I used to train the model into a new file, then do
variable_predictor = tf.estimator.Estimator( model_fn=my_model_fn, model_dir="/tmp/modeltest1")
to load the model. Having done this I'm able to evaluate the model just fine, getting the same loss it reported when I evaluated it right after training. Giving it 82,283 entries (a numpy array that's 82283*10*25), I get a MSE of 5400ish for a batch size of 25.
When I try to predict with it I get nonsensical answers. I'm doing this:
predict_input_fn = tf.estimator.inputs.numpy_input_fn( x=test_independent_variables, num_epochs=1, shuffle=False) results = variable_predictor.predict(input_fn=predict_input_fn) resultlist= for result in results: resultlist.append(result)
What I get back does not make sense. I'm feeding in 82,283 entries and I'm getting out 643 predictions that are not nearly as close to the actual values as evaluation says the model is.
82283/643 = 128ish, while as in training my batch size was 25 - no relation there. What's causing it to not give back as many predictions as I ask it to make, and how can I fix this?
See also questions close to this topic
How to rename files using python script according to downloading time?
- I was download a playlist in which all videos are not arranged in manner.
- so i want to rename each and every file according to downloading
EXAMPLE : suppose there are three videos :
First vdo downloaded at 7 : 56 AM
Second vdo downloaded at 7 : 56 : 5 AM
Third vdo download at 7 : 57 AM
after applying the script
the file must renamed as 1, 2, 3 at the starting of the file name. (shouldn't loss the real file name but add 1, 2, 3 at the start of the file name)
Transcrypt: How to check for empty list?
This a simple python program which causes an endless loop when compiled with Transcrypt 3.7.5:
stack = [ch for ch in "Hello"] while stack: print(stack.pop())
This can be solved by replacing
while len(stack):but this would only be an unacceptable workaround. Is there any other Transcrypt-related pragma switch to get around it?
Delete duplicated files
I hope you guys can help me. I have 2 PCs, 1 iMac and 2 External HD full of pictures and files. Some of them are duplicated and I was looking for a script that I can run on Windows, Mac, and HD to delete all the duplicated files. Can any of you please suggest the best way to get this accomplished. Best regards. Manoelito
How to deal with data contained in a list of elements with different sizes to train an LSTM?
First of all, I want to apologize about my english that might not be very good since it's not my native language. Afterthat, I expose my problem :
I'm a new in Python/Keras and currently, I'm working on a sequence-to-sequence problem using LSTM. Here it is : I got a dataset of a robot arm kinematics. All kinematics have the same number of columns but different numbers of rows (rows represent time steps). I got 24 kinematic as training data and I want to feed it to an LSTM. So, I've created a list containing the 24 elements. The problem is when I want to create an LSTM, I have to specify the "input_shape" argument (according to an error that occured after running a part of code). But the thing is I don't have any idea about the value I must put on "input_shape" since my training data is a list of kinematics with different amounts of time. All I know is that I have a list with 24 elements with 76 columns each. Here is the part of code that I run :
model = Sequential() model.add(LSTM(100)) model.add(Dense(1, activation='sigmoid')) model.compile(loss='binary_crossentropy', optimizer='adam', metrics= ['accuracy']) print(model.summary())
And here is the error that occured : `
Traceback (most recent call last): File "<input>", line 4, in <module> File "C:\Users\Pctec\Desktop\venv\lib\site-packages\keras\models.py", line 454, in add raise ValueError('The first layer in a ' ValueError: The first layer in a Sequential model must get an 'input_shape' or 'batch_input_shape' argument.`
Afterthat, I've searched a solution to the problem and found this : https://machinelearningmastery.com/prepare-univariate-time-series-data-long-short-term-memory-networks/
It gave me a little idea about what I should do, however, the author came up with an example where the number of time steps is known, unlike the case I'm working into.
I don't know if I gave enough details about my problem because like I said, I'm still a beginner but maybe someone can help me with this.
So, according to what should I specify the value of input_shape for the LSTM ?
Thank you :)
Apache Beam alternative
I would like to try out TFX on Windows, and for Transform and Model Analysis I need Apache Beam that only comes for linux and macOS. Is there an alternative, TFX could work with?
Much waste RecvTensor ops in distributed word2vec on tensorflow
Currently use original basic implementation in official document. It seems lots of time waste in RecvTensor ops which don't have a following logical computations. Maybe not a waste? Not sure...
Any suggestions or at least reason for this scenario? And, do need some help for a efficient word2vec model on tensorflow arch.
Can we use Eigenvalues and SVMs for image recognition?
I am trying to create a model for recognizing cars. In SciKit I found an example of using eignevalues and SVM to recognize faces, http://scikit-learn.org/stable/auto_examples/applications/plot_face_recognition.html#sphx-glr-auto-examples-applications-plot-face-recognition-py.
I used the example in Scikit to recognize cars given in the data set: https://ai.stanford.edu/~jkrause/cars/car_dataset.html. This data set has training and testing data sets.
However, I am unable to get good results. My question is can we use the face recognition method to recognize any other image, say cars. After all it is an image. There are 196 car classes. Is that an issue for SVM?
using label encoding in classification
I have made a classifier which uses a data-set of network traffic information such as ip, port, service etc. (dataset). Since this data contains words and the classifier only takes numbers, I have used pandas label encoding (category codes) to transform them into numbers. This has worked for training and testing the model.
When I perform the same method on live network data which I am scraping using the python library scapy all codes are returned as zero. I have tried sci-kit learns label encoder also which gives the same result. Because I have multiple columns in the data-frame I perform label encoding on each column separately, I am guessing that it sees each column as a list of one and is giving the value a zero since it is the first and only value in the list. I thought this type of encoding would work more like hashing does eg. the training data had many IP's, each having its unique encoding. If I pass the same IP string and perform label encoding on it, it will also return that same encoding. It seems like this may not be the case. Is this a case of me using label encoding incorrectly or that this type of encoding doesn't work in this way ? If it doesn't work in the way I assumed which type of encoding would be better suited for this ?
The full code I am using which returns zeros is here, and an example is below.
def codeit(df): obj_df = df.select_dtypes(include=['object']).copy() obj_df = df.astype('category') obj_df["srcip"] = obj_df["srcip"].cat.codes def sniffPackets(packet): # custom custom packet sniffer action method if packet.haslayer(IP): if packet[IP].src != host_ip: pckt_src = packet[IP].src df = pd.DataFrame() df["srcip"] = [pckt_src] codeit(df) def main(): sniff(filter="ip",iface="Intel(R) Ethernet Connection (2) I218-V",prn=sniffPackets)
Imputer on some columns in a Dataframe
I am trying to use Imputer on a singe column called age to replace missing values.But I get the error as " Expected 2D array, got 1D array instead:"
Following is my code
import pandas as pd import numpy as np from sklearn.preprocessing import Imputer dataset = pd.read_csv("titanic_train.csv") dataset.drop('Cabin',axis = 1,inplace = True) x = dataset.drop('Survived',axis = 1) y = dataset['Survived'] imputer = Imputer(missing_values ="nan",strategy = "mean",axis = 1) imputer=imputer.fit(x['Age']) x['Age']=imputer.transform(x['Age'])
tf.Dataset.from_tensor_slices performance problem
For my input fn, I am filling the tf.Data.Dataset.from_tensor_slices((np.array1, np.array2, np.array3)
And I am parsing them by calling dataset.map. as a result I am returning my dataset.
I would understand the initialization of the dataset can be slow but when I call the tf.estimator train_and_evaluate function to train and evaluate the model, it is performing very bad. Probably it is populating data in each epoch. What would be the reason? How can I overcome this problem. What would be your recommendations?
Loading a checkpoint from a trained model using estimator
I want to do very simple task. Let us assume that I have executed a model and saved multiple checkpoints and metada for this model using tf.estimator. We can again assume that I have 3 checkpoints. 1, 2 and 3. While I am evaluating the trained results on the tensorboard, I am realizing that checkpoint 2 is providing the better weights for my objective.
Therefore I want to load checkpoint 2 and make my predictions. What I want to ask simply is that, is it possible to delete checkpoint 3 from the model dir and let the estimator load it automatically from checkpoint 2 or is there anything I can do to load a specific checkpoint for.my predictions?
tf.summary.image seems not work for estimator prediction
I want visualize my input image use tf.estimator when predict, but it seems tf.summary.image not save image. But it work for training.
This is my code in model_fn:
... summary_hook = tf.train.SummarySaverHook( save_secs=2, output_dir='summary', scaffold=tf.train.Scaffold(summary_op=tf.summary.merge_all())) #summary_op=tf.summary.merge_all()) tf.summary.histogram("logit",logits) tf.summary.image('feat', feat) if mode == tf.estimator.ModeKeys.PREDICT: return tf.estimator.EstimatorSpec(mode, predictions=preds, prediction_hooks=[summary_hook]) ...
and this my prediction code:
config = tf.estimator.RunConfig(save_summary_steps=0) estimator = tf.estimator.Estimator(model_fn=model_fn, model_dir='logs', config=config) preds = estimator.predict(input_fn=eval_input_fn)
Is there something wrong for using