text generation using Estimator API
I have been trying to start transitioning to Estimator API since it is recommended by tensorflow people. However, I wonder how some of the basic stuff can be done efficiently with estimator framework. During weekend I tried to create GRU based model for text generation and followed tensorflow example for building custom estimators. I have been able to create a model that I can train relatively easily and its results match with nonestimator version. However for sampling (generating text) I faced with some trouble. I finally made it work but it is very slow, since every time it predict a character, the estimator framework load the whole graph which makes the whole thing slow. Is there a way to not load the graph every time or any other solution? Second issue: I also had to use state_is_tuple=False since I have to send back and forth the state of GRU (between model method and generator method) and I can't send tuples. Any one knows how to deal with this? thanks P.S. Here is a link to my code example: https://github.com/amirharati/sample_estimator_charlm/blob/master/RnnLm.py
See also questions close to this topic

ValueError: No way to determine width or height from video. Need `s` in `inputdict`. Consult documentation on I/O
the following below is my code,while i run the code , got an error like "ValueError: No way to determine width or height from video. Need
s
ininputdict
. Consult documentation on I/O."vid_path = '/home/dataset/MSVD/YouTubeClips/' vid = skvideo.io.vread(my_dir) # Select 16 frames from video vid = vid[40:56] x = preprocess_input(vid)

Preparing images to feed into tensorflow as datasets
AskTensorFlow
I have used the tensorflow packaged datasets like MNIST, IMDB to study the working of tensorflow. However, in practical applications we have to preprocess and prepare the dataset on our own. Suppose I am working with image dataset, so I want to preprocess them to a format that can be fed into a tensorflow model. How can I preprocess image dataset to a tensorflow format?

How can one set VLOG to specific level?
Getting the same error as Tensorflow keep printing something related to FusedBatchNorm. One of the answers on that post says "With the fix, the output should only get printed out if the VLOG level is set to 2 or higher."
I can't figure out how to set the VLOG level to a specific value.
How does one do that?

Can an RNN/LSTM have a different rank input than its output
I'm trying to implement an LSTM that takes the SWT of a time series as an input to predict the value of the time series in the near future steps.
The input is dimension M X t, where M > 1 The output is dimension t .
is it possible to have a higher rank input to a network in tensor flow or am I required to concatenate my input to match the output dimension?

Getting multidimensional predictions with Tensorflow session.run() and rnn.static_rnn()
After many hours stuck on that problem and searching for solutions, I come here for help. I am a beginner with Tensorflow.
I have setup an LSTM prediction algorithm in Python using Tensorflow, based on this and this. Unfortunately when I run the code, I get a multidimensional prediction. Which generates the error:
> symbols_out_pred = analysed_reversed_dictionary[int(tf.argmax(onehot_pred, 1).eval())]
TypeError: only size1 arrays can be converted to Python scalars
The prediction (variable onehot_pred) has shape (3, 7), which is (batch_size, vocab_size) instead of being a one dimensional array of 7 values (vocab_size). What does it mean? And how can I have a onedimensional prediction?
Here is the simplified code with comments:
# Parameters learning_rate = 0.001 training_iters = 50 display_step = 10 time_steps = len(feature_columns) batch_size = 3 analysed_column = 0 # number of units in RNN cell n_hidden = 512 # tf Graph input vocab_size = len(dictionaries[feature_columns[analysed_column]]) x = tf.placeholder("float", [None, time_steps, 1]) y = tf.placeholder("float", [None, vocab_size]) # RNN output node weights and biases weights = { 'out': tf.Variable(tf.random_normal([n_hidden, vocab_size])) } biases = { 'out': tf.Variable(tf.random_normal([vocab_size])) } # Target log path logs_path = '/tmp/tensorflow/rnn_words' writer = tf.summary.FileWriter(logs_path) def RNN(x, weights, biases): #processing the input tensor from [batch_size, time_steps, 1] to "time_steps" number of [batch_size, 1] tensors x=tf.unstack(x ,time_steps,1) # 2layer LSTM, each layer has n_hidden units. rnn_cell = rnn.MultiRNNCell([tf.nn.rnn_cell.LSTMCell(n_hidden),tf.nn.rnn_cell.LSTMCell(n_hidden)]) # generate prediction outputs, states = rnn.static_rnn(rnn_cell, x, dtype=tf.float32) # there are n_input outputs but # we only want the last output return tf.matmul(outputs[1], weights['out']) + biases['out'] pred = RNN(x, weights, biases) # Loss and optimizer cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(logits=pred, labels=y)) optimizer = tf.train.RMSPropOptimizer(learning_rate=learning_rate).minimize(cost) # Model evaluation correct_pred = tf.equal(tf.argmax(pred,1), tf.argmax(y,1)) accuracy = tf.reduce_mean(tf.cast(correct_pred, tf.float32)) # Initializing the variables init = tf.global_variables_initializer() # Launch the graph start_time = time() with tf.Session() as session: session.run(init) step = 0 offset = randint(0,batch_size+1) end_offset = batch_size + 1 acc_total = 0 loss_total = 0 writer.add_graph(session.graph) analysed_dictionary = dictionaries[feature_columns[analysed_column]] analysed_reversed_dictionary = reverse_dictionaries[feature_columns[analysed_column]] while step < training_iters: # Generate a minibatch. Add some randomness on selection process (once all set browsed). if offset > (len(training_data)end_offset): offset = randint(0, batch_size+1) symbols_in_keys = [] for i in range(offset, offset+batch_size): symbols_in_keys.append( [dictionaries[feature][training_data[i][feature_i]] for feature_i, feature in enumerate(feature_columns)] ) symbols_in_keys = np.reshape(np.array(symbols_in_keys), [batch_size,time_steps,1]) symbols_out_onehot = np.zeros([vocab_size], dtype=float) correct_pred = training_data[offset+batch_size][analysed_column] symbols_out_onehot[analysed_dictionary[correct_pred]] = 1.0 symbols_out_onehot = np.reshape(symbols_out_onehot,[1,1]) _, acc, loss, onehot_pred = session.run([optimizer, accuracy, cost, pred], \ feed_dict={x: symbols_in_keys, y: symbols_out_onehot}) loss_total += loss acc_total += acc if (step+1) % display_step == 0: print("Iter= " + str(step+1) + ", Average Loss= " + \ "{:.6f}".format(loss_total/display_step) + ", Average Accuracy= " + \ "{:.2f}%".format(100*acc_total/display_step)) acc_total = 0 loss_total = 0 symbols_in = [training_data[i] for i in range(offset, offset + batch_size)] symbols_out = training_data[offset + batch_size] symbols_out_pred = analysed_reversed_dictionary[int(tf.argmax(onehot_pred, 1).eval())] print("%s  [%s] vs [%s]" % (symbols_in,symbols_out,symbols_out_pred)) step += 1 offset += (batch_size+1)
The line generating the variable is
_, acc, loss, onehot_pred = session.run([optimizer, accuracy, cost, pred], \ feed_dict={x: symbols_in_keys, y: symbols_out_onehot})
and the function incriminated is RNN.
Thank you a lot for your answers!

How can I predict the next elements in a dataset with LSTM in Keras, python?
This is my first time with Keras and LSTMs and I am working in a project in which I have many time series data to train with.
I have around 13000 rows of data (1 column) with numerical values regarding to a degradation level of a component ending in a failure; and on the other side I have multiple datasets of 100 rows (1 column) with data regarding to a degradation level of a component, but ending some points before a failure.
The challenge is to predict when those datasets will record a failure.
So what I have done, is the next:
from pandas import DataFrame from pandas import Series from pandas import concat from pandas import read_csv from pandas import datetime from sklearn.metrics import mean_squared_error from sklearn.preprocessing import MinMaxScaler from keras.models import Sequential from keras.layers import Dense from keras.layers import LSTM from math import sqrt from matplotlib import pyplot from numpy import array # convert an array of values into a dataset matrix def create_dataset(dataset, look_back): dataX, dataY = [], [] for i in range(len(dataset)look_back1): a = dataset[i:(i+look_back), 0] dataX.append(a) dataY.append(dataset[i + look_back, 0]) return numpy.array(dataX), numpy.array(dataY) look_back = 600 epochs = 500 batch_size = 50 data = np.array(data).reshape(1,1) data = data.astype('float32') # Scale the data scaler = MinMaxScaler(feature_range=(0, 1)) dataset = scaler.fit_transform(data) # split into train and test sets train_size = int(len(dataset) * 0.67) test_size = len(dataset)  train_size train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:] trainX, trainY = create_dataset(train, look_back) testX, testY = create_dataset(test, look_back) # reshape input to be [samples, time steps, features] trainX = numpy.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1])) testX = numpy.reshape(testX, (testX.shape[0], 1, testX.shape[1])) # create and fit the LSTM network model = Sequential() model.add(LSTM(100, activation = 'tanh', inner_activation = 'hard_sigmoid', return_sequences=True)) model.add(LSTM(50, activation = 'tanh', inner_activation = 'hard_sigmoid')) model.add(Dense(1)) model.compile(loss='mean_squared_error', optimizer='adam') model.fit(trainX, trainY, epochs=epochs, batch_size=batch_size, verbose=2) # make predictions trainPredict = model.predict(trainX) testPredict = model.predict(testX) # invert predictions trainPredict = scaler.inverse_transform(trainPredict) trainY = scaler.inverse_transform([trainY]) testPredict = scaler.inverse_transform(testPredict) testY = scaler.inverse_transform([testY]) # calculate root mean squared error trainScore = math.sqrt(mean_squared_error(trainY[0], trainPredict[:,0])) print('Train Score: %.2f RMSE' % (trainScore)) testScore = math.sqrt(mean_squared_error(testY[0], testPredict[:,0])) print('Test Score: %.2f RMSE' % (testScore))
These line codes represents the training and the evaluation of the dataset, but how can I predict the next (for example) 50 elements of one dataset of 100 rows using that model?

ValueError: Error when checking target: expected dense_3 to have 2 dimensions, but got array with shape (500, 10, 14)
Keras: 2.1.6, python 3.6, tensorflow 1.8.0
I am trying to train a Sequence Model that has two LSTM layers and 3 dense layers. I had done some data preparation beforehand and setup my data in the format the LSTM layer requires it, i.e.
(n_samples, n_timesteps, n_features)
. My data has 14 features and is actually a long sequence of 5000 steps thus I have broken it down to 500 samples to 10 time steps each. Once I completed that, I started with below model, but soon ran into the error of Input shape for the last layer. I tried using Sequential and Functional API both yield the same error.import keras from keras import callbacks import tensorflow as tf from keras.models import Model from keras.layers import Input, Dense from keras.layers import LSTM X_input = Input(X_train.shape[1:]); ## First LSTM Layer X = LSTM(10, return_sequences=True, input_shape = (10,14), name = 'LSTM_1')(X_input); ## Second LSTM Layer X = LSTM(10)(X); ## First Dense Layer X = Dense(10, activation='relu', name = 'dense_1')(X) ## Second Dense Layer X = Dense(5, activation='relu', name = 'dense_2')(X) ## Final Dense Layer X = Dense(1, activation='relu', name = 'dense_3')(X) ##The model object model = Model(inputs = X_input, outputs = X, name='LSTMModel') model.compile(optimizer = "Adam" , loss = "mean_squared_error", metrics = ['mean_squared_error','cosine', 'mae']); Model.fit(x = X_train, y = Y_train, epochs = 300, callbacks=[tensorboard], validation_data=(X_eval,Y_eval));
My data is of shape
(500,10,14)
:>>> X_train.shape (500,10,14)
And my model summary looks like such:
model.summary() _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) (None, 10, 14) 0 _________________________________________________________________ LSTM_1 (LSTM) (None, 10, 10) 1000 _________________________________________________________________ LSTM_2 (LSTM) (None, 10) 840 _________________________________________________________________ dense_1 (Dense) (None, 10) 110 _________________________________________________________________ dense_2 (Dense) (None, 5) 55 _________________________________________________________________ dense_3 (Dense) (None, 1) 6 ================================================================= Total params: 2,011 Trainable params: 2,011 Nontrainable params: 0 _________________________________________________________________
Although, I still get the error:
ValueError: Error when checking target: expected dense_3 to have 2 dimensions, but got array with shape (500, 10, 14)
The shape of my labels are as follows:
X_train = np.reshape(Train_data_scaled.values,(500,10,14)); Y_train = np.reshape(Train_labels_scaled.values,(500,10,1)); X_eval = np.reshape(Validation_data_scaled.values,(10,10,14)); Y_eval = np.reshape(Validation_labels_scaled.values,(10,10,1));
After adding the layer of RepeatVector I find another issue here is the stack trace of the same.
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_1 (InputLayer) (None, 10, 14) 0 _________________________________________________________________ LSTM_1 (LSTM) (None, 10) 1000 _________________________________________________________________ repeat_vector_1 (RepeatVecto (None, 10, 10) 0 _________________________________________________________________ LSTM_2 (LSTM) (None, 10, 10) 840 _________________________________________________________________ dense_1 (Dense) (None, 10, 10) 110 _________________________________________________________________ dense_2 (Dense) (None, 10, 5) 55 _________________________________________________________________ dense_3 (Dense) (None, 10, 1) 6 ================================================================= Total params: 2,011 Trainable params: 2,011 Nontrainable params: 0 _________________________________________________________________ Traceback (most recent call last): File ".\lstm.py", line 76, in <module> tf.app.run() File "C:\Program Files\Python36\lib\sitepackages\tensorflow\python\platform\app.py", line 126, in run _sys.exit(main(argv)) File ".\lstm.py", line 67, in main Hist = Model.fit(x = X_train, y = Y_train, epochs = 300,batch_size=10, callbacks=[tensorboard], validation_data=(X_eval,Y_eval)); File "C:\Program Files\Python36\lib\sitepackages\keras\engine\training.py", line 1630, in fit batch_size=batch_size) File "C:\Program Files\Python36\lib\sitepackages\keras\engine\training.py", line 1480, in _standardize_user_data exception_prefix='target') File "C:\Program Files\Python36\lib\sitepackages\keras\engine\training.py", line 123, in _standardize_input_data str(data_shape)) ValueError: Error when checking target: expected dense_3 to have shape (10, 1) but got array with shape (10, 14)

Creating a module from a trained Tensorflow Estimator
I wrote a custom estimator and liked to do following:
Train the estimator as usual
Be able to use the trained variables to make predictions and calculations etc. and store this functionality into a Tensorflow module for easy reuse.
However, I'm a bit lost how to do that and more importantly what the intended way is to save such module.
I know how to create a module which uses constants (but no variables from the estimator graph), e.g., like here.
The problem is that the module_fn is called at a later point. Now, I could load the latest checkpoint of the estimator graph form disk in the module_fn (see here) but I wonder if there is a more logical way to do this. For instance, what exactly is the purpose of LatestModuleExporter , or is this intended to be used with canned instead of custom estimators?

TensorFlow Estimator tracking timeline?
A normal practice of timeline to track TensorFlow session is below:
import tensorflow as tf from tensorflow.python.client import timeline x = tf.random_normal([1000, 1000]) y = tf.random_normal([1000, 1000]) res = tf.matmul(x, y) # Run the graph with full trace option with tf.Session() as sess: run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE) run_metadata = tf.RunMetadata() sess.run(res, options=run_options, run_metadata=run_metadata) # Create the Timeline object, and write it to a json tl = timeline.Timeline(run_metadata.step_stats) ctf = tl.generate_chrome_trace_format() with open('timeline.json', 'w') as f: f.write(ctf)
But now I am using
tf.estimator
, without defining a session explicitly. So how and when should I define and usetensorflow.python.client.timeline
？ 
Tensorflow: train_and_evaluate always producing loss 0
I have a project using a canned estimator in Tensorflow and trying to produce train_and_evaluate method.
estimator = tf.estimator.DNNClassifier( feature_columns=my_feature_columns, hidden_units=hidden_units, model_dir=model_dir, optimizer=tf.train.ProximalAdagradOptimizer(learning_rate=0.01,l1_regularization_strength=0.001))
Every time that I see the console output it shows that the loss is always zero.
INFO:tensorflow:loss = 2.896826e06, step = 875 INFO:tensorflow:global_step/sec: 5.96785 INFO:tensorflow:loss = 1.9453131e05, step = 975 (16.756 sec) INFO:tensorflow:global_step/sec: 7.2834 INFO:tensorflow:loss = 8.6957414e05, step = 1075 (13.730 sec) INFO:tensorflow:global_step/sec: 7.36042 INFO:tensorflow:loss = 0.0004585028, step = 1175 (13.586 sec) INFO:tensorflow:global_step/sec: 7.38419 INFO:tensorflow:loss = 0.0012249642, step = 1275 (13.542 sec) INFO:tensorflow:global_step/sec: 7.3658 INFO:tensorflow:loss = 0.002194246, step = 1375 (13.576 sec) INFO:tensorflow:global_step/sec: 7.33054 INFO:tensorflow:loss = 0.0031063582, step = 1475 (13.641 sec)
This happened since I change my input_fn (I used to load my CSV into a pandas Dataframe and work from there, however my total Dataset is over 10GB (800x1500000 in dimensions), and every time that I used to save the model, the model folder size used to be crazy large (over 200GB), so I decided to use iterators instead (I found this input function in a tutorial somewhere and it works well):
def input_fn_train(filenames, num_epochs=None, shuffle=True, skip_header_lines=0, batch_size=200, modeTrainEval=True): filename_dataset = tf.data.Dataset.from_tensor_slices(filenames) if shuffle: filename_dataset = filename_dataset.shuffle(len(filenames)) dataset = filename_dataset.flat_map(lambda filename: tf.data.TextLineDataset(filename).skip(skip_header_lines)) dataset = dataset.map(parse_csv) if shuffle: dataset = dataset.shuffle(buffer_size=batch_size * 10) dataset = dataset.repeat(num_epochs) dataset = dataset.batch(batch_size) iterator = dataset.make_one_shot_iterator() features = iterator.get_next() features, labels = features, features.pop(LABEL_COLUMN) if not modeTrainEval: return features, None return features, labels
Unfortunately, this change caused my loss being always be zero, and as a consequence the predictions are terribly bad (50% accuracy), and I can't find the reason why.
(github link with sample dataset and my code)