Related questions

How to configure Keras model so training array can be larger than prediction arrays?

2018-10-11 19:39 James Adams imported from Stackoverflow

I have a Keras model that takes as input a 5-D array of geospatial-temporal data with an input shape (1, times, lats, lons, channels).

I want to train the model using more time steps than I'll typically use for the prediction input.

For example let's say my training dataset has 1000 time steps. After I've trained my model I will want to use it to make predictions on datasets that contain 100 timesteps.

How do I configure/train my model in a way that will allow for this flexibility in the time dimension?

My current model definition:

def define_model_cnn(num_times,
                     num_lats,
                     num_lons,
                     num_features,
                     num_labels):
    """
    Define a model using convolutional neural network layers.

    Input data is expected to have shape (1, times, lats, lons, features) and output data
    will have shape (1, times, lats, lons, labels).

    :param num_times: the number of times in the input's time dimension
    :param num_lats: the number of lats in the input's lat dimension
    :param num_lons: the number of lons in the input's lon dimension
    :param num_features: the number of features (input attributes) in the input's channel dimension
    :param num_labels: the number of labels (output attributes) in the output's channel dimension
    :return: a Keras neural network model that uses convolutional layers
    """

    # define the model
    cnn_model = Sequential()

    # add an initial 3-D convolutional layer
    cnn_model.add(Conv3D(filters=32,
                         kernel_size=(3, 3, 3),
                         activation="relu",
                         data_format="channels_last",
                         input_shape=(num_times, num_lats, num_lons, num_features),
                         padding='same'))

    # add a fully-connected hidden layer with twice the number of neurons as input attributes (features)
    cnn_model.add(Dense(num_features * 2, activation='relu'))

    # output layer uses no activation function since we are interested
    # in predicting numerical values directly without transform
    cnn_model.add(Dense(num_labels))

    # compile the model using the ADAM optimization algorithm and a mean squared error loss function
    cnn_model.compile(optimizer='adam', loss='mse')

    return cnn_model

I have no trouble using the above model for prediction when I have an input dataset with the same dimensions as the training input dataset, but it complains when I use an input dataset that has a different number of timesteps, i.e. the array has the same shape other than in the time dimension. For example I can train the model with an input array with shape (1, 720, 12, 23, 4) (720 time steps) but I get an error complaining about an unexpected input shape if I call predict() with an input array with shape (1, 360, 12, 23, 4) (360 time steps).

Thanks for any insight/suggestions.

See also questions close to this topic

  • How do we call bash functions from python

    Let's say i have a bash function that i source into a shell.

    # cat sample.sh 
function func()
{
    echo "Custom env : $CUSTOM_ENV"
}

    Now i source this script in the bash shell: #source sample.sh

    Then i define:

    export CUSTOM_ENV="abc"

    and then call func() from bash shell, it displays:

    #func
Custom env : abc

    Now if i am calling a python script from the same shell, i want to invoke the function func() from the python script. Anyway to achieve this ?

    What i tried:

    1. Tried os.system('func') - Doesn't work
    2. Tried subprocess.check_output('func', shell=True, env=os.environ.copy()) - Doesn't work

    Any guidance ?

  • How to run a file by another user in python Subprocess

    I am creating a python script which calls a shell script and run that shell script on a terminal using Subprocess . My problem is that I want to run that shell script by another user . My code is Given below

    Code: 

    import subprocess
filename = '/mount/test.sh'
p = subprocess.Popen([filename],shell=True,stdout=subprocess.PIPE)
out, err = p.communicate()
print(out)

    Can anyone tell me that how to run my shell script by another user ?

    Note:

    Only the subprocess part should be run by another user, not the main python script

  • Build new logic adapter

    I have 3 different types of csv files in which there is different data, I want to write 3 logic adapter for each csv files, in which I have to write code using conditions like if statement has confidence <007 then it should show matching information , elseif it should show other response else different repsonse .

    So how I can built logic for it using chatterbot?

  • How can I compute the tensor in Pytorch efficiently?

    I have a tensor x and x.shape=(batch_size,10), now I want to take 

     x[i][0] = x[i][0]*x[i][1]*...*x[i][9] for i in range(batch_size)

    Here is my code:

    for i in range(batch_size):
    for k in range(1, 10):
        x[i][0] = x[i][0] * x[i][k]

    But when I implement this in forward() and call loss.backward(), the speed of back-propagation is very slow. Why is it slow and is there any way to implement it efficiently?

  • How do I get all Gini indices in my decision tree?

    I have made a decision tree using sklearn, here, under the SciKit learn DL package, viz. sklearn.tree.DecisionTreeClassifier().fit(x,y).

    How do I get the gini indices for all possible nodes at each step? graphviz only gives me the gini index of the node with the lowest gini index, ie the node used for split.

    For example, the image below (from graphviz) tells me the gini score of the Pclass_lowVMid right index which is 0.408, but not the gini index of the Pclass_lower or Sex_male at that step. I just know the Gini index of Pclass_lower and Sex_male must be greater than (0.408*0.7 + 0) but that's it.

    decision tree

  • Fighting against overfitting in an RNN model

    We are currently trying to use an RNN model to build a classifier using text features. Our final accuracy on the training data is 87% but our accuracy on validation data flat out at 57% which is clearly overfitting. We think that the reason for overfitting is because of small data size since we only have about 4000 entries. What can we do to fix to the problem, we have also thought about doing data augmentation but all we can find is replacing words with synonyms which it wouldn't work in our case. Here's our code for the model and thank you in advance.

    model = Sequential()
model.add(Embedding(num_vocab+1,32))
model.add(SimpleRNN(64))
model.add(Dense(num_classes, activation='softmax'))

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['acc'])
history = model.fit(f_train, cause_train, epochs=10, batch_size=50, validation_split=0.2)
  • dimension errors in tf.scatter_nd in tensorflow/keras

    My codes:

    reshape_out = Reshape((21, 3), input_shape=(21*3,), name='reshape_to_21_3')(output3d)

def proj_output_shape(shp):
    return (None, 32, 32, 1)

def f(x):
    import tensorflow as tf
    batch_size = K.shape(x)[0]
    print('x.shape={0}'.format(x.shape))

    idx = K.cast(x[:, :, 0:2]*15.5+15.5, "int32")
    print('idx.shape={0}'.format(idx.shape))

    # z = mysparse_to_dense(idx, (K.shape(x)[0], 32, 32), 1.0, 0.0, name='sparse_tensor')
    updates = tf.ones([batch_size, 21])
    print('updates.shape={0}'.format(updates.shape))

    #shape = tf.Variable(np.array([batch_size, 32, 32]))
    #print('shape.shape={0}'.format(shape))

    z = tf.scatter_nd(indices=idx,
                      updates=updates,
                      shape=(batch_size, 32, 32),
                      name='cool')

    print('z={0}'.format(z))
    #z = tf.add(z, z)
    #z = tf.sparse_add(tf.zeros(z.dense_shape), z)
    z = K.reshape(z, (K.shape(x)[0], 32, 32, 1))
    print('z.shape={0}'.format(z.shape), z)

    fil = make_kernel(1.0)
    fil = K.reshape(fil, (5, 5, 1, 1))
    print('fil.shape={0}'.format(fil.shape), fil)

    r = K.conv2d(z,kernel=fil, padding='same', data_format="channels_last")
    print('r.shape={0}'.format(r.shape), r)

    return r

    Output:

    x.shape=(?, 21, 3)
idx.shape=(?, 21, 2)
updates.shape=(?, 21)

    errors:

    ValueError: The inner 1 dimensions of output.shape=[?,?,?] must match the inner 0 dimensions of updates.shape=[?,21]: Shapes must be equal rank, but are 1 and 0 for 'projection_4/cool' (op: 'ScatterNd') with input shapes: [?,21,2], [?,21], [3].

    How to fix this? Thanks

  • Getting worse results on gcloud vs local training

    I'm trying to understand why my local results are better than the gcloud results.

    In local, I run a job like this:

    gcloud ml-engine local train --module-name trainer.task --package-path trainer -- --vocabulary-file trainer/data/vocab.txt --class-files $CLASS_FILES --job-dir trainer/lr0001 --num-epochs 5000 --learning-rate 0.0001 --train-batch-size 4 --eval-batch-size 64 --export-format CSV

    And for gcloud I run

    gcloud ml-engine jobs submit training $JOBNAME --job-dir gs://.../lr0001 --module-name trainer.task --package-path trainer --region us-west1 --runtime-version 1.10 -- --vocabulary-file gs://.../vocab.txt --class-files $GS_CLASS_FILES --num-epochs 5000 --learning-rate 0.0001 --train-batch-size 4 --eval-batch-size 64 --export-format CSV

    I've fixed the seed, ran it multiple times, checked python 2 vs python 3, but the gcloud results are still worse than my local run.

    One last bit of clue that I've found is that local logs look like this:

    INFO:tensorflow:loss = 0.63639945, step = 401 (0.170 sec)
INFO:tensorflow:global_step/sec: 485.821
INFO:tensorflow:loss = 0.61793035, step = 501 (0.206 sec)
INFO:tensorflow:global_step/sec: 490.795
INFO:tensorflow:loss = 0.5869169, step = 601 (0.204 sec)
INFO:tensorflow:global_step/sec: 619.825
INFO:tensorflow:loss = 0.5738391, step = 701 (0.161 sec)
INFO:tensorflow:global_step/sec: 605.698
INFO:tensorflow:loss = 0.51589084, step = 801 (0.165 sec)

    whereas gcloud logs look like they are doubling up or something

    I  master-replica-0 loss = 0.40115586, step = 2202 (0.367 sec) master-replica-0 
I  master-replica-0 global_step/sec: 555.434 master-replica-0 
I  master-replica-0 global_step/sec: 498.601 master-replica-0 
I  master-replica-0 loss = 0.4367655, step = 2402 (0.470 sec) master-replica-0 
I  master-replica-0 global_step/sec: 366.906 master-replica-0 
I  master-replica-0 global_step/sec: 408.556 master-replica-0 
I  master-replica-0 loss = 0.41198668, step = 2602 (0.492 sec) master-replica-0 
I  master-replica-0 global_step/sec: 388.73 master-replica-0 
I  master-replica-0 global_step/sec: 380.982 master-replica-0 
I  master-replica-0 loss = 0.35386887, step = 2802 (0.522 sec) master-replica-0 
I  master-replica-0 global_step/sec: 401.002 master-replica-0 
I  master-replica-0 global_step/sec: 465.647 master-replica-0 
I  master-replica-0 loss = 0.4420835, step = 3002 (0.417 sec) master-replica-0

    Any pointers would be appreciated!

    The only thing I've found in the internet so far are these SO questions with no answer:

    Results of training a Keras model different on Google Cloud

    Differents outputs from predictions using Tensorflow from same data?

  • Keras Encoder Decoder expected to have 2 dimensions

    A Keras Encoder Decoder returns an InvalidArgumentError as the shapes of the inputs seem incompatible.

    I have:

    • X_numerical.shape gives (304, 2500, 4) as input data
    • y_numerical.shape gives (304, 40, 22) as output data

    The Keras encoder-decoder is the following:

    # Define an input sequence and process it. 
encoder_inputs = Input(shape=(None, 4)) 
encoder = LSTM(32, return_state=True) 
encoder_outputs, state_h, state_c = encoder(encoder_inputs)

# We discard `encoder_outputs` and only keep the states. 
encoder_states = [state_h, state_c]

# Set up the decoder, using `encoder_states` as initial state. 
decoder_inputs = Input(shape=(None, 22))
# We set up our decoder to return full output sequences,
# and to return internal states as well. We don't use the
# return states in the training model, but we will use them in inference. 
decoder_lstm = LSTM(32, return_sequences=True, return_state=True) 
decoder_outputs, _, _ = decoder_lstm(decoder_inputs,
                                     initial_state=encoder_states) 
decoder_dense = Dense(22, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)

# Define the model that will turn
# `encoder_input_data` & `decoder_input_data` into `decoder_target_data` 
model = Model([encoder_inputs, decoder_inputs], decoder_outputs)

# Run training 
model.compile(optimizer='rmsprop', loss='categorical_crossentropy') 

### THE ERROR OCCURS IN THE `.fit()` CALL
model.fit([X_numerical, y_numerical], y_numerical,
          batch_size=4,
          epochs=1)

    And I get the following error:

    InvalidArgumentError Traceback (most recent call last) in () 25 model.fit([X_numerical, y_numerical], y_numerical, 26 batch_size=4, ---> 27 epochs=1)

    InvalidArgumentError: Incompatible shapes: [4,40,22] vs. [1,22,1]
    [[Node: training_6/RMSprop/gradients/dense_15/add_grad/BroadcastGradientArgs = BroadcastGradientArgs[T=DT_INT32, _class=["loc:@training_6/RMSprop/gradients/dense_15/add_grad/Sum"], _device="/job:localhost/replica:0/task:0/device:CPU:0"](training_6/RMSprop/gradients/dense_15/add_grad/Shape, training_6/RMSprop/gradients/dense_15/add_grad/Shape_1)]]

    What I have tried to do is to reshape y_numerical to (304, 22, 40), but doens't work. I have also tried y_numerical .squeeze() and changing the batch_size in the model.fit() call, all returning various errors.

    What might be the cause of this dimensionality error?

    The summary of my model is:

    __________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_37 (InputLayer)           (None, None, 4)      0                                            
__________________________________________________________________________________________________
input_38 (InputLayer)           (None, None, 22)     0                                            
__________________________________________________________________________________________________
lstm_39 (LSTM)                  [(None, 32), (None,  4736        input_37[0][0]                   
__________________________________________________________________________________________________
lstm_40 (LSTM)                  [(None, None, 32), ( 7040        input_38[0][0]                   
                                                                 lstm_39[0][1]                    
                                                                 lstm_39[0][2]                    
__________________________________________________________________________________________________
dense_19 (Dense)                (None, None, 22)     726         lstm_40[0][0]                    
==================================================================================================
Total params: 12,502
Trainable params: 12,502
Non-trainable params: 0
__________________________________________________________________________________________________