How to configure Keras model so training array can be larger than prediction arrays?

2018-10-11 19:39 James Adams imported from Stackoverflow

I have a Keras model that takes as input a 5-D array of geospatial-temporal data with an input shape (1, times, lats, lons, channels).

I want to train the model using more time steps than I'll typically use for the prediction input.

For example let's say my training dataset has 1000 time steps. After I've trained my model I will want to use it to make predictions on datasets that contain 100 timesteps.

How do I configure/train my model in a way that will allow for this flexibility in the time dimension?

My current model definition:

def define_model_cnn(num_times,
                     num_lats,
                     num_lons,
                     num_features,
                     num_labels):
    """
    Define a model using convolutional neural network layers.

    Input data is expected to have shape (1, times, lats, lons, features) and output data
    will have shape (1, times, lats, lons, labels).

    :param num_times: the number of times in the input's time dimension
    :param num_lats: the number of lats in the input's lat dimension
    :param num_lons: the number of lons in the input's lon dimension
    :param num_features: the number of features (input attributes) in the input's channel dimension
    :param num_labels: the number of labels (output attributes) in the output's channel dimension
    :return: a Keras neural network model that uses convolutional layers
    """

    # define the model
    cnn_model = Sequential()

    # add an initial 3-D convolutional layer
    cnn_model.add(Conv3D(filters=32,
                         kernel_size=(3, 3, 3),
                         activation="relu",
                         data_format="channels_last",
                         input_shape=(num_times, num_lats, num_lons, num_features),
                         padding='same'))

    # add a fully-connected hidden layer with twice the number of neurons as input attributes (features)
    cnn_model.add(Dense(num_features * 2, activation='relu'))

    # output layer uses no activation function since we are interested
    # in predicting numerical values directly without transform
    cnn_model.add(Dense(num_labels))

    # compile the model using the ADAM optimization algorithm and a mean squared error loss function
    cnn_model.compile(optimizer='adam', loss='mse')

    return cnn_model

I have no trouble using the above model for prediction when I have an input dataset with the same dimensions as the training input dataset, but it complains when I use an input dataset that has a different number of timesteps, i.e. the array has the same shape other than in the time dimension. For example I can train the model with an input array with shape (1, 720, 12, 23, 4) (720 time steps) but I get an error complaining about an unexpected input shape if I call predict() with an input array with shape (1, 360, 12, 23, 4) (360 time steps).

Thanks for any insight/suggestions.

See also questions close to this topic

Keras Lambda Layer Before Embedding: Use to Convert Text to Integers
I currently have a keras model which uses an Embedding layer. Something like this:
```
input = tf.keras.layers.Input(shape=(20,) dtype='int32')
x = tf.keras.layers.Embedding(input_dim=1000,
                              output_dim=50,
                              input_length=20,
                              trainable=True,
                              embeddings_initializer='glorot_uniform',
                              mask_zero=False)(input)
```
This is great and works as expected. However, I want to be able to send text to my model, have it preprocess the text into integers, and continue normally.

Two issues:

1) The Keras docs say that Embedding layers can only be used as the first layer in a model: https://keras.io/layers/embeddings/

2) Even if I could add a Lambda layer before the Embedding, I'd need it to keep track of certain state (like a dictionary mapping specific words to integers). How might I go about this stateful preprocessing?

In short, I need to modify the underlying Tensorflow DAG, so when I save my model and upload to ML Engine, it'll be able to handle my sending it raw text.

Thanks!
PySpark: Sort or OrderBy DataFrame Column *Numerically* not working correctly
I have some fitness data that I'm trying to sort numerically, however the results are not turning out as the manual, and other examples, e.g. Spark DataFrame groupBy and sort in the descending order (pyspark), show:
```
display(df.sort(col("Calories Burned").desc()))#fails to sort correctly, shows 876, then 4756,
display(df.orderBy("Calories Burned", ascending = False))#fails to sort correctly, shows 876, then 4756
display(df.sort(desc("Calories Burned")))
```
All these examples display the following two columns data (there are more columns, but I'm abbreviating for space):
```
Date    Calories Burned
2018-10-18   876
2018-05-26    4756
2018-05-05    4440
```
As you can see, these are not sorting numerically. Spark isn't taking the number of digits into account, so 876 appears before 4756.

Whether or not I include the col function makes no difference, either.

How can this dataset be sorted numerically so the data looks more like this?
```
Date    Calories Burned
2018-05-26    4756
2018-05-05    4440
2018-10-18   876
```

Turtle is not reacting to onkeypress [SOLVED]

So, I am new in python so I took some time and watched some videos about how to make a simple "snake" game, I was doing everything that dude was saying, but when it came to the keyboard binding something went wrong and I can't move my turtle.. code:https://pastebin.com/GLSRNKLR

import turtle
import time

delay = 0.1
# Screen
wn = turtle.Screen()
wn.title("Snake Game By AniPita")
wn.bgcolor('black')
wn.setup(600, 600)
wn.tracer(0)
# Snake Head
head = turtle.Turtle()
head.speed(0)
head.shape("square")
head.color("white")
head.penup()
head.goto(0, 0)
head.direction = "stop"


# Functions
def go_up():
    head.direction == "up"


def go_down():
    head.direction == "down"


def go_left():
    head.direction == "left"


def go_right():
    head.direction == "right"


def move():
    if head.direction == "up":
        y = head.ycor()
        head.sety(y + 10)
    if head.direction == "down":
        y = head.ycor()
        head.sety(y - 10)
    if head.direction == "left":
        x = head.xcor()
        head.setx(x - 10)
    if head.direction == "right":
        x = head.xcor()
        head.setx(x + 10)


# Keyboard Bindings
wn.onkeypress(go_up(), 'w')
wn.onkeypress(go_down(), 's')
wn.onkeypress(go_left(), 'a')
wn.onkeypress(go_right(), 'd')
wn.listen()
# Main Game
while True:
    wn.update()
    time.sleep(delay)
    move()

wn.mainloop()

Freezing weight of neural network such that its output takes a particular value at a particular point (tensorflow)

Let's say I have a neural network that looks like this

def neural_net(x):
    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
    layer_1 = tf.nn.sigmoid(layer_1)

    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
    layer_2 = tf.nn.sigmoid(layer_2)

    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']
    return out_layer

Is there a way in tensorflow to fix the weights in such a way that neural_net(a) always returns b (where a,b are real numbers), e.g., f(1) = 0?

spaCy: optimizing tokenization

I'm currently trying to tokenize a text file where each line is the body text of a tweet:

"According to data reported to FINRA, short volume percent for $SALT clocked in at 39.19% on 12-29-17 http://www.volumebot.com/?s=SALT"
"@Good2go @krueb The chart I posted definitely supports ng going lower.  Gobstopper&#39; 2.12, might even be conservative."
"@Crypt0Fortune Its not dumping as bad as it used to...."
"$XVG.X LOL. Someone just triggered a cascade of stop-loss orders and scooped up morons&#39; coins. Oldest trick in the stock trader&#39;s book."

The file is 59,397 lines long (a day's worth of data) and I'm using spaCy for pre-processing/tokenization. It's currently taking me around 8.5 minutes and I was wondering if there were any way of optimising the following code to be quicker as 8.5 minutes seems awfully long for this process:

def token_loop(path):
    store = []
    files = [f for f in listdir(path) if isfile(join(path, f))]

    start_time = time.monotonic()
    for filename in files:
        with open("./data/"+filename) as f:
            for line in f:
                tokens = nlp(line.lower())
                tokens = [token.lemma_ for token in tokens if not token.orth_.isspace() and token.is_alpha and not token.is_stop and len(token.orth_) != 1]
                store.append(tokens)

    end_time = time.monotonic()
    print("Time taken to tokenize:",timedelta(seconds=end_time - start_time))

    return store

Although it says files, it's currently only looping over 1 file.

Just to note, I only need this to tokenize the content; I don't need any extra tagging etc.

Heat Map of Grid Search parameters

I'm tuning two parameters (alpha, gamma) with a grid search:

from sklearn.kernel_ridge import KernelRidge
from sklearn.model_selection import GridSearchCV

alphas = logspace(-5,5,10)
gammas = logspace(-3,3,10)

parameters = {
    'alpha' : alphas,
    'gamma' : gammas
}

krr = GridSearchCV(KernelRidge(kernel='poly'), parameters, n_jobs=-1, verbose=1)

krr.fit(train_X, train_Y)

All good so far. Now I would like to create a 2D heat map of the two parameters I've just tuned. I have followed this instruction and ended up with an approach like this:

from sklearn.metrics import make_scorer

scores = krr.cv_results_['mean_test_score'].reshape(len(alphas),len(gammas))

plt.figure(figsize=(8, 6))
plt.subplots_adjust(left=.2, right=0.95, bottom=0.15, top=0.95)
plt.imshow(scores, interpolation='nearest', cmap=plt.cm.hot)
plt.xlabel('Alpha')
plt.ylabel('Gammas')
plt.colorbar()
plt.xticks(arange(len(alphas)), alphas)
plt.yticks(arange(len(gammas)), gammas)
plt.title('Grid Search AUC Score')
plt.show()

Resulting to this ugly solution:

I am looking for a heat map that gives me a better output. Readable x, y labels and colors that don't hurt your eyes.. something like this:

If easy adaptable, I also would extend this solution for a 3D heat map for the case of three parameters.

Keras autoencoder optimizer and loss function

I've been implementing an autoencoder which receives as inputs vectors that consist only of 0 and 1, such as [1, 0, 1, 0, 1, 0, ...]. Likewise, another autoencoder that receives as inputs vectors that consist in values between 0 and 1, such as [0.123, 1, 0.9, 0.01, 0.9, ...]. In both cases each vector element is the input value of a node. The activation function of the hidden layers is relu and for the output layer is sigmoid.

I've seen some examples of autoencoders where adam/adadelta are used as optimizer and binary_crossentropy is used as a loss function. For that reason I implemented in both adadelta and binary_crossentropy, but I'm not sure if for both cases it's the correct configuration.

Thanks for any help!
tf.transform: add preprocessing to Keras model?

I have a keras model for text classification using a tensorflow backend. It currently assumes the input is a numpy array of integers.

I'd like to modify this so that I can train and predict on raw text. From what I've gathered, this involves using tf.transform to convert a tensor of strings into a tensor of integers.

I've done this using tf.transform but now am unsure how to add this preprocessing step to my model as the very first layer / step. To be clear, my input data looks like this:

[{"review":"movie is great}, {"review":"awful film"}]

and the output is:

[{"review_out": array([-1, -1, 1, 0, 2])}, {"review_out": array([-1, -1, -1, 3, 4])]

The function that does this is called preprocess. So I just want to include running preprocess as the first step in my DAG.

How might I do this?

For reference, this is important because I want to do live prediction on ML Engine.

Related questions

How to configure Keras model so training array can be larger than prediction arrays?

See also questions close to this topic