How to configure Keras model so training array can be larger than prediction arrays?

2018-10-11 19:39 James Adams imported from Stackoverflow

I have a Keras model that takes as input a 5-D array of geospatial-temporal data with an input shape (1, times, lats, lons, channels).

I want to train the model using more time steps than I'll typically use for the prediction input.

For example let's say my training dataset has 1000 time steps. After I've trained my model I will want to use it to make predictions on datasets that contain 100 timesteps.

How do I configure/train my model in a way that will allow for this flexibility in the time dimension?

My current model definition:

def define_model_cnn(num_times,
                     num_lats,
                     num_lons,
                     num_features,
                     num_labels):
    """
    Define a model using convolutional neural network layers.

    Input data is expected to have shape (1, times, lats, lons, features) and output data
    will have shape (1, times, lats, lons, labels).

    :param num_times: the number of times in the input's time dimension
    :param num_lats: the number of lats in the input's lat dimension
    :param num_lons: the number of lons in the input's lon dimension
    :param num_features: the number of features (input attributes) in the input's channel dimension
    :param num_labels: the number of labels (output attributes) in the output's channel dimension
    :return: a Keras neural network model that uses convolutional layers
    """

    # define the model
    cnn_model = Sequential()

    # add an initial 3-D convolutional layer
    cnn_model.add(Conv3D(filters=32,
                         kernel_size=(3, 3, 3),
                         activation="relu",
                         data_format="channels_last",
                         input_shape=(num_times, num_lats, num_lons, num_features),
                         padding='same'))

    # add a fully-connected hidden layer with twice the number of neurons as input attributes (features)
    cnn_model.add(Dense(num_features * 2, activation='relu'))

    # output layer uses no activation function since we are interested
    # in predicting numerical values directly without transform
    cnn_model.add(Dense(num_labels))

    # compile the model using the ADAM optimization algorithm and a mean squared error loss function
    cnn_model.compile(optimizer='adam', loss='mse')

    return cnn_model

I have no trouble using the above model for prediction when I have an input dataset with the same dimensions as the training input dataset, but it complains when I use an input dataset that has a different number of timesteps, i.e. the array has the same shape other than in the time dimension. For example I can train the model with an input array with shape (1, 720, 12, 23, 4) (720 time steps) but I get an error complaining about an unexpected input shape if I call predict() with an input array with shape (1, 360, 12, 23, 4) (360 time steps).

Thanks for any insight/suggestions.

See also questions close to this topic

Why is my if statement is always triggering in python

I'm new to Python and I decided to do a project by myself. In my project there is a if statement that always triggers. Also I'm still learning PEP 8, so tell me if I violated it.

yn = input('Do you need me to explain the rules. Y/N: ').lower()
if yn == 'y' or 'yes':
    print('I will think of a number between 1 - 100.')
    print('You will guess a number and I will tell you if it is higher or lower than my number.')
    print('This repeats until you guess my number.')

Dialogflow python client versioning
I am using python client for accessing dialogflow's functionality.
My question is: does
```
import dialogflow
```
and
```
import dialogflow_v2 as dialogflow
```
have any difference?
As per my experience, all the methods are the same. In the samples given by Google, import dialogflow_v2 as dialogflow has been used and I could not see any difference between the two.

Please note that here I am talking about version v2 in python client, and not the dialogflow API version.

Which device does .in_waiting and .out_waiting refer to in Pyserial?

I have a computer that is connected to a serial device at /dev/ttyUSB0 via a wire with USB2 and microUSB2 connectors.

My script writes:

ser = serial.Serial('/dev/ttyUSB0')
in_buffer = ser.in_waiting
in_data = ser.read( in_buffer )
out_buffer = ser.out_waiting
out_data = ser.read( out_buffer )

Output:

ser =  {'is_open': True, 'portstr': '/dev/ttyUSB0', 'name': '/dev/ttyUSB0', '_port': '/dev/ttyUSB0', '_baudrate': 9600, '_bytesize': 8, '_parity': 'N', '_stopbits': 1, '_timeout': None, '_write_timeout': None, '_xonxoff': False, '_rtscts': False, '_dsrdtr': False, '_inter_byte_timeout': None, '_rs485_mode': None, '_rts_state': True, '_dtr_state': True, '_break_state': False, '_exclusive': None, 'fd': 6, 'pipe_abort_read_r': 7, 'pipe_abort_read_w': 8, 'pipe_abort_write_r': 9, 'pipe_abort_write_w': 10}
in_buffer =  0 <class 'int'>
in_data =  b'' <class 'bytes'>
out_buffer =  0 <class 'int'>
out_data =  b'' <class 'bytes'>

Does in_buffer and out_buffer refer to the no. of bytes in the buffer in the UART chip of the computer and the device /dev/ttyUSB0, respectively? Why do they have zero byte size?

Input dimension error on pytorch's forward check

I am creating an RNN with pytorch, it looks like this:

class MyRNN(nn.Module):
    def __init__(self, batch_size, n_inputs, n_neurons, n_outputs):
        super(MyRNN, self).__init__()

        self.n_neurons = n_neurons
        self.batch_size = batch_size
        self.n_inputs = n_inputs
        self.n_outputs = n_outputs

        self.basic_rnn = nn.RNN(self.n_inputs, self.n_neurons)

        self.FC = nn.Linear(self.n_neurons, self.n_outputs)

    def init_hidden(self, ):
        # (num_layers, batch_size, n_neurons)
        return torch.zeros(1, self.batch_size, self.n_neurons)

    def forward(self, X):
        self.batch_size = X.size(0)
        self.hidden = self.init_hidden()

        lstm_out, self.hidden = self.basic_rnn(X, self.hidden)
        out = self.FC(self.hidden)

        return out.view(-1, self.n_outputs)

My input x looks like this:

tensor([[-1.0173e-04, -1.5003e-04, -1.0218e-04, -7.4541e-05, -2.2869e-05,
         -7.7171e-02, -4.4630e-03, -5.0750e-05, -1.7911e-04, -2.8082e-04,
         -9.2992e-06, -1.5608e-05, -3.5471e-05, -4.9127e-05, -3.2883e-01],
        [-1.1193e-04, -1.6928e-04, -1.0218e-04, -7.4541e-05, -2.2869e-05,
         -7.7171e-02, -4.4630e-03, -5.0750e-05, -1.7911e-04, -2.8082e-04,
         -9.2992e-06, -1.5608e-05, -3.5471e-05, -4.9127e-05, -3.2883e-01],
        ...

        [-6.9490e-05, -8.9197e-05, -1.0218e-04, -7.4541e-05, -2.2869e-05,
         -7.7171e-02, -4.4630e-03, -5.0750e-05, -1.7911e-04, -2.8082e-04,
         -9.2992e-06, -1.5608e-05, -3.5471e-05, -4.9127e-05, -3.2883e-01]],
       dtype=torch.float64)

and is a batch of 64 vectors with size 15.

When trying to test this model by doing:

BATCH_SIZE = 64
N_INPUTS = 15
N_NEURONS = 150
N_OUTPUTS = 1
model = MyRNN(BATCH_SIZE, N_INPUTS, N_NEURONS, N_OUTPUTS)
model(x)

I get the following error:

File "/home/tt/anaconda3/envs/venv/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 126, in check_forward_args
    expected_input_dim, input.dim()))
RuntimeError: input must have 3 dimensions, got 2

How can I fix it?

Removing stopwords; can't get rid of certain stopwords no matter what?

I'm trying to clean up a train set of newsarticles (input newsarticles) My output here is the 10 most common words in the articles.

When using the nltk stopwords certain words still got through: ['the','would','said','one','also','like','could','he'] So I added them to stopwords myself. I tried both the append method and the extend as shown in the code below. But the desired stopwords (words to be emitted) "the", and "he" is not removed.

Anyone knows why? Or know what I might be doing wrong?

(And yes; Ive googled it ALOT)
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import nltk
from nltk.corpus import stopwords
import string
import re
from IPython.display import display
from sklearn.feature_extraction.text import CountVectorizer


#importing dataset and making a copy as string
data = pd.read_csv('train.csv', encoding="ISO-8859-1")

data1 = data.copy()
data1.text = data1.text.astype(str)
to_drop = ['id',
               'title',
               'author',]
data1.drop(to_drop, inplace=True, axis=1)


#cleaning text for punctuation, whitespace, splitting, and set to lower 
data1['text'] = data1['text'].str.strip().str.lower().str.replace('[^\w\s] ', '').str.split()


#removing stopwords
stopwords = nltk.corpus.stopwords.words('english')
custom_words = ['the','would','said','one','also','like','could','he']
stopwords.extend(custom_words)
data1['text'] = data1['text'].apply(lambda x: [item for item in x if item not in stopwords ])
data1['text']= data1['text'].apply(lambda x: " ".join(x))

vectorizer = CountVectorizer(max_features=1500, analyzer='word')
train_voc = vectorizer.fit_transform(data1['text'])
sum_words = train_voc.sum(axis=0)
words_freq = [(word, sum_words[0, idx]) for word, idx in vectorizer.vocabulary_.items()]
words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True)



print (words_freq[:10])



display(data1.head())

Output:

[('the', 31138), ('people', 28975), ('new', 28495), ('trump', 24752), ('president', 18701), ('he', 17254), ('us', 16969), ('clinton', 16039), ('first', 15520), ('two', 15491)]

    text    label
0   house dem aidewe didnât even see comeyâs l...   1
1   ever get feeling life circles roundabout rathe...   0
2   truth might get fired october 292016 tension i...   1
3   videos 15 civilians killed single us airstrike...   1
4   print iranian woman sentenced six years prison...   1

Someone asked for an example. This is an example where you can see 2 outputs; one before removing stopwords, and one after removing them. The same goes for this data, only that its a much larger dataset, and the output is the most common words aswell. Example output:

without stopwords: ['This', 'is', 'a', 'sample', 'sentence', ',', 'showing', 'off', 'the', 'stop', 'words', 'filtration', '.']

With stopwords['This', 'sample', 'sentence', ',', 'showing', 'stop', 'words', 'filtration', '.']

I need help for my final year project about sentiment classification

I need help for my final year project. i am working on a sentimental classifier to classify peoples reactions in python. the available research is only classifying a text as either negative positive or neutral. but my research is about classifying peoples reaction basing on the post. we get facebook posts and the corresponding reactions and then classify them. so the sentiment depends on both the reaction and the post.

i also have to extract features that can determine the sentiment of a reaction like age, sex ,education background etc. any help is appreciated. thank you.

Keras LSTM different X timesteps to Y timesteps (e.g. learn on last 4 predict next 2)

I'm having some trouble getting an LSTM model in Keras to train on the last 4 timesteps and then just predict the next 2 timesteps.

I'm sure its possible but think I'm just confusing some of the keras api.

Here is a Google Colab workbook that generates some fake data, reshapes the X and Y to be passed into the model and then trains the model.

If I set X_N_TIMESTEPS to be the same as Y_N_TIMESTEPS it trains fine - so for example use the last 4 timesteps to predict the next 4.

But I'm trying to be a bit more general and be able to train on say last 4 timesteps and then predict the next 2. The make_xy() function reshapes the data as I think it needs to. e.g.

X.shape=(1995, 4, 3)
Y.shape=(1995, 2, 3)

I think what I'm missing is telling the last Dense() layer I want it to output just 2 timesteps. The error I get is:

ValueError: Error when checking target: expected dense_1 to have shape (4, 3) but got array with shape (2, 3)

Which sort of suggests the last dense layer does not know I just want 2 timesteps even though that's what I'm passing in as the Y values.

I found this which indicated that maybe I could pass an output_dim to the last dense layer but I get an error if I try that saying I need to use keras api v2, and when I look at the docs for Dense I think the api must have changed a bit since then.

Here is all the code (in case its preferred over the colab link):

import numpy as np
import pandas as pd
from numpy import concatenate
from matplotlib import pyplot
from keras.models import Sequential
from keras.callbacks import Callback
from keras.layers import LSTM, Dense, Activation
import matplotlib.pyplot as plt

# %matplotlib inline

# define some variables
N_FEATURES = 3
X_N_TIMESTEPS = 4
Y_N_TIMESTEPS = 2
N_DATA_ORIG = 3000
N_ROLLING = 1000
N_DATA = N_DATA_ORIG - N_ROLLING

# make some noisy but smooth looking data
data = np.sqrt(np.random.rand(N_DATA_ORIG,N_FEATURES))
df_data = pd.DataFrame(data)
df_data = df_data.rolling(window=N_ROLLING).mean()
df_data = df_data.dropna()
df_data = df_data.head(N_DATA)
print(df_data.shape)
data = df_data.values
print(data.shape)
print(df_data.head())

# plot the normal healthy data
fig, ax = plt.subplots(num=None, figsize=(14, 6), dpi=80, facecolor='w', edgecolor='k')
size = len(data)
for x in range(data.shape[1]):
    ax.plot(range(0,size), data[:,x], '-', linewidth=1)

def make_xy(data,x_n_timesteps,y_n_timesteps,print_info=True):
    ''' Function to reshape the data into model ready format, either for training or prediction.
    '''
    # get original data shape
    data_shape = data.shape
    # get n_features from shape of input data
    n_features = data_shape[1]
    # loop though each row of data and reshape accordingly
    for i in range(len(data)):
        # row to start on for x
        xi = i 
        # row to start on for y
        yi = i + x_n_timesteps
        x = np.array([data[i:(i+x_n_timesteps),]])
        y = np.array([data[yi:(yi+y_n_timesteps),]])
        # only collect valid shapes
        if (x.shape == (1,x_n_timesteps,n_features)) & (y.shape == (1,y_n_timesteps,n_features)):
            # if initial data then copy else concatenate
            if i == 0:
                X = x
                Y = y
            else:
                X = np.concatenate((X,x))
                Y = np.concatenate((Y,y))
    if print_info:
        print('X.shape={}'.format(X.shape))
        print('Y.shape={}'.format(Y.shape))
    return X, Y

# build network 
model = Sequential()
model.add(LSTM(10,input_shape=(X_N_TIMESTEPS,N_FEATURES),return_sequences=True))
model.add(LSTM(10,return_sequences=True))
model.add(Dense(N_FEATURES))
model.compile(loss='mae', optimizer='adam')

# print model summary
print(model.summary())

# reshape data for training
print(f'... reshaping data for training ...')
X, Y = make_xy(data,X_N_TIMESTEPS,Y_N_TIMESTEPS)

# fit model
model.fit(X, Y)

Changing variable in Lambda layer in pretrained model?

I have imported and pytorch model to keras using pytorch2keras and have made the input flexible from [None,3,224,224] to [None,3,224,224]. Unfortunately, in the original model there is a Lambda layer reducing the output of a convolutional layer by 1, e.g. [None,3,111,111] -> [None,3,110, 110].

How can I specify in my config that I would like to do [None,3,None,None] -> [None,3,None-1, None-1]?

The shape of the Lambda layer is hardcoded here (see below line: (3,0,110)):

[...,  
{'name': 'lambda_2',
  'class_name': 'Lambda',
  'config': {'name': 'lambda_2',
   'trainable': True,
   'function': ('4wQAAAAAAAAABAAAAAYAAABTAAAAc34AAAB8AWQBawJyFHwAfAJ8A4UCGQBTAHwBZAJrAnIwfABk\nAGQAhQJ8AnwDhQJmAhkAUwB8AWQDawJyUnwAZABkAIUCZABkAIUCfAJ8A4UCZgMZAFMAfAFkBGsC\ncnp8AGQAZACFAmQAZACFAmQAZACFAnwCfAOFAmYEGQBTAGQAUwApBU7pAAAAAOkBAAAA6QIAAADp\nAwAAAKkAKQTaAXjaBGF4aXPaBXN0YXJ02gNlbmRyBQAAAHIFAAAA+j4vdXNyL2xvY2FsL2xpYi9w\neXRob24zLjYvZGlzdC1wYWNrYWdlcy9weXRvcmNoMmtlcmFzL2xheWVycy5wedoMdGFyZ2V0X2xh\neWVypgQAAHMQAAAAAAEIAQwBCAEUAQgBGgEIAQ==\n',
    (3,0,110),
    None),
   'function_type': 'lambda',
   'output_shape': None,
   'output_shape_type': 'raw',
   'arguments': {}},
  'inbound_nodes': [[['lambda_1', 0, 0, {}]]]},
..]

Applying LSTM algorithm to Time Series Data

i am a neewbie when it comes to neural networks / artificial intelligence. I hope somebody can help me.

I have a server system by adding multiple hosts with the respective services. Here is an example of the data:

timestamp | value | host | Service

2018-05-04 09:16:00 | 2.000000e+00 | ddcrham505n2 | cpu

2018-05-04 21:10:00 | 1.061835e+10 | ddcrham505n1 | available_volume_space_e

2018-05-04 21:17:00 | 1.061835e+10 | ddcrham501n1 | available_volume_space_m

2018-05-04 21:24:00 | 1.061999e+10 | ddcrham514n2 | available_volume_space_l

2018-05-04 21:31:00 | 1.062202e+10 | ddcrham509n2 | available_volume_space_e

Every host has every service.

I want to make a prediction and at the end I want to use model.predict. However, I don't know how to create the newX. Because I want to use the newX to determine the time period of the prediction (for example a week or a month).

Importing Libraries

import numpy as np
import pandas as pd
import time
from numpy import newaxis
%matplotlib inline

import matplotlib.pyplot as plt  # this is used for the plot the graph
import seaborn as sns  # used for plot interactive graph.
from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error

## for Deep-learing:
import keras
from keras.layers import Dense
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dropout, Activation
import os

os.environ['KMP_DUPLICATE_LIB_OK']='True'

sns.set(style="whitegrid")

Function creation

# convert an array of values into a dataset matrix
def create_dataset(dataset, look_back=1):
    dataX, dataY = [], []
    for i in range(len(dataset) - look_back - 1):
        a = dataset[i:(i + look_back), 0]
        dataX.append(a)
        dataY.append(dataset[i + look_back, 0])
    return np.array(dataX), np.array(dataY)

def predict_and_score(model, X, Y):
    # Make predictions on the original scale of the data.
    pred = scaler.inverse_transform(model.predict(X))
    # Prepare Y data to also be on the original scale for interpretability.
    orig_data = scaler.inverse_transform([Y])
    # Calculate RMSE.
    score = np.math.sqrt(mean_squared_error(orig_data[0], pred[:, 0]))
    return(score, pred)

Data Importing & Cleaning

# date parser
dateparse = lambda dates: pd.datetime.strptime(dates, '%Y-%m-%d %H:%M')

# import data
server_capacity = pd.read_csv('data/dra_01-01-18.csv', names=["timestamp", "value", "host", "service"])

server_capacity['timestamp'] = pd.to_datetime(server_capacity['timestamp'], format='%Y-%m-%d %H:%M')

# harddisk in use with no data resampling
d = {}
for_modeling = {}
for i, hosts in enumerate(server_capacity.host.unique()):
    d["{0}".format(hosts)] = server_capacity.loc[server_capacity.host == hosts]
    d[hosts] = d.get(hosts).set_index('timestamp')
    d[hosts] = d.get(hosts).sort_index()
    for j, services in enumerate(d.get(hosts).service.unique()):
        for_modeling["{0} + {1}".format(services, hosts)] = d.get(hosts).loc[d.get(hosts).service == services]
        for_modeling[services + " + " + hosts] = for_modeling.get(services + " + " + hosts).sort_index()
        for_modeling[services + " + " + hosts] = for_modeling.get(services + " + " + hosts).resample('H').mean()

Omitted graphing for brevitiy

# Extracted Dataframe to test
df = for_modeling.get('cpu + ddcrham508n1')
df = df.dropna()

df.plot(title='Time Series Plot')
plt.show()

r = df.wert.resample('D').agg(['mean', 'std'])
r.plot(subplots=True, title='value resampled over day')
plt.show()

Data Preperation

df = df.wert.values.astype('float32')
df = df.reshape(df.size, 1)
df.shape

scaler = MinMaxScaler(feature_range=(0, 1))
df = scaler.fit_transform(df)

n_train_time = int(len(df) * 0.67)
train = df[:n_train_time, :]
test = df[n_train_time:, :]

# reshape into X=t and Y=t+1
look_back = 1
trainX, trainY = create_dataset(train, look_back)
testX, testY = create_dataset(test, look_back)

trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1]))
testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))

Model Building

model = Sequential()

model.add(LSTM(return_sequences=True, input_shape=(None, 1), units=50))
model.add(Dropout(0.2))

model.add(LSTM(
    100,
    return_sequences=False))
model.add(Dropout(0.2))

model.add(Dense(units=1))
model.add(Activation('linear'))

start = time.time()
model.compile(loss='mse', optimizer='adam')
print ('compilation time : ', time.time() - start)

Model Fitting

model.fit(
    trainX,
    trainY,
    batch_size=128,
    epochs=10,
    validation_split=0.05)

Validation

window_size = 1

rmse_train, train_predict = predict_and_score(model, trainX, trainY)
rmse_test, test_predict = predict_and_score(model, testX, testY)


print("Training data score: %.2f RMSE" % rmse_train)
print("Test data score: %.2f RMSE" % rmse_test)

Plotting Actual v Predicted

# Start with training predictions.
train_predict_plot = np.empty_like(df)
train_predict_plot[:, :] = np.nan
train_predict_plot[window_size:len(train_predict) + window_size, :] = train_predict

# Add test predictions.
test_predict_plot = np.empty_like(df)
test_predict_plot[:, :] = np.nan
test_predict_plot[len(train_predict) + (window_size * 2) + 1:len(df) - 1, :] = test_predict

# Create the plot.
plt.figure(figsize = (15, 5))
plt.plot(scaler.inverse_transform(df), label = "True value")
plt.plot(train_predict_plot, label = "Training set prediction")
plt.plot(test_predict_plot, label = "Test set prediction")
plt.xlabel("Time (Hours)")
plt.ylabel("value")
plt.title("Comparison true vs. predicted training / test")
plt.legend()
plt.show()

Forecasting / this is my problem

newX = ?????????
predict =model.predict(newX)

plt.plot(predict)

Related questions