How to configure Keras model so training array can be larger than prediction arrays?
I have a Keras model that takes as input a 5-D array of geospatial-temporal data with an input shape (1, times, lats, lons, channels).
I want to train the model using more time steps than I'll typically use for the prediction input.
For example let's say my training dataset has 1000 time steps. After I've trained my model I will want to use it to make predictions on datasets that contain 100 timesteps.
How do I configure/train my model in a way that will allow for this flexibility in the time dimension?
My current model definition:
def define_model_cnn(num_times,
num_lats,
num_lons,
num_features,
num_labels):
"""
Define a model using convolutional neural network layers.
Input data is expected to have shape (1, times, lats, lons, features) and output data
will have shape (1, times, lats, lons, labels).
:param num_times: the number of times in the input's time dimension
:param num_lats: the number of lats in the input's lat dimension
:param num_lons: the number of lons in the input's lon dimension
:param num_features: the number of features (input attributes) in the input's channel dimension
:param num_labels: the number of labels (output attributes) in the output's channel dimension
:return: a Keras neural network model that uses convolutional layers
"""
# define the model
cnn_model = Sequential()
# add an initial 3-D convolutional layer
cnn_model.add(Conv3D(filters=32,
kernel_size=(3, 3, 3),
activation="relu",
data_format="channels_last",
input_shape=(num_times, num_lats, num_lons, num_features),
padding='same'))
# add a fully-connected hidden layer with twice the number of neurons as input attributes (features)
cnn_model.add(Dense(num_features * 2, activation='relu'))
# output layer uses no activation function since we are interested
# in predicting numerical values directly without transform
cnn_model.add(Dense(num_labels))
# compile the model using the ADAM optimization algorithm and a mean squared error loss function
cnn_model.compile(optimizer='adam', loss='mse')
return cnn_model
I have no trouble using the above model for prediction when I have an input dataset with the same dimensions as the training input dataset, but it complains when I use an input dataset that has a different number of timesteps, i.e. the array has the same shape other than in the time dimension. For example I can train the model with an input array with shape (1, 720, 12, 23, 4) (720 time steps) but I get an error complaining about an unexpected input shape if I call predict()
with an input array with shape (1, 360, 12, 23, 4) (360 time steps).
Thanks for any insight/suggestions.
See also questions close to this topic
-
Why is my if statement is always triggering in python
I'm new to Python and I decided to do a project by myself. In my project there is a if statement that always triggers. Also I'm still learning PEP 8, so tell me if I violated it.
yn = input('Do you need me to explain the rules. Y/N: ').lower() if yn == 'y' or 'yes': print('I will think of a number between 1 - 100.') print('You will guess a number and I will tell you if it is higher or lower than my number.') print('This repeats until you guess my number.')
-
Dialogflow python client versioning
I am using python client for accessing dialogflow's functionality.
My question is: doesimport dialogflow
and
import dialogflow_v2 as dialogflow
have any difference?
As per my experience, all the methods are the same. In the samples given by Google,import dialogflow_v2 as dialogflow
has been used and I could not see any difference between the two.Please note that here I am talking about version v2 in python client, and not the dialogflow API version.
-
Which device does .in_waiting and .out_waiting refer to in Pyserial?
I have a computer that is connected to a serial device at
/dev/ttyUSB0
via a wire with USB2 and microUSB2 connectors.My script writes:
ser = serial.Serial('/dev/ttyUSB0') in_buffer = ser.in_waiting in_data = ser.read( in_buffer ) out_buffer = ser.out_waiting out_data = ser.read( out_buffer )
Output:
ser = {'is_open': True, 'portstr': '/dev/ttyUSB0', 'name': '/dev/ttyUSB0', '_port': '/dev/ttyUSB0', '_baudrate': 9600, '_bytesize': 8, '_parity': 'N', '_stopbits': 1, '_timeout': None, '_write_timeout': None, '_xonxoff': False, '_rtscts': False, '_dsrdtr': False, '_inter_byte_timeout': None, '_rs485_mode': None, '_rts_state': True, '_dtr_state': True, '_break_state': False, '_exclusive': None, 'fd': 6, 'pipe_abort_read_r': 7, 'pipe_abort_read_w': 8, 'pipe_abort_write_r': 9, 'pipe_abort_write_w': 10} in_buffer = 0 <class 'int'> in_data = b'' <class 'bytes'> out_buffer = 0 <class 'int'> out_data = b'' <class 'bytes'>
Does
in_buffer
andout_buffer
refer to the no. of bytes in the buffer in the UART chip of the computer and the device/dev/ttyUSB0
, respectively? Why do they have zero byte size? -
Input dimension error on pytorch's forward check
I am creating an RNN with
pytorch
, it looks like this:class MyRNN(nn.Module): def __init__(self, batch_size, n_inputs, n_neurons, n_outputs): super(MyRNN, self).__init__() self.n_neurons = n_neurons self.batch_size = batch_size self.n_inputs = n_inputs self.n_outputs = n_outputs self.basic_rnn = nn.RNN(self.n_inputs, self.n_neurons) self.FC = nn.Linear(self.n_neurons, self.n_outputs) def init_hidden(self, ): # (num_layers, batch_size, n_neurons) return torch.zeros(1, self.batch_size, self.n_neurons) def forward(self, X): self.batch_size = X.size(0) self.hidden = self.init_hidden() lstm_out, self.hidden = self.basic_rnn(X, self.hidden) out = self.FC(self.hidden) return out.view(-1, self.n_outputs)
My input
x
looks like this:tensor([[-1.0173e-04, -1.5003e-04, -1.0218e-04, -7.4541e-05, -2.2869e-05, -7.7171e-02, -4.4630e-03, -5.0750e-05, -1.7911e-04, -2.8082e-04, -9.2992e-06, -1.5608e-05, -3.5471e-05, -4.9127e-05, -3.2883e-01], [-1.1193e-04, -1.6928e-04, -1.0218e-04, -7.4541e-05, -2.2869e-05, -7.7171e-02, -4.4630e-03, -5.0750e-05, -1.7911e-04, -2.8082e-04, -9.2992e-06, -1.5608e-05, -3.5471e-05, -4.9127e-05, -3.2883e-01], ... [-6.9490e-05, -8.9197e-05, -1.0218e-04, -7.4541e-05, -2.2869e-05, -7.7171e-02, -4.4630e-03, -5.0750e-05, -1.7911e-04, -2.8082e-04, -9.2992e-06, -1.5608e-05, -3.5471e-05, -4.9127e-05, -3.2883e-01]], dtype=torch.float64)
and is a batch of 64 vectors with size 15.
When trying to test this model by doing:
BATCH_SIZE = 64 N_INPUTS = 15 N_NEURONS = 150 N_OUTPUTS = 1 model = MyRNN(BATCH_SIZE, N_INPUTS, N_NEURONS, N_OUTPUTS) model(x)
I get the following error:
File "/home/tt/anaconda3/envs/venv/lib/python3.6/site-packages/torch/nn/modules/rnn.py", line 126, in check_forward_args expected_input_dim, input.dim())) RuntimeError: input must have 3 dimensions, got 2
How can I fix it?
-
Removing stopwords; can't get rid of certain stopwords no matter what?
I'm trying to clean up a train set of newsarticles (input newsarticles) My output here is the 10 most common words in the articles.
When using the nltk stopwords certain words still got through: ['the','would','said','one','also','like','could','he'] So I added them to stopwords myself. I tried both the append method and the extend as shown in the code below. But the desired stopwords (words to be emitted) "the", and "he" is not removed.
Anyone knows why? Or know what I might be doing wrong?
(And yes; Ive googled it ALOT) import numpy as np import matplotlib.pyplot as plt import pandas as pd import nltk from nltk.corpus import stopwords import string import re from IPython.display import display from sklearn.feature_extraction.text import CountVectorizer #importing dataset and making a copy as string data = pd.read_csv('train.csv', encoding="ISO-8859-1") data1 = data.copy() data1.text = data1.text.astype(str) to_drop = ['id', 'title', 'author',] data1.drop(to_drop, inplace=True, axis=1) #cleaning text for punctuation, whitespace, splitting, and set to lower data1['text'] = data1['text'].str.strip().str.lower().str.replace('[^\w\s] ', '').str.split() #removing stopwords stopwords = nltk.corpus.stopwords.words('english') custom_words = ['the','would','said','one','also','like','could','he'] stopwords.extend(custom_words) data1['text'] = data1['text'].apply(lambda x: [item for item in x if item not in stopwords ]) data1['text']= data1['text'].apply(lambda x: " ".join(x)) vectorizer = CountVectorizer(max_features=1500, analyzer='word') train_voc = vectorizer.fit_transform(data1['text']) sum_words = train_voc.sum(axis=0) words_freq = [(word, sum_words[0, idx]) for word, idx in vectorizer.vocabulary_.items()] words_freq =sorted(words_freq, key = lambda x: x[1], reverse=True) print (words_freq[:10]) display(data1.head())
Output:
[('the', 31138), ('people', 28975), ('new', 28495), ('trump', 24752), ('president', 18701), ('he', 17254), ('us', 16969), ('clinton', 16039), ('first', 15520), ('two', 15491)] text label 0 house dem aidewe didnât even see comeyâs l... 1 1 ever get feeling life circles roundabout rathe... 0 2 truth might get fired october 292016 tension i... 1 3 videos 15 civilians killed single us airstrike... 1 4 print iranian woman sentenced six years prison... 1
Someone asked for an example. This is an example where you can see 2 outputs; one before removing stopwords, and one after removing them. The same goes for this data, only that its a much larger dataset, and the output is the most common words aswell. Example output:
without stopwords: ['This', 'is', 'a', 'sample', 'sentence', ',', 'showing', 'off', 'the', 'stop', 'words', 'filtration', '.']
With stopwords['This', 'sample', 'sentence', ',', 'showing', 'stop', 'words', 'filtration', '.']
-
I need help for my final year project about sentiment classification
I need help for my final year project. i am working on a sentimental classifier to classify peoples reactions in python. the available research is only classifying a text as either negative positive or neutral. but my research is about classifying peoples reaction basing on the post. we get facebook posts and the corresponding reactions and then classify them. so the sentiment depends on both the reaction and the post.
i also have to extract features that can determine the sentiment of a reaction like age, sex ,education background etc. any help is appreciated. thank you.
-
Keras LSTM different X timesteps to Y timesteps (e.g. learn on last 4 predict next 2)
I'm having some trouble getting an LSTM model in Keras to train on the last 4 timesteps and then just predict the next 2 timesteps.
I'm sure its possible but think I'm just confusing some of the keras api.
Here is a Google Colab workbook that generates some fake data, reshapes the X and Y to be passed into the model and then trains the model.
If I set
X_N_TIMESTEPS
to be the same asY_N_TIMESTEPS
it trains fine - so for example use the last 4 timesteps to predict the next 4.But I'm trying to be a bit more general and be able to train on say last 4 timesteps and then predict the next 2. The
make_xy()
function reshapes the data as I think it needs to. e.g.X.shape=(1995, 4, 3) Y.shape=(1995, 2, 3)
I think what I'm missing is telling the last
Dense()
layer I want it to output just 2 timesteps. The error I get is:ValueError: Error when checking target: expected dense_1 to have shape (4, 3) but got array with shape (2, 3)
Which sort of suggests the last dense layer does not know I just want 2 timesteps even though that's what I'm passing in as the Y values.
I found this which indicated that maybe I could pass an output_dim to the last dense layer but I get an error if I try that saying I need to use keras api v2, and when I look at the docs for Dense I think the api must have changed a bit since then.
Here is all the code (in case its preferred over the colab link):
import numpy as np import pandas as pd from numpy import concatenate from matplotlib import pyplot from keras.models import Sequential from keras.callbacks import Callback from keras.layers import LSTM, Dense, Activation import matplotlib.pyplot as plt # %matplotlib inline # define some variables N_FEATURES = 3 X_N_TIMESTEPS = 4 Y_N_TIMESTEPS = 2 N_DATA_ORIG = 3000 N_ROLLING = 1000 N_DATA = N_DATA_ORIG - N_ROLLING # make some noisy but smooth looking data data = np.sqrt(np.random.rand(N_DATA_ORIG,N_FEATURES)) df_data = pd.DataFrame(data) df_data = df_data.rolling(window=N_ROLLING).mean() df_data = df_data.dropna() df_data = df_data.head(N_DATA) print(df_data.shape) data = df_data.values print(data.shape) print(df_data.head()) # plot the normal healthy data fig, ax = plt.subplots(num=None, figsize=(14, 6), dpi=80, facecolor='w', edgecolor='k') size = len(data) for x in range(data.shape[1]): ax.plot(range(0,size), data[:,x], '-', linewidth=1) def make_xy(data,x_n_timesteps,y_n_timesteps,print_info=True): ''' Function to reshape the data into model ready format, either for training or prediction. ''' # get original data shape data_shape = data.shape # get n_features from shape of input data n_features = data_shape[1] # loop though each row of data and reshape accordingly for i in range(len(data)): # row to start on for x xi = i # row to start on for y yi = i + x_n_timesteps x = np.array([data[i:(i+x_n_timesteps),]]) y = np.array([data[yi:(yi+y_n_timesteps),]]) # only collect valid shapes if (x.shape == (1,x_n_timesteps,n_features)) & (y.shape == (1,y_n_timesteps,n_features)): # if initial data then copy else concatenate if i == 0: X = x Y = y else: X = np.concatenate((X,x)) Y = np.concatenate((Y,y)) if print_info: print('X.shape={}'.format(X.shape)) print('Y.shape={}'.format(Y.shape)) return X, Y # build network model = Sequential() model.add(LSTM(10,input_shape=(X_N_TIMESTEPS,N_FEATURES),return_sequences=True)) model.add(LSTM(10,return_sequences=True)) model.add(Dense(N_FEATURES)) model.compile(loss='mae', optimizer='adam') # print model summary print(model.summary()) # reshape data for training print(f'... reshaping data for training ...') X, Y = make_xy(data,X_N_TIMESTEPS,Y_N_TIMESTEPS) # fit model model.fit(X, Y)
-
Changing variable in Lambda layer in pretrained model?
I have imported and pytorch model to keras using pytorch2keras and have made the input flexible from [None,3,224,224] to [None,3,224,224]. Unfortunately, in the original model there is a Lambda layer reducing the output of a convolutional layer by 1, e.g. [None,3,111,111] -> [None,3,110, 110].
How can I specify in my config that I would like to do [None,3,None,None] -> [None,3,None-1, None-1]?
The shape of the Lambda layer is hardcoded here (see below line: (3,0,110)):
[..., {'name': 'lambda_2', 'class_name': 'Lambda', 'config': {'name': 'lambda_2', 'trainable': True, 'function': ('4wQAAAAAAAAABAAAAAYAAABTAAAAc34AAAB8AWQBawJyFHwAfAJ8A4UCGQBTAHwBZAJrAnIwfABk\nAGQAhQJ8AnwDhQJmAhkAUwB8AWQDawJyUnwAZABkAIUCZABkAIUCfAJ8A4UCZgMZAFMAfAFkBGsC\ncnp8AGQAZACFAmQAZACFAmQAZACFAnwCfAOFAmYEGQBTAGQAUwApBU7pAAAAAOkBAAAA6QIAAADp\nAwAAAKkAKQTaAXjaBGF4aXPaBXN0YXJ02gNlbmRyBQAAAHIFAAAA+j4vdXNyL2xvY2FsL2xpYi9w\neXRob24zLjYvZGlzdC1wYWNrYWdlcy9weXRvcmNoMmtlcmFzL2xheWVycy5wedoMdGFyZ2V0X2xh\neWVypgQAAHMQAAAAAAEIAQwBCAEUAQgBGgEIAQ==\n', (3,0,110), None), 'function_type': 'lambda', 'output_shape': None, 'output_shape_type': 'raw', 'arguments': {}}, 'inbound_nodes': [[['lambda_1', 0, 0, {}]]]}, ..]
-
Applying LSTM algorithm to Time Series Data
i am a neewbie when it comes to neural networks / artificial intelligence. I hope somebody can help me.
I have a server system by adding multiple hosts with the respective services. Here is an example of the data:
timestamp | value | host | Service
2018-05-04 09:16:00 | 2.000000e+00 | ddcrham505n2 | cpu 2018-05-04 21:10:00 | 1.061835e+10 | ddcrham505n1 | available_volume_space_e 2018-05-04 21:17:00 | 1.061835e+10 | ddcrham501n1 | available_volume_space_m 2018-05-04 21:24:00 | 1.061999e+10 | ddcrham514n2 | available_volume_space_l 2018-05-04 21:31:00 | 1.062202e+10 | ddcrham509n2 | available_volume_space_e
Every host has every service.
I want to make a prediction and at the end I want to use model.predict. However, I don't know how to create the newX. Because I want to use the newX to determine the time period of the prediction (for example a week or a month).
Importing Libraries
import numpy as np import pandas as pd import time from numpy import newaxis %matplotlib inline import matplotlib.pyplot as plt # this is used for the plot the graph import seaborn as sns # used for plot interactive graph. from sklearn.preprocessing import MinMaxScaler from sklearn.metrics import mean_squared_error ## for Deep-learing: import keras from keras.layers import Dense from keras.models import Sequential from keras.layers import LSTM from keras.layers import Dropout, Activation import os os.environ['KMP_DUPLICATE_LIB_OK']='True' sns.set(style="whitegrid")
Function creation
# convert an array of values into a dataset matrix def create_dataset(dataset, look_back=1): dataX, dataY = [], [] for i in range(len(dataset) - look_back - 1): a = dataset[i:(i + look_back), 0] dataX.append(a) dataY.append(dataset[i + look_back, 0]) return np.array(dataX), np.array(dataY) def predict_and_score(model, X, Y): # Make predictions on the original scale of the data. pred = scaler.inverse_transform(model.predict(X)) # Prepare Y data to also be on the original scale for interpretability. orig_data = scaler.inverse_transform([Y]) # Calculate RMSE. score = np.math.sqrt(mean_squared_error(orig_data[0], pred[:, 0])) return(score, pred)
Data Importing & Cleaning
# date parser dateparse = lambda dates: pd.datetime.strptime(dates, '%Y-%m-%d %H:%M') # import data server_capacity = pd.read_csv('data/dra_01-01-18.csv', names=["timestamp", "value", "host", "service"]) server_capacity['timestamp'] = pd.to_datetime(server_capacity['timestamp'], format='%Y-%m-%d %H:%M') # harddisk in use with no data resampling d = {} for_modeling = {} for i, hosts in enumerate(server_capacity.host.unique()): d["{0}".format(hosts)] = server_capacity.loc[server_capacity.host == hosts] d[hosts] = d.get(hosts).set_index('timestamp') d[hosts] = d.get(hosts).sort_index() for j, services in enumerate(d.get(hosts).service.unique()): for_modeling["{0} + {1}".format(services, hosts)] = d.get(hosts).loc[d.get(hosts).service == services] for_modeling[services + " + " + hosts] = for_modeling.get(services + " + " + hosts).sort_index() for_modeling[services + " + " + hosts] = for_modeling.get(services + " + " + hosts).resample('H').mean()
Omitted graphing for brevitiy
# Extracted Dataframe to test df = for_modeling.get('cpu + ddcrham508n1') df = df.dropna() df.plot(title='Time Series Plot') plt.show() r = df.wert.resample('D').agg(['mean', 'std']) r.plot(subplots=True, title='value resampled over day') plt.show()
Data Preperation
df = df.wert.values.astype('float32') df = df.reshape(df.size, 1) df.shape scaler = MinMaxScaler(feature_range=(0, 1)) df = scaler.fit_transform(df) n_train_time = int(len(df) * 0.67) train = df[:n_train_time, :] test = df[n_train_time:, :] # reshape into X=t and Y=t+1 look_back = 1 trainX, trainY = create_dataset(train, look_back) testX, testY = create_dataset(test, look_back) trainX = np.reshape(trainX, (trainX.shape[0], 1, trainX.shape[1])) testX = np.reshape(testX, (testX.shape[0], 1, testX.shape[1]))
Model Building
model = Sequential() model.add(LSTM(return_sequences=True, input_shape=(None, 1), units=50)) model.add(Dropout(0.2)) model.add(LSTM( 100, return_sequences=False)) model.add(Dropout(0.2)) model.add(Dense(units=1)) model.add(Activation('linear')) start = time.time() model.compile(loss='mse', optimizer='adam') print ('compilation time : ', time.time() - start)
Model Fitting
model.fit( trainX, trainY, batch_size=128, epochs=10, validation_split=0.05)
Validation
window_size = 1 rmse_train, train_predict = predict_and_score(model, trainX, trainY) rmse_test, test_predict = predict_and_score(model, testX, testY) print("Training data score: %.2f RMSE" % rmse_train) print("Test data score: %.2f RMSE" % rmse_test)
Plotting Actual v Predicted
# Start with training predictions. train_predict_plot = np.empty_like(df) train_predict_plot[:, :] = np.nan train_predict_plot[window_size:len(train_predict) + window_size, :] = train_predict # Add test predictions. test_predict_plot = np.empty_like(df) test_predict_plot[:, :] = np.nan test_predict_plot[len(train_predict) + (window_size * 2) + 1:len(df) - 1, :] = test_predict # Create the plot. plt.figure(figsize = (15, 5)) plt.plot(scaler.inverse_transform(df), label = "True value") plt.plot(train_predict_plot, label = "Training set prediction") plt.plot(test_predict_plot, label = "Test set prediction") plt.xlabel("Time (Hours)") plt.ylabel("value") plt.title("Comparison true vs. predicted training / test") plt.legend() plt.show()
Forecasting / this is my problem
newX = ????????? predict =model.predict(newX) plt.plot(predict)