Keras Reshape: total size of the new array must be unchanged
I'm trying to use Keras Reshape function API to reshape the output of a glove embedding (4D shape: (?, 9, 20, 100)) down to 3D (?, 9, 2000). However, when I tried Reshape((9, 2000))(text_layer), an error pops up and says total size of the new array must be unchanged, even though 9 * 20 * 100 = 9 * 2000. Any ideas why? Code is attached.
text = Input(shape=(9, news_text.shape[1]), name='text')
text_layer = Embedding(
embedding_matrix.shape[0],
embedding_matrix.shape[1],
weights=[embedding_matrix],
input_length=news_text.shape[1]
)(text)
text_layer = Reshape((9, text_layer.shape[2] * text_layer.shape[3]))(text_layer)
1 answer

Remove the
input_length
parameter from theEmbedding
layer.It is strange and I don't know the reason but when you indicate the parameter
input_length
the error is thrown.Anyway, the
Embedding
layer receives the dimension of theInput
layer. It seems that the parameterinput_length
has a very specific use, to know the dimension of the tensor after using aFlatten
layer, etc.In this case, the
Embedding
layer obtains the shape of the output tensor from the input tensor, ignoring theinput_length
parameter.(If you set an invalid value does not throw error until you add the next layer. Note that the
input_lenght
and the resultshape
):>>> inp = Input(shape=(9,20)) >>> emb = Embedding(100,100, input_length=84) (inp) >>> emb <tf.Tensor 'embedding_5/embedding_lookup:0' shape=(?, 9, 20, 100) dtype=float32> >>> res = Reshape((9,2000)) (emb) Traceback (most recent call last): File "<stdin>", line 1, in <module> ...
However, it seems that the
input_length
parameter is in conflict when you add theReshape
layer.Finally:
text = Input(shape=(9, news_text.shape[1]), name='text') text_layer = Embedding( embedding_matrix.shape[0], embedding_matrix.shape[1], weights=[embedding_matrix], )(text) text_layer = Reshape((9, text_layer.shape[2] * text_layer.shape[3]))(text_layer)
See also questions close to this topic

Building a model on Keras correctly
I'm new to neural networks and Keras, and I want to build a CNN that predicts certain values of an image. (the three values predict the size, length, and width of a blur put on top of the image). All 3 values can range from 0 to 1, and I have a large data set.
I am not exactly sure how to build the CNN to do this though, as all the prototype codes that I have built so far give me predictions of the format
[1.,0.,0.]
instead of ranges between 0 and 1 for each value. On top of that, despite changing the number of epochs and decay value in the SGD optimizer, I don't get any change in my loss function at all. Can you please tell me where I am going wrong? Here is what I have so far:images, labels = load_dataset("images") # function that loads images images = np.asarray(images) # images are flattened 424*424 arrays (grayscale) labels = np.asarray(labels) # Lables are 3arrays, each value is float from 01 # I won't write this part but here I split into train_imgs and test_imgs model = keras.Sequential() # explicitly define SGD so that I can change the decay rate sgd = keras.optimizers.SGD(lr=0.01, decay=1e6, momentum=0.9, nesterov=True) model.add(keras.layers.Dense(32, input_shape=(424*424,) )) model.add(keras.layers.Activation('relu')) model.add(keras.layers.Dense(3, activation='softmax')) model.compile(loss='mean_squared_error',optimizer=sgd) # note: I also tried defining a weighted binary crossentropy but it changed nothing checkpoint_name = 'Weights{epoch:03d}{val_loss:.5f}.hdf5' checkpoint = ModelCheckpoint(checkpoint_name, monitor='val_loss', verbose = 0, save_best_only = True, mode ='auto') callbacks_list = [checkpoint] model.fit(train_imgs, train_labls, epochs=20, batch_size=32, validation_split = 0.2, callbacks=callbacks_list) predictions = model.predict(test_imgs) # make predictions on same test set!
Now I know that I am leaving out dropout layers, but I WANT the CNN to overfit my data, at this point I just want it to do anything. When I predict on the same set of images, I would hopefully get exact results, no? I'm not quite sure what I'm missing. Thanks for the help!

Switching from Keras to tf.keras spams my screen with #010
I've built a simple Keras model for experimentation within Amazon SageMaker. I'm using Python 3.5 TensorFlow 1.12.0. Recently I switched my model to use TensorFlow.keras, but doing so has resulted in the printing of
#010
repeatedly followed by#015
, while downloading image net weights and displaying batch accuracy during the fit call.For example, with verbose=1 in
model.fit
:Epoch 1/1
015 1/1563 [..............................]  ETA: 5:50:36  loss: 2.2798  acc: 0.1875#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015
3/1563 [..............................]  ETA: 1:57:18  loss: 2.3002  acc: 0.1146#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 5/1563 [..............................]  ETA: 1:10:36  loss: 2.3088  acc: 0.1062#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010
Does anyone know why this might be happening or how I can prevent this? Reproducing with a minimal example may require running through SageMaker, but the code I switched from Keras to tf.keras is from this example, and specifically the
trainer/start.py
file:# Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved. # # Licensed under the Apache License, Version 2.0 (the "License"). You # may not use this file except in compliance with the License. A copy of # the License is located at # # http://aws.amazon.com/apache2.0/ # # or in the "license" file accompanying this file. This file is # distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF # ANY KIND, either express or implied. See the License for the specific # language governing permissions and limitations under the License. from __future__ import absolute_import from __future__ import print_function import keras from keras.preprocessing.image import ImageDataGenerator from keras.models import Sequential from keras.layers import Dense, Dropout, Activation, Flatten from keras.layers import Conv2D, MaxPooling2D import os import numpy as np from trainer.environment import create_trainer_environment NUM_CLASSES = 10 EPOCHS = 10 NUM_PREDICTIONS = 20 MODEL_NAME = 'keras_cifar10_trained_model.h5' # the trainer environment contains useful information about env = create_trainer_environment() print('creating SageMaker trainer environment:\n%s' % str(env)) # getting the hyperparameters batch_size = env.hyperparameters.get('batch_size', object_type=int) data_augmentation = env.hyperparameters.get('data_augmentation', default=True, object_type=bool) learning_rate = env.hyperparameters.get('learning_rate', default=.0001, object_type=float) width_shift_range = env.hyperparameters.get('width_shift_range', object_type=float) height_shift_range = env.hyperparameters.get('height_shift_range', object_type=float) EPOCHS = env.hyperparameters.get('epochs', default=10, object_type=int) # reading data from train and test channels train_data = np.load(os.path.join(env.channel_dirs['train'], 'cifar10npzcompressed.npz')) (x_train, y_train) = train_data['x'], train_data['y'] test_data = np.load(os.path.join(env.channel_dirs['test'], 'cifar10npzcompressed.npz')) (x_test, y_test) = test_data['x'], test_data['y'] model = Sequential() model.add(Conv2D(32, (3, 3), padding='same', input_shape=x_train.shape[1:])) model.add(Activation('relu')) model.add(Conv2D(32, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) model.add(Conv2D(64, (3, 3), padding='same')) model.add(Activation('relu')) model.add(Conv2D(64, (3, 3))) model.add(Activation('relu')) model.add(MaxPooling2D(pool_size=(2, 2))) model.add(Dropout(0.25)) model.add(Flatten()) model.add(Dense(512)) model.add(Activation('relu')) model.add(Dropout(0.5)) model.add(Dense(NUM_CLASSES)) model.add(Activation('softmax')) # initiate RMSprop optimizer opt = keras.optimizers.rmsprop(lr=learning_rate, decay=1e6) # Let's train the model using RMSprop model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy']) x_train = x_train.astype('float32') x_test = x_test.astype('float32') x_train /= 255 x_test /= 255 if not data_augmentation: print('Not using data augmentation.') model.fit(x_train, y_train, batch_size=batch_size, epochs=EPOCHS, validation_data=(x_test, y_test), shuffle=True) else: print('Using realtime data augmentation.') # This will do preprocessing and real time data augmentation: data_generator = ImageDataGenerator( featurewise_center=False, # set input mean to 0 over the dataset samplewise_center=False, # set each sample mean to 0 featurewise_std_normalization=False, # divide inputs by std of the dataset samplewise_std_normalization=False, # divide each input by its std zca_whitening=False, # apply ZCA whitening rotation_range=0, # randomly rotate images in the range (degrees, 0 to 180) width_shift_range=width_shift_range, # randomly shift images horizontally (fraction of total width) height_shift_range=height_shift_range, # randomly shift images vertically (fraction of total height) horizontal_flip=True, # randomly flip images vertical_flip=False) # randomly flip images # Compute quantities required for featurewise normalization # (std, mean, and principal components if ZCA whitening is applied). data_generator.fit(x_train) # Fit the model on the batches generated by data_generator.flow(). data_generator_flow = data_generator.flow(x_train, y_train, batch_size=batch_size) model.fit_generator(data_generator_flow, epochs=EPOCHS, validation_data=(x_test, y_test), workers=4) # Save model and weights model_path = os.path.join(env.model_dir, MODEL_NAME) model.save(model_path) print('Saved trained model at %s ' % model_path) # Score trained model. scores = model.evaluate(x_test, y_test, verbose=1) print('Test loss:', scores[0]) print('Test accuracy:', scores[1])

Hardware Requirements for Pandas Dataframe, Tensorflow and LightGBM
I ask for help with information on the minimum hardware requirements, such as HD space, memory, processor, video card, for the use of Pandas Dataframe, Tensorflow and LightGBM libraries.
Thank you!

Keras  Load images from generator into memory
I would like to load my images into memory utilizing the image augmentation opions provided by the keras
ImageDataGenerator
. As such, I am creating my generator like so:testgen = ImageDataGenerator(preprocessing_function=keras.applications.mobilenet.preprocess_input) test_generator = testgen.flow_from_dataframe(dataframe=df_test, classes=class_labels, directory=data_dir, x_col=training_on, y_col=target, has_ext=True, class_mode="categorical", target_size=(224,224), batch_size=batch_size, seed = 1, shuffle=False)
Now I can get a single batch using:
x,y = next(test_generator)
However, I would like to store the entire dataset (possibly augmented) into
x
andy
. How can I achieve this? 
Generate .net model using CNN Machine learning
Can anyone please help me to generate(train) a .net model of CNN using images data, and make it usable for prediction, as we generate .h5 model. I am struggling to find resource that can help me, or if anyone have useful reference or tutorial that have guidance about this, please let me know.

No overfitting by increasing the number of epochs
I use a feed foreward neural network with one hidden layer for my thesis. Threby I have 600 training data and 104 input and output values. There I want to show now the properties of the neural network and also want to show the overfitting when I increase the number of epochs. To do so, I first wanted to find the optimum for learning rate and number of hidden nodes, where I got the following results:
Based on that I decided to choose a learning rate of 0.0125 and 250 hidden nodes. But by using this set of parameters, I still have no overfitting when I increase the number of epochs, which can be seen here:
In this plot I showed in blue my old set of parameters and in theory I wanted to show how it improve when I use the best set of parameters, but it's just varying a bit. I also tested it until epoch 1000 but the accuracy with this value was still 0.830.
Does someone has an idea why this happen?
Thanks a lot for your help!

Serving multiple deep learning models from cluster
I was thinking about how one should deploy multiple models for use. I am currently dealing with tensorflow. I was referring this and this article.
But I am not able to find any article which targets need to serve several models distributed manner. Q.1. Does tensorflow serving serve models off from single machine? Is there any way to set up a cluster of machines running tensorflow serving? So that multiple machines serve same model somewhat working as master and slave or say load balance between them while serving different models.
Q.2. Does similar functionality exist for other deep learning frameworks, say keras, mxnet etc (not just restricting to tensorflow and serving models from different frameworks)?

Multi horizon forecast with multiple time series using LSTM
I am new to LSTM in timeseries. Most info on internet is for a single time series and for onestep forecasting. My problem is about multiple time series for multi horizon forecast, using LSTM in R.
1) Shape of X_train?
I want to use LSTM to forecast 100 time series, all monthly and length 54 months. The output should be 6 months forecast, so I need a 100*6 matrix as an output. I use sliding window approach with input window is 15 months, and output is 6 months. Therefore I have 34 different (X,Y) vectors for each time series. I reshaped X_train as (3400, 15, 1), is this correct?
2) How to choose "batch_size" and "units"?
In order to get 100*6 matrix as an output, how can I choose "batch_size" and "units"? In documentation of layer_dense and layer_lstm functions units is defined as dimensionality of the output space. So should I set units=6? I have difficulty in understanding what batch_size and units refer to, so any explanation would be appreciated.

When to use GlobalAveragePooling1D and when to use GlobalMaxPooling1D while using Keras for an LSTM model?
I have to make LSTM classification model for some text and I am confused between GlobalAveragePooling1D and GlobalMaxPooling1D in the pooling layer while using keras. Which one should I use and what are the things to consider while deciding a particular choice.

Coverting LSTM predicted data to original format
What I am trying to achieve
My code below is working on predicting "NG open" price but mostly in a scalar format. However, I want to compare the predicted price with actual values such as $4.30, $3.20, $2.60 etc for all the 100 rows. I am somehow not able to find a code that coverts my predicted values to forecasted numbers.
Dataset First few lines.
Contract NGLast NGOpen NGHigh NGLow NGVolumes COOpen COHigh COLow 20181201 4.487 4.50 4.60 4.03 100,000 56.00 58.00 50.00 20190101 4.450 4.52 4.61 4.11 93000 51.00 53.00 45.00
Code
import pandas as pd import numpy as np import matplotlib.pyplot as plt from keras.layers import Dense from keras.models import Sequential from keras.layers import LSTM import date time from keras import metrics from sklearn.preprocessing import MinMaxScaler data = pd.read_excel("C:\Futures\Futures.xls") data['Contract'] = pd.to_datetime(data['Contract'],unit='s').dt.date data['NG Last'] = data['NG Last'].str.rstrip('s') data['CO Last'] = data['CO Last'].str.rstrip('s') COHigh = np.array([data.iloc[:,8]]) COLow = np.array([data.iloc[:,9]]) NGLast = np.array([data.iloc[:,1]]) NGOpen = np.array([data.iloc[:,2]]) NGHigh = np.array([data.iloc[:,3]]) X = np.concatenate([COHigh,COLow, NGLast,NGOpen], axis =0) X = np.transpose(X) Y = NGHigh Y = np.transpose(Y) scaler = MinMaxScaler() scaler.fit(X) X = scaler.transform(X) scaler.fit(Y) Y = scaler.transform(Y) X = np.reshape(X,(X.shape[0],1,X.shape[1])) print(X.shape) model = Sequential() model.add(LSTM(100,activation='tanh',input_shape=(1,4),recurrent_activation='hard_sigmoid')) model.add(Dense(1)) model.compile(loss='mean_squared_error', optimizer='rmsprop', metrics = [metrics.mae]) model.fit(X,Y,epochs = 10,batch_size=1,verbose=2) Predict = model.predict(X,verbose=1) normalized = scaler.transform(Predict) print(normalized) inversed = scaler.inverse_transform(normalized) print(inversed)

Use extrapolation/interpolation to resize an array
I have an array in the shape
(22, 48)
and I'm comparing it to another array with shape(881,1121)
. In order to compare them I would like the shapes to map. I figured I could to do this in python using a form of extrapolation/interpolation, but I can't seem to find anything that would do it without simply filling the array with zeros or repeated values.Is there a simple function that would do it for me or another method to do it?
Edit 1:
By compare, I mean I'm doing further analysis after they've been resized. Specifically, finding a skill score using the mean square error, which means I need them to be the same size/shape.

Pandas data reshaping, turn multiple rows with the same index but different values into many columns based on incidence
I have the following table in pandas
user_id idaggregate_info num_events num_lark_convo_events num_meals_logged num_breakfasts num_lunches num_dinners num_snacks total_activity sleep_duration num_activity_events num_weights num_notifs idusermission completed mission_delta 0 0 406 94 20 7 2 2 2 1 4456 47738 72 0 18 1426 0 NaT 1 1 1247 121 48 26 8 7 2 9 48695 37560 53 14 48 1379 1 7 days 10:04:28 2 1 1247 121 48 26 8 7 2 9 48695 37560 53 14 48 1379 1 NaT 3 2 2088 356 32 15 6 6 1 2 41598 184113 314 1 21 967 1 8 days 00:03:05 4 2 2088 356 32 15 6 6 1 2 41598 184113 314 1 21 967 1 NaT
Some user_ids have multiple lines that are identical except for their different mission_delta values. How do I transform this into one line for each id, with a columns named "mission_delta_1", "mission_delta_2" (the number of them vary, it could be 1 per user_id to maybe 5 per user_id so naming has to be iterative_ etc so output would be:
user_id idaggregate_info num_events num_lark_convo_events num_meals_logged num_breakfasts num_lunches num_dinners num_snacks total_activity sleep_duration num_activity_events num_weights num_notifs idusermission completed mission_delta_1 mission_delta_2 0 0 406 94 20 7 2 2 2 1 4456 47738 72 0 18 1426 0 NaT 1 1 1247 121 48 26 8 7 2 9 48695 37560 53 14 48 1379 1 7 days 10:04:28 NaT 2 2 2088 356 32 15 6 6 1 2 41598 184113 314 1 21 967 1 8 days 00:03:05 NaT
Not a duplicate as those address exploding all columns, there is just one that needs to be unstacked. The solutions offered in the duplicate link fail:
df.groupby(level=0).apply(lambda x: pd.Series(x.values.flatten()))
produces the same df as the original with different labels
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 0 0 406 94 20 7 2 2 2 1 4456 47738 72 0 18 1426 0 NaT 1 1 1247 121 48 26 8 7 2 9 48695 37560 53 14 48 1379 1 7 days 10:04:28 2 1 1247 121 48 26 8 7 2 9 48695 37560 53 14 48 1379 1 NaT 3 2 2088 356 32 15 6 6 1 2 41598 184113 314 1 21 967 1 8 days 00:03:05
The next options:
result2.groupby(level=0).apply(lambda x: pd.Series(x.stack().values))
produces:
0 0 0 1 406 2 94 3 20 4 7
and
df.groupby(level=0).apply(lambda x: x.values.ravel()).apply(pd.Series)
produces the original dataframe:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 0 0 406 94 20 7 2 2 2 1 4456 47738 72 0 18 1426 0 NaT 1 1 1247 121 48 26 8 7 2 9 48695 37560 53 14 48 1379 1 7 days 10:04:28 2 1 1247 121 48 26 8 7 2 9 48695 37560 53 14 48 1379 1 NaT 3 2 2088 356 32 15 6 6 1 2 41598 184113 314 1 21 967 1 8 days 00:03:05
In essence, I want to turn a df:
id mission_delta 0 NaT 1 1 day 1 2 days 1 1 day 2 5 days 2 NaT
into
id mission_delta1 mission_delta_2 mission_delta_3 0 NaT NaT NaT 1 1 day 2 days 1 day 2 5 days NaT NaT

R wide to long reshape with column names
I have data in this format
A1 A2 B1 B2 C1 C2 10 5 11 5 21 10
And I want to convert it to:
1 2 A 10 5 B 11 5 C 21 10
How can I do it in R?