Concatenating vectors for CNN in Keras/tensorflow

I am trying to concatenate the flattened output of a CNN, and a vector of scalar values. I am trying to influence the network by both the image and the vector. So the output of the flattened CNN size is (1,1024), and the vector I want to concatenate is (1,5).

Of course keras wants them to be similar sizes. So what is the best practice in this situation? Expand the vector to 1024 with zeroes? I am trying to make the vector have an impact, what other options can I do? I am trying to do something similar to here (https://arxiv.org/abs/1603.02199).

1 answer

  • answered 2018-07-15 16:57 Aldream

    Why not concatenating them over the last dimension, to obtain a tensor of shape (1, 1029)?

    from keras.models import Model
    from keras.layers import Input, Concatenate
    
    img = Input(shape=(1,1024))
    vec = Input(shape=(1,5))
    res = Concatenate(axis=-1)([img, vec])
    model = Model(inputs=[img, vec], outputs=res)
    model.summary()
    # _______________________________________________________________________________
    # Layer (type)                    Output Shape         Param #     Connected to
    # ===============================================================================
    # input_1 (InputLayer)            (None, 1, 1024)      0              
    # _______________________________________________________________________________
    # input_2 (InputLayer)            (None, 1, 5)         0              
    # _______________________________________________________________________________
    # concatenate_1 (Concatenate)     (None, 1, 1029)      0           input_1[0][0]
    #                                                                  input_2[0][0]
    # ===============================================================================
    # Total params: 0
    # Trainable params: 0
    # Non-trainable params: 0
    # _______________________________________________________________________________