Give segmentation mask as input to segmentation network?

I want to train a semantic segmentation network. I have a dataset with Red, Blue, Green, Infrared, Low segmentation mask, high segmentation mask.

Low segmentation mask consists of class labels as pixel values like 0 for the forest, 1 for water. with dimension (256x256x1).

High segmentation mask consists of class labels as pixel values like 0 for the forest, 1 for water. with dimension (256x256x1). but it is more finely labeled according to the images.

Can I input the RGB IR and low segmentation mask as input to the semantic segmentation network like UNET? with outputs high segmentation mask.

I have very less data like only 1000 images (with high segmentation masks) and have to predict on 5000 images (whose high segmentation mask is not accessible to me). So that's why I want to try this technique.

I am currently working on CGAN implementation in which the loss of Generator is "categorical_crossentropy". The same is the case with UNET architecture.