Fiting a cuboid inside a segmented layout map

I need to find a 3D cuboid that fit into corresponding images and the camera position into this cuboid.

Segmented layout Polygon Edges

First image is a label map with at most 5 labels (left wall, front wall, right wall, floor, ceiling).

Second image is the rendered polygons I found in the first image, each polygon being either a wall or floor or ceiling. Polygons are found by applying scipy.spatial.ConvexHull function to each label blobs.

I also can compute the intersect corner points of those polygons if it can help.

Ressources I found: