Accessing GpuMat on another GPU

I'm pretty sure this is extremely dodgy code, but I just want to make sure. Assume I have two GPUs, and let us skip error checking:

cv::cuda::GpuMat foo(5, 5, CV_8UC1);<uchar>(2, 2) = 128;
cudaSetDevice(1);<uchar>(4, 4) = 192;

Is GpuMat or the CUDA runtime smart enough to know that foo data pointer points to GPU 0, or am I (as I suspect) accessing potentially unowned memory on GPU 1 with the<uchar>(4, 4) = 192; line? I'm using OpenCV 3.2 and CUDA 8.0 on Ubuntu 16.04.