Solvepnp CPP function gives different results

I am trying to undersatand how solvepnp works. I tried giving 8 corner points of an object (its 2D - 3D correspondence) and intrinsics of camera. I get the result as

rvec

-1.59 -1.6 -0.89

Tvec

-18 3000 1400

When i tried reprojecting using output of solvepnp rvec and tvec, the points get properly overlayed on the input image. When I increment value of one of my image points by one(say (400,300) earlier and now I just changed to (401,300)). My rvec changes sign and tvec value drastically varies . Now it is

rvec

1.6 1.6 -0.8

Tvec

9 -900 5000

Reprojection also fails. I am curious on how this change occurs with the minor change. How can it be solved?