mex matlab function c++ gives wrong answer relative to R c++ function

I've got an issue that I cannot get the answer after mex'ing c++ function. It gives zeros always no matter what.

code:

https://github.com/jstriaukas/codequestions/tree/master/code

main.m gives the answer in matlab. linSolver.cpp contains mexFunction that I think I made a mistake in but cannot find it. Any ideas?