Issue with RayCast picking algorithm

I'm trying to get a ray direction so I can pick vertices using my mouse. The following code works when rotating on the X or Y axis independently, it goes slightly off course when the view is rotated on X and Y axis, and the Z axis has no effect whatsoever. I don't know what I'm missing, I expected the inverse view to correct for the z axis ass well. Any help is appreciated.

static Vec3 RayCast3(Vec2 _homogeneousMouseCoordinates, float _fov, float _near, float _far, float _aspect, Mat4 _view)
    float dx = tanf(_fov*0.5f) * _homogeneousMouseCoordinates.x *_aspect;
    float dy = tanf(_fov*0.5f) * _homogeneousMouseCoordinates.y;

    Vec3 p1 = { dx*_near,   dy*_near,   _near };
    Vec3 p2 = { dx*_far,    dy*_far,    _far };

    bool valid;
    Mat3 invRotView = Mat3::GetInverse(Mat4::ExtractRotate(_view), valid);

    return Vec3::Normalize((p2 * invRotView) - (p1 * invRotView));