AWS CloudFront Error - The bucket you are attempting to access must be addressed using the specified endpoint

I get this XML error when accessing my static hosting S3 bucket via the CloudFront URL.

The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

I had a bucket called "mywebsite" located in Sydney, deleted it and created a bucket with the same name in N Virginia. I've tried disabling and recreating the CF distribution but it I think it's still pointing to the old bucket? The original bucket has disappeared from S3 already.

What should I do?