CURL - problems pulling the data from SharePoint API error

I am trying to download some data in the form of Json from a SharePoint API. I was able to do it using WGET:

wget --user user --password password --header='Accept: application/json;odata=verbose' "URL"

and I am trying to do the same thing with CURL.

I tried using different headers and authentication mechanisms, but it always gives me the bad request error. Below are the commands I tried:

curl -v --user user:password "URL"

curl -v --user user:password -H 'Accept: application/json;odata=verbose' "URL"

curl -v --ntlm --user user:password -H 'Accept: application/xml' "URL"

curl -v --anyauth --user user:password -H 'Accept: application/xml' "URL"

Below is the output I get:

* About to connect() to sharepoint2.domain.com port 80 (#0)
*   Trying xx.xx.xx.xx... connected
* Connected to sharepoint2.domain.com (xx.xx.xx.xx) port 80 (#0)
> GET <path> HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: sharepoint2.domain.com
> Accept: application/json;odata=verbose
>
< HTTP/1.1 400 Bad Request
< Content-Type: text/html; charset=us-ascii
< Server: Microsoft-HTTPAPI/2.0
< Date: Mon, 12 Feb 2018 21:45:06 GMT
< Connection: close
< Content-Length: 311
<
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN""http://www.w3.org/TR/html4/strict.dtd">
<HTML><HEAD><TITLE>Bad Request</TITLE>
<META HTTP-EQUIV="Content-Type" Content="text/html; charset=us-ascii"></HEAD>
<BODY><h2>Bad Request</h2>
<hr><p>HTTP Error 400. The request is badly formed.</p>
</BODY></HTML>
* Closing connection #0

I also tried using --ntlm and --negotiate arguments, as explained on this question:

Using curl with NTLM auth to make a post is failing

curl -v --ntlm --negotiate --user user:password -H 'Accept: application/json;odata=verbose' "URL"

But it still gives me the 400 bad request error!

I am not sure what's going wrong. How do I solve the issue and pull the data with CURL ?

UPDATE:

I tried putting the -L in the command in case there is an issue with redirect as pointed out here: Why does curl not work, but wget works? and here: https://askubuntu.com/questions/938391/url-works-in-wget-but-not-in-curl-related-to-apt,

but it's still the same error :( . I don't think it is really a redirect issue because I am getting the 400 bad request response from the API right?.

My URL is like this: .../_api/web/lists/GetByTitle('new list')/items. I guessed the error has something to do with GetByTitle('new list')..so I tried giving it a different URL: .../_api/web/lists and it didn't give me the 400 bad request Error! but instead I got HTTP/1.1 401 Unauthorized. SO my intial error had something to do with encoding the URL ?

I wanted to see why Wget is working and used the --debug option to print out the whole log and I found something interesting:

[~ /data/home/user/pir]$wget --user user --debug --password password --header='Accept: application/json;odata=verbose' "URL"
Setting --password (password) to password
Setting --header (header) to Accept: application/json;odata=verbose
DEBUG output created by Wget 1.12 on linux-gnu.

--2018-02-13 07:13:08--  URL
Host “sharepoint2.domain.com” has not issued a general basic challenge.
Resolving sharepoint2.domain.com... xx.xx.xx.xx
Caching sharepoint2.domain.com => xx.xx.xx.xx
Connecting to sharepoint2.domain.com|xx.xx.xx.xx|:80... connected.
Created socket 3.
Releasing 0x0000000002229e70 (new refcount 1).

---request begin---
GET path HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: application/json;odata=verbose
Host: sharepoint2.domain.com
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 401 Unauthorized
Content-Type: text/plain; charset=utf-8
Server: Microsoft-IIS/8.5
SPRequestGuid: 90174a9e-d921-1018-c709-d64cd2403a12
request-id: 90174a9e-d921-1018-c709-d64cd2403a12
SPRequestDuration: 29
SPIisLatency: 1
WWW-Authenticate: NTLM
WWW-Authenticate: Negotiate
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 15.0.0.4859
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
Date: Tue, 13 Feb 2018 13:13:08 GMT
Connection: keep-alive
Content-Length: 16

---response end---
401 Unauthorized
Registered socket 3 for persistent reuse.
Skipping 16 bytes of body: [401 UNAUTHORIZED] done.
Empty NTLM message, starting transaction.
Creating a type-1 NTLM message.
Reusing existing connection to sharepoint2.domain.com:80.
Reusing fd 3.

---request begin---
GET <path> HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: application/json;odata=verbose
Host: sharepoint2.domain.com
Connection: Keep-Alive
Authorization: NTLM TlRMTVNTUAABAAAAAgIAAAAAAAAgAAAAAAAAACAAAAA=

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/8.5
WWW-Authenticate: NTLM TlRMTVNTUAACAAAABAAEADgAAAAGAoEC33/kD9suGK8AAAAAAAAAAOAA4AA8AAAABgOAJQAAAA9DT1JQAgAIAEMATwBSAFAAAQAUAFcAUABOAE8AQQBTAFAAQwAwADYABAAsAGMAbwByAHAALgBiAGEAbgBrAG8AZgBhAG0AZQByAGkAYwBhAC4AYwBvAG0AAwBSAHcAcABuAG8AYQBzAHAAYwAwADYALgB1AHMAcABuAG8AYwB0AC4AYQBtAHIAcwAuAGIAYQBuAGsAbwBmAGEAbQBlAHIAaQBjAGEALgBjAG8AbQAFACIAYgBhAG4AawBvAGYAYQBtAGUAcgBpAGMAYQAuAGMAbwBtAAcACADOsXRkzKTTAQAAAAA=
SPRequestGuid: 90174a9e-8925-1018-c709-d7cd9dd3150d
request-id: 90174a9e-8925-1018-c709-d7cd9dd3150d
SPRequestDuration: 1
SPIisLatency: 0
WWW-Authenticate: Negotiate
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 15.0.0.4859
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
Date: Tue, 13 Feb 2018 13:13:08 GMT
Connection: keep-alive
Content-Length: 0

---response end---
401 Unauthorized
Skipping 0 bytes of body: [] done.
Received a type-2 NTLM message.
Creating a type-3 NTLM message.
Reusing existing connection to sharepoint2.domain.com:80.
Reusing fd 3.

---request begin---
GET <path> HTTP/1.0
User-Agent: Wget/1.12 (linux-gnu)
Accept: application/json;odata=verbose
Host: sharepoint2.domain.com
Connection: Keep-Alive
Authorization: NTLM TlRMTVNTUAADAAAAGAAYAEcAAAAYABgAXwAAAAAAAABAAAAABwAHAEAAAAAAAAAARwAAAAAAAAB3AAAAAYIAAHpraDl4Y2w02/8GDGRzkuXXcgWtte6c6h6ggw8yerZmu6ewFZpvd9hNAYfP/XDSQWvE5Eukuy0=

---request end---
HTTP request sent, awaiting response...
---response begin---
HTTP/1.1 200 OK
Cache-Control: private, max-age=0
Content-Type: application/json;odata=verbose;charset=utf-8
Expires: Mon, 29 Jan 2018 13:13:08 GMT
Last-Modified: Tue, 13 Feb 2018 13:13:08 GMT
Server: Microsoft-IIS/8.5
X-SharePointHealthScore: 0
SPClientServiceRequestDuration: 4936
X-AspNet-Version: 4.0.30319
SPRequestGuid: 90174a9e-c925-1018-c709-d9b4ad681772
request-id: 90174a9e-c925-1018-c709-d9b4ad681772
Persistent-Auth: true
X-Powered-By: ASP.NET
MicrosoftSharePointTeamServices: 15.0.0.4859
X-Content-Type-Options: nosniff
X-MS-InvokeApp: 1; RequireReadOnly
Date: Tue, 13 Feb 2018 13:13:13 GMT
Connection: close

---response end---
200 OK
Length: unspecified [application/json]
Saving to: “items.5”

The Wget request fails with HTTP/1.1 401 Unauthorized but it sill tries executing the request a total of 3 times and it gets connected 3rd time. and it does this all the time.

Could it be that I have to pass some argument to CURL to keep trying even after it failed with HTTP/1.1 401 Unauthorized ?

Any help would be appreciated. Thank you.