How to use Rest API as a source in dataflow in Azure data factory?

I am new to ADF. I am creating a small demo where I am trying to hit a sample API "https://dummy.restapiexample.com/api/v1/employees" and inserting data into the Employee table in the SQL database.

However, I would also like to add data flow to perform few select and other transformations on other related tables.
Below are the things that I have done so far in azure data factory:

  1. Created a linked service called "EmployeeRestService" to communicate with the above api.
    enter image description here

  2. Created a linked service called "AzureSqlDatabase" to communicate with azure sql database enter image description here

  3. Created a dataset "EmployeeApiDataset" of type "Rest" and linked it with "EmployeeRestService".

    enter image description here

  4. Created a dataset of type Azure sql database and Selected "EmployeeTable".

  5. Added one data flow (Now here I am getting the problem)
    data flow source settings

    data flow source options

And here is the error snapshot:
Error

In the data flow --> Source Settings, I selected "EmployeeApiDataset". But when I am trying to preview the data by enabling the debug, I am getting below error (refer above snapshot):
at Source 'EmployeeApi': Rest Sink - Error response received from the server (url:http://dummy.restapiexample.com/api/v1/employees,request body: None, request method: GET, status code: 429), response body: Some(

Is this a correct way of doing adding Rest data source in data flow? Note: I do not want to create an Azure data lake to store API results.

Can anybody suggest to me why I am getting the above error and what is the correct way of doing this?

1 answer

  • answered 2021-10-13 03:57 NiharikaMoola-MT

    As @esqew mentioned in the comments, there is nothing wrong with your dataflow Source settings. As you are using online dummy Rest API as your source dataset, there are high chances that too many requests are made to call the Rest API by users.

    1. I also got the same error when tried.

    enter image description here

    1. Also got the same 429 error when tested the URL.

    enter image description here

    1. I just hit the refresh and got the data in the source data preview.

    enter image description here

How many English words
do you know?
Test your English vocabulary size, and measure
how many words do you know
Online Test
Powered by Examplum