Browsed by
Tag: adf

Incrementally load data from SQL database to azure data lake using synapse

Incrementally load data from SQL database to azure data lake using synapse

Comments 9 Comments

In this post, we will cover how to create a simple and fully dynamic synapse pipeline to incrementally load data from an SQL database to parquet files stored in a data lake. Why should we move data from a database to a data lake In the traditional data warehouse approach, it is very common to move all the data sources and historize them into an ODS database.However, moving and storing data from all the data sources into a single database…

Read More Read More

pagination in azure data factory
How to perform pagination in azure data factory using Rest API

How to perform pagination in azure data factory using Rest API

Comments 8 Comments

REST APIs typically have some mechanism to transmit large volumes of records broken up into pages of results. In this post, I describe how to perform pagination in azure data factory or Synapse pipeline using the Azure REST API. What is pagination Azure REST API limits the number of items it returns per result, so when the results are too large to be returned in one response we need need to make multiple calls to the rest API. So when the results…

Read More Read More