synapse Archives - Ben's Blog

Load data from Synapse to Fabric

2023-11-05 Ben Comments 0 Comment

In this short post, we will cover how to load data from Synapse to Fabric or from Azure Data Factory using Sznapse/ADF pipelines. Note that at the time of writing this post the feature to load data from Synapse to Fabric is still in preview.Note that loading data from Synapse to Fabric using Synapse notebooks was already feasible, but these notebooks don’t natively support connections to on-premise SQL databases. Why loading data from Synapse to Fabric In the best world,…

Incrementally load data from SQL database to azure data lake using synapse

2022-12-14 Ben Comments 9 Comments

In this post, we will cover how to create a simple and fully dynamic synapse pipeline to incrementally load data from an SQL database to parquet files stored in a data lake. Why should we move data from a database to a data lake In the traditional data warehouse approach, it is very common to move all the data sources and historize them into an ODS database.However, moving and storing data from all the data sources into a single database…

How to perform pagination in azure data factory using Rest API

2022-07-01 Ben Comments 8 Comments

REST APIs typically have some mechanism to transmit large volumes of records broken up into pages of results. In this post, I describe how to perform pagination in azure data factory or Synapse pipeline using the Azure REST API. What is pagination Azure REST API limits the number of items it returns per result, so when the results are too large to be returned in one response we need need to make multiple calls to the rest API. So when the results…