Best Practices in Azure Synapse

Microsoft has rolled out an exciting new feature for Azure Synapse, specifically targeting data warehousing. Users can now leverage Change Data Capture (CDC) for the Analytical Store within Cosmos DB. This advancement allows for more efficient data integration through ETL or ELT processes, particularly by focusing on only the data that has changed. The significant update is that the analytical store of Azure Cosmos DB now supports CDC for both the NoSQL and MongoDB APIs.

To delve deeper into these capabilities, check out the following video:

This new functionality is currently available in Public Preview and enables continuous access to altered data—whether inserted, updated, or deleted—from the analytical store. The CDC feature operates seamlessly within Azure Synapse Analytics and Azure Data Factory, boasting scalability and a no-code experience for handling substantial data volumes.

The Zero ETL Approach in Data Engineering

As part of its strategy, Microsoft is aligning itself with industry leaders like Google and AWS by providing users with scalable, user-friendly, and nearly code-free data integration solutions. This approach is in line with the Zero ETL philosophy, allowing customers to incur minimal costs for data provision or direct analysis in the source system.

For further insights into Zero ETL, refer to the following video:

With this new feature, Microsoft users can access incremental data from a Cosmos DB container using either Azure Synapse Analytics or Azure Data Factory, provided that the Cosmos DB account is configured for Synapse Link, and the analytical store is activated on a new or existing container.

Additionally, along with this release, Microsoft has introduced a new ARM template designed to deploy an Azure Data Explorer database with a connection to Cosmos DB, which aims to streamline the process of deploying an Azure Data Explorer cluster.