Change data capture

Real-Time Data Streaming and Search Indexing with Azure - Part 1 - Introduction and Overview

Johan Olsson

• 5 min read
Real-Time Data Streaming and Search Indexing with Azure - Part 1 - Introduction and Overview

In today's digital age, we rely on seamless data integration across multiple systems to handle everyday tasks. Imagine ordering products from your favorite online store. You expect the process to be smooth and efficient, from payment and stock management to shipping and order confirmation. Behind the scenes, various systems work together to make this happen.

To address the challenges of such complex processes, several solutions exist. Today, we'll dive into the world of real-time data streaming and search indexing within Azure.

This article kicks off a multi-part series where we'll explore this topic in depth. We'll start with an introduction to the overall solution, including high-level architecture diagrams, and discuss the key components involved.
Ready to get started? Let's dive in!

High level architecture

The diagram below illustrates the design I plan to implement. Essentially, I aim to create a unified system where any changes made to a database trigger a stream of updates to a separate datastore, specifically Azure AI Search in this case. I've been eager to experiment with Change Data Capture (CDC) using Azure Event Hub and Debezium for a while now. While the components like SQL Server, CDC, Debezium, and Azure Event Hub can be easily swapped out for Azure CosmosDB with its built-in change stream, CDC with Debezium could be particularly useful for existing solutions that already use SQL Server.

This architecture could be adapted to a varity of use-cases that we will bring up as a seprate topic later in this post.

Components

Applications

Sample applications will sporadically make changes to the database. These applications can range from web applications to mobile apps, each performing various CRUD (Create, Read, Update, Delete) operations on the database. One application will use Azure Search to demonstrate near real-time search capabilities. This application will showcase how quickly changes in the database are reflected in the search results, providing a seamless user experience. Additionally, these applications will help in testing the robustness and efficiency of the CDC pipeline, ensuring that all changes are accurately captured and propagated to the Azure AI Search.

Azure SQL Server

Azure SQL Server is a robust platform for storing and managing relational data. Since its inception in 1989, it has evolved from on-premise servers to highly scalable and secure PaaS and IaaS services in Azure.

CDC (Change Data Capture)

Change Data Capture (CDC) is a feature that captures changes made to data in a database and makes this change data available for further use. In the context of Azure SQL Server, CDC captures insert, update, and delete activities applied to tables and stores the details of these changes in change tables. These change tables can then be consumed by applications or services to keep other data stores in sync, trigger business processes, or maintain audit logs.

CDC is particularly useful in scenarios where you need to:

  • Synchronize Data: Keep multiple data stores in sync by capturing changes in the primary database and applying them to secondary databases or data warehouses.
  • Audit Changes: Maintain a history of changes for auditing purposes, allowing you to track who changed what and when.
  • Trigger Business Processes: Initiate business processes or workflows in response to specific data changes, such as sending notifications or updating related systems.
  • Real-Time Analytics: Enable real-time analytics by streaming changes to analytics platforms or dashboards.

CDC works in conjunction with Azure Event Hub and Debezium to capture and stream changes from Azure SQL Server to Azure AI Search. This ensures that any changes made to the database are quickly reflected in the search index, providing up-to-date search results for users. CDC is also a fundamental element of event-driven architectures. It allows various loosely coupled services to interact by transmitting events through a shared data streaming platform. This approach facilitates real-time data processing and integration across different systems, enhancing the overall responsiveness and scalability of the architecture.

Debezium

Debezium is an open-source distributed platform for change data capture (CDC). It captures row-level changes in your databases and streams these changes in real-time to various downstream systems. Debezium supports a wide range of databases, including MySQL, PostgreSQL, MongoDB, and SQL Server, making it a versatile tool for integrating different data sources.

Key features of Debezium include:

  • Real-Time Data Streaming: Debezium captures changes as they happen and streams them in real-time, ensuring that downstream systems are always up-to-date.
  • Scalability: Designed to handle high-throughput environments, Debezium can scale to meet the demands of large-scale data processing.
  • Fault Tolerance: Debezium is built to be resilient, with mechanisms to handle failures and ensure data consistency.
  • Schema Evolution: It supports schema changes, allowing your database schema to evolve without disrupting the CDC pipeline.

Debezium works alongside Azure SQL Server and Azure Event Hub to capture changes from the database and stream them to Azure AI Search. By leveraging Debezium, we can ensure that any modifications to the database are reflected in the search index, providing users with accurate and up-to-date search results.

Azure Event Hub

Azure Event Hub is a powerful tool for handling lots of data quickly. It can take in and process millions of events every second, making it perfect for real-time data tasks.

In this setup, Event Hub is a key player. It's designed to work well with Debezium because it's Kafka-enabled. This means it can easily handle changes from our database and send them to Azure AI Search (via Azure Functions). This way, any updates in the database are quickly reflected in the search results, keeping everything up-to-date for users.

Azure Functions

Azure Functions will be used to process the change events captured by Debezium and sent to Azure Event Hub. When a change event is received, an Azure Function will be triggered to update the Azure AI Search index. This ensures that any updates in the database are quickly reflected in the search results, providing users with accurate and up-to-date information.

Azure AI Search is a great tool that makes it easy to add advanced search features to your applications. It offers full-text search, faceted navigation, and advanced filtering, so finding the right information is quick and simple.

In our setup, we will push data changes to Azure AI Search, which will then index these changes. This allows users to quickly and efficiently search through large volumes of data, with near real-time updates when data changes occur.

It also has cool features like cognitive search, which uses AI to pull insights from your data, and built-in analyzers that handle different languages and data types. With these features, we can give users a smooth and intuitive search experience, helping them find what they need with ease. Maybe that will be a short spinoff topic in the near future.

Use Cases for CDC

Change Data Capture (CDC) can be applied in various scenarios to enhance data management and processing. For data synchronization, CDC keeps multiple databases in sync by capturing changes in the primary database and applying them to secondary databases or data warehouses. For auditing, CDC maintains a detailed history of changes for compliance and auditing purposes. For real-time analytics, CDC streams changes to analytics platforms for up-to-date insights and decision-making. In event-driven architectures, CDC triggers business processes or workflows in response to specific data changes, enabling real-time reactions to data events that could fulfill a business value.

Wrapping Up

And that's a wrap! We've covered the key components for this experiment: Azure Event Hub, Azure Functions, and Azure AI Search. Each of these plays a vital role in keeping your data up-to-date and your search results precise.

Coming up next, we'll get into the details to setup this infrastructure in Azure.

So, grab a snack and a cup of coffee, and get ready to dive in. This series of articles is just beginning, and you won't want to miss the next part. Stay tuned!