A featured contribution from Leadership Perspectives: a curated forum reserved for leaders nominated by our subscribers and vetted by the Construction Tech Review Advisory Board.

YDUQS

Alexandre Gonzalez, Chief Technology Officer

Bridging the Gap: Effectively Managing Data in the Era of Digital Apps and Legacies

Alexandre Gonzalez

It is common to associate digital transformation with UX, agile methodologies, apps, microservices, and cloud computing. When we think about startups with no legacies, this combination seems perfect. However, for large corporations, the situation becomes more challenging as legacy integration remains a key component behind the scenes to support the new digital mindset. Unfortunately, legacy integration is often underestimated or neglected due to poor documentation and reliance on deprecated technologies. However, the challenge of handling real-time data to feed digital apps will eventually arise.

Traditional approaches typically rely on batch processing and data replicas, but the volume of data and infrastructure restrictions can quickly lead to frustration. Batch processing takes time, and even incremental replication can be difficult when dealing with outdated legacy systems that have incompatible data schemas. As the situation progresses, it becomes increasingly challenging when additional new digital apps require different types of data, yet they are still dependent on the same legacy systems. Eventually, the legacy systems may reach a breaking point where they are unable to handle the high volume of requests, leading to crashes and creating significant problems for both the legacy systems and the digital apps. The pressure mounts as the demand exceeds the capabilities of the legacy infrastructure.

Our approach involves adding a new computing layer for CDC, which enhances the process of data integration

Undoubtedly, this challenge is also present in the education sector. For instance, when it comes to attracting students, but not limited to that, we face the task of making course offers readily available across multiple digital channels in real-time, all supported by our existing legacy systems. In response to this need, our company has initiated a project to address and handle these sorts of requirements.

Idearevolves around harnessing the power of an event-driven architecture facilitated by change data capture (CDC) at the database level, all while ensuring transparency to the legacy application. By utilizing CDC, we can capture data in real-time and effortlessly stream it to multiple digital app consumers.

Our approach involves adding a new computing layer for CDC, which enhances the process of data integration. Despite this addition, our approach still offers the advantage of seamless integration with legacy applications, without the need for any additional computing resources or modifications to the existing codebase. Moreover, the data streaming process can be easily configured to capture a wide range of data events, including inserts, updates (considering the before and after state), and deletion indications. This flexibility enables comprehensive data synchronization, ensuring that all relevant data changes are efficiently captured and streamed in real-time. With our approach, we can read data once and make it available for reuse multiple times in real-time without causing any crashes or disruptions to the legacy systems.

On the other hand, our approach allows flexibility and respects the diverse paces and methodologies of different squads within the organization. Some squads solely consume data from our data stream, while others may create their own replicas for local persistence, such as our CRM solution. However, in all cases, data is available for consumption in real-time, eliminating the need for batch processing against the legacies that used to require hours to make data available to end users.

This shift in mindset also triggers a chain reaction, extending to digital apps themselves becoming part of this new paradigm. These apps can now trigger events, which are efficiently handled by microservices and middleware at an asynchronous level, enabling seamless integration with other applications. This eliminates the need for time-consuming batch processing.

On the other hand, it is crucial to emphasize the significance of peopleware in this context. Adopting an event-driven architecture requires a shift in the architectural mindset when designing and developing digital apps in the background. It also necessitates strong collaboration between squads, going beyond the requirements of traditional process and data integration. The interdependence among the applications developed in this environment relies on near real-time processing, making it essential for each squad to understand the behaviors of applications developed by other squads in order to effectively handle errors and ensure proper event directionality.

At the same time, despite the numerous advantages of this technique, our project is not without its challenges. Configuring data streaming in both the publisher and consumer sides is not a straightforward task and requires careful attention. It also involves understanding how to effectively utilize cloud offerings. For example, squads must familiarize themselves with concepts such as topics and queues rather than solely relying on development frameworks. Mishandling these concepts can lead to issues and bugs, such as record duplication. Therefore, having a strong architecture team is crucial to support the overall transformation that comes with adopting this new mindset.

Last but not least, adopting an event-driven architecture also brings positive implications for advanced analytics and AI. By leveraging the data streaming approach, these technologies can effectively consume real-time data, enabling valuable insights, AI calibration, and the development of predictive models.

Our company has embraced this new approach, leveraging it to deliver exceptional solutions to our students. Additionally, it has facilitated the modernization of our legacy systems, which play a key role in our IT ecosystem. This modernization enables us to reuse integrations without the need for peer-to-peer middleware development for each project. Our technology department assumes a leading role in delivering business solutions with an accelerated time-to-market, benefiting from the speed and efficiency offered by this approach.

The articles from these contributors are based on their personal expertise and viewpoints, and do not necessarily reflect the opinions of their employers or affiliated organizations.