Big Data in Finance: Strategies, Challenges, Future

Luis Carlos Cruz Huertas, Executive Director, Head of Infrastructure and Automation, Data Platform and Artificial Intelligence, DBS Bank

Luis Carlos Cruz Huertas, Executive Director, Head of Infrastructure and Automation, Data Platform and Artificial Intelligence, DBS Bank

1. What strategies have you implemented to harness the potential of innovative big data analytics techniques to fulfil your business requirements?

We have developed a modern data analytics platform leveraging cloud-based technologies. This hybrid on-premises and cloud-based infrastructure provides the scalability and flexibility needed for advanced analytics.

For example, we migrated our data warehouse into a large data lake, allowing queries across petabytes of financial data. We also optimised our workloads across a unified analytics engine for risk modelling and trade analysis, reducing run times by over 40%. Our data scientists can now train machine learning models faster using compute provisioned through data science workbenches.

Establishing strong data governance and talent practices has been equally important. To maintain regulatory compliance, we formulated policies for data access, encryption, and cross-border data flows. Our training programmes also allow the whole organisation to become proficient in Structure Query Language (SQL) and dashboarding tools to self-serve insights.

The combination of in-house infrastructure, skilled talent and external capabilities allows us to execute modern analytics while maintaining data security and control. We continue to evaluate new technologies like data fabric and auto-machine learning to further our analytics maturity.

2. Can you share your experiences from one of the projects that you were recently involved in?

To manage the increasing volume of data and complexity of analytics within the bank, we recently enhanced our data platform to increase productivity and optimise our data pipeline.

A key focus was optimising the data jobs that consumed the most resources and took the longest execution time. These revolved around the machine learning models that powered our risk management and trading analytics applications. By leveraging features like caching, broadcast variables, and partition pruning, we reduced shuffling operations and memory usage significantly.

Additionally, we tuned over 100 different configuration parameters across the execution, serialisation, memory management and hardware provisioning layers. This exercise of performance tuning was tedious but helped unlock massive gains. Finally, we also redesigned certain jobs to avoid unnecessary stages.

The combination of in-house infrastructure, skilled talent and external capabilities allows us to execute modern analytics while maintaining data security and control.

The impact was effective —our core analytics applications now run over ten times faster while consuming 25% less cluster resources. These performance gains have been crucial to scaling up the data science workloads as we build newer AI-driven features that will make our platforms more resilient.

3. What are some of the challenges in implementing big data analytics that make current services unable to provide an optimal solution?

Talent and technology gaps can make implementation challenging. Advanced techniques like machine learning require highly specialised skills. Existing tools also have limitations in enabling production-scale deployments. Close collaboration across teams is key to overcoming these obstacles.

4. How do you see the future of big data analytics evolving in your industry?

I see AI and machine learning being applied across the finance function - from client analysis and risk management to back-office automation with embedded Generative Capabilities. Cloud platforms will enable access to more data sources and computing power. Evolving regulations will also push financial institutions to leverage data and analytics further for transparency and reporting.

5. What advice would you give to other organisations leveraging big data analytics to drive innovation and improve their competitive edge?

Start small, focus on business needs, and make sure you have the right people involved. A crawl, walk, run approach allows you to demonstrate quick wins, build internal capabilities and gain stakeholder buy-in for larger initiatives. Finally, balance innovation with resiliency to manage technology risks effectively.