Accelerating Business Growth with Microservices Supporting the FX Infrastructure of the SBI Group with Confluent

Modernizing the FX trading platform for future business growth

As a member of the SBI Group handling online financial systems, SBI Liquidity Market is responsible for the development and provision of systems for foreign exchange margin trading (FX trading). It provides trading infrastructure to financial instruments business operators within the group, such as SBI Securities and SBI FXTRADE, delivering services to individual investors through these companies. By aggregating exchange rates through connections with 32 counterparty financial institutions, both domestic and overseas, the company has also delivered market functionality and liquidity for FX trading and supported the competitiveness of the SBI Group's FX business.

The FX trading infrastructure requires not only high speed capable of handling tens of thousands of updates per second, but also high reliability and availability to operate continuously 24 hours a day, all year long. In order to meet this goal, it faced the challenge of transforming its trading infrastructure to support further business growth. Yuta Yoshikawa, Executive Officer, CTO, and System Division General Manager, explained: "Before the transformation, our FX trading infrastructure was based on a database-centered monolithic architecture. Since its launch in 2011, repeated feature additions have made the system increasingly complex. As a result, modifications have had a broad impact, making performance improvements difficult. We also faced limitations in flexibility and scalability when adding or modifying services and handling increasing trading volumes. This made it urgent to upgrade the infrastructure to support future growth in transaction volumes."

Kafka was selected for its flexible and highly scalable microservices architecture

Upgrading its trading platform meant adopting a microservices architecture, which allowed them to functionally distribute the system, process disseminated rates by purpose, and scale execution services based on trading volume and load. Enter open-source Kafka.

 By adopting Kafka, they were able to enable "Message Queuing" (MQ) for data integration between services and ensure data "order" and "reliability." Yoshikawa reflected, "When it comes to data exchange, API integration may be the first approach that comes to mind. However, in rate dissemination, it is essential to have a mechanism that ensures reliable delivery of data in the correct order and supports retransmission in the event of data loss. In contrast, with API integration, we were concerned  that ensuring data order and consistency would be difficult when services are parallelized."

"With MQ, generated rates can be stored in order and reliably delivered to all services in the same sequence. Kafka is not the only MQ product, but we selected it after internal comparison because, in addition to its functionality, it can securely guarantee the level of reliability and order required by the financial industry, with no data loss," said Yoshikawa.

Confluent: The Key to the Stable Operation of Financial Systems 

Yoshikawa recalled, "Kafka is open-source software, and maintaining and operating it solely with our internal resources presents significant technical challenges and burdens." Given that it would be used in a financial system where downtime is unacceptable, the team had a key concern—if a failure occurs, who would resolve Kafka-related issues and how? Establishing an internal structure solely for that purpose was difficult, as was daily operational monitoring. For these reasons, in addition to operational and monitoring capabilities to support stable system operation, the team knew it needed reliable commercial support.

Confluent was the clear answer. As a fully managed service, it offered data processing and operational monitoring capabilities on top of Kafka's core functionalities. For operational monitoring, the GUI dashboard makes it easy to manage key operations and monitor the status and performance of each service.

"In addition to its strong technical capabilities with a large number of committers in the Kafka development community, the presence of a Japanese subsidiary and the availability of comprehensive support in Japanese were also key factors in our decision to choose Confluent. Confluent's extensive track record of deployments at financial institutions, both in Japan and overseas, also gave us a strong sense of confidence that they could give us all the support we needed," said Yoshikawa.

Comprehensive Support from Confluent Japan

The team began developing its new platform with Confluent in January 2022 and launched its operation in September 2023. Yoshikawa spoke highly of the support they received from Confluent Japan, noting, "Confluent Japan's engineers reviewed our programs down to the source-code level. This was extremely reassuring for us as an implementing company and helped us shorten the development period and improve quality."

Because of this support, application development proceeded smoothly, and they were able to allocate the saved time and resources to tasks such as data migration. In addition, by integrating with Datadog, they received concrete advice on which metrics to monitor and how to design operational monitoring, which contributed to the platform's current stable operation.

Real-Time Performance and Availability for Efficient Financial Transactions

Currently, transaction-related data, including exchange rates, customer orders, and trade results, is exchanged between services through Kafka. This enables parallel processing that ensures rate order, consistency, and duplication checks without relying on applications.

Previously, processing was sequential, waiting for database writes to complete before proceeding. In the new system, once data is fed into Kafka, each microservice -- such as rate dissemination, loss cut judgment, and notifications -- can acquire and process data in parallel. "As a result, even if some backend processes are delayed, rate dissemination to customers continues uninterrupted, allowing them to always access fresh data. The system has also met our expectations for availability and reliability, with zero data loss or inconsistencies since the migration," says Yoshikawa.

The processing that had been concentrated in the database for real-time integration, as well as the massive number of accesses from each service -- occurring hundreds of times per second -- has almost disappeared. As a result, the company no longer needs to constantly worry about database load and has more targeted and efficient system operations. "To handle rate data reaching 20 updates per second, we decided not  to speed up 100-millisecond batch processing, but to change our approach and transforming the architecture," says Yoshikawa. Dividing into microservices not only allowed load balancing and scaling the execution service processing, but also provided the flexibility and scalability to easily accommodate future system changes.

Going forward, the plan is to aggregate not only SBI Liquidity Market's trading data, but also vast amounts of customer activity logs and other log data, using them for self-service analytics, as well as for enhancing security through real-time fraud detection. They plan to leverage this data infrastructure as a foundation for AI integration, aiming to enable more advanced data analysis and the creation of new services.

With Confluent, SBI Liquidity Market can support the mission-critical infrastructure of its financial business. As Yoshikawa said, "The SBI Group is a company with a culture that encourages technological challenges, and adopting Confluent is right at the nexus of that culture. Because of Confluent's continued support, we will  pursue further system enhancements and the creation of business value."

Learn more about SBI Liquidity Market.