Singapore Exchange Updates its Trading Platform for Improved Latency and Throughput with Confluent

When the COVID-19 pandemic began to affect the global economy, news and information moved rapidly and investors needed to be able to react with confidence and manage their portfolio risks efficiently. Singapore Exchange (SGX)–a key part of Singapore’s financial market infrastructure–responded by working to ensure that its markets remained robust, uninterrupted, and accessible.

To meet that commitment to and to support the exchange’s continued growth, SGX is modernizing its core trading platform with a microservices-based architecture powered by real-time data streaming and Confluent.

The exchange’s legacy data platform, based on third-party software that SGX has been developing and customizing, had started to show signs of strain. “Our legacy platform was built using message queues and technologies that were between 10 and 20 years old,” says Sunarto Rahardja, SVP and Head of Software Development at SGX. “As our business growth has picked up over the last few years, the platform’s ability to scale was limited and the latency and throughput of its messaging layer was a weak point. That’s when we decided to bring in Apache Kafka and run some pilot projects.”

Once it became clear that Kafka was going to play a role in the exchange’s infrastructure, SGX worked with Confluent to maximize its business value. “Confluent engineers have worked hard to ensure our success, both before we went live and after,” says Rahardja. “For SGX, Confluent offers a resilient, low-latency messaging backbone for an architecture that has enabled us to increase agility, streamline DevOps, and shorten time to market.”

SGX is currently exploring ways to incorporate stream processing with ksqlDB into the trading platform as well as the best way to shift parts of its workload from on premises to the AWS cloud. In preparation, Confluent hosted a series of brown-bag sessions to share advanced topics with SGX engineers that are not covered through traditional operator and administrator training. “From an education perspective, the sessions have been very positive,” says Rahardja. “As we look to mature into a hybrid-cloud model, these sessions will help us do so in the most efficient way possible.”

Technical Solution

When SGX began looking for a more scalable alternative to the message queues that it had been using, SGX engineers conducted a proof of concept in which they used Apache Kafka® and Confluent Community software to integrate a few simple microservices. With the successful completion of that initiative, plans were put in place to incorporate Kafka into the core trading system, and at that point Rahardja and the team decided to engage Confluent.

“Once we saw that Kafka was going to enable us to move away from our legacy queueing mechanism, and that it was going to become a core part of our most critical tier-1 system, we needed to have a high level of support to ensure availability and business continuity,” says Rahardja.

Early Confluent support included guidance for designing an architecture that would enable the company to maximize the value of data in motion, as well as two, three-day, Confluent instructor led training sessions on developing for and managing the new data streaming platform. These sessions gave employees the dynamic classroom experience in which they could discuss relevant use cases with a peer group, complete with interactive labs to validate what they were learning in real-time.

Working with Confluent’s Professional Services team, SGX was able to accelerate their deployment into a more stable and secure Confluent Platform cluster. The Confluent team enabled the setup of Role-Based Access Control (RBAC) through which SGX would be able to prevent unauthorized access to cluster-stored data. In order to protect sensitive information in the configuration files for the Confluent Platform components, secrets management was used. The third securitization lever that SGX and the Confluent team employed was audit log configuration, in order to track authorization activity across the cluster.

Although rearchitecting the core trading platform will be a multiyear journey, SGX has gotten off to a strong start with substantial amounts of data already flowing through Confluent. “Currently we are pushing around one million events daily, but as we migrate more services, this is expected to grow significantly,” says Rahardja.

The new architecture, with microservices integrated via Confluent, has already started to pay dividends. For example, regression times have been reduced. When a change is made, the team needs to regression test only the microservice that was affected, instead of the entire system as they had done in the past.

Additionally, the architecture is making it easier to rapidly update risk profiles. SGX plans to integrate a risk module, which will consume from multiple topics and update the risk profiles of users based on orders and other events flowing through the platform.

As SGX continues its data streaming journey—including expanded use of Confluent Control Center, ksqlDB, and Schema Registry— the team is continuing to work with Confluent to build upon their successes. “We are confident because we know we have the right level of support for our messaging platform across the organization,” says Rahardja.

Business Results

100% uptime achieved. “Not only does Confluent provide lower latency and higher throughput than our legacy system, it is also more resilient,” says Rahardja.

Time to market reduced. “Confluent enabled us to implement a new architecture that would not have been possible with our legacy message queue,” says Rahardja. “That microservices based architecture, in turn, has enabled us to increase agility, reduce regression times, and improve our time to market.”

Timely, effective support responses received. “During deployment we ran into an integration issue with Active Directory,” says Rahardja. “When we contacted Confluent, the issue was escalated and then quickly resolved.”

Architectural alignment ensured. “The Confluent team was extremely helpful in ensuring that our planned architecture was in line with what we were trying to achieve,” says Rahardja. “Confluent engineers were professional and engaged well with our team to make sure everything we were doing was done right.”

Learn More About Singapore Exchange