SunGard, one of the world’s leading software and IT services companies, offers among its many services and solutions, Protegent, a suite of compliance solutions that help retail and institutional investment firms oversee business processes relating to compensation processing, client acquisition and suitability, and employee personal trading and codes of ethics. These Protegent solutions are fueled by massive amounts of internal and external data. As data volumes have mushroomed, the system, running on traditional relational databases, began to experience performance deterioration that was unacceptable to end-users. To remedy this problem and position itself for the inevitable continued data growth, the company implemented Sybase IQ, Sybase’s column-based analytics server designed specifically to deliver superior performance for mission-critical business intelligence, analytics and data warehouses.
Huge Data Volumes Can’t Slow This System Down
SunGard is one of the world’s leading software and IT services companies, working with more than 25,000 customers in more than 70 countries. The company provides software and processing solutions for financial services firms, higher education institutions and the public sector organizations. It also provides disaster recovery services, managed IT services, information availability consulting services and business continuity management software.
Among the solutions SunGard offers financial services companies are its
Protegent compliance solutions. Protegent solutions help retail and institutional investment firms oversee business processes relating to compensation processing, client acquisition and suitability, and employee personal trading and codes of ethics. Protegent supports supervision and surveillance practices, helps streamline the compliance life cycle, proactively monitors trades and provides comprehensive auditing and reporting for financial institutions and companies that trade in energy and commodities markets. In doing so, the solutions help reduce expenses and address regulatory requirements.
To do all of this effectively and efficiently, Protegent gathers huge volumes of internal and external data, which it runs through its synapse rules engine. This engine is designed to produce alerts that flag unusual or questionable activity.
“Our customers are the big trading firms,” says Steve Rider, product architect for SunGard’s Protegent business. “We currently operate in the U.S. market and we’re expanding into international markets. With our rules engine system, gathering all of this data together becomes challenging. As the data volumes have grown over the years, we’ve been faced with simply managing the volume of input data coming through the system. In addition, the volume of results is significant and we’re growing into ranges where traditional database technology struggles..”
Traditional Database Performance Suffered Under High Data Volumes
SunGard Protegent solutions were designed to interoperate with any database system, typically traditional, row-oriented databases developed primarily for transaction processing purposes. Protegent provided customers with an ad hoc user interface, allowing end users to select from over 100 fields to produce their own ad hoc queries. To optimize performance, Protegent created indexes based on its anticipation of the most commonly executed queries. But it had difficulties tuning every single, on-the-fly query.
As a result, as data volumes have grown, both from increased trading activity and from customers’ desires to retain more historical data, performance has become a concern for users. “It became quite obvious to us,” says Rider, “that we had to find a better technology – a technology that would give us a leap forward in our ability to handle very large and continuously growing volumes of data without performance degradation.”
Rider also notes that an additional factor contributing to the data volume/performance problem was that data was becoming more complex in and of itself. For example, a simple four-byte integer that used to store nine digits of an ID number, has now become alphanumeric and has expanded into the 50 to 100 character range. These are the keys Protegent uses to locate records. So now, instead of small, four byte fields, the company has to deal with hundred byte fields.
“We’re not simply pulling more records,” Rider explains. “We’re pulling more complex records based on more complex data. In addition, the complexity of what users are asking for is growing. And, of course, regulatory requirements are expanding and asking for data that is approaching real time. Data growth, for all of these reasons, is ongoing and massive, and represents a challenge for us.”
Solution: Sybase Column-Based Architecture
As Protegent searched for a solution that would enable it to address these challenges, it found one that stood out: Sybase IQ.
Protegent found that the column-oriented nature of Sybase IQ allowed it to put indexes on the specific columns containing the information users required.
This column-based architecture versus the previous row-based architecture means that Protegent employees do not have to spend time tuning to ensure high performance for ad hoc user queries. Additionally, Sybase IQ’s multi-column orientation allows it to run parallel without Protegent having to do anything. The result: every query becomes parallelized, every data load becomes parallelized and Protegent can truly leverage the power of modern, multi-processor computers and achieve necessary performance levels, regardless of data volumes, with ease.
“With the implementation of Sybase IQ, Protegent has grown from a single server system into a multi-node grid,” adds Rider. “The system currently holds approximately 80 terabytes of compressed data. We currently have some 660 billion rows of market data and load over a billion rows a day during market hours, at the same time interactive users on the system are pulling reports and running queries against this data.”
“We Feel Sybase IQ Can Support Whatever Growth We Experience”
With its Sybase IQ-powered architecture now in place, Protegent has a solution that will support its growth well into the future. The company no longer has concerns about data volumes. It no longer has to devote IT resources to tuning for ad hoc queries. In fact, very little administrative time is now spent on the system.
“With Sybase IQ,” says a very pleased Rider, “the system just runs.”
Of course, as the only constant in business is change, the future will undoubtedly present Protegent with new challenges. As it expands into other geographic markets and into other business areas, such as options trading, it expects to see data growth measured in orders of magnitude. But this is not something the company is concerned about. At some point, it may have to add another server to its grid, but that’s about it.
“We feel Sybase IQ can support our growing business without having to do much more than procuring a bit of hardware,” says Rider who expects that the system will exceed a trillion rows before too long and who envisions the system eventually growing into the petabyte range.
“The architecture we have in place now gives us a powerful platform to move forward on. We really do not worry about data volumes anymore. Sybase IQ is very simple to manage, very simple to operate and it’s enabling us to easily build really big systems in a way that is cost-effective, manageable and sustainable. Sybase IQ has really been a jewel for us. It’s easy to use and it always works. It doesn’t matter what we throw at it, it seems to take it in stride and give us a great response. It’s been an excellent tool for our data warehouse. We feel like it’s a platform that will carry us forward into uncharted territory. We see no limit to how far we can go with it.”