Complex Event Processing: Not as Complex as You Might Think

Complex event processing (CEP) is a rapidly growing software sector that is creating a new generation of real-time, event-driven business applications. It is the engine behind ultra-low latency trading systems in financial services, applications that monitor and proactively manage customer interactions with Web sites or call centers, and software that tracks assets, products and people using RFID (radio frequency identification) in a number of industries.

Here is a general definition of CEP that we like to use at Coral8:

“Complex event processing software allows you to process and analyze multiple streams of high-volume, high-speed business and system events, and to uncover opportunities and threats as they happen — not after the fact.”

A Powerful Platform

A CEP engine gives IT teams a powerful platform that drastically reduces the time and effort required to build event-driven applications and makes it much faster, easier and less costly to deploy them.

In some markets, such as financial services and security, CEP software is a more flexible solution than the custom-built event processing logic built into C or C++ applications. In other application areas, such as customer experience monitoring, developers are finding CEP to be the critical building block that “event-enables” their traditional applications.

Here are the five most common questions people ask us about CEP:

1. What is an event and an event stream?

   Events and event streams may be found almost anywhere in a typical enterprise. Some obvious examples include stock exchange tick and trade data, system and network management traps, RFID reads and Web site clicks. Thus, one common definition of an event is “a notification that something happened somewhere.”

   Other events are not so obvious. Many enterprise applications, whether packaged applications such as ERP (enterprise resource planning) and CRM (customer relationship management), or custom-built ones, output events when particular “states” are reached — for example, when a purchase order is received, goods arrive, or a customer transaction is completed. Thus, another common definition of an event is a “notification of a change of state.”

   An event stream is associated with a particular source of events. An exchange feed, a Web server log file, or an RFID reader are all sources of events, and can be viewed as event streams. CEP engines can be configured to “listen” to one or many of these event streams, and to analyze events and event patterns across streams.

2. How does CEP work?

   While most analytical engines are designed to process data stored in databases or files, complex event processing engines are designed to continuously analyze event streams in real time. The key words here are “continuously” and “real time.”

   Users register CEP queries, or rules, with the CEP engine. These queries are continuously run against the flowing events and maintain “state.” Some examples of state are the last known price for each stock, the last 10 pages visited by each Web site user and the log file data for the last hour.

   If CEP engines only looked at event streams, they would not be very useful. In event processing, context and history are vitally important. CEP engines are tightly integrated with databases to access or write data. For example, if real-time ticker data indicates that a stock is at US$50, it is very important to know whether this is a relative high or low within a specified period of time.

3. Do I have to learn a new language and skills to use CEP?

   A number of CEP engines use languages based on SQL. The designers of SQL-based CEP engines realized that starting with a familiar language has many advantages, including a large existing skills base. Event streams are structured very much like database tables and events are quite similar to rows.

   By itself, SQL is not a CEP language. A number of new features need to be added accommodate the different processing model of CEP, such as support for continuous queries, defining windows, maintaining time-based state and the ability to look for event patterns. These extensions are integrated into the CEP language, making the learning curve extremely low.

4. What gives CEP such high performance?

   CEP engines are designed to process a large volume of events at extremely high speeds. Throughput requirements are often well over 100,000 events per second, while processing latency demands can be as low as one millisecond, or less. This requires a number of specialized optimizations. Some of CEP engine’s speed comes from its ability to analyze data in memory, prior to storing it to disk. However, the biggest advantage CEP engines have is that queries are run continuously.

   Continuous queries enable incremental processing. If a database is asked to compute the average of a set of values in a table each time a new row is added (say 100 times per second), it generally scans the entire table 100 times each second (every time it executes the query). This is extraordinarily inefficient when processing real-time events. In a CEP engine, continuous queries update the average of the set of values incrementally as each new event arrives. There is no need to scan the entire table for every change.

   Other useful optimization techniques include parallel processing, minimization of data copying and locking, and thorough query optimization.

5. What benefits does CEP bring to my company and applications?

   CEP gives you a computing infrastructure and core building blocks for applications that can proactively monitor, manage and optimize your business. Event-enabled applications help you to take actions that maximize revenue, margins and customer satisfaction, while also helping to combat fraud and security violations.

   When compared to custom-coded solutions, a CEP engine offers faster development by a factor of 10 and provides more functionality. This new level of development speed and efficiency makes event-driven application development and deployment far more affordable.

   CEP software also provides the core component for event-enabling service-oriented architecture (SOA). CEP queries, modules and servers permit you to separate complex logic such as trading algorithms and customer interaction patterns into reusable services that are available for all applications.

Give It a Try

Complex event processing is driving the new paradigm of event-driven applications and architectures. One of the best things about CEP is that it is easy to get started. Many developers already understand the ideas behind event processing and SQL-based languages eliminate much of the learning curve.

Early adopters see dramatic results when using CEP software. A large financial services and loan origination company used CEP software to create over 40 new event-driven application modules in four months. This company is now event-enabling its entire customer management infrastructure in order to drive new revenue and lower the cost of interacting with its customers. A healthcare software solution provider replaced its entire RFID tracking system and logic with CEP engine, saving many hundreds of thousands of dollars per year.

So what’s holding you back? Some CEP vendors offer free developer versions of their software, so there is no risk if you get started immediately. Download a CEP product today and start event-enabling your applications and enterprise.