Complex event processing

Event processing is a method of tracking and analyzing streams of information (data) about things that happen (events), [1] and deriving a conclusion from them. Complex event processing , or CEP , is a process that combines data from multiple sources [2] to infer events or patterns that suggest more complicated circumstances. The goal of complex event processing is to identify meaningful events (such as opportunities or threats) [3] and respond to them as quickly as possible.

These events may be happening across the various layers of an organization as sales leads, orders or customer service calls. Or, they may be news items, [4] text messages, social media posts, stock market feeds, traffic reports, weather reports, or other kinds of data. [1] An event may also be defined as a “change of state,” when a measure exceeds a predefined threshold of time, temperature, or other value. Analysts suggest that CEP will give organizations a new way to analyze patterns in real-time and help the business side communicate better with IT and service departments. [5]

The vast amount of information available about events and events is limited. [1]

Conceptual description

Among thousands of incoming events, a monitoring system may for instance receive the following from the same source:

  1. Church bells ringing.
  2. The appearance of a man in a tuxedo with a woman in a flowing white gown.
  3. Rice flying through the air.

From these events the monitoring system may have a complex event : a wedding. CEP as a technique helps to discover complex events by analyzing and correlating other events: [6] the bells, the man and woman in wedding attracts and the rice flying through the air.

CEP relies on a number of techniques, [7] including:

  • Event- pattern detection
  • Event abstraction
  • Event filtering
  • Event aggregation and transformation
  • Modeling event hierarchies
  • Detecting relationships (such as causality , membership or timing )
  • Abstracting event-driven processes

Commercial applications of CEP exist in various industries and include algorithmic stock-trading , [8] the detection of credit-card fraud , business activity monitoring , and security monitoring. [9]

History

The CEP area has roots in discrete event simulation , the active database areas and some programming languages. The activity in the industry was preceded by a wave of research projects in the 1990s. Selon [10] the first project That paved the way to a generic CEP language and execution model Was the Rapide project in Stanford University , directed by David Luckham . In parallel there have been two other research projects: Infospheres in California Institute of Technology , directed by K. Mani Chandy , and Apama in University of Cambridge directed by John Bates. The commercial products were dependent on the concepts developed in these and some later research projects. Community efforts in a series of symposiums organized by the Event Processing Technical Society , and later by the ACM DEBS conference series. One of the efforts of the community was to produce the manifesto manifesto [11]

Related concepts

CEP is used in operational intelligence (OI) products to provide insight into business operations by running query analysis against live feeds and event data. OI collects real-time data and correlates against historical data to provide insight and analysis. Multiple sources of data can be combined to provide a common operating picture.

In network management , systems management , application management and service management , people usually refer to event correlation . As CEP engines, event correlation engines ( event correlators ) analyze a mass of events, pinpoint the most significant ones, and trigger actions. However, most of them do not produce new inferred events. Instead, they relate high-level events with low-level events. [12]

Inference engines , eg, rule-based reasoning engines , typically produce inferred information in artificial intelligence . However, they do not usually produce new information in the form of complex (ie, inferred) events.

Example

A more systemic example of CEP involves a car, some sensors and various events and reactions. Imagine that a car has several sensors-one that measures pressure, one that measures speed, and one that detects if someone sits on a seat or leaves a seat.

In the first situation, the moves and moves from 45 psi to 41 psi over 15 minutes. As the pressure in the pull is decreasing, a series of events containing the tire pressure is generated. In addition, a series of events containing the speed of the car is generated. The event processor may detect a situation whereby a loss of pressure occurs over a relatively long period of time in the creation of the “lossOfTirePressure” event. This new event may trigger a reaction process to note the pressure loss in the car’s maintenance log, and alert the driver via the car’s portal.

In the second situation, the car is moving and the pressure drops from 45 psi to 20 psi in 5 seconds. A different situation is detected because of the loss of pressure. The different situation results in a new event “blowOutTire” being generated. This new event triggers a different reaction process to immediately alert the driver and to initiate onboard computer routines to assist the driver in bringing the car to a stop without losing control through skidding.

In addition, events that represent detected situations can also be combined with other events in order to detect more complex situations. For example, in the final situation the car is moving normally and suffers a blown pull which results in the car leaving the road and striking a tree, and the driver is thrown from the car. A series of different situations are rapidly detected. The combination of “blowOutTire”, “zeroSpeed” and “driverLeftSeat” within a very short period of time in a new situation being detected: “occupantThrownAccident”. The results of this study are based on the results of the study of the detected situation. This is the essence of a complex (or composite) event. It is complex because one can not directly detect the situation; One has to infer or deduce that the situation has occurred from a combination of other events.

Integration with business process management

A natural fit for CEP has been with business process management (BPM). [13] BPM focuses on end-to-end business processes, in order to continuously optimize and align for its operational environment.

However, the optimization of a business does not rely solely on its individual, end-to-end processes. Seemingly disparate processes can affect each. Consider this scenario: In the aerospace industry, it is good practice to monitor breakdowns of vehicles to look for trends (determine potential weaknesses in manufacturing processes, material, etc.). Another separate process monitors current operational vehicles’ life cycles and decommissions them when appropriate. The second phase of the first cycle of the cycle is the first step in the evolution of the process. ) To issue a recall on vehicles using the same batch of metal discovered as faulty in the initial process.

The integration of CEP and BPM must exist at two levels, both at the business awareness level and the CEP can interact with BPM implementation). For a recent state of the art review on the integration of CEP with BPM, which is frequently labeled as Event-Driven Business Process Management, refer to. [14]

Computation-oriented CEP’s role can arguably be seen to overlap with Business Rule technology.

For example, customer service centers are using CEP for click-stream analysis and customer experience management. CEP software can factor real-time information about millions of events (clicks or other interactions) per second into business intelligence and other decision-support applications. These “recommendations applications” help agents provide personalized service based on each customer’s experience. The CEP application may collect data about what customers have recently interacted with the company in various channels, including in-branch, or on the web via self-service features, instant messaging and email. The application then analyzes the total customer experience and recommends the agent on the phone, And hopefully keep the customer happy. [15]

In financial services

The financial services industry was an early adopter of CEP technology, using complex transaction processing to structure and contextualize available data so that it could trade trading behavior, specifically algorithmic trading , by identifying opportunities or threats that indicate traders Or sell. [16] For example, CEP technology can track this an event. CEP technology can also track drastic rise and fall in number of trades. Algorithmic trading is already a practice in stock trading. It is estimated that around 60% of Equity trading in the United States is by way of algorithmic trades.

Recent improvements in CEP technologies have made it more affordable, helping smaller companies to create trading algorithms of their own. [3] CEP has evolved from an emerging technology to an essential platform of many capital markets. The technology is mostly growth in banking, serving fraud detection, online banking, and multichannel marketing initiatives. [17]

Today, a wide variety of financial applications use CEP, including profit, loss, and risk management systems, order and liquidity analysis, quantitative trading and signal generation systems, and others.

Integration with time series databases

A time series database is a software system that is optimized for the handling of data organized by time. Time series are finite or infinite sequences of data items, where each item has an associated timestamp and the sequence of timestamps is non-decreasing. Elements of a time series are often called ticks. The timestamps are not required to be ascending (merely non-decreasing) because in the practice of the time resolution of some systems such as financial data sources may be quite low (milliseconds, microseconds or even nanoseconds), so consecutive events may carry equal timestamps.

Time series provides a context to the analysis typically associated with complex event processing. This can apply to-any vertical industry Such As finance [18] and cooperatively with other Such As BPM technology.

Consider the scenario in which it is necessary to determine the statistical thresholds of future price movements. This is useful for both trade models and transaction cost analysis.

The ideal case for a continuous time-series analysis. What happened yesterday, last week or last month is simply an extension of what is occurring today and what may happen in the future. An example may involve comparing current volumes to historical volumes and volatility for trade execution logic. The volatility and volatility of the outlet market is a major concern.

See also

  • Event correlation
  • Event-driven architecture – (EDA) is a software architecture that promotes the production, detection, consumption and reaction to events.
  • SEDA – Staged event-driven architecture decomposes complex, event-driven architectures into internships
  • Event Processing Technical Society – (EPTS)
  • Event stream processing – (ESP) is a related technology that focuses on processing streams of related data.
  • Kinetic Rule Language – (KRL) is an event-condition-action rule with an embedded complex event expression language.
  • Operational intelligence – Both CEP and ESP are technologies that underpin operational intelligence.
  • Pattern matching
  • Real-time business intelligence – Business Intelligence is the application of knowledge derived from CEP systems
  • Real-time computing – CEP systems are typically real-time systems
  • Real time enterprise

Vendors and products

  • Apama by Software AG – monitors rapidly moving events. [19]
  • Azure Stream Analytics
  • Drools Fusion
  • Esper Complex for Java and C #.
  • Feedzai – Pulse
  • GigaSpaces XAP
  • Informatica RulePoint by Informatica
  • Microsoft StreamInsight Microsoft CEP Engine implementation [20]
  • OpenPDC – A set of applications for processing streaming time-series data in real-time.
  • Oracle Event Processing – for building applications to filter, correlate, and process events in real time.
  • BRMS – Red Hat based on Drools
  • SAP ESP – A low-latency, rapid development and deployment platform that allows multiple streams of data in real time [21]
  • SQLstream is a leading provider of enterprise- class data management solutions for enterprise- class, enterprise- class, enterprise- class data processing.
  • TIBCO BusinessEvents & Streambase – CEP platform and High Performance Low Latency Event Stream Processing
  • WebSphere Business Events
  • WSO2 Siddhi Complex event processing written in Java. Designed as part of a series of middleware components.
  • Apache Flink Open-source distributed streaming with a CEP API for Java and Scala.

References

  1. ^ Jump up to:a b c Luckham, David C. (2012). Event Processing for Business: Organizing the Real-Time Enterprise . Hoboken, New Jersey: John Wiley & Sons, Inc.,. p. 3. ISBN  978-0-470-53485-4 .
  2. Jump up^ Schmerken, Ivy (May 15, 2008), Deciphering the Myths Around Complex Event Processing , Wall Street & Technology
  3. ^ Jump up to:a b Bates, John, John Bates of Progress Explains how complex event processing works and how it can simplify the use of algorithms for finding and capturing Trading Opportunities Fix Global Trading , retrieved May 14, 2012
  4. Jump up^ Crosman, Penny (May 18, 2009), Aleri, RavenPack to Feed News Algos into Trading , Wall Street & Technology
  5. Jump up^ McKay, Lauren (August 13, 2009), Forrester Gives a Welcoming Wave to Complex Event Processing , Destination CRM
  6. Jump up^ D. Luckham, “The Power of Events: An Introduction to Complex Event Processing in Distributed Enterprise Systems”, Addison-Wesley, 2002.
  7. Jump up^ O. Etzion and P. Niblett, “Event Processing in Action”, Manning Publications, 2010.
  8. Jump up^ Complex Event Processing for Trading, FIXGlobal, June 2011
  9. Jump up^ Details of commercial products and use cases
  10. Jump up^ Leavit, Neal (April 2009), Complex-Event Processing Poised for Growth , Computer, vol. 42, no. 4, p. 17-20 Washington
  11. Jump up^ http://drops.dagstuhl.de/opus/volltexte/2011/2985/Mani Chandy and K. Opher Etzion and Rainer von Ammon (eds), 10201 Executive Summary and Manifesto – Event Processing, Dagstuhl seminar Procesdings 10201 , ISSN 1862-4405, 2011
  12. Jump up^ JP Martin-Flatin, G. Jakobson and L. Lewis, “Event Correlation in Integrated Management: Lessons Learned and Outlook”, Journal of Network and Systems Management, Vol. 17, No. 4, December 2007.
  13. Jump up^ C. Janiesch Mr. Matzner & O. Müller: “A Blueprint for Event-Driven Business Activity Management”, Lecture Notes in Computer Science, 2011, Volume 6896/2011, 17-28,doi:10.1007 / 978- 3-642-23059-2_4
  14. Jump up^ J. Krumeich, B. Weis, D. and P. Werth Loos: “Event-Driven Business Process Management: where are we now ?: A comprehensive synthesis and analysis of literature”, Business Process Management Journal, 2014, Volume 20, 615-633,doi:10.1108 / BPMJ-07-2013-0092
  15. Jump up^ Kobielus, James (September 2008), Really Happy in Real Time , Destination CRM
  16. Jump up^ The Rise of Unstructured Data in Trading , Aite Group, October 29, 2008
  17. Jump up^ Complex Event Processing: Beyond Capital Markets , Aite Group, November 16, 2011
  18. Jump up^ “Time Series in Finance”, Retrieved May 16, 2012
  19. Jump up^ Apama Real-Time Analytics Overview. Softwareag.com. Retrieved on 2013-09-18.
  20. Jump up^ Microsoft StreamInsight product page
  21. Jump up^ SAP ESP – Developers community

Leave a Comment

Your email address will not be published. Required fields are marked *