Esper to Apama migration use case story

Introduction

In this blog article I’d like to share some tips and best practices from a recent Esper to Apama migration our team completed.

Business Case

Our customer uses the Cumulocity IoT platform to provide one-stop support for equipment selection, IoT-PF introduction, and external system integration. Business use cases are built on spatial management, theft prevention, asset tracking, industrial equipment monitoring, usage management, driving safety and vital sensors. Use cases include:

1. Physical condition management system for workers.
2. Streamlining of failure prediction and preventive maintenance.
3. Realtime remote maintenance of products and equipment installed in multiple buildings.
4. Seat presence detection and visualisation of facility usage.
5. Device vacancy management.
6. Power consumption visualization with energy-saving initiatives for equipment and facilities.
7. Visualization of degree of congestion to prevent clusters.

Previously, Esper was used for real time streaming analytics in the platform. When Esper was deprecated post v10.5 (2018), the default Complex Event Processing (CEP) engine in Cumulocity changed to Apama. The Cumulocity platform permits any given tenant to have zero or one registered streaming analytics microservice. Scope of work involved migration of custom rules from Esper CEL language to Apama EPL along with associated migration of smart rules for all the tenants deployed in the environment.

This was Software AG’s biggest Esper to Apama migration, with a few hundred tenants and a few thousand custom rules requiring manual translation. There were also a few thousand smart rules auto translated to Apama in the customer’s multi-tenant multiple high-performance environments.

Technology Stack & Techniques

• Software AG Designer (v10.5) with Apama (Community Edition)
• Cumulocity (UI & backend) with Apama PAQ
• Apama Log Analyzer
• Apama EPL Apps Tools
• E2A-Tool v0.4 (limited use)
• Postman Collection

Challenges

• The migration had to be as seamless as possible for end customers. This was challenging, as complex rules and, in some cases, additional logic had to be incorporated to address real-life conditions.
• Stringent functional and performance testing with no tolerance for any resource consumption higher than existing usage.
• Multi-tenant (Esper) vs Per-Tenant (Apama) model.
• High precision requirement on Big Decimal (Esper) vs Float (Apama).
• Extensive testing cycles and lead time of end customer communication.
• Specific short single deployment window.

Mitigation

• Simulated real-time boundary conditions of use cases in multiple sessions with the customer. Hypothetical cases were strictly ring-fenced with actual business conditions filtering the necessary cases to be dealt with in the migrated code.
• Provided / adhered to specific code guidelines with the goal of reducing I/O calls to backend database.
• Highly complex rules required development of custom aggregate functions.
• Both development and deployment executions were done in a phased manner staggered over multiple overlapping sprints.
• Automated deployment scripts used with multiple checkpoints to ensure flawless execution

Key Implementations

• Cache Implementation - 2 approaches. Flat structure + Hierarchical.
• Use of Log Analyzer for troubleshooting and memory optimization.
• Complex mathematical calculations - Custom Round-Up/ValuetoFloat functions.
• Use of Postman Collection for testing.
• EPLApps for automated deployment
• Export Report Errors during migration
• Handling null values (considered as 0 in loupdatedwer versions) in Apama (discarded for higher versions).
• No distinction between created and notifications - WasCreated-WasUpdated functions provided by R&D
• Check null before FMO
• CEP-queue length configuration in core

Best Practices & Lessons Learned

The Software AG Professional Services team worked in close coordination with Apama R&D for key fixes inside the product and cache optimisations in the rules.

• Leveraged Log Analyzer for troubleshooting performance related issues, Postman Collections for automated testing, Apama EPL Apps Tools for deployment.
• Developed multiple scripts to automate multiple tasks involved in migration and reduce downtime during deployment.
• Developed custom functions and aggregates and reused across multiple cases.
• Impact Analysis communicated on any deprecation or compatibility of any EPLs in upgrade roadmap.
• Conducted additional sessions/workshops and code-walkthrough.
• Shared sample code and addressed technical queries at every level to enable the customer for Apama related work in future.

Customer Benefits

• Performance of Apama is better than Esper - Applications run faster in Apama and require less memory.
• Apama server is natively built and optimized for performance on each platform.
  • Apama does not require the performance overhead of working through a JVM.
  • No unpredictable slow performance spikes due to Java garbage collection.
• Tenant wise segregation of Apama pods. Any tenant specific issues will be limited to the specific pod and will not spill over (noisy neighbour protection) as had been the case for multi-tenant applications.
• Apama is fully supported within Cumulocity IoT. Integration between Apama and Cumulocity IoT provides several benefits, which will increase over time. Apama’s roadmap is aligned with Cumulocity IoT. Apama’s streaming analytics capabilities and tooling (Analytics Builder) will broaden to enhance what is possible today.

Relevant resources

• CEL (Esper) migration - Knowledge base - Cumulocity IoT
• Apama Product Documentation
• Rationale for migration to Apama in Cumulocity IoT
• Cumulocity IoT Streaming Analytics Documentation

Next Steps

Are you interested in the fastest way to leverage streaming analytics rules for Cumulocity IoT using Apama? We have created the Cumulocity IoT Streaming Analytics - Get Started to address your needs.

Have questions? Please feel free to contact us at professional.services@softwareag.com

Read full topic