OIC -Event-Driven Notification Framework in Oracle Integration Cloud (OIC) Using Apache Kafka

Working...

🧩 Use Case

Enterprises need a reliable, scalable, and asynchronous notification platform that can handle:

• High-volume events
• Multiple notification channels (SMS, App, Email)
• Controlled retries
• Failure isolation
• End-to-end observability

The challenge is to integrate Apache Kafka with Oracle Integration Cloud (OIC) while ensuring:

• Guaranteed message delivery
• Retry and dead-letter handling
• Clean separation between orchestration and channel delivery

---

🎯 Solution Overview

This solution leverages:

• Apache Kafka as the event backbone
• Oracle Integration Cloud (OIC) for orchestration
• Retry Topic + Dead Letter Topic (DLT) pattern
• Channel-specific child integrations
• External notification providers

---

🏗️ High-Level Architecture

Core Components

• Kafka Topics
  • notification-main-topic
  • notification-retry-topic
  • notification-dead-topic
• OIC Integrations
  • Publisher Integration
  • Main Consumer Integration
  • Retry Consumer Integration
  • Child Channel Integrations (SMS, App, Email)

---

🔄 End-to-End Flow

1️⃣ Event Publishing to Kafka

• Source system (CCS / upstream app) sends notification request to OIC.
• OIC Publisher Integration publishes the event to:
  • Kafka Main Topic

OIC connects to Kafka via REST Proxy or OCI Service Connector / API Gateway (depending on deployment).

---

2️⃣ Kafka Main Topic Consumption

• OIC Main Consumer Integration consumes messages from:
  • notification-main-topic
• Performs:
  • Payload validation
  • Header enrichment (correlationId, retryCount)

---

3️⃣ Notification Orchestration

The OIC Main Service Orchestrator:

• Identifies notification type
• Routes to the correct child integration:
  • SMS
  • App Push
  • Email

---

4️⃣ Channel-Specific Child Integrations

Child Integration	External System
OIC-Child-SMS	Twilio
OIC-Child-App	Azure
OIC-Child-Email	SendGrid

Each child:

• Invokes external provider
• Returns success or failure response

---

5️⃣ Retry Handling with Kafka Topics

If processing fails:

• Message is published to Kafka Retry Topic
• retryCount is incremented
• Optional delay logic applied (consumer-side or scheduler-based)

Retry Consumer Integration:

• Consumes from notification-retry-topic
• Reprocesses message
• Republishes to Main Topic if retry count < max

---

6️⃣ Dead Letter Topic (DLT) Handling

When retry count exceeds threshold:

• Message is published to:
  • notification-dead-topic
• Dead messages are:
  • Logged to Datadog
  • Stored for audit and manual reprocessing

---

🔁 Retry Logic Summary

Stage	Action
Main Topic Failure	Send to Retry Topic
Retry Count < Max	Reprocess
Retry Count = Max	Send to Dead Topic
Dead Topic	Monitoring + Alerting

---

📊 Design Diagram (Kafka + OIC)

Source System (CCS)
        |
        v
   OIC Publisher
        |
        v
+-----------------------+
| Kafka Main Topic      |
| notification-main     |
+-----------------------+
        |
        v
 OIC Main Consumer
        |
        v
 OIC Main Orchestrator
   |        |        |
   |        |        |
  SMS      APP     EMAIL
   |        |        |
 Twilio   Azure  SendGrid
   |
Failure?
   |
   v
+-----------------------+
| Kafka Retry Topic     |
| notification-retry    |
+-----------------------+
        |
        v
 OIC Retry Consumer
        |
Retry < Max?
   |        |
  Yes      No
   |        |
   v        v
Main     Kafka Dead Topic
Topic    notification-dead
             |
             v
          Datadog

---

🛠️ Kafka Design Best Practices

✅ Separate topics for main, retry, and dead messages
✅ Include retryCount and correlationId in headers
✅ Use consumer groups for scalability
✅ Idempotent processing in OIC
✅ Externalized retry thresholds
✅ Centralized logging and monitoring

---

📌 OIC Integration Design Notes

• Use App-Driven Orchestration for consumers
• Use Stage File or ATP for temporary persistence (optional)
• Secure Kafka via:
  • SASL / SSL
  • API Gateway
• Use Tracking ID for end-to-end visibility

---

🚀 Benefits of Kafka-Based Design

✔ High throughput & durability
✔ Decoupled producers and consumers
✔ Controlled retries without message loss
✔ Cloud-agnostic architecture
✔ Easy scaling across consumers

---

🔚 Conclusion

By integrating Apache Kafka with Oracle Integration Cloud, organizations can build a resilient, scalable, and fault-tolerant notification framework that supports enterprise-grade messaging with full observability and retry control.