OIC - Read latest file from SFTP server | Poll the file last time modified from a SFTP server

Usecase: Client has given a requirement that the file polling folder may have multiple files and we only need to read or poll latest file or the file last time modified.

Logic flow:

1. Create a sftp and rest connections.
2. Create an appdriven integration and configure rest to read the latest file and contents as response.
3. List the files using sftp adapter.
4. Read the latest file using sftp adapter. The tweaking, we have to do in the mapping file itself. 
1. For each file, we have to sort as descending order based on lastModifiedTine
2.  and if position =1 map the file name and ditectory
5. Then map the read file contents  and file name to rest response.

Used json:

{

"FileName":"",

"Employees":{

"Employee":[

{"Empid":1,"Fname":"","Lname":"","Dept":"","Age":"","Country":"","Skill":""},

"Empid":1,"Fname":"","Lname":"","Dept":"","Age":"","Country":"","Skill":""}

]

}

}

Map part for reading latest file:

<nstrgmpr:SyncReadFile>

<ns27:FileReadRequest>

<xsl:for-each select="$listFiles/nsmpr2:ListFileResponse/ns24:ListResponse/ns24:FileList/ns24:File">

<xsl:sort select="ns24:lastModifiedTime" order="descending"/>

<xsl:if test="position()=1">

<ns27:filename>

<xsl:value-of select="ns24:filename"/>

</ns27:filename>

<ns27:directory>

<xsl:value-of select="ns24:directory"/>

</ns27:directory>

</xsl:if>

</xsl:for-each>

</ns27:FileReadRequest>

</nstrgmpr:SyncReadFile>

Detailed screenshots:

Integration flow:

Configure rest trigger to read the latest file name and contents as response.

List the files:

Read the latest file:

Add tracking, save, activate and test

OIC - Gen3 - How to publish and subscribe events | What is oracle integration events | what is publish event action

Oracle has recently introduced a new feature called Event in Oracle integration generation-3 which is equivalent to the Publisher and Subscriber model of Oracle integration generation-2.

What is event?| what is oracle integration events? | Restrictions?

• Events are raised when something happens in the system such as employee onboarded, Order created etc. Once the events are published, multiple applications can subscribe to the events
• Starting OIC version 23.06 introduced this event concept.
• Support only JSON payload.
• The publish and subscribe feature enable us to decouple producers and subscribers.
• A maximum of 20 subacribers can subscribe to events per service instance.

Steps to follow:

1. Define event and Payload (Only JSON and xml schema supported)
2. Create Publish Integrations
3. Create Subscribe integrations

Detailed Steps with screenshots:

Create event:

Publish event from publisher integrations:

Create subscriber events

OIC - How many different scheduled integrations can run in parallel in Oracle Integration | Decoupled scheduler and Business Logic pattern.

Though OIC now provides dynamic threading model for scheduled orchestration integrations but it still has thread limitation. Thread count is internal to Oracle and may vary based on instance / subscription type.

Scheduled integration interfaces are widely used in batch integrations. Oracle Integration cloud provides feature to build scheduled integration flows, that can execute at defined frequency. However sometimes users complain about inconsistent behavior of scheduled flows like schedules are getting delayed by few minutes to few hours.

In this blog, We will discuss a common design issue that causes such incidents and solution for it.

What problem we faced and observed:

If we have a large of number of scheduled integrations running in parallel, then we will observe that few are running and others kept in queued and the backlog getting increased which delayed the processing of the backlog scheduled integrations.

For instance:

 I have created 6 scheduled integrations and run them in parallel and observed:

• 4 scheduled integrations are running in parallel. 
• whereas 2 integrations are in queued. 
• Once running ones completed, then the queued one started running. 

So what I understand for my OIC instance it has 4 scheduler threads limitation.

• o it can be made as re-usable component.

Solution:

We can decouple scheduler and integration logic into 2 different flows and let scheduled integration just invoke the child integration using Fire & Forget pattern.

1. For decoupled design, instead of creating many scheduled inegrations, we can create 1 or few  parent (Scheduled integrations) and others are Child (non-scheduled) application integration flows, simply exposed as REST endpoint.
2. Child flow exposes REST endpoint with "POST" verb and not returning any response hence making it a "One Way" flow. So it can be invoked as Fire & Forget .
3. Each Parent flow invokes its respective child flow. However since child is "one way" flow its not returning anything. So parent flow won't wait for the response after invoking child flow. It will either terminate and release the thread or move to next activity (if any).
4. Child integrations will use worker thread for processing. So this time, when one or few scheduled integration triggered at the same time, they were terminated within few seconds and there were 6 or more child integrations in "running" state, Running in parallel.

Reference:

https://www.ateam-oracle.com/post/oic-scheduler-decoupled-scheduler-and-business-logic-pattern

OIC - Best Practice - Run large number of scheduled integrations at the same time | avoid creating too many scheduled integrations

When there are too many schedule integrations configured, instances can become backlogged waiting for resources to become available or previous integration runs to complete. This can cause processing delays where some instances are in a waiting state longer than they must be and schedules may not start at the scheduled time.

Best practice:

A. Do not create too many schedule integrations that are scheduled to run at the same time. Where possible:

• Use an asynchronous REST Adapter trigger instead of a scheduled trigger when an active schedule is not absolutely required.
• Do not create any long-running schedule integrations (a schedule integration that takes longer than one hour, for example, to complete). This blocks scheduler resources impacting other scheduled runs.
• Spread schedules out over time to avoid schedule clusters.

B. We can convert a schedule integration to a REST Adapter trigger-based application integration.

If you absolutely require a large number of schedule integrations, and you encounter the previously described issue, the following design changes as a solution are recommended:

1. For each schedule integration, convert it to a REST Adapter trigger-based application integration.
2. Create a new schedule integration that only performs an asynchronous invoke of the application integration you converted in step 1 above.

   This solution enables the schedule integration to start at the scheduled time, invoke the REST Adapter trigger-based child integration asynchronously, and complete within milliseconds. This approach reduces the backlog and contention surrounding scheduler resources.

   If you have a large number of schedule integrations to convert, a staged approach starting with the following integrations is recommended.

   • Longest-running schedule integrations.
   • Schedule integrations configured with the shortest frequency (for example, integrations that run every 10 minutes or less).

   Design any new schedule integrations with the design practices described above.

Reference:

https://docs.oracle.com/en/cloud/paas/application-integration/integrations-user/common-integration-style-pitfalls-and-design-best-practices.html

OIC - Gen3 - Parallel action

Parallel action:

OIC 3 brings new features like Parallel flow, Cut -paste and publish events etc. 

Using Parallel flow one can define branches to run two or more actions in parallel. Parallel flows enable to perform multiple tasks at the same time, It is specially useful when we must perform several time consuming and independent tasks. Max 5 Branches we can create in parallel action.

Below screeshot: Where from a scheduled integration, we are running 3 rest services in parallel and in same time.

One more example:

Suppose we have to subscribe Purchase order events or employee events from ERP and then we need to send this information to multiple downstream applications that we can acheive using parallel action which will save a lot of processing time.

OIC - what is the maximum concurrent / parallel OIC scheduled flows limit?

• The maximum concurrent / parallel OIC Scheduled flows limit is 2 per managed server. 
• In case of a 2 node cluster, 4 scheduled flows can run in parallel.

Jobs of scheduled integration flows are controlled by IcsFlowIncompatibility property.

If user chooses to run the integration flow using ad hoc submit Now option, tgen IcsFlowIncompaProp of thr schedule job will be like:

<FlowCode>:<MajorVersion>:RunNow

If user defines a schedule for the integration flow and starts it, then IcsFlowIncompatProp of the scheduled job will be:

<FlowCode>:<MajorVersion>:SCHEDULENAME

Job incompatibility will restrict jobs with same IcsFlowIncompatprop value from running in parallel.

Hence,

• 2 submit runs of the same flow can not run in parallel.
• 1 submit run and 1 scheduled run can run in parallel.

WSDL - Asyn Callback service | Asyn BPEL WSDL vs Sync BPEL WSDL

Asyn callback users Wsdl file:

<?xml version="1.0" encoding="UTF-8"?>

<wsdl:definitions name="UserWSDL" targetNamespace="http://xmlns.oracle.com/Application1/Project1/UserWSDL" xmlns:tns="http://xmlns.oracle.com/Application1/Project1/UserWSDL"  xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:inp1="http://www.example.org">

<wsdl:types>

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:emp="http://www.example.org" targetNamespace="http://www.example.org" elementFormDefault="qualified">

<xsd:element name="Users">

<xsd:complexType>

<xsd:sequence>

<xsd:element name="User" maxOccurs="unbounded">

<xsd:complexType>

<xsd:sequence>

<xsd:element name="id" type="xsd:integer"/>

<xsd:element name="firstName" type="xsd:string"/>

<xsd:element name="lastName" type="xsd:string"/>

<xsd:element name="email" type="xsd:string"/>

<xsd:element name="country" type="xsd:string"/>

</xsd:sequence>

</xsd:complexType>

</xsd:element>

</xsd:sequence>

</xsd:complexType>

</xsd:element>

<xsd:element name="CallBackStatus">

<xsd:complexType>

<xsd:sequence>

<xsd:element name="Status" type ="xsd:string"/>

</xsd:sequence>

</xsd:complexType>

</xsd:element>

</xsd:schema>

</wsdl:types>

<wsdl:message name="requestMessgae">

<wsdl:part name="payload" element="inp1:Users"/>

</wsdl:message>

<wsdl:message name="callbackMessgae">

<wsdl:part name="payload" element="inp1:CallBackStatus"/>

</wsdl:message>

<wsdl:portType name="UserServicePort">

<wsdl:operation name="UserGetOP">

<wsdl:input message="tns:requestMessage"/>

</wsdl:operation>

</wsdl:portType>

<wsdl:portType name="callbackstatus_ptt">

<wsdl:operation name="callbackstatus">

<wsdl:input message="tns:callbackMessage"/>

</wsdl:operation>

</wsdl:portType>

</wsdl:definitions>

Async BPEL WSDL vs Sync BPEL WSDL:

Sync BPEL:

In SOA if our bpel is of synchronous type then we can make only synchronous calls and for which we will get the response immediately, So this case BPEL wsdl contains single port having two operations (request, response) in it.

Asyn BPEL:

If our BPEL is of asynchronous type then we can make sync and async calls. This case BPEL wsdl contains two ports having separate operations request, response. Here we need to observe one thing is if the response/callback takes long time then we should use separate port.