Friday, October 11, 2019

12c OSB - split large xml files

In current process, Source application sends New hire and change employee data into a apps DB. OSB service pulls the large payload (>2K employee elements) using apps adapter and sends the data as single instance to SOA service which again push the data in a JMS queue from where soa service pulls the data and sends to the EDN as events to a default JMS queue from where all the subscribing event composites processing the data.

Cons: This huge payload is processing as a single instance payload through soa service and this is taking huge time to process and the instance is also not loaded/not opening.

Now we are splitting the large payload into smaller chunks of batches (each batch size =500 employee elements) and write them in a directory using JCA based file adapter. Then using another proxy service we are reading all the files in 10 mins interval and then sends to SOA service.

Pros: This has resolved “the instance in not loading/opening issue”. Now all instances are getting loaded/opened with success.   

Pipeline flow :

Activity wise Split logic explained:

1.Assign: Take a batch Load
Value: number('500')
Variable: batchLoad
2.Assign: Count total employees
Value: fn:count($body/emp:WF_EVENT_T/EVENT_DATA/hrms_event_data/employee_data/employee)
Variable: varTotalEmpCounts

3.Assign : Taking the number of batches or iteration
fn:ceiling(xs:int($varTotalEmpCounts) div number($batchLoad))
4.Assign: Taking the body payload into a tempBody variable
5.Delete: Emptying the employee elements from the tempBody. This way it is keeping the header and other elements.Only employee elements are deleted.
6.Assign: Now assigning the emptied employee elements into 2nd tempBody2 variable.
7.Assign: Taking an initial batch count
8.For Each: For each employee we are now adding the the employee into the tempBody variable until the batch size(500) meets.
9.Insert: employees into the tempBody variable
10. IF: Below is the condition to stop the for each loop once the batch size meets:
$varIndex = (number($batchLoad) * number($batchCount)) or
$varIndex = $varTotalLoopCount
11.Publish: Publish the batch in a file directory.
13.Assign: then again assign tempbody2 payload into the tempBody variable
14.Assign: Then increasing the batch count

File Write JCA:
<adapter-config name="fileWrite" adapter="file" wsdlLocation="../WSDL/fileWrite.wsdl" xmlns="">
  <connection-factory location="eis/FileAdapter"/>
  <endpoint-interaction portType="Write_ptt" operation="Write">
    <interaction-spec className="oracle.tip.adapter.file.outbound.FileInteractionSpec">
      <property name="PhysicalDirectory" value="/soashare/abc/fileIn"/>    
      <property name="Append" value="false"/>
      <property name="FileNamingConvention" value="Employee%yyMMddHHmmssSS%.xml"/>
      <!--<property name="SingleThreadModel" value="true"/>-->


File Read JCA:
<adapter-config name="fileReadService" adapter="file" wsdlLocation="../WSDL/fileReadService.wsdl" xmlns="">
  <connection-factory UIincludeWildcard="*.xml" location="eis/FileAdapter"/>
  <endpoint-activation portType="Read_ptt" operation="Read">
    <activation-spec className="oracle.tip.adapter.file.inbound.FileActivationSpec">
      <property name="PhysicalDirectory" value="/soashare/abc/fileIn"/>
      <property name="PhysicalArchiveDirectory" value="/soashare/abc/archive"/>
      <property name="UseHeaders" value="false"/>
      <property name="MinimumAge" value="0"/>
      <property name="Recursive" value="false"/>
      <property name="PollingFrequency" value="600"/>
      <property name="MaxRaiseSize" value="1"/>
      <property name="SingleThreadModel" value="true"/>
      <property name="DeleteFile" value="true"/>
      <property name="IncludeFiles" value=".*\.xml"/>

Function of <property name="SingleThreadModel" value="true"/>
You want to configure File Adapters that it should pick one file at time from the given location with given polling interval.

Solution :
You set the "SingleThreadModel" and "MaxRaiseSize" properties for your file adapter.
Edit the adapter's jca file and add the following properties:
property name="SingleThreadModel" value="true"
property name="MaxRaiseSize" value="1"

No comments:

Post a Comment

Featured Post

11g to 12c OSB projects migration points

1. Export 11g OSB code and import in 12c Jdeveloper. Steps to import OSB project in Jdeveloper:   File⇾Import⇾Service Bus Resources⇾ Se...