OIC - Smart File Polling from OCI Object Storage with Dynamic Day-Based Logic and Datadog Logging

Use Case

A business requirement involves polling files from Oracle Cloud Infrastructure (OCI) Object Storage, processing them to a target file system via Agent, and logging any missing files in Datadog. The file polling count dynamically varies based on the day of the week:

• Friday: Expect 3 files
• Saturday: Expect 1 file
• Other days: Can be triggered manually via an adhoc flag

The solution ensures resilience through structured error handling and JSON-driven logic that categorizes files into MissingFiles and ProcessFiles.

Flow diagam

---

Solution Architecture Overview

The solution is designed using 3 OIC integrations and a supporting JSON structure:

{
  "MissingFiles": [
    {
      "FileName": ""
    }
  ],
  "ProcessFiles": [
    {
      "FileName": ""
    }
  ]
}

🔁 Integration 1: Scheduler Integration

• Purpose: Triggers the flow based on the scheduled time or ad-hoc execution.
• Steps:
  • Runs on a schedule (typically daily).
  • Accepts a flag adhocExecution = Y/N to override weekday logic.
  • Calls the Main Integration.

🔧 Integration 2: Main File Polling Integration

• Purpose: List and categorize files from OCI Object Storage.
• Steps:
1. List all files from a configured object storage bucket.
2. Determine required file count based on:
   • Day of the week (Friday = 3, Saturday = 1).
   • adhocExecution = Y allows polling on other days.
3. Compare expected vs actual files.
4. Populate a JSON object:
   • ProcessFiles: Files found and ready to process.
   • MissingFiles: Files not found (expected but missing).
5. For each ProcessFile, invoke the Child Integration.
6. Log MissingFiles to Datadog using REST API/log collector.

Missed file - throw fault

XSlT code for validating process files vs missed files

<xsl:template match="/xml:id_11">

  <ns1grp:write xml:id="id_117">

    <ns31:request-wrapper>

      <xsl:if test="($Var_dayOfWeek = &quot;Friday&quot;)">

        <xsl:if test="not($FileRef_Var/nsmpr0:response-wrapper/nsmpr0:objects[nsmpr0:name='BP_IDENTITY.csv'])">

          <ns31:MissingFiles>

            <ns31:FileName>

              <xsl:value-of select="'BP_IDENTITY.csv'"/>

            </ns31:FileName>

          </ns31:MissingFiles>

        </xsl:if>

        <xsl:if test="not($FileRef_Var/nsmpr0:response-wrapper/nsmpr0:objects[nsmpr0:name='BP_CREDITCARD.csv'])">

          <ns31:MissingFiles>

            <ns31:FileName>

              <xsl:value-of select="'BP_CREDITCARD.csv'"/>

            </ns31:FileName>

          </ns31:MissingFiles>

        </xsl:if>

        <xsl:if test="not($FileRef_Var/nsmpr0:response-wrapper/nsmpr0:objects[nsmpr0:name='DLP_CA_NUM.csv'])">

          <ns31:MissingFiles>

            <ns31:FileName>

              <xsl:value-of select="'DLP_CA_NUM.csv'"/>

            </ns31:FileName>

          </ns31:MissingFiles>

        </xsl:if>

      </xsl:if>

      <xsl:if test="($Var_dayOfWeek = &quot;Saturday&quot;)">

        <xsl:if test="not($FileRef_Var/nsmpr0:response-wrapper/nsmpr0:objects[nsmpr0:name='BP_NAMES.csv'])">

          <ns31:MissingFiles>

            <ns31:FileName>

              <xsl:value-of select="'BP_NAMES.csv'"/>

            </ns31:FileName>

          </ns31:MissingFiles>

        </xsl:if>

      </xsl:if>

      <xsl:if test="($Var_dayOfWeek = &quot;Friday&quot;) or (/nsmpr0:execute/ns17:request-wrapper/ns17:ProcessRequest/ns17:AdhocExecutionFlag = &quot;Y&quot;)">

        <xsl:if test="$FileRef_Var/nsmpr0:response-wrapper/nsmpr0:objects[nsmpr0:name='BP_IDENTITY.csv']">

          <ns31:ProcessFiles>

            <ns31:FileName>

              <xsl:value-of select="'BP_IDENTITY.csv'"/>

            </ns31:FileName>

          </ns31:ProcessFiles>

        </xsl:if>

        <xsl:if test="$FileRef_Var/nsmpr0:response-wrapper/nsmpr0:objects[nsmpr0:name='BP_CREDITCARD.csv']">

          <ns31:ProcessFiles>

            <ns31:FileName>

              <xsl:value-of select="'BP_CREDITCARD.csv'"/>

            </ns31:FileName>

          </ns31:ProcessFiles>

        </xsl:if>

        <xsl:if test="$FileRef_Var/nsmpr0:response-wrapper/nsmpr0:objects[nsmpr0:name='DLP_CA_NUM.csv']">

          <ns31:ProcessFiles>

            <ns31:FileName>

              <xsl:value-of select="'DLP_CA_NUM.csv'"/>

            </ns31:FileName>

          </ns31:ProcessFiles>

        </xsl:if>

      </xsl:if>

      <xsl:if test="($Var_dayOfWeek = &quot;Saturday&quot;) or (/nsmpr0:execute/ns17:request-wrapper/ns17:ProcessRequest/ns17:AdhocExecutionFlag = &quot;Y&quot;)">

        <xsl:if test="$FileRef_Var/nsmpr0:response-wrapper/nsmpr0:objects[nsmpr0:name='BP_NAMES.csv']">

          <ns31:ProcessFiles>

            <ns31:FileName>

              <xsl:value-of select="'BP_NAMES.csv'"/>

            </ns31:FileName>

          </ns31:ProcessFiles>

        </xsl:if>

      </xsl:if>

    </ns31:request-wrapper>

  </ns1grp:write>

</xsl:template>  

Integration 3: Child File Processor

• Purpose: Handles individual file transfer and cleanup.
• Steps:
• Download file from OCI Object Storage.
• Write the file to a local file system via Agent.
• Delete the file from OCI Object Storage post-processing.

---

Key Highlights

• ✅ Dynamic logic using weekday and ad-hoc flags.
• ✅ Robust processing pipeline using JSON mapping and loop controls.
• ✅ Clean-up mechanism ensures files aren't reprocessed.
• ✅ Monitoring integration using Datadog for transparency and alerting.