Processing large files (greater than 1 GB) in Oracle Integration Cloud (OIC) requires a well-thought-out design. OIC is an integration and orchestration platform, not a bulk file-processing engine. Therefore, the recommended approach is to offload heavy file handling and let OIC coordinate the process.
This blog covers the most effective and supported patterns to handle large files in OIC reliably.
Core Strategies for Large File Processing
1. Chunking (Segmentation)
File Chunking involves splitting a large file into smaller chunks and processing them one by one.
How It Works
The source system divides the file into smaller parts (for example, 50–100 MB chunks)
Each chunk is sent separately
OIC processes or routes each chunk
The target system reassembles the chunks into the original file
Benefits
Reduces memory pressure on OIC
Supports retries at chunk level instead of reprocessing the whole file
Improves reliability over unstable networks
Enables parallel processing if required
OIC Usage Notes
Pass chunk metadata (file name, chunk number, total chunks)
Use asynchronous integrations
Store chunk status externally if reassembly is required
2. Streaming
Streaming processes large files without loading the entire content into memory.
Benefits
Highly memory efficient
Faster processing start time
Suitable for continuous or large data flows
OIC Perspective
OIC does not support full custom streaming logic
Adapters such as FTP Adapter and OCI Object Storage Adapter internally stream files
Avoid mapping large payloads into integration variables
Best practice: Let adapters stream the file; keep OIC orchestration lightweight.
3. Cloud Object Storage (Preferred Pattern)
Using OCI Object Storage (or similar cloud storage) is the recommended approach for handling very large files.
Pattern Overview
Source uploads the file directly to Object Storage
OIC receives only file metadata (object name, bucket, URL)
OIC orchestrates downstream processing
Target system downloads the file directly from Object Storage
Benefits
Designed for multi-GB and TB-scale files
Built-in multipart upload support
High durability and scalability
Keeps OIC payload sizes small
Additional Recommended Approaches
4. OCI SDK / REST API (Multipart Uploads)
To bypass OIC’s payload size limitations:
Use OCI SDKs or OCI REST APIs
Perform multipart uploads directly to Object Storage
OIC only triggers or monitors the process
Benefits
No 1 GB payload limitation
Better performance and reliability
Ideal for custom upload/download logic
5. Oracle Functions for File Handling
Oracle Functions can act as a lightweight processing layer between systems.
Typical Use Case
Oracle Function downloads a large file from source
Performs validations or transformations if needed
Uploads the file to Object Storage using multipart upload
OIC is notified after completion
Benefits
Serverless and scalable
Keeps OIC free from heavy processing
Ideal for pre-processing or post-processing large files
Recommended Architecture
Source System → Object Storage / OCI SDK / Oracle Functions → OIC (Orchestration) → Target System
OIC’s role should be:
Triggering integrations
Orchestrating workflows
Handling errors and notifications
Tracking metadata (not file content)
Conclusion
When dealing with files larger than 1 GB in Oracle Integration Cloud:
✔ Use Object Storage as the central file repository
✔ Prefer chunking or streaming over full payload loads
✔ Leverage OCI SDKs, REST APIs, or Oracle Functions
✔ Keep OIC focused on orchestration, not file movement
Following these patterns ensures scalability, reliability, and compliance with OIC best practices.
No comments:
Post a Comment