Saturday, May 8, 2021

OIC - How to handle more than 10 mb file or large file poll

How to handle more than 10  mb file or large file poll???

If we use ftp or file adapter to poll files from a directory, it can only allow to poll or read file content max 10MB file.

So in order to read large files(>10 MB),  we’ll have to download the file to a Virtual file system or a stage location as it supports 1 GB data poll but  if you want to read the content, again it supports 10MB , so using file read in segment , best option to read contents in segments or smaller chunks and then marge it to create full target specific file.

Basically, we need to follow the following sequences:

  1. First list the files using ftp.
  2. Then for each file
  3. Download each file to stage location
  4. Using stage activity, read file in segments. In esch segment it support 200 records to process.
  5. Create the required file format and marge each segment to create full file.

No comments:

Post a Comment

Featured Post

OIC - How to Retrieve Email Attachments Using Microsoft Graph API: Automating Payment File Processing

Retrieving Email Attachments Using Microsoft Graph API: Automating Payment File Storage in Object Storage Use Case: A finance team needs t...