File BC : multiple files

4 messages Options
Embed this post
Permalink
hrishi09

File BC : multiple files

Reply Threaded More More options
Print post
Permalink
Hi all,
I want to know how File BC handles multiple files.
Is it beneficial to have a single file with large no of records or multiple files with less records.
I mean what would be efficient
1. Processing 1 file with 1 lakh records or
2. 10 files with 10000 records
It would be great if someone can can tell me pros and cons of above strategies.
Thanks,
Hrishi
Jim Fu

Re: File BC : multiple files

Reply Threaded More More options
Print post
Permalink
filebc is a jbi (jsr208) binding component, it is devised to help a
"message" flow in/out of ESB centered system(s), for filebc, the binding
'protocol' is file, so:

externally (among systems), the "message" flows through file system,
e.g. system 'A' write a file to path 'P' and system 'B' pick it up at
the path 'P'
internally (within a system), the "message" flows on the ESB bus (MxN
mix-match between BCs and SEs)

the unit of processing (routing) is the 'message', what makes up a
"message" depends on the application, e.g. a purchase order, a
prescription record for a patient, a medical provider's profile, etc.

in case of filebc, the "message" is persisted and transfered as file
system files, it can be one record per file or multiple-records per
file, the record can be delimited or fixed length enveloped.

most time, one record per file works fine, if it is needed to pack
multiple records into a physical file, then properly grouping (divide
into groups) the records into separate files are recommended...

the minimum unit of processing of filebc is "record" as described above,
if I found myself have to deal a record of 1GB, then I might want to
review by application design...

HTH
regards
Jim

hrishi09 wrote:

> Hi all,
> I want to know how File BC handles multiple files.
> Is it beneficial to have a single file with large no of records or multiple
> files with less records.
> I mean what would be efficient
> 1. Processing 1 file with 1 lakh records or
> 2. 10 files with 10000 records
> It would be great if someone can can tell me pros and cons of above
> strategies.
> Thanks,
> Hrishi
>  


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

hrishi09

Re: File BC : multiple files

Reply Threaded More More options
Print post
Permalink
Hi Jim,
Thanks for the reply.
Whatever i get from ur reply and as far as i know File BC processes a record at a time and is trated as a message within ESB.
Now let me specify my scenario.
I want to read a CSV file and put records from CSV into database.
Say my csv file structure is
col1,col2,col3,col4
My question is
If my file contains such 1 lakh records then is it efficient to process single file with 1 lakh records or
multiple files with less records
Thanks,
Hrishi
On Wed, Oct 21, 2009 at 11:18 PM, Jim Fu [via OpenESB Users] <[hidden email]> wrote:
filebc is a jbi (jsr208) binding component, it is devised to help a
"message" flow in/out of ESB centered system(s), for filebc, the binding
'protocol' is file, so:

externally (among systems), the "message" flows through file system,
e.g. system 'A' write a file to path 'P' and system 'B' pick it up at
the path 'P'
internally (within a system), the "message" flows on the ESB bus (MxN
mix-match between BCs and SEs)

the unit of processing (routing) is the 'message', what makes up a
"message" depends on the application, e.g. a purchase order, a
prescription record for a patient, a medical provider's profile, etc.

in case of filebc, the "message" is persisted and transfered as file
system files, it can be one record per file or multiple-records per
file, the record can be delimited or fixed length enveloped.

most time, one record per file works fine, if it is needed to pack
multiple records into a physical file, then properly grouping (divide
into groups) the records into separate files are recommended...

the minimum unit of processing of filebc is "record" as described above,
if I found myself have to deal a record of 1GB, then I might want to
review by application design...

HTH
regards
Jim

hrishi09 wrote:

> Hi all,
> I want to know how File BC handles multiple files.
> Is it beneficial to have a single file with large no of records or multiple
> files with less records.
> I mean what would be efficient
> 1. Processing 1 file with 1 lakh records or
> 2. 10 files with 10000 records
> It would be great if someone can can tell me pros and cons of above
> strategies.
> Thanks,
> Hrishi
>  

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


Jim Fu

Re: File BC : multiple files

Reply Threaded More More options
Print post
Permalink
Some javascript/style in this post has been disabled (why?)
in this case - i.e., input file contains multiple records, filebc will stage the file and assign one worker working on it, i.e. read record one by one and wrap each record as JBI message and send the message.
so, it is recommended that the CSV records be chunked into smaller files and feed to the filebc inbound, if such chunking is possible in your application.

the good news is that, the inbound throttling is effective on multiple records processing too - i.e. the worker will block if the pending NMR sends hit the "max concurrency limit"

regards

Jim

hrishi09 wrote:
Hi Jim,
Thanks for the reply.
Whatever i get from ur reply and as far as i know File BC processes a record at a time and is trated as a message within ESB.
Now let me specify my scenario.
I want to read a CSV file and put records from CSV into database.
Say my csv file structure is
col1,col2,col3,col4
My question is
If my file contains such 1 lakh records then is it efficient to process single file with 1 lakh records or
multiple files with less records
Thanks,
Hrishi
On Wed, Oct 21, 2009 at 11:18 PM, Jim Fu [via OpenESB Users] <[hidden email]> wrote:
filebc is a jbi (jsr208) binding component, it is devised to help a
"message" flow in/out of ESB centered system(s), for filebc, the binding
'protocol' is file, so:

externally (among systems), the "message" flows through file system,
e.g. system 'A' write a file to path 'P' and system 'B' pick it up at
the path 'P'
internally (within a system), the "message" flows on the ESB bus (MxN
mix-match between BCs and SEs)

the unit of processing (routing) is the 'message', what makes up a
"message" depends on the application, e.g. a purchase order, a
prescription record for a patient, a medical provider's profile, etc.

in case of filebc, the "message" is persisted and transfered as file
system files, it can be one record per file or multiple-records per
file, the record can be delimited or fixed length enveloped.

most time, one record per file works fine, if it is needed to pack
multiple records into a physical file, then properly grouping (divide
into groups) the records into separate files are recommended...

the minimum unit of processing of filebc is "record" as described above,
if I found myself have to deal a record of 1GB, then I might want to
review by application design...

HTH
regards
Jim

hrishi09 wrote:

> Hi all,
> I want to know how File BC handles multiple files.
> Is it beneficial to have a single file with large no of records or multiple
> files with less records.
> I mean what would be efficient
> 1. Processing 1 file with 1 lakh records or
> 2. 10 files with 10000 records
> It would be great if someone can can tell me pros and cons of above
> strategies.
> Thanks,
> Hrishi
>  

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]




View this message in context: Re: File BC : multiple files
Sent from the OpenESB Users mailing list archive at Nabble.com.