see comments in-line
hrishi09 wrote:
> Hi Jim,
> Thanks for the explanation.
> As per this discussion what i understand is,
>
> 1. File poller (inbound processor) just picks up files from polling
> directory and put them in staging area.
> 2. Worker threads *Actually Reads* *and Process* the files.
> 3. Max Concurrency Limit is the number of records, worker threads can
> process without bothering about ACK or NACK
> and worker threads will not process further untill ACK or NACK is
> received for (Max Concurrency Limit) no of previously sent
> records/messages.
correct
>
> As per ur previous post, I think No of worker threads are what we
> Specify as *Outbound Processor Threads*
now the number of per endpoint inbound worker thread are fix (hard coded 5)
it has nothing to do with *Outbound Processor Threads *count - 5 is just
coincident
regards
--Jim
>
> Is what i understood is right Jim?
> Thanks,
> Hrishi
> On Thu, Oct 22, 2009 at 10:16 PM, Jim Fu [via OpenESB Users] <[hidden
> email]
> <
http://n2.nabble.com/user/SendEmail.jtp?type=node&node=3876846&i=0>>
> wrote:
>
> there is no such a direct 1-to-1 mapping between max concurrency
> limit and the per endpoint inbound processor.
>
> here is the micro-details:
>
> for each every activated inbound endpoint, there is a dedicated
> inbound processor (thread responsible for periodically scan the
> input directory and move those matching files into a staging area
> - a sub directory by name 'filebc-in-processing' AND meanwhile put
> the file NAMES into a QUEUE - note, only files names are queued,
> there is no file content reading by the poller), and with each
> poller, there is 5 worker threads which are waiting on the QUEUE,
> and responsible for:
>
> (1) de-Q a staged file
> (2) read the file
> (3) normalize payload into a JBI message
> (4) send message into NMR
> (5) back to the Q for more messages
>
> the poller is not limited by "max concurrency limit" since it does
> not consume resources in proportion to the size of the files
> contents - so a greedy algorithm used to poll and staging files as
> much as possible
> the "max concurrency limit" is enforced by the workers in such a
> way that if there are still "max concurrency limit" 10 pending
> sends not completed yet, a worker will block until some pending
> sends complete...
>
> as described above, the poller is using a greedy algorithm when
> doing directory scan & poll, however, to prevent "starvation" of
> other poller instances on the same endpoint (this could occur when
> the SA is deployed in a clustered target server), the poller
> thread will yield (break out file name matching loop) for a small
> fraction of time if the input directory contains large number of
> file entries.
>
> this is the micro-details of how inbound processing is performed
> and how "max concurrency limit" is enforced.
>
> as an end user, on a high level, one could think the inbound
> processing is performed "concurrently"
>
> regards
> Jim
>
> hrishi09 wrote:
>> Hi Jim,
>> If set max concurrency limit to 10 and i put 10 files into input dir at the
>> same time
>> then will those files be read at same time and processed concurrently?
>> Thanks,
>> Hrishi
>>
>> Jim Fu wrote:
>>
>>> FYI,
>>>
>>> the outbound threading model might be different if you use a CDK
>>> (component development toolkit) to facilitate component development
>>> where a different outbound threading model is used - instead of the
>>> fixed configurable per component thread count.
>>>
>>> however, I am not sure if there is a adaptable worker thread pool used
>>> on the outbound. may be the CDK developers can comment on this..
>>> regards
>>> Jim
>>>
>>> Jim Fu wrote:
>>>
>>>> Hi,
>>>>
>>>> "Number of Outbound Processor" in the "Runtime Configuration" section
>>>> is the value set for the number of outbound processing threads that
>>>> are created and started when the filebc is started
>>>>
>>>> the threading model of binding components is implementation specific,
>>>> for most of the binding components in open jbi componets, the
>>>> threading model hasn't changed much since their early days, e.g filebc
>>>> (and I believe bunch of other BCs), the number of outbound processing
>>>> threads are per component and is a fixed configurable value (default
>>>> to 5 as shown in the screen shot), in another words, these fixed
>>>> number of outbound processing threads are responsible to process all
>>>> the out-going BI messages directed to any outbound file target...,
>>>> there are discussions about making the outbound threading more
>>>> adaptable to the inflow of messages but AFAIK not on the top of the to
>>>> do list.
>>>>
>>>>
>>>>
>>>>
>>>> on the inbound side, filebc will spin off a inbound processor (a
>>>> thread) per endpoint, this inbound processor thread is responsible to
>>>> poll the specified input directory for matching files, and move them
>>>> into a staging area and meanwhile queuing the staged files into a
>>>> Queue where a fixed number of worker threads (5) are waiting & doing
>>>> further processing (i.e. - the actual file content reading, JBI
>>>> messaging wrapping (normalization), and pump it into NMR), note that
>>>> the work threads are doing "hefty" part of the message routing, so the
>>>> FILEBC inbound threading model can be described as:
>>>>
>>>> per endpoint 1 (poller) +5 (workers)
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> the "Max Concurreny Limit" (Throttling) basically puts a cap on the
>>>> number of pending inbound message routing PER ENDPOINT, i.e. when the
>>>> cap has been hit, the poller for the particular ENDPOINT will not poll
>>>> any inbound message until the current pending message route on the
>>>> endpoint fall back below the cap - e.g. a NMR send get ACK or NACK etc.
>>>>
>>>> "Are Number of Outbound Processor Threads and Max concurrency limit
>>>> related"?
>>>> I'd say indirectly, throttling is enforced on each inbound endpoint,
>>>> in timing, it is before the message reach the outbound processing,
>>>> however, the outbound processing threading model could cause bottle
>>>> neck if even with all the inbound endpoints throttling combined, the
>>>> out-going messages flow targeted to the BC outbound are too many to
>>>> process for the pre-configured outbound processor threads...
>>>>
>>>> regards
>>>>
>>>> Jim
>>>>
>>>> hrishi09 wrote:
>>>>
>>>>> Hi all,
>>>>> I want to know how concurrency is achieved using File BC.
>>>>> What is Number of Outbound Processor Threads : i could't get that from
>>>>> documentation at
>>>>>
http://wiki.open-esb.java.net/Wiki.jsp?page=FileBCRuntimeConfiguration>>>>>
http://wiki.open-esb.java.net/Wiki.jsp?page=FileBCRuntimeConfiguration
>>>>> What is Max concurrency limit? i've read Throttling and Serial
>>>>> Processing
>>>>> topic, looking for more explanation.
>>>>> Are Number of Outbound Processor Threads and Max concurrency limit
>>>>> related
>>>>> Thanks,
>>>>> Hrishi
>>>>>
>>>>>
>>>
>>
>
>
>
> ------------------------------------------------------------------------
> View this message in context: Re: File Bc : Concurrency
> <
http://n2.nabble.com/File-Bc-Concurrency-tp3864018p3876846.html>
> Sent from the OpenESB Users mailing list archive
> <
http://n2.nabble.com/OpenESB-Users-f794670.html> at Nabble.com.
