file-bc managing input file size

[raffaelespazzoli]

Hi,
in our integration scenario we have a file-bc that read a very large
file (it could be several GigByte). Such large file should be encoded in
xml and brought in memory. Clearly this isn't possible with these size
since it would cause glassfish esb to exhaust the available memory.
More in general an application that receive incoming data should have
some degree of control on the size of data it reads.
So is the a feature in the file-bc that allows to split the read file on
chunks of configurable size before reading data into memory?
Or may is there a sort of streaming feature for reading such big files?

thanks
Raffaele


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Maurizio Turatti]

Ciao Raffaele,

AFAIK that's why the JCAPS Batch eWay is still quite popular, it allows to do exactly what you mentioned. Unfortunately it is not open-sourced.

Maurizio

raffaelespazzoli wrote:

Hi,
in our integration scenario we have a file-bc that read a very large
file (it could be several GigByte). Such large file should be encoded in
xml and brought in memory. Clearly this isn't possible with these size
since it would cause glassfish esb to exhaust the available memory.
More in general an application that receive incoming data should have
some degree of control on the size of data it reads.
So is the a feature in the file-bc that allows to split the read file on
chunks of configurable size before reading data into memory?
Or may is there a sort of streaming feature for reading such big files?

thanks
Raffaele


---------------------------------------------------------------------
To unsubscribe, e-mail: [e-mail]
For additional commands, e-mail: [e-mail]

[SherryWeng]

Chunking the payload could become a bit complicated, but we did
experiment passing on the payload in a (file) input stream to avoid
loading the entire xml in memory at the binding level, and the stream
can be processed as needed. This is especially memory friendly in a
pass-through kind of work flow. Unfortunately, there are details we
still need to iron out collaborating with other ESB components in terms
of managing the stream, for obvious things such as to establish the
protocol to close the stream.
In this case, we're dealing with a file stream, but the same
protocol/mechanism can apply to other binding resources. Could you
please log a RFE so that we can prioritize this on the component plan?

Regards
--Sherry

Raffaele Spazzoli wrote:

> Hi,
> in our integration scenario we have a file-bc that read a very large
> file (it could be several GigByte). Such large file should be encoded in
> xml and brought in memory. Clearly this isn't possible with these size
> since it would cause glassfish esb to exhaust the available memory.
> More in general an application that receive incoming data should have
> some degree of control on the size of data it reads.
> So is the a feature in the file-bc that allows to split the read file on
> chunks of configurable size before reading data into memory?
> Or may is there a sort of streaming feature for reading such big files?
>
> thanks
> Raffaele
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>  

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Michael.Czapski]

Some javascript/style in this post has been disabled (why?)

Hello, Raffaele.

Can you elaborate on what is in the file, how you would decide the size of the chunks, what would happen to each chunk and whether the payload will end up as a single XML document or as multiple XML document?

You say:

(it could be several GigByte). Such large file should be encoded in
xml and brought in memory. 

This means that however you read your file your payload will have to be in memory (several GB multiplied by at least a factor of 3 for XML encoding), possibly more then once at the same time to allow transformations, if any. You probably need to look more closely at the requirements to see how else you can achieve what you need to achieve.

Regards

Michael


Raffaele Spazzoli wrote:

Hi,
in our integration scenario we have a file-bc that read a very large
file (it could be several GigByte). Such large file should be encoded in
xml and brought in memory. Clearly this isn't possible with these size
since it would cause glassfish esb to exhaust the available memory.
More in general an application that receive incoming data should have
some degree of control on the size of data it reads.
So is the a feature in the file-bc that allows to split the read file on
chunks of configurable size before reading data into memory?
Or may is there a sort of streaming feature for reading such big files?

thanks
Raffaele


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]