ETL: can't perform 180,000 rows extraction

[Santiago De Bernardez]

Hi, all.

I'm trying to replicate a table from an Oracle instance to another through the Data Integrator SE. The source table has about 180,000 records, and a single field should be copied into the target table.

When running the ETL, the process is frozen in the extraction stage, and remains so for hours. I finish killing the process.
I tried changing some parameters like "batch size", but got no improvement. I can only insert about 800 records if I use a conditional extraction.

Does someone know how to solve this?
Tomorrow I'll send the logs files and the project, if that helps (I'm at home).

Thanks.

Santiago.

ESTE MENSAJE ES CONFIDENCIAL. Puede contener información amparada por el secreto profesional. Si usted ha recibido este e-mail por error, por favor comuníquenoslo inmediatamente vía e-mail y tenga la amabilidad de eliminarlo de su sistema; no deberá copiar el mensaje ni divulgar su contenido a ninguna persona. Muchas gracias.

THIS MESSAGE IS CONFIDENTIAL. It may also contain information that is privileged or otherwise legally exempt from disclosure. If you have received it by mistake please let us know by e-mail immediately and delete it from your system; should also not copy the message nor disclose its contents to anyone. Many thanks.

	
		


	

[Nilesh Apte]

Hi Santiago:

Which ETL build are you using? We have fixed a similar issue some time  
back. Are you using GFESBV2.1? Could you try installing latest ETL SE  
component and try?
With you existing installation, you can try changing the "Execution  
Strategy" from "Best Fit" to "Pipeline" or "Staging" and see if it  
makes any difference in execution.

Thanks,
Nilesh.

On 20-Oct-09, at 11:45 AM, Santiago De Bernardez wrote:

> Hi, all.
>
> I'm trying to replicate a table from an Oracle instance to another  
> through the Data Integrator SE. The source table has about 180,000  
> records, and a single field should be copied into the target table.
>
> When running the ETL, the process is frozen in the extraction stage,  
> and remains so for hours. I finish killing the process.
> I tried changing some parameters like "batch size", but got no  
> improvement. I can only insert about 800 records if I use a  
> conditional extraction.
>
> Does someone know how to solve this?
> Tomorrow I'll send the logs files and the project, if that helps  
> (I'm at home).
>
> Thanks.
>
> Santiago.
> ESTE MENSAJE ES CONFIDENCIAL. Puede contener información amparada  
> por el secreto profesional. Si usted ha recibido este e-mail por  
> error, por favor comuníquenoslo inmediatamente vía e-mail y tenga la  
> amabilidad de eliminarlo de su sistema; no deberá copiar el mensaje  
> ni divulgar su contenido a ninguna persona. Muchas gracias.
>
> THIS MESSAGE IS CONFIDENTIAL. It may also contain information that  
> is privileged or otherwise legally exempt from disclosure. If you  
> have received it by mistake please let us know by e-mail immediately  
> and delete it from your system; should also not copy the message nor  
> disclose its contents to anyone. Many thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

[Santiago De Bernardez]

Hi, Nilesh.

We are using sun-etl-engine 2.7.0 (Build 090829).

I also tried using Pipeline and Staging strategy, but in the case of Staging, Oracle shows me I do not have sufficient privileges, and in the case of Pipeline, Netbeans throws an Java null Pointer exception.

Thanks.

Santiago.

On Tue, Oct 20, 2009 at 9:34 AM, Nilesh Apte <[hidden email]> wrote:

Hi Santiago:

Which ETL build are you using? We have fixed a similar issue some time back. Are you using GFESBV2.1? Could you try installing latest ETL SE component and try?
With you existing installation, you can try changing the "Execution Strategy" from "Best Fit" to "Pipeline" or "Staging" and see if it makes any difference in execution.

Thanks,
Nilesh.

On 20-Oct-09, at 11:45 AM, Santiago De Bernardez wrote:

Hi, all.

I'm trying to replicate a table from an Oracle instance to another through the Data Integrator SE. The source table has about 180,000 records, and a single field should be copied into the target table.

When running the ETL, the process is frozen in the extraction stage, and remains so for hours. I finish killing the process.
I tried changing some parameters like "batch size", but got no improvement. I can only insert about 800 records if I use a conditional extraction.

Does someone know how to solve this?
Tomorrow I'll send the logs files and the project, if that helps (I'm at home).

Thanks.

Santiago.

ESTE MENSAJE ES CONFIDENCIAL. Puede contener información amparada por el secreto profesional. Si usted ha recibido este e-mail por error, por favor comuníquenoslo inmediatamente vía e-mail y tenga la amabilidad de eliminarlo de su sistema; no deberá copiar el mensaje ni divulgar su contenido a ninguna persona. Muchas gracias.

THIS MESSAGE IS CONFIDENTIAL. It may also contain information that is privileged or otherwise legally exempt from disclosure. If you have received it by mistake please let us know by e-mail immediately and delete it from your system; should also not copy the message nor disclose its contents to anyone. Many thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

ESTE MENSAJE ES CONFIDENCIAL. Puede contener información amparada por el secreto profesional. Si usted ha recibido este e-mail por error, por favor comuníquenoslo inmediatamente vía e-mail y tenga la amabilidad de eliminarlo de su sistema; no deberá copiar el mensaje ni divulgar su contenido a ninguna persona. Muchas gracias.

THIS MESSAGE IS CONFIDENTIAL. It may also contain information that is privileged or otherwise legally exempt from disclosure. If you have received it by mistake please let us know by e-mail immediately and delete it from your system; should also not copy the message nor disclose its contents to anyone. Many thanks.

	

[Nilesh Apte]

Some javascript/style in this post has been disabled (why?)

Hi Santiago:

The fix for this (or a similar issue) was checked in after 090829. Could you please download latest ETLSE installer and try your scenario again with it?

In the meantime you can still send out the log file and the project for analysis.

Thanks,

Nilesh.

On 20-Oct-09, at 7:37 PM, Santiago De Bernardez wrote:

Hi, Nilesh.

We are using sun-etl-engine 2.7.0 (Build 090829).

I also tried using Pipeline and Staging strategy, but in the case of Staging, Oracle shows me I do not have sufficient privileges, and in the case of Pipeline, Netbeans throws an Java null Pointer exception.

Thanks.

Santiago.

On Tue, Oct 20, 2009 at 9:34 AM, Nilesh Apte <[hidden email]> wrote:

Hi Santiago:

Which ETL build are you using? We have fixed a similar issue some time back. Are you using GFESBV2.1? Could you try installing latest ETL SE component and try?
With you existing installation, you can try changing the "Execution Strategy" from "Best Fit" to "Pipeline" or "Staging" and see if it makes any difference in execution.

Thanks,
Nilesh.

On 20-Oct-09, at 11:45 AM, Santiago De Bernardez wrote:

Hi, all.

I'm trying to replicate a table from an Oracle instance to another through the Data Integrator SE. The source table has about 180,000 records, and a single field should be copied into the target table.

When running the ETL, the process is frozen in the extraction stage, and remains so for hours. I finish killing the process.
I tried changing some parameters like "batch size", but got no improvement. I can only insert about 800 records if I use a conditional extraction.

Does someone know how to solve this?
Tomorrow I'll send the logs files and the project, if that helps (I'm at home).

Thanks.

Santiago.

ESTE MENSAJE ES CONFIDENCIAL. Puede contener información amparada por el secreto profesional. Si usted ha recibido este e-mail por error, por favor comuníquenoslo inmediatamente vía e-mail y tenga la amabilidad de eliminarlo de su sistema; no deberá copiar el mensaje ni divulgar su contenido a ninguna persona. Muchas gracias.

THIS MESSAGE IS CONFIDENTIAL. It may also contain information that is privileged or otherwise legally exempt from disclosure. If you have received it by mistake please let us know by e-mail immediately and delete it from your system; should also not copy the message nor disclose its contents to anyone. Many thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

ESTE MENSAJE ES CONFIDENCIAL. Puede contener información amparada por el secreto profesional. Si usted ha recibido este e-mail por error, por favor comuníquenoslo inmediatamente vía e-mail y tenga la amabilidad de eliminarlo de su sistema; no deberá copiar el mensaje ni divulgar su contenido a ninguna persona. Muchas gracias.

THIS MESSAGE IS CONFIDENTIAL. It may also contain information that is privileged or otherwise legally exempt from disclosure. If you have received it by mistake please let us know by e-mail immediately and delete it from your system; should also not copy the message nor disclose its contents to anyone. Many thanks.

[Santiago De Bernardez]

Hi, Nilesh.

The ETL SE is already update, but I could not test it yet.

However, I have another question now.

How would you do to extract non-duplicate records from two tables in two different databases using Data Integrator?

In MySQL could be work with a query like this:

SELECT s1.subscription_number FROM subscriptions s1 WHERE s1.subscription_number NOT IN (SELECT s2.subscription_number FROM subscribers s2 )

assuming that the tables are in the same database.

Is this possible to do in the ETL designer?

Thanks!

Santiago.

On Tue, Oct 20, 2009 at 4:20 PM, Nilesh Apte <[hidden email]> wrote:

Hi Santiago:

The fix for this (or a similar issue) was checked in after 090829. Could you please download latest ETLSE installer and try your scenario again with it?

In the meantime you can still send out the log file and the project for analysis.

Thanks,

Nilesh.

On 20-Oct-09, at 7:37 PM, Santiago De Bernardez wrote:

Hi, Nilesh.

We are using sun-etl-engine 2.7.0 (Build 090829).

I also tried using Pipeline and Staging strategy, but in the case of Staging, Oracle shows me I do not have sufficient privileges, and in the case of Pipeline, Netbeans throws an Java null Pointer exception.

Thanks.

Santiago.

On Tue, Oct 20, 2009 at 9:34 AM, Nilesh Apte <[hidden email]> wrote:

Hi Santiago:

Which ETL build are you using? We have fixed a similar issue some time back. Are you using GFESBV2.1? Could you try installing latest ETL SE component and try?
With you existing installation, you can try changing the "Execution Strategy" from "Best Fit" to "Pipeline" or "Staging" and see if it makes any difference in execution.

Thanks,
Nilesh.

On 20-Oct-09, at 11:45 AM, Santiago De Bernardez wrote:

Hi, all.

I'm trying to replicate a table from an Oracle instance to another through the Data Integrator SE. The source table has about 180,000 records, and a single field should be copied into the target table.

When running the ETL, the process is frozen in the extraction stage, and remains so for hours. I finish killing the process.
I tried changing some parameters like "batch size", but got no improvement. I can only insert about 800 records if I use a conditional extraction.

Does someone know how to solve this?
Tomorrow I'll send the logs files and the project, if that helps (I'm at home).

Thanks.

Santiago.

ESTE MENSAJE ES CONFIDENCIAL. Puede contener información amparada por el secreto profesional. Si usted ha recibido este e-mail por error, por favor comuníquenoslo inmediatamente vía e-mail y tenga la amabilidad de eliminarlo de su sistema; no deberá copiar el mensaje ni divulgar su contenido a ninguna persona. Muchas gracias.

THIS MESSAGE IS CONFIDENTIAL. It may also contain information that is privileged or otherwise legally exempt from disclosure. If you have received it by mistake please let us know by e-mail immediately and delete it from your system; should also not copy the message nor disclose its contents to anyone. Many thanks.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

ESTE MENSAJE ES CONFIDENCIAL. Puede contener información amparada por el secreto profesional. Si usted ha recibido este e-mail por error, por favor comuníquenoslo inmediatamente vía e-mail y tenga la amabilidad de eliminarlo de su sistema; no deberá copiar el mensaje ni divulgar su contenido a ninguna persona. Muchas gracias.

THIS MESSAGE IS CONFIDENTIAL. It may also contain information that is privileged or otherwise legally exempt from disclosure. If you have received it by mistake please let us know by e-mail immediately and delete it from your system; should also not copy the message nor disclose its contents to anyone. Many thanks.

ESTE MENSAJE ES CONFIDENCIAL. Puede contener información amparada por el secreto profesional. Si usted ha recibido este e-mail por error, por favor comuníquenoslo inmediatamente vía e-mail y tenga la amabilidad de eliminarlo de su sistema; no deberá copiar el mensaje ni divulgar su contenido a ninguna persona. Muchas gracias.

THIS MESSAGE IS CONFIDENTIAL. It may also contain information that is privileged or otherwise legally exempt from disclosure. If you have received it by mistake please let us know by e-mail immediately and delete it from your system; should also not copy the message nor disclose its contents to anyone. Many thanks.