Re: Single point of failure - AGAIN

[Tod Harter]

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Hmmm, yeah, hmmmm. Performance implications?

       First I would set the Connection AutoCommit to false.

       In the Session class sendRaw(...) method we have the following

       if (persistMessages) {
       state.set(msgSeqNum, messageString);
       }
       state.incrNextSenderMsgSeqNum();

       I would rather have it as:

       if (persistMessages) {
       state.persist(msgSeqNum, messageString);
       }else{
               state.incrNextSenderMsgSeqNum();
       }

       The state.persist(...) would call a new method
messageStore.persist(sequence, message);
       The messageStore.persist(sequence, message) would get a
connection from the pool, open a transaction, insert the message in the
messages table, update the sequence number, then commit or rollback
based on success/failure and return a code to the caller. All that would
happen before the actual socket send.

--
The Wise adapts himself to the world. The Fool adapts the world to himself. Therefore, all progress depends on the Fool.

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

[vstolea]

Some javascript/style in this post has been disabled (why?)

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/

I'm not concerned about performance yet.

What I need, is to be sure there's no way we'll ever get in the situations described in my initial post - which are:

A - the lost of a message from a client (overwritten in the messages table) wich I won't be able to recover (table updates not in a single transaction).

B - wrong values in the database compared with what the app has in the MemoryStore (two threads updating both sequence numbers when the message processor and the timer should update only the incoming and the sending thread only the outgioing)

The reuse of the prepared statement would eliminate some time spent in preparing the statement (10-15 ms in our case) that are now used on each and every db call.

Thanks!

===========Tod Harter=================

Hmmm, yeah, hmmmm. Performance implications?

               First I would set the Connection AutoCommit to false.
       
               In the Session class sendRaw(...) method we have the following
       
               if (persistMessages) {
               state.set(msgSeqNum, messageString);
               }
               state.incrNextSenderMsgSeqNum();
       
               I would rather have it as:
       
               if (persistMessages) {
               state.persist(msgSeqNum, messageString);
               }else{
                       state.incrNextSenderMsgSeqNum();
               }
       
               The state.persist(...) would call a new method
        messageStore.persist(sequence, message);
               The messageStore.persist(sequence, message) would get a
        connection from the pool, open a transaction, insert the message in the
        messages table, update the sequence number, then commit or rollback
        based on success/failure and return a code to the caller. All that would
        happen before the actual socket send.
       
       

===========Original Message=================

This message is long - please bear with me.

Hi,

    I have a few questions regarding the quickfix/j engine and it's JdbcStore (the one we're using).
    We are starting as an Initiator.
    I realize that the engine is self healing in most of the cases but not in all... so:

 

1. Why the message is first sent to the executor and then stored?

2. Why are the two statements (insert_message and update_sequnece_number) NOT executed in a single transaction or somehow atomic?

3. Why is only one statement that updates both sequence numbers (incoming and outgoing) and not two statements that would deal with each one since the call is from different threads ()?

4. Why are the sql statements prepared each time after a connection is aquired?

Now some arguments

==================

For #1
    Since I'm supposed to be able to resend messages I would store the message first, then send it to the socket.
    Scenario:
            a)Message 101 is sent to the executor - but not received by it - still in the socket output buffer.
            b)Message 101 is stored in the messages table
            c)application crashes
            d)sequence number is still 100 in the session table
            ===============
            e) application is restarted
            f) engine and executor agree to the right sequence number (100)
            g) new order comes in and send to the executor with sequence # 101.
            h) new message overwrites the one already in the messages table - old message is lost.

For #2

    First I would set the Connection AutoCommit to false.
    In the Session class sendRaw(...) method we have the following

    if (persistMessages) {
        state.set(msgSeqNum, messageString);
    }
    state.incrNextSenderMsgSeqNum();

    I would rather have it as:

    if (persistMessages) {
        state.persist(msgSeqNum, messageString);
    }else{
        state.incrNextSenderMsgSeqNum();
    }

    The state.persist(...) would call a new method messageStore.persist(sequence, message);

    The messageStore.persist(sequence, message) would get a connection from the pool, open a transaction, insert the message in the messages table, update the sequence number, then commit or rollback based on success/failure and return a code to the caller. All that would happen before the actual socket send.

    For #3
    Scenario:
        a) Message is sent to the executor.
        b) For some reason the thread that store the message in the DB is delayed from scheduling - because of this the sequence numbers are not updated yet.

        c) the executor sends back a report and updates the sequence numbers with the values it gets from the MemoryStore.

        d) the sending thread is finally scheduled and updates the sequence numbers with the values it already got from the MemoryStore, thus setting an old value for the incoming message sequence number.

    If there would be a separate statement for each sequence number update, the above would not happen.

    For #4
    I understand this is a generic implementation, but still, time is lost in the prepare-statement method.
    A specialized class could provide the actual statements (from a hashmap) for a connection provided as argument...
    When DB operation is needed, in this case only a bind/execute/clear would be necessary.

Thanks for your time.
Viorel

---
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, re-transmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this email in error, please contact the sender immediately by return electronic transmission and then immediately delete this transmission, including all attachments, without copying, distributing or disclosing same. No member of the Scotiabank Group is liable for any errors or omissions in the content or transmission of this email or accepts any responsibility or liability for loss or damage arising from the receipt or use of this transmission. Scotiabank Group may monitor, retain and/or review email. Trading instructions received by e-mail or voicemail will not be accepted or acted upon. Unless indicated in writing, opinions contained in this email are those of the author and are not endorsed by any member of the Scotiabank Group. For information on some members of the Scotiabank Group, click here. For authorized users of the Scotia Capital trademark, click here.

Pour obtenir la traduction en français: http://www.scotiacapital.com/EmailDisclaimer/French.htm
Traducción en español: http://www.scotiacapital.com/EmailDisclaimer/Spanish.htm

------------------------------------------------------------------------------
Come build with us! The BlackBerry(R) Developer Conference in SF, CA
is the only developer event you need to attend this year. Jumpstart your
developing skills, take BlackBerry mobile applications to market and stay
ahead of the curve. Join us from November 9 - 12, 2009. Register now!
http://p.sf.net/sfu/devconference
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users