Floating point values and problems with precision?

[Ivan Voras]

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Hi,

I am receiving a stream of market data that should nominally have the
scale of 4 decimal places to the right of the decimal dot, and seeing
raw protocol data I can confirm that this really is so. However, the
way I'm getting the data is via the getValue() method on the instance
of MDEntryPx, which returns the "double" data type.

Internally, the application immediately converts this value to
BigDecimal but I have noticed that the "double" value gets converted
literarily, with all the imprecision and problems in representation of
floating point numbers; e.g. all values have garbage decimals after
the 10th or so decimal place and some of them convert as a
semi-infinite sequences of 1s or 9s.

I'd like to ask:

a) why has this, seemingly dubious, design decision been make instead
of using BigDecimal all over the place and

b) if there is really no way to get a direct conversion between the
ASCII data as transmitted by the protocol and the application, is it
safe enough to use .scale() on the BigDecimal after the conversion
from double with 4 as the scale and the HALF_UP rounding mode?

--
f+rEnSIBITAhITAhLR1nM9F4cIs5KJrhbcsVtUIt7K1MhWJy1A==

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

[toli]

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Ivan,

By default QFJ uses doubles everywhere. You can re-generate the source
code to use BigDecimals instead - this is what we do in the
Marketcetera Trading Platform.

See http://www.quickfixj.org/quickfixj/usermanual/usage/codegen.html
for more information. You can specify -Dgenerator.decimal=true or
change the build.xml file to have
   <property name="generator.decimal" value="true" />

in order to recompile QFJ to use only BigDecimals.

On Tue, Sep 8, 2009 at 1:04 PM, Ivan Voras<[hidden email]> wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Hi,
>
> I am receiving a stream of market data that should nominally have the
> scale of 4 decimal places to the right of the decimal dot, and seeing
> raw protocol data I can confirm that this really is so. However, the
> way I'm getting the data is via the getValue() method on the instance
> of MDEntryPx, which returns the "double" data type.
>
> Internally, the application immediately converts this value to
> BigDecimal but I have noticed that the "double" value gets converted
> literarily, with all the imprecision and problems in representation of
> floating point numbers; e.g. all values have garbage decimals after
> the 10th or so decimal place and some of them convert as a
> semi-infinite sequences of 1s or 9s.
>
> I'd like to ask:
>
> a) why has this, seemingly dubious, design decision been make instead
> of using BigDecimal all over the place and
>
> b) if there is really no way to get a direct conversion between the
> ASCII data as transmitted by the protocol and the application, is it
> safe enough to use .scale() on the BigDecimal after the conversion
> from double with 4 as the scale and the HALF_UP rounding mode?
>
> --
> f+rEnSIBITAhITAhLR1nM9F4cIs5KJrhbcsVtUIt7K1MhWJy1A==
>
> ------------------------------------------------------------------------------
> Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
> trial. Simplify your report design, integration and deployment - and focus on
> what you do best, core application coding. Discover what's new with
> Crystal Reports now.  http://p.sf.net/sfu/bobj-july
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>

--
Toli Kuznets
http://www.marketcetera.com: Open-Source Trading Platform
download.run.trade.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

[Ivan Voras]

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


2009/9/8 Toli Kuznets <[hidden email]>:

> Ivan,
>
> By default QFJ uses doubles everywhere. You can re-generate the source
> code to use BigDecimals instead - this is what we do in the
> Marketcetera Trading Platform.
>
> See http://www.quickfixj.org/quickfixj/usermanual/usage/codegen.html
> for more information. You can specify -Dgenerator.decimal=true or
> change the build.xml file to have
>   <property name="generator.decimal" value="true" />
>
> in order to recompile QFJ to use only BigDecimals.

That's good to know, thanks!

On the other hand, we're really trying not to use custom builds of any
external library, so we're stuck.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

[toli]

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


> On the other hand, we're really trying not to use custom builds of any
> external library, so we're stuck.

Ivan, it's not technically a "custom build" - you are not modifying
any "real" code, just a compile-time flag and auto-generated code. But
i understand where you are coming from.

If that helps, you can either use Marketcetera
(http://www.marketcetera.com) out of the box, we are built on top of
QFJ, or you can use the BigDecimal libraries that we recompiled for
QFJ directly, which you can find at
http://repo.marketcetera.org/maven/quickfixj/quickfixj-all/1.3.3-bd/
However, we are still on QFJ-1.3.3 so we don't have the 1.4.0 ones
published yet.

hope this helps


--
Toli Kuznets
http://www.marketcetera.com: Open-Source Trading Platform
download.run.trade.

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

[Grant Birchmeier]

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


> b) if there is really no way to get a direct conversion between the
> ASCII data as transmitted by the protocol and the application, is it
> safe enough to use .scale() on the BigDecimal after the conversion
> from double with 4 as the scale and the HALF_UP rounding mode?

Would this clunky workaround help you?:
    quickfix.Message myMessage = <your callback's type>;
    BigDecimal myPrice = new BigDecimal(
        myMessage.getString(quickfix.field.MDEntryPx.FIELD));

This extracts the text from the fix message and uses it to construct
the BigDecimal.

-Grant

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

[jim_b_o]

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


You need to use BigDecimal.valueOf(double) not new BigDecimal(double) to
ensure accurate conversion.  See the JavaDoc of these two methods.
***********************************************************************************************
This email, its contents and any files attached are a confidential communication and are
intended only for the named addressees indicated in the message.

If you are not the named addressee or if you have received this email in error, you may not,
without the consent of Cognotec, copy, use or rely on any information or attachments in any way.
Please notify the sender by return email and delete it from your email system.

Unless separately agreed, Cognotec does not accept any responsibility for the accuracy or
completeness of the contents of this email or its attachments. Please note that any views,
opinion or advice contained in this communication are those of the sending individual
and not those of Cognotec and Cognotec shall have no liability whatsoever in relation to
this communication (or its content) unless separately agreed.
***********************************************************************************************

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

[Ivan Voras]

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


2009/9/9 James Olsen <[hidden email]>:

>
> You need to use BigDecimal.valueOf(double) not new BigDecimal(double) to
> ensure accurate conversion.  See the JavaDoc of these two methods.

I see now, and I think it's a terrible kludge :)

I like the suggestion given by Grant Birchmeier better - at least it
doesn't involve a trip through a floating point representation.

--
f+rEnSIBITAhITAhLR1nM9F4cIs5KJrhbcsVtUIt7K1MhWJy1A==

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now.  http://p.sf.net/sfu/bobj-july
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users