xmlrcp request takes very long time

[Juha Heinanen]

i wrote in python a simple text that sends domain_reload command over s
xmlrpc interface to sr:

---------------------------------------------------------------
import xmlrpclib

XMLRPC_PROTOCOL = 'http'
XMLRPC_SERVER   = '127.0.0.1'
XMLRPC_PORT = 6060

server_path = XMLRPC_PROTOCOL + '://' +  XMLRPC_SERVER + \
                ':' +  str(XMLRPC_PORT)

c = xmlrpclib.ServerProxy(server_path)

erg = c.domain_reload()
----------------------------------------------------------------

when i execute the program, ngrep immediately prints what one would
expect:

T 2009/07/04 17:20:02.455028 127.0.0.1:33418 -> 127.0.0.1:6060 [AP]
  POST /RPC2 HTTP/1.0..Host: 127.0.0.1:6060..User-Agent: xmlrpclib.py/1.0.1 (
  by www.pythonware.com)..Content-Type: text/xml..Content-Length: 107....    
##
T 2009/07/04 17:20:02.455358 127.0.0.1:33418 -> 127.0.0.1:6060 [AP]
  <?xml version='1.0'?>.<methodCall>.<methodName>domain_reload</methodName>.<
  params>.</params>.</methodCall>.                                          
##
T 2009/07/04 17:20:02.455718 127.0.0.1:6060 -> 127.0.0.1:33418 [AP]
  HTTP/1.0 200 OK..Via: SIP/2.0/TCP 127.0.0.1:33418..Server: SIP Route
  r (2.99.0-dev01-serk (i386/linux))..Content-Length: 274....<?xml version="1
  .0"?>.<methodResponse>.<fault>.<value>.<struct>.<member>.<name>faultCode</n
  ame>.<value><int>500</int></value>.</member>.<member>.<name>faultString</na
  me>.<value><string>Method Not Found</string></value>.</member>.</struct>.</
  value>.</fault>.</methodResponse>

however, python client program does not terminate until after more
than 2 minutes.  then it finally prints:

Traceback (most recent call last):
  File "sr-xmlrpc.py", line 15, in <module>
    erg = c.domain_reload()
  File "/usr/lib/python2.5/xmlrpclib.py", line 1147, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python2.5/xmlrpclib.py", line 1437, in __request
    verbose=self.__verbose
  File "/usr/lib/python2.5/xmlrpclib.py", line 1201, in request
    return self._parse_response(h.getfile(), sock)
  File "/usr/lib/python2.5/xmlrpclib.py", line 1340, in _parse_response
    return u.close()
  File "/usr/lib/python2.5/xmlrpclib.py", line 787, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault 500: 'Method Not Found'>

i also tested with a php xmlrcp client with same result.

any idea why it takes so long and why parsing of response fails?  is
something missing from the response?

-- juha

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Andrei Pelinescu-Onciul]

On Jul 04, 2009 at 17:30, Juha Heinanen <[hidden email]> wrote:
> i wrote in python a simple text that sends domain_reload command over s
> xmlrpc interface to sr:

[...]

> T 2009/07/04 17:20:02.455358 127.0.0.1:33418 -> 127.0.0.1:6060 [AP]
>   <?xml version='1.0'?>.<methodCall>.<methodName>domain_reload</methodName>.<
>   params>.</params>.</methodCall>.                                          
> ##
> T 2009/07/04 17:20:02.455718 127.0.0.1:6060 -> 127.0.0.1:33418 [AP]
>   HTTP/1.0 200 OK..Via: SIP/2.0/TCP 127.0.0.1:33418..Server: SIP Route
>   r (2.99.0-dev01-serk (i386/linux))..Content-Length: 274....<?xml version="1
>   .0"?>.<methodResponse>.<fault>.<value>.<struct>.<member>.<name>faultCode</n
>   ame>.<value><int>500</int></value>.</member>.<member>.<name>faultString</na
>   me>.<value><string>Method Not Found</string></value>.</member>.</struct>.</
>   value>.</fault>.</methodResponse>
>
> however, python client program does not terminate until after more
> than 2 minutes.  then it finally prints:

The reply looks ok to me. I've tried quickly writing an xmlrpc in perl
 (using XMLRPC::Lite) and tested using core rpcs and it works both with
  normal answers and with faults.

E.g:
 time perl ~/sr.git/xmlrpc_test.pl  foobar
 fault{
    faultCode: 500
    faultString: Method Not Found
  }

  real    0m0.240s

>
> Traceback (most recent call last):
>   File "sr-xmlrpc.py", line 15, in <module>
>     erg = c.domain_reload()
>   File "/usr/lib/python2.5/xmlrpclib.py", line 1147, in __call__
>     return self.__send(self.__name, args)
>   File "/usr/lib/python2.5/xmlrpclib.py", line 1437, in __request
>     verbose=self.__verbose
>   File "/usr/lib/python2.5/xmlrpclib.py", line 1201, in request
>     return self._parse_response(h.getfile(), sock)
>   File "/usr/lib/python2.5/xmlrpclib.py", line 1340, in _parse_response
>     return u.close()
>   File "/usr/lib/python2.5/xmlrpclib.py", line 787, in close
>     raise Fault(**self._stack[0])
> xmlrpclib.Fault: <Fault 500: 'Method Not Found'>
>
> i also tested with a php xmlrcp client with same result.
>
> any idea why it takes so long and why parsing of response fails?  is
> something missing from the response?

I don't know why is taking so long (especially since the reply is sent
immediately after the request). However I don't think the parsing fails.
I'm not familiar with python xmlrpc, but
xmlrpclib.Fault: <Fault 500: 'Method Not Found' looks ok given the
reply.

You get a fault reply because you try to call domain_reload which does
not exist (the correct rpc name is domain.reload).

Andrei

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Juha Heinanen]

when i try my test program with k mi_xmlrpc module, it works fine and
ngrep shows:

########
T 2009/07/06 18:12:34.028112 192.98.101.10:41014 -> 192.98.101.10:6060 [AP]
POST /RPC2 HTTP/1.0.
Host: 192.98.101.10:6060.
User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com).
Content-Type: text/xml.
Content-Length: 104.
.

##
T 2009/07/06 18:12:34.028289 192.98.101.10:41014 -> 192.98.101.10:6060 [AP]
<?xml version='1.0'?>
<methodCall>
<methodName>lcr_reload</methodName>
<params>
</params>
</methodCall>

##
T 2009/07/06 18:12:34.030189 192.98.101.10:6060 -> 192.98.101.10:41014 [AP]
HTTP/1.1 200 OK.

#
T 2009/07/06 18:12:34.030289 192.98.101.10:6060 -> 192.98.101.10:41014 [AP]
Content-type: text/xml; charset="utf-8".

#
T 2009/07/06 18:12:34.032315 192.98.101.10:6060 -> 192.98.101.10:41014 [AFP]
Content-length: 147.
Connection: close.
Date: Mon, 06 Jul 2009 16:12:34 GMT.
Server: XMLRPC_ABYSS/1.06.
.
<?xml version="1.0" encoding="UTF-8"?>.
<methodResponse>.
<params>.
<param><value><string></string></value></param>.
</params>.
</methodResponse>.

####

when i try domain.reload with xmlrpc module, it hangs and ngrep reports:

T 2009/07/06 18:18:25.496760 192.98.101.10:37656 -> 192.98.101.10:6060 [AP]
POST /RPC2 HTTP/1.0.
Host: 192.98.101.10:6060.
User-Agent: xmlrpclib.py/1.0.1 (by www.pythonware.com).
Content-Type: text/xml.
Content-Length: 107.
.

##
T 2009/07/06 18:18:25.496953 192.98.101.10:37656 -> 192.98.101.10:6060 [AP]
<?xml version='1.0'?>
<methodCall>
<methodName>domain.reload</methodName>
<params>
</params>
</methodCall>

##
T 2009/07/06 18:18:25.498713 192.98.101.10:6060 -> 192.98.101.10:37656 [AP]
HTTP/1.0 200 OK.
Via: SIP/2.0/TCP 192.98.101.10:37656.
Server: OpenXg SIP Router (2.99.0-dev01-serk (i386/linux)).
Content-Length: 108.
.
<?xml version="1.0"?>
<methodResponse>
<params>
<param>
<value></value>
</param>
</params>
</methodResponse>
#

one difference that could explain the hangup is that xmlrpc module does
not close the connection like mi_xmlrpc server does.

-- juha

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Andrei Pelinescu-Onciul]

On Jul 06, 2009 at 18:22, Juha Heinanen <[hidden email]> wrote:
[...]
> when i try domain.reload with xmlrpc module, it hangs and ngrep reports:
[...]

> one difference that could explain the hangup is that xmlrpc module does
> not close the connection like mi_xmlrpc server does.

You are right, it's the close. Looks like the default behaviour of
xmlrpclib is totally broken...

Here's how to make it work
(code ripped from ser_ctl : serctl/sercmlrpc.py --
 http://git.sip-router.org/cgi-bin/gitweb.cgi?p=ser;a=blob;f=ser_ctl/serctl/serxmlrpc.py
 )

---------------------------------------------------------------------------
import xmlrpclib, httplib

XMLRPC_SERVER = "127.0.0.1"
XMLRPC_PORT = 5060


class Transport:
    def __init__(self):
        self.conn=httplib.HTTPConnection(XMLRPC_SERVER, str(XMLRPC_PORT));

    def _http_request(self, uripath, body, host):
        self.conn.request("POST", uripath, body, {})

    def request(self, host, uripath, body, verbose=0):
        self._http_request(uripath, body, host)
        response=self.conn.getresponse()
        if response.status != 200:
            raise xmlrpclib.ProtocolError(host+uripath, response.status,
                            response.reason, response.msg)
        data=response.read()
        parser, unmarshaller=xmlrpclib.getparser()
        parser.feed(data)
        parser.close()
        return unmarshaller.close()


transport=Transport()
c=xmlrpclib.ServerProxy("http://" + XMLRPC_SERVER+ ":" + str(XMLRPC_PORT),
                        transport)

res=c.core.pwd()
print res;
---------------------------------------------------------------------------

Note: the code in ser_ctl is much more complex, it supports authentication
 and https, I reduced it to the basics for this example.


We could also close the connection in sr, but that would require additional
code and it's the xmlrpc lib that's broken (parsing the answer only on close()
is really braindamaged). However if there are lots of applications
relying on the broken behaviour, I might be convinced to add the close
option in sr xmlrpc.


Andrei

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Juha Heinanen]

Andrei Pelinescu-Onciul writes:

 > We could also close the connection in sr, but that would require
 > additional code and it's the xmlrpc lib that's broken (parsing the
 > answer only on close()
 > is really braindamaged). However if there are lots of applications
 > relying on the broken behaviour, I might be convinced to add the close
 > option in sr xmlrpc.

well, python and php both are very popular for writing management
software and a basic xmlrpc client written in either does not work with
sr xmlrpc module.  they do work with k mi_xmlrpc module which makes
migration from mi_xmlrpc to xmlrpc module painful.

i would suggest to do whatever is necessary make this simple call work:

server_path = XMLRPC_PROTOCOL + '://' +  XMLRPC_SERVER + \
                ':' +  str(XMLRPC_PORT) + XMLRPC_DIR
c = xmlrpclib.ServerProxy(server_path)

if you look at any examples on the web on python xmlrpc clients, the
above is what you will find.  if sr requires something more complicated,
it is not a good idea.

-- juha

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Andrei Pelinescu-Onciul]

On Jul 06, 2009 at 22:39, Juha Heinanen <[hidden email]> wrote:

> Andrei Pelinescu-Onciul writes:
>
>  > We could also close the connection in sr, but that would require
>  > additional code and it's the xmlrpc lib that's broken (parsing the
>  > answer only on close()
>  > is really braindamaged). However if there are lots of applications
>  > relying on the broken behaviour, I might be convinced to add the close
>  > option in sr xmlrpc.
>
> well, python and php both are very popular for writing management
> software and a basic xmlrpc client written in either does not work with
> sr xmlrpc module.  they do work with k mi_xmlrpc module which makes
> migration from mi_xmlrpc to xmlrpc module painful.
>
> i would suggest to do whatever is necessary make this simple call work:
>
> server_path = XMLRPC_PROTOCOL + '://' +  XMLRPC_SERVER + \
> ':' +  str(XMLRPC_PORT) + XMLRPC_DIR
> c = xmlrpclib.ServerProxy(server_path)
>
> if you look at any examples on the web on python xmlrpc clients, the
> above is what you will find.  if sr requires something more complicated,
> it is not a good idea.

My point is that the library is broken.
The perl xmlrpc:lite  works perfectly:

-------------------------------------------------------
use XMLRPC::Lite;

my $rpc=shift @ARGV;
my $res= XMLRPC::Lite-> proxy("http://127.0.0.1:5060")
         -> call($rpc, @ARGV)->result;

# result printing skipped
-------------------------------------------------------

Moreover with the transport workaround I posted, it works even in python.

So unless the effort of fixing existing python or php applications is not
significantly greater then the effort of adding the silly close
connection mode (which will disallow keeping a connection open for a
long time and piping xmlrpc requests through it), I do not intend to
waste any time on it.
For future implementations we can document the "persistent" transport
python stuff (I assume there is something similar for php too).

So whoever is interested in xmlrpc close connection after reply mode,
should vote for it (but please vote only if you really have "legacy"
hard to change applications that would be affected by it).


Andrei

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Juha Heinanen]

Andrei Pelinescu-Onciul writes:

 > So whoever is interested in xmlrpc close connection after reply mode,
 > should vote for it (but please vote only if you really have "legacy"
 > hard to change applications that would be affected by it).

i would be interested, but perhaps i'm the only one who as xmlrpc client
apps written in python and php.

it is simpler for me to add mi xmlrpc support to domain module that try
to figure out what is wrong with python and php libs.  i can then keep
on using mi_xmlrpc module an forget xmlrpc module.

user should need only one of those two modules, not both.

-- juha

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Juha Heinanen]

i may be wrong, but it is hard for me to believe that python xmlrpclib
would be badly broken, because it is very widely used.

if i look more closely at body of sr response:

##
T 2009/07/06 18:18:25.498713 192.98.101.10:6060 -> 192.98.101.10:37656 [AP]
HTTP/1.0 200 OK.
Via: SIP/2.0/TCP 192.98.101.10:37656.
Server: OpenXg SIP Router (2.99.0-dev01-serk (i386/linux)).
Content-Length: 108.
.
<?xml version="1.0"?>
<methodResponse>
<params>
<param>
<value></value>
</param>
</params>
</methodResponse>
#

i notice that each body line is terminated only by one char (cr or lf)
and the last line is not terminated by any char.  content-length 108
seems to verify this.

i don't know if it is ok that lines end with only one char and that the
last line does not end with anything.

for comparison, mi xmlrpc module terminates each body line (including
the last) with both cr and lf:

#
T 2009/07/06 18:12:34.032315 192.98.101.10:6060 -> 192.98.101.10:41014 [AFP]
Content-length: 147.
Connection: close.
Date: Mon, 06 Jul 2009 16:12:34 GMT.
Server: XMLRPC_ABYSS/1.06.
.
<?xml version="1.0" encoding="UTF-8"?>.
<methodResponse>.
<params>.
<param><value><string></string></value></param>.
</params>.
</methodResponse>.

####

-- juha

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Andrei Pelinescu-Onciul]

On Jul 06, 2009 at 23:41, Juha Heinanen <[hidden email]> wrote:
> i may be wrong, but it is hard for me to believe that python xmlrpclib
> would be badly broken, because it is very widely used.

The parser is not broken, it works. The problem is how it uses the
transport by default. It looks like it waits for the remote close before
it starts parsing the reply, which is wrong.

If one changes the transport (like in the example I've sent or in
ser_ctl), then it works perfectly. One has only to create a new
Transport class and then import it in all his code and instead of:

c=xmlrpclib.ServerProxy("http://" + XMLRPC_SERVER+ ":" +
str(XMLRPC_PORT))

use

transport=Transport()
c=xmlrpclib.ServerProxy("http://" + XMLRPC_SERVER+ ":" +
str(XMLRPC_PORT), transport)

(this will removes the wait for the remote close() and it will work also
 with mi_xmlrpc)

Anyway IMHO for more complex applications the default transport should
be changed anyway if there's any need for some sort of security (and for
that a look at ser_ctl will really pay off: you can authenticate the
HTML requests using the transport class defined there and all you need
to do on ser side is to add a www_authorize/challenge in the xmlrpc
 route).

>
> if i look more closely at body of sr response:
>

[...]

>
> i notice that each body line is terminated only by one char (cr or lf)
> and the last line is not terminated by any char.  content-length 108
> seems to verify this.
>
> i don't know if it is ok that lines end with only one char and that the
> last line does not end with anything.

The separator doesn't matter as long as it's some kind of whitespace.
We can even get rid of all of them (everything on one line) and it
should still work.
>
> for comparison, mi xmlrpc module terminates each body line (including
> the last) with both cr and lf:
>

Try the latest sr version and add at the end of the xmlrpc route
 a return -1 or drop -1 (e.g.: route[XMLRPC]{dispatch_rpc(); drop -1; })
and it should close the connection after sending a reply.
(I've come to the conclusion that is faster to implement it then
answering to emails :-)).
You could even do the return -1 only if the user agent is xmlrpclib.py
(so that non-broken clients would still be able to keep 1 open
connection and send xmlrpc requests on it).


Andrei

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Raúl Alexis Betancor Santana]

On Monday 06 July 2009 21:41:40 Juha Heinanen wrote:
> i may be wrong, but it is hard for me to believe that python xmlrpclib
> would be badly broken, because it is very widely used.

I also doub that xmlrpclib it's the problem

> i don't know if it is ok that lines end with only one char and that the
> last line does not end with anything.

It's not ok, as per HTTP/1.1 RFC, headers lines must end with <CR><LF> then an
empy <CR><LF> line must be used as boundary between headers and body and
finaly HTTP request or response must end with <CR><LF> on its own line.
So taking into account that xmlrpc it's XML over HTTP, we must follow the RFC
on that.

> for comparison, mi xmlrpc module terminates each body line (including
> the last) with both cr and lf:

I have not read HTTP/1.1 RFC line by line .. so I could not tell that it's
true for all body lines, but we must follow the specs about header/body
boundary and all other specs.

--
Raúl Alexis Betancor Santana
Dimensión Virtual

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Raúl Alexis Betancor Santana]

On Monday 06 July 2009 22:13:27 Andrei Pelinescu-Onciul wrote:

> On Jul 06, 2009 at 23:41, Juha Heinanen <[hidden email]> wrote:
> > i may be wrong, but it is hard for me to believe that python xmlrpclib
> > would be badly broken, because it is very widely used.
>
> The parser is not broken, it works. The problem is how it uses the
> transport by default. It looks like it waits for the remote close before
> it starts parsing the reply, which is wrong.
>
> If one changes the transport (like in the example I've sent or in
> ser_ctl), then it works perfectly. One has only to create a new
> Transport class and then import it in all his code and instead of:
>
> c=xmlrpclib.ServerProxy("http://" + XMLRPC_SERVER+ ":" +
> str(XMLRPC_PORT))
>
> use
>
> transport=Transport()
> c=xmlrpclib.ServerProxy("http://" + XMLRPC_SERVER+ ":" +
> str(XMLRPC_PORT), transport)
>
> (this will removes the wait for the remote close() and it will work also
>  with mi_xmlrpc)

Umm, re-checking your example and the responses that Juha pasted ... you are
wrong, I explain myself ...

xmlrpc responses are sended as HTTP/1.0 transport, SO AS PER HTTP/1.0 RFC,
only ONE request/response per conection are allowed so xmlrpclib is waiting
for the remote end to close the conection because it see HTTP/1.0 on the
headers.

Could someone try to check what happens if transport is HTTP/1.1 instead of
HTTP/1.0 ?
I think that will run without having to change "transport" on the python
script.

> Anyway IMHO for more complex applications the default transport should
> be changed anyway if there's any need for some sort of security (and for
> that a look at ser_ctl will really pay off: you can authenticate the
> HTML requests using the transport class defined there and all you need
> to do on ser side is to add a www_authorize/challenge in the xmlrpc
>  route).

I use twisted when developing that kind of apps, so transport and security
it's covered by the framework.

> The separator doesn't matter as long as it's some kind of whitespace.
> We can even get rid of all of them (everything on one line) and it
> should still work.

It matter for the headers/body boundary and for the body/endDoc boundary, you
are right saying that you could send the response in one line if you want,
XML-spec's allows that, BUT not for the HTTP transport

--
Raúl Alexis Betancor Santana
Dimensión Virtual

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Andrei Pelinescu-Onciul]

On Jul 06, 2009 at 22:37, Ra?l Alexis Betancor Santana <[hidden email]> wrote:
> On Monday 06 July 2009 21:41:40 Juha Heinanen wrote:
> > i may be wrong, but it is hard for me to believe that python xmlrpclib
> > would be badly broken, because it is very widely used.
>
> I also doub that xmlrpclib it's the problem

Being standard HTTP, the xmlrpc client should close the connection after
receiving the answer (if it doesn't want a connection persistent mode)
and not the reverse.

>
> > i don't know if it is ok that lines end with only one char and that the
> > last line does not end with anything.
>
> It's not ok, as per HTTP/1.1 RFC, headers lines must end with <CR><LF> then an
> empy <CR><LF> line must be used as boundary between headers and body and
> finaly HTTP request or response must end with <CR><LF> on its own line.
> So taking into account that xmlrpc it's XML over HTTP, we must follow the RFC
> on that.

Juha meant only the xml body. All the HTTP header lines are CRLF
terminated.

>
> > for comparison, mi xmlrpc module terminates each body line (including
> > the last) with both cr and lf:
>
> I have not read HTTP/1.1 RFC line by line .. so I could not tell that it's
> true for all body lines, but we must follow the specs about header/body
> boundary and all other specs.

It's true only for the headers. How the body looks depends on what does
it carry. For the case of xmlrpc it's xml which is quite free-form (you
 can have the whole xml part on one line if you want to).

Andrei

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Raúl Alexis Betancor Santana]

On Monday 06 July 2009 22:50:38 Andrei Pelinescu-Onciul wrote:
> On Jul 06, 2009 at 22:37, Ra?l Alexis Betancor Santana
<[hidden email]> wrote:
> > On Monday 06 July 2009 21:41:40 Juha Heinanen wrote:
> > > i may be wrong, but it is hard for me to believe that python xmlrpclib
> > > would be badly broken, because it is very widely used.
> >
> > I also doub that xmlrpclib it's the problem
>
> Being standard HTTP, the xmlrpc client should close the connection after
> receiving the answer (if it doesn't want a connection persistent mode)
> and not the reverse.

But server is sending a response not ended by CR+LF, so client doesn't know
that it must close the connection and also as serving HTTP/1.0 resquest
server MUST close the connection when it send its response.

What I see is that server is sending out HTTP/1.0 responses so you could NEVER
have a client that could wait for persistent connections because if it does,
it will be breaking HTTP/1.0 RFC

> Juha meant only the xml body. All the HTTP header lines are CRLF
> terminated.

And boundary between headers and body ?

> It's true only for the headers. How the body looks depends on what does
> it carry. For the case of xmlrpc it's xml which is quite free-form (you
>  can have the whole xml part on one line if you want to).

Yes, but I was not telling that body must follow cr+lf spec .. I told that
response should be:

HTTP/1.0
header: value<cr><lf>
<cr><lf>
{xml response(no matter if one line or more)}<cr><lf>
<cr><lf>

It should be easy to check, I don't use xmlrpc on s-r so I could not check it
by now

--
Raúl Alexis Betancor Santana
Dimensión Virtual

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Andrei Pelinescu-Onciul]

On Jul 06, 2009 at 23:16, Ra?l Alexis Betancor Santana <[hidden email]> wrote:

> On Monday 06 July 2009 22:50:38 Andrei Pelinescu-Onciul wrote:
> > On Jul 06, 2009 at 22:37, Ra?l Alexis Betancor Santana
> <[hidden email]> wrote:
> > > On Monday 06 July 2009 21:41:40 Juha Heinanen wrote:
> > > > i may be wrong, but it is hard for me to believe that python xmlrpclib
> > > > would be badly broken, because it is very widely used.
> > >
> > > I also doub that xmlrpclib it's the problem
> >
> > Being standard HTTP, the xmlrpc client should close the connection after
> > receiving the answer (if it doesn't want a connection persistent mode)
> > and not the reverse.
>
> But server is sending a response not ended by CR+LF, so client doesn't know
> that it must close the connection and also as serving HTTP/1.0 resquest
> server MUST close the connection when it send its response.

AFAIK the request does not have to end in anything specific. The
headers end with a double CRLF and then there are Content-Lenght bytes
of the body.

>
> What I see is that server is sending out HTTP/1.0 responses so you could NEVER
> have a client that could wait for persistent connections because if it does,
> it will be breaking HTTP/1.0 RFC

The server server sends exactly the same HTTP version in the reponse
that it has received in the request (if the request  is HTTP/1.1 the
reply is HTTP/1.1).
>
> > Juha meant only the xml body. All the HTTP header lines are CRLF
> > terminated.
>
> And boundary between headers and body ?

It's ok (or else the body would not be at all parse-able and no xmlrpc
client would work).

>
> > It's true only for the headers. How the body looks depends on what does
> > it carry. For the case of xmlrpc it's xml which is quite free-form (you
> >  can have the whole xml part on one line if you want to).
>
> Yes, but I was not telling that body must follow cr+lf spec .. I told that
> response should be:
>
> HTTP/1.0
> header: value<cr><lf>
> <cr><lf>
> {xml response(no matter if one line or more)}<cr><lf>
> <cr><lf>
>
> It should be easy to check, I don't use xmlrpc on s-r so I could not check it

It is. The problem is the python xmlrpc client waits for the server to
close the connection. It might be related to the fact that it uses
http/1.0 by default (but I don't know how can that be easily changed).
However the perl client works, the python client works with a transport
class and the latest sr version has support for closing the connection
 after sending a reply if one really wants to and prefer this rather
 then updating his python/php program.

Andrei

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Jan Janak]

On 06-07 23:41, Juha Heinanen wrote:
> i may be wrong, but it is hard for me to believe that python xmlrpclib
> would be badly broken, because it is very widely used.

I do not recall all the details, but I remember that I encountered the same
problem long time ago. I even reported it to the maintainer of the xmlrpc
library in python. He confirmed that this was a "shortcomming" in the library,
but refused to fix it because "all other xmlrpc servers close the connection".

In ser_ctl (the tools written in python) we patched the library, as far as I
can remember.

  Jan.

>
> if i look more closely at body of sr response:
>
> ##
> T 2009/07/06 18:18:25.498713 192.98.101.10:6060 -> 192.98.101.10:37656 [AP]
> HTTP/1.0 200 OK.
> Via: SIP/2.0/TCP 192.98.101.10:37656.
> Server: OpenXg SIP Router (2.99.0-dev01-serk (i386/linux)).
> Content-Length: 108.
> .
> <?xml version="1.0"?>
> <methodResponse>
> <params>
> <param>
> <value></value>
> </param>
> </params>
> </methodResponse>
> #
>
> i notice that each body line is terminated only by one char (cr or lf)
> and the last line is not terminated by any char.  content-length 108
> seems to verify this.
>
> i don't know if it is ok that lines end with only one char and that the
> last line does not end with anything.
>
> for comparison, mi xmlrpc module terminates each body line (including
> the last) with both cr and lf:
>
> #
> T 2009/07/06 18:12:34.032315 192.98.101.10:6060 -> 192.98.101.10:41014 [AFP]
> Content-length: 147.
> Connection: close.
> Date: Mon, 06 Jul 2009 16:12:34 GMT.
> Server: XMLRPC_ABYSS/1.06.
> .
> <?xml version="1.0" encoding="UTF-8"?>.
> <methodResponse>.
> <params>.
> <param><value><string></string></value></param>.
> </params>.
> </methodResponse>.
>
> ####
>
> -- juha
>
> _______________________________________________
> sr-dev mailing list
> [hidden email]
> http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Jan Janak]

On 06-07 22:46, Raúl Alexis Betancor Santana wrote:

> On Monday 06 July 2009 22:13:27 Andrei Pelinescu-Onciul wrote:
> > On Jul 06, 2009 at 23:41, Juha Heinanen <[hidden email]> wrote:
> > > i may be wrong, but it is hard for me to believe that python xmlrpclib
> > > would be badly broken, because it is very widely used.
> >
> > The parser is not broken, it works. The problem is how it uses the
> > transport by default. It looks like it waits for the remote close before
> > it starts parsing the reply, which is wrong.
> >
> > If one changes the transport (like in the example I've sent or in
> > ser_ctl), then it works perfectly. One has only to create a new
> > Transport class and then import it in all his code and instead of:
> >
> > c=xmlrpclib.ServerProxy("http://" + XMLRPC_SERVER+ ":" +
> > str(XMLRPC_PORT))
> >
> > use
> >
> > transport=Transport()
> > c=xmlrpclib.ServerProxy("http://" + XMLRPC_SERVER+ ":" +
> > str(XMLRPC_PORT), transport)
> >
> > (this will removes the wait for the remote close() and it will work also
> >  with mi_xmlrpc)
>
> Umm, re-checking your example and the responses that Juha pasted ... you are
> wrong, I explain myself ...
>
> xmlrpc responses are sended as HTTP/1.0 transport, SO AS PER HTTP/1.0 RFC,
> only ONE request/response per conection are allowed so xmlrpclib is waiting
> for the remote end to close the conection because it see HTTP/1.0 on the
> headers.

I tend to disagree. The RFC you refer to says that if the response carries
Content-Length header field then the value of the header field determines the
end of the reply. If there is no Content-Length header field in the reply then
the reply is complete when the server closes the connection. Even for HTTP/1.0
requests and responses.

This really is a bug in the xmlrpc client library, because it ignores the
Content-Length header field. Unfortunately it is not triggered by most other
servers because they close the connection immediately.

> Could someone try to check what happens if transport is HTTP/1.1 instead of
> HTTP/1.0 ?

Nothing, in fact the xmlrpc module returns in reply whatever the client sent
in the request.

   Jan.

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev

[Jan Janak]

On 06-07 23:16, Raúl Alexis Betancor Santana wrote:

> On Monday 06 July 2009 22:50:38 Andrei Pelinescu-Onciul wrote:
> > On Jul 06, 2009 at 22:37, Ra?l Alexis Betancor Santana
> <[hidden email]> wrote:
> > > On Monday 06 July 2009 21:41:40 Juha Heinanen wrote:
> > > > i may be wrong, but it is hard for me to believe that python xmlrpclib
> > > > would be badly broken, because it is very widely used.
> > >
> > > I also doub that xmlrpclib it's the problem
> >
> > Being standard HTTP, the xmlrpc client should close the connection after
> > receiving the answer (if it doesn't want a connection persistent mode)
> > and not the reverse.
>
> But server is sending a response not ended by CR+LF, so client doesn't know
> that it must close the connection and also as serving HTTP/1.0 resquest
> server MUST close the connection when it send its response.

Where in the RFC do you see the MUST? The closest thing that I can find in the
HTTP/1.0 RFC is that "except for experimental setups, current practise
requires that the server closes the connection after sending the response".

And further in the text, it is said that if Content-Length is present in the
reply, the header field is used to determine the end of the reply, not the end
of the connection.

> What I see is that server is sending out HTTP/1.0 responses so you could NEVER
> have a client that could wait for persistent connections because if it does,
> it will be breaking HTTP/1.0 RFC

Not true.

   Jan.

_______________________________________________
sr-dev mailing list
[hidden email]
http://lists.sip-router.org/cgi-bin/mailman/listinfo/sr-dev