Hello World! and Welcome to jersey-multipart
|
|
| 1 2 3 |
Craig McClanahan
|
For those who don't know me, I have been around the Java web tier for
quite a while, being the original author of the Struts framework (<http://struts.apache.org>), as well as co-spec-lead for JavaServer Faces 1.0 and 1.1. My more recent interests have focused on RESTful web services, which led me naturally towards JAX-RS and the Jersey implementation. I've been one of the folks inside Sun who has been leveraging Jersey for some internal projects over the last few months. We had a particular need to support MIME multipart/* media types, and it made sense to generalize this into a reusable module -- hence, I've just uploaded the "jersey-multipart" module to the "contribs" directory. It relies on 1.0 or later Jersey code, and provides what I hope are found to be elegant solutions to the problems of multipart handling, while leveraging all the nice JAX-RS providers support for dealing with the entity content of body parts, just like we've grown spoiled by on complete message bodies. And, it works both on the server side and the client side, when you use jersey-client. Example server code to build a multipart response might look like this: // I have also provided an appropriate MessageBodyWriter for the MyBean class MyBean bean = ...; return Response.ok(new MultiPart(). type(new MediaType("multipart", "mixed"). bodyPart("This is the first body part in plain text", new MediaType("text", "plain")). bodyPart(bean, new MediaType("x-application", "x-format"))).build(); (Of course, you can do things in a more fine-grained fashion, but the builder pattern utilized all over the JAX-RS and Jersey APIs was so cool that Paul suggested I use it here too, so I did :-). To read a MultiPart entity (produced by code like the previous example) that was uploaded to the server you might do something like this: // I have also provided an appropriate MessageBodyReader for the MyBean class @Path("...") @PUT @Consumes("multipart/mixed") @Produces(...) public Response handler(MultiPart multiPart) { BodyPart part0 = multiPart.getBodyParts().get(0); String text = part0.getEntityAs(String.class); BodyPart part1 = multiPart.getBodyParts().get(1); MyBean bean = part1.getEntityAs(MyBean.class); ... multiPart.cleanup(); } The need for cleanup() is because the implementation knows how to buffer "large" body parts to temporary files on disk, so you don't blow away your JVM heap on a multi-gigabyte upload or download. I'm looking for a way to avoid the need for the application to call this, but haven't found one yet -- in the mean time, everything else about dealing with multipart files has seemed pretty easy to deal with. As mentioned above, this module works on the client side as well, if you're using jersey-client. The unit tests have some more worked-out examples of the lower level details. Give it a try and see what you think! And, for sure, if you see anything that needs to be improved, please ask here and/or file an issue in the issue tracking system. Craig McClanahan PS: Among my other interests will be working with the Atom Publishing Protocol support, again with the idea of leveraging JAX-RS providers to do format translations for custom <content> payloads. --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
|||||||||||||||||||
Paul Sandoz
|
Hi Craig,
Welcome to the list of Jersey committers :-) Thanks very much for the multipart MIME contribution, it looks really good. I would like to leverage this to improve the support for multipart/form-data and also consider support for Multipart Java Beans. Classes supporting AtomPub support would be great! Re: the cleanup aspect. I think we need to review Jersey's IoC support and integration capabilities with the likes of Spring/Guice. It is really hard to be abstract from an IoC framework :-( If the returned MultiPart was instantiated in the life-cycle of the request then when the response is about to the sent a pre destroy method could be called to do the clean up. That would not work for the conditions of the client working standalone (perhaps cleanup could also be called before being GC'ed although that cannot be relied upon). Paul. On Oct 17, 2008, at 1:00 AM, Craig McClanahan wrote: > For those who don't know me, I have been around the Java web tier > for quite a while, being the original author of the Struts framework > (<http://struts.apache.org>), as well as co-spec-lead for JavaServer > Faces 1.0 and 1.1. My more recent interests have focused on RESTful > web services, which led me naturally towards JAX-RS and the Jersey > implementation. > > I've been one of the folks inside Sun who has been leveraging Jersey > for some internal projects over the last few months. We had a > particular need to support MIME multipart/* media types, and it made > sense to generalize this into a reusable module -- hence, I've just > uploaded the "jersey-multipart" module to the "contribs" directory. > It relies on 1.0 or later Jersey code, and provides what I hope are > found to be elegant solutions to the problems of multipart handling, > while leveraging all the nice JAX-RS providers support for dealing > with the entity content of body parts, just like we've grown spoiled > by on complete message bodies. And, it works both on the server > side and the client side, when you use jersey-client. > > Example server code to build a multipart response might look like > this: > > // I have also provided an appropriate MessageBodyWriter for the > MyBean class > MyBean bean = ...; > return Response.ok(new MultiPart(). > type(new MediaType("multipart", "mixed"). > bodyPart("This is the first body part in plain text", new > MediaType("text", "plain")). > bodyPart(bean, new MediaType("x-application", "x- > format"))).build(); > > (Of course, you can do things in a more fine-grained fashion, but > the builder pattern utilized all over the JAX-RS and Jersey APIs was > so cool that Paul suggested I use it here too, so I did :-). > > To read a MultiPart entity (produced by code like the previous > example) that was uploaded to the server you might do something like > this: > > // I have also provided an appropriate MessageBodyReader for the > MyBean class > @Path("...") > @PUT > @Consumes("multipart/mixed") > @Produces(...) > public Response handler(MultiPart multiPart) { > BodyPart part0 = multiPart.getBodyParts().get(0); > String text = part0.getEntityAs(String.class); > BodyPart part1 = multiPart.getBodyParts().get(1); > MyBean bean = part1.getEntityAs(MyBean.class); > ... > multiPart.cleanup(); > } > > The need for cleanup() is because the implementation knows how to > buffer "large" body parts to temporary files on disk, so you don't > blow away your JVM heap on a multi-gigabyte upload or download. I'm > looking for a way to avoid the need for the application to call > this, but haven't found one yet -- in the mean time, everything else > about dealing with multipart files has seemed pretty easy to deal > with. > > As mentioned above, this module works on the client side as well, if > you're using jersey-client. The unit tests have some more worked- > out examples of the lower level details. > > Give it a try and see what you think! And, for sure, if you see > anything that needs to be improved, please ask here and/or file an > issue in the issue tracking system. > > Craig McClanahan > > PS: Among my other interests will be working with the Atom > Publishing Protocol support, again with the idea of leveraging JAX- > RS providers to do format translations for custom <content> payloads. > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [hidden email] > For additional commands, e-mail: [hidden email] > --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
||||
Jeff Schmidt
|
Some javascript/style in this post has been disabled (why?)
Hi Craig:Thanks for your work on jersey-multipart. I have some SOAP based endpoints where a request can consist of various XML meta data, and then there is an element of type base64Binary where an arbitrary file can be 'attached' to the request. I would like to do something equivalent in a RESTful manner. It seems like a request comprised of an XML entity and a binary entity would be the way to go. The JAX-RS resource does not need to understand the binary entity, just make it available to some server-side code. Looking at some of the jersey-multipart code, it looks like I can use BodyPart.getEntityAs(MyMetaData.class) to access the XML meta data, and BodyPart.getEntity().getInputStream() to access the binary data. When I'm done, MultiPart.cleanup() will delete the temporary binary file (if > threshold bytes). Am I reading that right? Also, I don't know if doing this on the server side will make it difficult for clients. You said the jersey-client can easily work with jersey-multipart. I've not yet looked into the Commons HttpClient or other client libraries to see if they have issues with making multipart requests. Thanks, Jeff On Oct 17, 2008, at 12:58 AM, Paul Sandoz wrote:
-- Jeff Schmidt |
|||||||||||||||||||
|
Craig McClanahan
|
Some javascript/style in this post has been disabled (why?)
Jeff Schmidt wrote:
Hi Craig:Yep, that is exactly the sort of thing I use it for. That's exactly right. You can also look at the Jersey client code itself. One of the really cool things about jersey-client is you can use the same providers (including the jersey-multipart stuff) on the client just like you can use it on the server. There's some small samples of this in the jersey-multipart unit tests. Craig
|
|||||||||||||||||||
|
Jeff Schmidt
|
Some javascript/style in this post has been disabled (why?)
Thanks Craig! This all sounds good.I guess my client side question was more generic. Potential clients may be using Java, Python, Ruby etc. and whatever related libraries. I don't know if issuing multi-part requests is unusual or difficult in general for such clients. I don't want to put up anymore barriers to entry to use the API then I have to. :) What has been your experience in this regard? Any complaints from clients? Cheers, Jeff On Oct 23, 2008, at 1:10 PM, Craig McClanahan wrote:
-- Jeff Schmidt |
|||||||||||||||||||
|
Craig McClanahan
|
Jeff Schmidt wrote:
> Thanks Craig! This all sounds good. > > I guess my client side question was more generic. Potential clients > may be using Java, Python, Ruby etc. and whatever related libraries. I > don't know if issuing multi-part requests is unusual or difficult in > general for such clients. I don't want to put up anymore barriers to > entry to use the API then I have to. :) Agreed -- this is one of the nicest things about REST. > What has been your experience in this regard? Any complaints from > clients? > Constructing a multipart request is actually very easy, with pretty much any client language and HTTP framework. From the point of view of such a framework, it's a single request of type "multipart/mixed" (or whatever), and the interesting part of the problem is constructing the message body. For that, it's mostly a matter of alternating between a bunch of headers and content for each body part, with a little extra glue for the "boundary" string. In jersey-client, the code that does this is in MultiPartWriter.java, and as you can see it is pretty straightforward (assuming your library already has a way to format the request body for your data entity -- which is the same code you'd need to send that body part by itself). Parsing a multipart response, on the other hand, is somewhat more complicated. In the case of jersey-multipart I punted :-) and used JavaMail APIs, because systems that understand mail streams already have to know how to do this. If you use a different Java HTTP library you can still use JavaMail for this part; in other languages, I'd start by looking for parsing libraries that already exist, and then just glue them in to the client. One of my upcoming tasks will be to create a Ruby client library for some of the services I'm working on, and that will ultimately include some multipart processing -- so I'll soon get to practice what I preach above :-). Craig > Cheers, > > Jeff > > On Oct 23, 2008, at 1:10 PM, Craig McClanahan wrote: > >> Jeff Schmidt wrote: >>> Hi Craig: >>> >>> Thanks for your work on jersey-multipart. I have some SOAP based >>> endpoints where a request can consist of various XML meta data, and >>> then there is an element of type base64Binary where an arbitrary >>> file can be 'attached' to the request. I would like to do something >>> equivalent in a RESTful manner. It seems like a request comprised of >>> an XML entity and a binary entity would be the way to go. The JAX-RS >>> resource does not need to understand the binary entity, just make it >>> available to some server-side code. >> Yep, that is exactly the sort of thing I use it for. >>> >>> Looking at some of the jersey-multipart code, it looks like I can >>> use BodyPart.getEntityAs(MyMetaData.class) to access the XML meta >>> data, and BodyPart.getEntity().getInputStream() to access the binary >>> data. When I'm done, MultiPart.cleanup() will delete the temporary >>> binary file (if > threshold bytes). Am I reading that right? >>> >> That's exactly right. >>> Also, I don't know if doing this on the server side will make it >>> difficult for clients. You said the jersey-client can easily work >>> with jersey-multipart. I've not yet looked into the Commons >>> HttpClient or other client libraries to see if they have issues with >>> making multipart requests. >>> >> You can also look at the Jersey client code itself. One of the >> really cool things about jersey-client is you can use the same >> providers (including the jersey-multipart stuff) on the client just >> like you can use it on the server. There's some small samples of >> this in the jersey-multipart unit tests. >> >>> Thanks, >>> >>> Jeff >>> >> Craig >> >>> On Oct 17, 2008, at 12:58 AM, Paul Sandoz wrote: >>> >>>> Hi Craig, >>>> >>>> Welcome to the list of Jersey committers :-) >>>> >>>> Thanks very much for the multipart MIME contribution, it looks >>>> really good. I would like to leverage this to improve the support >>>> for multipart/form-data and also consider support for Multipart >>>> Java Beans. >>>> >>>> Classes supporting AtomPub support would be great! >>>> >>>> Re: the cleanup aspect. I think we need to review Jersey's IoC >>>> support and integration capabilities with the likes of >>>> Spring/Guice. It is really hard to be abstract from an IoC >>>> framework :-( If the returned MultiPart was instantiated in the >>>> life-cycle of the request then when the response is about to the >>>> sent a pre destroy method could be called to do the clean up. That >>>> would not work for the conditions of the client working standalone >>>> (perhaps cleanup could also be called before being GC'ed although >>>> that cannot be relied upon). >>>> >>>> Paul. >>>> >>>> >>>> On Oct 17, 2008, at 1:00 AM, Craig McClanahan wrote: >>>> >>>>> For those who don't know me, I have been around the Java web tier >>>>> for quite a while, being the original author of the Struts >>>>> framework (<http://struts.apache.org>), as well as co-spec-lead >>>>> for JavaServer Faces 1.0 and 1.1. My more recent interests have >>>>> focused on RESTful web services, which led me naturally towards >>>>> JAX-RS and the Jersey implementation. >>>>> >>>>> I've been one of the folks inside Sun who has been leveraging >>>>> Jersey for some internal projects over the last few months. We >>>>> had a particular need to support MIME multipart/* media types, and >>>>> it made sense to generalize this into a reusable module -- hence, >>>>> I've just uploaded the "jersey-multipart" module to the "contribs" >>>>> directory. It relies on 1.0 or later Jersey code, and provides >>>>> what I hope are found to be elegant solutions to the problems of >>>>> multipart handling, while leveraging all the nice JAX-RS providers >>>>> support for dealing with the entity content of body parts, just >>>>> like we've grown spoiled by on complete message bodies. And, it >>>>> works both on the server side and the client side, when you use >>>>> jersey-client. >>>>> >>>>> Example server code to build a multipart response might look like >>>>> this: >>>>> >>>>> // I have also provided an appropriate MessageBodyWriter for the >>>>> MyBean class >>>>> MyBean bean = ...; >>>>> return Response.ok(new MultiPart(). >>>>> type(new MediaType("multipart", "mixed"). >>>>> bodyPart("This is the first body part in plain text", new >>>>> MediaType("text", "plain")). >>>>> bodyPart(bean, new MediaType("x-application", >>>>> "x-format"))).build(); >>>>> >>>>> (Of course, you can do things in a more fine-grained fashion, but >>>>> the builder pattern utilized all over the JAX-RS and Jersey APIs >>>>> was so cool that Paul suggested I use it here too, so I did :-). >>>>> >>>>> To read a MultiPart entity (produced by code like the previous >>>>> example) that was uploaded to the server you might do something >>>>> like this: >>>>> >>>>> // I have also provided an appropriate MessageBodyReader for the >>>>> MyBean class >>>>> @Path("...") >>>>> @PUT >>>>> @Consumes("multipart/mixed") >>>>> @Produces(...) >>>>> public Response handler(MultiPart multiPart) { >>>>> BodyPart part0 = multiPart.getBodyParts().get(0); >>>>> String text = part0.getEntityAs(String.class); >>>>> BodyPart part1 = multiPart.getBodyParts().get(1); >>>>> MyBean bean = part1.getEntityAs(MyBean.class); >>>>> ... >>>>> multiPart.cleanup(); >>>>> } >>>>> >>>>> The need for cleanup() is because the implementation knows how to >>>>> buffer "large" body parts to temporary files on disk, so you don't >>>>> blow away your JVM heap on a multi-gigabyte upload or download. >>>>> I'm looking for a way to avoid the need for the application to >>>>> call this, but haven't found one yet -- in the mean time, >>>>> everything else about dealing with multipart files has seemed >>>>> pretty easy to deal with. >>>>> >>>>> As mentioned above, this module works on the client side as well, >>>>> if you're using jersey-client. The unit tests have some more >>>>> worked-out examples of the lower level details. >>>>> >>>>> Give it a try and see what you think! And, for sure, if you see >>>>> anything that needs to be improved, please ask here and/or file an >>>>> issue in the issue tracking system. >>>>> >>>>> Craig McClanahan >>>>> >>>>> PS: Among my other interests will be working with the Atom >>>>> Publishing Protocol support, again with the idea of leveraging >>>>> JAX-RS providers to do format translations for custom <content> >>>>> payloads. >>>>> >>>>> >>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: [hidden email] >>>>> <mailto:[hidden email]> >>>>> For additional commands, e-mail: [hidden email] >>>>> <mailto:[hidden email]> >>>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [hidden email] >>>> <mailto:[hidden email]> >>>> For additional commands, e-mail: [hidden email] >>>> <mailto:[hidden email]> >>>> >>> >>> >>> >>> -- >>> Jeff Schmidt >>> >>> > -- > > Jeff Schmidt > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
|||||||||||||||||||
|
Jeff Schmidt
|
Some javascript/style in this post has been disabled (why?)
Thanks for your thoughts on that Craig. I feel better about going this route. :)Cheers, Jeff
On Oct 23, 2008, at 5:51 PM, Craig McClanahan wrote:
-- Jeff Schmidt |
|||||||||||||||||||
Gili
|
In reply to this post
by Craig McClanahan
Hi Craig,
On the topic of jersey-multipart: 1) Is it really Jersey-specific or can it be used with any JAX-RS implementation? 2) I'm curious whether you looked into using Apache Commons FileUpload instead of Javamail under the hood. FileUpload's jar is only 50k and deals exclusively with parsing this kind of data, versus Javamail which is an entire email client. It might also be more flexible, better integrated than JavaMail. Gili
|
||||||||||||||||||
Robertson, Jeff
|
In reply to this post
by Craig McClanahan
Some javascript/style in this post has been disabled (why?)
Isn't javamail required to be provided by a JEE server? That could be a reason to use it instead of something you'd always have to bundle with the app. |
|||||||||||||||||||
|
Robertson, Jeff
|
In reply to this post
by Craig McClanahan
Some javascript/style in this post has been disabled (why?)
Also I'm sure Craig is familiar with commons fileupload since it is used in struts and (iirc) was a spinoff from struts in the first place. |
|||||||||||||||||||
|
Craig McClanahan
|
In reply to this post
by Gili
Gili wrote:
> Hi Craig, > > On the topic of jersey-multipart: > > 1) Is it really Jersey-specific or can it be used with any JAX-RS > implementation? > Currently, the only Jersey-specific dependencies are in the unit tests. In principle the runtime should only require the JAX-RS APIs, JavaMail, and JAF. But I haven't tested it with anything else. > 2) I'm curious whether you looked into using Apache Commons FileUpload > instead of Javamail under the hood. FileUpload's jar is only 50k and deals > exclusively with parsing this kind of data, versus Javamail which is an > entire email client. It might also be more flexible, better integrated than > JavaMail. > > I hadn't looked at Commons FileUpload, based on the (perhaps mistaken?) assumption that it only supported "multipart/form-data". I'll definitely go take a look. On the other hand, basically all the server side apps I'm working on are running on Glassfish anyway, so JavaMail comes integrated "for free" :-). > Gili > > Craig > Craig McClanahan wrote: > >> For those who don't know me, I have been around the Java web tier for >> quite a while, being the original author of the Struts framework >> (<http://struts.apache.org>), as well as co-spec-lead for JavaServer >> Faces 1.0 and 1.1. My more recent interests have focused on RESTful web >> services, which led me naturally towards JAX-RS and the Jersey >> implementation. >> >> I've been one of the folks inside Sun who has been leveraging Jersey for >> some internal projects over the last few months. We had a particular >> need to support MIME multipart/* media types, and it made sense to >> generalize this into a reusable module -- hence, I've just uploaded the >> "jersey-multipart" module to the "contribs" directory. It relies on 1.0 >> or later Jersey code, and provides what I hope are found to be elegant >> solutions to the problems of multipart handling, while leveraging all >> the nice JAX-RS providers support for dealing with the entity content of >> body parts, just like we've grown spoiled by on complete message >> bodies. And, it works both on the server side and the client side, when >> you use jersey-client. >> >> Example server code to build a multipart response might look like this: >> >> // I have also provided an appropriate MessageBodyWriter for the >> MyBean class >> MyBean bean = ...; >> return Response.ok(new MultiPart(). >> type(new MediaType("multipart", "mixed"). >> bodyPart("This is the first body part in plain text", new >> MediaType("text", "plain")). >> bodyPart(bean, new MediaType("x-application", "x-format"))).build(); >> >> (Of course, you can do things in a more fine-grained fashion, but the >> builder pattern utilized all over the JAX-RS and Jersey APIs was so cool >> that Paul suggested I use it here too, so I did :-). >> >> To read a MultiPart entity (produced by code like the previous example) >> that was uploaded to the server you might do something like this: >> >> // I have also provided an appropriate MessageBodyReader for the >> MyBean class >> @Path("...") >> @PUT >> @Consumes("multipart/mixed") >> @Produces(...) >> public Response handler(MultiPart multiPart) { >> BodyPart part0 = multiPart.getBodyParts().get(0); >> String text = part0.getEntityAs(String.class); >> BodyPart part1 = multiPart.getBodyParts().get(1); >> MyBean bean = part1.getEntityAs(MyBean.class); >> ... >> multiPart.cleanup(); >> } >> >> The need for cleanup() is because the implementation knows how to buffer >> "large" body parts to temporary files on disk, so you don't blow away >> your JVM heap on a multi-gigabyte upload or download. I'm looking for a >> way to avoid the need for the application to call this, but haven't >> found one yet -- in the mean time, everything else about dealing with >> multipart files has seemed pretty easy to deal with. >> >> As mentioned above, this module works on the client side as well, if >> you're using jersey-client. The unit tests have some more worked-out >> examples of the lower level details. >> >> Give it a try and see what you think! And, for sure, if you see >> anything that needs to be improved, please ask here and/or file an issue >> in the issue tracking system. >> >> Craig McClanahan >> >> PS: Among my other interests will be working with the Atom Publishing >> Protocol support, again with the idea of leveraging JAX-RS providers to >> do format translations for custom <content> payloads. >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [hidden email] >> For additional commands, e-mail: [hidden email] >> >> >> >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
|||||||||||||||||||
|
Gili
|
I took a look at Jersey's built-in integration for MimeMultipart (from Javamail) and your implementation of jersey-multipart. I found the former API to be messy, out-of-date and return incorrect values (for example, BodyPart.getFilename() returns null even if Content-Disposition contains a filename). Your API was better but:
1) Given: Content-Disposition: form-data; name="files" you'd expect an API call that returns a Map with two rows: [Content-Disposition, form-data] [name, files] Unfortunately your API returns: Content-Disposition = [form-data; name="files"] and expects me to parse it myself. Any chance you'll improve upon this? 2) I took a quick glance at the implementations and I see some problematic //FIXME comments, such as not being able to configure how big an attachment may get before being dumped to disk, requiring explict calls to cleanup(), etc. Gili
|
||||||||||||||||||
|
Gili
|
Craig,
Here is what I am proposing: Let "example header" refer to "Content-Type: application/xml; charset=UTF-8" 1a) BodyPart.getHeaders should return MultivaluedMap<String, MatrixParameters> instead of MultivaluedMap<String, String>. Where MatrixParameters extends Map<String, String>. The example header would return MatrixParameters = {[Content-Type,application/xml], [charset, UTF-8]} *or* 1b) BodyPart.getHeaders should return MultivaluedMap<String, HeaderValue> instead of MultivaluedMap<String, String>. Where HeaderValue defines two methods: String HeaderValue.getValue() and Map<String, String> HeaderValue.getAttributes() The example header would return: HeaderValue.getValue() = "application/xml" HeaderValue.getAttributes() = "[charset, UTF-8]" Let me know if you can think of a cleaner way to represent this... 2) Here is the algorithm I used to parse the matrix parameters: /** * Converts matrix parameters to key-value pairs. * * @param key the header name * @param value the header value * @return the matrix parameters */ private Map<String, String> getMatrixParameters(String key, String value) { Map<String, String> result = new HashMap<String, String>(); String[] pairs = value.split(";"); if (pairs.length > 0) result.put(key, unquoted(pairs[0])); for (int i = 1, size = pairs.length; i < size; ++i) { String[] tokens = pairs[i].split("="); assert (tokens.length == 2): Arrays.toString(tokens); result.put(tokens[0], unquoted(tokens[1])); } return result; } /** * Removes any quotes surrounding the input text. * * @param text input text * @return unquoted text */ private String unquoted(String text) { text = text.trim(); if (text.charAt(0) == '\"' && text.charAt(text.length() - 1) == '\"') return text.substring(1, text.length() - 1); return text; } 3) Modify the package name so it's obvious that this implementation isn't specific to Jersey (I think it's okay that the tests are Jersey-specific) and publish it on JAX-RS's mailing list in the hopes of getting other people to use and support it. It looks to me like pretty much all major JAX-Rs implementations have built-in MimeMultipart (from javax.mail) providers but this API is not as easy to use as it should be. The value proposition of your provider over those implementations would be a cleaner, more modern API. What do you think? Gili |
||||||||||||||||||
|
Gili
|
The more I look at Commons FileUpload, the more I think it does everything we need. They provide a better parser than my own: http://commons.apache.org/fileupload/apidocs/org/apache/commons/fileupload/ParameterParser.html
Clearly my code is mishandling quotes. Also I noticed that they mention multipart/mixed here: http://commons.apache.org/fileupload/apidocs/org/apache/commons/fileupload/portlet/PortletFileUpload.html though it's not clear if other components support multipart/mixed properly. I'm not a fan of their API (and their documentation is full of typos) but it looks like it does the job from an implementation point of view. Perhaps we can throw it underneath your API. Gili
|
||||||||||||||||||
|
Craig McClanahan
|
In reply to this post
by Gili
Gili wrote:
> I took a look at Jersey's built-in integration for MimeMultipart (from > Javamail) and your implementation of jersey-multipart. I found the former > API to be messy, out-of-date and return incorrect values (for example, > BodyPart.getFilename() returns null even if Content-Disposition contains a > filename). Your API was better but: > > 1) Given: > > Content-Disposition: form-data; name="files" > > you'd expect an API call that returns a Map with two rows: > [Content-Disposition, form-data] > [name, files] > > Unfortunately your API returns: Content-Disposition = [form-data; > name="files"] and expects me to parse it myself. Any chance you'll improve > upon this? > > HeaderValue) that gives you an accessor to get to the value itself ("form-data" in this case), plus a Map<String,String> giving you access to all the parameters (just one in the case shown here, but there could be several for general header values)? We'd still want convenient string-only ways to *construct* the header value, but it would be nice to have something like this on BodyPart: MultivaluedMap<String,HeaderType> getHeaders() { ... } instead of the current return value of "MultivaluedMap<String,String>". Then you could call: HeaderType header = bodyPart.getHeaders().getFirst("Content-Disposition"); System.out.println("Disposition value is " + header.getValue()); // "form-data" System.out.println("Name is " + header.getParameters().get("name")); // "files" What do you think? > 2) I took a quick glance at the implementations and I see some problematic > //FIXME comments, such as not being able to configure how big an attachment > may get before being dumped to disk, requiring explict calls to cleanup(), > etc. > > Dealing with the configuration seems solvable fairly easily, although the interesting bit is to find a solution that works for both a servlet deployment (where there could be multiple independent apps deployed in the same server, so system properties don't work well) and a non-servlet deployment (where we don't have any access to servlet APIs. Probably the best bet is an optional properties file loaded via the thread context class loader (if any; otherwise the class loader that loaded the jersey-multipart classes) that can be used to set configuration stuff like this. Regarding the need for the cleanup call, I'm open to suggestion for how to improve this. I've started looking at Jersey filters (which would impose a Jersey-specific implementation dependency), but haven't settled on anything yet. Separately, I took your suggestion to look at Commons FileUpload. It turns out that there *is* a single important class (org.apache.commons.fileupload.MultipartStream) -- plus a couple of small helpers -- that performs the only real task I'm currently delegating to JavaMail. That's the actual parsing of the multipart/* input stream. I need to do some experiments, and take heed of the potential concerns in the javadocs about nested multipart/* body parts, but it may well be that we could incorporate a variant of just this class and not even need the entire Commons FileUpload package (or it's dependence on Commons IO). > Gili > > Craig > Craig McClanahan wrote: > >> Gili wrote: >> >>> Hi Craig, >>> >>> On the topic of jersey-multipart: >>> >>> 1) Is it really Jersey-specific or can it be used with any JAX-RS >>> implementation? >>> >>> >> Currently, the only Jersey-specific dependencies are in the unit tests. >> In principle the runtime should only require the JAX-RS APIs, JavaMail, >> and JAF. But I haven't tested it with anything else. >> >>> 2) I'm curious whether you looked into using Apache Commons FileUpload >>> instead of Javamail under the hood. FileUpload's jar is only 50k and >>> deals >>> exclusively with parsing this kind of data, versus Javamail which is an >>> entire email client. It might also be more flexible, better integrated >>> than >>> JavaMail. >>> >>> >>> >> I hadn't looked at Commons FileUpload, based on the (perhaps mistaken?) >> assumption that it only supported "multipart/form-data". I'll >> definitely go take a look. >> >> On the other hand, basically all the server side apps I'm working on are >> running on Glassfish anyway, so JavaMail comes integrated "for free" :-). >> >>> Gili >>> >>> >>> >> Craig >> >> >>> Craig McClanahan wrote: >>> >>> >>>> For those who don't know me, I have been around the Java web tier for >>>> quite a while, being the original author of the Struts framework >>>> (<http://struts.apache.org>), as well as co-spec-lead for JavaServer >>>> Faces 1.0 and 1.1. My more recent interests have focused on RESTful web >>>> services, which led me naturally towards JAX-RS and the Jersey >>>> implementation. >>>> >>>> I've been one of the folks inside Sun who has been leveraging Jersey for >>>> some internal projects over the last few months. We had a particular >>>> need to support MIME multipart/* media types, and it made sense to >>>> generalize this into a reusable module -- hence, I've just uploaded the >>>> "jersey-multipart" module to the "contribs" directory. It relies on 1.0 >>>> or later Jersey code, and provides what I hope are found to be elegant >>>> solutions to the problems of multipart handling, while leveraging all >>>> the nice JAX-RS providers support for dealing with the entity content of >>>> body parts, just like we've grown spoiled by on complete message >>>> bodies. And, it works both on the server side and the client side, when >>>> you use jersey-client. >>>> >>>> Example server code to build a multipart response might look like this: >>>> >>>> // I have also provided an appropriate MessageBodyWriter for the >>>> MyBean class >>>> MyBean bean = ...; >>>> return Response.ok(new MultiPart(). >>>> type(new MediaType("multipart", "mixed"). >>>> bodyPart("This is the first body part in plain text", new >>>> MediaType("text", "plain")). >>>> bodyPart(bean, new MediaType("x-application", >>>> "x-format"))).build(); >>>> >>>> (Of course, you can do things in a more fine-grained fashion, but the >>>> builder pattern utilized all over the JAX-RS and Jersey APIs was so cool >>>> that Paul suggested I use it here too, so I did :-). >>>> >>>> To read a MultiPart entity (produced by code like the previous example) >>>> that was uploaded to the server you might do something like this: >>>> >>>> // I have also provided an appropriate MessageBodyReader for the >>>> MyBean class >>>> @Path("...") >>>> @PUT >>>> @Consumes("multipart/mixed") >>>> @Produces(...) >>>> public Response handler(MultiPart multiPart) { >>>> BodyPart part0 = multiPart.getBodyParts().get(0); >>>> String text = part0.getEntityAs(String.class); >>>> BodyPart part1 = multiPart.getBodyParts().get(1); >>>> MyBean bean = part1.getEntityAs(MyBean.class); >>>> ... >>>> multiPart.cleanup(); >>>> } >>>> >>>> The need for cleanup() is because the implementation knows how to buffer >>>> "large" body parts to temporary files on disk, so you don't blow away >>>> your JVM heap on a multi-gigabyte upload or download. I'm looking for a >>>> way to avoid the need for the application to call this, but haven't >>>> found one yet -- in the mean time, everything else about dealing with >>>> multipart files has seemed pretty easy to deal with. >>>> >>>> As mentioned above, this module works on the client side as well, if >>>> you're using jersey-client. The unit tests have some more worked-out >>>> examples of the lower level details. >>>> >>>> Give it a try and see what you think! And, for sure, if you see >>>> anything that needs to be improved, please ask here and/or file an issue >>>> in the issue tracking system. >>>> >>>> Craig McClanahan >>>> >>>> PS: Among my other interests will be working with the Atom Publishing >>>> Protocol support, again with the idea of leveraging JAX-RS providers to >>>> do format translations for custom <content> payloads. >>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: [hidden email] >>>> For additional commands, e-mail: [hidden email] >>>> >>>> >>>> >>>> >>>> >>> >>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [hidden email] >> For additional commands, e-mail: [hidden email] >> >> >> >> > > --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
|||||||||||||||||||
|
Gili
|
That sounds exactly like the kind of thing I was looking for. BTW, I just discovered this related discussion: http://forums.sun.com/thread.jspa?threadID=5333195 This, in turn, led me to javax.mail.internet.ContentDisposition and more importantly javax.mail.internet.HeaderTokenizer. We should be able to use these fairly easily to implement the aforementioned APIs. Why do you need to support non-servlet deployments? As soon as we start talking about class loaders things get ugly real quick. Ideally I'd like to cheat by simplifing the requirements if possible ;) FileUpload has a background thread that monitors when its File instances get garbage-collected (Guice does something similar) and when this happens it goes and deletes the files off the hard-drive automatically. It also uses a Servlet Filter to hook servlet shutdown to force immediate deletion of all outstanding files. At least, that's what I understood from their Javadoc. The only reason I suggested looking at FileUpload is I thought it was an active project that it handles temporary files and configuration under the hood for us. Now I'm not so sure anymore. If you believe that adding their configuration on top of Javamail isn't much hassle I think I'd prefer that than relying on their code. I say this because I noticed that Javamail has been open-sourced and is now part of Glassfish and I trust those guys a heck of a lot more than I do your average Apache project. My main concern now is stability and simplicity. I guess I no longer care about the underlying JAR size ;) Gili |
||||||||||||||||||
|
Craig McClanahan
|
Some javascript/style in this post has been disabled (why?)
Gili wrote:
Cool. I love it when great minds think alike :-).Craig McClanahan wrote: Ugh ... that (parsing MIME headers) is something else that I didn't have to worry about when I was "constructively lazy" and depended on JavaMail for parsing. We'd definitely need to build something on top of the MultipartStream from Commons FileUpload, since that only gives you body parts as a whole, and doesn't deal with the headers.BTW, I just discovered this related discussion: http://forums.sun.com/thread.jspa?threadID=5333195 This, in turn, led me to javax.mail.internet.ContentDisposition and more importantly javax.mail.internet.HeaderTokenizer. We should be able to use these fairly easily to implement the aforementioned APIs. Two reasons:Craig McClanahan wrote: * On the server side, JAX-RS explicitly declares support for non-servlet deployments (although lots of details are left as "exercise for the implementor" in the 1.0 spec). There is no a priori reason I can think of that we should restrict multipart support to only work for servlet based server side deployments. * A lot of my use cases involve *client* applications using RESTful web services. Jersey has a very nice client API that can leverage much of the JAX-RS infrastructure on the client side as well, and it would be really weird to "import javax.servlet.*" into an applet, or a RIA app based on Swing or Java FX. I know that pain :-), having been pretty heavily involved in the servlet container inside Tomcat from version 4.x -- the basic architecture today is pretty similar although quite refined. But let me state a requirement I think we must satisfy in a different way: it must be possible to deploy two or more different server apps, each using jersey-multipart, in the same instance of a servlet container, with different configuration settings for things like the threshold size before a body part gets spooled to a disk file.Craig McClanahan wrote: The algorithm I described above is quite typical of the way your average web framework (including Struts) locates application specific resources. Fortunately, we can make it work transparently in a non-servlet world too, by using things like ClassLoader.getResourceAsStream() instead of ServletContext.getResourceAsStream(). I'm not usually a fan of "background thread cleanup" type solutions because they are (a) asynchronous, and therefore consume resources for longer than they should be consumed, and (b) they encourage sloppy coding -- "someone else will clean up my mess, so I don't have to worry about it." It might not be quite so bad in this use case, although we'd need to again deal separately with the servlet versus non-servlet case (servlet based temp files should be cleaned up when the app is undeployed, not just when the container is shut down), but that might be feasible.Craig McClanahan wrote: Unless performance considerations dictate otherwise, I'm much more into synchronously cleaning up after myself after each request. But it's definitely not optimal to make the app developer responsible for ensuring that this happens. I'm afraid I'm biased both ways so can't help you much in determining who to trust more.Craig McClanahan wrote: I work for Sun (originally in the J2EE (now Java EE) group where Glassfish now comes from), and have also been heavily involved in many Apache projects -- most particularly Tomcat and Struts, the latter being where many of the Commons projects got a lot of initial developers and initial codebases. I trust them both :-). +1 for stability and simplicity. But this is a case where the "reuse" argument might tends towards fork-and-copy-a-few-classes (with appropriate attributions, of course, to satisfy the relevant open source licenses), rather than importing reasonably large packages when we're only using a few classes from them.My main concern now is stability and simplicity. I guess I no longer care about the underlying JAR size ;) On the other hand, the JavaMail based support for parsing multipart messages "just works" ... CraigGili |
|||||||||||||||||||
|
Craig McClanahan
|
In reply to this post
by Robertson, Jeff
Some javascript/style in this post has been disabled (why?)
Robertson, Jeff wrote:
JavaMail is indeed required to be available for a J2EE (older) or Java EE (newer) app server. That's only true for an out-of-the-box deployment of something like Tomcat or Jetty. Every servlet container I'm aware of provides a mechanism for you to add JAR files to the set of libraries that are made available to *all* webapps that are deployed on that container. For example, on Tomcat 4.x/5.x, just put javamail.jar and jaf.jar into "$TOMCAT_HOME/common/lib". On Tomcat 6.x, use "$TOMCAT_HOME/lib" instead. You'll find some similar capability for all the other containers too. Craig |
|||||||||||||||||||
|
Paul Sandoz
|
In reply to this post
by Craig McClanahan
Some javascript/style in this post has been disabled (why?)
On Nov 4, 2008, at 9:06 AM, Craig McClanahan wrote:
private static Map<String, BodyPart> getFormData(MimeMultipart mm) throws Exception { Map<String, BodyPart> m = new HashMap<String, BodyPart>(); for (int i = 0; i < mm.getCount(); i++) { BodyPart b = mm.getBodyPart(i); if (b.getDisposition() != null && b.getDisposition().equalsIgnoreCase("form-data")) { String name = getName(b.getHeader("content-disposition")[0]); if (name != null) m.put(name, b); } } return m; } private static String getName(String disposition) throws ParseException { HttpHeaderReader reader = new HttpHeaderReaderImpl(disposition); // Skip any white space reader.hasNext(); // Get the "form-data" reader.nextToken(); while (reader.hasNext()) { reader.nextSeparator(';'); // Ignore a ';' with no parameters if (!reader.hasNext()) break; // Get the parameter name String name = reader.nextToken(); reader.nextSeparator('='); // Get the parameter value String value = reader.nextTokenOrQuotedString(); if (name.equalsIgnoreCase("name")) { return value; } } return null; } I am reusing the HTTP header parsing code for parsing the MIME header. There are some close similarities between HTTP headers and MIME headers but there may also be some subtle differences that results failure for edge cases.
1) Have a class in jersey core called FeatureAndProperties; 2) The ClientConfig and ResourceConfig extend this class. 3) FeaturesAndProperties can be injected. Then message body readers/writers requiring access to config information are independent of client/server.
There is an issue with isolated clients though, and may be that is where a thread clean up solution is appropriate and we may be able to hide that functionality through different life-cycle implementation support. This does of course mean that the mail support is dependent on Jersey specific APIs.
Paul.
|
|||||||||||||||||||
|
Gili
|
In reply to this post
by Craig McClanahan
Correct me if I'm wrong, but it doesn't look like parsing the headers is difficult at all. I believe that HeaderTokenizer takes care of the quotes and internal whitespace for you. All that's left for you to do is tell it to scan for semi-colons and equal-signs, take the parser output and pass it into HeaderType. Gili |
||||||||||||||||||
| Free Embeddable Forum Powered by Nabble | Help |
