Opening "a shadowbox window" when the user clicks on a submit button is simple. but only if you are not actually submitting the form!
However, I suspect that what you
really want to do is submit the form
into the newly opened shadowbox window? That is, the main page submits the form, but the result of the submittal (the action) returns into a shadowbox iframe without changing the main page?
This is not so simple, and the best way is to submit the form via ajax, load the returned result into a hidden inline div, then open an inline shadowbox displaying that div.
(Tip : use a library, such as jQuery)
Combine literacy with curiosity and a whole world of information opens up to you