The script you have posted enables the following...
Page A
iframe[1], containing Page B (has the posted script)
Then, Page B tells Page A to open Page C (containing the form) in a Shadowbox iframe...
Page A
iframe[1], containing Page B (has the posted script)
iframe[2], containing Page C (the form, with a standard submit to itself)
When the form on Page C is submitted using a standard submit, the resultant page will appear in the current frame, ie. iframe[2]. My (limited) understanding is that the resultant page may be Page C again - or similar - (if the form submittal was invalid), or Page D (if the form submittal was valid). If Page D is the resultant page (ie. the form was valid) then you want it open in iframe[1] instead of iframe[2]?
To do that, you could return a different page, Page E, from a valid submittal. Page E runs a script located on Page A. That script on Page A closes its Shadowbox (iframe[2]), and then changes the src of iframe[1] to Page D. (The url to Page D can be returned as part of Page E, and then passed to Page A when calling the script).
This is all complicated by the fact that you have Shadowbox running from the top frame, but with content initially controlled by and subsequently affecting an entirely different iframe. You need to think very carefully about your cross-frame controls, and whether you actually need the different iframes.
This is all based on my limited understanding of what you are trying to achieve, which may be incorrect!
Combine literacy with curiosity and a whole world of information opens up to you