What you have in your diagram is not a great idea.
Yes it would work, but it places the turbo farther from the exhaust manifolds, and it won't spool as quick. You're better off getting rid of the crossover pipe connecting the two manifolds and having a pipe from each manifold meet at the turbo. Then, as you said, just plug the hole where the stock downpipe was.
What does the back manifold look like on a 3400? The exhaust would essentially have to flow backwards to where the crossover pipe was, and if the manifold is shaped more like a header instead of a log, it won't flow very well.
__________________
|