How to force rerouting of a packet after return to netfilter from NFQUEUE

I have a caching application that runs in userspace. Normally, it uses DPDK to interact with the NICs directly without kernel interaction, grabbing packets off of the wire, and either passing them through, generating a response from cache, or in some occasions blocking them. It's transparent to the endpoints, and is written to work on raw packet data straight from the NIC. Notably, I'm not doing normal socket programming here - everything's handled at the packet level, without interaction with the TCP/IP stack.

For reasons that are long and boring, I wanted to add some NAT functionality. As a proof of concept, I was hoping to put a front end on my application using iptables/netfilter. So I did the NAT parts with commands like these:

sudo iptables --table nat --append PREROUTING --in-interface seg1a  -j DNAT --to-destination $SERVERIP
sudo iptables --table nat --append POSTROUTING --out-interface seg1b -j MASQUERADE
sudo route add $SERVERIP seg1b

This works well for my purposes. The clients now attach to an interface on my system, but their traffic gets redirected to the server. I'm guaranteed that their traffic passes through my system, so all's good.

But to be useful as a cache, I need to be able to respond to some requests that I get from clients. I had thought I could use NFQUEUE for this, with a small application that reads from the netfilter queue and passes packets to and from my application via IPC. I used an iptables rule like this:

sudo iptables --table mangle --append FORWARD -j NFQUEUE

This works OK as long as my application doesn't respond to anything. But when my cache attempts to respond to something from one of the endpoints, things go wrong. The cache reverses all of the L2-4 headers, manages sequence and ack numbers, etc. But the packets don't get back to the client.

I think what's happening is that routing decisions for the packet were made before it was sent to NFQUEUE. So even though the cache returns something whose source and destination IP addresses are reversed, that's irrelevant for the routing of the packet.

I've tried a number of things to change the routing of my response packets, but nothing seems to work. How does one go about changing the routing on a packet read from netfilter queues? Failing that, is there a good way to just inject packets onto the wire from netfilter queues? If that were doable, I could block the original request and then send the cache's reply as an entirely new thing.

Thanks in advance for any suggestions.