These audio problems are always sporadic. Don't stress about you not being able to replicate it always, that is the very nature of this type of problem.
can you post what you have in your sip_nat.conf and your rtp.conf?
They should match the ports above.
Also,
1: does your piaf have only one network card?
2: Is there ABSOLUTELY NO CHANCE that another box may have an IP conflict with your PIAf's internal IP?
4: What about your handsets? no chance of an IP conflict there?
5: Could it possibly be a contention issue (ie is anyone / thing sharing your Piaf Server's bandwidth.
6: please also provide your hardware details. primarily brand of NIC. I can;t really help you there but there are pleanty of Linux Geeks who could tell you if you are using the right drivers etc.
7: disable IP tables (or just allow ICMP echo if you know how) and setup a ping -t ip.add.of.pbx > c:\pinglog.txt
when you get a drop out stop the ping and review the log.
8: replace your server's NIC. The only problems I have had with PIAF was hardware related!!! I had a NIC intermittently fail.
Also which way is the one way audio?
It is almost always a networking problem. and networking problems are almost always the simplest things that goes wrong.
Today I installed a cisco phone for someone (they had complained about poor speakerphone performance) & it tested fine. They rang me (from thier mobile) complaining that the phone was broken. I knew it wasn't because I had tested it. Turns out I had plugged in the handset to the headset socket. I was so stressed about getting the speakerphone just right I never tested the obvious!