I found this solution here:
This is an issue with Core Audio on OS X getting caught in some sort of resource availability loop with Apple TV. If you restart Core Audio on OS X and make sure there are no other iOS or OS X devices requesting access to the ATV, then it’ll fix this issue. To restart Core Audio you can either:
Method 1:
1) Open Activity Monitor (control + space then start typing “activity monitor”)
2) Find the process coreaudiod
3) Select and quit
Method 2:
1) Open terminal
2) type command: sudo killall coreaudiod
3) enter admin password and confirm
OS X will cycle Core Audio on and off and voila- your audio streaming will work again.
There’s no way to prevent this issue right now. It’s a Mountain Lion bug, I think because I can swap control between my iPhone and iPad without any issues like this coming up.