Why do most RTP sessions use separate streams for audio and video? How can a receiving client synchronize these streams?

Sending audio and video in separate RTP streams provides a great deal of flexibility. For example, this makes it possible for a player to receive only the audio stream, but not video (or vice-versa). It would even be possible to have one computer receive and play audio, and a separate computer receive and play video.

These audio and video streams are synchronized using RTCP "Sender Report" (SR) packets - which map each stream's RTP timestamp to 'wall clock' (NTP) time. For more information, see the IETF's RTP/RTCP specification.

Receivers can then use this mapping to synchronize the incoming RTP streams. The LIVE555 Streaming Media code does this automatically: For subclasses of "RTPSource", the "presentationTime" parameter that's passed to the 'afterGettingFunc' of "getNextFrame()" (see "liveMedia/include/FramedSource.hh) will be an accurate, time-synchronized time. (For this to work, you need to have also created a "RTCPInstance" for each RTP source.)

For example, if you use "openRTSP" to receive RTSP/RTP streams, then the contents of each RTP stream (audio and video) are written into separate files. This is done using the "FileSink" class. If you look at the "FileSink::afterGettingFrame()" member function, you'll notice that there's a "presentationTime" parameter for each incoming frame. Some other receiver could use the "presentationTime" parameter to synchronize audio and video.