Learning VoIP, RTP and SIP (aka awesome pjsip)

Before working with Windows Phone and iOS, my life involved researching VoIP. That was to build a C library for voice over IP functionality for a very popular app, and that was how I got started in open source.

The library I was working with were Linphone and pjsip. I learn a lot of UDP and SIP protocol, how to build C library for consumption in iOS, Android and Windows Phone, how challenging it is to support C++ component and thread pool in Windows Phone 8, how to tweak entropy functionality in OpenSSL to make it compile in Windows Phone 8, how hard it was to debug C code with Android NDK. It was time when I needed to open Visual Studio, Xcode and Eclipse IDE at the same time, joined mailing list and followed gmane. Lots of good memories.

Today I find that those bookmarks I made are still available on Safari, so I think I should share here. I need to remove many articles before they are outdated or not available anymore. These are the resources that I actually read and use, not some random links. Hopefully you can find something useful.

This post shows more resources for pjsip on client and how to talk directly and with/without a proxy server.

First of all

Here are some of the articles and open sources made by me regarding VoIP, hope you find it useful

• rtpproxy: I forked from http://www.rtpproxy.org/ and changed code to make it support for IP handover. It means the proxy can handle when IP changes from 3G, 4G to Wifi and to reduce chances of attacks

• Jitter buffer in VoIP

• How to calculate packet size in VoIP

VoIP overview

Voice over Internet Protocol (also voice over IP, VoIP or IP telephony) is a methodology and group of technologies for the delivery of voice communications and multimedia sessions over Internet Protocol (IP) networks, such as the Internet

• Voice over IP Overview: introduction to VoIP concepts, H.323 and SIP protocol

• Voice over Internet Protocol the wikipedia article contains very foundation knowledge

• Open Source VOIP Software: this is a must read. Lots of foundation articles about client and server functionalities, SIP, TURN, RTP, and many open sources framworks

• VOIP call bandwidth: a very key factor in VoIP application is bandwidth consumption, it’s good to not going far beyond the accepted limit

• Routers SIP ALG: this is the most annoying, because there is NAT and many types of NAT, also router with SIP ALG

• SIP SIMPLE Client SDK: introduction to SIP core library, but it gives an overview of how

SIP

The Session Initiation Protocol (SIP) is a communications protocol for signaling and controlling multimedia communication sessions in applications of Internet telephony for voice and video calls, in private IP telephone systems, as well as in instant messaging over Internet Protocol (IP) networks.

• Session Initiation Protocol

• RFC 3261: to understand SIP, we need to read its standard. I don’t know how many times I read this RFC.

• OpenSIPS: OpenSIPS is a multi-functional, multi-purpose signaling SIP server

• SIP protocol structure through an example: this is a must read, it shows very basic but necessary knowledge

• Relation among Call, Dialog, Transaction & Message: basic concepts about call, dialog, transaction and message

• microSIP: Open source portable SIP softphone for Windows based on PJSIP stack. I used to use this to test my pjsip tweaked library before building it for mobile

• What is SIP: introduction to SIP written by the author of CSipSimple

• What is SIP proxy server

• SIP by Wireshack: introduction to SIP written by Wireshack. I used Wireshack a lot to intercept and debug SIP sessions

• Solving the Firewall/NAT Traversal Issue of SIP: this shows how NAT can be a problem to SIP applications and how NAT traversal works

• Introduction to SIP for Java, C#, and VB Developers

• SipML5 SIP client written in Javascript

• SIP Retransmissions: what and how to handle retransmission

• draft-ietf-sipping-dialogusage-06: this is a draft about Multiple Dialog Usages in the Session Initiation Protocol

• Creating and sending INVITE and CANCEL SIP text messages: SIP also supports sending text message, not just audio and video packages. This isa good for chat application

SIP server

• Kamailio: this is the server that I used, and it plays well with lots of standard SIP clients, including pjsip. Debugging on this server was also a fun story

• Configuring NAT traversal using Kamailio 3.1 and the Rtpproxy server: I don’t know how many times I had read this post

• How to set up and use SIP Server on Windows: I used this to test a working SIP server on Windows

• Build your own VoIP System

• OpenSIPS/Kamailio serving far end nat traversal: discussion about how Kamailio deals with NAT traversal

• NAT Traversal Module: how NAT traversal works in Kamailio as a module

RFC

RTP, SIP clients and server need to conform to some predefined protocols to meet standard and to be able to talk with each other. You need to read RFC a lot, besides you need to read some drafts.

• RFC 3550 - RTP: A Transport Protocol for Real-Time Applications

• RFC 3261 - SIP: Session Initiation Protocol

• Symmetric NAT Traversal using STUN

• RFC 3842 - A Message Summary and Message Waiting Indication Event Package for the Session Initiation Protocol (SIP)

NAT

NAT solves the problem with lack of IP, but it causes lots of problem for SIP applications, and for me as well 😂

• Network address translation: Network address translation (NAT) is a method of remapping one IP address space into another by modifying network address information in the IP header of packets while they are in transit across a traffic routing device

• SIP and NAT: Why is it a problem?

• Configuring Port Address Translation (PAT): how to configure port forwarding

• Types Of NAT Explained (Port Restricted NAT, etc): This is a must read. I didn’t expect there’s many kinds of NAT in real life, and how each type affects SIP application in its own way

• One Way Audio SIP Fix: sometimes we get the problem that only 1 person can speak, this talks about why

• NAT traversal for the SIP protocol: explains RTP, SIP and NAT

• A New Method for Symmetric NAT Traversal in UDP and TCP

• SIP NAT Traversal: This is a must read. How to make SIP work under NAT

• NAT and Firewall Traversal with STUN / TURN / ICE: pjsip and Kamailio actually supports STUN, TURN and ICE protocol. Learn about these concepts and how to make it work

• Introduction to Network Address Translation (NAT) and NAT Traversal

TCP

Learn how TCP helps SIP in initiating session and to turn in TCP mode for package sending

• Transmission Control Protocol: The Transmission Control Protocol (TCP) is one of the main protocols of the Internet protocol suite. It originated in the initial network implementation in which it complemented the Internet Protocol (IP)

• Datagram socket: A datagram socket is a type of network socket which provides a connectionless point for sending or receiving data packets.[2] Each packet sent or received on a datagram socket is individually addressed and routed

• TCP RST packet details: learn the important of RST bit

• RST packet sent from application when TCP connection not getting closed properly

• Why will a TCP Server send a FIN immediately after accepting a connection?

• Where do resets come from? (No, the stork does not bring them.): learn about 3 ways handshake in TCP connection

• TCP listen() Backlog

• Sockets and Ports: Do not confuse between socket and port

• TCP Wake-Up: Reducing Keep-Alive Traffic in Mobile IPv4 and IPsec NAT Traversal

• failed to register using tcp only

• TCP vs UDP

TLS

Learn about Transport Layer Security and SSL, especially openSSL for how to secure SIP connection. The interesting thing is to read code in pjsip about how it uses openSSL to encrypt messages

• Configuring PJSIP with TLS

• Why TLS for SIP

• SIP Signaling Over TLS

• SSL/TLS certificates: What you need to know

• Configuring TLS support in Kamailio 3.1 — Howto: learn how to enable TLS mode in Kamailio

• SIP TLS: how to configure TLS in Asterisk

ICE

Learn about Interactive Connectivity Establishment, another way to workaround NAT

• SIP 2012:: ICE — NAT traversal for media

• Introducing pjnath — Open Source ICE, STUN, and TURN for NAT Traversal

• ICE Media Transport

STUN and TURN

Learn about Session Traversal Utilities for NAT and Traversal Using Relays around NAT, another way to workaround NAT

• STUN: STUN (Simple Traversal of UDP through NATs (Network Address Translation)) is a protocol for assisting devices behind a NAT firewall or router with their packet routing. RFC 5389 redefines the term STUN as ‘Session Traversal Utilities for NAT’.

• What is STUN and does it need a port-forwarded server?

• TURN server

ALG

Learn about Application Layer Gateway and how it affects your SIP application. This component knows how to deal and modify your SIP message, so it might introduce unexpected behaviours.

• What is SIP ALG and why does Gradwell recommend that I turn it off?

• Understanding the SIP ALG

• What Is Sip ALG (Application Layer Gateway) Voip firewall

• About the SIP ALG

• Understanding SIP with Network Address Translation (NAT): This is a must read, a very thorough document

Voice quality

Learn about voice quality, bandwidth and fixing delay in audio

• VoIP — fixing voice quality

• What is Delay in VoIP?

• Understanding Delay in Packet Voice Networks

• Reducing the SIP Packet Size in VoIP

• What Affects Voice Quality in VoIP Calls

• 5 Curable Causes of Poor VoIP Call Quality

• RTP, Jitter and audio quality in VoIP: learn about the important of jitter and RTP

• An Adaptive Codec Switching Scheme for SIP-based VoIP: explain codec switching during call in SIP based VoIP

• How to master VoIP bandwidth fundamentals

• Voice Over IP — Per Call Bandwidth Consumption

Echo

This is a very common problem in VoIP, sometimes we hear voice from the other and also from us. Learn how echo is made, and how to effectively do echo cancellation

• Echo Analysis for Voice over IP

• Echo Cancellation: How to use Speex to cancel echo

• Echo suppression and cancellation

• Echo and Sidetone: A telephone is a duplex device, meaning it is both transmitting and receiving on the same pair of wires. The phone network must ensure that not too much of the caller’s voice is fed back into his or her receiver

• VoIP Echo and how to correct it

• Causes of Echo

• How software echo canceller works?: I asked about how we use software to do echo cancellation

Dual Tone

Learn how to generate dual tone to make signal in telecommunication

• DTMF (Dual-Tone Multi-Frequency)

• The dance of DTMF, SIP & RFC 2833 — An introduction

• How can I send inband DTMF tones?

pjsip

PJSIP is a free and open source multimedia communication library written in C language implementing standard based protocols such as SIP, SDP, RTP, STUN, TURN, and ICE. It combines signaling protocol (SIP) with rich multimedia framework and NAT traversal functionality into high level API that is portable and suitable for almost any type of systems ranging from desktops, embedded systems, to mobile handsets.

• PJSUA API — High Level Softphone API: high level usage of pjsip

• pjsip library architecture

• pjsip documentation

• Stateful Operations: common functions to send request statefully

• Message Creation and Stateless Operations: functions related to send and receive messages

• Understanding Media Flow: this is a must read. The media layer is so important, it controls sound, codec and conference bridge.

• Getting Started: Building and Using PJSIP and PJMEDIA: This article describes how to download, customize, build, and use the open source PJSIP and PJMEDIA SIP and media stack

• Codec Framework: pjsip supports multiple codec

• Adaptive jitter buffer: this takes sometime to understand, but it plays an important part in making pjsip work properly regarding buffer handling

• PJSUA-API Accounts Management: how to register account in pjsua

• Building Dynamic Link Libraries (DLL/DSO): how to build pjsip as a dynamic library

• Compile time configuration: lots of configuration we can apply to pjsip

• Fast Memory Pool: pjsip has its own memory pool. It’s very interesting to look at the source code and learn something new

• SIP and Media Features

• Using SIP TCP Transport: How to enable TCP mode in SIP and to initiate SIP session

• Monochannel and multichannel audio frame converter: interesting read about mono and multi channel

• IOQueue: I/O Event Dispatching with Proactor Pattern: the code for this is very interesting and plays a fundamental in how pjsip handles events

• DNS Asynchronous/Caching Resolution Engine: how pjsip handles DNS resolution by itself

• Secure socket I/O: the code for this is important if you want to learn how to use SSL under the hood

• Multi-frequency tone generator: I learn a lot how pjsip uses sin wave to generate tone

• SIP SRV Server Resolution (RFC 3263 — Locating SIP Servers): learn the mechanism for how pjsip finds a particular SIP server

• Exception Handling: how to do Try Catch in C

• Mutex Locks Order in PJSUA-LIB: how multiple locks at each layer helps ensure correction and avoid deadlocks. I had lots of nightmare debugging deadlocks with pjsip 😱

Threading

pjsip uses Local Thread Storage which introduces very cool behaviors

• Thread — Operating System Dependent Functionality

• Threads question: how pjlib handles thread

• Using Thread Local Storage: how to use TlsAlloc and TlsFree in Windows

• The Windows Processes and Threads 8

• Example: Thread local storage in a Pthread program: how Pthread works

• Thread Local Storage: learn about pj_thread

Resampling

How to work with sample rate of the media stream

• Resample Port: how to perform resampling in pjmedia

• Resampling Algorithm: code to perform resampling

• Samples: Using Resample Port: very straightforward example to change sample rate of the media stream

Memory and Performance

• Memory allocation strategy

• PJSIP — High Performance Open Source SIP Stack

Audio

• How to Record Audio with pjsua: how to use pjsua to record audio.

• Memory/Buffer-based Capture Port: believe me, you will jump into pjmedia_mem_capture_create a lot

• File Writer (Recorder): record audio to .wav file

• Using pjsua to create a mp3 stream

• AMR Audio Encoding: understands AMR encoding

• Audio Device API: how pjsip detects and use Audio device

• Sound Device Port: Media Port Connection Abstraction to the Sound Device

• audio bursting

• Buffer problem

• Problem with PJMEDIA’s play callback

• Audio Manipulation Algorithms: lots of cool algorithm written in C for audio manipulation. The hardest and most imporant one is probably Adaptive jitter buffer

• bad quality on iphone 2G with os 3.0: No one would use iPhone 2G now, but it’s good to be aware of older phones

• getting Underflow, buf_cnt=0, will generate 1 frame continuessly: how to handle underflow in pjmedia

• Measuring Sound Latency: This article describes how to measure both sound device latency and overall (end-to-end) latency of ​pjsua

• Checking for Network Impairments of Incoming RTP Packets

• Master/sound: How master sound works and deal with no sound on the mic input port

Video

I learn a lot regarding video capture, ffmpeg and color space, especially YUV

• siphon — VIdeoSupport.wiki: How siphon deals with video before pjsip 2.0

• Video Device API; PJMEDIA Video Device API is a cross-platform video API appropriate for use with VoIP applications and many other types of video streaming applications.

• PJSUA-API Video: Uses video APIs in pjsua with pjsip 2.1.0

• PJSIP Video User’s Guide: all you need to know about video support in pjsip

• Video streams: I can’t never forget pjmedia_vid_stream_create

• Video source duplicator: duplicate video data in the stream.

• AVI File Player: Video and audio playback from AVI file

• PJSIP Version 2.0 Release Notes: starting with 2.0, pjsip supports video. Good to read

• Video API for PJSUA-LIB

• How to make a loopback video call with AVI file?

• What is lib Swscale used for by ffmpeg programers?

• FFmpeg-iOS-build-script: details how to build ffmpeg for iOS

CSipSimple

There are many SIP client for mobile and desktop, microSIP, Jitsi, Linphone, Doubango, … They all follow strictly SIP standard and may have their own SIP core, for example microSIP uses pjsip, Linphone uses liblinphone, …

Among that, I learn a lot from the Android client, CSipSimple, which offers very nice interface and have good functionalities. Unfortunately Google Code was closed, so I don’t know if the author has plan to do development on GitHub.

I also participated a lot on the Google forum for user and dev. Thanks for Regis, I learn a lot about open source and that made me interested in open source.

You can read What is a branded version

I don’t make any money from csipsimple at all. It’s a pure opensource and free as in speech project.
I develop it on my free time and just so that it benefit users.
That’s the reason why the project is released under GPL license terms. I advise you to read carefully the license (you’ll learn a lot of things on the spirit of the license and the project) : http://www.gnu.org/licenses/gpl.html
To sump up, the spirit of the GPL is that users should be always allowed to see the source code of the software they use, to use it the way they want and to redistribute it.

RTP Proxy

Because of NAT or in case users want to talk via a proxy, then a RTP proxy is needed. RTPProxy follows standard and works well with Kamailio

• Making RTPproxy work

• Sippy B2BUA and RFC3261 SIP Stack

• rtpproxy(8) — Linux man page

• rtpproxy address filling

Other related articles

• Sample Rate and Bitrate: The Guts of Digital Audio

• VoIP Packet Size

• How to measure brandwith consumption of an rtp stream?

• Sending Reliable Provisional Responses

• SILK audio codec wrapper implementation

• conference bridge questions

• multiple audio devices, multiple calls, conferencing, recording and mix all of the above

• How to Know Offline Call

• Symmetric RTP

• Account specific NAT settings: STUN, ICE, and TURN

• SIP stories, part 3: INVITE retransmission

• TCP/TLS reconnect

• Configuring TCP keep alive and connection lifetime

• Max TCP connections

• VAD detection scenario

• Handle native capture preview

• Video orientation support

• Keep-alive mechanism for TCP and TLS transports

• Periodically transmit RTP packet on silence

• Conference bridge should transmit silence frame when level is zero

• Add user defined NAT hole-punching and keep-alive mechanism to media stream

IP change

IP change during call can cause problem, such as when user goes from Wifi to 4G mode

• Proper way to handle Ip address changes in Android

• support WiFi and 3G simultaneously

RTP and RTCP

Learn about Realtime transport control protocol and how that works with RTP

• Voice over IP: RTP/RTCP — The transport layer

• RTCP, RTP Control Protocol

• Protocol overview: RTP and RTCP

• The Multimedia Control Protocol RTCP

• Using Conference Bridge

• SIP and RTP Stack

• Media Transport

• RTP Session and Encapsulation (RFC 3550)

Codec

To reduce payload size, we need to encode and decode the audio and video package. We usually use Speex and Opus. Also, it’s good to understand the .wav format

• Digital Audio — Creating a WAV (RIFF) file

• Streaming Data from a WAV File

• Programming with Speex (the libspeex API)

• Speex narrowband mode

• Developing with libopus (API reference)

Building pjsip for Windows Phone 8

Windows Phone 8 introduces C++ component , changes in threading, VoIP and audio background mode. To do this I need to find another threadpool component and tweak openSSL a bit to make it compile on Windows Phone 8. I lost the source code so can’t upload the code to GitHub 😢. Also many links broke because Nokia was not here any more

• Building for Other Platforms

• Getting Started: Building for UWP and Windows Phone 8.x

• Porting to New CPU Architecture: pjlib is the foundation of pjsip. Learn how to port it to another platform

• VoIP apps for Windows Phone 8

• How to implement audio streaming for VoIP calls for Windows Phone 8

• In-process, Out-of-process, and Remote Servers

• Basics of an IDL file

Porting OpenSSL to Windows Phone 8

Firstly, learn how to compile, use OpenSSL. How to call it from pjsip, and how to make it compile in Visual Studio for Windows Phone 8. I also learn the important of Winsock, how to port a library. I struggled a lot with porting openSSL to Windows RT, then to Windows Phone 8

A lot of links were broken 😢 so I can’t paste them all here.

• An Introduction to OpenSSL Programming

• SSL Socket support in Windows Phone

• Building Secure Windows Phone 8 Apps — APIs and Techniques

• Is Winsock available on Windows Phone 8?

• Why i am not been able to use all the header files (eg->ssl.h) under open source openssl library

• Getting Started with Winsock

• Building OpenSSL for Visual Studio

• How to Compile OpenSSL for Visual Studio 2010/2012

• OpenSSL for Windows

• OpenSSL for Windows RT

• 2911: enhancement request: Windows RT support

• Need a fast random generator for c++

• Window C/C++ Crypto API Examples and tips

• CryptGenRandom function

• EGD: The Entropy Gathering Daemon

C and C++

Since pjsip, rtpproxy and kamailio are all C and C++ code. I needed to have a good understanding about them, especially pointer and memory handling. We also needed to learn about compile flags for debug and release builds, how to use Make, how to make static and dynamic libraries.

• What does a const pointer-to-pointer mean in C and in C++?

• comp.lang.c Frequently Asked Questions: there’s lot of things about C we haven’t known about

• What is external linkage and internal linkage?

• Bit Twiddling Hacks: how to apply clever hacks with bit operators. Really really good reading here

• Better types in C++11 — nullptr, enum classes (strongly typed enumerations) and cstdint

• Microsoft Visual C++ Static and Dynamic Libraries

• Managed C++ — Learn by Example

• Preprocessor directives

❤️ Support my apps ❤️

• Push Hero - pure Swift native macOS application to test push notifications
• PastePal - Pasteboard, note and shortcut manager
• Quick Check - smart todo manager
• Alias - App and file shortcut manager
• My other apps

❤️❤️😇😍🤘❤️❤️