WebRTC Security

How can I make my WebRTC solution secure?

In one of my previous posts I have mentioned about Security threats to WebRTC Solution . It includes mainly 4 ways in which WebRTC Solution Providers and Users are vulnerable . It includes Identity Management , Browser Security , Authentication and Media encryption. Since I have already covered these topics here( https://altanaitelecom.wordpress.com/2014/10/03/security-for-webrtc-applications/ ) I will not repeat the same here. This post is about making WebRTC secure so that they can be used inn area which require sensitive data to be communicated and need to be secure enough to withstand and hacks and attacks.

In the recent months everyone has been trying to get into the WebRTC ” kool space ” but at the same time fearing that hackers might be able to listen in on conferences, access user data, or even private networks. Although development and usage around WebRTC is so simple , the security and encryption aspecets of it are in the dim light.


So does existing WebRTC model offer security ?

We know that the forces behind WebRTC standardization are WHATWG, W3C, IETF and strong internet working groups . WebRTC security was already taken into consideration when standards were being build for it . The encryption methods and technologies like DTLS and SRTP were included to safeguard users from intrusions so that the information stays protected.

WebRTC media stack has native built-in features that address security concerns. The peer-to-peer media is already encrypted for privacy . Figure below:

WebRTC media stack Solution Architecture - Google Slides (1)

WebRTC media stack

For WebRTC to transfer real time data, the data is first encrypted using the DTLS (Datagram Transport Layer Security) method. This is a protocol built into all the WebRTC supported browsers from the start (Chrome, Firefox and Opera). On a DTLS encrypted connection, eavesdropping and information tampering cannot take place.

Other than DTLS, WebRTC also encrypts video and audio data via the SRTP (Secure Real-Time Protocol) method ensuring that IP communications – your voice and video traffic – can not be heard or seen by unauthorized parties.

What is SRTP ?

The Secure Real-time Transport Protocol (or SRTP) defines a profile of RTP (Real-time Transport Protocol), intended to provide encryption, message authentication and integrity, and replay protection to the RTP data in both unicast and multicast applications.

Earlier models of VOIP communication such as SIP based calls had an option to use only RTP for communication thereby subjecting the endpoint users to lot of problem like compromising media Confidentiality  . However the WebRTC model mandates the use of SRTP hence ruling out insecurities of RTP completely. For encryption and decryption of the data flow SRTP utilizes the Advanced Encryption Standard (AES) as the default cipher.

What is DTLS ?

DTLS allows datagram-based applications to communicate in a way that is designed to prevent eavesdropping, tampering, or message forgery. The DTLS protocol is based on the stream-oriented Transport Layer Security (TLS) protocol .

Together DTLS and SRTP enables the exchange of the cryptographic parameters so that the key exchange takes place in the media plane and are multiplexed on the same ports as the media itself without the need to reveal crypto keys in the SDP.

Today the browser acts as a TRUSTED COMPUTING BASE (TCB) where the HTML and JS act inside of a sandbox that isolates them both from the user’s computer.

A script cannot access user’s webcam , microphone , location , file , desktop capture without user’s explicit consent. When the user allows access, a red dot will appear on that tab, providing a clear indication to the user, that the tab has media access.

Figure depicting browser asking for user’s consent to access Media devices for WebRTC .

Untitled drawing

Figure depicting Media Capture active on browser with red dot .

Untitled drawing (1)
we know that XMLHttpRequest() API can be used to secretly send data from one origin to other and this can be used to secretly send information without user’s knowledge . However now , SAME ORIGIN POLICY (SOP) in browser’s prevents server A from mounting attacks on server B via the user’s browser, which protects both the user (e.g., from misuse of his credentials) and the server B (e.g., from DoS attack).

What is SOP ?
SOP enables webpages and scripts from the same origin server to interact with each other’s JS variables, but prevents pages from the different origins or even iframes on the same page to not exchange information.

WebSockets use masking technique to randomize the bits that are being transmitted , thus making it more difficult to generate traffic which resembles a given protocol , thus making it difficult for inspection from flowing traffic .
In-spite of all this the security challenges with Web Server based WebRTC service are many for example :

1. If the both the peers have WebRTC browser then one can place a WebRTCcall to callee anytime this might result in denial of service .
2. Since the media is p2p and also can override firewalls settings through TURN server , it can result in unwanted data being send to peer .
3. One may secretly make calls to users through website and extract information .
4. Threat from screen sharing, for example user might mistakenly share his internet banking screen or some confidential information.
5. Giving long-term access to the camera and microphone for certain sites is also a concern . for example : since next time you visit a site that has access to your microphone and camera , they can secretly be viewing youe webcam and microphone inputs .
6. Clever use of User Interface to mask a ongoing call can mislead the user into believing that call has been cut while it is secretly still ongoing.
7. Network attackers can modify an HTTP connection through my Wifi router or hotspot to inject an IFRAME (or a redirect) and then forge the response to initiate a call to himself.
8. As WebRTC doesn’t yet have an congestion control mechanism , it can eat up a large chunk of user’s bandwidth.
9. By visiting chrome://webrtc-internals/ in chrome browser alone , one can view the full traces of all webRTC communication happening through his browser . The traces contain all kinds of details like signalling server used , relay servers , TURN servers , peer IP , frame rates etc .

WebRTC Internals

Ofcourse other challenges that arrive with any other webservice based architecture are also applicable here such as :

1. Malicious Websites which automatically execute the attacker’s scripts.
2. User can be induced to download harmful executable files and run them.
3. Improper use of W3C Cross-Origin Resource Sharing (CORS) to bypass SAME ORIGIN POLICY (SOP) .


Best practices to make your VOIP Solution more secure

A simple WebRTC architecture is shown in the figure below :

WebRTC media stack Solution Architecture - Google Slides (2)

By following the simple steps described below one can ensure a more secure WebRTC implementation . The same applies to healthcare and banking firms looking forth to use WebRTC as a communication solution for their portals .

1. Ensure that the signalling platform is over a secure protocol such as SIP / HTTPS / WSS .

2. User’s that can participate in a call , should be pre registered / Authenticated with a registrar service. Unauthenticated entities should be kept away from session’s reach .

WebRTC authentication certificate

WebRTC authentication certificate

2. Make sure that ICE values are masked thereby not rendering the caller/ callee’s IP and location to each other through tracing in chrome://webrtc-internals/ or packet detection in Wirehsark on user’s end.

3. Also since media is p2p , the media contents like audio video channel are between peers directly in full duplex. Thus

4. As the signalling server maintains the number of peers , it should be consistently monitored for addition of suspicious peers in a call session. If the number of peers actually present on signalling server is more that the number of peers interacting on WebRTC page then it means that someone is eavesdropping secretly and should be terminated from session access by force.

5. It is observed these days that users simply agree to all permissions request from browser without actually consciously giving consent . Therefore user’s should be made aware of API in websites which ask for undue permissions . For example permission to :

Screenshot from 2015-04-22 15:22:15

6. To protect against Man-In-The-Middle (MITM) attack the media path should be monitored regularly for no suspicious relay.

7. Third party API should be thoroughly verified before sending their data on WebRTC DataChannel.

8. Before Desktop Sharing user’s should be properly notified and advised to close any screen containing sensitive information .


What happens if your VOIP solution is on the verge of being compromised ?

As the media connections are p2p , even if we restart the signalling server , it will not affect the ongoing media sessions . Only the time duration ( probably 3 – 4 minutes ) it takes to restart the server , is when the users will not be able to connect to signalling server for creating new sessions .

Most browsers today like Google Chrome and Mozilla Firefox have a goof record of auto-updating themselves withing 24 hours of a vulnerability of threat occurring .

If a call is confirmed to be compromised , it should be within the power of Web Application server rendering the WebRTC capable page to cut off the call .


TURN server for WebRTC – RFC5766-TURN-Server , Coturn , Xirsys

STUN (Session Traversal Utilities for NAT) and TURN (Traversal Using Relays around NAT) are protocols that can be used to provide NAT traversal for VoIP and WebRTC. These projects provide a VoIP media traffic NAT traversal server and gateway.

TURN Server is a VoIP media traffic NAT traversal server and gateway.

I come accross the question of difference between turn and stun a lot . Here I wanted to specify in very clear words that TURN is an extension of STUN .

rfc5766-turn-server

This is a VoIP gateway for inter network communication which is popular and MIT based .

platforms supported :

Any client platform is supported, including Android, iOS, Linux, OS X, Windows, and Windows Phone. This project can be successfully used on other *NIX platforms ( Aamazon EC2) too. It supports flat file or Database based user management system ( MySQL , postgress , redis ). The source code project contains ,  TURN server ,  TURN client messaging library and some sample scripts to test various modules like protocol , relay , security etc .

Protocols :

protocols between the TURN client and the TURN server – UDP, TCP, TLS, and DTLS. Relay protocol – UDP , TCP .

Authentication

The authentication mechanism is using key which is calculated over the user name, the realm, and the user password. Key for the HMAC depends on whether long-term or short-term credentials are in use. For long-term credentials, the key is 16 bytes:
key = MD5(username “:” realm “:” SASLprep(password))

Installation

Since I used my Ubuntu Software center for installing the RFC turn server 5677 .

Screenshot from 2015-03-05 15:22:30

More information is on Ubuntu Manuals : http://manpages.ubuntu.com/manpages/trusty/man1/turnserver.1.html

The content got stored inside /usr/share/rfc5766-turn-server.

Also install mysql for record keeping

sudo apt-get install mysql-server

mysql

mysql2mysql4

Intall MySQL workbench to monitor the values feed into the turn database server in MysqL. connect to MySQL instance using the following screenshot

mysql5

The database formed with mysql after successful operation is as follows . We  shall notice that the initial db is absolutely null

mysql8empty

Terminal Commands

These terminal command ( binary images ) get stored inside etc/init.d after installing

turnadmin –

Its turn relay administration tool used for generating , updating keys and passwords . For generating a key to get long term crdentaial use -k command and for aading or updateing a long -term user use the -a command. Therefore a simple command to generate a key is

format : turnadmin -k -u <username> -r <realm> -p <password>
examples : turnadmin -k -u turnwebrtc -r mycompany.com -p turnwebrtc

The generated key is displayed in console . For example the following screenshot shows this :

rfc5677turnkey

To fill in user with long term credentails

Format : turnadmin -a [-b <userdb-file> | -e <db-connection-string> | -M <db-connection-string> | -N <db-connection-string> ] -u <username> -r <realm> -p <password>

exmaple : turnadmin -a -M “host=localhost dbname=turn user=turn password=turn” -u altanai -r mycompany.com -p 123456

Check the values reflected in MySQL workbench for long term user table . ( screenshot depicts two entries for altanai and turnwebrtc user )

turnkeylongterm

you can also check it on console using the -l command

format :turnadmin -l –mysql-userdb=”<db connection string >”

example :  turnadmin -l –mysql-userdb=”host=127.0.0.1 dbname=turn user=turnwebrtc password=turnwebrtc connect_timeout=30″

longtermuserlcommand

or we can also check using the terminal based mySQL client

mysql> use turn;
Database changed

mysql> select * from turnusers_lt;
+------------+----------------------------------+
| name | hmackey |
+------------+----------------------------------+
| altanai | 57bdc681481c4f7626bffcde292c85e7 |
| turnwebrtc | 6066cbe0b5ee14439b2ddfc177268309 |
+------------+----------------------------------+
2 rows in set (0.00 sec)

turnserver –

Its command to handle the turnserver itself . We can use the simple turnserver command to start it without any db support using just turnserver. Screenshot for this is

turnserverstart

We can use a database like mysql to start it with db connection string

Format : turnserver –mysql-userdb=”<db connection string >”

Example : turnserver –mysql-userdb=”host=127.0.0.1 dbname=turn user=turnwebrtc password=turnwebrtc connect_timeout=30″

turnservermysqldb

turnutils_uclient:

emulates multiple UDP,TCP,TLS or DTLS clients.

turnutils_peer:

simple stateless UDP-only “echo” server. For every incoming UDP packet, it simply echoes it back.

turnutils_stunclient:

simple STUN client example that implements RFC 5389 ( using STUN as endpoint to determine the IP address and port allocated to it , keep-alive , check connectivity etc) and RFC 5780 (experimental NAT Behavior Discovery STUN usage) .

turnutils_rfc5769check:

checks the correctness of the STUN/TURN protocol implementation. This program will perform several checks and print the result on the screen. It will exit with 0 status if everything is OK, and with (-1) if there was an error in the protocol implementation.

Specifications :

TURN specifications include :

  • RFC 5766 – base TURN specs
  • RFC 6062 – TCP relaying TURN extension
  • RFC 6156 – IPv6 extension for TURN
  • DTLS
  • Mobile ICE (MICE)

STUN specifications :

  • RFC 3489 – Simple Traversal of User Datagram Protocol (UDP) Through Network Address Translators (NATs) (STUN) to discover the presence and public IP
  • RFC 5389 – STUN serves as a tool for other protocols in NAT traversal. It can be used by an endpoint to determine the IP address and port allocated to it , keep-alive  , check connectivity etc .
  • RFC 5769 – test vectors for STUN protocol . FINGERPRINT, MESSAGE-INTEGRITY, and XOR-MAPPED-ADDRESS involving binary-logical operations (hashing, xor)
  • RFC 5780 – experimental NAT Behavior Discovery STUN usage

ICE specifications :

  • RFC 5245 – ICE
  • RFC 5768 – ICE–SIP
  • RFC 6336 – ICE–IANA Registry
  • RFC 6544 – ICE–TCP
  • RFC 5928 – TURN Resolution Mechanism

Test :

1. Test vectors from RFC 5769 to double-check that our
STUN/TURN message encoding algorithms work properly. Run the utility to check all protocols :

$ cd examples
$ ./scripts/rfc5769.sh

2. TURN functionality test (bare minimum TURN example).

If everything compiled properly, then the following programs must run
together successfully, simulating TURN network routing in local loopback
networking environment:

console 1 :

$ cd examples
$ ./scripts/basic/relay.sh

console2 :

$ cd examples
$ ./scripts/peer.sh

If the client application produces output and in approximately 22 seconds
prints the jitter, loss and round-trip-delay statistics, then everything is
fine.

Usage

iceServers:[
{ ‘url': ‘stun: altanai@mycompany.com’},
{ ‘url': ‘turn: altanai@mycompany.com’, ‘credential': ‘123456’}]

Insert the above piece of code on peer connection config .

Now call from one network environment to another . For example call from a enterprise network behind a Wifi router to a public internet datacard webrtc agent . The call should connect with video flowing smoothly between the two .

tooltips

website : https://code.google.com/p/rfc5766-turn-server/

Download the executable from : http://turnserver.open-sys.org/downloads/v3.2.5.4/

you can read about setting a carrier grade TURN infrastructure on amazon EC2 here –

coturn

Project Coturn evolved from rfc5766-turn-server project with many new advanced TURN specs beyond the original RFC 5766 document.
Here the databses supported are : SQLite , MySQL , PostgreSQL , Redis , MongoDB

Protocols :

The implementation fully supports the following client-to-TURN-server protocols: UDP  , TCP  , TLS  SSL3/TLS1.0/TLS1.1/TLS1.2; ECDHE , DTLS versions 1.0 and 1.2. Supported relay protocols UDP (per RFC 5766) and TCP (per RFC 6062)

Authetication :

Supported message integrity digest algorithms:

  • HMAC-SHA1, with MD5-hashed keys (as required by STUN and TURN standards)
  • HMAC-SHA256, with SHA256-hashed keys (an extension to the STUN and TURN specs)

Supported TURN authentication mechanisms:

Installation :

Install libopenssl and libevent plus its dev or extra libraries .
OpenSSL has to be installed before libevent2 for TLS beacuse When libevent builds it checks whether OpenSSL has been already installed, and its version.

Download coturn readonly  from

svn checkout http://coturn.googlecode.com/svn/trunk/ coturn-read-only

extract the tar contents
$ tar xvfz turnserver-<…>.tar.gz

go inside the extracted folder and run the following command to build
$ ./configure
$ make
$ make install

Adding users in the format using turnadmin
$ Sudo turnadmin -a -u <user name> -r <realm> -p <password>

Example
$ Sudo turnadmin -a -u altanai -r myserver.com -p 123456

Start the turn Server using turnserver from inside of /etc/init.d using the start command

$ sudo /etc/init.d/coturn start

Screenshot from 2015-01-06 12-08-15

The logs are usually stored in /var/log . Screenshot of log file

tuenlog2

The default configured port is 3478.If other port is needed, change the file /etc/turnserver.conf

Usuage:

Specify the  values in Peer Connection

Format:
iceServers: [
{ ‘url': ‘stun: <username>@<serverip>: <port>’},
{ ‘url': ‘turn: <username>@<serverip>: <port>’, ‘credential': ‘<password>’}]

example:

iceServers:[
{ ‘url': ‘stun: altanai@myserver.com’},
{ ‘url': ‘turn: altanai@myserver.com’, ‘credential': ‘123456’}]

Specifications:

TURN specs:

STUN specs:

  • RFC 3489 – STUN – Simple Traversal of User Datagram Protocol (UDP) Through Network Address Translators (NATs)
  • RFC 5389 – Session Traversal Utilities for NAT (STUN)
  • RFC 5769 – test vectors for STUN protocol testing
  • RFC 5780 – NAT behavior discovery support
  • RFC 7443 – Application-Layer Protocol Negotiation (ALPN) Labels for STUN and TURN

ICE :

  • RFC 5245 – ICE
  • RFC 5768 – ICE–SIP
  • RFC 6336 – ICE–IANA Registry
  • RFC 6544 – ICE–TCP
  • RFC 5928 – TURN Resolution Mechanism

website : https://code.google.com/p/coturn/

Xirsys

Xirsys is a provider for WebRTC infrastructure which included stun and turn server hosting as well .

The process of using their services includes singing up for a account and choosing whether you want a paid service capable of handling more calls simultaneously or free one handling only upto 10 concurrent turn connections .

The dashboard appears like this :

xirsys1

To receive the api one need to make a one time call to their service , the result of which contains the keys to invoke the turn services from webrtc script .


&lt;script src=&quot;http://ajax.googleapis.com/ajax/libs/jquery/2.1.0/jquery.min.js&quot;&gt;&lt;/script&gt;&lt;script&gt;// &lt;![CDATA[

$.post(&quot;https://api.xirsys.com/getIceServers&quot;, {
ident: &quot;altanai&quot;,
secret: &quot;&lt; your secret key &gt;&quot;,
domain: &quot; &lt; your doemain &gt;&quot;,
application: &quot;default&quot;,
room: &quot;default&quot;,
secure: 1
},
function(data, status) {
alert(&quot;Data: &quot; + data + &quot;n Status: &quot; + status);
console.log(&quot;Data: &quot; + data + &quot;nnStatus: &quot; + status);
});
});
&lt;/script&gt;

The resulting output should look like ( my keys are hidden with a red rectangle ofcourse )

xirsysedited

The process of adding a TURN / STUN to your webrtc script in JS is as follows :

iceServers:[
{“url”:”stun:turn2.xirsys.com”},
{“username”:”< put your API username>”,”url”:”turn:turn2.xirsys.com:443?transport=udp”,”credential”:”< put your API credentail>”},
{“username”:”< put your API username>”,”url”:”turn:turn2.xirsys.com:443?transport=tcp”,”credential”:”< put your API credentail>”}]

website : http://xirsys.com/technology/

NAT traversal using STUN and TURN

We know that WebRTC is web based real-time communications on browser-based platform using the browser’s media application programming interface (API) and adding our JavaScript & HTML5 t control the media flow .
WebRTC has enabled developers to build apps/ sites / widgets / plugins capable of delivering simultaneous voice/video/data/screen-sharing capability in a peer to peer fashion.

But womething which escapes our attention is the way in which media ia traversing accross the network. Ofcourse the webrtc call runs very smoothly when both the peers are on open public internet without any restructions or firewall blocks . But the real problem begins when one of the peer is behind a Corporate/Entreprise network or using a different Internet service provider with some security restrcitions . In such a case the normal ICE capability of WebRTC is not enough , what is required is a NAT traversal mechanism .

STUN and TURN server protocols handle session initiations with handshakes between peers in different network envrionments . In case of a firewall blocking a STUN peer-to-peer connection, the system fallbacks to a TURN server which provides the necessary traversing mechanism through the NAT.


Lets study from the start ie ICE . What is it and why is it used ?

ICE (Interactive Connectivity Establishment )  framework ( mandatory by WebRTC standards  ) find network interfaces and ports in Offer / Answer Model to exchange network based information with participating communication clients. ICE makes use of the Session Traversal Utilities for NAT (STUN) protocol and its extension, Traversal Using Relay NAT (TURN)

ICE is defined by RFC 5245 – Interactive Connectivity Establishment (ICE): A Protocol for Network Address Translator (NAT) Traversal for Offer/Answer Protocols.

Sample WebRTC offer holding ICE candidates :

type: offer, sdp: v=0
o=- 3475901263113717000 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE audio video data
a=msid-semantic: WMS dZdZMFQRNtY3unof7lTZBInzcRRylLakxtvc
m=audio 9 RTP/SAVPF 111 103 104 9 0 8 106 105 13 126
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:/v5dQj/qdvKXthQ2
a=ice-pwd:CvSEjVc1z6cMnhjrLlcbIxWK
a=ice-options:google-ice
a=fingerprint:sha-256 F1:A8:2E:71:4B:4E:FF:08:0F:18:13:1C:86:7B:FE:BA:BD:67:CF:B1:7F:19:87:33:6E:10:5C:17:42:0A:6C:15
a=setup:actpass
a=mid:audio
a=sendrecv
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:9 G722/8000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:126 telephone-event/8000
a=maxptime:60
m=video 9 RTP/SAVPF 100 116 117 96
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:/v5dQj/qdvKXthQ2
a=ice-pwd:CvSEjVc1z6cMnhjrLlcbIxWK
a=ice-options:google-ice
a=fingerprint:sha-256 F1:A8:2E:71:4B:4E:FF:08:0F:18:13:1C:86:7B:FE:BA:BD:67:CF:B1:7F:19:87:33:6E:10:5C:17:42:0A:6C:15
a=setup:actpass
a=mid:video
a=sendrecv
a=rtcp-mux
a=rtpmap:100 VP8/90000
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=rtcp-fb:100 goog-remb
a=rtpmap:116 red/90000
a=rtpmap:117 ulpfec/90000
a=rtpmap:96 rtx/90000
a=fmtp:96 apt=100
m=application 9 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
a=ice-ufrag:/v5dQj/qdvKXthQ2
a=ice-pwd:CvSEjVc1z6cMnhjrLlcbIxWK
a=ice-options:google-ice
a=fingerprint:sha-256 F1:A8:2E:71:4B:4E:FF:08:0F:18:13:1C:86:7B:FE:BA:BD:67:CF:B1:7F:19:87:33:6E:10:5C:17:42:0A:6C:15
a=setup:actpass
a=mid:data
a=sctpmap:5000 webrtc-datachannel 1024

Notice the ICE candidates under video and audio . Now take a look at the SDP answer

type: answer, sdp: v=0
o=- 6931590438150302967 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE audio video data
a=msid-semantic: WMS R98sfBPNQwC20y9HsDBt4to1hTFeP6S0UnsX
m=audio 1 RTP/SAVPF 111 103 104 0 8 106 105 13 126
c=IN IP4 0.0.0.0
a=rtcp:1 IN IP4 0.0.0.0
a=ice-ufrag:WM/FjMA1ClvNb8xm
a=ice-pwd:8yy1+7x0PoHZCSX2aOVZs2Oq
a=fingerprint:sha-256 7B:9A:A7:43:EC:17:BD:9B:49:E4:23:92:8E:48:E4:8C:9A:BE:85:D4:1D:D7:8B:0E:60:C2:AE:67:77:1D:62:70
a=setup:active
a=mid:audio
a=sendrecv
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=fmtp:111 minptime=10
a=rtpmap:103 ISAC/16000
a=rtpmap:104 ISAC/32000
a=rtpmap:0 PCMU/8000
a=rtpmap:8 PCMA/8000
a=rtpmap:106 CN/32000
a=rtpmap:105 CN/16000
a=rtpmap:13 CN/8000
a=rtpmap:126 telephone-event/8000
a=maxptime:60
m=video 1 RTP/SAVPF 100 116 117 96
c=IN IP4 0.0.0.0
a=rtcp:1 IN IP4 0.0.0.0
a=ice-ufrag:WM/FjMA1ClvNb8xm
a=ice-pwd:8yy1+7x0PoHZCSX2aOVZs2Oq
a=fingerprint:sha-256 7B:9A:A7:43:EC:17:BD:9B:49:E4:23:92:8E:48:E4:8C:9A:BE:85:D4:1D:D7:8B:0E:60:C2:AE:67:77:1D:62:70
a=setup:active
a=mid:video
a=sendrecv
a=rtcp-mux
a=rtpmap:100 VP8/90000
a=rtcp-fb:100 ccm fir
a=rtcp-fb:100 nack
a=rtcp-fb:100 nack pli
a=rtcp-fb:100 goog-remb
a=rtpmap:116 red/90000
a=rtpmap:117 ulpfec/90000
a=rtpmap:96 rtx/90000
a=fmtp:96 apt=100
m=application 1 DTLS/SCTP 5000
c=IN IP4 0.0.0.0
b=AS:30
a=ice-ufrag:WM/FjMA1ClvNb8xm
a=ice-pwd:8yy1+7x0PoHZCSX2aOVZs2Oq
a=fingerprint:sha-256 7B:9A:A7:43:EC:17:BD:9B:49:E4:23:92:8E:48:E4:8C:9A:BE:85:D4:1D:D7:8B:0E:60:C2:AE:67:77:1D:62:70
a=setup:active
a=mid:data
a=sctpmap:5000 webrtc-datachannel 1024

Call Flow for ICE

STUN call flow for WebRTC Offer Answer

STUN call flow for WebRTC Offer Answer


WebRTC needs SDP Offer to be send to the clientB Javascript code from clientA Javascript code . Client B uses this SDP offer to generate an SDP Answer for client A. The SDP ( as seen on chrome://webrtc-internals/ ) includes ICE candidates which punchs open ports in the firewalls.
However incase both sides are symmetric NATs the media flow gets blocked. For such a case TURN is used which tries to give a public ip and port mapped to internal ip and port so as to provide an alternative routing mechanism like a packet-mirror. It can open a DTLS connection and use it to key the SRTP-DTLS media streams, and to send DataChannels over DTLS.

In order to Understand this better consider various scenarios

1 . No Firewall present on either peer . Both connected to open public internet .

Diagrammatic representation of  this shown as follows :

WebRTC signalling and media flow on Open public network

WebRTC signalling and media flow on Open public network

In this case there is no restriction to signal or media flow and the call takes places smoothly in p2p fashion.

2.  Either one or both the peer ( could be many in case of multi conf call ) are present behind a firewall  or  restrictive connection or router configured for intranet .

In such a case the signal may pass with the use of default ICE candidates or simple ppensource google Stun server such as

iceServers:[
{ ‘url': “stun:stun.l.google.com:19302″}]

Diagram :

WebRTC signalling when peers are behind  firewalls

WebRTC signalling when peers are behind firewalls

However the media is restricted resulting in a black / empty / no video situation for both peers  . To combat such situation a relay mechanism such as TURN is required which essentially maps public ip to private ips thus creating a alternative route for media and data to flow through .

WebRTC media flow when peers are behind NAT . Uses TURN relay mechanism

WebRTC media flow when peers are behind NAT . Uses TURN relay mechanism

Peer config should look like :

var configuration =  {
iceServers: [
{ “url':”stun:<username>:<domain>”},
{ “url”:”turn:<username>:<domain>”}
]
};

var pc = new RTCPeerConnection(configuration);

3. When the TURN server is also behind a firewall .  The config file of the turn server need to be altered to map the public and private IP

The diagrammatic description of this is as follows :

WebRTC media flow when peers are behind NAT and TURN server is behind NAT as well . TURN config files bind a public interface to private interface address.

WebRTC media flow when peers are behind NAT and TURN server is behind NAT as well . TURN config files bind a public interface to private interface address .


continue : Streaming / broadcasting Live Video call to non webrtc supported browsers and media players

This blog is in continuation to the attempts / outcomes and problems in building a WebRTC to RTP media framework that successfully stream / broadcast WebRTC content to non webrtc supported browsers ( safari / IE ) / media players ( VLC )


Attempt 4: Stream the content to a WebRTC endpoint which is hidden in a video call . Pick the stream from vp8 object URL send to a streaming server

This process involved the following components :

  • WebRTC API : simplewebrtc on Chrome
  • Transfer mechanism from client to Streaming server:  webrtc media channel

Problems : No streaming server is qualified to handle a direct webrtc input and stream it on network .


Attempt 4.1 : Stream the content to a WebRTC endpoint . Do WebRTC Endpoint to RTP Endpoint bridge using Kurento APIs. 

Use the RTP port and ip address to input into a ffmpeg or gstreamer or VLC terminal command and out put a live H264 stream on another ip and port address .  

This process involved the following components :

  • API : Kurento
  • Transfer mechanism : HTML5 webrtc client -> application server hosting java -> media server -> application for webrtc media to RTP media conversation -> RTP player

Screenshots of attempts with Wowza to stream from a ip and port

kurentowowoza

problems :

  • The stream was black ie no video content .

Attempt 4.2 : Build a WebRTC Endpoint to Http endpoint in kurento and force the video audio encoding to be that of H264 and PCMU.

code for adding constraints to output media and forcing choice of codecs

MediaPipeline pipeline = kurento.createMediaPipeline();
    WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
    HttpGetEndpoint httpEndpoint=new HttpGetEndpoint.Builder(pipeline).build();

    org.kurento.client.Fraction fr= new org.kurento.client.Fraction(1, 30);         
    VideoCaps vc= new VideoCaps(VideoCodec.H264,fr);
    httpEndpoint.setVideoFormat(vc);

    AudioCaps ac= new AudioCaps(AudioCodec.PCMU, 65536);
    httpEndpoint.setAudioFormat(ac);

    webRtcEndpoint.connect(httpEndpoint);

code for using gstreamer filter to force the output in raw format . It is a alternate solution to above

//basic media operation of 1 pipeline and 2 endpoinst
MediaPipeline pipeline = kurento.createMediaPipeline();
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
RtpEndpoint rtpEndpoint = new RtpEndpoint.Builder(pipeline).build();

//adding Gstream filters 
GStreamerFilter filter1 = new GStreamerFilter.Builder(pipeline, &quot;videorate max-rate=30&quot;).withFilterType(FilterType.VIDEO).build();
GStreamerFilter filter2 = new GStreamerFilter.Builder(pipeline, &quot;capsfilter caps=video/x-h264,width=1280,height=720,framerate=30/1&quot;).withFilterType(FilterType.VIDEO).build();
GStreamerFilter filter3 = new GStreamerFilter.Builder(pipeline, &quot;capsfilter caps=audio/x-mpeg,layer=3,rate=48000&quot;).withFilterType(FilterType.AUDIO).build();

//connecting all poin ts to one another 
webRtcEndpoint.connect (filter1); 
filter1.connect (filter2); 
filter2.connect (filter3); 
filter3.connect (rtpEndpoint);

// RTP SDP offer and answer
String requestRTPsdp = rtpEndpoint.generateOffer();
rtpEndpoint.processAnswer(requestRTPsdp);

problem : The output is still webm


Attempt 5  : Use a RTP SDP Endpoint ( ie a SDP file valid for a given session ) and use it to play the WebRTC media over Wowza streaming server

This process involved the following components

  1. WebRTC Stream and object URL of the blob containing VP8 media
  2. Kurento  WebRTC Endpoint  bridge to generate SDP
  3. Wowza Streaming server

code for kurento to generate a SDP file from WebRTC to RTP bridge

@RequestMapping(value = &quot;/rtpsdp&quot;, method = RequestMethod.POST)
private String processRequestrtpsdp(@RequestBody String sdpOffer)
throws IOException, URISyntaxException, InterruptedException {

//basic media operation of 1 pipeline and 2 endpoinst
MediaPipeline pipeline = kurento.createMediaPipeline();
WebRtcEndpoint webRtcEndpoint = new WebRtcEndpoint.Builder(pipeline).build();
RtpEndpoint rtpEndpoint = new RtpEndpoint.Builder(pipeline).build();

//connecting all poin ts to one another 
webRtcEndpoint.connect (rtpEndpoint);

// RTP SDP offer and answer
String requestRTPsdp = rtpEndpoint.generateOffer();
rtpEndpoint.processAnswer(requestRTPsdp);

// write the SDP conector to an external file
PrintWriter out = new PrintWriter(&quot;/tmp/test.sdp&quot;);
out.println(requestRTPsdp);
out.close();

HttpGetEndpoint httpEndpoint = new HttpGetEndpoint.Builder(pipeline).build();
PlayerEndpoint player = new PlayerEndpoint.Builder(pipeline, requestRTPsdp).build();
httpEndpoint.connect(rtpEndpoint);
player.connect(httpEndpoint);

// Playing media and opening the default desktop browser
player.play();
String videoUrl = httpEndpoint.getUrl();
System.out.println(&quot; ------- video URL -------------&quot;+ videoUrl);

// send the response to front client
String responseSdp = webRtcEndpoint.processOffer(sdpOffer);

return responseSdp;
}

problems : wowza doesnt not recognize the WebRTC SDP and play the video

screenshot of wowza with SDP input

Screenshot from 2015-01-30 15:28:59


Attempt 5.1 : Use a RTP SDP Endpoint ( ie a SDP file valid for a given session ) and use it to play the WebRTC media over Default Ubuntu media player 

SDP file formed contains contents such as :

v=0
o=- 3631611195 3631611195 IN IP4 192.168.0.119
s=Kurento Media Server
c=IN IP4 192.168.0.119
t=0 0
m=audio 42802 RTP/AVP 98 99 0
a=rtpmap:98 OPUS/48000/2
a=rtpmap:99 AMR/8000/1
a=rtpmap:0 PCMU/8000
a=ssrc:2713728673 cname:user59375791@host-ad1117df
m=video 35946 RTP/AVP 96 97 100 101
a=rtpmap:96 H263-1998/90000
a=rtpmap:97 VP8/90000
a=rtpmap:100 MP4V-ES/90000
a=rtpmap:101 H264/90000
a=ssrc:93449274 cname:user59375791@host-ad1117df

problem : deformed media

screenshot of playing from a SDP file

Screenshot from 2015-01-29 17:42:21


Attempt 5.2 : Use a RTP SDP Endpoint ( ie a SDP file valid for a given session ) and use it to play the WebRTC media over VLC using socket input

problem : nothing plays

screenshot of VLC connected to play from socket and failure to play anything

Screenshot from 2015-01-21 17:49:52

Attempt 5.3: Create a WebRTC endpoint and connected it to RTP endpoint via media pipelines . Also make the RTP SDP offer and answering the same . Play with ffnpeg / ffplay / gst playbin

String requestRTPsdp = rtpEndpoint.generateOffer();
rtpEndpoint.processAnswer(requestRTPsdp);

Write the requestRTPsdp to a file and obtain a RTP connector endpoint with Application/SDP .It plays okay with gst playbin ( 10 secs without audio )

Successful attempt to play from a gst playbin

gst-launch -vvv playbin uri=file:///tmp/test.sdp 

donekurento streaming

but refuses to be played by VLC , ffplay and even wowza . The error generated with

ffmpeg -i test.sdp -vcodec copy -acodec copy -f mpegts output-file.ts

or

ffmpeg -re -i test.sdp -vcodec h264 -acodec mp3 -f mpegts “udp://192.168.4.26:5000″

are

Could not find codec parameter for stream1 ( video:h263, none ) .Other errors types are , Could not write header for output file <incorrect codec parameter > output file is empty nothing was encoded

Error screenshots of trying to play the RTP SDP file with ffmpeg

ffmpeg error kurebto1 ffmpeg error kurebto2


Attempt 6 : Use a WebRTC capable media and streaming server ( eg Kurento )  to pick a live stream of VP8 . Convert the VP8 to H268  ( ffmpeg / RTP endpoint ) . Convert H268 to Mp4 using MP4 parser and pass to a streaming server  ( wowza)

In process . to be updated .

Streaming / broadcasting Live Video call to non webrtc supported browsers and media players

As the title of this article suggests I am going to pen my attempts of streaming / broadcasting Live Video WebRTC call to non WebRTC supported browsers and media players such as VLC , ffplay , default video player in Linux etc .

I am currently attempting to do this by making my own MP4 engine from WebRTC feed . However I am sharing my past experiments in hope of helping someone whose objective is not the same as mine and might get some help from these threads .


Attempt 1 : use one to many brodcasting API :

&lt;!DOCTYPE html&gt;
&lt;html id=”home” lang=”en”&gt;

&lt;head&gt;
&lt;meta http-equiv=”Content-Type” content=”text/html; charset=UTF-8″&gt;
&lt;meta charset=utf-8&gt;
&lt;meta name=”viewport” content=”width=device-width, initial-scale=1.0, user-scalable=no”&gt;
&lt;meta name=”author” content=”altanai”&gt;
&lt;meta http-equiv=”X-UA-Compatible” content=”IE=edge,chrome=1″&gt;

&lt;link rel=”stylesheet” type=”text/css” href=”style.css”&gt;

&lt;script&gt;
document.createElement(‘article’);
&lt;/script&gt;
&lt;/head&gt;

&lt;body&gt;
&lt;article&gt;

&lt;table class=”visible”&gt;
&lt;tr&gt;
&lt;td style=”text-align: right;”&gt;
&lt;input type=”text” id=”conference-name” placeholder=”Broadcast Name”&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;select id=”broadcasting-option”&gt;
&lt;option&gt;Audio + Video&lt;/option&gt;
&lt;option&gt;Only Audio&lt;/option&gt;
&lt;option&gt;Screen&lt;/option&gt;
&lt;/select&gt;
&lt;/td&gt;
&lt;td&gt;
&lt;button id=”start-conferencing”&gt;Start Broadcasting&lt;/button&gt;
&lt;/td&gt;
&lt;/tr&gt;
&lt;/table&gt;
&lt;table id=”rooms-list” class=”visible”&gt;&lt;/table&gt;

&lt;div id=”participants”&gt;&lt;/div&gt;

&lt;script src=”RTCPeerConnection-v1.5.js”&gt;&lt;/script&gt;
&lt;script src=”firebase.js”&gt;&lt;/script&gt;
&lt;script src=”broadcast.js”&gt;&lt;/script&gt;
&lt;script src=”broadcast-ui.js”&gt;&lt;/script&gt;

&lt;/article&gt;

&lt;/body&gt;

&lt;/html&gt;
 

It uses API fromwebrtc-experiment.com. The broadcast is in one direction only where the viewrs are never asked for their mic / webcam permission .

problem : The broadcast is for WebRTC browsers only and doesnt support non webrtc players / browsers


Attempt 1.1: Stream the media directly to nodejs through websocket


window.addEventListener('DOMContentLoaded', function() {

var v = document.getElementById('v');
navigator.getUserMedia = (navigator.getUserMedia || 
navigator.webkitGetUserMedia || 
navigator.mozGetUserMedia || 
navigator.msGetUserMedia);

if (navigator.getUserMedia) {
// Request access to video only
navigator.getUserMedia(
{
video:true,
audio:false
}, 
function(stream) {
var url = window.URL || window.webkitURL;
v.src = url ? url.createObjectURL(stream) : stream;
v.play();

var ws = new WebSocket('ws://localhost:3000', 'echo-protocol');
waitForSocketConnection(ws, function(){

console.log(&quot; url.createObjectURL(stream)-----&quot;, url.createObjectURL(stream))
ws.send(stream);

console.log(&quot;message sent!!!&quot;); 
});

},
function(error) {
alert('Something went wrong. (error code ' + error.code + ')');
return;
}
);
}
else {
alert('Sorry, the browser you are using doesn\'t support getUserMedia');
return;
}
});

//Make the function wait until the connection is made...
function waitForSocketConnection(socket, callback){
setTimeout(
function () {
if (socket.readyState === 1) {
console.log(&quot;Connection is made&quot;)
if(callback != null){
callback();
}
return;

} else {
console.log(&quot;wait for connection...&quot;)
waitForSocketConnection(socket, callback);
}

}, 5); // wait 5 milisecond for the connection...
}

problem : The video is in form of buffer and doesnot play


Attempt 2: Record the WebRTC media ( 5 secs each ) into chunks of webm format->  transfer them to other end -> append the chunks together like a regular file 

This process involved the following components :

  • Recorder Javascript library : RecordJs
  • Transfer mechanism : Record using RecordRTC.js -> send to other end for media server -> stitching together the small webm files into big one at runtime and play
  • Programs :

Code for video recorder

navigator.getUserMedia(videoConstraints, function(stream) {

video.onloadedmetadata = function() {
video.width = 320;
video.height = 240;

var options = {
type: isRecordVideo ? 'video' : 'gif',
video: video,
canvas: {
width: canvasWidth_input.value,
height: canvasHeight_input.value
}
};

recorder = window.RecordRTC(stream, options);
recorder.startRecording();
};
video.src = URL.createObjectURL(stream);
}, function() {
if (document.getElementById('record-screen').checked) {
if (location.protocol === 'http:')
alert('&lt;https&gt; is mandatory to capture screen.');
else
alert('Multi-capturing of screen is not allowed. Capturing process is denied. Are you enabled flag: &quot;Enable screen capture support in getUserMedia&quot;?');
} else
alert('Webcam access is denied.');
});

Code for video append-er

var FILE1 = '1.webm';
var FILE2 = '2.webm';
var FILE3 = '3.webm';
var FILE4 = '4.webm';
var FILE5 = '5.webm';

var NUM_CHUNKS = 5;
var video = document.querySelector('video');

window.MediaSource = window.MediaSource || window.WebKitMediaSource;
if (!!!window.MediaSource) {
alert('MediaSource API is not available');
}

var mediaSource = new MediaSource();

video.src = window.URL.createObjectURL(mediaSource);

function callback(e) {

var sourceBuffer = mediaSource.addSourceBuffer('video/webm; codecs=&quot;vorbis,vp8&quot;');

GET(FILE1, function(uInt8Array) {

var file = new Blob([uInt8Array], {type: 'video/webm'});
var i = 1;

(function readChunk_(i) {

var reader = new FileReader();

reader.onload = function(e) {

sourceBuffer.appendBuffer(new Uint8Array(e.target.result));

if (i == NUM_CHUNKS) mediaSource.endOfStream();

else {
if (video.paused) {
video.play(); // Start playing after 1st chunk is appended.
}
readChunk_(++i);
}

};

reader.readAsArrayBuffer(file);

})(i); // Start the recursive call by self calling.
});
}

mediaSource.addEventListener('sourceopen', callback, false);
mediaSource.addEventListener('webkitsourceopen', callback, false);
mediaSource.addEventListener('webkitsourceended', function(e) {
logger.log('mediaSource readyState: ' + this.readyState);
}, false);

// function get the video via XHR
function GET(url, callback) {

var xhr = new XMLHttpRequest();
xhr.open('GET', url, true);
xhr.responseType = 'arraybuffer';
xhr.send();

xhr.onload = function(e) {

if (xhr.status != 200) {
alert(&quot;Unexpected status code &quot; + xhr.status + &quot; for &quot; + url);
return false;
}

callback(new Uint8Array(xhr.response));
};
}

Shortcoming of this approach

  1. The webm files failed to play on most of the media players
  2. The recorder can only either record video or audio file at a time .

Attempt 2.1: Record the WebRTC media ( 5 secs each ) into chunks of webm format ( RecordRTC.js) >  Use Kurento JS script ( kws-media-api,js) to make a HTTP Endpoint to recorded Webm files  -> append the chunks together like a regular file at runtime 


function getByID(id) {
return document.getElementById(id);
}

var recordAudio = getByID('record-audio'),
recordVideo = getByID('record-video'),
stopRecordingAudio = getByID('stop-recording-audio'),
stopRecordingVideo = getByID('stop-recording-video'),
broadcasting=getByID('broadcasting');

var canvasWidth_input = getByID('canvas-width-input'),
canvasHeight_input = getByID('canvas-height-input');

var video = getByID('video');
var audio = getByID('audio');

var videoConstraints = {
audio: false,
video: {
mandatory: {},
optional: []
}
};

var audioConstraints = {
audio: true,
video: false
};

const ws_uri = 'ws://localhost:8888/kurento';
var URL_SMALL=&quot;http://localhost:8080/streamtomp4/approach1/5561840332.webm&quot;;


var audioStream;
var recorder;

recordAudio.onclick = function() {
if (!audioStream)
navigator.getUserMedia(audioConstraints, function(stream) {

if (window.IsChrome) stream = new window.MediaStream(stream.getAudioTracks());
audioStream = stream;

audio.src = URL.createObjectURL(audioStream);
audio.muted = true;
audio.play();

// &quot;audio&quot; is a default type
recorder = window.RecordRTC(stream, {
type: 'audio'
});
recorder.startRecording();
}, function() {});
else {
audio.src = URL.createObjectURL(audioStream);
audio.muted = true;
audio.play();
if (recorder) recorder.startRecording();
}


window.isAudio = true;

this.disabled = true;
stopRecordingAudio.disabled = false;
};

stopRecordingAudio.onclick = function() {
this.disabled = true;
recordAudio.disabled = false;
audio.src = '';

if (recorder)
recorder.stopRecording(function(url) {
audio.src = url;
audio.muted = false;
audio.play();

document.getElementById('audio-url-preview').innerHTML = '&lt;a href=&quot;' + url + '&quot; target=&quot;_blank&quot;&gt;Recorded Audio URL&lt;/a&gt;';
});
};

recordVideo.onclick = function() {
recordVideoOrGIF(true);
};


function recordVideoOrGIF(isRecordVideo) {
navigator.getUserMedia(videoConstraints, function(stream) {

video.onloadedmetadata = function() {
video.width = 320;
video.height = 240;

var options = {
type: isRecordVideo ? 'video' : 'gif',
video: video,
canvas: {
width: canvasWidth_input.value,
height: canvasHeight_input.value
}
};

recorder = window.RecordRTC(stream, options);
recorder.startRecording();
};
video.src = URL.createObjectURL(stream);
}, function() {
if (document.getElementById('record-screen').checked) {
if (location.protocol === 'http:')
alert('&lt;https&gt; is mandatory to capture screen.');
else
alert('Multi-capturing of screen is not allowed. Capturing process is denied. Are you enabled flag: &quot;Enable screen capture support in getUserMedia&quot;?');
} else
alert('Webcam access is denied.');
});

window.isAudio = false;

if (isRecordVideo) {
recordVideo.disabled = true;
stopRecordingVideo.disabled = false;
} else {
recordGIF.disabled = true;
stopRecordingGIF.disabled = false;
}
}

stopRecordingVideo.onclick = function() {
this.disabled = true;
recordVideo.disabled = false;

if (recorder)
recorder.stopRecording(function(url) {
video.src = url;
video.play();
document.getElementById('video-url-preview').innerHTML = '&lt;a href=&quot;' + url + '&quot; target=&quot;_blank&quot;&gt;Recorded Video URL&lt;/a&gt;';

});
};


/*--------------------------broadcasting -----------------------------------*/

function onerror(error)
{
console.log( &quot; error occured&quot;);
console.error(error);
};

broadcast.onclick = function() {
var videoOutput = document.getElementById(&quot;videoOutput&quot;);

KwsMedia(ws_uri, function(error, kwsMedia)
{
if(error) return onerror(error);

// Create pipeline
kwsMedia.create('MediaPipeline', function(error, pipeline)
{
if(error) return onerror(error);

// Create pipeline media elements (endpoints &amp; filters)
pipeline.create('PlayerEndpoint', {uri: URL_SMALL},
function(error, player)
{
if(error) return console.error(error);

pipeline.create('HttpGetEndpoint', function(error, httpGet)
{
if(error) return onerror(error);

// Connect media element between them
player.connect(httpGet, function(error, pipeline)
{
if(error) return onerror(error);
// Set the video on the video tag
httpGet.getUrl(function(error, url)
{
if(error) return onerror(error);

videoOutput.src = url;

console.log(url);

// Start player
player.play(function(error)
{
if(error) return onerror(error);

console.log('player.play');
});
});
});

// Subscribe to HttpGetEndpoint EOS event
httpGet.on('EndOfStream', function(event)
{
console.log(&quot;EndOfStream event:&quot;, event);
});
});
});
});
},
onerror);

}

problem : dissecting the live video into small the files and appending to each other on reception is an expensive , time and resource consuming process . Also involves heavy buffering and other problems pertaining to real-time streaming .


Attempt 2.2 : Send the recorded chunks of webm to a port on linux server . Use socket programming to pick up these individual files and play using  VLC player from UDP port of the Linux Server

Screenshot from 2015-01-22 15:32:51


Attempt 2.3: Send the recorded chunks of webm to a port on linux server socket . Use socket programming to pick up these individual webm files and convert to H264 format so that they can be send to a media server. 

This process involved the following components :

  • Recorder Javascript library : RecordJs
  • Transfer mechanism :WebRTC endpoint -> Call handler ( Record in chunks ) -> ffmpeg / gstreamer to put it on RTP -> streaming server like wowza – > viewers
  • Programs : Use HTML webpage Webscoket connection -> nodejs program to write content from websocket to linux socket -> nodejs program to read that socket and print the content on console

Program to transfer the webm recorder files over websocket to nodejs program

//Make the function wait until the connection is made...
function waitForSocketConnection(socket, callback){
setTimeout(
function () {
if (socket.readyState === 1) {
console.log(&quot;Connection is made&quot;)
if(callback != null){
callback();
}
return;

} else {
console.log(&quot;wait for connection...&quot;)
waitForSocketConnection(socket, callback);
}

}, 5); // wait 5 milisecond for the connection...
}

function previewFile() {
var preview = document.querySelector('img');
var file = document.querySelector('input[type=file]').files[0];
var reader = new FileReader();

reader.onloadend = function () {

preview.src = reader.result;
console.log(&quot; reader result &quot;, reader.result);

var video=document.getElementById(&quot;v&quot;);
video.src=reader.result;
console.log(&quot; video played &quot;);

var ws = new WebSocket('ws://localhost:3000', 'echo-protocol');

waitForSocketConnection(ws, function(){
ws.send(reader.result); 
console.log(&quot;message sent!!!&quot;); 
});

}

if (file) {
// converts to base64 encoded string of the file data
//reader.readAsDataURL(file);

reader.readAsBinaryString(file);

} else {
preview.src = &quot;&quot;;
}
}

Program for Linux Sockets sender which creates the socket for the webm files

var net = require('net');
var fs = require('fs');
var socketPath = '/tmp/tfxsocket';
var http = require('http');
var stream = require('stream');
var util = require('util');

var WebSocketServer = require('ws').Server;
var port = 3000;
var serverUrl = &quot;localhost&quot;;

var socket;
/*--------------------------------http server -----------------------------*/
var server= http.createServer(function (request, response) {

});

server.listen(port, serverUrl);

console.log('HTTP Server running at ',serverUrl,port);

/*--------------------------------websocket server -----------------------------*/

var wss = new WebSocketServer({server: server});

wss.on(&quot;connection&quot;, function(ws) {
console.log(&quot;websocket connection open&quot;);

ws.on('message', function (message) {
console.log(&quot; stream recived from broadcast client on port 3000 &quot;);

var s = require('net').Socket();
s.connect(socketPath);
s.write(message);

console.log(&quot; send the stream to socketPath&quot;,socketPath); 
});

ws.on(&quot;close&quot;, function() {
console.log(&quot;websocket connection close&quot;)
});

});

Program for Linux Socket Listener using nodejs and socket . Here the socket is in node /tmp/mysocket

var net = require('net');

var client = net.createConnection(&quot;/tmp/mysocket&quot;);

client.on(&quot;connect&quot;, function() {
console.log(&quot;connected to mysocket&quot;);
});

client.on(&quot;data&quot;, function(data) {
console.log(data);
});

client.on('end', function() {
console.log('server disconnected');
});

Output 1: Video Buffer displayed

Screenshot from 2015-01-22 15:35:06 (copy)

Output 2 : Random data from Video displayed

Screenshot from 2015-01-23 12:57:35

ffmpeg format of transfering the content from socket to UDP IP and port

ffmpeg -i unix://tmp/mysocket -f format udp://192.168.0.119:8083

problems of this approach : The video was on a passing stage from the socket and contained no information as such when tried to play / show console


Attempt 3 : Send the live WebRTC stream from Kurento WebRTC endpoint to Kurento HTTP endpoint . play using  Mozilla VLC web plugin

VLC mozilla plugin can be embedded by :


name=”video2″
autoplay=”yes” loop=”no” hidden=”no”
target=”rtp://@192.165.0.119:8086″ />

screenshot of failure on part of Mozilla VLC plugin to play from a WebRTC endpoint

Screenshot from 2015-01-29 10:37:06Screenshot from 2015-01-29 10:37:17

Screenshot from 2015-01-29 12:06:14

problem : VLC mozilla plugin was unable to play the video

………………………………………………………………………………………………………………..

The 4th , 5th and 6th sections of this article are in the next blog :

continue : Streaming / broadcasting Live Video call to non webrtc supported browsers and media players

Building WebRTC API , platform and deploying as a browser component

So I haven’t written anything worthy in a while , just published some posts that were lying around in my drafts . Here I write about the main thing . some thing awesome that I was trying to accomplish in the last quarter .

TFX platform

<< TFX is now live in chrome store , open and free for public use . No signin or account required , no adds : https://chrome.google.com/webstore/detail/tfx-sessions/aochimdcllmgleokpnlabijehdlmkdga >>

TFX Sessions is a plug and play platform for VoIP ( voice over IP ) scenarios.  Intrinsically it  is a very lightweight API package and shipped in form of a Chrome Extension . It is a turn-key solution when parties want instant audio/audio communication without any sign-in ,plugin installation or additional downloads  . Additionally TFX Sessions is packaged with some interesting plugins which enable the communicating parties to get the interactive and immersive experience as in a face to face meeting.

So here is the thing . I have been thinking of making  a utterly simple WebRTC API  that has everything needed to build bigger aggregate projects but the available solutions are either just to basic or much too complex . So I initially stardted writing my own getuserMedia APIs, but left it midway and picked up simplewebrtc API instead for want of time .Then I focused on the main crux  which was my plugin/ widget/ applications based platform .

TFX WebRTC platform architecture . socket io signalling

TFX WebRTC platform architecture . socket io signalling

Note that when I say plugin it isnt actually a plugin for chrome but for my TFX platform alone . Essentially it is any web project that wants communication over webrtc channel .

So now the platform is ready I have core APIs , widgets and signalling server. Then ofcourse came up the subject of enterprise internet blocking my communication stream . Time for TURN . I used a coturn server which mingled with my platform just as water dissolves coffee :) .

So here is the final architecture .

TFX whitepaper v2.0

TFX platform Server client components . WebRTC media and socketio communication . Build as chrome Extension

Alright so that’s there . Tada the platform is alive and kicking . Right now in beta stage however . Intensive testing going on here . However here are some screenshots that are from my own developer version .

TFX recording widget

TFX recording widget

TFX face detection and overlay widget

TFX face detection and overlay widget

TFX multilingual communication

TFX multilingual communication

TFX screen-sharing

TFX screen-sharing

TFX video Filters

TFX video Filters

TFX audio visualizer

TFX audio visualizer

TFX text messaging widget

TFX text messaging widget

TFX cross domian access . flicker here

TFX cross domain access . flicker here

TFX draw widget

TFX draw widget

TFX code widget supportes many programming languages

TFX code widget supportes many programming languages

TFX  webrtc dynamic stats

TFX webrtc dynamic stats

TFX  introduction widget

TFX introduction widget

please note that the plugins . widgets / applications described above have been made with the help of third party APIs. The detailed summary of every widget and its procedure , development , setup is to be described in my next blog .

TFX startup screen

TFX startup screen

TFX create / join room

TFX create / join room

My objective behind all this work was to come up with a modular solution which can exists as a API , platform as a service  , simple browser extension and web project .

SIP messages

1. Request Message

Request Message

Description

REGISTER A Client use this message to register an address with a SIP server
INVITE A User or Service use this message to let another user/service participate in a session. The body of this message would include a description of the session to which the callee is being invited.
ACK This is used only for INVITE indicating that the client has received a final response to an INVITE request
CANCEL This is used to cancel a pending request
BYE A User Agent Client use this message to terminate the call
OPTIONS This is used to query a server about its capabilities

 

2. Response Message

Code

Category

Description

1xx Provisional The request has been received and processing is continuing
2xx Success An ACK, to indicate that the action was successfully received, understood, and accepted.
3xx Redirection Further action is required to process this request
4xx Client Error The request contains bad syntax and cannot be fulfilled at this server
5xx Server Error The server failed to fulfill an apparently valid request
6xx Global Failure The request cannot be fulfilled at any server 

, based on RFC 3261

Sip server Brekeke

We used Brekeke SIP server to run our SIP applications . Although there are newer versions of Brekeke SIP server out now . More awesome than before , we prefer using the old one for the sake of not messing with legacy SIP applications . The official site for brekeke is – http://www.brekeke.com/sip/ .

A general architecture of Brekeke SIP server is . brekeke

Here are the steps of installing and configuring a Brekeke SIP server .

Step 1: Download the Server form http://www.brekeke.com/sip/ and run the setup file .

brekeke0

brekeke01

brekeke1

brekeke8

brekek2 brekek3 brekek6

brekeke4 brekeke5 brekeke7  brekeke9

Step 2: It is always good to give a look to README file . brekeke11

Step 3: Run the local implementation of SIP server at localhost or 127.0.0.1 at port 8080brekeke12

Step 4: Important is to get the license which will help us activate the SIP server . One can obtain a free license from http://www.brekeke.com/downloads/sip-server-trial-license.phpbrekeke12_001

Step 5 : Once the license is activates , we can goto the console screen after loggin with default username and password sa . brekeke13

Step 6 : Once we are at console , we could add/ delete / modify parameters like port , start/shutdown status etc . brekeke14 brekeke14_001Step 7 : Once the server is all setup , just add the IP and port of SIP server to SIP clients server filed . Now all the SIP request and response will be catered by this SIP Server

Steps for building and deploying WebRTC solution

Step 1  : Do it myself 

Pick any WebRTC API and run its demos . It works kool . download and run in local-machine with nodejs server . Awesome . Everything is Awesome !!

You can learn more about some WS based WebRTC API here:  https://altanaitelecom.wordpress.com/2014/12/02/current-state-of-webrtc/.

If you are a diehard telecom engineer and only want SIP based WebRTC solutions go here : https://altanaitelecom.wordpress.com/2014/07/16/interoperability-between-webrtc-sip-phones-and-others/

step 1 of building and deploying a WebRTC solution

Step 2 : Give it to friends 

Now what good is it doing to anyone if its running locally on my machine with addresses like localhost and 127.0.0.1  . Let us put it on the cloud and atleast let my colleague / friends enjoy it .  Now I need two things

1. Cloud Web Server and Nodejs signalling server . That is okay use amazon’s Ec2. works for most of the people most of the time .

2. STUN server for address mapping and NAT . For this I have to rely on google’s default STUN server stun.l.google.com:19302. Easy and free .

step 2 : building and deploying a WebRTC solution

Note that this step only works if everyone you want to connect to is either on same intranet or on public internet without and UDP blocks / firewalls / restriction .

There you go everything is looking good from here now . But wait a sec what about step 3 . Lets see what that has to say .

Step 3: Call people in a inter network fashion 

Sure the architecture I have setup is bound to work everywhere , but wait it doesnt . Error in connectivity , errors in console , blank video are the problems that might appear . So well err things begin to get a bit complicated from here . To bypass network firewalls , corporate net policies , UDP blocks and filters we require a TURN server .

Now we have 3 options to choose from

1.  Use a wildly popular http://numb.viagenie.ca/

2. Build your own TURN server with RFC 5766 , or rather easier would be to use any open source TURN server code available in Github

3. Pay and use a commercial TURN service provider or you can even use their trail version to see if things work out for you .

Remember you can use any TURN service it does not affecr your WebRTC API functionality . All we need to do is add it to Peerconnection confih configuration like

peerConnectionConfig: {
iceServers:[
{“url”: < stunserver address >},
{“username”:”xx”,”url”:< turn server address transport=udp>,”credential”:”yy”},
{“username”:”xx”,”url”:< turn server address transport=tcp> , “credential”:”yy”}]
},

step 3 : Call between Inter network machines

There we go , now anyone from anywhere should be able to use our WebRTC setup for making audio , video calls or just exchanging data via DataChannel ( like screen-sharing , file transfer , messages , playing games , collaborative office work etc )  .

The setups covers scenarios wherein user is on office corporate network , home network , mobile network , no problem as long as he / she has a webrtc enables browser ( read Chrome , Mozilla , Opera ) .

It is noteworthy that ideally voice should be traversing on TCP while video and data can go around in UDP however unless restrained the WebRTC API’s self determine the best protocol to route the packets / stream .

………………………………………………………..

you can read more about best WebRTC frameworks and code in this book

Current State of WebRTC

In the last few months I have been observing how the course for WebRTC is turning out so far . Unfortunately contrary to my expectations the fundamental holes in WebRTC specification are still the same with less being done to fulfill them . Ofcourse now there are abundance of interactive
WebRTC API each using a new masking method to call the same old WebRTC API function of getusermedia and peer-connection . Few of these I will list down in this blog but no concrete stable reliable  guide to setup the backbone network ( yes i am referring to Media inter conversion , relay , TURN , STUN servers ) which is left to telecom software engineer / developer to find out and configure . Instead I see many commercial service providers who claim of providing their backend for our WebRTC implementation but that in my opinion completely defeats the objective of WebRTC based communication .  WebRTC was meant to *everything you can’t do with proprietary communication tools and networks* .

Well moving on , here are some nice API implementations of WebRTC ( only for Websockets no SIP )

1.appRTC

apprtc apprtc2 apprtc3 apprtc4 apprtc5

2. talky.io

talky3 talky2 talky1 talky4 talky5 talky63. tokbox

tokbox2tokbox1tokbox7 tokbox6 tokbox5 tokbox4 tokbox3

IPTV ( Internet Based Television ) appliactions

We know the power of Internet protocol suit as it takes on the world of telecom . Alreday half of Communication has been transferred from legacy telecom signalling protocols like SS7 to IP based communication ( Skype , Hangouts , whatsapp , facebook call ) . The TV service providers too are largely investing in IP based systems like SIP and IMS to deliver their content over Telecom’s IP based network ( Packet switched ).

A consumer today wants HD media content anytime anywhere . The traditional TV solutions just dont match upto the expectations anymore . The IPTV provider in todays time must make investments to deliver content that is media-aware, and device-aware. Not only this it should be  personal, social, and interactive . after all its all about user  experience.

Few popular applications for IPTV solutions developers are

  • Menu overlay with detailed description of channels , categories , programs , movies
  • Replay option also referred to as timeshift . It allows a user to pause , resume and  record the show in his absence and view it later
  • Video on demand which concerns paying and viewing music albums , movies etc on demand
  • Live streaming of events such as president speech , tennis match etc .

Application that can be build around the IPTV context

  • Parental Control to realtime view , monitor and control what your child is watching on the IPTV
  • Watch the surveillance  footage from IP cameras anywhere
  • Real time communication on IPTV  with advanced features like call continuity , content sync .
IPTV overview

IPTV overview

Developing a OTT ( Over the Top ) Communication application

Market trends are really not in favor of Telecom Service /providers with increasing use of OTT application like watsapp , Facebook messenger , Google hangouts , skype  , viber , etc .

Four Depleting waves of revenue – Operator’s dilemma

  1. Messaging – OTT messaging cost operators $13.9 billion, or 9% of message revenue in 2013
  2. Voice – Voice services under threat from VOIP services like Skype, Viber
  3. OTT apps – Voice & Message apps have been the operator’s biggest headache. Its time Operator should launch its own OTT Services
  4. Data Traffic – The utilization is yet to reach its peak. Will face challenges from  WiFi access
  5. Critical Pain areas – Erosion of Operator’s revenue from voice and (especially) messaging

At this stage it is crucial for a telecom Service provider / Operator to enter the Apps market and bring forth a Messenger which is more powerful , interactive and awesome than a OTT application.  Fortunately the Operator can always couple this application with his background telecom infrastructure to provide the edge in performance and functionalists .

Let us analyse the current roadblocks for the Telecom Operator through the following figure

Road block while developing a OTT application for a Telecom Service Provider

Road block while developing a OTT application for a Telcom Service Provider

Next we find the way of solving the problems and integrating them together to form a Solution that is described in the figure below :

OTT Application for Telecom Service provider

OTT Application for Telecom Service provider

This writeup outlines the process of creating a OTT application for a Telecom Service Provider .

Components for the application include cloud Address Book , Video Chatting , Location share , Contact synchronization ,REST based thin  client , OS and device agnostic etc shown in the figure below

Ott3

The Application  is designed to close knit with Operator’s own infrastructure hence the crucial entities like Network Address Book , Location Service are synced and fetched from Backend Network .

High level design of the OTT application is provided below :

Technical high level digram for developing Telecom Operator's own OTT application

Technical high level digram for developing Telecom Operator’s own OTT application

Feature Overview

Smart Address Book

  • Automatic: Get contacts from Gmail, Facebook
  • Fast search by first, last name, frequently
  •   dialed number
  • Roadmap: View calendar events
  • Personal: Get image from Gmail and display in   contacts list

Geo Location

  • Share own location during chatting
  • Get map for calculating the distance between two chat users
  • Roadmap : Trigger device (say Switch on/off AC before reaching home) from a threshold distance away from home   location

Messaging

  • Ad-hoc Chat
  • Session Based Chat
  • Voice Input for texting
  • Presence information of contacts
  • RoadMap: Legacy message integration

Telephony

  • Voice call to mobile
  • Voice call to PSTN
  • Video call to other @imAll user
  • Share images during voice call to other

Device agnostic

  • Compatible with IOS, windows
  • Can run as native app on ipad
  • Can run as browser client on windows
  • RoadMap: native app for android, windows phone,blackberry10

Roadmap

  • To upgrade the application and provide enganced and enrich service support the I propose the following roadmap.
  • From plain vanilla voice and video calling ( supported by every other OTT application ) our application should progress towards  legacy telecom support whihc included PSTN , GSM , ISDN etc . This requires backbone of telecom network and a good setup for media codec conversion to suit various legacy media codecs .
  • Ott4
  • To keep the interest of customers it is essential that the application be supported on other popular OTT services like skype  , Gtalk . for exmaple a caller should be able to make call from Skype  / Gtalk to our application .
  • Multilingual capabilities, support for larger protocol spectrum will just act like icing on the cake .
  • How does it benefit the Operator??
  1.  Saves on development cost and time
  2.  Device Agnostic OTT Applications
  3. Simplified Service deployment
  4. Saves licensing cost per client
  5. Reuses existing Messaging and   Address Book service logic.
  6. Open New Revenue Streams for operator
  7. No separate SIP stack required for the client
  8.  Faster Time to Market

Developing a Service Creation Environment for SIP Applications

I hoped of making a SIP application Development environment a year back and worked towards it earnestly . Sadly I wasn’t able to complete the job yet I have decided to share a few things about it here .

Aim :

Develop  a SCE ( Service Creation Environment ) to addresses all aspects of lifecycle of a Service, right from creation/development, orchestration, execution/delivery, Assurance and Migration/Upgrade of services.

Similar market products :

  • Open/cloud Rhino
  • Mobicents and Telestax

Limitations of open source/other market products:

  • Free versions of the Service Creation Environments do not offer High Availability.
  • High Cost of Deployment grade versions.

Solution Description

I propose a in-house Java based Service Creation Environment “SLC SCE”. The SLC SCE will enable creation of JAINSLEE based SIP  services. It can be used to develop and deploy carrier-grade applications that use SS7 and IMS based protocols such as INAP, CAP, Diameter and SIP as well as IT / Web protocols such as HTTP and XML.

Benefits:

  • Service Agility
  • Significantly Lower price points
  • Open Standards eliminate Legacy SCP Lock-in

Timeline

Java-based service creation environment (SCE) – 1.5 Months

Graphical User Interface (GUI) and schematic representations to help in the design, maintenance and support of applications – 1.5 months

SIP Resource Adapter – 1 month

—————————————————————

In essence it encompasses the idea of developing the following

  1. SIP stack
  2. Javascript API’s
  3. Java Libraries for calling SIP stack
  4. Eclipse plugin to work with the SIP application development process
  5. Visual Interface to view the logic of application and possible errors / flaws
  6. SDKs (  Service Development Kit) , which are development Environment themselves

Extra Effort required to put in to make the venture successful

  1. Demo applications for basic SIP logic like Call screening , call rerouting .
  2. tutorial to create , deploy and run application from scratch . Aimed at all sections ie web developer , telecom engineer , full stack developer etc .
  3. Some opensource implementation on public repositories like Github , Google code , SourceForge
  4. Perform active problem solving on Stackoverflow , CodeRanch , Google groups and  other forums .

—————————————————————

Call Continuity from Mobile GSM network to WebRTC

In  the present age of IP telephony when telecom convergence is the big thing all around the world , need of the hours is to enable fixed and mobile Service Providers ( SP )  to monetize the subscriber’s phone number by extending it to new web based services.SPs can offer a WebRTC Communicator endpoint that uses the same phone number as the subscriber’s fixed or mobile phone.

Advanced features enable calls to be transferred between fixed-line, mobile and WebRTC endpoints.

Find the diagram depicting this below :

Transfer mobile callto WebRTC session

Transfer mobile callto WebRTC session

SPs can offer 3rd Party WebRTC endpoints to access the user’s phone number and subscription . E.g. enable web applications such as Facebook, Amazon or Netflix to allow their users to make/receive calls or messages directly from the web applications

Revenue Streams :

  • monthly fee for access to WebRTC endpoints and for receiving calls from by 3rd Party WebRTC endpoints
  • One time upgrade fees for Accessing the Web service integration with telecom network like a plan upgrade

Brownie points

  • No software is required to be downloaded on the subscriber’s computer, tablet or mobile phone
  • No desktop support required for the service provider

Plans For Consumer Customers:

  • Subscribers can use the WebRTC endpoints on their computers, tablets or mobile phones as a fixed-line device at home, as a desktop solution when away from home and to avoid international tolls when traveling
  • Subscribers can connect their web services (e.g. Websites , Facebook, Amazon, Netflix) to their fixed or mobile services subscriptions using their SP-provided phone number

Plans For SP Enterprise Customers:

  • Enterprises can deploy a WebRTC endpoint for their employees that provides a single corporate communications endpoint that can be connected to any of the corporation’s UC/PBX and Call Recording systems
  • Employees can use the WebRTC endpoint as their office phone at work, home or when traveling
  • Connects to all leading UC/PBX and Recording platforms simultaneously
  • Enterprises can deploy a single WebRTC endpoint across all their UC/PBX and Recording platforms – current and future
  • Easy for IT departments to deploy – no software is required to be downloaded to employees’ computers, tablets or mobile phones
  • Enables corporate policies and features from the WebRTC endpoint including
  • Displaying the corporate identity
  • Routing calls via corporate networks
  • Tracking and Recording calls and messages

Service Harmonization between various telecom generations

I shall be editing this post to discuss more on the process of Service Harmonization to save the Telecom Service Provider the trouble of rewriting call logic with every telecom generation evolution ie IN to SIP to Web based .

The Service Harmonization Layer does the job of holding all new and legacy services while providing uniform interface to interact with access network regardless of the back-end Call program logic .

Diagrammatic depiction of roles of Service Harmonization

Role of Service Broker and Service Harmonization Layer in Telecom Network

Role of Service Broker and Service Harmonization Layer in Telecom Network