2 years ago I started with Patrick the Whenever.is project, an agenda where new comers in the Dutch Japanse popculture scene can see which events are held and where. We also added a livestream so that they can watch what the so called anime conventions are.
Patrick and I are both webdevelopers so the website wouldn’t be a problem, but the livestream was a whole other story 😛
After some trials with open source software we decided to use the commercial software Wowza, I had at the time no time to find out what was the best solution for us because an upcoming Dutch exam.
Because Wowza worked well for us it was kind of sticky.
This was until Animecon 2014. we had to tweak our setup to provide a platform for a theoretical 2000 viewers. 1 of the tweaks was move from the statefull protocol RTMP to the stateless HTTP protocol HDS.
We still used Wowza as our origin server (the server we push our livestream to).
Our viewers connect to so called edge nodes, these are local caches located around the world (Amsterdam, Singapore and New York in our case) that serve the content fetched from the origin to our viewers. In our analytics we saw disturbing figures. The max speed of the local caches, which were connected with 1Gbps uplinks, was 100mbps (the uplink of our origin).
Because this was something we had to eliminate we were researching other products.
For our local caches we used the Russian open source webserver nginx. nginx is a highly scalable webserver that can handle more than 10.000 clients at any given time. by extending nginx it can handle RTMP.
Based on this we decided to create a new infrastructure.
Because we use Wirecast for recording and broadcasting our livestreams we had to support RTMP.
nginx supports RTMP and can transmux the livestream to the HTTP based protocol HLS.
Caching the HLS files created by our origin will not work, so every edge node will create its own HLS files based on the RTMP stream fetched from the origin.
It will be something like this:
Wirecast on premise > origin (based in Amsterdam at PCextreme) > edge nodes (based in Amsterdam, Singapore and New York at Digital Ocean) > viewers around the world
because of the connection between the origin and edge nodes are RTMP and the edge node transmux it to HLS we now for sure that there is only 1 connection and that we can support around 700 viewers per server.
The reason why we use HTTP based livestream protocols is that we can use DNS based load balancing.
It’s an easy form of load balancing, since the clients pick up the closest server.
Now we have our infrastructure we have to create the custom software that control our content delivery network 🙂