How to enable broadcast and multicast support on Amazon (AWS) EC2

This article explains how to enable broadcast and multicast support on Amazon (AWS) EC2, which is required for certain enterprise applications and protocols.

Contents

  1. Introduction
  2. Software used
  3. Installation guide
  4. Post installation instructions
  5. Tests and performance
  6. Transfer of one large file to establish consistent throughput
  7. ZMQ Performance, throughput test
  8. OpenVPN, without encryption
  9. TIPC - An example of multicast and broadcast in action using n2n
  10. Summary

Introduction

Broadcast and multicast support is a requirement of many enterprise applications and clustering stacks. However, in our experience, the choice of hosting providers who can effectively execute these networking features is somewhat limited. This means that more expensive co-location is required. It is much the same situation with many cloud providers, who do not provide broadcast and multicast support. Currently, AWS do not officially support broadcast or multicast out of the box, so the only solution if you require instant results is to use another cloud provider, such as Rackspace.

However, as AWS certified consultants, we wanted broadcast and multicast support within the AWS EC2 network as we currently utilise other AWS products and services and wanted to engineer our way out of the problem, so we set about creating our own solution.

Firstly, we're using AWS VPC (Virtual private cloud) instances (any size is supported). To enable Layer 2 communication over the network we are using n2n Peer-to-Peer VPN software which is bound to the internal Ethernet adapter of the AWS VPC instance. As n2n is a secure VPN server it ships by default with encryption and compression. We removed the encryption and compression as it's not required over a AWS VPC connection (Read more about VPC security). For this demonstration we created a VPC with one subnet, in one availability zone. We used four "m1.small" EM2 instances with public IP addresses. Do not forget adjust Security group(s) to only allow access from the internal network, which in our example is: 192.168.100.0/16

Network configuration of VPC Instances

Server name Role IP Address VPN IP Address
test1 supernode 192.168.100.1 none
test2 VPN client 192.168.100.2 192.168.1.2
test3 VPN client 192.168.100.3 192.168.1.3
test4 VPN client 192.168.100.4 192.168.1.4

Software used

  • Operating system: Ubuntu (Saucy 13.10 amd64 server)
  • n2n Peer-to-Peer VPN
  • ZeroMQ

Installation Guide

Although n2n is available via the official Ubuntu repository, we will instead download the source code, as small tweaks have to be made to disable compression and encryption. In order to checkout the source code, subversion is required. Once subversion is installed, proceed to the commands below.

You can use your favorite text editor, but for this demonstration we're using vi, you may substitute with nano or any other

root@test1 $ svn co https://svn.ntop.org/svn/ntop/trunk/n2n
root@test1 $ cd n2n/n2n_v2

Disable encryption

root@test1 $ vi MakeFile

Search for

N2N_OPTION_AES

and change it to

N2N_OPTION_AES=no

Now disable compression

root@test1 $ vi n2n.h

Search for

#define N2N_COMPRESSION_ENABLED 1

Change it to

#define N2N_COMPRESSION_ENABLED 0

Once the files have been edited, we need to compile n2n by typing the following command from within the n2n folder

root@test1 $ make

Once make has been successfully run, you will see two binaries, supernode and edge.

supernode is only required on the supernode server, and edge is required on the others.

Compile n2n on every test server in the same way, or simply copy the edge binary to each server.

The supernode which is responsible for introducing new VPN clients and the edge is used to connect. The VPN clients initially discover each other by using the supernode, but after a connection has been established they never directly communicate through the supernode. This allows us to avoid network bottlenecks and to also mitigate the single point of failure which is present in a star architecture VPN network. If the supernode were to fail, the VPN would continue to function, but no new VPN nodes would be able to register. If we wanted a hot standby for redundancy then another EC2 instance with supernode configured, can be started up using standard AWS failover over procedures.

Post Installation Instructions

On server test1, start supernode

root@test1 $ sudo ./supernode -l 1200

If you haven't blocked access from public Internet traffic to the supernode server, you can achieve this with Iptables by executing the following command

root@test1 $ sudo iptables -I INPUT ! -s 192.168.100.0/16 -m udp -p udp --dport 1200  -j DROP

Now on server test2, start the VPN client

root@test2 $ sudo ./edge -l 192.168.100.1:1200 -c Buckhill -a 192.168.1.2

Check interface by typing

root@test2 $ ifconfig edge0

You should see the following output

edge0     Link encap:Ethernet  HWaddr c6:9b:6f:bf:cb:49
inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
inet6 addr: fe80::c49b:6fff:febf:cb49/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1400  Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:5 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 B)  TX bytes:418 (418.0 B)

On server test3, start the VPN client

root@test3 $ sudo ./edge -l 192.168.100.1:1200 -c Buckhill -a 192.168.1.3

Now check VPN connectivity on server test3 by typing

root@test3 $ ping -c 1 192.168.1.2

The results should look like this

PING 192.168.1.2 (192.168.1.2) 56(84) bytes of data.
64 bytes from 192.168.1.2: icmp_seq=1 ttl=64 time=0.906 ms

--- 192.168.1.2 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.906/0.906/0.906/0.000 ms

You can now connect as many nodes as required.

Testing and performance

Depending on your network requirements it's important to establish the throughput and overhead created by operating over a VPN. In order to determine this we performed two type of tests, between two nodes over the newly created VPN network.

Transfer of one large file to establish consistent throughput

On test2 server netcat was started in listen mode

root@test2 $ nc -l 5001 > /dev/null

On test3 server the 1GB file was created and sent over the network to test2

root@test3 $ dd if=/dev/urandom bs=1M count=1000 of=/tmp/big_file

The results look like this

1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 309.952 s, 3.4 MB/s

root@test3 $ cat if=/tmp/big_file |pv|nc 192.168.1.2 5001
1e+03MB 0:00:25 [38.7MB/s] [                                        ]

If you wish to re-produce our ZeroMQ demonstrations shown below, you will need to install ZeroMQ 4.1 with TIPC support. Due to the complexity of this process, it is outside the scope of this article.

ZMQ Performance, throughput test

The tests used by us are standard ZMQ tests which come built in with libzmq

On server test2 start receiver

root@test2 $ ./local_thr tcp://*:5555 4096 100000

For this test we used a 4KB message size, which is bigger than the average request in our own applications. It is above the MTU of the network adapter.

On server test3, start the test

root@test3 $ ./remote_thr tcp://192.168.100.2:5555 4096 100000

Reference: http://zeromq.org/results:perf-howto

In order to compare results with another VPN service or product, we performed tests using OpenVPN in a Point 2 Point topology.

OpenVPN, without encryption

From server test2

root@test2 $ sudo openvpn --cipher none --proto udp  --dev tun --comp-lzo --auth none --prng none --mode p2p
--ifconfig 192.168.1.2 192.168.1.3  --port 1194

From server test3

root@test3 $ sudo openvpn --cipher none --proto udp  --dev tun --comp-lzo --auth none --prng none --mode p2p
--ifconfig 192.168.1.3 192.168.1.2  --remote 192.168.100.2 1194

OpenVPN without encryption and compression

From server test 2

root@test2 $ sudo openvpn --cipher none --proto udp  --dev tun --auth none --prng none --mode p2p
--ifconfig 192.168.1.2 192.168.1.3  --port 1194

From server test3

root@test3 $ sudo openvpn --cipher none --proto udp  --dev tun --auth none
--prng none --mode p2p --ifconfig 192.168.1.3 192.168.1.2  --remote 192.168.100.2 1194

Test results

Legend

Name in graph Description
eth Ethernet device
n2n n2n vpn with encryption and compression enabled
n2n -e n2n vpn with encryption disabled
n2n -e -c n2n vpn with encryption and compression disabled
OVPN -e OpenVPN with encryption disabled
OVPN -e -c OpenVPN with encryption and compression disabled

Transfer rate, large file

Transfer rate

ZeroMQ messages per second

ZeroMQ messages per second

ZeroMQ bandwidth test

ZeroMQ bandwidth test

Messages per second, based on message size

Message Size

We also performed a ZMQ benchmark of the throughput (messages per second), in relation to the messages size.

If the message fits into n2n's MTU, which is 1400, it will not be fragmented. Usually request headers are below 1KB in size, so a higher amount of messages per second can be achieved compared to a 4KB request.

Another conclusion we've drawn is that bandwidth maximum capacity doesn't differ more than 10% in relation to the message size.

An example of multicast and broadcast in action using n2n and TIPC

Our applications make use of the TIPC protocol, which operates on top of Layer 2 packet networks. Using Amazon's EC2 we can't access that network layer. The following demonstrates how a customised n2n VPN solves this problem.

TIPC reference: http://tipc.sourceforge.net

Configuring TIPC nodes is taken from http://hintjens.com/blog:71

For this demonstration we needed one extra server, called test4, which is also a member of the n2n VPN network

Our TIPC configuration

root@test2 $ sudo modprobe tipc && sudo tipc-config -a=1.1.2 -be=eth:edge0
root@test3 $ sudo modprobe tipc && sudo tipc-config -a=1.1.3 -be=eth:edge0
root@test4 $ sudo modprobe tipc && sudo tipc-config -a=1.1.4 -be=eth:edge0

The multicast_demo from TIPC utilities is used for the following demonstration

On servers test2 and test3 run the listener

root@test2 $ ./multicast_demo/server_tipc
Server: port {18888,300,399} created
Server: port {18888,200,299} created
Server: port {18888,100,199} created
Server: port {18888,0,99} created
root@test3 $ ./multicast_demo/server_tipc
Server: port {18888,300,399} created
Server: port {18888,200,299} created
Server: port {18888,100,199} created
Server: port {18888,0,99} created

On the fourth server a message is sent over multicast

root@test4 $ sudo ./multicast_demo/client_tipc
****** TIPC client multicast demo started ******
Client: sending message to {18888,99,100}
Client: sending message to {18888,150,250}
Client: sending message to {18888,200,399}
Client: sending message to {18888,0,399}
Client: sending termination message to {18888,0,399}
****** TIPC client multicast demo finished ******

Both servers (test2 and test3) receive the message and return output which looks like this

Server: port {18888,200,299} received: message to {18888,150,250}
Server: port {18888,100,199} received: message to {18888,99,100}
Server: port {18888,100,199} received: message to {18888,150,250}
Server: port {18888,0,99} received: message to {18888,99,100}
Server: port {18888,300,399} received: message to {18888,200,399}
Server: port {18888,300,399} received: message to {18888,0,399}
Server: port {18888,300,399} terminated
Server: port {18888,200,299} received: message to {18888,200,399}
Server: port {18888,200,299} received: message to {18888,0,399}
Server: port {18888,200,299} terminated
Server: port {18888,100,199} received: message to {18888,0,399}
Server: port {18888,100,199} terminated
Server: port {18888,0,99} received: message to {18888,0,399}
Server: port {18888,0,99} terminated

Summary

Success, a customised n2n VPN set up can be used to effectively operate broadcast and multicast over the AWS EC2 network.

While the average throughput is less than the standard ethernet interface with no VPN for both file transfer and ZMQ, n2n is still comparable with OpenVPN, considered to be a very fast point to point VPN server, but with the added benefit of running a VPN cluster.

Depending on the packet size, we can send between 3000 and 35000 messages per second over ZMQ, and over 80Mbit/second throughput on a small instance.

If more throughput is required, larger instances can be used.

To conclude this article, depending on your exact requirements, using a customised n2n unlocks broadcast and multicast over the EC2 network. If you require more messages per second than you can achieve with EC2, then we recommend Rackspace performance cloud instances which are more expensive but officially support broadcast and multicast.

For our own requirements, the messages per second achieved with n2n provides plenty of excess capacity.

If you have any feedback please comment below.

Tags:

Cloud Networking EC2 VPC

Published by: Stuart Buckell

Published on: 19.01.2016

Related blog posts

10 things to check if your website or web application is offline

Follow our free 10 point checklist to get your website or application back online.


Managed AWS Cloud

As AWS Certified Partners we provide jargon free, fully managed AWS Cloud services for organisations looking to utilise the flexibility and cost savings AWS can offer.


MySQL Performance Audits

We are excited to announce that we are now providing highly competitive MySQL performance audits which include a complete overview of your current MySQL setup.


"Buckhill Ltd is very knowledgeable and works quickly. We got bogged down and stuck trying to implement SAML on our own but Buckhill Ltd stepped in and immediately executed a working setup and made insightful recommendations for enhanced security."

Ken Snyder - Chief Technology Officer
Sharpr, Salt Lake City, Utah, USA

"I was impressed by the enthusiasm and professionalism of the Buckhill team and their 'can do' attitude."

Matthew Oliver - Director
M3 Corporate Finance LLP, Bristol, United Kingdom

"Working with Buckhill was a pleasure. Right from the consultation process, through to their depths of knowledge & obvious enterprise experience - I would have no hesitation recommending their services."

Reece Griffin, Technical Director
TBA Digital, Canada

"We were working with one of our vendors to configure SSO via ADFS to access their web application. We needed additional expertise and guidance to move the project forward so we contacted Buckhill Ltd. In a matter of days they had their developers working on it. They quickly determined the configuration issues we were having and provided explanation on how to get them resolved. We were very impressed with their knowledge, professionalism and diligence in assisting us on this project."

David O'Boyle CCNA, MCSE - Technology Services
The Richards Group, Dallas, Texas, USA

"I contacted Buckhill as I needed expert advice regarding a complex SAML integration requirement. The project involved integration with the US Health Care API (Obamacare eligibility). The developers assigned to the project were highly capable and professional, with real passion and determination to see the project through to completion even when issues presented themselves. Looking forward to working with you guys in the future, you're awesome! All the best, Carlos."

Carlos Arias - Owner
Carlos Ja Design, Florida, USA

"Buckhill provided us with consultancy to implement SimpleSamlPHP and configure it for use with a major client's SSO.
They were incredibly helpful, easy to work with and delivered in very short timescales within our agreed budget.
I am extremely happy with the service they provided."

Eddie Robins - Technical Director
Groovy Gecko, London, UK

"We asked Buckhill to provide us with a robust, reliable and faster server infrastructure so that we could deliver a better and more consistent service to our customers. Buckhill delivered results quickly, in a friendly and professional manner. We have been impressed by the knowledge they bring to the table and with the honest and helpful communication throughout the process. We are happy to receive continued support and maintenance for a core part of the business."

Brian Kavanagh
Field Gibson Media Ltd, UK

"We have been using Buckhill for several years for all our industry software requirements.

Their systems are reliable and affordable and the team are always friendly and knowledgeable.

If you're looking for an innovative software solutions provider, I would highly recommend them."

Mandy Brown, Director
Litigation Protection, UK

"Working with Buckhill was a pleasure. Right from the consultation process, through to their depths of knowledge & obvious enterprise experience - I would have no hesitation recommending their services."

Reece Griffin, Technical Director
TBA Digital, Canada

"Buckhill provided us with excellent service throughout our project. From the detailed initial meeting, to explaining the process, to final delivery, prompt response and assistance with follow up questions. We have been thoroughly impressed.

Kevin Hoggard, Director
CRM Innovations, Canada

"I must say that working with Buckhill has been a pleasure. The C2MS system has really helped us get our wholesale division up and running and the features within the system are very comprehensive. It allows us to trade with our brokers the way our brokers want to trade with various trading options available. I especially like the way Buckhill respond to you new idea’s and do look to implement them in days rather than months."

Hassan Soyer, Director
Nova Insurance, UK

"Buckhill were very helpful scoping up a vastly improved and flexible server environment - from the very first phone call we were able to define the problem and agree on a route towards solution. The project was completed with the minimum of fuss, on time and on budget. Real time technical support was available throughout the process and we now have a vastly improved operational environment that is future proofed for our strategic objectives. We are very happy with the results and have quickly realised the ROI."

Iain Wilcox, Marketing Director
Digitonic, UK

"We are delighted with our choice of service provider, and have nothing but good things to say about Buckhill. Stuart and his team have acted with a high level of professionalism and competence in all respects. We cannot recommend them highly enough ..."

Matthew Dickinson, Director
Linex Systems, UK

"Buckhill was able to bring a solution in a complex interoperability problem (ADFS/SAML with SOAP between Linux/Windows platform). I was often impressed by the level of their expertise and deep knowledge. When problems arose they were able to find creative solutions with an emphasis on reliability and security. I can say that during the whole project Buckhill was never the bottleneck, our success with this project is definitely due to Buckhill's talented team."

David Wulliamoz, Head of IT Services
Compassion Suisse, Switzerland

Our Partners

VMware Partner
Learn more about Reseller and Partnership opportunities

Join the Partner Program

Our offices are open 8.30am - 7pm GMT, Monday to Friday - but you can always contact us via email. When we receive your email during opening hours, we aim to respond within 30 minutes or less. Should your email reach us out of hours, we will contact you when the office re-opens.

You can contact us using live chat

Contact Information

Speak to an expert on +44(0)1903 250250, email us at contact@buckhill.co.uk, or start a live chat, located in the bottom right

Alternatively, complete the form below

We're happy to talk

Our offices are open 8.30am - 7pm GMT, Monday to Friday - but you can always contact us via email. When we receive your email during opening hours, we aim to respond within 30 minutes or less. Should your email reach us out of hours, we will contact you when the office re-opens.

You can contact us using live chat

Buckhill