Skip to main content

Initial Setup

For the purpose of this guide, I will assume that you are using an Ubuntu 22.04 based virtual machine with a public IP address that is routeable across the internet. Instructions for other systems are similar, but may be slightly more involved.

Pre Setup

Before we get going on anything, you'll want to disable rp_filter, and allow all forwarding. The reason this is required is that you will likely end up serving as transit for some NX3 traffic. If you don't allow forwarding, a user that is connected through you may be unable to access the entirety of the NX3 network. You can accomplish this through these commands:

cat <<'EOF' >> /etc/sysctl.conf
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.forwarding=1
net.ipv6.conf.all.forwarding=1
EOF
sysctl -p

Install WireGuard

In order to create the overlay network, each machine needs to have a "direct" connection with the other it wishes to peer with. This can be done by using WireGuard, or any other similar VPN technologies.

In theory multiple peers could be served with a single WireGuard interface, however internal routing is performed, and thus multiple interfaces are needed to allow for the BGP daemon to do routing.

The simplest way to install WireGuard is by using apt:

sudo apt update
sudo apt install wireguard

Following the installation of WireGuard, you'll want to generate your private and public keys:

mkdir -p /etc/wireguard
cd /etc/wireguard
wg genkey | tee private | wg pubkey | tee public

These commands create a directory to store all of the WireGuard configuration details, as well as the keys for encryption.

Now, you'll want to create a connection with your peer, you can create this file as nx3_030002.conf where 030002 are the last 6 digits of the AS you'll be peering with. You'll want the contents as follows:

[Interface]
ListenPort = 20002
PrivateKey = {Private Key}
Address = {First v4 Address}/32
Address = {First v6 Address}/128
Table = off
PostUp = /usr/sbin/ip -4 route add dev nx3_030002 {Remote first v4 Address}/32
PostUp = /usr/sbin/ip -6 route add dev nx3_030002 {Remote first v6 Address}/128

[Peer]
PublicKey = {Remote Public Key}
EndPoint = {Remote Public IP}:20001
AllowedIPs = 172.24.0.0/14, fd00::/8
PersistentKeepalive = 60

You'll want to take note of the ListenPort and the Endpoint port of the peer. In my configuration, the ports are also based off of the AS you're peering with. In this case 2 + the last 4 digits of the AS number.

If you have the port numbers or IP addresses wrong, your tunnel will not connect, and/or traffic will not be routeable.

Additionally, you'll want to replace each of the {} above with the appropriate values for your environment.

Repeat this step for the other side of the tunnel, again using the appropriate values.

In order to bring these tunnels up, you'll want to run the command, replacing {TUNNEL} for the last 6 digits of the peer AS:

wg-quick up nx3_{TUNNEL} && systemctl enable wg-quick@nx3_{TUNNEL}

Install BIRD2

In order for the routes to be shared amongst the (now) connected machines, you'll want to use a routing daemon. The solution we'll be using is BIRD2.

To get started, make sure you install BIRD:

sudo apt update
sudo apt install bird2

From here, we'll want to set up the BIRD config in /etc/bird/bird.conf.


################################################
#               Variable header                #
################################################

define OWNAS =  {YOUR AS};
define OWNIP =  {YOUR IP};
define OWNIPv6 = {YOUR IP6};
define OWNNET = {YOUR NET};
define OWNNETv6 = {YOUR NET V6};
define OWNNETSET = [{YOUR NET}+];
define OWNNETSETv6 = [{YOUR NET V6}+];

################################################
#                 Header end                   #
################################################

router id OWNIP;

protocol device {
    scan time 10;
}

/*
 *  Utility functions
 */

function is_self_net() {
  return net ~ OWNNETSET;
}

function is_self_net_v6() {
  return net ~ OWNNETSETv6;
}

function is_valid_network() {
  return net ~ [
    172.24.0.0/14{21,29}, # nx3
    172.24.0.0/24{28,32}, # nx3 Anycast
    172.25.0.0/24{28,32}, # nx3 Anycast
    172.26.0.0/24{28,32}, # nx3 Anycast
    172.27.0.0/24{28,32}  # nx3 Anycast
  ];
}


#roa4 table nx3_roa;
#roa6 table nx3_roa_v6;

#protocol static {
#    roa4 { table nx3_roa; };
    # include "/etc/bird/roa4_nx3.conf";
#};

#protocol static {
#    roa6 { table nx3_roa_v6; };
    # include "/etc/bird/roa6_nx3.conf";
#};

function is_valid_network_v6() {
  return net ~ [
    fd00::/8{44,64} # ULA address space as per RFC 4193
  ];
}

protocol kernel {
    scan time 20;
    learn;
    ipv6 {
        import where net ~ [fd00::/8+];
        export filter {
            if source = RTS_STATIC then reject;
            krt_prefsrc = OWNIPv6;
            accept;
        };
    };
};

protocol kernel {
    scan time 20;
    learn;
    ipv4 {
        import where net ~ [172.24.0.0/14+];
        export filter {
            if source = RTS_STATIC then reject;
            krt_prefsrc = OWNIP;
            accept;
        };
    };
}

protocol static { ipv4; route OWNNET unreachable; route 172.24.0.53/32 unreachable; }


protocol static { ipv6; route OWNNETv6 unreachable; }


template bgp nxpeers {
    local as OWNAS;
    multihop;

    ipv4 {
        import filter {
          if is_valid_network() && !is_self_net() then {
            #if (roa_check(nx3_roa, net, bgp_path.last) != ROA_VALID) then {
            #  # Reject when unknown or invalid according to ROA
            #  print "[nx3] ROA check failed for ", net, " ASN ", bgp_path.last;
            #  reject;
            #} else accept;
            accept;
          } else reject;
        };

        export filter { if is_valid_network() && source ~ [RTS_STATIC, RTS_BGP] then accept; else reject; };
        import limit 9000 action block;
    };

    ipv6 {
        import filter {
          if is_valid_network_v6() && !is_self_net_v6() then {
            #if (roa_check(nx3_roa_v6, net, bgp_path.last) != ROA_VALID) then {
            #  # Reject when unknown or invalid according to ROA
            #  print "[nx3] ROA check failed for ", net, " ASN ", bgp_path.last;
            #  reject;
            # } else accept;
            accept;
          } else reject;
        };
        export filter { if is_valid_network_v6() && source ~ [RTS_STATIC, RTS_BGP] then accept; else reject; };
        import limit 9000 action block;
    };
}


include "/etc/bird/peers/*";

The biggest things of note here: Update the Variable Header section with your own information. Additionally. ROA is not configured by default on the BIRD instance. This is for ease of use. If ROA is desired in the future, please read the related page.

This file only sets up your local session, in order to get peers up and running, you'll want to create an /etc/bird/peers/asNUM.conf for each of your peers as well:

protocol bgp AS{Remote AS} from nxpeers {
  neighbor {Remote first v4 Address} as {Remote AS};

  ipv6 {
    import none;
    export none;
  };
}

protocol bgp AS{Remote AS}_v6 from nxpeers {
  neighbor {Remote first v6 Address} as {Remote AS};

  ipv4 {
    import none;
    export none;
  };
}

Again, make sure that you replace the values in {} with the appropriate values for your environment.

In order for BIRD to take your new config, you'll want to run:

birdc configure

To confirm that you're receiving routes, you can run:

birdc show route

You should see an output similar to this:

BIRD 2.0.8 ready.
Table master4:
172.24.32.0/24       unicast [AS4266030002 15:02:25.321 from 172.24.32.1] * (100/?) [AS4266030002i]
        dev nx3_030002
172.24.33.0/24       unicast [AS4266030003 15:03:20.346 from 172.24.33.1] * (100/?) [AS4266030003i]
        dev nx3_030003
172.24.32.1/32       unicast [kernel2 15:02:21.444] * (10)
        dev nx3_030002
172.24.33.1/32       unicast [kernel2 15:02:21.444] * (10)
        dev nx3_030003
172.24.34.1/32       unicast [kernel2 15:02:21.444] * (10)
        dev nx3_030004
172.24.31.0/24       unreachable [static3 15:02:21.440] * (200)
172.24.0.53/32       unicast [AS4266030002 15:02:25.321 from 172.24.32.1] * (100/?) [AS4266030002i]
        dev nx3_030002

Table master6:
fd06:108f:6f8e::1/128 unicast [kernel1 15:02:21.443] * (10)
        dev nx3_030003
fd48:c420:f618::/48  unreachable [static4 15:02:21.440] * (200)
fdd8:4045:c53c::1/128 unicast [kernel1 15:02:21.443] * (10)
        dev nx3_030002
fdaa:6145:8e73::1/128 unicast [kernel1 15:02:21.443] * (10)
        dev nx3_030004

Table nx3_roa:
172.24.32.0/24-24 AS4266030002  [static1 15:02:21.440] * (200)
172.24.33.0/24-24 AS4266030003  [static1 15:02:21.440] * (200)
172.24.31.0/24-24 AS4266030001  [static1 15:02:21.440] * (200)
172.24.0.53/32-32 AS4266030002  [static1 15:02:21.440] * (200)

Table nx3_roa_v6:
fd48:c420:f618::/48-48 AS4266030001  [static2 15:02:21.440] * (200)

Note that this output contains ROA data. The default configuration above has ROA disabled.

When looking at this output, you'll notice a couple of things:

  1. An unreachable route is created for your announced network. This ensures that your network will be announced over BGP.
  2. Kernel routes have been inserted for peers over WireGuard
  3. BGP Peers share other BGP routes they have learned.

If you want to see more detailed information regarding routes, you may run

birdc show route all

The output of this command has been omitted for brevity.

Back to top