Xplornet and its confounded double NAT

Edit: I’ve taken a stab at the OpenVPN version of this, which I consider an improvement, here.

Contents

Who is Xplornet

You can check their site, but they’re a nation-wide Internet service provider in Canada. They generally focus on rural areas and offer both satellite and fixed wireless products (previously 4G, but now LTE). If you have experience with satellite Internet, you’re likely aware it’s at just about the bottom of the list for desirable connections. Its typically expensive, has low bandwidth, restrictive data caps, and high latency. Contrast that with Xplornet’s fixed wireless offerings which historically have been expensive, have relatively low bandwidth, restrictive data caps, but acceptable latency! We jumped at the opportunity to upgrade from 4G to LTE – the pitch was “Up to1,2,3 25 Mbps down and 1 up with a 500GB data cap for 99$/month”. If you live in an urban area you may find that surprising, but that is in fact a pretty decent deal. We were paying about 10$/month less for 10/1 connection and a 100GB data cap. So we switched (eventually, after Xplornet’s labored roll out).

One static IP, please

A couple months after switching, I wanted to expose a small web server as part of a hobby project. I looked at the Xplornet website and found that they offer static IP addresses – see? They seem to have forgotten to mention that static IPs were not something they could offer with their LTE service, only with their now obsolete WiMAX. There was no amount of money (even the 10$/month was expensive enough…) which could purchase a static IP from Xplornet on their LTE equipment. So, how to host a server? The go to solution in situations like this is a dynamic DNS(DDNS) – for most purposes its functionally equivalent so long as the DNS is updated every time the IP address changes. I’ve personally used (and enjoyed) FreeDNS, but there are other providers like NoIP. There are yet other providers that are not free, but I don’t see a compelling reason to use them. So, problem solved, right? Update the DNS entries every time Xplornet issues a new IP address and we’re off to the races! Not a chance. Xplornet provides users with an IP address through network infrastructure that involves a double NAT.

What is a double NAT?

Network Address Translation. Is that helpful? Someone who actually knows about it can explain it better (e.g. here). Essentially, with finite IP addresses and an ever-growing number of devices, the number of devices that are directly accessible from the public Internet is a small subset of all public Internet connected devices. For many home Internet users, their ISP gives them a dynamically allocated IP address and that conventionally ends up associated with the user’s router. The router provides the means to translate addresses for incoming and outgoing packets – this is where you get your computers IP address (usually looks like 192.168.1.10). Machines on your local network can “see out” via the router, but public Internet users can only see so far as the router (wild generalizations happening here). This means that if you want to host a website or run a server that is accessible from anything other than your LAN, you need to work around the NAT that the router is offering you. Everyday users run into this with the concept of port forwarding for video games or other software products – the router sends traffic that shows up to a given port directly to the target computer.

The issue with what Xplornet does is that it has its own router that serves many users routers. One public IP address hits Xplornet’s router, is split into multiple private subnets (one layer of NAT), and those subnets are provided to end users who then use NAT (the second layer) to route to all the devices in their house. Generally, this doesn’t cause any issues and makes better use of the increasingly scarce IPv4 addresses, but if you actually want to have anything visible from the public Internet, it’s a nightmare. There’s no direct connection from the equipment that is in/around your house (including Xplornet’s antenna) to the public Internet. So without cooperation from Xplornet, there is no way to accomplish the same effect as a static IP address with similar methods.

Feel free to check out other people running into this problem (and far more people fundamentally misunderstanding it) over on the RedFlagDeals site.

What to do?

A double NAT means you cannot use a DDNS or port forwarding or any of the other common suggestions people have for resolving what they think the issue is. They address the single NAT problem, but do a whole lot of nothing for this situation. So what are the options?

Free

What I believe to be the easiest is a “local tunnel” solution. There are plenty out there, one of the more popular is ngrok. It has a free tier and as long as you don’t need multiple ports on a single domain (like I did), it gets you 4 tunnels and is actually super handy. Something as simple as:

ngrok http 80

Gets you:

ngrok by @inconshreveable                                                 
                                                                          
Session Status                online                                      
Version                       2.1.18                                      
Region                        United States (us)                                               
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://d75b33b2.ngrok.io -> localhost:80
Forwarding                    https://d75b33b2.ngrok.io -> localhost:80
                                                                          
Connections                   ttl     opn     rt1     rt5     p50     p90 
                              0       0       0.00    0.00    0.00    0.00

Similarly, Pagekite is easy to get up and running. The difference there is the free tier of ngrok doesn’t expire, while Pagekite’s does.

Neither of these (so far as I know) offers any way to get many ports on the same domain (or subdomain). You can have many ports fed from a single machine, but they’re made accessible on separate subdomains. This part didn’t work for me.

As close as you can get to free

I think there are two workable options. One would be to set up a publicly accessible VPN server, the other is to set up a publicly accesible SSH server. All else held equal, I imagine the VPN server would be the best option, but I’ve never done that before. SSH servers are present by default on many Linux distributions, so most of the work is done.

With that said, my answer here is: reverse SSH tunnel. Cloud computing has driven the price of VPSs into the ground. Whether it be with DigitalOcean, AWS, or Linode (see comparison here), you can pick up a pretty beefy machine for 10$ per month of runtime. DigitalOcen even offers a 5$ tier, which is pretty astounding. With that taken into consideration, it’s cheaper (or nearly the same price) to get a cloud instance than it is to pay for something like ngrok or Pagekite. You need barely anything computing resource wise to run this kind of set up.

There are two pieces to this, I’ll refer to them as the “public computer” (cloud instance”) and the “private computer” (the one you’re wanting to expose).

Public computer

The public computer set up is pretty straightforward:

  1. Already have, or sign up for a cloud instance with DigitalOcean, AWS, Linode or similar
  2. Provision the machine with some kind of Linux (e.g. Ubuntu 16.04)
  3. Know enough about securing a publicly accessible machine that you don’t need to be told how to (you were already exposing something to the internet, so presumably you have some idea). You can look at this guide for a quick intro.
  4. SSH into it ssh user@cloud-instance-ip.totallyreal.com
  5. Make sure it’s up to date – sudo apt update && sudo apt upgrade
  6. Here’s the tricky part. It took me more than an hour to find out why I could only see the tunneled ports from the remote machine. Feel free to read up on the documentation. You have to open up your public machine so that the SSH tunnels are exposed to the world at large (hopefully not everybody, you have a firewall set up – right?). This is possible by modifying /etc/ssh/sshd_config – note, you’ll likely have a ssh_config as well, so pay attention to what you’re modifying (ssh vs sshd). The ssh_config file is for the public machine’s SSH clients (i.e. when it connects to other servers), we need to change the settings for the public machine as a server – sshd_config. Anyways,
    1. sudo sed -i '$a GatewayPorts clientspecified' /etc/ssh/sshd_config
    2. Restart the SSH server: sudo service ssh restart

And that’s it. The public computer is ready to act as a gateway to your private computer. All it needs to do is stay accessible.

Private computer

  1. First things first, you’re going to want to ensure your private machine can communicate with the public machine securely. The easiest way to do this is with SSH keys – they’ll allow the computers to connect without having to type in a username and password everytime. If anything here is confusing, follow the better written guide here.
    1. Ensure you have SSH installed (as well as autossh, we’ll be using that): sudo apt install ssh autossh
    2. Ensure you’re logged in as the user you’d like to connect to the public computer with
    3. Create a private/public key pair to use for authentication with the public computer
      1. ssh-keygen
      2. You can use the defaults for all the prompts or change them as you wish
      3. This will yield a key pair – ~/.ssh/id_rsa (the private part that you never share with anyone) and ~/.ssh/id_rsa.pub (the public part, designed to be shared)
    4. Add the public key that corresponds to your private machine to the “authorized keys” on the public machine
      1. ssh-copy-id user@cloud-instance-ip.totallyreal.com
  2. You can now test the connection with your own configuration, for me it was running on port 7080:
    1. ssh -R "[::]:7080:localhost:7080" -N root@23.239.2.79
    2. This sets up a reverse tunnel from the public machine’s port 7080 to the private machine’s port 7080
  3. Using SSH directly works, but for a more robust and persistent connection we’ll use autossh. It’s basically just a managed SSH connection, perfect for this use case.
    1. To make this easy, we’ll use a SSH client config file. In ~/.ssh/config, put the details of your connection. Mine, for example, looked something like this:
HOST unifi-video-tunnel
    HostName            li838-22.members.linode.com
    User                toby
    Port                22
    IdentityFile        /home/toby/.ssh/id_rsa
    ServerAliveInterval 30
    ServerAliveCountMax 3
    RemoteForward       :7080 localhost:7080
    RemoteForward       :7443 localhost:7443
    RemoteForward       :7445 localhost:7445
    RemoteForward       :7446 localhost:7446
  1. To add yet another layer, everything will be easier with a service wrapping around the autossh session. In Ubuntu 16, that would be a systemd service.
    1. Add a .service file that reflects the application to /etc/systemd/system/ – I called mine autossh-unifi-video.service
    2. Flesh out the service to invoke autossh for you
[Unit] 
Description=Make Unifi Video available remotely
After=network.target 

[Service] 
User=toby
ExecStart=/usr/bin/autossh -M 0 -N unifi-video-tunnel

[Install] 
WantedBy=multi-user.target]
  1. As with any service, get it moving:
    1. systemctl daemon-reload
    2. systemctl enable autossh-unifi-video.service
    3. systemctl start autossh-unifi-video.service

And that’s pretty much it! The reverse tunnel should be working, and everything you’ve mapped should be exposed to the public.

Improvements

  • Add a separate user for autossh
  • Use a VPN solution instead of this

9 thoughts on “Xplornet and its confounded double NAT

  1. wee_fla

    I ran into this double nat issues when I switched to their lte service and wished to connect to my home security system when I was travelling. The work-around I implemented was to drag an old laptop that was gathering dust from a drawer. Set the laptop up with teamviewer. Enable teamviewer for unattended access and to automaticially start at boot-time. I authorized the laptop that I take with me when travelling to have access to the old in-house laptop. It all works great and I can access my security cameras.

    Reply
    1. tobymurray Post author

      That’s a neat solution – handful of questions:
      – How stable is your connection with TeamViewer? Does it drop when you’re out and about at all?
      – I guess with this solution you’re effectively watching the video stream through a remote desktop session – does that suck or is it passable?
      – Have you tried a mobile client? Does TeamViewer reconcile resolution differences nicely?
      – Is there a “pairing” you have to go through to authorize a given device, or could you add a new device while away from home?

      Reply
  2. RedDirtRiders.com

    I used to use a DDNS service duckdns.org and while it is updating my WAN ip correctly, traffic just can’t get past the xplornet front end, and I found this blog which addresses the reason why. Thanks for clarifying what is going on and a possible solution.

    Reply
    1. RedDirtRiders.com

      SSH access over ngrok:
      $ ngrok tcp 22
      Response from ngrok:
      Forwarding tcp://0.tcp.ngrok.io:19856 -> localhost:22

      To access from a remote computer or device:
      ssh user@0.tcp.ngrok.io -p19856

      I run this in a “screen” session so it remains running in the background until reboot.

      I hope this helps another Xplornet user gain access to their machine. 😉

      Reply
  3. Genny

    Is it possible to do the private computer config setup on a router (maybe running DD-WRT), without a permanent computer located at that end, or do you have to have a computer? I want to look at cameras connected to the router, but currently no computer permanently resides at the location.
    This is a bit above my head, but I’m trying to learn.

    Reply
    1. tobymurray Post author

      I believe the short answer is “yes”. If you have a video stream heading somewhere, you can redirect it with a router (e.g. everything from 192.168.1.32 gets forwarded to the VPN server). In this case you’d have to set up your router as the VPN client. In a lot of ways I think it would actually be preferable, as the camera wouldn’t know it’s being forwarded somewhere, so it may simplify some of the device set up.

      A quick google shows something like this: https://charleswilkinson.co.uk/2016/05/14/selective-routing-using-ddwrt-and-openvpn/

      Reply
  4. oucil

    Not sure if you’re still dealing with this, but here’s what I’ve found…. there are only a small handful of ports that they “reserve” which you cannot forward on through the double NAT, 80 (http) is one of them unfortunately. If you’re stuck using 80, then your other solutions are reasonable, otherwise, you can simply use “double port forwarding” to get around this. Set up port forwarding for an outside port like 8000 on the LTE router, then forward it to 8000 on the internal network, then set up another port forward on your own router from 8000 to port 80 and the specific IP of your machine. I’ve used this for HTTP, Plex, RDP, SSH, etc. and am able to use all of these services without issue now.

    Reply
    1. tobymurray Post author

      Interesting. Naively, it seems like it shouldn’t be possible though… You’re saying there’s a way to have public traffic to an IP address that is shared across many Xplornet customers forwarded to your own server? Have you actually done this? What happens if another user with the same public IP sets the same port forwarding rule, do both users get the traffic?

      I managed to screw up my Internet access by mucking with the LTE router, I’d be interested in a bit of a walkthrough if you know how it works.

      > am able to use all of these services without issue now
      To be clear, you’re able to use these services over the public Internet without any tunneling? You go to oucil.ca:8000 and it sends traffic to your locally hosted server?

      Reply
    2. Genny

      Generally Xplornet and other LTE type providers (like Sasktel Fusion) in rural areas block any and all inbound internet requests. No matter how good your port forwarding is it won’t get to you. DDNS does not work. double port forwarding doesn’t work. That is why you have to set up the tunnel. I was able to get a static IP from Sasktel, but Xplornet would not provide one. Before I set up the static IP on Sasktel, the WAN address on my router was a private range address. “What’s my IP” gave a completely different address I did not see on any router I had access to at all. There is no way to double port forward through that.

      Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: