Xplornet and its confounded double NAT

19 Replies

Edit: I’ve taken a stab at the OpenVPN version of this, which I consider an improvement, here.

Contents

Who is Xplornet

You can check their site, but they’re a nation-wide Internet service provider in Canada. They generally focus on rural areas and offer both satellite and fixed wireless products (previously 4G, but now LTE). If you have experience with satellite Internet, you’re likely aware it’s at just about the bottom of the list for desirable connections. Its typically expensive, has low bandwidth, restrictive data caps, and high latency. Contrast that with Xplornet’s fixed wireless offerings which historically have been expensive, have relatively low bandwidth, restrictive data caps, but acceptable latency! We jumped at the opportunity to upgrade from 4G to LTE – the pitch was “Up to1,2,3 25 Mbps down and 1 up with a 500GB data cap for 99$/month”. If you live in an urban area you may find that surprising, but that is in fact a pretty decent deal. We were paying about 10$/month less for 10/1 connection and a 100GB data cap. So we switched (eventually, after Xplornet’s labored roll out).

One static IP, please

A couple months after switching, I wanted to expose a small web server as part of a hobby project. I looked at the Xplornet website and found that they offer static IP addresses – see? They seem to have forgotten to mention that static IPs were not something they could offer with their LTE service, only with their now obsolete WiMAX. There was no amount of money (even the 10$/month was expensive enough…) which could purchase a static IP from Xplornet on their LTE equipment. So, how to host a server? The go to solution in situations like this is a dynamic DNS(DDNS) – for most purposes its functionally equivalent so long as the DNS is updated every time the IP address changes. I’ve personally used (and enjoyed) FreeDNS, but there are other providers like NoIP. There are yet other providers that are not free, but I don’t see a compelling reason to use them. So, problem solved, right? Update the DNS entries every time Xplornet issues a new IP address and we’re off to the races! Not a chance. Xplornet provides users with an IP address through network infrastructure that involves a double NAT.

What is a double NAT?

Network Address Translation. Is that helpful? Someone who actually knows about it can explain it better (e.g. here). Essentially, with finite IP addresses and an ever-growing number of devices, the number of devices that are directly accessible from the public Internet is a small subset of all public Internet connected devices. For many home Internet users, their ISP gives them a dynamically allocated IP address and that conventionally ends up associated with the user’s router. The router provides the means to translate addresses for incoming and outgoing packets – this is where you get your computers IP address (usually looks like 192.168.1.10). Machines on your local network can “see out” via the router, but public Internet users can only see so far as the router (wild generalizations happening here). This means that if you want to host a website or run a server that is accessible from anything other than your LAN, you need to work around the NAT that the router is offering you. Everyday users run into this with the concept of port forwarding for video games or other software products – the router sends traffic that shows up to a given port directly to the target computer.

The issue with what Xplornet does is that it has its own router that serves many users routers. One public IP address hits Xplornet’s router, is split into multiple private subnets (one layer of NAT), and those subnets are provided to end users who then use NAT (the second layer) to route to all the devices in their house. Generally, this doesn’t cause any issues and makes better use of the increasingly scarce IPv4 addresses, but if you actually want to have anything visible from the public Internet, it’s a nightmare. There’s no direct connection from the equipment that is in/around your house (including Xplornet’s antenna) to the public Internet. So without cooperation from Xplornet, there is no way to accomplish the same effect as a static IP address with similar methods.

Feel free to check out other people running into this problem (and far more people fundamentally misunderstanding it) over on the RedFlagDeals site.

What to do?

A double NAT means you cannot use a DDNS or port forwarding or any of the other common suggestions people have for resolving what they think the issue is. They address the single NAT problem, but do a whole lot of nothing for this situation. So what are the options?

Free

What I believe to be the easiest is a “local tunnel” solution. There are plenty out there, one of the more popular is ngrok. It has a free tier and as long as you don’t need multiple ports on a single domain (like I did), it gets you 4 tunnels and is actually super handy. Something as simple as:

ngrok http 80

Gets you:

ngrok by @inconshreveable                                                 
                                                                          
Session Status                online                                      
Version                       2.1.18                                      
Region                        United States (us)                                               
Web Interface                 http://127.0.0.1:4040
Forwarding                    http://d75b33b2.ngrok.io -> localhost:80
Forwarding                    https://d75b33b2.ngrok.io -> localhost:80
                                                                          
Connections                   ttl     opn     rt1     rt5     p50     p90 
                              0       0       0.00    0.00    0.00    0.00

Similarly, Pagekite is easy to get up and running. The difference there is the free tier of ngrok doesn’t expire, while Pagekite’s does.

Neither of these (so far as I know) offers any way to get many ports on the same domain (or subdomain). You can have many ports fed from a single machine, but they’re made accessible on separate subdomains. This part didn’t work for me.

As close as you can get to free

I think there are two workable options. One would be to set up a publicly accessible VPN server, the other is to set up a publicly accesible SSH server. All else held equal, I imagine the VPN server would be the best option, but I’ve never done that before. SSH servers are present by default on many Linux distributions, so most of the work is done.

With that said, my answer here is: reverse SSH tunnel. Cloud computing has driven the price of VPSs into the ground. Whether it be with DigitalOcean, AWS, or Linode (see comparison here), you can pick up a pretty beefy machine for 10$ per month of runtime. DigitalOcen even offers a 5$ tier, which is pretty astounding. With that taken into consideration, it’s cheaper (or nearly the same price) to get a cloud instance than it is to pay for something like ngrok or Pagekite. You need barely anything computing resource wise to run this kind of set up.

There are two pieces to this, I’ll refer to them as the “public computer” (cloud instance”) and the “private computer” (the one you’re wanting to expose).

Public computer

The public computer set up is pretty straightforward:

  1. Already have, or sign up for a cloud instance with DigitalOcean, AWS, Linode or similar
  2. Provision the machine with some kind of Linux (e.g. Ubuntu 16.04)
  3. Know enough about securing a publicly accessible machine that you don’t need to be told how to (you were already exposing something to the internet, so presumably you have some idea). You can look at this guide for a quick intro.
  4. SSH into it ssh user@cloud-instance-ip.totallyreal.com
  5. Make sure it’s up to date – sudo apt update && sudo apt upgrade
  6. Here’s the tricky part. It took me more than an hour to find out why I could only see the tunneled ports from the remote machine. Feel free to read up on the documentation. You have to open up your public machine so that the SSH tunnels are exposed to the world at large (hopefully not everybody, you have a firewall set up – right?). This is possible by modifying /etc/ssh/sshd_config – note, you’ll likely have a ssh_config as well, so pay attention to what you’re modifying (ssh vs sshd). The ssh_config file is for the public machine’s SSH clients (i.e. when it connects to other servers), we need to change the settings for the public machine as a server – sshd_config. Anyways,
    1. sudo sed -i '$a GatewayPorts clientspecified' /etc/ssh/sshd_config
    2. Restart the SSH server: sudo service ssh restart

And that’s it. The public computer is ready to act as a gateway to your private computer. All it needs to do is stay accessible.

Private computer

  1. First things first, you’re going to want to ensure your private machine can communicate with the public machine securely. The easiest way to do this is with SSH keys – they’ll allow the computers to connect without having to type in a username and password everytime. If anything here is confusing, follow the better written guide here.
    1. Ensure you have SSH installed (as well as autossh, we’ll be using that): sudo apt install ssh autossh
    2. Ensure you’re logged in as the user you’d like to connect to the public computer with
    3. Create a private/public key pair to use for authentication with the public computer
      1. ssh-keygen
      2. You can use the defaults for all the prompts or change them as you wish
      3. This will yield a key pair – ~/.ssh/id_rsa (the private part that you never share with anyone) and ~/.ssh/id_rsa.pub (the public part, designed to be shared)
    4. Add the public key that corresponds to your private machine to the “authorized keys” on the public machine
      1. ssh-copy-id user@cloud-instance-ip.totallyreal.com
  2. You can now test the connection with your own configuration, for me it was running on port 7080:
    1. ssh -R "[::]:7080:localhost:7080" -N root@23.239.2.79
    2. This sets up a reverse tunnel from the public machine’s port 7080 to the private machine’s port 7080
  3. Using SSH directly works, but for a more robust and persistent connection we’ll use autossh. It’s basically just a managed SSH connection, perfect for this use case.
    1. To make this easy, we’ll use a SSH client config file. In ~/.ssh/config, put the details of your connection. Mine, for example, looked something like this:
HOST unifi-video-tunnel
    HostName            li838-22.members.linode.com
    User                toby
    Port                22
    IdentityFile        /home/toby/.ssh/id_rsa
    ServerAliveInterval 30
    ServerAliveCountMax 3
    RemoteForward       :7080 localhost:7080
    RemoteForward       :7443 localhost:7443
    RemoteForward       :7445 localhost:7445
    RemoteForward       :7446 localhost:7446
  1. To add yet another layer, everything will be easier with a service wrapping around the autossh session. In Ubuntu 16, that would be a systemd service.
    1. Add a .service file that reflects the application to /etc/systemd/system/ – I called mine autossh-unifi-video.service
    2. Flesh out the service to invoke autossh for you
[Unit] 
Description=Make Unifi Video available remotely
After=network.target 

[Service] 
User=toby
ExecStart=/usr/bin/autossh -M 0 -N unifi-video-tunnel

[Install] 
WantedBy=multi-user.target]
  1. As with any service, get it moving:
    1. systemctl daemon-reload
    2. systemctl enable autossh-unifi-video.service
    3. systemctl start autossh-unifi-video.service

And that’s pretty much it! The reverse tunnel should be working, and everything you’ve mapped should be exposed to the public.

Improvements

  • Add a separate user for autossh
  • Use a VPN solution instead of this

19 thoughts on “Xplornet and its confounded double NAT”

  1. wee_fla

    I ran into this double nat issues when I switched to their lte service and wished to connect to my home security system when I was travelling. The work-around I implemented was to drag an old laptop that was gathering dust from a drawer. Set the laptop up with teamviewer. Enable teamviewer for unattended access and to automaticially start at boot-time. I authorized the laptop that I take with me when travelling to have access to the old in-house laptop. It all works great and I can access my security cameras.

    Reply ↓
    1. tobymurray Post author

      That’s a neat solution – handful of questions:
      – How stable is your connection with TeamViewer? Does it drop when you’re out and about at all?
      – I guess with this solution you’re effectively watching the video stream through a remote desktop session – does that suck or is it passable?
      – Have you tried a mobile client? Does TeamViewer reconcile resolution differences nicely?
      – Is there a “pairing” you have to go through to authorize a given device, or could you add a new device while away from home?

      Reply ↓
  2. RedDirtRiders.com

    I used to use a DDNS service duckdns.org and while it is updating my WAN ip correctly, traffic just can’t get past the xplornet front end, and I found this blog which addresses the reason why. Thanks for clarifying what is going on and a possible solution.

    Reply ↓
    1. RedDirtRiders.com

      SSH access over ngrok:
      $ ngrok tcp 22
      Response from ngrok:
      Forwarding tcp://0.tcp.ngrok.io:19856 -> localhost:22

      To access from a remote computer or device:
      ssh user@0.tcp.ngrok.io -p19856

      I run this in a “screen” session so it remains running in the background until reboot.

      I hope this helps another Xplornet user gain access to their machine. 😉

      Reply ↓
  3. Genny

    Is it possible to do the private computer config setup on a router (maybe running DD-WRT), without a permanent computer located at that end, or do you have to have a computer? I want to look at cameras connected to the router, but currently no computer permanently resides at the location.
    This is a bit above my head, but I’m trying to learn.

    Reply ↓
    1. tobymurray Post author

      I believe the short answer is “yes”. If you have a video stream heading somewhere, you can redirect it with a router (e.g. everything from 192.168.1.32 gets forwarded to the VPN server). In this case you’d have to set up your router as the VPN client. In a lot of ways I think it would actually be preferable, as the camera wouldn’t know it’s being forwarded somewhere, so it may simplify some of the device set up.

      A quick google shows something like this: https://charleswilkinson.co.uk/2016/05/14/selective-routing-using-ddwrt-and-openvpn/

      Reply ↓
  4. oucil

    Not sure if you’re still dealing with this, but here’s what I’ve found…. there are only a small handful of ports that they “reserve” which you cannot forward on through the double NAT, 80 (http) is one of them unfortunately. If you’re stuck using 80, then your other solutions are reasonable, otherwise, you can simply use “double port forwarding” to get around this. Set up port forwarding for an outside port like 8000 on the LTE router, then forward it to 8000 on the internal network, then set up another port forward on your own router from 8000 to port 80 and the specific IP of your machine. I’ve used this for HTTP, Plex, RDP, SSH, etc. and am able to use all of these services without issue now.

    Reply ↓
    1. tobymurray Post author

      Interesting. Naively, it seems like it shouldn’t be possible though… You’re saying there’s a way to have public traffic to an IP address that is shared across many Xplornet customers forwarded to your own server? Have you actually done this? What happens if another user with the same public IP sets the same port forwarding rule, do both users get the traffic?

      I managed to screw up my Internet access by mucking with the LTE router, I’d be interested in a bit of a walkthrough if you know how it works.

      > am able to use all of these services without issue now
      To be clear, you’re able to use these services over the public Internet without any tunneling? You go to oucil.ca:8000 and it sends traffic to your locally hosted server?

      Reply ↓
    2. Genny

      Generally Xplornet and other LTE type providers (like Sasktel Fusion) in rural areas block any and all inbound internet requests. No matter how good your port forwarding is it won’t get to you. DDNS does not work. double port forwarding doesn’t work. That is why you have to set up the tunnel. I was able to get a static IP from Sasktel, but Xplornet would not provide one. Before I set up the static IP on Sasktel, the WAN address on my router was a private range address. “What’s my IP” gave a completely different address I did not see on any router I had access to at all. There is no way to double port forward through that.

      Reply ↓
  5. Clement Morton

    Just a quick comment on this – After much digging around and experimentation. I have found that.

    Xplornet gives out a IPV6 /61 & an IPv4 CGNAT /29

    This is 8 ipv4 CGNAT addresses in the range of 100.64.0.0/10 subnet – I believe this is standard to RFC 6598 – https://datatracker.ietf.org/doc/html/rfc6598

    The IPv6 range is a /61 which will give you 8 /64 prefixes. I believe that is around 147,573,952,589,676,000,000 globally routable IPv6 addresses.

    I have successfully used both pfsense and edgerouter to route ipv6 addresses in and out of my LAN.

    If I find myself stranded in IPV4 only land. The solution I have devised in to utilize a cheap VPS that has both IPV4 and IPV6.

    Reply ↓
    1. Mack

      Clement,
      You may have just changed my mind, I’ve spend the last month back and forth with Xplornet Sales and their “IT rep” to try and get a guarantee that if I sign up they’ll give me a public IP… short answer is they won’t unless you pay for “commercial” then I think they’ll do a p2p link and starts at $500/month. I didn’t even think they would have v6 deployed to their wireless gear yet, this changes everything Thank You. I’m going to email them back and confirm they are doing IPv6 in my area. Little droplet with 4-6 tunnelling and done…

      Reply ↓
    2. Josh V

      Hey Clement,

      Could you give me some details on how you were able to get the IPV6 working with Xplornet LTE?

      I have been reading about it for a long time and cannot seem to piece it together.

      Thank you

      Reply ↓
  6. Bill

    Hello, thanks for this post, it was very helpful. I have been stymied by Xplornet’s double NAT for so long I didn’t think there was a relatively easy solution until I tried your NGROK suggestion. It works like a charm.

    My question, though, is how secure is it for a redirect for my VPN Server? What I mean is that I have a Synology NAS running an OpenVPN server on my LAN and use NGROK to forward the OpenVPN port TCP (UDP not supported on the free tier) 1194. Is that secure? I am armed and dangerous with enough networking knowledge to get this working and have certain security aspects working, but I don’t know if I have blown a hole in my whole system of trying to have end to end encryption if I throw NGROK in the mix. Thanks!

    Reply ↓
    1. tobymurray Post author

      Prefacing this with the disclaimer that I’m just some person on the internet, I have no qualifications to give network security advice

      > Is that secure?

      I think the answer to that could be researched with “is a proxy secure?”. By definition, all your traffic will be visible to ngrok in whatever fashion it is passed to them. Assuming you’re visiting sites with TLS (HTTPS), the actual content would not be available but there is a significant amount of metadata being shared. Actual end-to-end encryption will not be impacted (otherwise it wouldn’t be end-to-end). If ngrok were trying to man-in-the-middle your traffic you’d see that as all sorts of TLS errors for everything you try and do. If you’re at all shy, hosting your own HTTP tunnel on a locked down ultra cheap VPS + static IP (~5$/month) seems like it’d be a step better, but you’re still trusting someone else.

      I guess the one thing to be mindful of is ngrok exposes whatever you expose with ngrok – if you can access something on your local network via ngrok, anyone else can as well unless you’re actively preventing it. So long as you’re only exposing the OpenVPN port, I would think that’s just fine and your risk is roughly “how secure is OpenVPN?”. It becomes awfully convenient to start exposing other applications that may not have as secure a public interface though.

      Reply ↓
      1. Bill

        Thanks so much for the reply, I totally understand the disclaimer on dispensing security advice on the internet. 😉 Yes, I am only exposing the OpenVPN port via ngrok. You have at least given me a bit more confidence that I haven’t ripped a hole in the VPN tunnel process by using ngrok. I will only really be leaving the port open via ngrok for when I am away. Thanks again.

      2. Bill

        Hey, it’s Bill bugging you again. Thanks so much for the information you are so freely sharing (I feel like I should buy you a beer or something 😉 ). My NGROK solution is working very well, but I am trying to move to the other solution you shared (the VPN edition https://technicallyrural.ca/2017/11/05/xplornet-double-nat-vpn-edition/) and have successfully set that up using WireGuard instead of OpenVPN in a Digital Ocean Droplet. Yay!

        So right now I have a Digital Ocean droplet that I create a VPN tunnel to using WireGuard. I have successfully forwarded port 80 to expose the webserver on my internal computer. As I said this works great. The problem I am running into is that I don’t need to use it to expose a webserver, I need it to expose the VPN Server port and my VPN Server is not on the same device as the VPN tunnel. The end game is to be able to VPN into my home network from outside.

        My current NGROK solution is pretty simple. I have a Mac computer that run the NGROK command which forwards the traffic to my internal Firewalla (firewalla.com – Linux-ish box) that runs my VPN Server. The obvious solution (I would assume) is to use the Firewalla box to run the VPN client to the Droplet and just forward the correct port, but that is beyond my abilities to setup as the box is a closed system and the UI doesn’t seem to let me.

        My question then is, is it possible to forward the VPN port from the Droplet directly to the Firewalla box on my internal network the same way the NGROK command does it? I.e. the NGROK command runs on my Mac but the forwarding happens between NGROK directly to my Firewalla box. Can the VPN Tunnel run on my Mac but forwarding happen direct to the Firewalla box?

        Any thoughts you might have on this would be greatly appreciated.

        Thanks!

      3. tobymurray Post author

        > is it possible to forward the VPN port from the Droplet directly to the Firewalla box on my internal network

        I _think_ so, but I may be misunderstanding. Your droplet will have connectivity to your LAN via the VPN. With a very hand-wavy explanation, the connectivity to an arbitrary public IP vs. a LAN address is equivalent from the perspective of the droplet. I would expect you’d be able to forward the same way as proposed in the linked post.

        Filling in your own values, on your droplet something like:
        sudo iptables -t nat -A PREROUTING -p tcp –dport [VPN_PORT] -j DNAT –to-destination [FIREWALLA_IP]:[VPN_PORT]

        In other words, forward traffic received on the VPN port of the droplet to the box-running-Firewalla’s IP address with the same VPN port.

        As a note, you’d have to ensure the Firewalla box will accept this incoming traffic as well so that both the transmitter and receiver are happy.

  7. Bill

    Thanks for the fast reply. I tried something similar like this:

    iptables -t nat -A PREROUTING -d 37.XX.XX.XX -p tcp –dport 1194 -j DNAT –to-dest 192.168.1.5:1194

    This didn’t work, but this seems to mixing IPs between the VPN and my local network (I am sooooo not a network guy ;)) The destination IP seems like it needs to be something like 10.8.0.2 and not one from my internal network 192.168…..

    Side note, I am also assuming this should work the same whether UDP or TCP forwarding? As in I could forward either the same way?

    Reply ↓
  8. Neil

    No IPv6 here behind xplore’s LTE CGNAT modem/router, SE Manitoba.

    I’ve come up with Twingate as the easiest zero-trust remote access system for me to set up and use. The free trial for up to 2 simultaneous client connections never expires, so free for my usage connecting from my Ubuntu laptop or Android phone. Works on iphones, macs, windows, android clients…

    I have a linux desktop running as my TV media server 24×7 anyway so adding their Connector service on it works perfectly.

    Basically I’ve set Twingate up so all I need to do is start the client on my device when away from my LAN (but internet connected obv) and magically get a peer-to-peer tls encrypted lan connection to my home network. In my case I set the entire subnet, that’s visible to my media server connector, as available to the clients. So I can go browse files, or run RDP sessions, play plex shows, as if there at home.

    The Twingate client access can be strictly controlled (ZERO-TRUST) down to client security checks, 2-factor auth, individual remote ip address and ports restrictions, and more.

    /

    Reply ↓

Leave a comment