Quick Blurb

Homelab Ubiquiti Mesh Link

In my apartment I needed to get wired networking with VLANs across the apartment. I didn’t want to run a wire since I thought my roommate would not appreciate that. I wanted to have a switch near my desk, that allowed different devices I have like file server, desktop, and a few other things to have a wired link; then, connect to the modem/firewall and rest of the networking gear across the apartment.

Long story short, I ended up using a trick I didn’t know would work till I tried it. I have 2 x UAP-AC-M, they work decently well, topping out at 867Mbps and 2×2 MIMO; as well as being able to get them on sale in a 2 pack for a decent price made them a great deal. I have run 1 of them for 4 years as my main access point. Then when I wanted to get this wire connection in a new room configuration I tried to do a wireless uplink to the second one. This makes it mesh with the first access point. Now the important item I don’t seem written anywhere but works well (caveats below):

Ubiquiti access points in wireless uplink/mesh will bridge that network to the wired port on the device

This means if you have a trunk port going into your original/base mesh AP, you will have the same trunk port coming out the other end. This also means anyone who is running mesh points, and hasn’t secured the wired port may want to think about doing so. I am will skip over HOW to set this up, Ubiquiti has a good guide https://help.ui.com/hc/en-us/articles/115002262328 to walk you through it, and most APs can do wireless uplink at this point; this is more about saying it can be done, and works well from my experience to anyone thinking about implementing this or wants a solution for their home/apartment that is not powerline networking. The APs I have are 2×2 802.11AC, I’m sure with a 4×4 AP like the AC-Pro as your base you may see better performance on higher trafficked lines.

This setup has worked well for me for over 6 months now, I can easily hit the 300Mbps I get from my internet connection on a desktop plugged into this meshed AP’s port; I also get 6ms pings to servers while playing games. You get the benefit of real commercial grade antennas and radios in the APs you are using instead of a tiny wifi chip in a laptop, desktop, or device. This also lowers the number of wireless devices (since all the wired devices would have been wireless instead). I also disabled the secondary AP from hosting any of the SSIDs I have in the apartment, so it just works as a wireless uplink. My apartment is not big enough for 2 AP’s for devices.

Caveats

I am looking to move away from this setup for a few reasons. It has worked well and if you are in a pinch I would recommend this setup much more than powerline networking which I have also tried and used several times. I am hoping to move to 10gb/s networking at home with my growing homelab setup; thus, no more wireless link. The other limitation that 99% of people probably would not care about is that you can not do jumbo packets over wireless, so that means it can not be done from all I have read over a wireless link of this type.

Network Topology

The first caveat is that this configuration slightly confuses the access point when it first starts up. The first 60 seconds or so when the access point is online it will think the wired connection is its uplink and attempt to ping out over it. After that it realizes it cant hit anything and will go to wireless uplinking. Sometimes everything just works then, sometimes I have had my switch be confused about where traffic should go and had to power cycle it; in this case it was just a Netgear Prosafe switch with VLANs, not especially smart, but not the dumbest switch. This is similar to a enterprise networks re-converge time when a link is downed. Overall it is rarely a problem and these APs are solid and can go months between restarts, but this is something to lookout for.

Remember that if a Ubiquiti AP cant get an IP, then it doesn’t broadcast SSIDs; this is important since if the base AP boots (like after a power outage) and doesn’t get a DHCP address quick enough, it wont broadcast, then the mesh side will never find an uplink to connect to.

Management

With the earlier mentioned topology issues you can run into, that can make management difficult. You need to make sure the base side of the network is stable. You can get into a position where you did a bad config push or a setting is wrong on the secondary/mesh side and the only way to fix the config is bringing that AP back to the original wired network and pushing a config to it, before the secondary AP can go back into wireless uplink mode.

US Patent US10530642B1

One of the projects I currently work on at work, and have for the last few years is how to go from a blank stack of servers to a fully configured cluster with my companies software running on it. While some projects were starting and getting going in the open source field when I started this project 5+ years ago, a lot of them kept rewriting their API every minor version rev. That started my down a path that has now become a decently large internal network booting infrastructure, and managing interconnects to our inventory system as well as other systems such as Tenable Nessus. I recently was awarded my first patent! This one is specifically about how my system interacts with the inventory to dynamically assign systems as they come online to clusters.

https://patents.google.com/patent/US10530642B1/en?oq=US10530642

My part of the code was all written in Java and continues to evolve as a platform, I hope to open source a good amount of it down the road. I started the project by reading the RFCs for DHCP/PXE and then writing code. I have grown to enjoy writing libraries and some project this way of adhering to the standard (more on that some other time). The general platform can handle ProxyDHCP PXE booting, and then uses iPXE to create menus and boot systems. I spent many hours debugging different vendors PXE code and BIOS vs UEFI to get all the systems to work. The platform now supports plugins for many different aspects of server configuration.

I could write page about small details I have learned a long the way; one issue that has been driving me crazy recently, if you want to ProxyDHCP instead of using your main DHCP stacks these days is Secure Boot. iPXE does not have a Secure Boot signed image, I have tried to get Microsoft to sign it but they will not unless you are selling a product using that the sign iPXE. I am not I just wanted it for internal use. That means you may want to use grub2 as your loader, but there is a bug that has been outstanding for over 6 years and makes ProxyDHCP with grub basically impossible, https://savannah.gnu.org/bugs/?55636 which is sad.

Dell Inspiron 3050 Ram Upgrade

This is a short post about a Dell Inspiron 3050 I upgraded a little bit ago. This is a tiny pc, similar to an Intel NUC. Its a Intel Celeron, and came with a 32GB SSD. I got it for around $150, with an Office 365 subscription; thus it was worth it to me. It came with 2GB of ram, and a 32GB SSD, these days those are not expensive to swap; I wanted to swap the components for 8GB of ram and a 512GB SSD. Below is a short guide with some photos of opening this thing up.

First we needed to remove the case, this involves flipping it over, and taking the 4 screws out that are in the little feet.

Inspiron Bottom

That gives you access to the RAM DIMM. Easy to swap if you want to do just that. Now there are 4 screws at the outer corners, those come out then the board can fold out keeping the antenna and other cables connected. Flipping that over and putting on the table shows the CMOS battery, as well as the SSD.

Underside of board

After replacing the SSD its just a matter of flipping the board back onto the posts, and screwing it all back together. Fairly easy to do, but I couldn’t find a ton of photos online so I thought I would put some up. I ended up installed Hyper-V 2019 on it, the box is fairly slow with its Celeron dual core J1800 processor; but can run a Linux VM or two. Plus its a cute little computer that uses very little power.

One last note about putting it back together, there are little metal spokes that stick out from the top metal mount, those need to line up with the motherboard the system wont go back together correctly.

Little metal spokes

Sonnet Labs Firmware Poking

A few weeks ago I got the Sonnet Labs, Sonnet One long after we thought the Kickstarter was just dead. A pleasant surprise, until it was missing any instructions, and the setup link didn’t go anywhere. I ended up writing up what I could piece together on the web into an Unofficial manual of sorts, but then decided I wanted to know a bit more about the firmware currently on it.

Finding bits like the FCC manual, which didnt have much (https://fccid.io/2AN8Z-SONNET/Users-Manual/user-manual-4003660). Then someone on a comment back in the Kickstarter’s past mentioned a draft of a manual was put up (https://www.dropbox.com/s/avmes7rhanx2vli/Sonnet%20User%20Manual%20v0.4.pdf) that gave some actual guidance about the device, but then one user wrote on a forum where he was early testing the device that the devs had given him SSH access to control the backend of the device.

Now I was interested! You can get SSH and control the whole device? How does one get that. First I looked at the code of the web app that was running, and just wanted to see if there were any admin pages I could click and just not see. Nothing big stood out, except the code on the device, and the code on the GitHub looked a bit different than each other, and there is not even a README about how to get the repo up and working.

Running nmap against the device just displayed the few ports we knew it had to have open: web server, DNS, DHCP.

That’s when I saw a reference for http://repo-test.sonnetlabs.com(backed up on archive.org), a place where all the different firmware versions has been stored. Some are marked “stable” some are “beta” or “alpha”. With a bit of searching around I found a site that walked through easily extracting OpenWRT firmware. After a quick Brew install on mac, I was able to binwalk the file and open the firmware that I seemed to have on my device, vs one of the beta ones. Looking around, its a fairly standard OpenWRT firmware with some tweaks done for the long range radios. It also has services like dropbear for SSH. In the beta/alpha releases they are missing one line that the stable ones has: “option enable ‘0’” in the /etc/config/dropbear file disabling SSH.

Looking more, there are some init scripts the system starts, one for the app, and another for the radios. The code is actually decently documented,

If we (the internet) had a build of the firmware with SSH enabled, it would make checking and seeing information about the mesh much easier (since the UI seems to have no indication of what is going on with that), there appears to be a backend app for managing the mesh. This is used in the startup of the radios:

    ${MESH_CONTROLLER_CMD} reset
    echo "mesh: using channel ${channel}"
    ${MESH_CONTROLLER_CMD} set NCP:Channel ${channel}
    ${MESH_CONTROLLER_CMD} set Network:Name Sonnet
    ${MESH_CONTROLLER_CMD} set Network:PANID 0x4700

Most of the operations for the app seem to handled by /usr/bin/sonnet_server. And the web part of the app is in /usr/share/sonnet_app where it has all of its node modules setup, and cordova.js for some offline stuff. This code is much different than the one on the Github. Which makes the timeline/code confusing. If they shipped around October, they had to have the firmware more or less finalized a while before that. How do we have this one code base with some stuff from August 30th, then this firmware from 18-Jun-2019.

Anyway that was a bit of playing around with it. What I would like is Sonnet Labs to put a firmware out for us with SSH on, and everything else stock. Then go and add documentation to the repo, perhaps a easy way to iterate on the code and put it on the device. After that, if the community wants to help make the app better and perhaps put a UI on the actual mesh part. Or Sonnet Labs can update their product. The fact that http://repo.sonnetlabs.com has a bunch of 0 byte files last updated May 2018 does not fill me with hope…

Sonnet Labs, Sonnet One Unofficial Manual

Today a device I forgot I ordered arrived, the Sonnet Labs One, a mesh point that reports to connect two places over lower frequency radio to allow large mesh networks. It works by getting one of the units and connecting it to your local wifi. Then the other one, according to the box can be up to 1ish miles in a city, and up to 10 miles out in the middle of no where. When I got it I unpacked it, and all that was in the box was the device, a micro USB cable, and a charger. Where are the instructions? I check the other (you need at least 2, or hope your town has some other people who got them years ago on Kickstarter or Indiegogo (yes they did both) ) and that also didn’t have instructions.

Ah the top of the box says sonnetlabs.com/start, perfect! 404 page not found, with a few other links, which also 404. Looking on the Indiegogo and Kickstart pages most people are still waiting for theirs, so I guess I am too early for the website? The sonnetlabs website also shows an early mockup of the device and nothing what it looks like now. I started digging through comments, and looking at their GitHub. I have pieced together some aspects of the device and figured I would start a manual since so far non exists. And at this point, the project seems to have maybe 1 or 2 people working to try to fulfill it. I would be happy to chat with the devs.

Turning on the device:

  • The first flag contains a port for an antenna, the second has the micro USB charging port, and a micro SD card slot, the last has a USB host port to charge your devices.
  • Plug in the included charger into the middle micro USB port
  • Hold the Orange button for about 3 seconds and a green light should appear on the top of the unit

Connecting to the device, and to the internet:

  • When the unit is on, you should see a new Wifi network near by, Sonnet-XXXX connect to this with the password of “sonneteer”
    • Note: I had issues with my Mac doing this and had to manually enter the credentials
  • The web server that hosts the app takes a minute to start, if you try to go to it too fast it will fail to load
  • In a browser go to https://app.sonnetlabs.com, the IP is usually 192.168.47.1; but if you go to the IP the settings menu seemed total for me, it looks like they hard coded that URL in some locations
  • There will be an error about the certificate, this is because the device made it, accept to continue, you may have to hit “Details” or “Advanced”
  • You should see the login screen, click “Register New User”
    • This makes a user to send messages, and use the basic aspects of the device, fun fact all this data is stored locally in your browser, so the users generally don’t matter. This does secure your chat message data, because you need to use that exact browser and that exact device to ever see these messages again. If you change the address you are going to, or the browser, or anything you need to go re-register a user.
    • The Administrative password, set later, does actually store on the device and persist
  • Put in a name, username, and password
  • Once you see “Registration Successful” go back to the login screen and login
    • Note: Hitting the “Enter” key on the password field doesn’t trigger a login, you need to click “login” (at least in Firefox)
  • Now you are at the default page, “Contacts”, go to the “Settings Page”
  • At the bottom of the “Settings” page it will say “Set New Password” and “New password”, you have to put something in here to get Admin access to the device, that isn’t clear, but it is needed. This is saved to the device.
    • Note: This password is entered later in the UI and is just displayed on your screen, don’t make it something you don’t want people to see
  • Once you set a password there, you will get a real “Admin” page
  • Here you can click “Wifi” and start the process to connect to your own wifi network
  • This took a minute, then mine displayed a green check mark and I could connect to the internet, through my internet, even though I was bouncing through the device
  • Note at this time, I have gotten the two devices to use the “Chat” function, but not the mesh internet functions

Using Chat:

I was able to get chat to work between two of the devices, setting up one, then turning the other one seems to auto pair them. At this time I can not find any user interface on how to confirm/configure/see anything about the mesh. But Chat worked… so that’s something.

  • Login to the web interface
  • Go to “Contacts” on BOTH devices
  • Click the + in the top right
  • Select “From Network”
  • If both sessions are online, you should see the other user
  • Then if you can, you can click there name and this will send a request to add
  • On the second device, there should be a red dot on the + in the top right of “Contacts” screen, go to the +
  • Select “Requests” and approve the request
  • Then you can chat, I haven’t done much testing on how much caching of messages the devices do, but in my first test one device missed a message because the window timed out and was “offline” again
  • Note: If you see “Offline” as a bar at the top of the Window, your browser has disconnected from the device itself, clicking “Offline” should reconnect

I have not gotten QR codes to work, even with a very clear photo.

I took some screen shots of other screens. If you want more info, or have more info please leave a comment!

Additional Info:

I found a bunch of info poking around online, here are some notes:

Github for the app the device runs, but an OLD build: https://github.com/SonnetLabs/sonnet-webapp/

People talking about the device: https://community.gotennamesh.com/t/sonnet-devices-beta/4328/22

From what I can tell at the address above, a user states he was working with the devs and got SSH access to the device. I believe the image he had was a dev build with SSH installed, and the normal image we all have on the production units have this disabled.

The creators posted a early draft of a manual a while ago, copy below for archiving: https://www.dropbox.com/s/avmes7rhanx2vli/Sonnet%20User%20Manual%20v0.4.pdf

The FCC registered manual, very light: https://fccid.io/2AN8Z-SONNET/Users-Manual/user-manual-4003660

And the last laugh, there is a subdomain under sonnetlabs.com that the Digital Ocean server now belongs to someone else, so hilariously redirects. I give you lithium.sonnetlabs.com

FusionIO ioMemory VSL4 on CentOS/Rhel 7.5

With CentOS 7.5, ioMemory VSL 4.3.3 kernel module would no longer load; I could not get it to recompile from source either. I tried a bunch of things including moving my CentOS 7.5 box to the EL7 4.17 kernel to see if that helped me compile from source, no luck. Then I found a forum post, https://forums.servethehome.com/index.php?threads/centos-7-fusionio-users-do-not-upgrade-to-kernel-3-10-0-862-2-3-el7-yet.19760/ where they speak of patching VSL 3.2.15. Using this and some playing around I got VSL 4.2.1 to work with my system. This method may work for some later versions, yet 4.3.3 had some other code changes that were causing it not to compile, so I used 4.2.1. Below are the steps to get a working ioMemory VSl 4.2.1 for Centos 7.5; comments if it worked or didnt are welcome.
Note: I did all these steps as my user, and not as root. My card is a FusionIO ioMemory SX350
  1. Before we begin, I am assuming you have GCC, kernel-devel, if you do not; go to https://wiki.centos.org/HowTos/I_need_the_Kernel_Source to get kernel and the rpm build parts.
  2. Go to link.sandisk.com, register, and login
  3. Browse to the software download page, https://link-app.sandisk.com/Home/SoftwareDownload
  4. Download the “iomemory-vsl4-4.2.1.1137-1.0.src.rpm” source package
  5. In a terminal, change to the directory you downloaded it to
  6. Extract the contents of the RPM to disk
    1. rpm2cpio iomemory-vsl4-4.2.1.1137-1.0.src.rpm

  7. That gives us some metadata and a tar, extract the contents of the tar.gz file
    1. tar xvzf iomemory-vsl4-4.2.1.1137.tar.gz

  8. Change directory to the kernel modules folder
    1. cd iomemory-vsl4-4.2.1.1137/root/usr/src/iomemory-vsl4-4.2.1/

  9. Using your favorite file editor, edit kblock.c
    1. vim kblock.c

  10. Edit line 2592
    1. Before
      1. elevator_exit(q->elevator);

    2. After
      1. elevator_exit(q, q->elevator);

  11. Save the file, and quit the text editor
  12. Compile the kernel module
    1. make modules

  13. If that completes without errors, then install
    1. sudo make modules_install

  14. Let’s add the module to the running kernel
    1. sudo modprobe iomemory_vsl4

    2. If you have issues, you may need to do “sudo modprobe -r iomemory_vsl4” to force a reload of the module if one was already present
  15. You should now have the fio in /dev/, or after installing the utils from the Sandisk site, see the card under “fio-status”

Mini Mac on Twitter

I was very excited to see Susan Kare borrow of of the mini Macs from a friend and give it a shout out on Twitter! These kind words from someone responsible for a lot of the original Macintosh design are quite humbling. 🙂

Mini Mac