Out of the box the two switches shipped with firmware 1.0.9.16014 from 07/24/2020. The latest version is 1.3.0.527638 which is from 11/23/22. There are two processes for upgrading the firmware. One is automatic and the other is manual. Unfortunately I was unable to get either of them to directly upgrade using either method. The automatic upgrade fails and says there is no internet connection and the manual process just hangs on updating. However, using the following process I was able to get them upgraded to the latest version.

1. Download 1.0.12.17336 (1.0.12 build 20210506) from here (no idea if it has to be this file but it was the first one I grabbed and it worked)
2. Apply the upgrade through the UI under Firmware Update > Firmware Update > Browse > {File.img}
3. The switch will process the firmware upgrade and after a few minutes restart
4. Login to the switch and click Firmware Update > Live Update > Check for Update
5. This should pull 1.1.0.29653 and install it (switch will process it for a few minutes and then restart)
6. Login to the switch again and it should automatically tell you that 1.3.0.527638 is available
7. Follow the prompts to install 1.3.0.527638
8. Done

I did not see any immediate differences in the UI by moving from 1.0 to 1.3 and I didn’t find or look for a corresponding release note document to tell me what had changed but I’m sure it’s better!

The post QNAP QSW-M408-2C Firmware Upgrade appeared first on Jeremy Gotcher.

Install Docker

Reference: https://docs.docker.com/engine/install/ubuntu/

sudo apt-get remove docker docker-engine docker.io containerd runc
sudo apt-get update
sudo apt-get install \
    ca-certificates \
    curl \
    gnupg \
    lsb-release
sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io docker-compose-plugin
sudo docker run hello-world

Install Portainer

Reference: https://docs.portainer.io/start/install/server/docker/linux

sudo docker volume create portainer_data
sudo docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest
sudo docker ps

Pi-Hole

Reference: https://www.reddit.com/r/portainer/comments/wdm7j6/portainer_request_failed_with_status_code_500/

Before deploying pi-hole, run the following commands:

sudo systemctl stop systemd-resolved
sudo systemctl disable systemd-resolved

Transcribe the following .yaml to Portainer settings: https://hub.docker.com/r/pihole/pihole

Optionally I set Hostname and Domain Name under Network.

The post Ubuntu 22.04, Docker, Portainer, Pi-Hole appeared first on Jeremy Gotcher.

VM Config
2 vCPU
4GB RAM
50GB boot disk
NIC1 untagged vm traffic
NIC2 iSCSI A – VLAN 100
NIC3 iSCSI B – VLAN 101

The goal is setup a multipath disk using the iSCSI A and B NICs to a SAN and ensure that the mount persists after a reboot. Working through multiple articles, I had some issues.

Setup Process

I intend to do a full write up of the steps I followed but here are the references I used and changes I made along the way.

I followed these IBM Steps. However this line needs to be changed.

#iscsiadm -m discovery -T sendtargets -p <target IP> # the T didn't work for me
iscsiadm -m discovery -t sendtargets -p <target IP> # lowercase t

In order to mount on reboot I followed the fstab step from here but adapted it for my needs

/dev/mapper/mpatha-part1 /mnt/ubuntu01-vol01 ext4 _netdev,rw 0 0

The post iSCSI Ubuntu Initiator appeared first on Jeremy Gotcher.

Process to Change iSCSI Paths in TrueNAS

Edit /conf/base/etc/iscsi.conf and change the IP in t0 and t1 to the new IPs. My IQNs remained the same.

Run the following commands to manually connect to the new IPs and then check paths:

iscsictl -An t0
iscsictl -An t1
iscsisct -L

Run the following commands to remove the paths that are being abandoned and check the paths:

iscsictl -R -p 192.168.1.151
iscsictl -R -p 192.168.1.152
iscsictl -L

Check the pool and see if the iSCSI disk is online. Then write to a file.

References

man iscsictl

people.freebsd.org

The post Working With TrueNAS Initiator iSCSI Paths appeared first on Jeremy Gotcher.

But why you ask? Well, what if you happen to have a spare SAN laying around and the only thing in your Home Lab that is useful for serving data is TrueNAS? While this is very much an edge case, it happened to me!

TrueNAS Background

Just like Synology and QNAP you can setup a pile of disks, protect the data on those disks with a form of RAID, and then serve up the data over NFS or SMB. This is a very basic use case of these three products. In addition to serving data through NFS and SMB you can use these devices to present storage over iSCSI.

Of course TrueNAS, QNAP, and Synology can do much more than just serve data but those features are beyond the scope of this article.

iSCSI in Two Paragraphs

Before we talk more about iSCSI let’s briefly cover what a Target is and what an Initiator is. This is mostly for me because I always forget!

The Target is the SAN. Think EMC, NetApp, Nimble, 3Par, Pure, etc. The Initator is the server that is going to do something with that storage. Think servers running ESXi.

Why iSCSI?

Why would you want to serve data over iSCSI in your home? This is a good question. If you are running a home network with the intention of serving media to Plex and setting up file shares for Windows you are going to be better off using NFS and SMB. However, having the ability to present volumes over iSCSI is helpful if want to set up shared volumes on ESXi or experiment with block storage on Windows Server or Linux.

TrueNAS has all the GUI options you need to setup volumes on TrueNAS to serve through iSCSI which could be consumed by ESXi or Windows.

Sharing Local Disk and a SAN Disk With TrueNAS

In the following diagram the ZFS Pool that is sharing Data, Movies, and etc. is 6 local SATA disks which is sharing data to Windows, Linux, and Plex over NFS and SMB. The ZFS “Pool” is a single large iSCSI volume that is sharing MoreData to Plex over NFS.

Setting up TrueNAS as an iSCSI Initiator

I followed the process from this post to get TrueNAS running as an iSCSI initiator.

From the TrueNAS shell I created /conf/base/etc/iscsi.conf and entered the following information. TargetAddress and TargetName will be specific to your setup. CHMOD 600 the file when complete.

t0 {
  TargetAddress = 192.168.1.151
  TargetName = iqn.2010-06.com.etc.etc.etc
}
t1 {
  TargetAddress = 192.168.1.152
  TargetName = iqn.2010-06.com.etc.etc.etc
}

Next I added these lines to the end of /conf/base/etc/rc.conf:

iscsid_enable="YES"
iscsictl_enable="YES"
iscsictl_flags="-Aa"

Then I found the following line in /conf/base/etc/ix.rc.d/ix-zfs…

# REQUIRE: hostid mountcritlocal

… and changed it to:

# REQUIRE: hostid mountcritlocal iscsictl

Finally I rebooted and ran this command…

iscsictl -L

…which showed me my iSCSI targets and confirmed that I was connected.

More information can found here.

Remaining Issues

There are a couple of issues to sort out. The first is a timing issue between TrueNAS and the SAN when power is lost. The second issue is that my console is displaying the following warnings which I have yet to research.

iSCSI on TrueNAS Summary

Issues aside, the configuration works well and the volume shared over iSCSI has been as performant as the local ZFS Pool. In hindsight it might be preferable to use TrueNAS tunables rather then modifying the files directly. This article shows how to do that but I have yet to try the steps.

The post TrueNAS as an iSCSI Initiator appeared first on Jeremy Gotcher.

This morning, after a snow storm, I had a power outage and discovered a few issues that I hadn’t counted on. I have to remind myself: “whenever I build something, pull the power and plug it back in”. Only then can I know that it’s working.

The Orbi’s are Purple!

The First and obvious issue is that my Orbi’s were purple. I don’t know what this means as most everything with my Orbi’s is a black box to me. This is partially due to Netgear’s software but more because I’m to stubborn to dig in deeper with them. All I know is that when they are purple I have to pull the power and plug them back in. I sorta hate the Orbi’s. I mean when they work they do what they are supposed to but they tend to be flakey for me. And they have a dumb name.

Pi-Hole Fail

Next, my pi-hole didn’t come back up. Even after power-cycling them it didn’t come back up. I had to switch to regular old 8.8.8.8 and 8.8.4.4 like a chump and experience elevated levels of ads. It wasn’t until later in the day that I discovered there was no link light. I still haven’t figured out what’s up but honestly I’d like to ditch the pi and move pi-hole to a container.

ESXi and VM Auto Start on Restored AC

Then I realized that the mini PC that I just got ESXi 7.0.2 installed on wasn’t powered on. I have a write up coming soon for this build… whatta pain that was. This wasn’t that big of a deal because the only thing it has on it is a single VM running Ubuntu and Docker/Portainer. That’s literally it, I haven’t decided what containers to put on it yet. Pi-hole is obviously now on the list but I’d like to get a second host and vCenter running so that I can have HA. Solving this was easy. I set the BIOS to restore power on AC loss and then set the VM to start automatically after the host started.

The Actual Issue to Solve

On a power loss, TrueNAS boots quicker than my SAN. The iSCSI backed block storage pool cannot come on-line. My Jails, which serve data from the iSCSI SAN pool, won’t load. For now the only solution I have is to reboot TrueNAS after the SAN comes back on-line.

Possible solutions to investigate:

• Automatically delay booting TrueNAS – BIOS Setting?
• Figure out a way to rescan iSCSI through TrueNAS
  • iscsictl command?
  • Figure out a boot script (crontab?)
• Figure out a way to periodically rescan iSCSI or rescan when a volume doesn’t come up
  • This would be a variation of the above solution

Update 12.17.2022
Following this research I think that in order to reconnect to an iSCSI target after a power fail you would run the following:

iscsictl -An t0
iscsictl -An t1
iscsictl -L

Now the question is how to automate that after a power failure.

OK but why tho?

I had been patting myself on the back lately as I was able to configure TrueNAS as an iSCSI initiator so that I can serve up data from a SAN using TrueNAS to manage NFS sharing. This is absolutely an edge case that may leave you scratching your head and asking “ok but why tho?”

I plan to write that up soon but for now just assume it’s an awesome idea that totally makes sense!

Update 12.17.2022
Here is the write-up and you can still assume that it is an awesome idea that totally makes sense.

The post TrueNAS, iSCSI, and Power Loss appeared first on Jeremy Gotcher.

In the Beginning…

Going back as far as 1995 I have had some sort of network in my home. I shared an ISDN connection with two roommates using a Farrallon Netopia ISDN Router and 10Base2 (Thinnet or coax as we called it) networking. It worked well for awhile but after many sessions of troubleshooting cables, terminators, and T-Connectors we upped our game and bought a hub and ethernet cards.

Having ISDN, Cable, DSL, and now Fiber connected with a router and switches carries on to this day. Add in some wi-fi and we’re not really talking about anything special. Just about anyone on-line these days is going to have this configuration in their home.

In 1999 I started a new job as a Server Coordinator at a bio technology company. I was disappointed that my title wasn’t Server Administrator or Network Administrator but what are you going to do? In this role I took care of servers, networking, phones, desktops, printers, scanners, and all the software in between. The achievement I was most proud of was setting up a T1 and brand new Cisco 2500s (probably 2501) with RIP running between two sites. As you can see from the following pictures, the bar was pretty low. Yes, these are photos of my server room.

Around 2002 I had left that job and moved on to a State job where I was the Desktop Team Lead. My focus was building Windows NT 4.0 Workstation Ghost images, evaluating software, and writing install procedures.

I was having a good time and I was good at what I was doing but I wanted to work on servers. I had a bit of server experience from the previous job and I was also enrolled in a technical college where I was taking IT related classes.

Server Tinkering at Home

The prevalent server job at that time (from my observation) was a Windows Server administrator. Novell was still around but dying, Unix was rare, and Linux wasn’t being seriously considered at the organizations I was looking at. So, I took an old desktop computer and installed Windows 2000 Server.

I did all the Windows server things that one should do such as promoting it to a Domain Controller, setting it up as a print server and creating users. I even joined all the computers in my house to the domain. This was my first Home Lab.

Next up I installed Red Hat Linux on another old computer. I don’t remember why I installed that Linux server. Most likely I wanted experience installing it and experience tinkering at the command line. I am sure that I set it up as a SSH server and likely as an Apache server that probably served up a lame webpage. Security professionals may wince, but I definitely opened RDP, SSH, and HTTP to their respective servers. Later I would learn how to tunnel through SSH but in the bad old days of 2002-2007 I just opened stuff up directly. I don’t believe that my Windows server was ever compromised but who knows…

Managing a Windows Server and domain got old after awhile so I eventually took all of that apart. From there I always maintained a Linux server of some sort. The Linux server would have been for SSH, experimentation with Drupal and WordPress, Samba, and tunneling to get access to everything in my home.

Adventures in Media Servers

Around 2008 I bought a Popcorn Hour A110 and started down the path of building various media tanks. One solution was a large USB hard drive attached to whatever router I had at the time. Another solution I had was to put a large hard drive in the Popcorn Hour and move media to it. My recollection of the Popcorn Hour was that, when it worked, it worked well. There were some codec types that it had issues with, the UI was less than stellar, and it was slow. I remember “upgrading” the Popcorn Hour to an HP Laptop which I install Windows XP Media Center Edition on. When I wanted to play a movie I put the laptop on it’s docking station and plugged in the digital audio cable. The laptop could even be controlled by my universal remote control!

I switched over to XBMC and I know that a lot of people loved XBMC but I hated it. I’m sure I was “doing it wrong” but I could never get 100% of my media scanned or the metadata downloaded and I was not a fan of the UI. What I did like was Boxee (which I know was fork of XBMC). I installed that on some hardware about 6 months before they pulled the free version and went all in on the Boxee Box. Eventually I landed on Plex.

Modernizing My Game

Ok, so what does all that talk of media servers have to do with a Home Lab? As I said I started with an external USB disk attached to a router. Then I moved that to a local disk on the Popcorn Hour. Eventually I added more disks to Linux server and used it as a makeshift NAS (no RAID though). As my media grew and the services I wanted to use to maintain that media also grew so did my “server” and my storage. Eventually I would install FreeNAS on an old Athlon system. FreeNAS on that little Athlon which probably only had 8GB of RAM was woefully underpowered. The experience I had on it was a nightmare. However, this gave me my first taste of FreeBSD jails which it leant itself well to better understanding container concepts.

Anyway, the experience was so terrible that I wiped FreeNAS off the system and installed some version of Ubuntu. The data on the FreeNAS ZFS pools was mostly disposable but would have been time consuming to recreate so I was delighted when I learned that I could mount ZFS on Ubuntu. So, out of laziness I ran Ubuntu with two large ZFS pools that had all my data on them. Recently (as in earlier in 2022) I did actual research and purchased equipment specifically for a TrueNAS (renamed from FreeNAS) build. Despite missing that the NIC was not entirely supported (damn you Realtek!), the build came out great and has been very performant. However… will I keep it running? Stay tuned.

So, that takes care of the back story. Now on to what I’m doing next.

The post Home Lab 2022 Part 1 appeared first on Jeremy Gotcher.