Having gone through this struggle myself, here's the cheat sheet. You want a device that uses the Linux switchdev driver and is supported by dentOS whether or not you actually choose to run dentOS on it (I run NixOS on my switch):
Switchdev support means you don't need hardware-specific userspace tools (with their own bizarre syntax to learn) in order to configure the switch.
DentOS support means the device uses a sane bootloader (uboot or grub) and the only binary blobs on the device will be the ones built into the bootloader (IntelME, Arm Trusted Firmware) and the switch firmware which will be part of linux-firmware (and therefore very easy to manage/update).
In particular, looking for these two keywords is how you make sure that the hardware vendor is staying on "their side of the line" between hardware and software. Violations of this line are endemic to 10G+ switching.
This is really, really cool and I didn't know the platform was open to the extent that you could install your own upstream Linux and just get going.
I am curious though what configuration option prevents this from ending up with software switching. I understand the mellanox kernel module was compiled and loaded, but certainly that doesn't mean that anything you do in the network stack gets converted to switch fabric code and uses hardware packet switching. How do you make sure that you don't errantly wind up with poor latency and capped throughput?
But also, short of making and selling your own networked device for whatever reason, what are the real benefits of going this approach? I can see crazy use cases for where you have full control of the network stack (but again, see my point above — how do you guarantee you are not doing this in software?) but for most purposes, especially with qsfp and fiber, how much are you really gaining by doing it on-device? What is the killer use case here?
EDIT
Upon rereading, it seems the switch is hard-coded to be hardware-switched and cannot end up in a situation where you are accidentally using software packet switching in the first place (i.e. it does not just optimize to a hardware packet switching state). But that limits what you can do considerably, to the point that an off-the-rack Juniper or CSCO or whatever probably has more features than you can do here without writing your own code to hook into the mellanox sdk?
The Linux switchdev driver is the awesome magic that says "make hardware-offloaded switching ASICs look just like software switching". It's beautiful and amazing, as you'd expect from Mellanox.
To configure Linux to do software switching, you need bridge interfaces and NICs to be made part of the bridge with the appropriate `ip` commands (or `brctl` if you're still using that).
This is common with small home routers - the WLAN and wired LAN ports (all typically appearing as one NIC) will be made part of a bridge `br0`. The four LAN ports aren't typically exposed as separate NICs so there is hardware switching going on there (some devices do let you split them out because they are VLANed internally though).
If `ip link show type bridges` doesn't show any bridges then you aren't software switching unless your drivers are lying to you.
> But that limits what you can do considerably, to the point that an off-the-rack Juniper or CSCO or whatever probably has more features than you can do here without writing your own code to hook into the mellanox sdk?
I mean, I'm not touching any mellanox sdk here, I am using the a very similar stack that someone on a "software router" would use, on a switch that can automatically accelerate it to 800G+ throughputs, while hitting a 60W power target.
You can hit some of those performance/power numbers in vendor hardware like Juniper/Cisco/Arista, however you have to also put up with their software, I (and others in my group of peers) have not had great experiences with vendor software, and in this setup I am able to patch/fix the software on my own terms.
If there is a security vuln in one section, I can fix that, and call it a day, I won't be forced to upgrade parts of the system I do not want to. I cannot do this with Juniper/Cisco/Arista always.
Yes. The switchdev "sw1p[0-9]+" ports are special; the any data the software kernel injects to them is discarded and they never emit packets to the kernel. They exist only to allow you to use `ip bridge` and `ip route` on them. So if you accidentally configure software switching on these ports no data will flow -- it will be totally obvious. You might get "no packets" by accident but you will never get "software switching" by accident.
If you really want software switching you have to use the management port (there's only one or two of these) whose name is "eth0" or "eth1" or something like that. So avoiding "accidental software switching" is really easy -- if you're typing "eth" you're doing it wrong. You can even explicitly delete this interface if you don't need the CPU to be able to snoop/inject traffic to/from the switch ports.
It really depends on how much you know what you are doing, If you stick to:
*) IP Routing that would normally "fit" in a vendor switch
*) Bridging
*) VRFs
You will be fine
If you try and do some weird stuff then it's best to check with "ip route" to see if it was actually installed into hardware or not, but I would simply not do anything weird on such hardware
Really tempting price/performance, and I love the idea of getting away from questionable switch vendor OSes.
How disruptive are switch config changes? If I edit /etc/network/interfaces to add a new port, vlan, etc - does a `systemctl restart networking` (or whatever equivalent) bounce ports or halt switching for a moment while changes are applied?
> How disruptive are switch config changes? If I edit /etc/network/interfaces to add a new port, vlan, etc
They are seamless as long as your configuration does not do something stupid like tear down interfaces to reconfigure them. The switch takes no noticeable time to program from the Linux space to the ASIC
> - does a `systemctl restart networking` (or whatever equivalent) bounce ports or halt switching for a moment while changes are applied?
"systemctl restart networking" will typically blow away even the well most well configured systems in my experience.
In the post I suggest using ifupdown (the first one), since it's the most "easy" to debug, but I'm sure networkctl works too, with a healthy amount of systemd restraint
IIRC the /etc/network/interfaces does a reconfiguration that's pretty disruptive.
Things like brctl and ethtool worked on the fly without issues (note though that I mostly used Arista years ago).
It is usually non-disruptive if it gets applied as deltas. If your config tool does a teardown/recreate then that's disruptive. Within the bounds of ethernet and routing protocols (OSPF DR/DBR changes are disruptive, STP can be fun, ....).
I commend Ben for pushing the boundary of open Software networky-type things.
Even though I have been running HPC clusters for years, I still really don't like core switching infra[1]. Its just a pain in the arse to investigate and run.
[1] I realise that this is very much not a core switch, or even a TOR switch.
Cool project… for homelab stuff though, I lean towards simpler, cheaper solutions…. Like Mikrotik hardware, and low power mini PCs. I’m not hosting a business out of my house, it all functions perfectly fine.
Great, seems like a setup that works for you. I'm considering going to find one of these switches for my lab - it'll be like the 5th switch in my rack. I like playing with high performance networking, and your mikrotik isn't going to cut it for the stuff I like to experiment/play around with.
(Remember... this is a hobby where people persue their interests and experiment with things they want to know more about, not a proscribed set of exercises for just one path towards devops engineer).
Personally I don't like Mikrotik very much. It's just too easy to turn on some feature that disables offloading. I run Ruckus/Brocade ICX6610 and ICX7150 in my lab currently, though the 6610 uses more power than makes sense.
Is there an OS or another Linux distribution that matches Debian's performance in this respect, without the complexity of an entire Linux system? Could Debian be stripped down (and then how are updates applied)?
Sure you can install basically any EFI+amd64 distro you want, that also has the correct kernel module.
I don't really know why you would want do that (though, RHEL based would be a reasonable 2nd option)
Performance of the distro isnt really a big deal because the OS has so little to do with the day to day packet shuffling, My switch runs at a load average (crappy metric I know, but to give you a idea) of 0.04.
I apply updates with apt update and apt upgrade, as you would a normal Debian server. Only the kernel is pinned since it is special for the switch, however you can rebuild the kernel deb's when you want, as the driver is in the mainline kernel repo.
I think you are misunderstanding what the author means by "stock Debian." The issue here is that the hardware is so unique that we need to add kernel modules that are not included in the standard Linux kernel that ships with Debian. So we need a custom Linux kernel. Close to stock might also be referring to any config changes the author needed to make in order for the switch to function properly.
This could be done with any Linux. I'm not sure what 'complexity' you are referring to, but the complex open modular nature of the Linux kernel allows us the flexibility to build a custom kernel for this interesting application without unnecessary modules. To be frank all modern operating systems are complex because modern hardware is complex and user expectations are high. Some just do a great job of hiding the complexity from the user. Linux makes an attempt to expose it's complexity, but it's complexity is about the same as iOS, macOS, Windows, etc.
If you wanted to do without Debian or any distribution you could compile your own kernel, your own user space tools, and then install it on the hardware. At this point updates would need to be applied by pulling code from upstream and compiling it yourself. Look into Linux From Scratch to get an idea of how Linux without a distribution works. However, I don't think the juice is worth the squeeze in this situation.
EDIT: My answer to "is there another OS", to which the answer is yeah, probably a dozen. I think a blog post of doing this with openBSD would be interesting because I'm not sure what the exact steps are to install custom drivers on a BSD and openBSD has a very high standard for security. I think the reason it's done on Linux is because there is a lot of expertise about Linux and this type of project is relativity straightforward with Linux.
Plan 9. Seriously. One person can understand the whole OS including the kernel. File servers coupled with 9p provide an excellent abstraction for distributing services across networks. Source code is included and the build system is designed with cross platform baked in OUT OF THE BOX.
All you need is a driver for the switch chip that serves it as a set of files you can read and write configuration commands to. With the right driver setup you could then configure the chip using human readable textual commands and use a script and standard cmd line tools to configure the switch. We could PXE boot plan 9 and leave the switch diskless. PXE booting plan 9 networks is brain dead simple and works out of the box.
Familiarity with plan 9 and its kernel along with a solid understanding of c. Plan 9 had its own c library which is close to c90 but much cleaner imo. Networking and threading libs are really nice.
Driver might be in two parts like usb where the kernel driver serves up the usb controller and attached devices while user space file servers open the devices and serves them. e.g. a webcam is served as a video stream file.
I really enjoyed that writeup. I wonder has anyone played with DPUs in a similar way? I have been trying to think of how to hairpin traffic through some standalone DPUs like an nvidia bluefield or pensando. To make my own >100G east west stateful firewall for a small fraction of the cost of a real option.
Switch like OPs with all traffic passing to and from a DPU to make a poor man's Aruba CX10000
I encourage playing with these + cumulus linux via the nvidia air lab environment. It may seem limiting at first, but being able to upload a graphviz topology as part of a CI/CD pipeline is extremely powerful.
Not sure that's a fit for home network enthusiasts, and for business, just pay the bucks to be able to pass the buck - ie get a packaged solution not homebrew.
For what it's worth I did not pay 5k USD for it, that is significantly overpriced for it on the 2nd hand market.
> for business, just pay the bucks to be able to pass the buck - ie get a packaged solution not homebrew
There is a significant cost difference here,
The overall point of the post is that the "packaged solution" is not viable when most switch vendors have what I can describe as "crap" software quality and/or support.
So if you are in the land of simple L2 switching and L3 routing, this switch is amazing because you can escape the crap vendor software.
eBay shows me a whole bunch of these SN2010-series switches in the $1000-2000 USD range. Totally reasonable for such a switch. You can step up to the SN2700 mentioned in the article for prices in the $2000-3000 USD range. All used from sellers claiming "tested".
If you're really into homelab stuff or trying to run a small business where networking is critical, they're quite a good deal.
The "packaged solutions" as a new product from a known vendor with a support contract (because honestly if you're not buying the support contract you're going to spend a LOT of time messing with the switch no matter what vintage it is) are at least 10x the price, which very well could be much too much for a home user or small business.
I've seen a few chinese knock off switches on Amazon and Alibaba that are cheap and layer 3/"managed". I messaged a few of the alibaba sellers telling them I'm "building a custom switch OS and want their hardware to prototype" and they seemed amicable to helping me get a custom firmware installed, but never pursued it more.
I'd love to have managed 2.5Gb switches with 10Gb uplinks in my house using a custom linux OS that I can use standard config management tools with...
> I'd love to have managed 2.5Gb switches with 10Gb uplinks in my house using a custom linux OS that I can use standard config management tools with...
100%, this has been the most frustrating thing about following the various "open" switching worlds. There's a massive gap in the middle between the sorts of 4-8 port switches that end up in OpenWRT-compatible routers and this sort of enterprise switch that's barely accessible to the "homelab" class user.
I would absolutely love to have some open switching in the "Ubiquiti" class, desktop and 1U rackmount devices with gigabit through 10G as their primary interfaces. I'm personally in the VoIP world and if I could install Asterisk directly on a 48 port PoE switch I'd be deploying them by the dozens.
> I'd love to have managed 2.5Gb switches with 10Gb uplinks in my house using a custom linux OS that I can use standard config management tools with
Me too. I have been eyeballing "SparX-5" based switches, which do run Linux ("SMBStaX"), but you'd need something like million bucks to get anywhere :( Could be candidate for a Kickstarter style project maybe...
> So, I was very happy to learn that a friend had a Mellanox SN2010 that they were not using and were willing to sell to me.
That thing retails new for 10k. You got awesome friends if they have such a thing lying around unused and willing to sell it to you at a discount!
For those of us with less fortunate bank accounts: what's the smallest and reasonably affordable Mellanox model that has a similar featureset in terms of native Linux support?
> That thing retails new for 10k. You got awesome friends if they have such a thing lying around unused and willing to sell it to you at a discount!
Tactical eBay (or whatever is the robust 2nd handmarket in your region) can yield similar discounts on such hardware. Retail price is often list price, and that is often a large mark up regardless. The 32x100G port version of the same switch goes for around £2000 in the UK 2nd hand market
> what's the smallest and reasonably affordable Mellanox model that has a similar featureset in terms of native Linux support?
The SN2010 is likely the smallest, the SN2700 is likely the cheapest
Wait a few years and then root around in the datacenter dumpster once they toss the old gear after tech refresh (between 3-10yrs depending on the gear)
No such things around any more, unless you know people who're willing to risk their jobs. Everything's gotta be environmentally certified and whatnot... no one's putting stuff in a dumpster, it all goes to some commercial reseller that either refurbishes and sells on old equipment or hands it to a certified scrapyard.
Having experience with both; I am confident this would not work as well as you would want it to if even at all. VyOS has no native awareness of switchdev interfaces or their limitations
I began wondering about this and planning to investigate. My first impulse was that Vyos would know about anything Debian would know about, so if I could get Debian to deal with it, so also could I vyos.
Until I can buy one of these things or something similar, which I am now highly motivated to do, if you have any experiences or insights you can share, I’d love to see them.
Benjojo wasn't trying to make something for home or office (or even lab) use, he runs DC installations. The point of the article was to have an as-open-as-possible switch. Mikrotik doesn't qualify for that.
I'm curious what did this guy connect to several 100 GbE ports and how does the upstream connections he mentions look like, and from which provider. The device is second hand, so the likely use-case is a Home(?)Lab type of setup.
They have 100GBASE-PLR4 optics in them, that allow the 100G ports to be split up into 4x25G ports (or actually in this switch, 2x25G ports due to a hardware limitation with this switch)
> how does the upstream connections he mentions look like, and from which provider
They are just normal 10G-LR Single mode optics, in a data center.
> The device is second hand, so the likely use-case is a Home(?)Lab type of setup.
The high speed (25G+) does not have good solutions for multimode, and the length limits that physics enforces with multimode mean that it's not "no-brainer" applicable for going any more than between the same rack row.
So if you are dealing with datacenter cross connects that can exceed the max distance for MultiMode then you might spend hours debugging broken stuff for no real gain. MultiMode is slightly cheaper, but it's a false economy the moment stuff does not work correctly. I've spoken to people with DC cross connects that go into the 5km+ of cable distance. So it's easier to just stock one kind of optic per speed, and call it a day.
I stopped dealing with large data center providers in 2014. By that point we had switched to using single mode within our colocation spaces and most of the colocation data center providers we dealt with had stopped providing any multimode cross connects prior to that.
For them unfamiliar with fiber optics. Single mode fiber is better than multi mode in nearly every category. The only advantage multimode has is when terminating it. The looser tolerances and large fibers make it easier to attach an end.
Single-mode fiber is future-proof. Utilities bury that stuff and depreciate it over a 30-year lifetime. There has been like one spec change since the 1970s.
I have an SN2700 in my rack, next to a pair of Arista 7060CXs (as a point of comparison). These are wildly under-rated devices outside of the STH fanbase.
You may be surprised at how quiet and low power these Mellanox switches can be.
Quiet in the terms of DC grade switching, I'm not sure you could get away with such a switch in a home environment. The 100G optics need plenty of airflow to keep cool so it's not just a case of swapping the fans for something smaller
You can use a 100g DAC within a single rack (which is also cheaper). I only have a tiny bit of 100g at home and just do 10g optics for the connection out of the rack. Of course, that's my weird setup and it won't work once I want 100g to my office.
All the switches in this series except the one in this review actually have two separate boards inside with a PCIe-over-cable connection between them. For example the SN2700 here (the sleeved cable in the third photo; you can't see the connector from the angle it's taken from):
AFAIK DPUs are only ~$2,000 but if you put one in each server it adds up to far more money than a traditional network. They're really not intended to replace TOR switches.
No mention of ptp, I’m curious how the switch handles it (and how is it connfiged from Linux)
I find understanding how Linux networking really helpful in uunderstanding mikrotiks (which I use a lot in prod, although tend to shy away from for the most critical and demanding of services)
IIRC the half-width version of these switches are not advertised as having PTP support. I wouldn't have the first clue if it's straightforward to configure it to work with upstream linux but I do use PTP with Dell SmartFabric OS10 so there is likely some way to achieve it.
As impressive as it is, switchdev is a big departure from "Linux networking" though. This is not a great platform if that is your main objective. The interfaces are not "normal" in that respect and huge subsystems of the Linux network stack do not apply.
Yeah, the whole "(hyper)converged" network gear experience has been... let's say less than overwhelming. I'm happy that this person managed to get an experience they're happy with, but keep in mind that:
-The SN2010 retailed for about 10K (Euro/Dollars)
-It has never been truly available, as in: you could go somewhere, order it, and expect it to turn up in 2-3 days
-Even though small, these units tend to be loud, with at least 2 tiny fans making a lot of high-pitched noise
But the most salient point is that, even with "Linux on a router/switch", there's no guarantee that you'll get decent performance, as that entirely depends on how well the kernel understands the (proprietary) onboard chipset, which usually means that you're squarely in "well, here's this blob that works on certain kernel versions, and good luck with that!" territory.
> But the most salient point is that, even with "Linux on a router/switch", there's no guarantee that you'll get decent performance
As long as the ASIC is programmed the performance is the same between kernel versions. If the ASIC does not get programmed then it's like the entire device does not work at all (so you likely need to roll back the update you made).
> which usually means that you're squarely in "well, here's this blob that works on certain kernel versions, and good luck with that!" territory.
As mentioned in the post, the switch in question has blobless drivers, unlike the broadcom stuff you likely have experience with based on what you are saying
Having gone through this struggle myself, here's the cheat sheet. You want a device that uses the Linux switchdev driver and is supported by dentOS whether or not you actually choose to run dentOS on it (I run NixOS on my switch):
https://www.kernel.org/doc/html/latest/networking/switchdev....
https://github.com/dentproject/dentOS
Switchdev support means you don't need hardware-specific userspace tools (with their own bizarre syntax to learn) in order to configure the switch.
DentOS support means the device uses a sane bootloader (uboot or grub) and the only binary blobs on the device will be the ones built into the bootloader (IntelME, Arm Trusted Firmware) and the switch firmware which will be part of linux-firmware (and therefore very easy to manage/update).
In particular, looking for these two keywords is how you make sure that the hardware vendor is staying on "their side of the line" between hardware and software. Violations of this line are endemic to 10G+ switching.
I had no idea there were switches you could run NixOS on. What would be an example of such a switch?
That's what the blog post is about. Mellanox Spectrum switches can run any Linux distro.
For 10gbit+: Mellanox SN2xxx. Probably the SN3xxx, SN4xxx, and SN5xxx too but I can't afford those.
For 1gbit: Ubiquiti Edgerouter ER-4 and ER-6; both have SFP cages for fiber optic transcievers https://github.com/NixOS/nixpkgs/pull/194153 https://store.ui.com/us/en/products/er-4
Really useful, thanks a bunch!
This is really, really cool and I didn't know the platform was open to the extent that you could install your own upstream Linux and just get going.
I am curious though what configuration option prevents this from ending up with software switching. I understand the mellanox kernel module was compiled and loaded, but certainly that doesn't mean that anything you do in the network stack gets converted to switch fabric code and uses hardware packet switching. How do you make sure that you don't errantly wind up with poor latency and capped throughput?
But also, short of making and selling your own networked device for whatever reason, what are the real benefits of going this approach? I can see crazy use cases for where you have full control of the network stack (but again, see my point above — how do you guarantee you are not doing this in software?) but for most purposes, especially with qsfp and fiber, how much are you really gaining by doing it on-device? What is the killer use case here?
EDIT
Upon rereading, it seems the switch is hard-coded to be hardware-switched and cannot end up in a situation where you are accidentally using software packet switching in the first place (i.e. it does not just optimize to a hardware packet switching state). But that limits what you can do considerably, to the point that an off-the-rack Juniper or CSCO or whatever probably has more features than you can do here without writing your own code to hook into the mellanox sdk?
I am curious though what configuration option prevents this from ending up with software switching
The answer is "switchdev":
https://www.kernel.org/doc/html/latest/networking/switchdev....
The Linux switchdev driver is the awesome magic that says "make hardware-offloaded switching ASICs look just like software switching". It's beautiful and amazing, as you'd expect from Mellanox.
To configure Linux to do software switching, you need bridge interfaces and NICs to be made part of the bridge with the appropriate `ip` commands (or `brctl` if you're still using that).
This is common with small home routers - the WLAN and wired LAN ports (all typically appearing as one NIC) will be made part of a bridge `br0`. The four LAN ports aren't typically exposed as separate NICs so there is hardware switching going on there (some devices do let you split them out because they are VLANed internally though).
If `ip link show type bridges` doesn't show any bridges then you aren't software switching unless your drivers are lying to you.
None of that applies to switchdev; it's a somewhat different world than normal Linux networking.
> But that limits what you can do considerably, to the point that an off-the-rack Juniper or CSCO or whatever probably has more features than you can do here without writing your own code to hook into the mellanox sdk?
I mean, I'm not touching any mellanox sdk here, I am using the a very similar stack that someone on a "software router" would use, on a switch that can automatically accelerate it to 800G+ throughputs, while hitting a 60W power target.
You can hit some of those performance/power numbers in vendor hardware like Juniper/Cisco/Arista, however you have to also put up with their software, I (and others in my group of peers) have not had great experiences with vendor software, and in this setup I am able to patch/fix the software on my own terms.
If there is a security vuln in one section, I can fix that, and call it a day, I won't be forced to upgrade parts of the system I do not want to. I cannot do this with Juniper/Cisco/Arista always.
Is it obvious when you try to use a config that won't be accelerated? Or is the config silently ignored?
Yes. The switchdev "sw1p[0-9]+" ports are special; the any data the software kernel injects to them is discarded and they never emit packets to the kernel. They exist only to allow you to use `ip bridge` and `ip route` on them. So if you accidentally configure software switching on these ports no data will flow -- it will be totally obvious. You might get "no packets" by accident but you will never get "software switching" by accident.
If you really want software switching you have to use the management port (there's only one or two of these) whose name is "eth0" or "eth1" or something like that. So avoiding "accidental software switching" is really easy -- if you're typing "eth" you're doing it wrong. You can even explicitly delete this interface if you don't need the CPU to be able to snoop/inject traffic to/from the switch ports.
It really depends on how much you know what you are doing, If you stick to:
*) IP Routing that would normally "fit" in a vendor switch
*) Bridging
*) VRFs
You will be fine
If you try and do some weird stuff then it's best to check with "ip route" to see if it was actually installed into hardware or not, but I would simply not do anything weird on such hardware
The author mentioned another similar write up which provided more useful explanations and context to me:
https://ipng.ch/s/articles/2023/11/11/mellanox-sn2700.html
Really tempting price/performance, and I love the idea of getting away from questionable switch vendor OSes.
How disruptive are switch config changes? If I edit /etc/network/interfaces to add a new port, vlan, etc - does a `systemctl restart networking` (or whatever equivalent) bounce ports or halt switching for a moment while changes are applied?
> How disruptive are switch config changes? If I edit /etc/network/interfaces to add a new port, vlan, etc
They are seamless as long as your configuration does not do something stupid like tear down interfaces to reconfigure them. The switch takes no noticeable time to program from the Linux space to the ASIC
> - does a `systemctl restart networking` (or whatever equivalent) bounce ports or halt switching for a moment while changes are applied?
"systemctl restart networking" will typically blow away even the well most well configured systems in my experience.
In the post I suggest using ifupdown (the first one), since it's the most "easy" to debug, but I'm sure networkctl works too, with a healthy amount of systemd restraint
It just depends on what you use for management.
IIRC the /etc/network/interfaces does a reconfiguration that's pretty disruptive.
Things like brctl and ethtool worked on the fly without issues (note though that I mostly used Arista years ago).
It is usually non-disruptive if it gets applied as deltas. If your config tool does a teardown/recreate then that's disruptive. Within the bounds of ethernet and routing protocols (OSPF DR/DBR changes are disruptive, STP can be fun, ....).
I commend Ben for pushing the boundary of open Software networky-type things.
Even though I have been running HPC clusters for years, I still really don't like core switching infra[1]. Its just a pain in the arse to investigate and run.
[1] I realise that this is very much not a core switch, or even a TOR switch.
Cool project… for homelab stuff though, I lean towards simpler, cheaper solutions…. Like Mikrotik hardware, and low power mini PCs. I’m not hosting a business out of my house, it all functions perfectly fine.
Great, seems like a setup that works for you. I'm considering going to find one of these switches for my lab - it'll be like the 5th switch in my rack. I like playing with high performance networking, and your mikrotik isn't going to cut it for the stuff I like to experiment/play around with.
(Remember... this is a hobby where people persue their interests and experiment with things they want to know more about, not a proscribed set of exercises for just one path towards devops engineer).
Personally I don't like Mikrotik very much. It's just too easy to turn on some feature that disables offloading. I run Ruckus/Brocade ICX6610 and ICX7150 in my lab currently, though the 6610 uses more power than makes sense.
> as close to stock Debian as possible
Is there an OS or another Linux distribution that matches Debian's performance in this respect, without the complexity of an entire Linux system? Could Debian be stripped down (and then how are updates applied)?
Sure you can install basically any EFI+amd64 distro you want, that also has the correct kernel module.
I don't really know why you would want do that (though, RHEL based would be a reasonable 2nd option)
Performance of the distro isnt really a big deal because the OS has so little to do with the day to day packet shuffling, My switch runs at a load average (crappy metric I know, but to give you a idea) of 0.04.
I apply updates with apt update and apt upgrade, as you would a normal Debian server. Only the kernel is pinned since it is special for the switch, however you can rebuild the kernel deb's when you want, as the driver is in the mainline kernel repo.
I think you are misunderstanding what the author means by "stock Debian." The issue here is that the hardware is so unique that we need to add kernel modules that are not included in the standard Linux kernel that ships with Debian. So we need a custom Linux kernel. Close to stock might also be referring to any config changes the author needed to make in order for the switch to function properly.
This could be done with any Linux. I'm not sure what 'complexity' you are referring to, but the complex open modular nature of the Linux kernel allows us the flexibility to build a custom kernel for this interesting application without unnecessary modules. To be frank all modern operating systems are complex because modern hardware is complex and user expectations are high. Some just do a great job of hiding the complexity from the user. Linux makes an attempt to expose it's complexity, but it's complexity is about the same as iOS, macOS, Windows, etc.
If you wanted to do without Debian or any distribution you could compile your own kernel, your own user space tools, and then install it on the hardware. At this point updates would need to be applied by pulling code from upstream and compiling it yourself. Look into Linux From Scratch to get an idea of how Linux without a distribution works. However, I don't think the juice is worth the squeeze in this situation.
EDIT: My answer to "is there another OS", to which the answer is yeah, probably a dozen. I think a blog post of doing this with openBSD would be interesting because I'm not sure what the exact steps are to install custom drivers on a BSD and openBSD has a very high standard for security. I think the reason it's done on Linux is because there is a lot of expertise about Linux and this type of project is relativity straightforward with Linux.
What I meant was simpler OSes like BSDs, as you discuss in the edit, or Linux stripped down to networking essentials.
Plan 9. Seriously. One person can understand the whole OS including the kernel. File servers coupled with 9p provide an excellent abstraction for distributing services across networks. Source code is included and the build system is designed with cross platform baked in OUT OF THE BOX.
All you need is a driver for the switch chip that serves it as a set of files you can read and write configuration commands to. With the right driver setup you could then configure the chip using human readable textual commands and use a script and standard cmd line tools to configure the switch. We could PXE boot plan 9 and leave the switch diskless. PXE booting plan 9 networks is brain dead simple and works out of the box.
Interesting. What would it take to write the driver? Also, is Plan 9 maintained sufficiently?
> What would it take to write the driver?
Familiarity with plan 9 and its kernel along with a solid understanding of c. Plan 9 had its own c library which is close to c90 but much cleaner imo. Networking and threading libs are really nice.
Driver might be in two parts like usb where the kernel driver serves up the usb controller and attached devices while user space file servers open the devices and serves them. e.g. a webcam is served as a video stream file.
> Also, is Plan 9 maintained sufficiently?
A community fork known as 9front is maintained and receives patches almost daily. http://git.9front.org/plan9front/plan9front/HEAD/info.html and http://9front.org/releases/
If you like videos here is a really nice channel of a plan 9 hacker: https://www.youtube.com/channel/UC7qFfPYl0t8Cq7auyblZqxA
network booting core network infrastructure like switches seems like a good way to create interesting bootstrapping problems on a power outage.
I really enjoyed that writeup. I wonder has anyone played with DPUs in a similar way? I have been trying to think of how to hairpin traffic through some standalone DPUs like an nvidia bluefield or pensando. To make my own >100G east west stateful firewall for a small fraction of the cost of a real option.
Switch like OPs with all traffic passing to and from a DPU to make a poor man's Aruba CX10000
Bluefield can also run regular Linux distros and it can run standalone without a host (but you have to power it somehow). https://www.servethehome.com/zfs-without-a-server-using-the-...
All other DPUs seem to be NDA-encrusted.
I encourage playing with these + cumulus linux via the nvidia air lab environment. It may seem limiting at first, but being able to upload a graphviz topology as part of a CI/CD pipeline is extremely powerful.
See: https://github.com/na-son/nvidia-air for bootstrapping a non-EVPN topology.
Woah these are around $5k.
Not sure that's a fit for home network enthusiasts, and for business, just pay the bucks to be able to pass the buck - ie get a packaged solution not homebrew.
> Woah these are around $5k.
For what it's worth I did not pay 5k USD for it, that is significantly overpriced for it on the 2nd hand market.
> for business, just pay the bucks to be able to pass the buck - ie get a packaged solution not homebrew
There is a significant cost difference here,
The overall point of the post is that the "packaged solution" is not viable when most switch vendors have what I can describe as "crap" software quality and/or support.
So if you are in the land of simple L2 switching and L3 routing, this switch is amazing because you can escape the crap vendor software.
Sort of an openwrt for hard lines.
eBay shows me a whole bunch of these SN2010-series switches in the $1000-2000 USD range. Totally reasonable for such a switch. You can step up to the SN2700 mentioned in the article for prices in the $2000-3000 USD range. All used from sellers claiming "tested".
If you're really into homelab stuff or trying to run a small business where networking is critical, they're quite a good deal.
The "packaged solutions" as a new product from a known vendor with a support contract (because honestly if you're not buying the support contract you're going to spend a LOT of time messing with the switch no matter what vintage it is) are at least 10x the price, which very well could be much too much for a home user or small business.
I've seen a few chinese knock off switches on Amazon and Alibaba that are cheap and layer 3/"managed". I messaged a few of the alibaba sellers telling them I'm "building a custom switch OS and want their hardware to prototype" and they seemed amicable to helping me get a custom firmware installed, but never pursued it more.
I'd love to have managed 2.5Gb switches with 10Gb uplinks in my house using a custom linux OS that I can use standard config management tools with...
> I'd love to have managed 2.5Gb switches with 10Gb uplinks in my house using a custom linux OS that I can use standard config management tools with...
100%, this has been the most frustrating thing about following the various "open" switching worlds. There's a massive gap in the middle between the sorts of 4-8 port switches that end up in OpenWRT-compatible routers and this sort of enterprise switch that's barely accessible to the "homelab" class user.
I would absolutely love to have some open switching in the "Ubiquiti" class, desktop and 1U rackmount devices with gigabit through 10G as their primary interfaces. I'm personally in the VoIP world and if I could install Asterisk directly on a 48 port PoE switch I'd be deploying them by the dozens.
> I'd love to have managed 2.5Gb switches with 10Gb uplinks in my house using a custom linux OS that I can use standard config management tools with
Me too. I have been eyeballing "SparX-5" based switches, which do run Linux ("SMBStaX"), but you'd need something like million bucks to get anywhere :( Could be candidate for a Kickstarter style project maybe...
Checkout RCHD-SPARX [0]. You'll need to draw the rest of the owl, but the really hard part's there. I hope someone builds a switch on that.
[0] https://conclusive.tech/products/rchd-sparx-networking-som/
> I have been eyeballing "SparX-5" based switches
Have you found any for sale other than the Microchip dev board?
I stumbled across this recently: https://www.servethehome.com/insane-48-port-2-5gbe-2x-25gbe-... but I haven't been able to find where you can buy one these days.
Sadly my PCB design skills are not quite up to dense BGA escape + 25gig lanes...
I'd love to have managed 2.5Gb switches with 10Gb uplinks in my house using a custom linux OS that I can use standard config management tools with...
Check out DENT NOS: https://dent.dev/ There's a Delta 32x1G (PoE+) + 16x2.5G (PoE++) + 6x25G SFP28 switch that can run DENT.
Those Delta ones look nice but I can't find a reseller for them unfortunately (specifically the DVS-G106W02-2GF)
All I need now is a 1U switch that has:
1. 48-port 10/40 GbE SFP+ with mostly copper UPoE/++
1.5. 2-4 100 GbE QSFP28 uplinks
2. Doesn't sound like a jet engine
3. Doesn't phone home to a cloud in another country
4. Doesn't use an obscure, fragile configuration language
5. Doesn't cost $5k
Replying to myself (minus 2): Edgecore AS5610-52X on FleaBay. It even has dual hot-swap PSUs. Nothing a little resistor or different fan can't solve.
Overriding your points, I'd be happy with:
1. 8-port nbase-t (2.5/5/10G supported in all ports)
5. Doesn't cost $500
That's easy. I think I have a 10 GbE POE++ switch that does that.
Would appreciate some pointers.
This would suffice for the fast part of my home LAN... the rest is handled well by a gigabit ethernet switch with one 10g uplink.
Arr.. there be lots of treasures on the high fleas.
https://www.ebay.com/itm/386453004153 Arista DCS-7050SX-64-R 48P 10GbE SFP+ 4P 40GbE QSFP+ RA Switch $180
Actually looks really nice for the price, but I don't think it does support nbase-t.
> So, I was very happy to learn that a friend had a Mellanox SN2010 that they were not using and were willing to sell to me.
That thing retails new for 10k. You got awesome friends if they have such a thing lying around unused and willing to sell it to you at a discount!
For those of us with less fortunate bank accounts: what's the smallest and reasonably affordable Mellanox model that has a similar featureset in terms of native Linux support?
> That thing retails new for 10k. You got awesome friends if they have such a thing lying around unused and willing to sell it to you at a discount!
Tactical eBay (or whatever is the robust 2nd handmarket in your region) can yield similar discounts on such hardware. Retail price is often list price, and that is often a large mark up regardless. The 32x100G port version of the same switch goes for around £2000 in the UK 2nd hand market
> what's the smallest and reasonably affordable Mellanox model that has a similar featureset in terms of native Linux support?
The SN2010 is likely the smallest, the SN2700 is likely the cheapest
Wait a few years and then root around in the datacenter dumpster once they toss the old gear after tech refresh (between 3-10yrs depending on the gear)
No such things around any more, unless you know people who're willing to risk their jobs. Everything's gotta be environmentally certified and whatnot... no one's putting stuff in a dumpster, it all goes to some commercial reseller that either refurbishes and sells on old equipment or hands it to a certified scrapyard.
I like the photo of smuggling gear out of the data center in a backpack.
I want this and I want to put VyOS on it.
Having experience with both; I am confident this would not work as well as you would want it to if even at all. VyOS has no native awareness of switchdev interfaces or their limitations
I began wondering about this and planning to investigate. My first impulse was that Vyos would know about anything Debian would know about, so if I could get Debian to deal with it, so also could I vyos.
Until I can buy one of these things or something similar, which I am now highly motivated to do, if you have any experiences or insights you can share, I’d love to see them.
VyOS is a great piece of software. I wish more people knew about it!
I wish the edgerouter line by ubiquiti continued as another way to use vyatta.
I have two Mikrotik CRS305-1G-4S+IN (10GbE) plus one CRS504-4XQ-IN (100GbE) switches in my office. Along with a load of other Mikrotik gear.
The 10GbE ones are silent, and the 100GbE one isn't exactly loud, unlike lots of second-hand kit which has come from a DC...
Benjojo wasn't trying to make something for home or office (or even lab) use, he runs DC installations. The point of the article was to have an as-open-as-possible switch. Mikrotik doesn't qualify for that.
I'm curious what did this guy connect to several 100 GbE ports and how does the upstream connections he mentions look like, and from which provider. The device is second hand, so the likely use-case is a Home(?)Lab type of setup.
Hi, I am the guy.
> what did this guy connect to several 100 GbE
They have 100GBASE-PLR4 optics in them, that allow the 100G ports to be split up into 4x25G ports (or actually in this switch, 2x25G ports due to a hardware limitation with this switch)
> how does the upstream connections he mentions look like, and from which provider
They are just normal 10G-LR Single mode optics, in a data center.
> The device is second hand, so the likely use-case is a Home(?)Lab type of setup.
Nope, this device now runs my business bgp.tools
That suddenly makes a lot more sense! Thank you for the explanations and for the writeup!
Stupid question, why does the datacenter provide you with LR smf instead of SR mmf?
In my eyes, MultiModeFibre is on it's way out.
The high speed (25G+) does not have good solutions for multimode, and the length limits that physics enforces with multimode mean that it's not "no-brainer" applicable for going any more than between the same rack row.
So if you are dealing with datacenter cross connects that can exceed the max distance for MultiMode then you might spend hours debugging broken stuff for no real gain. MultiMode is slightly cheaper, but it's a false economy the moment stuff does not work correctly. I've spoken to people with DC cross connects that go into the 5km+ of cable distance. So it's easier to just stock one kind of optic per speed, and call it a day.
Equinix I think already phased out MMF XCs
I stopped dealing with large data center providers in 2014. By that point we had switched to using single mode within our colocation spaces and most of the colocation data center providers we dealt with had stopped providing any multimode cross connects prior to that.
Thanks for the info.
For them unfamiliar with fiber optics. Single mode fiber is better than multi mode in nearly every category. The only advantage multimode has is when terminating it. The looser tolerances and large fibers make it easier to attach an end.
Stupid answer: Multimode sucks. The only reason to use it is if the person before you failed to know that it sucks and now it cannot be replaced.
Friends don't let friends use multi-mode fiber.
Single-mode fiber is future-proof. Utilities bury that stuff and depreciate it over a 30-year lifetime. There has been like one spec change since the 1970s.
Single-mode fiber. Always single-mode fiber. Nothing else, ever.
I have an SN2700 in my rack, next to a pair of Arista 7060CXs (as a point of comparison). These are wildly under-rated devices outside of the STH fanbase.
You may be surprised at how quiet and low power these Mellanox switches can be.
Quiet in the terms of DC grade switching, I'm not sure you could get away with such a switch in a home environment. The 100G optics need plenty of airflow to keep cool so it's not just a case of swapping the fans for something smaller
You can use a 100g DAC within a single rack (which is also cheaper). I only have a tiny bit of 100g at home and just do 10g optics for the connection out of the rack. Of course, that's my weird setup and it won't work once I want 100g to my office.
The rack I’m referring to is in my home. YMMV, of course.
Would buy something like this as a device you can connect via pcie to your server.
No need to have switches and servers in your rack anymore, every server is a switch and every switch is a server with a 192 threads CPU. Insane.
All the switches in this series except the one in this review actually have two separate boards inside with a PCIe-over-cable connection between them. For example the SN2700 here (the sleeved cable in the third photo; you can't see the connector from the angle it's taken from):
https://ipng.ch/s/articles/2023/11/11/mellanox-sn2700.html
The cable has a SAS (SFF-8087) connector on each end. I bet you can replace it with an SFF-8087-to-Oculink cable:
https://www.amazon.com/chenyang-SFF-8611-SFF-8087-PCI-Expres...
and a PCIe-to-oculink card like one of these:
https://www.amazon.com/Ableconn-PEX-OL153-OCuLink-SFF-8612-A...
(none of the links are affiliate links)
It ends up crazy expensive and complex to configure but go for it.
I bet Nvidia is selling something like that but with only 2-4 ports for the same price as a 48 port switch...
AFAIK DPUs are only ~$2,000 but if you put one in each server it adds up to far more money than a traditional network. They're really not intended to replace TOR switches.
No mention of ptp, I’m curious how the switch handles it (and how is it connfiged from Linux)
I find understanding how Linux networking really helpful in uunderstanding mikrotiks (which I use a lot in prod, although tend to shy away from for the most critical and demanding of services)
IIRC the half-width version of these switches are not advertised as having PTP support. I wouldn't have the first clue if it's straightforward to configure it to work with upstream linux but I do use PTP with Dell SmartFabric OS10 so there is likely some way to achieve it.
As impressive as it is, switchdev is a big departure from "Linux networking" though. This is not a great platform if that is your main objective. The interfaces are not "normal" in that respect and huge subsystems of the Linux network stack do not apply.
https://github.com/Mellanox/mlxsw/wiki/Precision-Time-Protoc...
cool "100000baseSR4"
I remember 10base*, so I find 100000base* pretty amazing.
Yeah, the whole "(hyper)converged" network gear experience has been... let's say less than overwhelming. I'm happy that this person managed to get an experience they're happy with, but keep in mind that:
-The SN2010 retailed for about 10K (Euro/Dollars)
-It has never been truly available, as in: you could go somewhere, order it, and expect it to turn up in 2-3 days
-Even though small, these units tend to be loud, with at least 2 tiny fans making a lot of high-pitched noise
But the most salient point is that, even with "Linux on a router/switch", there's no guarantee that you'll get decent performance, as that entirely depends on how well the kernel understands the (proprietary) onboard chipset, which usually means that you're squarely in "well, here's this blob that works on certain kernel versions, and good luck with that!" territory.
You're FUDding yourself pretty hard. All of these points have been addressed in various comments in this thread.
Hyperconverged infra is basically bundled networking, storage, and hypervisor. Think nutanix, vxrail, Cisco ucs.
> But the most salient point is that, even with "Linux on a router/switch", there's no guarantee that you'll get decent performance
As long as the ASIC is programmed the performance is the same between kernel versions. If the ASIC does not get programmed then it's like the entire device does not work at all (so you likely need to roll back the update you made).
> which usually means that you're squarely in "well, here's this blob that works on certain kernel versions, and good luck with that!" territory.
As mentioned in the post, the switch in question has blobless drivers, unlike the broadcom stuff you likely have experience with based on what you are saying