In my last post Windows 2012 R2 Decommission I mentioned about a problem that would change my whole architecture.
Transfer Speeds
Whilst copying all the data over Samba from my M1 Max Macbook Pro over Gigabit Ethernet and having the Plex server access the files using NFS over Gigabit Ethernet it seemed like everything was going perfect. Transfer speeds of > 100MB/s were achievable in both directions.
However, over WIFI connecting over Wireless AC I was only getting around 10MB/s. Transfer speeds to the old Windows 2012 R2 server over WIFI were about 40-45MB/s. Nothing else in the network had changed.
Tuning Samba
After much Googling and reading of the Samba documentation I tried out a lot of settings to see if it would improve the performance over WIFI. Some of the settings that I tried are listed below. Don’t take any of them as advice as they didn’t help in my situation and are listed for illustration only.
log level = 1
#socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=131072 SO_SNDBUF=131072 SO_KEEPALIVE
#oplocks = yes
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536
#min protocol = SMB2
#max protocol = SMB3
#read size = 65536
#max xmit = 65536
#aio read size = 16384
#aio write size = 16384
#socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536
#max protocol = SMB3
#read raw = yes
#write raw = yes
#oplocks = yesBackup NAS Performance Testing
For a sanity check I created a new share on the “backup” Synology DS920+ NAS to test both Ethernet and WIFI Samba transfers from my Macbook Pro. Wired was 110MB/s and WIFI was around the 45MB/s mark (which is what is expected).
To ensure this test was accurate I verified that this was also an issue with a Windows 11 laptop and desktop as well.
Back To The Future
At this point I felt like I was repeating history. I again was having problems with performance of Samba and it was not immediately apparent what the issue was and how to fix it. The last time I had this issue I ditched Samba and used Windows Server 2012 R2.
My plan was to investigate replacing the hyper-converged server (compute + storage) with separate compute and storage boxes that communicate over the network. However I had planned to do this on the main server so that I wouldn’t be penalised with Gigabit Ethernet networking and I could monitor the performance with Prometheus/Grafana. This would allow me to size the performance required for the NAS before purchasing the replacement.
Although I probably could have figured out what the issue was I decided against spending the time on this. As any parent of small children knows you do not have a lot of time to spend on projects. Things need to work reliably, easily and with little fettling. It was disappointing that a decade later it was not a great out of the box experience using Samba.
New Architecture
I decided that it would be easier and more time effective to just cut the ties and move all the data to the Synology NAS and make the main server the backup target for the daily backups. This means that the primary source and backup destinations need to swap for all the data.
This required quite a bit of thought and care but the first step was to make sure that I had an up to date backup (off grid) before starting just in case anything went wrong.
I then could start the process which comprised of the following steps:
- Disable daily backup for Windows Server 2012 R2
- Copy Windows Server 2012 R2 Data to NAS (~13TB)
- Setup NFS/SMB to allow read write access for data
- Edit backup scripts to pull from Synology NAS back to main server
- Run backups manually as they will take over 6 hours (the max my on premises GitHub Actions can run)
- Enable automatic daily backups
- De-provision Windows Server 2012 R2 server
As you can probably see this means that to keep things up to date and running at the same time I need to have multiple copies of the data at one time. Normally this is not a problem with small amounts of data but when you have ~13TB of data it presents a little more of a challenge. Luckily the main server has 32TB of total storage and the NAS has 29.7TB of total storage so I could keep 2 copies of all the data at the same time. If I had less available storage I would have had to do the scary action of deleting production (or backup) data before I could sync the new data reducing my redundancy to only the production data and the offsite backup.
Migration Complete?
It was the 21 days ago (7th of May) that I last updated my blog on this project and I still have yet to fully migrate all the data and have the automatic backups running as expected. Although slow it has been thankfully uneventful and I have not been at any point at risk of data loss or corruption.
However I am yet to find out if the Synology DS920+ can run as my storage mechanism (performance wise) and Gigabit Ethernet could still be a bottleneck between my compute and storage (I may seperate NFS and SMB to different network adapters to alleviate some of this.
Leave a Reply