2x10Gb network : Transfer Huge files with NFS limitied to 6 Gbps
Feb 12
I'm transferring big files (78GB) through network, with NFS, but the transfer is finally limited to 6 Gbps (on a 2x10Gb NIC). I cannot find the bottleneck for now. Here my setup:
1 Server
- MB: Supermicro X10DRH-C (pcie 3.0), only one cpu
- NIC: 2 x 10Gb (intel X710-T2L)
- 64GB of DDR4 (2400Mhz)
- HBA: LSI 9311-8i => 2 mini SAS hd connected to SAS 12Gb backplane
- 12 HDD: consumer grade, SATA. Toshiba MG09 18TB (So SAS is limited to 6 Gbps per line)
NFS /etc/exports : xxx(rw,async,fsid=0,no_subtree_check,crossmnt). 24 threads for nfsd
I tried to dd big files into all the disks, no slow down, they all write at 200MB/s
2 Clients
NFS configuration: async.
MTU: 9000.
- threadripper 3970X, 192GB of ram, copy from a RAID0 of 4 NVME gen 3
- 2 xeon E5, 512GB of ram, copy from a RAID0 of 4 ssd
Clients are directly connected to the server (no switch), on their own network (10.1.0.0/24 and 10.1.2.0/24).
I tried to play with ring buffer and flow control with ethtool, but it doesn't change anything.
No high CPU load.
I'm generally copying 6-8 files at the same time, onto 6-8 different disks => so I'm expecting an average of 9-12Gbps transfer rate. The network spikes sometime to more than 9 Gbps on one interface, for a few seconds, but finally stabilize around 3Gbps on both interfaces. I guess it's due to caching in server ram
I need some help to find the bottleneck please.
linux
1 answer
Accepted answer · original discussion
Feb 22
What are you using to copy data? I found standard tools such as cp, mv, rsync don't scale very well and will use bbcp for such transfers. Even though the project is unfortunately abandoned, it's still better. The best I can do is between our nfs v4.0 mounts over shared 10 Gb/s Ethernet line but at night when it's unused I've seen it saturated at 1.2 GB/s before.
bbcp is at https://www.slac.stanford.edu/~abh/bbcp/
last I checked there were forks on Github but they were abandoned as well.
Sometimes the Stanford git doesn't work but if you retry throughout the day you should be able to grab v17.12.00.00.0 from there...
If not, I keep a clean copy of the repo so I can put it on new machines. I just archived it for you with git archive -o bbcp-17.12.00.00.0.tar master then gzip'd it and put it here: https://ava.genome.arizona.edu/pub/bbcp-17.12.00.00.0.tar.gz
Best,
2 question comments
Use comments to ask for clarification. Post a solution as an answer.
Feb 13
Feb 13