For context:
I’m copying the same files to the same USB drive for comparison from Windows and from my Fedora 41 Workstation.
Around 10k photos.
Windows PC: Dual Core AMD Athlon from 2009, 4GB RAM, old HDD, takes around 40min to copy the files to USB
Linux PC: 5800X3D, 64GB RAM, NVMe SSD, takes around 3h to copy the same files to the same USB stick
I’ve tried chagning from NTFS to exFAT but the same result. What can I do to improve this? It’s really annoying.
Silly question perhaps, but are you sure you’re using the correct port on your Linux system? If I plug my external HD into a USB2 port, I’m stuck at 30-40MB/sec, while on a USB3 port I get ~150-180MB/sec. That’s proportionally similar to the difference you described so I wonder if that’s the culprit.
You can verify this in a few different ways. From Terminal, if you run
lsusb
you’ll see a list of all your USB hubs and devices.It should look something like this:
Bus 002 Device 001: ID xxxx:yyyy Linux Foundation 3.0 root hub Bus 002 Device 002: ID xxxx:yyyy <HDD device name> Bus 003 Device 001: ID xxxx:yyyy Linux Foundation 2.0 root hub Bus 004 Device 001: ID xxxx:yyyy Linux Foundation 3.0 root hub
So you can see three hubs, one of which is 2.0 and the other two are 3.0. The HDD is on bus 002, which we can see is a USB 3.0 hub by looking at the description of
Bus 002 Device 001
. That’s good.If you see it on a 2.0 bus, or on a bus with many other devices on it, that’s bad and you should re-organize your USB devices so your low-speed peripherals (mouse, keyboard, etc.) are on a USB2 bus and only high-speed devices are on the USB3 bus.
You can also consult your motherboard’s manual, or just look at the colors of your USB ports. By convention, gray ports are USB 1.0, blue ports are 2.0, and green ports are 3.x.
If you’re running KDE, you can also view these details in the GUI with kinfocenter. Not sure what the Gnome equivalent is.
For sanity check I’ve tried.
Bus 004 Device 004
and it’s USB 3.0 as it should be.Also I’ve disabled caching and I’m now copying 6 video files at only just 15MB/s (and it’s slowing down, byt the time I went to make screenshot for this post it dropped again). And it’s quite a bit slower than on Windows still.
Now down to 9MB/s and still going down
EDIT: 12 min later…
Still going…
it just finished, 4h for 40GB (6 files)
Did the USB drive get excessively warm during this because it looks like the drive is throttling?
Incidentally, this is why I switched to using external SSDs. A group of 128GB flash drives I had would slowly fall over when I would write 100GB off files to it.
This is a really good point. I generally have the opposite experience re: Linux vs windows file handling speed. But I have been throttled before by heat.
OP, start again tomorrow and try the reverse, and tell us the results.
single 4.4GB video file
- Zen1 based Windows PC - 2min 10s (could unmount right after progress finished)
- Zen4 based Fedora41 laptop - 3min 1s (progress finished within 1min but then took 2min to unmount)
- Zen3 based Fedora41 PC - 5min 16s (progress finished within 20s but took almost 5min to unmount)
No it did not get warm at all because it was barely doing anything. But so that I’m fair I’ve borrowed different USB 3.2 Gen1 drive and tested also with my Fedora Laptop.
single 4.4GB video file
- Zen1 based Windows PC - 2min 10s (could unmount right after progress finished)
- Zen4 based Fedora41 laptop - 3min 1s (progress finished within 1min but then took 2min to unmount)
- Zen3 based Fedora41 PC - 5min 16s (progress finished within 20s but took almost 5min to unmount)
Linux, and macOS, enables write caching by default and Windows does not. This is what you’re seeing.
Mounting the drive with “noatime,flush” (preferred) would adjust the write caching and mounting with “sync,dirsync” would turn off write caching.
I’m using USB 3 10Gbps on my Linux system. The USB stick is USB 3-1.0 and the Windows PC only has USB 2.0 so it should be the slowest but it’s actually several times faster.
rsync -aP <source>/ <dest>
I find it faster and more reliable than most GUI explorers
-avP
I do
-azP
for compressionthat’s good if your data is not already compressed, otherwise is more of a CPU waste
images, videos, audio, game assets, standardized data files, and a bunch of other things are likely already compressed.
That’s just the state of things. I have experienced this as well, trying to copy a 160 GB usb stick to another one (my old itunes library). Windows manages fine, but neither Linux nor MacOS do it properly. They crawl, and in macos’ case, it gets much slower as time goes by, and I had to stop the transfer. Overall, it’s how these things are implemented. It’s ok for a few gigabytes, but not a good case for many small files (e.g. 3-5 mb each) with many subfolders, and many GBs overall. Seems to me that some cache is overfilling, while windows is more diligent to clear up that cache in time, before things get into a crawl. Just a weak implementation for both Linux and MacOS IMHO, and while I’m a full time Linux user, I’m not afraid to say it how I experienced it under a debian and ubuntu.
I/O scheduler issue?
Connect that USB stick, and run command:
grep '' /sys/block/*/queue/scheduler
show output.
Can I run it while transferring?
Sure!
I see nothing wrong here. So issue must be somewhere else.
Random peripherals get tested against windows a lot more than Linux, and there are quirks which get worked around.
I would suggest an external SSD for any drive over 32GB. Flash drives are kind of junk in general, and the external SSDs have better controllers and thermals.
Out of curiosity, was the drive reformatted between runs, and was a Linux native FS tried on the flash drive?
The Linux native FS doesn’t help migrate the files between Windows and Linux, but it would be interesting to see exFAT or NTFS vs XFS/ext4/F2FS.
Depends on the distro and desktop environment but some will “transfer” files to a software buffer that doesn’t actually write the data immediately. Works for limiting unnecessary writes on Flash memory but not USB sticks that are designed to be inserted and removed at short notice.
You can force Linux to commit pending writes using the ‘sync’ command. Note it won’t give you any feedback until the operation is finished (multiple minutes for a thumbdrive writing GBs of data) so append & to your command (‘sync &’) to start it as its own process so you don’t lock the terminal.
You can also watch the progress using the command form this Linux Stack Exchange Q;
https://unix.stackexchange.com/questions/48235/can-i-watch-the-progress-of-a-sync-operation#48245
Side question though, it seems that there are faster options. How come we don’t use those in GUI file explorers if they’re faster?
As I’ve already mentioned, sync does absolutely nothing. The copy took so long that the
sync
command exited 4 times while the files were still transfering and were nowhere near finishing. Regarding thewatch -d grep -e Dirty: -e Writeback: /proc/meminfo
command, I did not mention it in this thread but I did try it and yes, there was some almost 900k kB of data in the “Dirty” buffer that went up and down constantly even after I’ve disabled the caching.