Community Forum

Mansoor
Mansoor
Offline
Resolved
0 votes
I want to copy some 1TB of files from Apple Time Capsule to my clearOS server. I have a gigabit ethernet link between the two machines. I did a test using ipref, and found that Time Capsule is capable of pushing upto 650mb/s (~80MB/s) via its network ports.

I tried SMB, but got around 17MB/s only of transfer speed using rsync with all possible optimization I know. So, I decided to try AFP instead knowing that Apple always prefers its own tech! (I'm aware that Apple decided to drop AFP in favor of SMB3, but my time capsule is an old fellow!)

I downloaded and compiled afpfs-ng ver. 0.8.2 (updated 2 years ago) using the following commands:

cd /usr/src
svn checkout https://github.com/simonvetter/afpfs-ng.git
yum install gmp-devel libgcrypt-devel readline-devel fuse-devel
#yum install fuse-afp # no need maybe!
cd /usr/src/afpfs-ng.git/trunk/
./configure --prefix=/usr
make && make install
ldconfig

Then started the daemon:
afpfsd

and it worked :) So, I mounted the Time Capsule:
mount_afp afp://user:pass@192.168.2.1/Data /mnt/airport

and started synchronizing the files:
rsync -avh --exclude '.*' --progress /mnt/airport/Users/user/MyFiles/ /var/flexshare/shares/dm

What an improvement! It started from 25MB/s and escalated up to 80MB/s!

But that did not have a happy ending! The transfer is always interrupted with an error message saying: Danger, recursive loop

When I look at the log, I find complaints from afpfsd saying: I have no idea what this is a reply to, id 3148.

I'd like to know if anyone here has tried this afpfs-ng with clearOS 7.3, and how was it?

Thank you.
Monday, September 11 2017, 05:19 PM
Share this post:
Responses (8)
  • Accepted Answer

    Tuesday, September 12 2017, 04:39 AM - #Permalink
    Resolved
    0 votes
    Use "rsync -e ssh ......." here to transfer files between machines. Would have thought Apple would support ssh natively...
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, September 12 2017, 11:52 AM - #Permalink
    Resolved
    0 votes
    With SSH you have to watch the speed as encryption is relatively slow. When I did my transfers, I ran the rsync as a daemon on one machine and rsync'ed to it from the other which gets round using ssh. At a guess that is not possible to run rsync on the Time Capsule.
    The reply is currently minimized Show
  • Accepted Answer

    Tuesday, September 12 2017, 01:17 PM - #Permalink
    Resolved
    0 votes
    Speed is dependant on your CPUs - sure on an old single core CPU it will be slow. Find it considerably faster than CIFs and more consistent. Biggest bottle neck I find is when a multitude of small files are being transferred - the receiving system cannot create and update the file-system quick enough showing down the average transfer rate.

    Just looked for the manual for the Time Capsule - guess it probably doesn't support rsync or ssh - it fact it didn't specify any file transfer protocols or anything it supports except "Use the Time Machine application in Mac OS X v10.5.7 (or later)". Only specifications were for standard ethernet and wireless standards. At least that was all I could find on https://manuals.info.apple.com - maybe there's more elsewhere... :p
    The reply is currently minimized Show
  • Accepted Answer

    Mansoor
    Mansoor
    Offline
    Tuesday, September 12 2017, 09:32 PM - #Permalink
    Resolved
    0 votes
    The Time Capsule is a locked up device running NetBSD. Many have tried to jail-break it over the years but without success. Any user of the time capsule is limited for now to what Apple had chosen to offer in terms of file transfer, which is SMB or AFP.

    Based on my observation, I think the problem with afpsf-ng is time-related. It seems that the Time Capsule and afpfs driver get out of synchronization somehow, especially at high transfer speed. So in order to make the TC to concentrate on file transfers, I relinquished it from its network routing duties and made it to operate in bridged mode. I also limited the rsync bandwidth using --bwlimit operand. I then ran a test over WiFi and the process went smoothly up to 4MB/s. I'll test again with a wired link between the TC and the clearOS server and will report the results here.
    The reply is currently minimized Show
  • Accepted Answer

    Thursday, September 14 2017, 11:29 PM - #Permalink
    Resolved
    0 votes
    Just to close out the speed issue with ssh transport for rsync in case anybody searching the issue hits this thread. Currently copying 1.6TB from a Raid 5 on one system to a Raid 1 on another...

    Here is the mrtg graph of progress so far after a few hours. As you can see we are almost hitting 60 MB/second most of the time...

    Transmitting System - i7-3770 Clearos 6.9 clocked at 3.2 GHz
    Receiving System - AMD Phenom 570 ClearOS 7.3 clocked at 3.5 GHz

    Network 1G using common consumer grade equipment components.

    http://www.sraellis.tk/net_eth0-day-danda.png
    The reply is currently minimized Show
  • Accepted Answer

    Mansoor
    Mansoor
    Offline
    Friday, September 15 2017, 12:16 AM - #Permalink
    Resolved
    0 votes
    Thank you Tony for this valuable addition.

    I've have just finished transferring my first portion of files to the clearOS server. Here is the rsync summary:
    sent 407.64G bytes received 25.01K bytes 27.82M bytes/sec
    total size is 407.59G speedup is 1.00

    afpfs-ng worked after all in clearOS 7.3. I used these commands:
    mount_afp afp://user:pass@192.168.1.53/Data /mnt/airport
    rsync -e ssh -avh --bwlimit=40000 --exclude '.*' --progress /mnt/airport/Users/.... /var/flexshare/shares/....

    The afpfs-ng couldn't transfer files with UTF-8 characters in their names. So, I used CIFS to transfer them and here is the rsync summary:
    sent 119.43M bytes received 1.01K bytes 2.27M bytes/sec
    total size is 6.05G speedup is 50.63

    Notice the difference in speed! Apple always favors its own tech!

    All transfers went from a 2014 Time Capsule to a dual Xeon 3.2GHz clearOS 7.3 server over gigabit ethernet link.
    The reply is currently minimized Show
  • Accepted Answer

    Friday, September 15 2017, 03:33 AM - #Permalink
    Resolved
    0 votes
    The first 1.39T completed - the rest has to be done separately... (about 200M)

    sent 1388602421142 bytes received 23169454 bytes 47196043.52 bytes/sec
    total size is 1388347858342 speedup is 1.00

    Here is the complete transfer - obvious when thousands of tiny files were encountered - brought the average down a bit...
    http://www.sraellis.tk/net_eth0-complete_danda.png
    The reply is currently minimized Show
  • Accepted Answer

    Friday, September 15 2017, 12:52 PM - #Permalink
    Resolved
    0 votes
    Hmm. Between the server (Core-i3 4130) and my Windoze PC (Core-i3 4130T, SSD), using WinSCP I get c. 16MBps with the PC virus scanner enabled and 33.9MB/s with the virus scanner disabled. Need more horsepower!
    The reply is currently minimized Show
Your Reply