Otdfbt

I'm using a script like that:

rsync -avu --delete --progress user@host:~/backup .

and every time, after a while, a receive this error:

rsync: connection unexpectedly closed (10251754 bytes received so far) [receiver]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [receiver=3.1.0]
rsync: connection unexpectedly closed (45482 bytes received so far) [generator]
rsync error: unexplained error (code 255) at io.c(226) [generator=3.1.0]

# generator = Debian 5, rsync version 3.0.3 protocol version 30 (Generator[A])
# receiver = Ubuntu 14.04, rsync version 3.1.0 protocol version 31 (Receiver[A])

I have over ten years of experience in rsync, I tried everything, but I can't figure out why it happens.

On both sides: I have no errors in logs, no strange things in tcpdump, no connection problems, no firewalls. The transfer just suddently stops after sometime.

The problem is not limited to one server.

I thought it could be a some king of incompatibility between rsync versions, so I switch the generator with a fresh new server (Debian 8) on a completely different server farm and I have update rsync on the receiver, but I get this error:

rsync: connection unexpectedly closed (31931964 bytes received so far) [receiver]
rsync: [generator] write error: Broken pipe (32)
rsync error: unexplained error (code 255) at io.c(820) [generator=3.1.2]
rsync error: error in rsync protocol data stream (code 12) at io.c(226) [receiver=3.1.2]

# generator = Debian 8, rsync version 3.1.1 protocol version 31 (Generator[B])
# receiver = Ubuntu 14.04, rsync version 3.1.2 protocol version 31 (Receiver[B])

Then I tried to sync Receiver[A] with Generator[B] throught a proxy and it works

Receiver[A](Italy) <=> HTTP Proxy(Paris) <=> Generator[B](Strasbourg)

So now I have

Receiver[A] <=> Generator[A] // ERROR
Receiver[A] <=> Generator[B] // ERROR 
Receiver[B] <=> Generator[B] // ERROR

Receiver[A] <=> HTTP Proxy <=> Generator[A] // OK
Receiver[A] <=> HTTP Proxy <=> Generator[B] // OK 

I tried too without HTTP Proxy

Receiver[A] <=> Generator[C] // OK
Receiver[A] <=> Generator[D] // OK 
Receiver[A] <=> Generator[E] // ERROR

where generators [C], [D] and [E] are servers in Paris (same server farm of my HTTP Proxy) with rsync 3.0.9, prot. v. 30.

So the bottom line here is: I have multiple servers in multiple locations. On Receiver[A] I have no firewall and things works well since few days ago. Now rsync return unexplained error and I can't figure out why.

I think it's a some king of phisical network error (not on my machines), but I don't know if I can find, isolate and reproduce this error.

Thanks for your help!

It seems that you are having a hard time pinpointing the issue but no problems reproducing it. In this case, it may help if you troubleshoot the SSH connection itself, and also the data that you are getting over it.

You can run your rsync command like below, it would produce some data du:

rsync -e 'bash -x -c "ssh -vvvv $0 $@ 2>/tmp/rsync-ssh.stderr | tee /tmp/rsync.stdout"' -avu --delete --progress user@host:~/backup .

Looking at the end of those files might have something useful. E.g., if the SSH connections was interrupted, something should show up in /tmp/rsync-ssh.stderr. And if it is actually a problem on the other side, the actual rsync protocol dump in /tmp/rsync.stdout may have some stringified hint (strings /tmp/rsync.stdout).

If I had to take a guess, it is either a nasty firewall or a physical issue corrupting packets causing the SSH connection to drop.

It is possible that rsync might be writing faster to the disk cache than the disk on the server is able to keep up with, see for example https://unix.stackexchange.com/a/331189/163108 for a more detailed explanation, and https://unix.stackexchange.com/a/523896/163108 for a similar problem but different case when using rsync over a samba/cifs mounted drive.

A possible solution could be to throttle rsync using the --bwlimit= option. Depending on the network/disk performance of the server, I would recommend trying values between 2000-10000 (in kbit/s).