Tags
tool

This documentation provides optimized and parallelized methods for transferring Peertube media data from an LXC container to a remote server. It focuses on speed, reliability, and efficient handling of large file sets. All examples assume the source directory:

/var/lib/lxc/CTNAME/rootfs/var/www/peertube/storage/

and the destination:

root@TARTGETSERVERIP:/srv/peertube-media/

Overview

Peertube media often includes a very large number of small and medium‑sized files. Traditional tools like scp -r are slow and not optimal for repeated syncs or large datasets. The following tools offer significantly better performance through parallelism, connection reuse, and efficient delta transfers.

1. rsync + GNU Parallel

This method parallelizes transfers by launching multiple rsync jobs at once. It is especially effective when handling thousands or millions of small files.

Command:

find /var/lib/lxc/CTNAME/rootfs/var/www/peertube/storage/ -type f \
  | parallel -j 8 rsync -Ravh --progress {} root@TARTGETSERVERIP:/srv/peertube-media/

Notes:

  • -j 8 defines the number of parallel rsync processes.
  • -R keeps directory structure intact at the destination.
  • Very useful when file count is high and file sizes vary greatly.

2. rsync with SSH Multiplexing

This option speeds up transfers without true parallelization by reusing a persistent SSH connection. It reduces overhead and significantly improves throughput for many small files.

SSH Control Path Setup

mkdir -p ~/.ssh/control

rsync Command

rsync -avh --progress -e "ssh -o ControlMaster=auto -o ControlPath=~/.ssh/control/%r@%h:%p -o ControlPersist=600" \
  /var/lib/lxc/CTNAME/rootfs/var/www/peertube/storage/ \
  root@TARTGETSERVERIP:/srv/peertube-media/

Notes:

  • No configuration changes are needed on the remote server.
  • Ideal when rsync should behave normally but faster.

3. lftp Mirror (True Parallel SFTP Synchronization)

lftp mirror provides real parallel SFTP transfers with an adjustable number of concurrent connections. It is extremely fast and robust for large directory trees.

Command:

lftp -e "
  set ssl:verify-certificate no
  set net:timeout 20
  set net:max-retries 3
  set net:reconnect-interval-base 5
  set mirror:parallel-transfer-count 10
  mirror --reverse --verbose \
    /var/lib/lxc/CTNAME/rootfs/var/www/peertube/storage \
    /srv/peertube-media
  bye
" sftp://root@TARTGETSERVERIP

Notes:

  • mirror:parallel-transfer-count controls the number of parallel file transfers.
  • Highly stable even with millions of files.
  • Resume‑safe and ideal for repeated synchronization.

4. rclone (Cloud‑Grade Parallel Synchronization)

rclone offers powerful parallel syncing with advanced caching and consistency checks. It often outperforms rsync in very large directories.

4.1 Configure Remote

rclone config

Create an SSH remote named remote.

4.2 Sync Command

rclone sync /var/lib/lxc/CTNAME/rootfs/var/www/peertube/storage \
  remote:/srv/peertube-media \
  --transfers=16 --checkers=16 --progress

Notes:

  • Extremely efficient for large file sets.
  • Often noticeably faster than rsync due to parallel operations.

Recommendation

For migrating Peertube media efficiently:

  • Best overall performance: lftp mirror — fast, robust, highly parallel.
  • Best option within the rsync ecosystem: rsync + GNU Parallel.

Both significantly outperform classic scp -r and standard rsync when dealing with large datasets.

End of documentation.