Automated PikPak Monitor & Downloader for Debian 13

1. Introduction

This guide details how to set up and use a Bash script designed to automate file synchronization from a PikPak cloud drive (mounted via rclone) to a local Debian 13 server.

Key Features:

• Automated Monitoring: Runs in the background (Daemon mode) and scans for new files every 5 minutes.
• Smart Mount Management: Automatically refreshes the rclone mount to detect new files, but uses a concurrency lock to prevent restarting the service if a download is currently in progress.
• Stability Checks: Verifies the existence of the mount point before attempting scans to prevent errors.
• Bandwidth Limiting: Limits downloads to 10 MiB/s to prevent network congestion.
• Interactive Menu: Provides a user-friendly CLI menu to add, delete, and view tasks.

---

2. Prerequisites

Before installing the script, ensure your system meets the following requirements:

1. Operating System: Debian 13 (Trixie) or a compatible Linux distribution.
2. Root Privileges: You must have sudo or root access to manage system services and write to /mnt.
3. Rclone Installed & Configured:

• Rclone must be installed.
• You must have a valid PikPak remote configured (e.g., pikpak:).

4. Systemd Mount Service:

• You must have a systemd service (e.g., rclone-pikpak.service) that mounts your PikPak drive to a local directory (e.g., /mnt/pikpak).

---

3. Installation

Step 1: Install Dependencies

While the script attempts to install missing tools, it is best practice to update your system and install them manually first.

sudo apt update
sudo apt install -y curl jq procps

Step 2: Create the Script

Create a new file named pikpak_sync.sh and open it with your preferred editor (nano or vim).

nano pikpak_sync.sh

Paste the following finalized code into the file:

#!/bin/bash

# =========================================================
# Script Name: PikPak Auto-Monitor & Downloader (Final)
# System: Debian 13
# Dependencies: rclone, jq, systemd, procps
# Features: Concurrency locks, Smart mount detection, Bandwidth limit
# =========================================================

# Configuration Directory
BASE_DIR="/mnt/pikpak_monitor"
mkdir -p "$BASE_DIR"

# === USER CONFIGURATION ===
# The name of your systemd service for the rclone mount
SERVICE_NAME="rclone-pikpak"

# Critical directory to check if the mount is successful
# The script will only scan if this directory exists
CHECK_MOUNT_DIR="/mnt/pikpak/AVs" 

# Colors
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color

# ---------------------------------------------------------
# Dependency Check
# ---------------------------------------------------------
check_dependencies() {
    local missing=0
    for cmd in rclone jq pgrep; do
        if ! command -v $cmd &> /dev/null; then
            echo -e "${RED}Error: Command $cmd not found.${NC}"
            missing=1
        fi
    done
    
    if [ $missing -eq 1 ]; then
        echo -e "${YELLOW}Installing missing dependencies (jq, procps)...${NC}"
        apt-get update && apt-get install -y jq procps
    fi
}

# ---------------------------------------------------------
# Background Daemon Logic
# ---------------------------------------------------------
monitor_daemon() {
    local src_path="$1"
    local dest_path="$2"
    local task_name="$3"
    local json_file="${BASE_DIR}/${task_name}.json"
    local log_file="${BASE_DIR}/${task_name}.log"

    while true; do
        # === 1. Concurrency Lock & Service Refresh ===
        # Check if ANY rclone copy process is running to prevent service interruption
        if pgrep -f "rclone copy" > /dev/null; then
            echo "[$(date '+%Y-%m-%d %H:%M:%S')] Download in progress (rclone copy detected). Skipping service restart." >> "$log_file"
        else
            echo "[$(date '+%Y-%m-%d %H:%M:%S')] No active downloads. Restarting $SERVICE_NAME to refresh cache..." >> "$log_file"
            systemctl restart "$SERVICE_NAME" >> "$log_file" 2>&1
        fi
        
        # === 2. Smart Mount Detection ===
        local wait_count=0
        local max_retries=24  # 120 seconds timeout
        local mount_ready=0

        while [ $wait_count -lt $max_retries ]; do
            if [ -d "$CHECK_MOUNT_DIR" ]; then
                mount_ready=1
                break
            fi
            sleep 5
            ((wait_count++))
        done

        if [ $mount_ready -eq 0 ]; then
            echo "[$(date '+%Y-%m-%d %H:%M:%S')] Error: Mount timeout! $CHECK_MOUNT_DIR not found. Skipping scan." >> "$log_file"
            sleep 60
            continue
        fi

        # === 3. Scanning & Downloading ===
        # Get current file list
        current_files_json=$(rclone lsf --files-only --format "p" "$src_path" 2>>"$log_file" | jq -R -s -c 'split("\n")[:-1]')
        
        if [ -z "$current_files_json" ]; then
            echo "[$(date '+%Y-%m-%d %H:%M:%S')] Scan empty or failed." >> "$log_file"
            sleep 300 
            continue
        fi

        # Initialize JSON if missing
        if [ ! -f "$json_file" ]; then
            echo "[]" > "$json_file"
        fi
        
        # Compare for new files
        new_files=$(echo "$current_files_json" | jq -r --argjson old "$(cat "$json_file")" '. - $old | .[]')

        if [ -n "$new_files" ]; then
            IFS=$'\n'
            for file in $new_files; do
                echo "[$(date '+%Y-%m-%d %H:%M:%S')] New file found: $file. Downloading (Limit: 10MB/s)..." >> "$log_file"
                
                rclone copy "${src_path}/${file}" "$dest_path" --bwlimit 10M --log-file="$log_file" --log-level INFO
                
                if [ $? -eq 0 ]; then
                    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Download Success: $file" >> "$log_file"
                    # Update JSON record
                    tmp_json=$(mktemp)
                    jq --arg new_file "$file" '. + [$new_file]' "$json_file" > "$tmp_json" && mv "$tmp_json" "$json_file"
                else
                    echo "[$(date '+%Y-%m-%d %H:%M:%S')] Download Failed: $file" >> "$log_file"
                fi
            done
            unset IFS
        else
            echo "[$(date '+%Y-%m-%d %H:%M:%S')] No new files." >> "$log_file"
        fi

        # Wait 5 minutes before next cycle
        sleep 300
    done
}

# ---------------------------------------------------------
# Interactive Menu
# ---------------------------------------------------------

add_task() {
    echo -e "${BLUE}=== Add New Task ===${NC}"
    echo -e "${YELLOW}Mount Check Directory: $CHECK_MOUNT_DIR ${NC}"
    read -e -p "Enter Monitor Folder (Absolute Path, e.g., pikpak:Video): " src_path
    read -e -p "Enter Download Destination (Absolute Local Path): " dest_path

    src_path=${src_path%/}
    task_name=$(basename "$src_path")
    
    json_file="${BASE_DIR}/${task_name}.json"
    pid_file="${BASE_DIR}/${task_name}.pid"
    log_file="${BASE_DIR}/${task_name}.log"

    if [ -f "$pid_file" ]; then
        if kill -0 $(cat "$pid_file") 2>/dev/null; then
            echo -e "${RED}Error: Task [$task_name] is already running.${NC}"
            return
        fi
    fi

    echo -e "${YELLOW}Initializing file list...${NC}"
    rclone lsf --files-only --format "p" "$src_path" | jq -R -s -c 'split("\n")[:-1]' > "$json_file"
    
    echo -e "Starting background process..."
    nohup bash "$0" --daemon "$src_path" "$dest_path" "$task_name" >/dev/null 2>&1 &
    
    pid=$!
    echo $pid > "$pid_file"
    
    echo -e "${GREEN}Task [$task_name] started (PID: $pid)${NC}"
    echo -e "Log file: $log_file"
    read -p "Press Enter to return..."
}

delete_task() {
    echo -e "${BLUE}=== Delete Task ===${NC}"
    files=(${BASE_DIR}/*.pid)
    if [ ! -e "${files[0]}" ]; then
        echo "No running tasks found."
        read -p "Press Enter to return..."
        return
    fi

    echo "Running Tasks:"
    i=1
    declare -a tasks
    for file in "${files[@]}"; do
        name=$(basename "$file" .pid)
        pid=$(cat "$file")
        if kill -0 "$pid" 2>/dev/null; then
            status="${GREEN}Running${NC}"
        else
            status="${RED}Stopped (Zombie)${NC}"
        fi
        echo -e "$i. Name: $name (PID: $pid) - $status"
        tasks[$i]=$name
        ((i++))
    done

    echo ""
    read -p "Select task number to delete (0 to cancel): " choice

    if [[ "$choice" =~ ^[0-9]+$ ]] && [ "$choice" -ge 1 ] && [ "$choice" -lt "$i" ]; then
        target_name=${tasks[$choice]}
        target_pid_file="${BASE_DIR}/${target_name}.pid"
        target_pid=$(cat "$target_pid_file")
        
        echo "Stopping process $target_pid ..."
        kill "$target_pid" 2>/dev/null
        rm "$target_pid_file"
        echo -e "${GREEN}Task [$target_name] stopped.${NC}"
        
        read -p "Delete JSON records and Logs for this task? (y/n): " del_files
        if [[ "$del_files" == "y" ]]; then
            rm -f "${BASE_DIR}/${target_name}.json"
            rm -f "${BASE_DIR}/${target_name}.log"
            echo "Files cleaned."
        fi
    else
        echo "Cancelled."
    fi
    read -p "Press Enter to return..."
}

view_tasks() {
    echo -e "${BLUE}=== View Task Logs ===${NC}"
    files=(${BASE_DIR}/*.log)
    if [ ! -e "${files[0]}" ]; then
        echo "No log files found."
        read -p "Press Enter to return..."
        return
    fi

    i=1
    declare -a logs
    for file in "${files[@]}"; do
        name=$(basename "$file" .log)
        echo "$i. Task: $name"
        logs[$i]=$file
        ((i++))
    done

    echo ""
    read -p "Select log to view (Ctrl+C to exit log view): " choice

    if [[ "$choice" =~ ^[0-9]+$ ]] && [ "$choice" -ge 1 ] && [ "$choice" -lt "$i" ]; then
        target_log=${logs[$choice]}
        echo -e "${YELLOW}Streaming Log (Press Ctrl+C to exit)...${NC}"
        sleep 1
        tail -f "$target_log"
    else
        echo "Invalid selection."
    fi
}

# ---------------------------------------------------------
# Main Entry Point
# ---------------------------------------------------------

if [ "$1" == "--daemon" ]; then
    if [ -z "$2" ] || [ -z "$3" ] || [ -z "$4" ]; then
        exit 1
    fi
    monitor_daemon "$2" "$3" "$4"
    exit 0
fi

if [[ $EUID -ne 0 ]]; then
   echo -e "${YELLOW}Please run as root (sudo).${NC}"
   sleep 2
fi

check_dependencies

while true; do
    clear
    echo -e "${BLUE}=======================================${NC}"
    echo -e "   PikPak Monitor & Downloader (Debian 13)"
    echo -e "${BLUE}=======================================${NC}"
    echo "1. Add Monitor Task"
    echo "2. Delete Monitor Task"
    echo "3. View Running Tasks (Logs)"
    echo "4. Exit"
    echo -e "${BLUE}=======================================${NC}"
    
    read -p "Enter Option [1-4]: " choice
    
    case $choice in
        1) add_task ;;
        2) delete_task ;;
        3) view_tasks ;;
        4) echo "Exiting."; exit 0 ;;
        *) echo "Invalid option."; sleep 1 ;;
    esac
done

Step 3: Configure the Script

Before running, you must edit lines 16 and 20 to match your environment:

• SERVICE_NAME: Change this to the exact name of your systemd service file (e.g., rclone-mount, pikpak).
• CHECK_MOUNT_DIR: Set this to a folder inside your mounted drive that is guaranteed to exist. The script uses this to verify the mount is healthy.

Step 4: Make Executable & Install System-wide

Make the script executable and install it to /usr/local/bin under the name piksync for easy access.

chmod +x pikpak_sync.sh
sudo install -m 755 pikpak_sync.sh /usr/local/bin/piksync

---

4. Usage Guide

To start the program, simply type piksync in your terminal from anywhere:

sudo piksync

Menu Option 1: Add Monitor Task

Use this to start monitoring a specific folder.

1. Monitor Folder: Enter the rclone path (e.g., pikpak:Movies/Action).
2. Download Destination: Enter the absolute local path (e.g., /home/user/Downloads).
3. Process: The script creates a .json snapshot of current files, then launches a background process.

Menu Option 2: Delete Monitor Task

Use this to stop a background process.

1. The menu lists all active tasks (PIDs).
2. Select the number corresponding to the task you want to stop.
3. You can choose to keep or delete the JSON logs for that task.

Menu Option 3: View Running Tasks

Use this to check the real-time activity of your tasks.

1. Select a task to view its log.
2. You will see entries like “Restarting Service,” “Downloading,” or “Download Success.”
3. Press Ctrl+C to return to the menu (this does not stop the background download).

---

5. Technical Architecture

1. The Concurrency Lock (pgrep)

To support multiple monitoring tasks running simultaneously (e.g., Movies, TV Shows, Music) without conflict:

• Before any task attempts to restart the systemd service (to refresh the file list), it runs pgrep -f "rclone copy".
• If any download process is detected (even from another task), the restart command is skipped.
• This ensures that Task A does not kill Task B’s active connection.

2. Smart Mount Detection

Instead of blindly scanning, the script verifies CHECK_MOUNT_DIR:

• After a service restart (or skip), it polls for the directory every 5 seconds (up to 2 minutes).
• If the directory is not found, the scan is aborted to prevent errors.

3. State Management

• Logs & PIDs: Stored in /mnt/pikpak_monitor/.
• History: A JSON file (taskname.json) tracks downloaded files to prevent duplicates.

---

6. Troubleshooting

Q: The script says “Mount timeout!” in the logs.

• A: Check your internet connection and ensure the CHECK_MOUNT_DIR path in the script matches a real folder in your PikPak drive.

Q: I added a file to PikPak, but it isn’t downloading.

• A: The script scans every 5 minutes. Also, if another task is currently downloading a large file, the service restart (list refresh) is skipped until that download finishes.

Q: Can I close the SSH terminal?

• A: Yes. The tasks are launched using nohup, so they will continue running in the background even after you disconnect.

Below is a modular explanation of the provided Bash script, organized by functional blocks and written in formal English. The script’s overall objective is to continuously monitor an rclone-accessible remote folder (e.g., PikPak), detect newly appeared files, and download them to a local destination while preserving state across cycles.

---

Detailed Description

1. Script Header and High-Level Intent

Purpose. The script is titled “PikPak Auto-Monitor & Downloader.” It is designed for a Debian-based system and assumes the presence of:

• rclone for remote listing and copying,
• jq for JSON processing (tracking previously seen files),
• systemd for restarting an rclone mount service,
• procps (specifically pgrep) for process inspection.

Operational model. It offers an interactive menu (add/delete/view tasks), but the actual monitoring work runs as background daemons started via nohup and controlled using PID files.

---

2. Global Configuration and Directory Initialization

2.1 State directory (BASE_DIR)

BASE_DIR="/mnt/pikpak_monitor"
mkdir -p "$BASE_DIR"

• Establishes a persistent directory to store per-task artifacts:

  • *.json (state: already known files),
  • *.pid (daemon process identifiers),
  • *.log (task logs).

2.2 User configuration: systemd service and mount check path

SERVICE_NAME="rclone-pikpak"
CHECK_MOUNT_DIR="/mnt/pikpak/AVs"

• SERVICE_NAME is the systemd unit the script restarts to refresh mount/cache state.
• CHECK_MOUNT_DIR is a “health check” directory used to confirm the mount is present before scanning. The script will not proceed to scan unless this directory exists.

2.3 ANSI color constants

Defines color codes for menu output. These are cosmetic and do not affect logic.

---

3. Dependency Verification Module (check_dependencies)

3.1 Detection of required commands

for cmd in rclone jq pgrep; do
    if ! command -v $cmd &> /dev/null; then
        ...
    fi
done

• Verifies rclone, jq, and pgrep exist in PATH.
• If any are missing, missing=1 is set.

3.2 Automatic installation (partial)

apt-get update && apt-get install -y jq procps

• If any command is missing, it attempts to install jq and procps.
• Note: even though it checks for rclone, it does not install it. rclone must be installed separately.

---

4. Core Background Worker Module (monitor_daemon)

This is the primary automation loop. Each monitoring task runs this function with parameters:

• src_path: the remote folder (e.g., pikpak:Video)
• dest_path: the local download directory
• task_name: derived from the basename of src_path (used for file naming)

Artifacts created per task:

• json_file="${BASE_DIR}/${task_name}.json"
• log_file="${BASE_DIR}/${task_name}.log"

The daemon runs indefinitely:

while true; do
   ...
   sleep 300
done

4.1 Concurrency guard and service refresh logic

if pgrep -f "rclone copy" > /dev/null; then
    ... Skipping service restart.
else
    systemctl restart "$SERVICE_NAME"
fi

• The script avoids restarting the mount service if any process matching rclone copy is currently running.
• The intent is to prevent disrupting ongoing downloads and to reduce the risk of transient mount failures during copying.
• If no active copy is detected, it restarts the mount service to “refresh cache.”

4.2 Smart mount detection (readiness loop)

local max_retries=24  # 120 seconds timeout
while [ $wait_count -lt $max_retries ]; do
    if [ -d "$CHECK_MOUNT_DIR" ]; then
        mount_ready=1
        break
    fi
    sleep 5
    ((wait_count++))
done

• Checks for existence of CHECK_MOUNT_DIR every 5 seconds, up to 24 times (about 120 seconds).
• If the mount is not ready after the timeout, it logs an error, sleeps 60 seconds, and starts a new cycle without scanning.

4.3 Scanning remote folder: obtaining current file list as JSON

current_files_json=$(rclone lsf --files-only --format "p" "$src_path" | jq -R -s -c 'split("\n")[:-1]')

• rclone lsf --files-only --format "p" lists file paths (no directories) from the remote folder.
• The output is piped into jq to convert it into a JSON array of strings.
• split("\n")[:-1] removes the trailing empty entry caused by a terminal newline.

If the scan output is empty or failed, it logs and sleeps 300 seconds, then retries.

4.4 Persistent state initialization

if [ ! -f "$json_file" ]; then
    echo "[]" > "$json_file"
fi

• Ensures there is a JSON record file.
• This file represents the set of already seen/processed files.

4.5 Delta computation: identify new files

new_files=$(echo "$current_files_json" | jq -r --argjson old "$(cat "$json_file")" '. - $old | .[]')

• Uses jq’s array subtraction: current - old to compute newly observed files.
• Emits each new file as a separate line.

4.6 Download workflow with bandwidth limiting

For each new file:

rclone copy "${src_path}/${file}" "$dest_path" --bwlimit 10M --log-file="$log_file" --log-level INFO

• Executes rclone copy for the single file.
• Applies --bwlimit 10M (10 MB/s), controlling bandwidth usage.
• Logs rclone output to the per-task log file.

On success, it appends the filename to the JSON record:

jq --arg new_file "$file" '. + [$new_file]' "$json_file" > "$tmp_json" && mv "$tmp_json" "$json_file"

• Uses a temporary file and atomic mv to reduce risk of JSON corruption.
• On failure, it logs the failure and does not update the record, allowing retries in subsequent cycles.

4.7 Scheduling cadence

After each complete scan-and-download round, the daemon sleeps for 300 seconds (5 minutes).

---

5. Interactive Task Management Modules

The interactive interface is a menu-driven loop that creates and controls daemon processes.

5.1 Add task (add_task)

Key steps:

1. Prompts user for:

   • src_path (remote folder)
   • dest_path (local folder)

2. Normalizes remote path by removing a trailing slash.

3. Derives task_name from basename "$src_path".

Artifacts per task:

• ${task_name}.json: initial snapshot and later file history
• ${task_name}.pid: background daemon PID
• ${task_name}.log: running logs

Existing task detection. If the PID file exists and the process is alive (kill -0), it refuses to start a second copy.

Initialization snapshot.

rclone lsf ... | jq ... > "$json_file"

• This baseline prevents the task from immediately downloading all existing files; it will download only files appearing after task creation.

Start daemon.

nohup bash "$0" --daemon "$src_path" "$dest_path" "$task_name" ...
pid=$!
echo $pid > "$pid_file"

• Launches the same script in daemon mode with --daemon.
• Stores PID for later control.

5.2 Delete task (delete_task)

Key steps:

1. Lists *.pid files in BASE_DIR to find tasks.

2. For each, checks if PID is alive (kill -0) and reports status:

   • Running
   • Stopped (Zombie) — meaning PID file exists but process is not running

3. User selects a task number; the script kills the process and removes PID file.

4. Optionally removes the JSON state and log file.

5.3 View logs (view_tasks)

Key steps:

1. Lists available *.log files.
2. User selects one.
3. Streams with:

tail -f "$target_log"

• This provides real-time monitoring until the user interrupts with Ctrl+C.

---

6. Daemon Entry Point and Main Menu Loop

6.1 Daemon mode

if [ "$1" == "--daemon" ]; then
    monitor_daemon "$2" "$3" "$4"
fi

• Validates three required parameters exist.
• Runs the monitoring loop for the specified task.

6.2 Root privilege check

if [[ $EUID -ne 0 ]]; then
   echo ... "Please run as root (sudo)."
   sleep 2
fi

• Warns if not root, but does not hard-exit.

• This is relevant because:

  • dependency installation via apt-get typically requires root,
  • restarting systemd services requires appropriate privileges.

6.3 Startup dependency check and interactive loop

After calling check_dependencies, it enters an infinite loop that displays the menu and dispatches to add_task, delete_task, or view_tasks.

---

7. Summary of Key Design Choices

• Stateful “new file” detection: a JSON array of previously seen filenames per task.
• Service refresh with concurrency check: restarts the mount service only when no rclone copy is active.
• Mount readiness gating: scanning is permitted only if a specific mount directory exists.
• Bandwidth control: --bwlimit 10M limits download speed.
• Multi-task management: PID files allow multiple independent monitoring tasks to run concurrently, each with separate JSON and log files.