Why Splitting Files into Chunks Makes Backup & Restore Faster, Safer, and More Reliable

When dealing with WordPress backups, especially for large sites with thousands of files and multi-gigabyte databases, traditional "all-at-once" approaches often fail. Chunked file processing—splitting large operations into smaller, manageable pieces—is the key to reliable, scalable backup and restore operations.

The Problem with Monolithic File Operations

WordPress sites can grow massive. A typical e-commerce site might have:

50,000+ product images in wp-content/uploads/
A 500MB+ database with years of order history
Hundreds of plugins and themes
Custom upload directories

Attempting to process these files in a single operation leads to:

Memory Exhaustion

// ❌ This will fail with large files
$file_content = file_get_contents('/path/to/500mb-backup.zip');
$extracted = zip_open($file_content);
// Fatal error: Allowed memory size exhausted

PHP's default memory limits (often 128MB-256MB) are quickly exceeded when loading entire files into memory.

Execution Timeouts

// ❌ This will timeout
foreach ($all_files as $file) {
    copy($file, $backup_location);
    // Takes 5+ minutes, exceeds max_execution_time
}

WordPress and PHP have execution time limits (typically 30-60 seconds). Large operations exceed these limits, causing incomplete backups.

Network Transfer Failures

Uploading a 2GB backup file through a browser or single HTTP request often fails due to:

Browser timeout limits
Server request size limits (upload_max_filesize, post_max_size)
Network interruptions
Proxy/gateway timeouts

How Chunking Solves These Problems

Chunking breaks large operations into smaller, sequential pieces that can be processed reliably within memory and time constraints.

Memory Efficiency

Instead of loading entire files, we process them in fixed-size chunks:

/**
 * Read file in chunks to avoid memory exhaustion
 * 
 * @param string $file_path Path to file
 * @param int $chunk_size Size of each chunk in bytes (default 1MB)
 * @return Generator Yields file chunks
 */
function read_file_in_chunks($file_path, $chunk_size = 1048576) {
    $handle = fopen($file_path, 'rb');
    
    if (!$handle) {
        throw new Exception("Cannot open file: {$file_path}");
    }
    
    while (!feof($handle)) {
        yield fread($handle, $chunk_size);
    }
    
    fclose($handle);
}

// Usage: Process large file without loading it entirely
foreach (read_file_in_chunks('/path/to/large-backup.zip') as $chunk) {
    // Process each 1MB chunk
    upload_chunk($chunk);
    // Memory usage stays constant
}

Benefits:

Constant memory usage regardless of file size
Can process files larger than available RAM
Works within PHP memory limits

Timeout Prevention

By processing in chunks and resetting execution timers, we avoid timeout errors:

/**
 * Process files with chunking and execution time management
 * 
 * @param array $files List of file paths
 * @param int $chunk_size Number of files per batch
 */
function process_files_in_chunks($files, $chunk_size = 100) {
    $total = count($files);
    $processed = 0;
    
    // Process in batches
    $chunks = array_chunk($files, $chunk_size);
    
    foreach ($chunks as $chunk) {
        // Reset execution time for each chunk
        set_time_limit(30);
        
        foreach ($chunk as $file) {
            backup_file($file);
            $processed++;
        }
        
        // Optional: Log progress
        error_log("Processed {$processed}/{$total} files");
        
        // Optional: Small delay to prevent server overload
        usleep(10000); // 10ms
    }
}

Benefits:

Each chunk completes within timeout limits
Progress can be tracked and resumed
Server resources are managed better

Reliable Network Transfers

For uploads, chunking enables resumable, reliable transfers:

/**
 * Upload file in chunks via REST API
 * 
 * @param string $file_path Local file path
 * @param string $session_id Upload session identifier
 * @param int $chunk_size Chunk size in bytes
 */
function upload_file_chunks($file_path, $session_id, $chunk_size = 5242880) {
    $file_size = filesize($file_path);
    $total_chunks = ceil($file_size / $chunk_size);
    $handle = fopen($file_path, 'rb');
    
    $chunk_number = 0;
    
    while (!feof($handle)) {
        $chunk_data = fread($handle, $chunk_size);
        $chunk_hash = md5($chunk_data);
        
        // Upload single chunk
        $response = wp_remote_post(rest_url('worry-proof-backup/v1/upload-chunk'), [
            'body' => [
                'session_id' => $session_id,
                'chunk_number' => $chunk_number,
                'total_chunks' => $total_chunks,
                'chunk_data' => base64_encode($chunk_data),
                'chunk_hash' => $chunk_hash,
            ],
            'timeout' => 60,
        ]);
        
        if (is_wp_error($response)) {
            // Retry logic can be implemented here
            throw new Exception("Chunk upload failed: " . $response->get_error_message());
        }
        
        $chunk_number++;
    }
    
    fclose($handle);
    
    // Finalize upload
    wp_remote_post(rest_url('worry-proof-backup/v1/finalize-upload'), [
        'body' => ['session_id' => $session_id],
    ]);
}

Benefits:

Each chunk is small enough to upload reliably
Failed chunks can be retried individually
Progress can be tracked
Works around browser/server upload limits

WordPress-Specific Considerations

Database Chunking

WordPress databases can be massive. Chunking SQL exports prevents memory issues:

/**
 * Export WordPress database in chunks
 * 
 * @param string $output_file Output SQL file path
 * @param int $rows_per_chunk Number of rows per chunk
 */
function export_database_chunks($output_file, $rows_per_chunk = 1000) {
    global $wpdb;
    
    $tables = $wpdb->get_col("SHOW TABLES");
    $handle = fopen($output_file, 'w');
    
    foreach ($tables as $table) {
        // Write table structure
        $create_table = $wpdb->get_row("SHOW CREATE TABLE `{$table}`", ARRAY_N);
        fwrite($handle, "DROP TABLE IF EXISTS `{$table}`;\n");
        fwrite($handle, $create_table[1] . ";\n\n");
        
        // Export data in chunks
        $total_rows = $wpdb->get_var("SELECT COUNT(*) FROM `{$table}`");
        $offset = 0;
        
        while ($offset < $total_rows) {
            $rows = $wpdb->get_results(
                "SELECT * FROM `{$table}` LIMIT {$rows_per_chunk} OFFSET {$offset}",
                ARRAY_A
            );
            
            if (empty($rows)) break;
            
            foreach ($rows as $row) {
                $values = array_map([$wpdb, '_escape'], $row);
                $values_str = "'" . implode("','", $values) . "'";
                fwrite($handle, "INSERT INTO `{$table}` VALUES ({$values_str});\n");
            }
            
            $offset += $rows_per_chunk;
        }
    }
    
    fclose($handle);
}

File System Chunking

WordPress wp-content/uploads/ can contain millions of files. Chunked iteration prevents timeouts:

/**
 * Backup WordPress uploads directory in chunks
 * 
 * @param string $source_dir Source directory
 * @param string $backup_dir Backup destination
 * @param int $files_per_chunk Number of files per batch
 */
function backup_uploads_chunks($source_dir, $backup_dir, $files_per_chunk = 500) {
    $iterator = new RecursiveIteratorIterator(
        new RecursiveDirectoryIterator($source_dir),
        RecursiveIteratorIterator::LEAVES_ONLY
    );
    
    $files = [];
    foreach ($iterator as $file) {
        if ($file->isFile()) {
            $files[] = $file->getPathname();
        }
    }
    
    // Process in chunks
    $chunks = array_chunk($files, $files_per_chunk);
    
    foreach ($chunks as $chunk_index => $chunk) {
        set_time_limit(60); // Reset timer
        
        foreach ($chunk as $file_path) {
            $relative_path = str_replace($source_dir, '', $file_path);
            $backup_path = $backup_dir . $relative_path;
            
            // Create directory if needed
            $backup_file_dir = dirname($backup_path);
            if (!is_dir($backup_file_dir)) {
                wp_mkdir_p($backup_file_dir);
            }
            
            // Copy file
            copy($file_path, $backup_path);
        }
        
        // Log progress
        $progress = (($chunk_index + 1) / count($chunks)) * 100;
        update_option('backup_progress', $progress);
    }
}

Performance Comparison

Without Chunking

Memory Usage: 500MB+ (entire file in memory)
Execution Time: 300+ seconds (often times out)
Success Rate: ~60% (fails on large sites)
Resumability: No (must restart on failure)

With Chunking

Memory Usage: ~10MB (constant, regardless of file size)
Execution Time: 30-60 seconds per chunk (completes reliably)
Success Rate: ~99% (handles large sites)
Resumability: Yes (can resume from last chunk)

Implementation in Worry Proof Backup

Worry Proof Backup implements chunking for:

ZIP Archive Creation: Large backup files are created incrementally
ZIP Archive Extraction: Restore operations process files in batches
File Uploads: CLI and web uploads use chunked transfer
Database Operations: SQL exports are streamed in chunks
File System Operations: Directory operations are batched

This ensures reliable backups and restores even for sites with:

100GB+ of files
Databases with millions of rows
Thousands of plugins and themes

Best Practices

Choose Appropriate Chunk Sizes
- File reads: 1-5MB chunks
- Database rows: 500-2000 rows per chunk
- Network uploads: 5-10MB chunks
- File operations: 100-500 files per batch

Reset Execution Timers

set_time_limit(30); // Before each chunk

Track Progress

update_option('backup_progress', $percentage);

Handle Errors Gracefully

try {
    process_chunk($chunk);
} catch (Exception $e) {
    log_error($e);
    // Retry or skip
}

Provide User Feedback
- Show progress percentage
- Display current operation
- Allow cancellation

Conclusion

Chunked file processing is essential for reliable WordPress backup and restore operations. By breaking large operations into manageable pieces, we achieve:

Faster operations through better resource management
Safer processing by staying within memory and time limits
More reliable results with higher success rates and resumability

Worry Proof Backup leverages chunking throughout its architecture, ensuring your backups complete successfully regardless of site size.

Want to see chunking in action? Download Worry Proof Backup and experience reliable backups for sites of any size.