Laravel Chores: Resumable, Checkpointed Data Operations for Large Datasets

The Problem: Interrupted Bulk Operations

Backfilling ten million rows is straightforward until a deploy restarts the server four hours in. Without checkpointing, you're left guessing which records were already processed and restarting from scratch. Laravel Chores, by Amr Lotfy Saleh, solves this by persisting progress to the database after every batch, inspired by Shopify's maintenance_tasks gem for Rails.

Key Features

• Checkpointed progress — the last processed ID is written to a chore_runs table after each batch; a crash loses at most one batch
• Keyset pagination — pages by primary key rather than offset, avoiding the classic bug where updating rows shifts them out of the current chunk
• Failure isolation — exceptions are logged to a failures table and skipped; the run continues
• No extra infrastructure — state lives in your database; no Redis or queue workers required
• Six Artisan commands — scaffold, run, list, pause, inspect failures, and retry
• CI-friendly output — JSON mode and distinct exit codes: 0 clean, 1 completed with failures, 2 fatal

Writing a Chore

A chore class needs two methods: collection() returns the query, and process() handles one record. Scaffold one with:

php artisan make:chore NormalizePhoneNumbers

namespace App\Chores;

use AmrLotfy\Chores\Chore;
use App\Models\User;
use Illuminate\Contracts\Database\Eloquent\Builder;

class NormalizePhoneNumbers extends Chore
{
    public int $batchSize = 500;

    public function collection(): Builder
    {
        return User::whereNotNull('phone')
            ->where('phone', 'not like', '+%');
    }

    public function process($record): void
    {
        $record->update([
            'phone' => PhoneNumber::parse($record->phone, 'EG')->toE164(),
        ]);
    }
}

Batching, progress tracking, and failure logging are handled by the package around your two methods. The default batch size is 500 and can be overridden per class or via the config file.

Running, Pausing, and Resuming

php artisan chore:run NormalizePhoneNumbers

Progress checkpoints after each batch. If the run is interrupted, the same command resumes from the last saved position. Use chore:pause to stop at the next batch boundary, and chore:list to view available chores and their run history.

Idempotency note: The checkpoint is per batch, not per record. Records inside an in-flight batch may be re-examined after a resume. Write process() to be idempotent where possible — the phone normalization example above is safe because already-normalized numbers no longer match the collection() query.

For recurring work, compose with Laravel's task scheduler:

$schedule->command('chore:run PurgeExpiredRecords')->monthly();

Handling Failures

A record that throws an exception is logged and skipped; the run continues. Afterwards:

php artisan chore:failures NormalizePhoneNumbers
php artisan chore:retry NormalizePhoneNumbers

This split is valuable on long runs: a hundred malformed records out of ten million shouldn't kill a four-hour job, and retrying only the failures after a fix is far cheaper than rerunning everything.

Installation

Requires PHP 8.2+, Laravel 12 or 13, and supports MySQL, PostgreSQL, and SQLite.

composer require amrlotfy/laravel-chores
php artisan vendor:publish --tag=chores-migrations
php artisan migrate

The config file lets you adjust the chore class directory, default batch size, table names, and a sleep interval between batches for throttling load on busy databases.

Limitations to Know

Chores run in the foreground with a single worker per chore, and the collection requires an orderable primary key (auto-increment or ULID). Queued and parallel execution are on the roadmap.

Takeaways

• Checkpointed batches mean interrupted runs resume rather than restart
• Keyset pagination prevents the offset-drift bug on mutable datasets
• Failure isolation keeps a long job alive despite bad records
• Zero infrastructure overhead — just two database tables
• Idempotent process() methods are the key to safe resumption

Source: Laravel Chores: Resumable Data Operations and Cleanups — Laravel News