What to Audit First in a Monthly Cleanup Script Before It Deletes Wrong Data

You wrote a cleanup script to delete old logs. It passes dry-run with no errors. You run it for real, and suddenly your production server is missing yesterday's transaction files. The date filter was off by one day. That's the kind of mistake that makes you double-check everything before a monthly cleanup run.

Here's what to audit first in that script—before it deletes wrong data.

Where Monthly Cleanup Scripts Go Wrong in Real Ops

The false sense of security from dry-run

Dry-run modes feel like a safety net. You set --dry-run, watch the script print what it would delete, everything looks clean—so you flip the flag off and run it live. I have watched this pattern blow up three times in the last year alone. The problem is not the dry-run logic itself; it's that dry-runs often use a different code path. One team I worked with had a Python script where the dry-run mocked the filesystem with fake timestamps—it never actually queried the real metadata. So the dry-run printed perfect output, and the live run deleted client invoices from the wrong folder because the live path resolution used a different source of truth. The catch? The developer who wrote the dry-run helper hadn't tested against production-shaped data. That hurts.

Date math pitfalls across time zones

Most cleanup scripts work with file timestamps. "Delete anything older than 30 days." Simple, right? Then a server in UTC+2 stores logs, but the script runs on a machine set to UTC-5. Midnight calculation lands on the wrong boundary. Or the script uses os.path.getmtime() for a file that was modified by an FTP upload that preserved the original time zone—thirty-one days ago in GMT, but thirty-two in local time when DST flips. I once debugged a failure where a cleanup script deleted daily backups from the 30th of the month instead of the 1st because the comparison used floating-point epoch seconds while the retention policy was defined in calendar days. That mismatch took an hour to find and cost a weekend restore. Quick reality check—always convert all timestamps to a single zone before the comparison, not after.

File name patterns that look right but aren't

*.tmp feels bulletproof until someone names a production report Q4_final_tmp.xlsx. Or the glob temp_* matches files like temp_engine_config_backup.json that are not temporary at all. The real-world failure I see most often is when a script uses endswith(tmp) or a prefix match without confirming file intent—not just file name. A team once built a cleanup script targetting *.log files older than 7 days, and accidentally swept the debug logs for the monitoring system that rotated them hourly. No alert for six days because the logger stopped writing. The fix was cheap: add a double-check against file size (empty logs get deleted, non-empty ones get archived). But the damage? A full week of missing metrics.

'The script ran fine for three months. Then a contractor uploaded a file with a single character typo in the name, and the regex matched the whole archive folder.'

— Senior SRE recalling a Monday morning postmortem

Most teams skip this: auditing the shape of what the pattern matches, not just the name. Run the glob against a sample set first, then grep the results for edge cases. A one-line find . -type f -name 'temp_*' | head -20 beats staring at regex in a dark terminal. That said, even that won't save you from the real killer—a pattern that semantically looks right but drifts as the system grows. New team members add files that incidentally fit the pattern, and nobody updates the cleanup rules. Six months later, the script is deleting the archives it was never supposed to touch. That's why the first audit step is not the code—it's asking what the pattern actually matches right now, on disk, with today's data.

Foundations: What Readers Confuse About Cleanup Scripts

Difference between 'last modified' and 'last accessed'

Most teams treat these two timestamps as interchangeable. They're not. I have watched an ops engineer wipe six months of archival logs because his cleanup script targeted files with a last accessed date older than 90 days. The script ran at 3 AM — cron had never opened those logs for read, so atime sat frozen since creation. The files were accessed daily, just not by stat or cat. The database team found out the hard way, during a compliance audit. Wrong order: the script deleted by access age, not modification age. The catch is that most Linux filesystems now mount with relatime or noatime, meaning last accessed updates only sporadically or never. If your cleanup script uses atime without checking the mount options, you're not cleaning — you're gambling.

That sounds fine until a developer stores temporary build artifacts in /tmp but never modifies them again. The modification timestamp stays locked to the install date. The access timestamp? Also fixed, because the build tool reads each file exactly once. Thirty days later your script nukes the entire artifact cache, and your CI pipeline grinds to a halt for six hours. We fixed this by forcing every cleanup script to print both timestamps for a sample of ten files before it runs any delete — a simple --dry-run that logs mtime versus atime side by side. The difference is rarely zero, and that difference tells you which clock the script is actually reading.

Why recursive globbing can surprise you

**/*.tmp looks safe. It's not. Recursive globbing descends into every subdirectory — including hidden ones, symlinked directories, and mount points you forgot existed. I once saw a cleanup script glob /data/cache/**/*.old and follow a symlink that pointed to /var/lib/postgresql. The script deleted zero files there because the extension didn't match, but the traversal alone triggered an fsnotify storm that took down a production replica. The real pitfall: ** in Python's glob or Bash 4+ matches broken symlinks too, and a broken symlink to a deleted target still counts as a match. Your script tries to os.remove that path — it fails silently or raises an exception that halts the entire loop. Half the directory stays untouched; the other half gets purged. That asymmetry corrupts whatever invariant you were trying to enforce.

Quick reality check — most engineers test their globs on a clean staging directory with three subfolders. They never test against a filesystem with 200,000 files spread across 800 directories, where the first hundred directories are fine and the hundred-and-first hits an NFS mount point that lags for 12 seconds per readdir call. The script appears to freeze. Someone kills the process mid-glob; next month a stale pid file causes the cron job to skip entirely. Two months later, no cleanup has run at all, and the disk hits 97% usage. The real fix is not ** — it's an explicit whitelist of directory trees with depth limits and a find -maxdepth guard that stops traversal before it hurts.

Flag this for productivity: shortcuts cost a day.

The myth of safe deletion with -rf

rm -rf is a weapon, not a tool. Yet I still see tutorials that call it "safe" if you prefix it with a sanity check. It's not safe. Ever. The command has no undo, no journal, and no --warn-if-target-exceeds flag. One trailing space in a variable expansion — rm -rf $DIR / when $DIR is empty — and your server becomes a brick. We fixed this by aliasing rm to rm -I (interactive once per more than three files) or, better, by never calling rm directly in scripts at all. Instead we move files to a .trash directory with a timestamp, then schedule a separate process to purge items older than seven days. That buffer has saved three teams I know from catastrophic date-offset bugs where the script ran a week early and targeted /var instead of /var/log/old.

'The files you think you're deleting are never the files the script actually matches. Trust the output before you trust the command.'

— senior SRE, after a post-mortem on a cleanup script that deleted the backup repository instead of the temp directory

The deeper issue is psychological: -rf looks deterministic, so engineers stop double-checking. They echo the file list once, see twenty lines of .log files, and hit execute. They never re-run the echo after a filesystem change or a mount repoint. Next month the same script deletes 200 GB of customer uploads. Mitigation is boring but effective: force every cleanup to write the deletion plan to a dated JSON file before acting, then require a second script to confirm that plan against current disk state. That's two extra lines in a cron file and zero data loss since deployment.

Patterns That Usually Work: Audit Checklist

Verify date range with inclusive/exclusive boundaries

The most common blowup I see: a script that deletes through a date instead of before it. Quick reality check—does your condition use < or <=? That single character difference evaporates a month of customer orders. One team at a logistics shop I worked with lost three days of shipping records because their cron job used --older-than 30 which turned out to be inclusive of day 31. Wrong order. The fix: always write the boundary as an explicit stop—created_at < '2025-03-01' not <= '2025-02-28'. Then test with a known edge record one second before and one second after the cut. Most teams skip this: they check a random sample from the middle of the range, where nothing is ambiguous. The tricky bit is that database timestamps often round up or down depending on the driver—Postgres stores microsecond precision, MySQL may truncate to second, and your script's comparison logic rarely accounts for that drift.

What usually breaks first is the timezone assumption. Your server runs UTC, but the data arrived with a local timestamp baked in during ingestion. I have seen a cleanup script that ran at 01:00 UTC delete everything "older than 90 days"—which, in UTC, was still 89 days 23 hours for every record logged in a Pacific-time system. That's not a corner case; that's a regular Tuesday. So pin your comparison to a single known timezone, store the cutoff as an explicit UTC string, and add a one-line assertion that the boundary you pass is not null or zero. A dry-run will catch this, but only if you actually look at the row count and not just the green exit code.

Run dry-run and capture output for diff

A dry-run flag that prints "would delete N rows" is nearly useless. You need the identities of those rows. The pattern that works: write a dry-run mode that dumps primary keys into a file, then run diff against last month's dry-run output. A jump from 1,200 rows to 17,000 rows is your signal to stop. Not yet—read the diff. I use a three-line wrapper: dry-run-2025-03-01.txt, dry-run-2025-03-08.txt, diff the two. If the new set includes records with updated_at timestamps from last week, something upstream changed the data model without telling ops. The catch is that teams automate the dry-run but forget to review the diff. They set it and walk away. That hurts. So schedule a human step—a Slack notification with the row count change highlighted in bold, and a three-hour window to cancel before the real delete fires.

Test on a copy of the data directory

Unpopular opinion: your staging database is not a good test bed for cleanup scripts because it has different row counts, different date distributions, and zero production locking patterns. Clone production instead—or at least snapshot a recent replica and run the script against it. The pattern: pg_dump --schema-only for structure, then COPY a random 10% slice of the target table, skewed toward the deletion boundary. That catches index bloat, foreign-key cascades, and timeouts that only appear when the script tries to delete 50,000 rows through an unindexed column. I once saw a script finish in four seconds on staging and run for forty-seven minutes on production—because staging had 2,000 abandoned carts and production had 300,000. The diff between those run times is not a performance bug; it's a design failure.

Avoid the temptation to test on a Monday morning with production traffic. Take the weekend replica from Saturday 02:00, spin it up on a separate server, and run your audit checklist there. If the script deletes something it should not, you restore from the Saturday snapshot—no incident, no panic. One concrete edge case that surfaces only on production-like data: soft-deleted rows that still carry a deleted_at timestamp older than the cleanup threshold. Your script sees them as eligible, deletes them hard, and now you have orphaned history in related tables. That's the kind of surprise a copy reveals before you wake up to a Slack pile-on at 06:00 on a Tuesday.

“The first time I ran a cleanup script on a production clone, it tried to delete the admin user’s account because the admin had never logged in. The copy caught it. The real database would not have.”

— Engineer at a mid-market e-commerce platform, after their third incident

Anti-Patterns and Why Teams Revert to Manual Cleanup

Wildcards without quoting — the silent deletion magnet

You write rm -rf /tmp/$PROJECT/* and it works in staging. Then $PROJECT is empty in production. The shell expands /tmp/* — now you're wiping the entire /tmp partition. I have watched a senior engineer do this during a demo. One unquoted variable, and the script deleted three weeks of build artifacts. The fix is brutal but simple: quote everything, even when you think the variable is safe. "$PROJECT"/* is not pedantry — it's the difference between a targeted cleanup and a carpet bomb.

Most teams skip this: they test with a populated directory and never simulate the null case. The catch is that Unix shells treat empty variables as literal empty strings before the wildcard. * then matches everything. So the script that worked fine for six months suddenly eviscerates a shared filesystem on the first Monday after a colleague clears a config file. That hurts. And it erodes trust faster than any logic bug — because the damage looks intentional.

Honestly — most productivity posts skip this.

Hardcoded paths that change — the brittle foundation

You hardcode /opt/app/logs/ into the script. Six months later the ops team migrates the app to a containerized deployment. Logs now live under /var/data/app-2024/volumes/logs/. Nobody updates the script. The cleanup silently targets a directory that still exists but is now used for something else — a shared NFS mount for a different team's backups. That sound you hear is the seam blowing out.

Anti-pattern: using absolute paths without a single source of truth. The team that reverts to manual cleanup usually does so because they got burned by a path change that caused a cascade failure — the script deleted something important, but worse, it didn't delete the intended targets. So archives grew until disk filled up, alerts fired at 3 AM, and the on-call engineer swore off automation forever. Quick reality check — hardcoding is faster to write, but it costs roughly an hour of debugging per path per incident. I have seen teams tolerate four of those before declaring the script untrustworthy.

Skipping validation for edge cases like empty directories

A common script flows like this: list files older than 30 days, delete them, report success. What about empty directories that remain after the deletion? They don't consume much space, sure. But they break downstream processes that expect a certain file count — monitoring agents that tail missing logs, backup jobs that error on empty folders, report generators that crash on zero-row CSVs. One team I worked with had a script that cleaned temp files but left stale symlinks pointing to nothing. Those dangling links confused their archiver, which then skipped the entire volume because it hit a "Too many levels of symbolic links" error. The whole backup chain collapsed.

The fix is boring but effective: add a validation step that confirms the state after deletion matches expectations, not just the count of deleted items. Check for zero-byte files, dangling symlinks, and permission changes that crept in since last run. Without this, your script becomes a black box — and teams revert to manual cleanup because at least when they delete something by hand, they know what they deleted.

'The only thing worse than a script that deletes the wrong data is one that deletes the right data silently and leaves the mess behind.'

— overheard in a post-mortem after a 2 AM PagerDuty flood, ops team explaining exactly why they were rolling back to manual procedures despite the extra labor

Stop ignoring the empty case. Next month when your script runs and $PROJECT is unset, or the log directory moved, or the retention policy changed without a ticket — that's the moment trust breaks. And trust, once lost in ops automation, takes months of clean runs to rebuild. Teams don't revert to manual because they love clicking folders. They revert because the script burned them, and burning twice is one time too many.

Maintenance, Drift, and Long-Term Costs of Cleanup Scripts

When cron jobs drift due to filesystem changes

A cleanup script written in January assumes /var/log/app stays flat. By March, someone moves logs into date-stamped subdirectories. The script doesn't break — it runs successfully, deleting nothing because its glob pattern misses everything. That feels safe. Until the next rotation adds a symlink back to the old path and the script suddenly slurps an entire production directory. I’ve watched this exact pattern surprise a team mid-quarter. The cron job hadn’t failed; the filesystem shape had drifted under it. Filesystem changes are the silent killers — they rarely trigger errors, just wrong counts or silent skips that eventually compound into a deletion event.

Cost of restoring from backup after accidental deletion

One restore costs more than the script saved in a year. That’s not hyperbole — factor in the manual validation, the recreated configs, the engineer hours explaining to the CTO why Tuesday’s archive vanished. Most teams underestimate the blast radius. A cleanup script that deletes /tmp/cache for ten services looks contained until one service writes user uploads to that same temp directory under a race condition nobody documented. The backup restore assumes yesterday’s state works, but yesterday’s state might already corrupt because the script ran Wednesday too. Rebuilding from backup isn’t a button push; it’s a multi-hour investigation with pressure and half-awake on-call rotations. That hurts.

“Restoring from backup after a bad cleanup isn’t recovery — it’s triage with a seven-day delay.”

— DevOps engineer who spent a weekend rebuilding a log archive from raw tcpdumps

Ask yourself: does your backup RTO actually fit inside the outage window your users tolerate? Most don’t. Teams discover this when the restoration process itself reveals gaps — missing indexes, stale database snapshots, permission mismatches that the original script respected but the backup doesn’t mirror. That’s the real cost: not the storage space, but the confidence loss.

Updating scripts when log formats change

Log rotation software updates. JSON fields rename. Timestamps shift from UTC to local with no announcement. The cleanup script that parsed YYYY-MM-DD now silently skips records formatted as epoch_ms. Wrong order again — the script deletes everything older than 90 days, but the parsed date is zero, so everything qualifies. One format change undoes a year of careful thresholds. We fixed this once by adding a dry-run flag that emitted the final deletion list to a diff-able file. That caught the format shift before the cron job ate thirty days of audit logs. The catch: someone has to review that diff manually every rotation, and manual review gets skipped when the team ships features instead. Maintenance isn’t a one-time cost; it’s a subscription you pay in attention every time the schema changes upstream.

Field note: productivity plans crack at handoff.

When Not to Use This Approach

Databases that need point-in-time recovery

If your database lives under a PITR policy—say, you must be able to roll back to any minute within the last 72 hours—automated cleanup becomes a gamble dressed as efficiency. I have watched a script happily purge "stale" backup snapshots only to discover, three hours later, that the compliance team needed that exact 3:14 AM copy for an audit trail audit. The catch is obvious once you feel it: your script can't know tomorrow's subpoena. The DELETE runs clean, rows vanish, and the recovery window shrinks to whatever the transaction log still holds. That hurts. For databases governed by PITR, manual review isn't slower—it's the only safe floor. You keep human eyes on what gets truncated, and you tag each purge candidate with a reason that survives beyond the script's exit code.

Production hours with high I/O

Cleanup scripts that scan tables during peak traffic—that's where reputations get quietly demolished. A SELECT COUNT(*) that runs for six seconds on a 20 GB table might be harmless at 3 AM; at 2:47 PM on Cyber Monday it locks a critical index and sends latency through the roof. Quick reality check—I once saw a cleanup job hold a row-level lock for eleven minutes because the transaction log filled mid-delete. The site didn't crash. It just got slow enough that the on-call engineer pulled the plug on the script, leaving half the records orphaned. The trade-off is brutal: you can schedule around traffic, but if your environment can't guarantee a quiet window, manual triage wins. One batch delete, handled by a human who monitors the I/O counters live, beats an automated script that ignores the disk queue depth until it's too late. Not every process needs automation—some need a human holding the brake.

Environments with strict compliance requirements

“The script deleted 1,200 rows that matched an open litigation hold. We got the subpoena two hours later.”

— Engineer at a fintech startup, post-mortem notes

Compliance environments look like they should benefit from automation—repeatable steps, documented schedules, deterministic outcomes. Except compliance also demands that no deletion outruns a legal hold, and no automated pipeline can inspect a litigation hold spreadsheet while it scans for orphaned temp tables. The fragmentation is the problem: your cleanup script sees created_at < 90 days; the compliance officer sees a custodian who logged in 89 days ago and whose data must be preserved for another 18 months. Those overlaps kill automation. Best practice in regulated shops is to let the script generate a report (list of candidate rows, size estimates, age buckets) and then have a human approve the actual deletion. Slower? Yes. Auditable? Absolutely. The script still does the heavy lifting—it just never pulls the trigger without a signature.

Avoid the reflex to automate everything because you can. The scenarios above share a pattern: the cost of a single mistake exceeds the cumulative benefit of a hundred clean runs. When the data's provenance is fuzzy, the recovery window is tight, or a regulator might ask why row 4732 disappeared last Tuesday, lean toward manual review. Your future self—and your auditor—will thank you.

Open Questions and FAQ

Should you always run dry-run twice?

One-and-done dry-runs lull you into a false sense of safety. I've debugged scripts where dry-run #1 showed zero deletions because the target path was unmounted — it ran against an empty local folder, reported "nothing to clean," and looked perfect. The second dry-run, if you force a mount check first, catches that. Two runs with different entry points: first from the cron environment, then from an explicit interactive shell that mimics the real runtime context. The gap between those two environments — missing mounts, stale symlinks, permission masks that shift between user contexts — is where the second run earns its keep. One team I know now treats the second dry-run as a diff check: run it, pipe output to a file, run it again, diff the two. If they match, you have a stable snapshot. If they diverge, something is flapping. Flag it, don't deploy it.

How to handle files that are locked or in use?

Skip-and-log is the only sane default. Don't force-unlock, don't inject lsof kill logic into your cleanup script. Here's the trap: an application writes a temp file, the script sees a file older than 30 minutes, deletes it, and the application crashes mid-write because it held a descriptor open against a now-orphaned inode — the file is gone but the handle isn't released. That hurts.

Most teams skip this: add a pre-delete check for open file handles using fuser or lsof on the exact path. If the check returns a PID, skip that file and bubble a warning into your alert channel, not just the log. The catch is performance — scanning hundreds of thousands of files with lsof per entry kills wall-clock time. Workaround: invert the pattern. Have the application itself drop a *.lock or *.inprogress marker file before it starts writing, and teach your cleanup to skip anything with a companion marker less than N hours old. Cooperative design beats brute-force OS audits every time.

What's the best way to log what was deleted?

Structured JSON to a separate, append-only log file. Not the syslog. Not stdout interleaved with debug noise. A single file per script invocation with ISO 8601 timestamps, full absolute path, file size in bytes, and the deletion trigger rule that matched. Like this: {"ts":"2025-03-11T04:22:01Z","path":"/data/tmp/cache_9832.tmp","size":4096,"rule":"older_than_24h"}. Why? Because when someone asks "what did last night's run delete?" — and they will, at 3 AM — you don't want to grep through mixed output. You want a replayable audit trail that a non-operations person can load into a spreadsheet and reason about.

We logged to syslog for two years and never caught a drift until a single line wrapped across three buffer-broken rows during a filesystem error. JSON to a flat file would have saved us a post-mortem.

— Operations lead, mid-sized adtech firm

One more thing: pipe that JSON log into a cheap retention bucket — S3, GCS, whatever — and set a lifecycle policy to expire after 90 days. Then add a daily summary report that counts total files deleted, total bytes freed, and any skip/error counts. A summary tames the firehose and keeps stakeholders out of your raw logs. Without that summary, you will get paged for a 5 MB cleanup that ran flawlessly.