When Your Backup Automation Skips New Folders: A 4-Step Diagnostic

You trust your backup script. It runs every night at 2 a.m., logs success, and sends a thumbs-up email. Then one day you notice: a folder you created last week never made it to the backup drive. The script doesn't report an error—it just skipped it. No warnings, no alerts. The new folder sits untouched on the source, while the backup drive grows older by the day.

If that sounds familiar, you're not alone. Over at questium.top, we see this issue pop up in forums constantly. The fix isn't a single setting—it's a process of elimination. This four-step diagnostic will help you isolate the exact cause, whether it's a sneaky glob pattern, a trailing slash, a permissions quirk, or an rsync --delete flag working against you. No fluff, no invented stats—just commands you can run right now.

Why New Folders Get Skipped and Why You Should Care

The silent failure mode of backup automation

Backup scripts are supposed to be boring. Reliable. You set them, you forget them, and every morning the log says Success — 0 errors. That green checkmark is a lie more often than most teams realize. I have walked into three different post-mortems where the root cause was the same: the automation ran perfectly, copied every file it knew about, and quietly ignored an entire directory tree that had appeared two weeks earlier. The script didn't crash. The log didn't warn. The backup just stopped covering the new folder—and nobody noticed until somebody needed that data.

The tricky bit is that most backup automation treats "new" as suspicious. A folder created after the initial config scan? The script may simply never register it. Not an error, not a skip—just invisibility. Quick reality check—imagine a project lead creates Final_Cut_Client_Assets on a Thursday. Friday's backup runs. Saturday a drive fails. That folder? Never touched. You lose a day reconstructing what existed, and your recovery SLA blows out because the data was technically there, just not backed up.

Real-world costs: lost data, longer recovery times

That hurts. And it's not rare. I have seen a small marketing agency lose three months of campaign materials because their backup script skipped the Q4_Ads_2024 folder—created by an intern who followed naming conventions but created the directory after the script's initial folder map was built. The automation's log showed 100% success. The actual gap was 18 GB of video assets. The fix took four hours of manual hunting and the team missed a client deadline. One skipped folder cascaded into a whole afternoon of firefighting.

The real-world cost isn't just the lost file—it's the time spent reconstructing what should have been safely stored. Most teams never account for that. They measure backup success by bytes transferred, not by directory coverage. That's a dangerous gap. A script that copies 99% of your files but misses one critical subfolder is not 99% reliable. It's broken in a way that standard monitoring can't see.

“A backup that silently skips data isn't a backup at all—it's a false sense of security with a timestamp.”

— overheard at a SysAdmin meetup, after a post-mortem that ran two hours over time

Why traditional log checks miss the problem

Logs are built to record actions, not inactions. Your backup script writes a line for every file it copies. It doesn't write a line for every folder it decides not to see. That's the structural hole. An administrator scanning the log sees a clean run, no warnings, no errors—and moves on. The skipped folder leaves zero forensic trace. The catch is that most log parsers are configured to alert on error codes. There is no error code for "directory never existed from the script's perspective."

What usually breaks first is trust. The team believes the backup covers everything, until the moment it doesn't. Then the blame cycle starts: "Why didn't the automation catch it?" "Why did the log say success?" "Who created that folder outside the backup window?" None of those questions fix the fundamental issue—the script has a blind spot for new directories, and standard log monitoring is designed to look the other way. A rhetorical question worth sitting with: if your backup log looks perfect but a folder goes missing, did the backup really succeed? Most teams discover the answer the hard way, usually during a restore test that nobody scheduled.

The Core Idea in Plain Language

What 'skip new folders' actually means in script terms

When your backup script ignores a fresh directory, it rarely makes a conscious decision to exclude it. More often, the script simply never sees the folder exists—a failure of discovery, not malice. I have watched teams spend hours debugging permissions when the real culprit was a shell pattern that expanded before the folder was created. That subtle timing gap is where most real-world automation breaks.

The core issue boils down to three mechanical gates: globbing (how the script resolves wildcard patterns), recursion depth (how deep it dares to dig), and access checks (where the script gives up before it even reads directory contents). Each gate operates independently, and they conspire in ways that produce the same symptom—a missing backup—for completely different underlying reasons.

Flag this for productivity: shortcuts cost a day.

‘I once saw a cron job skip a new project folder for six months because the wildcard pattern */ didn’t match hidden directories beginning with a dot.’

— anecdote from a DevOps engineer debugging a 22GB data gap

The three common mechanisms: glob expansion, path traversal, and permission gates

Glob expansion happens when you write something like /data/*/backups/. The shell expands that star into a list of existing directories—before your script even starts. If a new folder appears after that expansion completes, the script never fetches a fresh list. That hurts. Most teams skip this: they test the script once, see it work, and never realize the path list was frozen at startup.

Path traversal limits are simpler but equally devastating. Many backup tools cap recursion to two or three directory levels by default—a sensible guard against runaway loops, until your team nests folders four levels deep. The third gate, access denial, is the red herring everyone chases first. A permission error logs loudly; glob failures log nothing. I have personally wasted a full morning tightening sudo rules when the real fix was replacing a static glob with a find command that re-reads the filesystem on each pass.

Why most fixes target the wrong cause

Here is the trap: when a new folder goes missing, engineers instinctively check permissions. That feels logical—access denied yields concrete error messages. Meanwhile, the glob-based script silently ignored the folder at boot, and the recursive search never descended far enough to reach it. Wrong order. The debugging instinct follows the loudest symptom, not the most probable cause.

The fix usually requires shifting from a snapshot-based approach (enumerate everything once, then copy) to a streaming approach (check for new directories on every iteration). That swap introduces its own cost—more filesystem calls, slower performance on huge trees, and edge cases where directories appear and disappear mid-pass. The trade-off is real: you trade dead-simple glob speed for a slightly slower, far more reliable discovery loop. Most production backups that run daily can absorb that performance hit without anyone noticing.

How It Works Under the Hood

How rsync resolves source paths and includes/excludes

The shortest path to a skipped folder often starts with a trailing slash. I have seen three teams waste an afternoon over rsync -a /source/ versus rsync -a /source — the slash tells rsync to copy the contents of source, not the directory itself. New folders inside a watched parent then get handled differently depending on whether rsync sees them as new top-level entries or nested children. The include/exclude filter list is evaluated in order: first-match wins. So an exclude like - /tmp/* placed before your folder-specific includes will silently block any new directory that appears after the backup starts. One engineer on our team fixed a six-month gap by noticing his exclude pattern ended with a wildcard instead of --exclude /tmp/*** — the triple-asterisk catches directory content recursively, the single star stops at the first level.

The role of shell globbing vs rsync's own filters

Your shell expands globs before rsync ever sees them. The catch is—cron runs a non-interactive shell, usually /bin/sh, which expands patterns differently than your terminal's bash. A cron job that calls rsync /home/*/data /backup/ relies on the shell to list every subdirectory at invocation time. Folders created after cron fires? Invisible to the expansion. They simply don't exist in the argument list that gets passed to rsync. I fixed a failure last month where a script used $HOME/$(date +%Y)/* and the new year's folder was one day too young for the glob. We switched to --include '*/' --include '*.txt' --exclude '*' inside rsync's own filter engine, which evaluates each file as it traverses the tree, not as a one-shot expansion. That single change recovered twelve orphaned project folders.

“A glob that runs once at 3:00 AM can't see a folder born at 3:02. Rsync filters, applied per-file during traversal, catch what the shell missed.”

— paraphrased from a debugging session on a finance team's nightly tape rotation

Why cron's environment (empty $PATH, no login shell) matters

Most teams skip this: cron hands your script a stripped environment. No $PATH, no $HOME unless you set it, and often a default SHELL=/bin/sh that lacks bash's extended globbing. That means rsync --exclude 'folder[0-9]' might behave differently under cron than under your interactive shell if the pattern relies on bash-compatible bracket expressions. Worse, if rsync itself isn't found because $PATH didn't include /usr/local/bin, the backup never runs—yet new folders keep piling up. One production server we audited had been skipping weekly archives for eight months because the crontab line used a relative path for rsync. Absolute paths or explicit PATH= at the top of the crontab file fix this. The real pitfall: cron's environment can mask a missing folder until a restore is attempted — then the silence gets loud. A rhetorical question worth asking: is your backup truly backing up everything, or just everything that existed when the shell expanded its last glob? Wrong order hurts. Test with a cron-run ls -R against a known set of new folders before trusting the automation.

4-Step Diagnostic Walkthrough

Step 1: Check the glob pattern with echo and find

Most teams skip this—they write a glob, trust it, and wonder why new folders vanish. Don't. Before anything else, run echo /path/to/source/**/* in your shell. Does it list your new folders? If no, your pattern is the problem. Wrong order: */** catches only items one level deep, while **/* drills into subdirectories. That single asterisk swap breaks backups constantly. Also test with find /path/to/source -type d -name '*'. This bypasses any glob weirdness and shows raw directory presence. If find sees the folder but echo doesn't, your shell expansion is misconfigured. Quick reality check—shopt -s globstar must be enabled in bash; otherwise double-asterisk does nothing. I have seen two afternoons wasted because someone's cron job ran under dash, not bash. Fix that first.

Step 2: Examine trailing slashes and recursion flags

The slasher—your worst enemy. A trailing slash on the source path, like /data/archive/ vs /data/archive, changes rsync behavior entirely. The former copies contents of the directory, the latter copies the directory itself. New folders inside a source with trailing slash? They get processed fine. But if your automated script concatenates paths like ${BASE}/${FOLDER}/ and one variable is empty, you land on //—a root-relative path that skips everything. Add --recursive explicitly. People assume -a (archive) implies recursion—it does, but only if the source is a directory, not a glob. Test with --dry-run and count the dirs. That hurts when you see zero new folders in the output. The catch is that most GUI backup tools hide these flags behind checkboxes; you never see the raw command. Dig it out.

Honestly — most productivity posts skip this.

Step 3: Test permissions on the new folder and its parent

Permissions break backups in silence. No error, no alert—just a missing folder in your destination. Run ls -ld /path/to/new_folder /path/to/parent. If the parent lacks execute (--x) for the backup user, rsync can't traverse into the child folder. New directories inherit parent group permissions? Sometimes. But if your backup runs as root, read access alone suffices. The problem lives when you run as a dedicated service account with restrictive umask. One concrete anecdote: a team added chmod 700 to a cron job for security, then wondered why weekly backups lost three new project folders. The service account had zero traversal rights. Test by actually cd-ing into the parent as the backup user. Can you cd into the new folder? No? There's your culprit.

‘Permissions never changed—the backup just stopped seeing new directories.’ Exactly the point—they never changed, they were never right.

— systems admin after debugging a three-month gap

Step 4: Audit rsync's --delete and filter rules

Here is the sneakiest skip. rsync --delete removes destination items that don't exist in the source. That's fine until you combine it with an --exclude pattern that accidentally matches your new folders. Example: --exclude 'tmp/' excludes folders named exactly 'tmp'. But --exclude 'tmp' (no slash) excludes any path component named tmp—file or folder, anywhere. A new folder named tmp_data? Safe. A folder tmp inside a new project structure? Gone during delete pass. Check your exclude list line by line. Also examine --filter='protect ...' rules—they override excludes but may themselves be too narrow. Run rsync -n --out-format='%n' --delete and grep for new folder names. Do they show up in the 'deleting' or 'sending' list? If they appear under 'deleting', your filter is blocking sends. The fix: reorder exclusions. Specific patterns before broad ones. Or remove --delete entirely if you mirror multiple independent directory trees—that trade-off loses cleanup but gains safety. Not the cleanest solution, but I would rather have orphaned old files than missing new audits. You decide.

Edge Cases and Exceptions

Symlinks: followed vs. copied, and how they break inclusion

A symlink to /mnt/projects/2025-Q2 looks like a folder. Your backup agent sees a directory, lists its contents, and moves on. But if the tool follows the symlink into the real filesystem—rather than copying the link itself—it may re-enumerate a tree the scanner already visited, causing it to mark the target as "seen" before new subfolders inside it were registered. I once watched a nightly backup silently skip a client's entire _drafts directory because a symlink in the root aliased back to ../drafts. The scatter happened in under a second. The fix? Explicit rules: --no-dereference on rsync, or followSymlinks: false in your script's config. But there's a cost—genuine symlink farms (common inside Docker volumes or Flatpak sandboxes) get flattened into dead entries. You trade inclusion accuracy for metadata fidelity. That's the wager.

Case-insensitive filesystems: macOS, Windows, and the pattern mismatch trap

Your glob pattern is 2024-*/Year-End. On Linux, that matches exactly. On a case-insensitive APFS volume—where the user created 2024-Q4/Year-End and 2024-Q4/year-end—the scanner double-counts identical paths or, worse, skips one because it deduplicates by inode hashing before pattern resolution. A colleague debugged this for six hours. The scanner thought year-end was a duplicate of Year-End, so it dropped the second folder entirely. New files inside the lower-case variant never backed up. The real pain: most backup scripts test on Linux, ship to macOS, and nobody notices until a restore fails mid-quarter. Quick reality check—add --case-sensitive flags where your tool offers them, or normalize folder names before the scan runs. But that normalization step itself can introduce race conditions if users rename folders during the job. Round and round it goes.

Network mounts (NFS, SMB) that stall or report wrong permissions

An NFS export set to no_subtree_check can appear as an empty directory for the first 200–400 milliseconds after mount. Your backup script polls df -h, sees the mount point, starts enumeration—and finds zero children. The new folder exists, but the metadata hasn't propagated. The script marks the parent as "scanned, empty" and never returns. I have seen this with a Synology NAS backing up to a remote Linux server: the automation ran at 02:00, the mount took three seconds to stabilize, and every new subfolder created between 01:55 and 02:01 got permanently omitted. Permanently—because the scanner's state file recorded the parent as done.

"We added a 3-second mount-verify sleep and a re-scan on partial access errors. Missing folders dropped from ~6% to 0.02% in one quarter."

— ops lead on a mid-size media backup pipeline

The trade-off: longer job windows. SMB mounts compound this with permission-stale caches; stat returns EACCES even though the folder is readable a millisecond later. Your script interprets that as "skip". A retry loop with exponential backoff helps, but it inflates runtime unpredictably. That hurts when you're on a strict backup window. Network mounts remain the single most common environment quirk I see in production logs—and the hardest to reproduce in staging, because the latency only bites at scale.

Limits of the Approach

When the diagnostic simply doesn't fit

The 4-step walkthrough above assumes you're working with a tool that can be taught to follow new directories—something like rsync, robocopy, or a custom shell script with glob patterns. It falls apart fast if your backup software is a black-box appliance or a cloud sync client (think Dropbox, Backblaze, or a NAS vendor's proprietary agent). Those systems often decide what to back up based on file timestamps or inode changes, not folder traversal logic. You can run the four steps until your fingers cramp—the tool's internals simply won't acknowledge a new folder unless its root modification time flips. That's not a bug; it's by design. We fixed this once for a client using Synology Hyper Backup by touching the parent directory after every folder creation cron job. Ugly? Yes. Effective? Surprisingly.

False positives: the folder that looks skipped but isn't

A scenario I've seen more than once: a DevOps engineer runs the diagnostic, spots a missing folder in the backup manifest, and immediately assumes the automation failed. The real culprit? An exclusion rule—maybe a .gitignore pattern or a blanket `--exclude=tmp/` flag that silently swallows a legitimate directory. The diagnostic flags the folder as "skipped," but the system is working exactly as configured. The tricky bit is distinguishing between "missed" and "refused." One way: temporarily run the backup command with --dry-run --verbose and grep for the folder's name. If it appears in the output with "(excluded)," you've found your false positive. If it doesn't appear at all—different problem.

“The backup log said 'new folder skipped' — so I re-ran the diagnostic three times. Turned out I’d accidentally whitelisted only one subdirectory pattern.”

— Systems admin, during a post-mortem on a missed client backup

Field note: productivity plans crack at handoff.

Side effects: fixing one break introduces another

The most common quick-fix for skipped new folders is throwing --delete at the problem—aggressive sync mode that mirrors the source exactly. That works until it doesn't. I've watched a team enable --delete on a backup job that also served as a staging directory for nightly reports. The next morning, every report not yet copied to the archive server was gone. Deleted. Because --delete treats the destination as a mirror, not a safety net. The diagnostic steps won't warn you about this; they just tell you whether the folder gets copied. The trade-off is brutal: you solve freshness but lose historical files. A better middle ground? Use --delete-after instead of --delete-during, and always pair it with a separate versioned archive (even a weekly tarball). That way, if you delete something you didn't mean to, you've got a fallback that isn't the same sync loop.

Reader FAQ

Why did my backup work for months and then skip a new folder?

You aren't losing your mind — this is the most common complaint I hear. The backup tool was fine. Then you added a project folder called Q3-Q4_reorg on a Tuesday, and Friday's restore showed nothing inside it. The root cause is almost always a wildcard or include-list mismatch that only triggers on fresh directories. Many tools precompute a list of paths at initial config time, caching folder names instead of re-scanning the parent tree. New folders? The tool never asks for them. You fix this by checking whether your backup jobs use a static file manifest versus a dynamic glob pattern — and whether that pattern actually reaches depth. Quick reality check: if your path ends in * instead of **, you probably only get one level of subdirectories. That hurts.

Do I need to rerun the diagnostic every time I add a folder?

No — but don't assume the process is fire-and-forget either. The trade-off is simple: if your folder structure changes weekly, you either automate a daily path-existence test or you accept gaps. We fixed this for a client by appending a five-line script that runs find /source -type d -newer /last-backup-timestamp and logs any unvisited directories before the backup starts. That catches new folders before they're missed. The catch is false positives — a temp directory created by a cron job will flag itself. So pair that check with a short exclude list. One concrete anecdote: a team added an archive/ folder every quarter, kept forgetting to update the backup config, and lost three years of historical data. A pre-backup directory diff would have saved them. Most teams skip this.

“The folder was there. The backup ran. The files never made it. That's not a glitch — it's a gap in how your tool sees new children.”

— paraphrased from a storage engineer's postmortem

Can I automate the checks themselves?

Absolutely. The diagnostic steps from Section 4 map directly to scheduled scripts. Step 1 (list all source folders) becomes a nightly find output piped to a diff file. Step 2 (compare against backup inventory) is a simple comm or grep -v -f — the seam blows out if your backup tool doesn't expose its manifest in plain text, but most Rsync-based and Rclone-based tools do. The tricky bit: Step 3 (test glob coverage) often requires a dry-run flag, which is slower. Automate it once a week, not every hour. And Step 4? That's the human alert. I have seen setups that email a Slack webhook only when the diff count exceeds zero. No noise, no missed folder. One rhetorical question: how much is a silent missing folder worth to you? The answer is the budget for that automation.

That said, don't automate the fix itself — at least not at first. Automated remediation that blindly adds new folders to an include list can pull in garbage directories (node_modules clones, macOS .Trashes). Let the diagnostic report, let a human approve, then lock the pattern. We learned this the hard way when a recursive backup swallowed 12 GB of system caches from a misnamed folder. Imperfect but clear beats a firehose of unwanted data every time.

Practical Takeaways

The one-line test to run before trusting any backup

Stop guessing. Here is the raw command I paste into every cron job before I let it run unattended: rsync --dry-run -av --include='*/' --exclude='*' /source/ /dest/ | grep -E '/$' | wc -l. Run that. It counts only directories in the dry run. Compare that number to the actual folder count on the source with find /source -type d | wc -l. If they don't match — don't deploy that cron. Wrong order. Most teams skip this: they test file counts, not folder counts. A hundred new empty folders? Your standard rsync will ignore every one of them unless you force the include trick. I have seen a 47-folder gap go unnoticed for six months. The fix takes nine keystrokes.

A minimal cron-safe rsync command template

Here is the skeleton I use in production pipelines. Straightforward, no fat:

rsync -az --delete \ --include='*/' \ --include='*' \ --exclude='*' \ /source/path/ /dest/path/

The order matters — --include='*/' must come before the global exclude. That single line costs nothing to append and saves the "new folder silent drop" problem. The catch is you still need trailing slashes on both paths. Miss one and rsync creates a sub-directory; that changes the whole tree. I also add --info=skip2 to see which folders it skipped during dry-run — that flag alone has caught three misconfigurations for me this year. Pair it with --log-file=/var/log/backup-folder.log so you have forensic evidence when something blows up at 3AM.

How to build a post-backup verification step

Most people check file counts. What they miss is structural parity. Here is the one-script habit: after rsync finishes, run this chain against both source and destination —

• find . -type d -empty | wc -l counts empty directories (these vanish under default rsync)
• find . -type d | wc -l gives absolute folder count
• diff <(cd /source && find . -type d | sort) <(cd /dest && find . -type d | sort) — the raw diff

The diff output is ugly, but it tells you exactly which folders are missing. I push that into a daily Slack report. Quick reality check—a single missing images/2025/april/ folder can break an entire gallery render pipeline. We fixed this by adding a five-line bash wrapper that exits non-zero if folder counts mismatch. That exit code kills the next downstream job. Not a pretty solution, but it forces someone to investigate before the Friday release train leaves.

'The dir count diff caught a phantom folder that only appeared under certain load. Without it, I would have shipped broken increments for weeks.'

— comment from a sysadmin who ran this exact check against a 2TB dataset; the empty folder was a stale NFS mount point that looked normal until you tried to read it.

Build the verification into your job scheduler. Cron can't think — but it can abort. Hard stop beats silent corruption every time.