AE Find WebLinks 1.7.4

AE-Find-WebLinks-v1.7.4.zip · 206.2 KB

Changelog

Added

• New option: -StuckUrlTimeoutSeconds <n|Disabled> (Aliases: -SameLinkTimeoutSeconds, -UrlWatchdogSeconds, -StuckLinkTimeoutSeconds)
• New option: -RetryFailedOnFinish (Aliases: -AutoRetryFailed, -RetryFailedAtEnd)
• New feature: Crawl mode runs are now resumable.
• Generates a crawl journal (.crawl.progress) to track the frontier of scheduled (ENQUEUED) and completed (DONE) URLs.
• Allows interrupted crawls to be safely resumed with -Resume.

Example use case: Automatically abort and restart processing for a single URL if it gets stuck for more than 5 minutes (-StuckUrlTimeoutSeconds 300), and run one automatic retry pass over any failures when the job ends (-RetryFailedOnFinish).

Updated

• Help text now documents the new stuck URL watchdog, automatic retry switch, and crawl resume behavior.
• Interactive mode now prompts for the stuck URL timeout duration and whether to enable the automatic retry on finish.
• Generated command/config output now includes the new watchdog and retry options.
• Resume/run signature now dynamically includes crawl boundaries (FollowDepth, FollowScope, FollowPathScope, etc.) when crawl mode is enabled.
• Default progress file extension adapts to <OutputFile>.crawl.progress for crawl mode instead of the standard <OutputFile>.progress.
• Internal sleep, delay, and HTTP functions now integrate with the stuck URL watchdog to consistently enforce the timeout bound.

Safety / Validation

• -StuckUrlTimeoutSeconds safely accepts friendly string values like 0, Disabled, Off, or None to bypass the watchdog, and throws clear errors for invalid formats.
• The stuck URL timeout enforces a maximum operational guardrail of 604,800 seconds (1 week) to prevent typo-induced crashes.
• -RetryFailedOnFinish is strictly limited to a single extra pass to prevent infinite recursive execution loops against permanently broken URLs.
• The automatic retry pass safely forces append modes behind the scenes (-Resume $true, -Mode Append, -LogMode Append, -FailedUrlMode Append) to prevent accidental data loss of the primary run.
• The run signature gracefully omits crawl-related variables for standard File runs, ensuring backward compatibility with older progress files.

Possible Gotcha

Old resume progress files from interrupted crawl mode runs (if you were manually tracking them before native support) will not match the new run signature, because the signature now correctly includes crawl-specific settings to ensure the resumed frontier is identical. However, standard non-crawl File mode progress files remain fully compatible.

Download now

If the download does not start automatically after the countdown, click the button above.