AE Find Weblinks

AE-Find-WebLinks.zip · 26.7 KB

Release Notes

Summary

Find-WebLinks 1.5 is a major capability update over 1.1. The tool keeps the original URL/file scraping workflow, but adds stronger filtering, resumable runs, parallel processing, maintenance commands, better logging controls, and safer operational limits for large jobs.

New and Improved

• Added support for multiple search patterns with -SearchPatterns.
• Added Any / All matching through -SearchMode.
• Added exclusion filtering with -ExcludePattern, -ExcludePatterns, and -ExcludeMode.
• Added resumable file-mode runs with -Resume and -ProgressFile.
• Added parallel URL processing with -ThrottleLimit for PowerShell 7+.
• Added blacklist scope control with -BlacklistScope Input|Output|Both.
• Added URL fragment control with -KeepFragments.
• Added custom -UserAgent and -Proxy support.
• Added standalone file maintenance commands:
• -Command Deduplicate
• -Command Sort
• -Command Maintain
• Added maintenance timing controls with -DeduplicateWhen and -SortWhen.
• Added output sorting options with -SortDirection.
• Added large-file protection for maintenance operations, with override options.
• Added configurable operational limits for page size, redirects, regex timeout, URL length, retry-after handling, file write retries, file move retries, and HTTP connection limits.
• Added high-failure-rate warning support with -HighFailureRatePercent.
• Added -AllowExtremeOperationalValues for intentional high-value overrides while keeping typo guardrails by default.

Reliability Improvements

• Better handling of interrupted runs.
• Safer append/write behaviour for output, logs, failed URL files, and progress files.
• Better protection against very large pages or files causing memory problems.
• More robust retry and redirect handling.
• Improved deduplication and sorting workflows for large URL lists.

Compatibility

Existing 1.1-style commands continue to work. The original positional syntax remains supported:

.\Find-WebLinks.ps1 <Source> <SearchPattern> <OutputFile> [Append|New] [Url|File]

Download now

If the download does not start automatically after the countdown, click the button above.