AE Find Weblinks

AE-Find-WebLinks.zip · 61.7 KB

Changelog

AE-Find-WebLinks 1.7.0 is a functional crawler release. The biggest change from 1.6.2 is that the script can now follow links beyond the initial page while still preserving the old default behaviour when crawling is not requested.

Added

• Added recursive link-following / crawl mode.
• Added -FollowDepth <n>:
• 0 = default behaviour, fetch only the supplied source URL(s).
• Positive values = follow links up to that many link-hop levels.
• -1 = crawl until the allowed frontier is exhausted.
• Added -FollowUntilExhausted.
• Added aliases:
• -FollowToEnd
• -UnlimitedFollowDepth
• Added -FollowScope:
• SameDomain
• SameHost
• Any
• Added -FollowPathScope:
• Any
• SeedPath
• Added -FollowSubdomains.
• Added -MaxSubdomainDepth.
• Added -MaxFollowPages.
• Added optional robots.txt enforcement with -EnforceRobotsTxt.
• Added -RobotsUserAgent.
• Added crawl queue/frontier handling with per-page depth tracking.

Improved

• File input handling now works better as crawl seed input.
• Multiple URLs can be extracted from a source-file line more reliably.
• Private/internal URL checks are applied to crawl seeds and crawl candidates.
• Blacklist checks can now prevent URLs from being crawled.
• Crawl candidates are followed by scope/depth rules, not only by search-pattern matches.
• Matching/output writing remains separate from crawl discovery.
• Help text was expanded with crawl examples.
• Interactive configuration now includes crawl-related options.
• CSV logging now records ROBOTS_BLOCKED and ROBOTS_UNAVAILABLE.

Behaviour Changes

• Default behaviour is unchanged: -FollowDepth 0 means no crawling.
• Unlimited crawling is now explicit: use -FollowDepth -1, -FollowUntilExhausted, or -FollowToEnd.
• -FollowPathScope SeedPath keeps crawling inside the starting URL path/subtree.
• SameHost is stricter than SameDomain.
• robots.txt is not enforced unless -EnforceRobotsTxt is supplied.

Current Limitations

• Crawl mode currently requires sequential execution with -ThrottleLimit 1.
• -Resume is not supported in crawl mode.
• -FollowPathScope is ignored when crawl mode is disabled.
• Exact Public Suffix List handling is not implemented; SameDomain uses an approximation.

Summary

Version 1.7.0 turns AE-Find-WebLinks from a single-page / URL-list extractor into a controlled crawler, adding depth-based crawling, same-domain/same-host boundaries, seed-path restriction, subdomain controls, crawl page caps, and optional robots.txt enforcement.

Download now

If the download does not start automatically after the countdown, click the button above.