Here is a complete, standalone Python script designed to handle large-scale crawls (e.g., tens or hundreds of thousands of URLs from a Screaming Frog export).
It automatically detects the URL column, categorizes links by their top-level directory, paginates categories exceeding your target threshold (default: 1,000 links per page), generates an index hub (sitemap_index.html), and converts URL slugs into readable anchor text.
Paid subscribers can download it below
