fix: ignore whitespace-only Disallow paths in extractUrlsFromRobotsTxt

[juliosuas]

Summary

Fixes #701. Resolves the TODO in the docstring.

Problem

The regex r'disallow:\s*(.[^ #]*)' used . as the first character of the capture group, which matches any character including a space. This caused Disallow: (whitespace-only path) to be returned as ' ', adding an invalid disallowed path to the crawl-exclusion list.

# Before
extractUrlsFromRobotsTxt('Disallow: ')   # returns [' '] ← wrong
extractUrlsFromRobotsTxt('Disallow:  ')  # returns [' '] ← wrong

Per the robots.txt spec, Disallow: with no path (or only whitespace) means allow all and should produce no exclusion entries.

Fix

Replace the leading . with \S so only paths that begin with a non-whitespace character are captured:

# After
m = re.match(r'disallow:\s*(\S[^ #]*)', line, re.IGNORECASE)
extractUrlsFromRobotsTxt('Disallow: ')          # returns []  ✓
extractUrlsFromRobotsTxt('Disallow: /admin')    # returns ['/admin']  ✓
extractUrlsFromRobotsTxt('Disallow: /p#comment') # returns ['/p']  ✓

[juliosuas]

A bit more detail: per the robots.txt spec, Disallow: with no path (or only whitespace) means "allow all" — it should not add any entry to the disallowed list. The old regex r'disallow:\s*(.[^ #]*)' used . which matches any character including space, so Disallow: (space-only) was returned as ' ' — an invalid path. The \S fix ensures only paths starting with a real non-whitespace character are captured. Also removed the TODO comment from the docstring since it's now addressed.