Improvement: Regex support for crawler #290
Description
The current version of the crawler extracts all paths under the given URL. We can allow the user to add regular expressions to filter the crawl paths. This would also improve crawling speed by avoiding unnecessary pages. This is discussed in the Discord thread.