Recursion not behaving as expected

[jkanel]

Could you please advise? I'm having trouble with recursive behavior, or perhaps I am misunderstanding the functionality.

DESIRED: I'm specifying a rootUrl and would like the scraper to recurse through all hyperlinks. The rootUrl will not be downloaded in this scenario. When the scraper finds a hyperlink ending in .abc it should download the file.

OBSERVED: The rootUrl (see code below) does not meet the urlFilter criteria and the scraper stops with no recursion. The scraper should find a hyperlink to http://trillian.mit.edu/~jc/music/book/SCD/Book45.abc in the rootUrl among other .abc urls, but it does not. Note that when I set the rootUrl equal to an .abcurl, e.g. the example above, the file downloads as expected.

const rootUrl = "http://trillian.mit.edu/~jc/music/book/SCD"

scrape({
  urls: rootUrl,
  recursive: true,
  maxRecursiveDepth: 5,
  urlFilter: function(url) {
    let match = url.match(/\.abc$/);
    return (match && match[0]);
  },
  directory: savePath
});

Many thanks for your consideration!
Jeff

[jkanel]

Seems like this could be a bug, however I found a workaround!

WORKAROUND: Add exception logic to allow the rootUrl to be downloaded. After that change, the remainder of DESIRED functionality has been achieved.

[s0ph1e]

Hi @jkanel 👋

Thank you for reporting the issue and sorry for the late response

As I understand you want to download all resources on rootUrl which end with .abc but since rootUrl does not end with .abc - it's filtered out and not downloaded, correct?
And you suggest that urlFilter should not be applied to root urls? I think it makes sense.

Feel free to submit a PR with a fix or I can do it a bit later

[jkanel]

Hi Sophia - thank you for your reply. You correctly understand the issue and proposed fix, i.e. urlFilter should not be applied to root urls. I won't have time to work on a PR and will leave it with you. Happily, I was able to work around the issue.

I created an issue for this topic.

This is a wonderful library. Thanks for building it!

[s0ph1e]

Issue was fixed in version 5.0.0