Crawling Subpages

[maxin9966]

Trying to figure out how to get an LLM to pick the right subpages to scrape from a site. For instance, I want to grab the top news articles in a specific category, plus all the data from their individual pages.

[dosubot]

You can use the DepthSearchGraph in Scrapegraph-ai to crawl subpages and extract data from them in a single workflow. Set the depth parameter to control how many levels of links to follow (for example, depth=2 to get both the category page and its articles), and use only_inside_links to restrict crawling to the same domain if needed. Tailor your prompt to specify that you want the top news articles in a category and details from their pages.

Here's a practical example:

from scrapegraphai.graphs import DepthSearchGraph

graph_config = {
    "llm": {
        "model": "your-llm-provider/model-name",
        "api_key": "your-api-key",
    },
    "verbose": True,
    "headless": False,
    "depth": 2,  # Adjust as needed
    "only_inside_links": True,  # Set to False if you want to follow external links
}

search_graph = DepthSearchGraph(
    prompt="List the top news articles in the 'Technology' category and extract all data from their individual pages.",
    source="https://example.com/news/technology",
    config=graph_config
)

result = search_graph.run()
print(result)

This will crawl the category page, follow links to articles up to the specified depth, and use the LLM to extract the requested information from each page. For more advanced filtering (like picking only certain articles), you may need to refine your prompt or post-process the results. See more examples in the official docs.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[maxin9966]

You can use the DepthSearchGraph in Scrapegraph-ai to crawl subpages and extract data from them in a single workflow. Set the depth parameter to control how many levels of links to follow (for example, depth=2 to get both the category page and its articles), and use only_inside_links to restrict crawling to the same domain if needed. Tailor your prompt to specify that you want the top news articles in a category and details from their pages.

Here's a practical example:

from scrapegraphai.graphs import DepthSearchGraph

graph_config = {
"llm": {
"model": "your-llm-provider/model-name",
"api_key": "your-api-key",
},
"verbose": True,
"headless": False,
"depth": 2, # Adjust as needed
"only_inside_links": True, # Set to False if you want to follow external links
}

search_graph = DepthSearchGraph(
prompt="List the top news articles in the 'Technology' category and extract all data from their individual pages.",
source="https://example.com/news/technology",
config=graph_config
)

result = search_graph.run()
print(result)
This will crawl the category page, follow links to articles up to the specified depth, and use the LLM to extract the requested information from each page. For more advanced filtering (like picking only certain articles), you may need to refine your prompt or post-process the results. See more examples in the official docs.

To reply, just mention @dosu.

How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other    

@dosu

Why is the graph fetching unrelated URLs instead of following instructions?

I ran this code:

search_graph = DepthSearchGraph(
    prompt="Summarize this repository and output in markdown format. Then review the repository's Issues and summarize the most popular Issues.",
    source="https://github.com/ScrapeGraphAI/Scrapegraph-ai",
    config=graph_config
)

result = search_graph.run()

But instead of focusing on the repository and its issues, it fetched many unrelated URLs.

Why is it not following my prompt instructions and instead crawling random pages? This doesn't seem like the correct behavior for summarizing a repository and its issues.

logs:

Max input tokens for model openai/gpt not found,
please specify the model_tokens parameter in the llm section of the graph configuration.
Using default token size: 8192
--- Executing FetchLevelK Node ---
--- (Fetching HTML from: https://github.com/ScrapeGraphAI/Scrapegraph-ai) ---
Extracted 300 links.
--- (Fetching HTML from: https://github.com/ScrapeGraphAI/Scrapegraph-ai#start-of-content) ---
Extracted 300 links.
--- (Fetching HTML from: https://github.com/) ---
Extracted 144 links.
--- (Fetching HTML from: https://github.com/login?return_to=https%3A%2F%2Fgithub.com%2FScrapeGraphAI%2FScrapegraph-ai) ---
Extracted 11 links.
--- (Fetching HTML from: https://github.com/solutions/industry/nonprofits) ---
Extracted 124 links.
--- (Fetching HTML from: https://github.com/solutions/use-case/devsecops) ---
Extracted 127 links.
--- (Fetching HTML from: https://github.com/solutions/use-case/devops) ---
Extracted 128 links.
--- (Fetching HTML from: https://github.com/solutions/use-case/ci-cd) ---

[dosubot]

DepthSearchGraph crawls links based only on depth and domain restriction—it does not filter by URL subpaths or patterns. The prompt you provide is used for extracting information from the fetched pages, but it does not influence which links are followed during crawling. This means that on a site like GitHub, the crawler will follow any internal link it finds, including navigation, login, and unrelated sections, which is why you see so many irrelevant URLs being fetched. This is expected behavior and not a bug—it's a current design limitation of DepthSearchGraph docs.

If you want to restrict crawling to specific subpaths (like /issues), you’ll need to build a custom graph pipeline that uses the SearchLinkNode with a custom filter_config to filter links by keywords or patterns. This node isn’t part of DepthSearchGraph by default, but it’s designed for advanced link filtering code. Community discussions confirm this is a known limitation, and users are advised to implement their own filters or use an external crawler (like Scrapy) to pre-filter URLs before passing them to Scrapegraph-ai for extraction discussion.

A practical workaround: use SmartScraperGraph (with ignore_links=False in Html2TextTransformer) to extract a list of relevant URLs, then pass those URLs to a multi-page scraper. This is a manual, two-step process but gives you more control over which pages are scraped discussion.

Let me know if you want a code example for a custom graph or more details on configuring link filters!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[maxin9966]

@dosu Are there plans to develop an agent that can intelligently filter URLs based on natural language instructions? This would be a highly versatile feature for many scraping scenarios. Thank you for your hard work and the incredible open-source contribution.

[dosubot]

There aren’t any documented plans or released features for an agent that can filter URLs based on natural language instructions in Scrapegraph-ai yet. Current modules like DepthSearchGraph and GraphBuilder use your prompt for extraction and graph construction, but not for controlling which links are crawled—link selection is still rule-based, not LLM-driven docs.

This is a known limitation and has come up in community discussions. Advanced, prompt-driven link filtering would require integrating LLM reasoning into the crawling process, which isn’t available out of the box. For now, users build custom graphs with manual filters or use external crawlers for pre-filtering.

Your suggestion aligns with what many users want, and it’s a natural direction for future development. If you’re interested in experimenting, you could combine SmartScraperGraph for link extraction with your own filtering logic, but it’s not a single-agent solution yet.

Thanks for the feedback—ideas like this help shape the roadmap!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}