-
Notifications
You must be signed in to change notification settings - Fork 29
Description
Hi i am using the crawl api endpoint to crawl an webpage and also added the webhook parameter in body so that i can get the crawled response, however, the webhook only fire the started event and then no other event fires. is it expected?
const firecrawlOptions = { method: 'POST', headers: { Authorization:
Bearer ${env.FIRECRAWL_API_KEY}, 'Content-Type': 'application/json' }, body: JSON.stringify({ url: base_url, excludePaths: websiteExcludePaths, maxDepth: 5, limit: 100, allowBackwardLinks: true, allowExternalLinks: true, delay: 1000, scrapeOptions: { formats: ["json"], onlyMainContent: true, includeTags: ["article", "main", "section"], excludeTags: ["header", "footer", "nav", "aside"], timeout: 15000, jsonOptions: { schema: schema, systemPrompt: contentPrompt, prompt: contentPrompt } }, webhook: { url:
${env.NEXT_PUBLIC_APP_URL}/api/crawl-webhook, metadata: { jobId, university_id }, events: ["started", "page", "completed", "failed"] } }) }; const response = await fetch(
${fireCrawlApiUrl}/crawl`, firecrawlOptions);
Response:
{
body: {
success: true,
type: 'crawl.started',
id: '66f557bd-ac40-4d86-aac1-6b0af0cfdb4a',
data: [],
metadata: {
jobId: 'd5aac043-14a2-4e45-9e5c-88a2292a8563',
university_id: '4b7f22a3-bbc3-4c53-81d3-d6d34cb77ee9'
}
}
}
`