GuidesAPI Reference
API Reference

Crawl

post
https://api.fetchfox.ai/api/crawl

Crawl a site for URLs matching a pattern

Recent Requests
Log in to see full request history
TimeStatusUser Agent
Retrieving recent requests…
LoadingLoading…

Crawl takes a URL pattern, and finds matching URLs on the target site. A URL pattern is a URL that contains at least one wildcard, like https://example.com/category/*.

You can control the crawls maximum depth, how it prioritizes pages, starting URLs, and more.

Body Params
string
required
Defaults to https://pokemondb.net/pokedex/*

The URL pattern to match

start_urls
array of strings

Specify where the crawl starts. If not provided, default includes the top level domain.

start_urls
int32

Limit the number of URLs the crawler visits before returning results.

int32

Limit the distance from the starting URLs. Each link away from a page adds 1 to the distance, but only if the path changes. Query strings and fragments are ignored when computing distance.

Response
200

Successful response

Updated over 1 year ago

Did this page help you?
Language
Credentials
Bearer
LoadingLoading…
Response
Click Try It! to start a request and see the response here!

Updated over 1 year ago

Did this page help you?