Understand the Crawl Setup tab (API detail page)

Your crawl setup tab is where you can manage your API settings, view your API status, and input the URLs you would like your API to crawl

The crawl setup tab:


Cheat sheet for the fields on the crawl setup tab:

FeatureDescription
Crawl Strategy Determine the URLS your API will crawl - the default is to ONLY crawl the source URL, but you can crawl more pages by using a kimono API to source the URLs, generating the URLs or manually inputting the URLs.
Crawl frequency Choose frequency settings to scrape your data.  You can set your API to only trigger a scrape when called - e.g., a manual crawl - or automatically on a schedule.  
Rows returned Number of data rows extracted in the last crawl
URLs crawled Number of URLs successfuly crawled, number that required a second or third attempt, and number of failed URLS
Start crawl Button to manually trigger a crawl
URLs to crawl The URLs will crawl - this depends on your API crawl strategy 
Powered by Zendesk