Crawl pages via following a 'next page' or 'more' link

You can scrape data across a series of pages that are linked together sequentially with a 'next page' or 'more'. You can watch a video on how to set up pagination. If you want to crawl multiple pages of data but pagination doesn't meet your needs, you can look into our other options.

To create a paginating API:

1. Create an API as you normally would. Once you've setup all of the desired collections/datatypes, click the  button in the toolbar to activate pagination.

While pagination is activated, you are in a special mode in which kimono is waiting for you to identify the 'Next page' link. If there is no link, behind the next page icon then kimono cannot currently support this type of page. Next page links often look like this:

Once you select the next page link, it will be highlighted in grey:

When you save your paginated API, you will be prompted to enter the maximum number of pages to crawl. You should set your page limit to the lowest threshold that will still return all your data (e.g., 25 pages if you want to scrape 23 pages)

From the 'API Setup' tab on the API detail page, you can adjust your crawling API settings. Note that with multi-page crawling, you can either scheduled your crawls or trigger them manually or with the kimono RESTful API.

Powered by Zendesk