Troubleshoot a paginating API that won't crawl past the first page

There are a few reasons why your API could be getting stuck on the first page:

1. Your CSS selector for pagination (e.g., supposed to select the link to the subsequent pages) may be retriving an img or a span tag, instead of an a or btn tag. An incorrect selector does not show kimono how to navigate to the subsequent page.

You can check your example in advanced mode:

Correct selector example 

NOT

if your pagination selector is not pointing to an a or a btn tag , then edit your API and adjust your CSS selectors.

2. Make sure that your pagination CSS selector selects only one a or a btn tag  on the page. If your pagination CSS selector is too broad, it will pick up several links and your kimono API will not be able to determine which leads to the next page.

which gives you

NOT:

which gives you

 

3. Finally, if your pagination link is indicated by an icon, it can often help to reset your regex to standard regex, as regex for icons can prove problematic.

NOT:

 

If you are still having issues, there could be part of the pagination workflow that is not yet supported by kimono. A good work around for these instances is to use targeted URL crawling to either scrape the links (by scraping the links to each page) or generate URLs for the pages to be crawled (by picking up the pagination pattern with URL generator http://somewebiste.com/1, http://somewebiste.com/2 etc).

 

Powered by Zendesk