This feature will be reintroduced to the Kimono offering after some maintenance. It is not currently available as part of the free plan. Pre-existing APIs will not be affected.
URL parameter pass through allows you to scrape data across similarly structured pages by replacing the query and path parameters in the source URL with new values that you provide. This is particularly helpful for sites where you often filter search results (e.g., shopping & review sites). If you are unsure if URL parameter pass through is the right choice for your crawling scenario, look into our other options for crawling multiple pages.
Imagine you've kimonified a specific restaurant page on Yelp. You're at the page
/biz/pylos-new-york. For this site, the current URL structure happens to look like this:
You can view the different components of the URL that you can toggle on the 'API Docs' tab of the detail page. Note that using URL parameters will set your frequency to on demand and will automatically turn off any multi-page crawling you have. If you would like to toggle your URL parameters and use crawling, you can use an alternate method.
|Parameter||Default value||Parameter to append|
Then append one or more of the parameters as a query parameter on the API call. So for example if an on demand API call normally looks like this:
And you want your API to fetch data from the page for Chipotle instead, you would pass a new value for the
kimpath2parameter onto your API call like this:
This will temporarily modify the target url for your API and fetch data. The new target URL would effectively be:
You can pass through as many parameters as you have in your path/query. If you do not set a new parameter, the default value will be used.
Note that this will only work as long the structure for the Chipotle yelp page is identical to the Pylos yelp page