Crawl Links For: Root
  • The crawler will fetch titles, descriptions and keywords from meta tags where available.
  • Do not add too many URLs at once as it may cause the browser to time-out or cause slow loading pages.
_
Add URLs to crawl below: (Each url should be on a new line)

Ignore Crawled Items that doesn't have a Meta Description
Ignore Crawled Items that doesn't have Meta Keywords

Threads:
Threads speeds up crawling be creating more than one instance of the crawler to crawl multiple URLs at the same time)

_
Extract URLs from Web Page:

The 'Extract URLs' functionality lets you extract URLs from web pages (or URL addresses) so that they can be crawled.

URL to web page where URLs should be extracted from:

URL Filter:
(URLs specified here will not be included when extracting URLs from the address specified above. You can only specify a domain name for example and any URLs from this domain name won't be included.)
_