Create a New Web Data Source

The Web crawler uses Aperture, an open source crawler, which is intended as a small-scale crawler not designed as a large-scale internet crawler (for example, it is single-threaded, which means a large list of URLs may take a long time to fully crawl). For more large scale crawling, you may want to consider [Integrating Nutch|lweug:Integrating Nutch].

The following table explains the fields in this form:

|| Field || Description ||
| Name | A name you give for the data source. Data source names may contain any combination of letters, digits, spaces and other characters, and data source names are case sensitive. |
| URL | The URL for the site. Web data sources are expected to be unique by URL, which means that creating two data sources for the same URL is not possible. This is the base URL; by default, all pages linked from this URL will be crawled, unless you select some of the limiting options available on the configuration form. \\
{warning:title=Working with HTTP redirects}The embedded crawler does not work well with HTTP redirects which are URLs that direct the user to another page at a different address. If a Web data source is created with a redirected URL, LucidWorks may initially fail to crawl the content. A second attempt to crawl, however, will usually successfully crawl the site. {warning} |
| Ignore robots.txt | By default, the crawler will respect a robots.txt file found at a site, which is a way for site owners to request no crawling of parts or all of their site.  This box should only checked if the site is known or if the site owner has granted permission to ignore it. |
| Proxy Server | The crawler can communicate with Web sites through an HTTP proxy.  Click *Show* to open the configuration area, then enter the proxy server host, port, username and password. |
| Authentication credentials | The crawler can access websites that require authentication. Currently only Basic, Digest and NTLMv1 types of HTTP authentication are supported. Click *Show* to open the configuration area, then click *Add* to enter the username, password, realm, and host where the authentication should be used. |
| Commit within (seconds) | Defines the maximum interval between commits, the process that adds the crawled documents to the index, which allows crawled documents to be searchable while crawling continues. The default is 900 seconds (or 15 minutes). Commits can be resource-heavy so if your index is large, you may want to set this higher to avoid system slowdowns in either crawling or searching. For v2.1, the interval has changed from minutes to seconds. |
| Commit when crawl finishes | If this box is checked, the commit will not start until the end of the crawl.  This may be preferable if documents do not need to be available to users immediately, or if your index is large and you want to avoid system resource conflicts. |

| Exclude paths | Directories on the site that should not be crawled and that should be excluded from the index. You can use the same regular expression syntax to specify Exclude Paths as you used for Include Paths. |
| Skip files larger than (bytes) | The crawler can skip files larger than a specified size. This may be useful if index size or the length of time it takes to crawl a data source is a concern. You can enter any number, but if you leave it blank, a default limit of 10MB will be imposed (entered in the form already, as 10,485,760 bytes). |
| Log extra detail | By default, LucidWorks limits the amount of information that is printed to the logs about crawl activities to reduce the size of the log files. &nbsp;However, this information can be helpful if documents aren't being indexed as expected. &nbsp;Check this box for more information to be printed to the&nbsp;{{data/logs/core.<YYYY_MM_DD>.log}}. &nbsp;The default is for one line to be printed to the log showing the status of the document accessed ("New", "Update", "Delete" or "Unchanged"). &nbsp;When this setting is enabled, two lines will be shown in the logs for every document: one to say the file is being accessed, and a second to show the status of the document. &nbsp;If there is an error in accessing the document, the first line of the extra log detail may be helpful in figuring out the problem. |
| Fail unsupported file types | By default, documents that cannot be parsed by the crawler (either due to a failure or there is no recognizable data in the file) are not logged as errors. &nbsp;Checking this box will cause those failures to be reported in the log. If "Add failed docs" is also checked, the result of these failures will also be added to the index with whatever metadata could be extracted from the content. This is different from the "Log warnings for unknown mime types" in the sense that this parameter only applies to content that fails parsing and not content that doesn't have a defined mime type. |
| Add failed docs | If this option is selected, documents that failed processing (due to invalid formats, parsing errors, IO failures, etc.) will be added to the index with whatever metadata that could be extracted from the content (if it was only partially parsed, for example) and the reason for the error in the "parse" or "fetch" field, which would be added to the document. *This is a new parameter for v2.1.* |
| Log warnings for unknown mime types | By default, the crawler attempts to process documents with no mime type defined as plain text files. &nbsp;If that is not possible, the file will be skipped. &nbsp;If this happens, it is not reported in the log as errors or failures. Check this box if you'd like such files to be reported in the log. This parameter differs from "Fail unsupported file types" in the sense that it only applies to content that does not have a defined mime type. |

When you have filled in the form, click *Create data source*.

{info:title=Use Constrain to, Include Paths and/or Exclude Paths to limit Web crawling}
If you do not specify to Constrain to a subtree, define any Include Paths or Exclude Paths, and your Crawl Depth is higher than 0, LWE will follow *all* links on the page, even if they are to other websites. This may result in acquiring more documents than required, the crawl taking much longer than expected, or performance issues with the system. As noted above, the crawler is not designed for large-scale web crawling so it's recommended to limit crawls only to that content that's required for your implementation.
{info}

h2. Related Topics

* [lweug:Integrating Nutch]
* [lweug:Processing Documents in Batches]