Web Content (Crawler) Connector Limitations

Learn about the functional limitations of the Web Content (Crawler) connector that affect certain behaviors and outcomes.

Unsupported File Types Aren't Ingested

HTML and PDF pages are supported.
Supported image formats: .jpg, .jpeg, .png.
Video and audio files (.mp3, .mp4, etc.) aren’t supported.

Language Metadata Isn't Supported

The Web Content (Crawler) connector only identifies the ingested content as English, without providing other language metadata.

Client-Side Rendering Isn't Supported

The Web Content (Crawler) connector doesn’t support content that is dynamically rendered via JavaScript at runtime. Any content rendered in this manner isn’t ingested.

URL Query Strings Aren't Supported

When parsing links, the connector automatically removes the query string from the URL—all text following the ?. This limitation applies to all pages except the root page.

Rate Limit

The connector’s rate limit is 5 records (pages) per second. If your website doesn’t support this rate limit, the connector may fail.

Apache HttpClient Requests

The ingested site needs to allow Apache HttpClient Requests.

General Data 360 Limitations

Learn about Data 360’s general limitations which may create issues for this specific connector.