Google Drive Unstructured Connector Limitations
The Google Drive Unstructured data connector enforces specific constraints concerning file format compatibility, size limits, and content age during data ingestion. Files that don’t meet these criteria are automatically skipped, and their status is logged in the unstructured data lake object (UDLO) for auditing and review.
The connector is configured to process the following explicit MIME types and matching file patterns. This includes a wide range of documents, images, and text formats.
| Category | Supported MIME Types |
|---|---|
| Documents & Spreadsheets | application/msword, application/rtf, application/vnd.openxmlformats-officedocument.wordprocessingml.document, application/vnd.oasis.opendocument.text, application/vnd.sun.xml.writer, application/vnd.ms-excel, libre/doc, text/csv, application/json |
| application/vnd.pdf, application/acrobat, application/pdf, application/x-pdf, text/pdf, text/x-pdf | |
| Google Workspace | application/vnd.google-apps.document, application/vnd.google-apps.presentation, application/vnd.google-apps.spreadsheet |
| Images | image/bmp, image/gif, image/jpeg, image/png, image/svg+xml, image/tiff, image/webp |
| Web & Text | html, htm, text/html, text/markdown, text/x-markdown, text/plain |
The following regular expressions are used to capture supported file types that match broad patterns in their MIME type or file extension:
| Regex Pattern | Description |
|---|---|
| /^(?!.*google).doc. | .-doc./i |
| /.html. | .-html./i |
| /.image. | .-image./i |
| /.md. | .markdown./i |
| /.pdf./i | Matches MIME types containing pdf anywhere. |
| /^application/vnd.google-apps..*/i | Matches Google Apps-specific MIME types. |
The following Google Workspace MIME types are explicitly excluded from ingestion and are skipped:
- application/vnd.google-apps.shortcut
- application/vnd.google-apps.script
- application/vnd.google-apps.site
- application/vnd.google-apps.form
- application/vnd.google-apps.vid
To maintain processing stability and efficiency, the connector enforces maximum file size limits. Files are categorized, and a maximum size check is performed based on the content type. Any file exceeding its category limit will be skipped.
| File Type | Size Limit | MIME types & Regex Patterns |
|---|---|---|
| PDF, CSV & similar formats | 100 MB | MIME Types: application/pdf, application/x-pdf, application/acrobat, applications/vnd.pdf, text/pdf, text/x-pdf, text/csv, application/vnd.openxmlformats-officedocument.wordprocessingml.document Regex Patterns: ^application/vnd.google-apps..*, .pdf. |
| Image & Similar format | 20 MB | MIME Types: image/jpeg, image/png, image/gif, image/bmp, image/tiff, image/webp, image/svg+xml Regex Patterns: `.image. |
| All other supported Formats | 4 MB | Any supported file format that doesn’t fall into the PDF or Image categories are checked against the 4-MB maximum size limit. |
Content deemed stale based on its age is skipped to maintain relevance.
Rule: Any content where the last modified time is more than 2 years before the current sync time will be skipped.
When a file fails any constraint check (Format, Size, or Last Modified Time), it’s skipped. The outcome is recorded in the UDLO. The UDLO contains the below fields to detail the failure.
| Field | Value | Description |
|---|---|---|
| Sync Status | SUCCESS, FAILURE or SKIPPED | Indicates the file ingestion status. |
| Sync Status Detail | Detailed Error Message | Provides the explicit reason why the file was skipped. |
| File Path | NULL or Empty String | The file path field is explicitly cleared for all skipped records to mark them as non-ingested. |
Files are processed through a series of checks. If a check fails, the file is immediately skipped and the appropriate reason is logged in the UDLO.
| Failure Condition | Status Details Message |
|---|---|
| Unsupported Format | Explicitly Unsupported: If the file’s MIME type matches an unsupported pattern (for example, application/vnd.google-apps.shortcut). Unsupported content type: %s (for example, unsupported content type: audio files) |
| Default Format Skip | Implicitly Unsupported: If the file’s type doesn’t match any supported format or regex patterns, it will be skipped. Unsupported content type: %s (for example., Unsupported content type: audio files) |
| Content Age | Last modified time %s is older than %d years (for example, Last modified time 2022-01-15T10:30:00Z is older than 2 years) |
| Size Exceeded (PDF, CSV & Similar) | %s must be less than %d MB (found %.2f MB) (for example, PDF must be less than 100 MB (found 204.20 MB)) |
| Size Exceeded (Image & Similar) | %s must be less than %d MB (found %.2f MB) (for example, IMAGE must be less than 20 MB (found 25 MB)) |
| Size Exceeded (All Other) | %s must be less than %d MB (found %.2f MB) (for example, text/other files must be less than 4 MB (found 30 MB)) |
| Export Failed (Size) | Export of Google Workspace file failed: file size exceeds export limit |
| Export Failed (Complexity) | Export of Google Workspace file failed: file too complex to export |
Google Drive supports ingestion of drives up to 270 GB in a single ingestion job.
Ingestion jobs that run for longer than 24 hours may time out depending on system execution limits, so plan accordingly.
These limitations apply to the connector’s ingestion scope:
- Shared Drives require explicit Drive ID configuration. When ingesting from a Shared Drive, provide the Shared Drive ID during UDLO creation. The service account must have domain-wide delegation enabled and appropriate access to the Shared Drive. Files in Shared Drives that the service account can’t access are silently skipped.
- My Drive (personal drive) is not supported. Only Shared Drives can be ingested. Files stored in individual users’ My Drive are not accessible to this connector.
- Google Drive shortcuts are not followed. Files referenced via Google Drive shortcuts (
application/vnd.google-apps.shortcut) are excluded from ingestion. Only original files in their actual location are processed. - Folder-level filtering is not available. The connector ingests all supported files from the configured drive. You can’t scope ingestion to a specific folder or subfolder within a drive.
- Google Workspace files are exported during ingestion. Google Docs, Sheets, and Slides are native Google formats without a downloadable binary. The connector exports these to standard formats (PDF for Docs/Slides, CSV for Sheets) before ingestion. Export failures due to size or complexity result in the file being skipped.
To ingest files from a specific folder within a large drive:
- Folder scoping: Move the target files into a dedicated Shared Drive and configure the connector to ingest only that drive.
If the connector raises a MISSING_REQUIRED_PROPERTIES exception during data stream refresh, it typically indicates one of the following configuration issues:
- Missing Shared Drive ID. The connector is configured to ingest from a Shared Drive, but no Drive ID was provided during UDLO creation. Verify that the Shared Drive ID is correctly specified in the UDLO configuration.
- Domain-wide delegation not enabled. The service account used for authentication does not have domain-wide delegation enabled in the Google Workspace Admin console. Enable domain-wide delegation and ensure the required OAuth scopes are authorized.
- Insufficient service account permissions. The service account lacks access to the target Shared Drive. Grant the service account at least “Content Manager” or “Viewer” access on the Shared Drive.
To resolve, verify the Shared Drive ID in your UDLO configuration, confirm domain-wide delegation is enabled for the service account, and ensure the service account has been granted access to the target Shared Drive.