eCDN Logpush delivers eCDN logs in batches to specified destinations. eCDN logs contain valuable information, such as the client IP address, the requested URL, the user agent string, and details about firewall events such as blocked requests, firewall rule triggers, and more.
When creating a Logpush job through eCDN Logpush, view the list of available fields for http_requests, firewall, and pageshield events. Also specify which log types to deliver, including HTTP requests and firewall events, and specify different destination types:
Create an S3 bucket. For details, see AWS S3 documentation.
In the Amazon S3 console, navigate to S3 > Bucket > Permissions > Bucket Policy, and edit and paste the policy, replacing the Resource value with your own bucket path. The AWS Principal is owned by the CDN provider and must not be changed.
To avoid incomplete files and minimize storage costs, use the AbortIncompleteMultipartUpload action when setting up a rule for S3 multipart uploads with eCDN Logpush. For details, see Uploading and copying objects using multipart upload_.
Note
Create Amazon S3 Bucket Ownership Token
eCDN Logpush supports pushing eCDN logs directly to Amazon S3 through CDN-API Logpush API.
However, before you can create an eCDN Logpush job to directly push logs to an AWS S3 bucket, you must demonstrate ownership of the S3 bucket. You do this by writing the ownership token file to the bucket destination. You must also create a unique ownership challenge token for each destination path within the same bucket.
You can use the special string {DATE} in the destination path to separate logs into daily subdirectories. For example, s3://customer-bucket/logs/{DATE}?region=us-east-1\&sse=AES256. When the logs are stored, the name of the directory is replaced with the date in YYYYMMDD format, for example: 20230215.
Populate the `ownershipChallengeToken` based on the call made in the previous section. The destinationPath must be the same as the one specified while generating the ownership token. For details, see the Parameters, Optional Parameters and Limitations sections.
Cloudflare uses Google Cloud Identity and Access Management (IAM) to gain access to your Google Cloud Storage bucket. The Cloudflare IAM service account needs admin permission for the bucket. Add the member logpush@cloudflare-data.iam.gserviceaccount.com with Storage Object Admin permission. This makes sure that Cloudflare can push the ownership verification and logs to the GCS bucket.
Create GCS Ownership Token
eCDN Logpush supports pushing eCDN logs directly to Google Cloud Storage (GCS) through the CDN Logpush API.
However, before you can create an eCDN Logpush job for pushing logs to a GCS bucket, you must demonstrate ownership of the GCS bucket. You do this by writing the ownership token file to the bucket destination. You must also create a unique ownership challenge token for each destination path within the same bucket.
You can use the special string {DATE} in the destination path to separate logs into daily subdirectories. For example gs://customer-bucket/logs/{DATE}. When the logs are stored, the name of the directory is replaced with the date in YYYYMMDD format, for example: 20230215.
Populate the ownershipChallengeToken based on the call made in the previous section. The `destinationPath` must be the same as one specified while generating the ownership token. For details, see the Parameters, Optional Parameters and Limitations sections.
Use Storage Explorer ↗ to navigate to your container and right click on the container to create a signature. Set the signature to expire at least five years from the current date and only provide write permission.
Provide the SAS URL as part of your destinationPath attribute.
Logpush stops pushing logs if your SAS token expires, which is why an expiration period of at least five years is needed.
Cloudflare makes a test call to push test data to the Azure destination specified. If successful, the logpush job is created. If not, the logpush job creation fails.
Specify your destination configuration hostname as either http-intake.logs.datadoghq.com/v1/input or http-intake.logs.datadoghq.com/api/v2/logs
Retrieve the Datadog API Key using the Datadog documentation. This is a required field.
You can set the service, host name, Datadog ddsource field, and ddtags fields as URL parameters. While these parameters are optional, they can be useful for indexing or processing logs. For details, see the Logs section ↗ in Datadog’s documentation. Note that the values of these parameters can contain special characters, which should be URL encoded.
Cloudflare makes a test call to push test data to the specified Datadog destination configuration. If successful, the logpush job is created. If not, the logpush job creation fails.
API requests fail with a 504 error if you add an incorrect URL. Typically, the Splunk Cloud endpoint URL includes text like http-inputs- before the hostname.
<SPLUNK_CHANNEL_ID>: A unique channel ID. This is a random GUID that you can generate by using:
The command line, for example: python \-c 'import uuid; print(uuid.uuid4())'.
<INSECURE_SKIP_VERIFY>: A boolean value. Cloudflare recommends setting this value to false, because setting this value to true is equivalent to using the -k option with curl as shown in Splunk examples and is not recommended unless HEC uses a self-signed certificate.
Cloudflare makes a test call to push test data to the endpoint specified. If successful, the logpush job is created. If not, the logpush job creation fails.
Make sure that the file upload to validate the destination accepts a gzipped test.txt.gz file with compressed content {"content":"tests"}. Otherwise, it returns an error, for example: error validating destination: error writing object: error uploading.
Specify “header_*” URL parameters for setting request headers.
The HTTPS endpoint cannot have custom URL parameters that conflict with any “header_*” URL parameters you have set.
These parameters must be properly URL-encoded, for example: use “%20” for a whitespace. Otherwise, some special characters can be decoded incorrectly.
For destinationconf , specify URL parameters in addition to special “header*” parameters.
Non URL-encoded special characters are encoded when uploaded.
Example destination configuration for HTTPS endpoint:
Cloudflare makes a test call to push test data to the endpoint specified. If successful, the logpush job is created. If not, the logpush job creation fails.
Example destination configuration for setting up logpush to AWS Kinesis:
kinesis://<STREAM_NAME>?region=<AWS_REGION>&sts-assume-role-arn=arn:aws:iam::<YOUR_AWS_ACCOUNT_ID>:role/<IAM_ROLE_NAME>
Name: Your site name, which can’t be changed after the job is created.
Destination Path: The destination path for receiving logs. The prerequisites information for each destination type describes how to get the destinationPath attribute.
Ownership Challenge Token: This attribute is required for AWS S3 and Google Cloud Storage destinations. It isn’t required for other destinations such as HTTPS, Datadog, Azure, Splunk, or Kinesis.
Log Type: The type of logs you want to receive in your S3 bucket. Currently, the available types are http_requests, firewall_events, and page_shield_events.
Log Fields: The available log fields, which vary depending on the type of logs. For details, see eCDN Logpush Log Fields.
Optional Parameters
Filter: Use filters to select specific types of events to include in your logs or remove events irrelevant to your analysis. By applying filters to your logs, focus on the most important data and avoid unnecessary noise.
You can create a maximum of two Logpush jobs per zone.
Enable Logpush Job
An eCDN Logpush Job isn’t enabled upon creation. You must enable the Logpush Job to start receiving logs. For additional details, see CDN Zones API Overview.
The Logpush job was accidentally turned off, and we missed some logs for a certain time period. Is there a way to retrieve those missed logs?
No, eCDN Logpush pushes the logs into the S3 bucket as soon as they become available and isn’t able to backfill the missing logs.
Can I adjust how often logs are pushed?
No, eCDN Logpush pushes the logs in batches to the destination specified at the earliest possible time.
I created an ownership token, but I don’t see the file in my S3 bucket.
Check your S3 bucket policy for allowing eCDN Logpush to write a file to your bucket. Edit and paste the policy into S3 > Bucket > Permissions > Bucket Policy, replacing the Resource value as shown in the Enable eCDN Logpush to Amazon S3 section.