Raw Logs

The raw logs functionality allows you to send logs associated with your CDN resources to your distant server in real time. The raw log allows customers to understand, analyze, and debug files delivered via OnApp CDN, or can be served as an audit trailed. Once the user creates CDN resource(s), the raw logs are enabled for this account and the user can configure and receive raw logs.  Logs are sent for all the CDN Resources associated with the user. There are three different types of delivery: Syslog, SFTP, and FTP. The frequency of uploading the log to client destination is every 10 minutes for SFTP and FTP protocols. For the Syslog protocol, logs are uploaded instantaneously.

If the user does not have any CDN resources, they cannot configure raw logs.

To view and set the raw log configuration:

  1. Go to your Control Panel > CDN > Resources menu.
  2. Click the Raw Log tab.
  3. The page that loads shows the current raw log configuration. On this page you can also set the raw log configuration:

    • For the FTP/SFTP delivery protocol:

      • Hostname - fill in the hostname of the server to which the log will be delivered
      • Username - specify the user name of the FTP/SFTP client on the server to which the log will be delivered
      • Password - fill in the password of the FTP/SFTP client on the server to which the log will be delivered
    • For the Syslog delivery protocol:
      • Hostname - fill in the hostname of the server to which the log will be delivered
      • Syslog protocol - select the protocol that will be used for sending the log: TCP or UDP
      • Port - specify the port number of the syslog server to which the log will be delivered
    • Choose Disabled to disable raw logs.
  4. Click Save to save the configuration.

To edit, set new configuration parameters and click Save.

To disable, select Disabled from the raw log configuration delivery protocol drop-down list.

The list of raw log parameters includes the following:

  • $upstream_response_time - keeps time spent on receiving the response from the upstream server; the time is kept in seconds with millisecond resolution. Times of several responses are separated by commas and colons like addresses in the $upstream_addr variable.
  • $upstream_http_x_cache - a value of the X-Cache HTTP response header field from the upstream server.
  • $request_time - the time elapsed between the first bytes that have been read from the client and the writing log after the last bytes have been sent to the client.
  • $tcpinfo_rtt - the round trip time (ping) estimate of the TCP socket in microseconds.
  • $http_referer - the HTTP request header field that identifies the address of the webpage (i.e., the URI or IRI) linked to the resource requested.
  • $for_operator - a buyer (if any) subscribed to the edge server.

If the SFTP or FTP protocol is applied, raw logs are delivered as an archive. If the Syslog delivery protocol is selected, user will receive the text of the logs.

There are different log formats for the Syslog and FTP/SFTP delivery protocols.

An example of the HTTP log format (V8) for the Syslog delivery protocol:

$time_local<TAB>$edge_server_id<TAB>$remote_addr<TAB>$request_method<TAB>$http_host<TAB>$request_uri<TAB>$status<TAB>$bytes_sent<TAB>$upstream_http_x_cache<TAB>$upstream_response_time<TAB>$request_time<TAB>$tcpinfo_rtt<TAB>$for_operator<TAB>$resource_id<TAB>$server_name<TAB>$server_addr<TAB>$http_user_agent<TAB>$http_referer<TAB>$http_range<TAB>$body_bytes_sent<TAB>$publisher_id<TAB>$tls_version<TAB>$placeholder1<TAB>$placeholder2<TAB>$placeholder3<TAB>$placeholder4<TAB>$placeholder5<TAB>$placeholder6<TAB>$placeholder7<TAB>$placeholder8<TAB>V8

An example of the HTTP log format (V8) for the Syslog delivery protocol:

00:00:00 +0000 123456789 1.2.3.5 GET test-site.com /testfile.txt 200 1086 HIT 0.000 0.002 110972 234567891 345678912 test-site.com 1.2.3.4 Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.88 Safari/537.36 https://www.test-site.com/test-page - 671 456789123 TLSv1.3 - - - - - - - - V8

An example of the Stream log format for the Syslog delivery protocol:

date<TAB>time<TAB>edge-id<TAB>type<TAB>x-app<TAB>x-severity<TAB>x-category<TAB>x-event<TAB>date<TAB>time<TAB>c-client-id<TAB>c-ip<TAB>c-port<TAB>cs-bytes<TAB>sc-bytes<TAB>x-duration<TAB>x-sname<TAB>x-stream-id<TAB>x-spos<TAB>sc-stream-bytes<TAB>cs-stream-bytes<TAB>x-file-size<TAB>x-file-length<TAB>x-ctx<TAB>x-comment<TAB>x-sname-query

An example of the HTTP log format for FTP/SFTP delivery protocols:

$remote_addr - - [$time_local] "$request_method $request_uri HTTP/1.1" $http_status $bytes_sent "$http_referer" "$http_user_agent"

An example of the HTTP log format for FTP/SFTP delivery protocols:

1.2.3.4 - - [02/Jan/2020:07:59:42 +0000] "GET /test-file.txt HTTP/1.1" 200 1000 "https://www.test-site.com/test-page" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36"

An example of the Stream log format for FTP/SFTP delivery protocols:

x-app<TAB>x-severity<TAB>x-category<TAB>x-event<TAB>date<TAB>time<TAB>c-client-id<TAB>c-ip<TAB>c-port<TAB>cs-bytes<TAB>sc-bytes<TAB>x-duration<TAB>x-sname<TAB>x-stream-id<TAB>x-spos<TAB>sc-stream-bytes<TAB>cs-stream-bytes<TAB>x-file-size<TAB>x-file-length<TAB>x-ctx<TAB>x-comment<TAB>x-sname-query