P
P
Product Manual - 2.X
v2.20
Search…
⌃K

Access log-structure

Log line structure is as follows, note that some fields are quoted and some are not.if we will add more fields in the future, they'll be added on the right side.

Field names:

Field
Description
remote_addr
Client IP
timestamp
Timestamp
status
Response Status code sent to client
bytes_sent
Number of bytes sent in the response
method
HTTP Request method
request
The complete URL (including the query string)
proto_version
Protocol Version (1.0/1.1)
blocked
Was it blocked by Reblaze?
is_human
Was it marked as human?
block_reason
If blocked / Exceptionally passed - for what reason
geoip_country_name
Country Name
geoip_country_code
Country Code
request_id
Unique ID of this request within Reblaze
captured_vector
The vector attack we captured
request_time
The time it took our system to process the request
upstream_addr
The address of the upstream server(s) reblaze approached
upstream_response_time
The time Reblaze was waiting to the upstream server(s) to return the response
domain_name
The domain name of the server group in reblaze
host
The Host header of this request (same as domain name, or one of its aliases)
referer
The HTTP Referer header
http_user_agent
HTTP User Agent
http_cookie
The Cookie header string
request_headers
Request headers encoded in base6
organization
The complete organisation name owning the IP address
upstream_status
The status code returned by the upstream server
uri
The request URI without query part
hostname
The Proxy (Reblaze) server that process the request.
is_cloud
is_tor
is_vpn
is_anonymizer
is_proxy
rbzsessionid
Reblaze Session ID
request_length
The upload request size in bytes
sent_http_cache_control
The Cache-Control header we sent as part of the response
sent_http_expires
The Expires header we sent as part of the response
cookie_rbzid
Hash of the RBZID Cookie
sent_http_content_type
The MIME - Type (Content-Type response header)
browsersig
A unique signature of the visiting browser (future use)
ssl_protocol
SSL Protocol Version
ssl_cipher
Selected SSL Cipher
cache_status
Upstream Cache Status
anything_else
Future use place holder

Example line (sliced into smaller parts):

15.14.13.12 1465461803.223 200 3158 \
"POST /payment_service_api/get_all_payment_methods.json HTTP/1.1" "0" "0" \
"" "Singapore" "SG" "foobar-rbzr131343635343631383032ce9df9a6a90fa3bc" "-" \
0.729 "8.12.40.38:443" "0.418" "secure.foobar.com" "secure.foobar.com" "-" \
"REEBONZ 5.2.2 rv:3 (iPhone; iPhone OS 9.3.2; en_SG)" "-" \
"eyJob3N0Ijoic2VjdXJlLnJlZWJvbnouY29tIiwieC1uZXdyZWxpYy1pZCI6IlVnWURVRkJBQ1FzSlZGbGFCUT09IiwiY29udGVudC10eXBlIjoiYXBwbGljYXRpb25cL3gtd3d3LWZvcm0tdXJsZW5jb2RlZDsgY2hhcnNldD11dGYtOCIsImNvbm5lY3Rpb24iOiJjbG9zZSIsImNvbnRlbnQtbGVuZ3RoIjoiMTQ0IiwiYWNjZXB0LWVuY29kaW5nIjoiZ3ppcCIsInBvc3RfcmVxdWVzdF9ib2R5Ijp7ImNvdW50cnlfY29kZSI6IlNHIiwic2lnbmF0dXJlIjoiMGE2YzQwZTI3ZGRmMGYyNjcxMzM2YjhiYTljYjVmYzIiLCJkYXRldGltZSI6IjIwMTYtMDYtMDkgMTY6NDM6MjEiLCJkcnVwYWxfdWlkIjoiNTI0NTgxNiIsInBsYXRmb3JtX25hbWUiOiJNb2JpbGUiLCJidV9jb2RlIjoiMDEifSwidXNlci1hZ2VudCI6IlJFRUJPTlogNS4yLjIgcnY6MyAoaVBob25lOyBpUGhvbmUgT1MgOS4zLjI7IGVuX1NHKSJ9" \
"AS4773 MobileOne Ltd. Mobile/Internet Service Provider Singapore" "200" \
"/payment_service_api/get_all_payment_methods.json" "foobar-rbzr1" "0" "0" "0" "0" \
"0" "83d90abb12608623ea23442273249146" "466" "max-age=0, private, must-revalidate" \
"-" "-" "text/html; charset=utf-8" "-" "TLSv1.2" "ECDHE-RSA-AES128-GCM-SHA256" "-"
Headers are base64 encoded JSON string contains headers names and values.
if we decided to block a request, either by ACL, or WAF/IPS it will be marked as "1" in the blocked field,
as well there will be a description at "block_reason" field.

Decoding headers filed shall result with the following:

{
"host": "secure.reebonz.com",
"x-newrelic-id": "UgYDUFBACQsJVFlaBQ==",
"content-type": "application/x-www-form-urlencoded; charset=utf-8",
"connection": "close",
"content-length": "144",
"accept-encoding": "gzip",
"post_request_body": {
"country_code": "SG",
"signature": "0a6c40e27ddf0f2671336b8ba9cb5fc2",
"datetime": "2016-06-09 16:43:21",
"drupal_uid": "5245816",
"platform_name": "Mobile",
"bu_code": "01"
},
"user-agent": "FOOBAR 5.2.2 rv:3 (iPhone; iPhone OS 9.3.2; en_SG)"
}
Note that within the headers, we add an entry for the post_request_body.

Python Parser Example:

import re
access_line_rec = re.compile(''' # -- line by line
(\S+)\s # remote_addr
(\S+)\s # timestamp
(\S+)\s # status
(\S+)\s # bytes_sent
"(\S+)\s # METHOD
(.+) # request
\s(HTTP/\d\.\d)"\s # PROTOCOL/Version
"([^"]*)"\s # blocked
"([^"]*)"\s # is_human
"([^"]*)"\s # block_reason
"([^"]*)"\s # geoip_city_country_name
"([^"]*)"\s # geoip_city
"([^"]*)"\s # request_id
"([^"]*)"\s # captured_vector
(\S+)\s # request_time
"([^"]*)"\s # upstream_addr
"([^"]*)"\s # upstream_response_time
"([^"]*)"\s # canonical_domain_name
"([^"]*)"\s # http_host
"([^"]*)"\s # referer
"([^"]*)"\s # user-agent
"([^"]*)"\s # cookie
"([^"]*)"\s # headers
"([^"]*)"\s # organization
"([^"]*)"\s # upstream_status
"([^"]*)"\s # uri
"([^"]*)"\s # hostname
"([^"]*)"\s # is_cloud
"([^"]*)"\s # is_tor
"([^"]*)"\s # is_vpn
"([^"]*)"\s # is_anonymizer
"([^"]*)"\s # is_proxy
"([^"]*)"\s # rbzsessionid
"([^"]*)"\s # request_length
"([^"]*)"\s # sent_http_cache_control
"([^"]*)"\s # sent_http_expires
"([^"]*)"\s # cookie_rbzid
"([^"]*)"\s # sent_http_content_type
"([^"]*)"\s # browsersig
"([^"]*)"\s # ssl_protocol
"([^"]*)"\s # ssl_cipher
"([^"]*)" # cache_status
(.*)''', re.X) # anything else
names = ("remote_addr","timestamp","status",
"bytes_sent","method","request","proto_version",
"blocked","is_human","block_reason",
"geoip_city_country_name","geoip_city",
"request_id", "captured_vector", "request_time", "upstream_addr",
"upstream_response_time", "domain_name", "host", "referer",
"user_agent", "cookie", "request_headers", "organization", "upstream_status",
"uri", "hostname", "is_cloud", "is_tor", "is_vpn", "is_anonymizer",
"is_proxy", "rbzsessionid", "request_length", "sent_http_cache_control",
"sent_http_expires", "cookie_rbzid", "sent_http_content_type", "browsersig",
"ssl_protocol", "ssl_cipher", "cache_status", "anything_else")
def parse_line(line, as_dict=False):
rmatch = access_line_rec.match(line)
if rmatch:
g_match = rmatch.groups()
if not as_dict:
return g_match
else:
# to do, check if using re-group names would be faster
return dict(zip(names, g_match))
else:
return None