Access log-structure

Log line structure is as follows, note that some fields are quoted and some are not.if we will add more fields in the future, they'll be added on the right side.

Field names:

Field	Description
remote_addr	Client IP
timestamp	Timestamp
status	Response Status code sent to client
bytes_sent	Number of bytes sent in the response
method	HTTP Request method
request	The complete URL (including the query string)
proto_version	Protocol Version (1.0/1.1)
blocked	Was it blocked by Reblaze?
is_human	Was it marked as human?
block_reason	If blocked / Exceptionally passed - for what reason
geoip_country_name	Country Name
geoip_country_code	Country Code
request_id	Unique ID of this request within Reblaze
captured_vector	The vector attack we captured
request_time	The time it took our system to process the request
upstream_addr	The address of the upstream server(s) reblaze approached
upstream_response_time	The time Reblaze was waiting to the upstream server(s) to return the response
domain_name	The domain name of the server group in reblaze
host	The Host header of this request (same as domain name, or one of its aliases)
referer	The HTTP Referer header
http_user_agent	HTTP User Agent
http_cookie	The Cookie header string
request_headers	Request headers encoded in base6
organization	The complete organisation name owning the IP address
upstream_status	The status code returned by the upstream server
uri	The request URI without query part
hostname	The Proxy (Reblaze) server that process the request.
is_cloud
is_tor
is_vpn
is_anonymizer
is_proxy
rbzsessionid	Reblaze Session ID
request_length	The upload request size in bytes
sent_http_cache_control	The Cache-Control header we sent as part of the response
sent_http_expires	The Expires header we sent as part of the response
cookie_rbzid	Hash of the RBZID Cookie
sent_http_content_type	The MIME - Type (Content-Type response header)
browsersig	A unique signature of the visiting browser (future use)
ssl_protocol	SSL Protocol Version
ssl_cipher	Selected SSL Cipher
cache_status	Upstream Cache Status
anything_else	Future use place holder

Example line (sliced into smaller parts):

15.14.13.12 1465461803.223 200 3158 \
"POST /payment_service_api/get_all_payment_methods.json HTTP/1.1" "0" "0" \
"" "Singapore" "SG" "foobar-rbzr131343635343631383032ce9df9a6a90fa3bc" "-" \
0.729 "8.12.40.38:443" "0.418" "secure.foobar.com" "secure.foobar.com" "-" \
"REEBONZ 5.2.2 rv:3 (iPhone; iPhone OS 9.3.2; en_SG)" "-" \
"eyJob3N0Ijoic2VjdXJlLnJlZWJvbnouY29tIiwieC1uZXdyZWxpYy1pZCI6IlVnWURVRkJBQ1FzSlZGbGFCUT09IiwiY29udGVudC10eXBlIjoiYXBwbGljYXRpb25cL3gtd3d3LWZvcm0tdXJsZW5jb2RlZDsgY2hhcnNldD11dGYtOCIsImNvbm5lY3Rpb24iOiJjbG9zZSIsImNvbnRlbnQtbGVuZ3RoIjoiMTQ0IiwiYWNjZXB0LWVuY29kaW5nIjoiZ3ppcCIsInBvc3RfcmVxdWVzdF9ib2R5Ijp7ImNvdW50cnlfY29kZSI6IlNHIiwic2lnbmF0dXJlIjoiMGE2YzQwZTI3ZGRmMGYyNjcxMzM2YjhiYTljYjVmYzIiLCJkYXRldGltZSI6IjIwMTYtMDYtMDkgMTY6NDM6MjEiLCJkcnVwYWxfdWlkIjoiNTI0NTgxNiIsInBsYXRmb3JtX25hbWUiOiJNb2JpbGUiLCJidV9jb2RlIjoiMDEifSwidXNlci1hZ2VudCI6IlJFRUJPTlogNS4yLjIgcnY6MyAoaVBob25lOyBpUGhvbmUgT1MgOS4zLjI7IGVuX1NHKSJ9" \
"AS4773 MobileOne Ltd. Mobile/Internet Service Provider Singapore" "200" \
"/payment_service_api/get_all_payment_methods.json" "foobar-rbzr1" "0" "0" "0" "0" \
"0" "83d90abb12608623ea23442273249146" "466" "max-age=0, private, must-revalidate" \
"-" "-" "text/html; charset=utf-8" "-" "TLSv1.2" "ECDHE-RSA-AES128-GCM-SHA256" "-"

Headers are base64 encoded JSON string contains headers names and values.

if we decided to block a request, either by ACL, or WAF/IPS it will be marked as "1" in the blocked field,

as well there will be a description at "block_reason" field.

Decoding headers filed shall result with the following:

{
  "host": "secure.reebonz.com",
  "x-newrelic-id": "UgYDUFBACQsJVFlaBQ==",
  "content-type": "application/x-www-form-urlencoded; charset=utf-8",
  "connection": "close",
  "content-length": "144",
  "accept-encoding": "gzip",
  "post_request_body": {
    "country_code": "SG",
    "signature": "0a6c40e27ddf0f2671336b8ba9cb5fc2",
    "datetime": "2016-06-09 16:43:21",
    "drupal_uid": "5245816",
    "platform_name": "Mobile",
    "bu_code": "01"
  },
  "user-agent": "FOOBAR 5.2.2 rv:3 (iPhone; iPhone OS 9.3.2; en_SG)"
}

Note that within the headers, we add an entry for the post_request_body.

Python Parser Example:

import re
access_line_rec = re.compile(''' # -- line by line
    (\S+)\s         # remote_addr
    (\S+)\s         # timestamp
    (\S+)\s         # status
    (\S+)\s         # bytes_sent
    "(\S+)\s        # METHOD
    (.+)            # request
    \s(HTTP/\d\.\d)"\s     # PROTOCOL/Version
    "([^"]*)"\s     # blocked
    "([^"]*)"\s     # is_human
    "([^"]*)"\s     # block_reason
    "([^"]*)"\s     # geoip_city_country_name
    "([^"]*)"\s     # geoip_city
    "([^"]*)"\s     # request_id
    "([^"]*)"\s     # captured_vector
    (\S+)\s         # request_time
    "([^"]*)"\s     # upstream_addr
    "([^"]*)"\s     # upstream_response_time
    "([^"]*)"\s     # canonical_domain_name
    "([^"]*)"\s     # http_host
    "([^"]*)"\s     # referer
    "([^"]*)"\s     # user-agent
    "([^"]*)"\s     # cookie
    "([^"]*)"\s     # headers
    "([^"]*)"\s     # organization
    "([^"]*)"\s     # upstream_status
    "([^"]*)"\s     # uri
    "([^"]*)"\s     # hostname
    "([^"]*)"\s     # is_cloud
    "([^"]*)"\s     # is_tor
    "([^"]*)"\s     # is_vpn
    "([^"]*)"\s     # is_anonymizer
    "([^"]*)"\s     # is_proxy
    "([^"]*)"\s     # rbzsessionid
    "([^"]*)"\s     # request_length
    "([^"]*)"\s     # sent_http_cache_control
    "([^"]*)"\s     # sent_http_expires
    "([^"]*)"\s     # cookie_rbzid
    "([^"]*)"\s     # sent_http_content_type
    "([^"]*)"\s     # browsersig
    "([^"]*)"\s     # ssl_protocol
    "([^"]*)"\s     # ssl_cipher
    "([^"]*)"       # cache_status
    (.*)''', re.X)  # anything else

names = ("remote_addr","timestamp","status",
    "bytes_sent","method","request","proto_version", 
    "blocked","is_human","block_reason",
    "geoip_city_country_name","geoip_city",
    "request_id", "captured_vector", "request_time", "upstream_addr", 
    "upstream_response_time", "domain_name", "host", "referer", 
    "user_agent", "cookie", "request_headers", "organization", "upstream_status", 
    "uri", "hostname", "is_cloud", "is_tor", "is_vpn", "is_anonymizer", 
    "is_proxy", "rbzsessionid", "request_length", "sent_http_cache_control", 
    "sent_http_expires", "cookie_rbzid", "sent_http_content_type", "browsersig", 
    "ssl_protocol", "ssl_cipher", "cache_status", "anything_else")

def parse_line(line, as_dict=False):
   rmatch = access_line_rec.match(line)
   if rmatch:
       g_match = rmatch.groups()
       if not as_dict:
           return g_match
       else:
           # to do, check if using re-group names would be faster
           return dict(zip(names, g_match))
   else:
       return None