Understanding and Diagnosing Traffic Issues

Leveraging Reblaze's traffic data

Introduction

A typical security solution provides information about the traffic that it has blocked. Contrary to this, Reblaze shows all of the traffic that it receives.

This provides you with a powerful ability to dig into your traffic data and gain deep understanding about the nature and disposition of incoming requests.

Metrics about blocked requests are useful, but their usefulness is multiplied when you can compare them to passed requests. By showing all requests, Reblaze allows you to understand what "normal" requests look like. This gives you more insights into abnormal traffic.

The discussion below assumes you are already familiar with Query Filter Syntax and Best Practices.

Gaining General Insights

Reblaze provides multiple ways to view your traffic. The discussion below will focus on the Events Log. Similar tactics can be applied when viewing different parts of the Dashboard.

Sometimes the Events Log is used to answer a specific question about a request or a traffic source. At other times, it might be a more general exploration; for example, a beginning-of-the-day review of what happened over the last 24 hours.

Let's discuss the latter scenario (a general exploration). Because the Events Log shows all requests, it can be overwhelming. It's helpful to start by excluding requests that aren't as relevant to your current purpose. Examples:

• Exclude passed requests: result!="Passed"

• Exclude requests being rejected by the origin (i.e., the upstream server): result!="Blocked by origin"

• Exclude requests from a banned IP: tags!="ip:192-168-2-3".

• Exclude requests that are obviously invalid. For example, exclude those with unrecognized host headers: reason~"unrecognized".

Eventually, you can filter the display down to a list of challenges or blocked requests that might produce some insights.

From this point, you are looking for possible patterns, or unusual outliers, and considering possible actions to take. For example:

• Are there a lot of requests for a Wordpress file, but your site does not use Wordpress? These are coming from malicious traffic sources. It might be useful to set up a Dynamic Rule, e.g. to Ban requestors who submit more than one request for that file in a three-minute period.

• Is the same IP failing multiple challenges? It might be interesting to filter on that IP only, and go through all of its activity, to see what that traffic source was trying to do. (You can see that a challenge is being failed when the challenge itself appears in the logs, but it is not followed by a successful request by the IP for the same URI.)

• Are there many blocked requests for the same URI, but from different traffic sources? This might be a False Positive alarm. See below for more on this.

There is no set procedure for this process. Essentially, you are browsing the list of challenges or blocked requests, thinking about why you are seeing the entries there, and asking further questions.

Translating Encoding in Requests

Sometimes, request data will be encoded. Here's an example of an XSS attempt:

default.site/api/?a== %3c%73%63%72%69%70%74%3e%77%69%6e%64%6f%77%2e%6f%6e%6c%6f%61%64%20%3d%20%66%75%6e%63%74%69%6f%6e%28%29%20%7b%76%61%72%20%6c%69%6e%6b%3d%64%6f%63%75%6d%65%6e%74%2e%67%65%74%45%6c%65%6d%65%6e%74%73%42%79%54%61%67%4e%61%6d%65%28%22%61%22%29%3b%6c%69%6e%6b%5b%30%5d%2e%68%72%65%66%3d%22%68%74%74%70%3a%2f%2f%61%74%74%61%63%6b%65%72%2d%73%69%74%65%2e%63%6f%6d%2f%22%3b%7d%3c%2f%73%63%72%69%70%74%3e

In plain text, this is:

default.site/api/?a==window.onload = function() {var link=document.getElementsByTagName("a");link[0].href="http://attacker-site.com/";}]

Reblaze will decode many requests, but not necessarily all. For example, double-encoded requests will only have their first level of encoding undone.

In these cases, you can right-click on the string in the UI (for example, the Block Reason field), select Copy Value To Clipboard, and run it through decoding tools. (For example, http://0xcc.net/jsescape/.)

False Positive Alarms

A “False Positive” (FP) alarm occurs when a security system misinterprets a non-malicious activity as an attack. These errors are a critical issue for cybersecurity.

Although it might seem that FP errors do not necessarily have serious consequences, incorrect security alerts can lead to significant monetary losses. For example, an e-commerce site might incorrectly exclude real online shoppers. Or, it might reject "good" web crawlers, which would reduce its Internet visibility.

FP diagnosis is often done when Reblaze is first deployed, before it has been fine-tuned for the customer's applications. It can also be done later, when you discover that certain requests are being blocked, but you do not understand why.

Determining if a block is a FP

When examining blocked request(s), it can be helpful to ask questions such as the following.

Is Reblaze's configuration appropriate for the application?

Sometimes an application will expect and accept inputs that Reblaze blocks by default. Example: the site might use a CMS which accepts HTML/Javascript POST requests. However, out of the box, Reblaze is often configured to block requests containing POST. Thus, Reblaze would need to be re-configured to match the application it is protecting.

Has the web application been updated, without Reblaze being updated as well?

If the range of valid inputs has changed, but Reblaze was not re-configured, then FP errors can result.

Is the block happening to multiple users?

If a single IP is being blocked, but other requestors for the same URI are being allowed through, the block is more likely to be the correct response to this IP's request. This is especially true when the single IP has attempted other questionable activities.

Conversely, if multiple requestors are being blocked, and they seem to be innocuous otherwise, this indicates the problem might be a FP.

Is the block being done by Reblaze, or is it coming from elsewhere?

There are some situations where a request is blocked even though Reblaze has no obvious reason to be blocking it. This can occur when Reblaze is not actually the entity making the decision.

• The upstream server can reject requests. These requests can be displayed in the Events Log with result~"origin" as the filter.

• Reblaze can use external sources of information, e.g. Spamhaus. Out of the box, Reblaze includes a number of Global Filters based on threat intelligence feeds. Although a given traffic source might not be triggering any of Reblaze's explicitly-configured security settings, it can still be blocked based on factors such as these.

Is this a FP that resulted from incorrect user actions?

Example: a web user visited a landing page, entered contact details into a form, and then tried to proceed to the next page. However, the request to proceed was blocked.

This could be a FP due to "junk input." Perhaps the user entered a phone number of "1111111111" and it was rejected by the upstream application. Or perhaps the page itself autocompleted a field and inadvertently created junk input on its own.

Is this a FP that resulted from faulty or too-restrictive parameters within Reblaze?

If requestors are being blocked for violating rate limits, and the rate limits are very stringent, then perhaps the limits are too tight, and they need to be relaxed.

Or, perhaps the block is the result of content filtering. This feature is powerful, and it is possible to mistakenly configure it to be too restrictive.

Example: requests are being blocked because of a Content Filter Rule that contains this Match condition:

(?i)(select|create|drop|\<|>|alert)\W

The admin wrote this regex in order to identify SQL injection attempts (i.e, SELECT, CREATE, and DROP commands. Normally SQL and code injection attempts are blocked automatically by Reblaze, but occasionally customers choose to disable this).

Now let's examine one of the blocked requests in the Events Log. It contains this:

https://www.pinterest.com/pin/create/button/?url=https%3A%2F%2Fexample.com%2Fpitbull-2012%3Frt%3Dstorefront%26rp%3Dpitbull%26rn%3DPitbull%26s%3Dhanes-S04V%26c%3DBlack%26p%3DFRONT&media=&description=

This can be decoded with a tool such as http://0xcc.net/jsescape/. It now becomes this:

https://www.pinterest.com/pin/create/button/?url=https://example.com/pitbull-2012?rt=storefront&rp=pitbull&rn=Pitbull&s=hanes-S04V&c=Black&p=FRONT&media=&description=

Now let's see why the regex condition matched the request. On https://www.debuggex.com/, it's possible to paste in regex and a string, and see if/where a match occurs.

The result from deguggex.com

Notice the highlighted (in yellow) part of the string in the bottom textbox. This shows which part of the bottom string matches with the regex in the box above it.

We see that the expression that was meant to identify an SQL command of CREATE, is also matching with the URL generated by a user who was trying to feature this site's product on Pinterest.

This indicates that the regex should probably be modified, so that it only accomplishes its intended purpose. In this example, it could be modified as follows:

(?i)(\sselect|\screate|\sdrop|\<|>|alert)\W

This would require a space to be found before each potential SQL command, thus eliminating matches when those words are found in a URL.

Summary of FP Detection

As mentioned earlier, there is no set procedure for the process of identifying the underlying reasons why requests are blocked. It requires digging into the requests, gathering data, and asking questions.

Hopefully the examples above are helpful in illustrating the necessary thought process, and some of the tools that are available.