Rate Limit Rules
Controlling the rates of incoming requests
Last updated
Controlling the rates of incoming requests
Last updated
There are many web threats that are built upon requests which otherwise might seem benign: credential stuffing, enumeration, payment card validation, coupon/gift code discovery, and more.
Rate limiting defends against these malicious activities by restricting the rates at which traffic sources can successfully submit requests. When a limit is exceeded, an action can be taken; for example, subsequent requests from that traffic source can be blocked for a defined period of time.
For more information, see the detailed discussion below of how rate limits are enforced.
A Rate Limit Rule can be enforced globally by enabling its Global mode setting. Otherwise, the Rule's scope will be the paths from its connected Security Policies. In both cases, enforcement is also subject to the Rule's Active mode setting and the constraints of its Include/Exclude filters.
If you define a Rate Limit Rule, but its Global mode setting is not enabled and the Rule is not connected to any Security Policies, it will not have any effect within your applications/APIs.
The main page lists all current Rate Limit Rules.
The administration (addition/deletion/editing/versioning) of these Rules follows the conventions described here.
Each Rate Limit Rule consists of the following:
Settings to define the scope (the Global mode, Active mode, Include/Exclude lists, and Connections to Security Policies). These specify the requests upon which this Rule will be enforced.
A rate limit (defined by the combination of Time Frame, Count by / Event, Number of events, and Time frame).
Action(s) to perform when a violation occurs (Action, and Ban mode)
General parameters for administration.
A name for this Rule, to be used within the Reblaze interface. An Automatic Tag (shown below the Tags field) will include it as well.
It can be helpful to use descriptive names based on the Rule's parameters. For example, a Rule that limits 50 requests per username, per 60 seconds, could be named "RL 50/60s username". This makes administration straightforward, and also clarifies events in the traffic logs.
When this setting is enabled, this Rate Limit Rule will be enforced globally (if the Global mode setting is enabled) or within the paths configured in all connected Security Policies, subject to the scope defined by the Include and Exclude lists. To deactivate the Rule, deselect this setting.
This can be helpful when a Rule is not performing as expected, and needs to be reconsidered. Rather than having to go find all the places where the Rule is being used, admins can simply deactivate it here, which will take effect across the entire platform.
When enabled, this Rule will be enforced globally, subject to the Active mode setting and the scope defined by the Include and Exclude lists.
Information about this Rule, for use within the interface.
These parameters tell the system how to count incoming events for rate limiting purposes. See "How rate limits are enforced," below.
The maximum allowable number of events within the Time frame.
The time period within which the Number of events limit is enforced, specified in seconds.
Frequently, an admin will want to constrain user activity within short periods of time (e.g., only allow three login attempts per minute). However, the optimal setting for Time frame will often be a much larger period of time. This is explained in more detail below: multi-tiered rate limiting.
An action to execute when the Number of events limit is exceeded within the Time frame. The available actions are defined on the Actions page. (Note that this can be, but doesn't need to be, a blocking action.)
Enables the blocking of a Rule violator for a longer time period than the Time frame. See discussion below: Ban mode.
A list of one or more tags, separated by spaces. These will be attached to requests from traffic sources that have exceeded a rate limit.
In addition to these admin-defined tags, the system also shows some Automatic Tags that will be attached as well.
By default, an active Rate Limit Rule will be enforced upon all requests globally (if the Global mode setting is enabled), or upon incoming requests targeting URLs to which this rule has been assigned via a Security Policy.
The Include and Exclude tag lists can be used to further constrain the enforcement of this Rule. Their use is described on the UI Overview page.
This list contains all of the Security Policies that use this Rate Limiting Rule, and allows the Rule to be connected to additional Security Policies.
Above, Rate Limit Rules were described as if the system will count incoming requests, while monitoring the count to see if it has exceeded the Number of events. This is true sometimes, but not always.
A more accurate description is that a Rate Limit Rule counts events, as specified in the Event field. Frequently, this field will be set to HTTP request, which means that indeed, an internal counter will be incremented every time a request is received. However, there are more powerful ways to configure a Rule beyond this.
To illustrate how Rate Limit Rule parameters can be used, we'll discuss several examples. We'll start with simple use cases and finish with more complicated situations.
The examples below are for individual Rate Limit Rules. Note that It's possible for multiple Rules to be in-scope simultaneously, e.g. when several Rules have their Global setting enabled. When a request is received, all applicable Rate Limit Rules are evaluated. If it exceeds the thresholds of multiple Rules, Reblaze will compare their Actions and execute the one that has the highest priority.
Example 1: Let's say an admin wants to define a rate limit as follows:
"Allow each unique IP address to send up to three requests to /login. If more than three requests are sent within one minute, block them."
To configure this, the following parameters would be set:
Time frame: 60 seconds
Count by: Attribute / IP Address
Event: HTTP request
Number of events: 3
...and then this Rate Limit Rule would be connected to a Security Policy that includes /login in its Path Mapping.
As shown in the example above, Reblaze maintains an internal counter for every unique value of the Count by parameter(s).
In the example, a separate counter is maintained for each IP address that is sending traffic. When a request is received from an IP, that IP's counter is incremented, and the system checks to ensure that the counter's value hasn't exceeded the Number of events.
Each counter is reset to zero once the Time frame has passed (starting when the first request was received).
The Count by parameter can be multifactorial. In other words, admins can select "New Entry" and add additional Count by settings. When this is done, an internal counter is maintained for every unique combination of these settings.
Example 2: an admin configures the Count by section with two lines: Attribute / IP Address and Header / user_id. When the first request is received, an internal counter is created (set to a value of one) for this unique combination of IP and user_id. A second request is then received, originating from the same IP and from the same user; this causes the internal counter to be incremented up to two. A third request is then received from the same IP but with a different user_id; this causes a new internal counter to be created (and set to a value of one) for this combination.
Example 3: multiple Count by settings can be used to create site-level rate limiting. Let's say that there are three sites: www.example.com, api.example.com, and mobile.example.com. If the Count by section has Attribute / Session ID and Header / host, then the system will enforce separate rate limit counts for each site.
Example 4: an admin wants to rate limit the activity within each user session, regardless of the IP that is being used. For this purpose, the admin can set the Count by parameter to Attribute / Session ID.
Note that the composition of a Session ID might vary, according to context. Therefore, the system allows admins to define it within Security Policies, as explained here.
Each time a request is received, the system examines its destination's path to see which Security Policy applies. That Policy's definition of Session ID is used for any active Rate Limit Rules connected to it.
If a Rate Limit Rule is "Active" and "Global", but is not connected to a Security Policy, then the system will use the requestor's IP Address as the default Session ID for rate limiting purposes. In this situation, if actual session-based rate limiting is needed, then admins will need to specify the appropriate combination of headers, cookies, arguments, etc. manually in the Count by section.
As mentioned earlier, the rate limiting system does not directly count requests; instead, it counts events.
The four examples above have the Event field set to its default of HTTP request. This means that the receipt of a request is an event, and therefore, each request will cause an internal counter to be incremented.
Other Event settings are possible; admins can specify a header, cookie, argument, or attribute. When this is done, an internal counter is maintained for each Count by value, and incremented each time a new, previously unobserved Event value is encountered in a request.
Therefore, if an Event is defined as something other than HTTP request, the Rate Limit Rule constrains the number of allowable Event values for any given Count by value.
Example 5: Let's say we want to allow an individual user to login from a maximum of two ASNs within one hour. (Perhaps the user is accessing our web application from a coffee shop's WiFi, and then a few minutes later, leaves the coffee shop and begins using the cell network instead.) We want to allow this possibility; however, if we receive requests from the same user originating from three or more ASNs within an hour, we want to treat this traffic more suspiciously.
This is not possible merely by specifying two Count by conditions, as described above in the "Multiple Count by settings" section. If we set up two conditions (Argument / Username and Attribute / Organization) with a Number of events of 2, and assign it to our login form, then this will merely limit the number of times that the user can login from each ASN within an hour.
Instead, we set up one Count by condition (Argument / Username), and then set the Event to Attribute / Organization. Now the Number of events will apply to the number of Organizations that are observed for each specific Username.
Example 6: Your Reblaze deployment includes some Rate Limit Rules out of the box. One of them might be a global Rule with settings similar to these:
Time frame 3600
Count by Cookie / rbzid
Event Attribute / IP Address
Number of events 5
Reblaze sets a cookie (rbzid) for all users who interact with it. This can serve as a global identifier for users, across all domains, sites, applications, etc., even if individual sites/apps track users with different methods (whether user id, authentication tokens, and so on).
The example Rule shown above allows any particular user to have up to five IP addresses within an hour. When a sixth IP is encountered, it will execute the Action.
A common use case for Rate Limit Rules is the following:
Allow incoming requests until the Number of events has been reached.
Each subsequent request within the same Time frame will trigger the Action.
However, it is often necessary to create multiple tiers of Rate Limiting:
Allow the requests until the first Number of events is reached.
Each subsequent event will trigger the first tier's Action, until the second Number of events is reached.
Each subsequent event will trigger the second tier's Action, until the third Number of events is reached.
And so on.
For this purpose, multiple Rate Limit Rules can be created and set to be active within the same scope. This allows for increasingly restrictive actions to be taken as the number of events within the same Time frame increases.
Example 7: one possible use of multi-tier Rate Limits is to allow a certain amount of activity, and then verify that the user is human before allowing the activity to continue, while still enforcing reasonable boundaries. To do this, the Rules might look like this:
Number of events 3, Action monitor [tag only]
Number of events 7, Action challenge
Number of events 10, Action default blocking
The system would behave as follows:
The first three events would be allowed.
Events 4-7 would be allowed, but the requests would be tagged.
Event 8 would subject the user to a bot challenge. If the challenge is failed, subsequent requests would be blocked.
Events 9 and 10 would be allowed.
Subsequent events would be blocked.
Earlier, it was mentioned that the optimal Time frame for a Rate Limit Rule might be quite long. Multi-tier Rate Limits are the reason for this; the Time frame must be long enough to accommodate all of the previous tiers.
When its Time frame expires, a rate limit will reset, and its internal counter will be set to zero.
This means that the rate limiting described above cannot, on its own, fully defeat persistent attackers.
Example: Access to a login form is rate-limited to four requests per minute. An attacker tries to brute-force the login, and sends 60 requests per minute. The Rate Limit Rule allows the first four requests, but then blocks the next 56 requests. However, after the minute has passed, the rate limit resets. The attacker is allowed another four attempts before being temporarily blocked again. This cycle can continue for as long as the attacker wishes. In effect, the rate limit is not preventing the attack; it is merely slowing it from 60 attempts per minute down to four attempts per minute.
To accomplish this, Reblaze provides two capabilities: Ban mode, and Dynamic Rules.
Both allow admins to define granular Rate Limit Rules with short time frames and maintain precise control of various situations, while still being able to block persistent violators for much longer periods of time.
When Ban mode is enabled, the Ban duration field appears. These two values allow admins to configure a longer-term blocking action for rate limit violators.
When a Rate Limit Rule is violated (i.e., an event occurs that exceeds the Number of Events), and Ban mode is enabled, the offending traffic source is banned immediately.
Beginning with the request that triggered the ban, Reblaze will reject all activity from the offending traffic source, for the length of time specified in Ban duration.
A Dynamic Rule is similar to a Rate Limit Rule with Ban mode enabled. It too defines an allowable threshold of events that can occur within a certain period of time. If that threshold is violated, an action occurs.
However, there are some important differences:
When configuring a Dynamic Rule, an admin can choose to block violators, but can also choose a non-blocking Action.
If blocking is selected, the blocking can be offloaded to Link11's Layer 3 protection.
Dynamic Rules are enforced globally, unless their scope is otherwise constrained.
Violators of a Dynamic Rule will appear in the Quarantined list. Admins can manually remove them from this list (which ends the quarantine early) if desired.
