← Back to Documentation

Creating a Cloudflare WAF Rule to Skip Security Checks for NewsCrawler/2.0

A step-by-step guide for configuring Cloudflare WAF rules to allow Newstex's NewsCrawler bot.

Table of contents

Overview
Steps
Verification
Additional Resources
Notes

For platforms

License original, human-reviewed content at scale.

Request a demo

Overview

This guide explains how to create a Cloudflare WAF rule that skips security checks for requests from the NewsCrawler/2.0 user agent.

Steps

1. Access Cloudflare WAF Settings

  1. Log in to your Cloudflare dashboard
  2. Navigate to Security > WAF in the left sidebar
  3. Select Custom rules

2. Create a New Skip Rule

  1. Click Create rule
  2. Give your rule a descriptive name, e.g., “Skip WAF checks for NewsCrawler/2.0”

3. Configure the Rule Condition

In the When incoming requests match section, set up the condition:

  • Field: User Agent
  • Operator: equals
  • Value: NewsCrawler/2.0

4. Set the Action

In the Then section, select Skip from the dropdown. Choose which security features to skip:

  • Skip all security features (recommended for trusted crawlers)
  • Or select specific features to skip (e.g., Rate Limiting, Bot Management, etc.)

5. Deploy the Rule

  1. Click Deploy to activate the rule
  2. The rule will take effect immediately

Verification

To verify the rule is working:

  1. Make a request with the NewsCrawler/2.0 user agent
  2. Check the request logs in Cloudflare Analytics
  3. Confirm that WAF security checks are being skipped for matching requests

Additional Resources

Notes

  • Ensure that NewsCrawler/2.0 is a trusted source before skipping security checks
  • Monitor the rule’s impact on your security posture
  • Consider using more specific conditions if needed (e.g., combining with IP address restrictions)