Firewall: Automatic update of blocklists (that block IP ranges from certain countries)

Background

Cloudron has a feature to block connections:

Using the blocklist configuration, one or more IP addresses and/or networks can be blocked from connecting to Cloudron.

Based on the discussion here, @imc67 came up with a great script to update such blocklists, whereas I only changed the script a little bit to meet my needs. Below you'll find the script with all comments to guide you through.

The basic idea:

• Save code as as script
• Modify blocklist urls to your liking
• Modify cloudron_api_endpoint and api_key with your values
• run the script daily via cron (for information on cron and crontab, see here).
• The script pulls updated lists from https://iplists.firehol.org/ and https://www.ipdeny.com/ipblocks/ and
• uploads the blocklists to Cloudron's firewall via the API

Caveats

Note: Blocking certain countries is only one small part of securing Cloudron but it can have a significant impact on the amount of spam and mail server abuse.

Note: If you block certain country IP ranges this way, it also means, you might not be able to receive mails from servers / vendors located in these countries! So blocking the US is probably a bad idea; if you do regular business e.g. with China, the same applies.

Improvements are welcome as there are certainly people out there with better bash skills!

Script

#!/bin/bash
# Save script in cd ~/urlfilter/, make executable, and run via cron e.g. daily, some time at night
#
# Variable for Current date and time
current_datetime=$(date +"%Y%m%d_%H%M%S")
#
# Location of script and the blocklist archive
cd ~/urlfilter/
#
# Array containing the URLs of the IP lists and their descriptions
# The first two from firehol.org are standard blocklists
# The next four are an example how to block connections from Afghanistan and Azerbaijan - first two are for IPv4 addresses, the next two for IPv6 addresses.
# In order to add countries, go to https://www.ipdeny.com/ipblocks/ to find out the relevant country names and abbreviations and add URLs accordingly on this scheme:
# IPv4:  "https://www.ipdeny.com/ipblocks/data/aggregated/[ABBREVIATIONS, small caps]-aggregated.zone,[ABBREVIATIONS, all caps] - [COUNTRY NAME]”
# IPv6:  "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/[ABBREVIATIONS, small caps]-aggregated.zone,[ABBREVIATIONS, all caps] - [COUNTRY NAME]”
# So for Brunei it would be:
# Abbreviation: bn / BN
# Country Name: BRUNEI DARUSSALAM
# i.e.  "https://www.ipdeny.com/ipblocks/data/aggregated/bn-aggregated.zone,BN - Brunei Darussalam"
#
declare -a urls=(
  "https://iplists.firehol.org/files/spamhaus_drop.netset,Spamhaus - Drop"
  "https://iplists.firehol.org/files/spamhaus_edrop.netset,Spamhaus - eDrop"
  "https://www.ipdeny.com/ipblocks/data/aggregated/af-aggregated.zone,AF - Afganistan"
  "https://www.ipdeny.com/ipblocks/data/aggregated/az-aggregated.zone,AZ - Azerbaijan"
  "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/af-aggregated.zone,AF - Afganistan"
  "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/az-aggregated.zone,AZ - Azerbaijan"
 )

# File name with the current date and time
output_file="merged_list_${current_datetime}.txt"

# Download and merge the IP lists
for url_info in "${urls[@]}"
do
    # Splitting the URL information
    IFS=',' read -r url description <<< "$url_info"
    # Add comment with the URL and description
    echo "# URL: $url" >> "$output_file"
    echo "# Description: $description" >> "$output_file"
    echo "Download IP list from $url"
    # Download the IP list and add it to the file
    curl -sS "$url" >> "$output_file"
done
echo "Merge completed! The merged list is stored in $output_file"
# Formatting the file for the Cloudron Blocklist API
formatted_file="formatted_$output_file"

# Add "\n" to the end of each line
awk '{printf "%s\\n",$0}' "$output_file" > "$formatted_file"
# Cloudron Blocklist API endpoint
cloudron_api_endpoint="https://my.yourdomain.com/api/v1/network/blocklist"

# API Key for authentication (replace 'your-api-key' with your API key)
api_key="#######"

# Upload to Cloudron Blocklist API with wget
echo "# Upload to Cloudron Blocklist API with wget..."

# The data in the required format for the API
data="{\"blocklist\":\"$(cat "$formatted_file" | tr '\n' '\\n')\"}"

# Send the file with a POST request via wget
echo "$data"> temp_data.txt
curl -X POST "$cloudron_api_endpoint" \
     -H "Content-Type: application/json" \
     -H "Authorization: Bearer $api_key" \
     --data-binary "@temp_data.txt" \
     --verbose \
     -o output.txt
# Delete temporary files
rm temp_data.txt
# Compress lists and create an archive of the last 10 updates 
7z a -mx9 "${current_datetime}.7z" "formatted_$output_file"
rm "formatted_$output_file"
rm "$output_file"
ls -td *.7z | grep -v '/$' | tail -n +10 | while IFS= read -r f; do rm -f "$f"; done

---

Authors:

• @necrevistonnezr
• @imc67

---