+ Post New Thread
Results 1 to 2 of 2

Thread: Linux bash script to check if we do not host scams listed on Artists Against 419...

  1. #1
    Administrator Fli's Avatar
    Join Date
    Mar 2013
    Posts
    2,406
    Post Thanks / Like
    Blog Entries
    1

    Linux bash script to check if we do not host scams listed on Artists Against 419...



    Bash script to check Artists Against 419 and CLEAN MX .de webpages against websites hosted on local Linux Apache webserver & report via email if any is hosted.

    No guarantee provided that this script will work or will not damage anything on the server. For me worked.


    1. On the webserver, create folder /root/scamcheck/

    then inside, create file /root/scamcheck/webpagestoextract
    which will contain following 2 lines:

    Then create another file: /root/scamcheck/scamcheck
    with content:

    # This script working for me to check webpages that lists fake websites.
    # Then it check server apache virtualhost file to discover if any of the mentioned sites is not hosted on local server.
    # If is hosted, then notiffication is sent to an email address.

    # i saved this script as /root/scamcheck/scamcheck

    # then create file /root/scamcheck/webpagestoextract which contains URLs to be checked against scam sites, one url per line.

    # setup cronjob like:
    # */10 * * * * /bin/sh /root/scamcheck/scamcheck

    webpagestoextract=webpagestoextract
    suspiciousdomains=suspiciousdomains
    suspiciousdomainshosted=suspiciousdomainshosted
    hosteddomains=$(cat /etc/httpd/conf/httpd.conf | grep "ServerName" | tr -d " " | sed -e "s/ServerName//g")
    thisscriptdir=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
    adminmail=[email protected]

    # Empty suspiciousdomainshosted file if its not younger 4 months (is older), so if abuser start hosting same domain, im alerted again (number is in hours)
    find $thisscriptdir -name "suspiciousdomainshosted" ! -ctime -2880 -delete

    for webpage in $(cat $thisscriptdir/webpagestoextract);do
    #echo "-------------------------------------------------------------------------------------------"
    #echo "Download webpage $webpage content"
    #echo "Check if any of listed domains (suspicious domains) on that page are hosted on this server"
    #echo "-------------------------------------------------------------------------------------------"

    curl --silent "$webpage" | grep -ahoP 'http[[email protected]:%_\+.~#?&//=]{2,256}\.[a-z]{2,4}\b(\/[[email protected]:%_\+.~#?&//=]*)?' | grep -v "419" | grep -v "w3." | awk -F/ '{print $3}' | sed -e "s/http:\/\/www.//g" | sed -e "s/www.//g" | sed -e "s/http:\/\///g" | sort -u > $suspiciousdomains > /dev/null 2>&1

    #echo "Extracted suspicious domains:"
    #cat $suspiciousdomains
    #echo ""

    for suspiciousdomain in $(cat $suspiciousdomains);do
    # is suspicious domain $suspiciousdomain hosted?
    if [[ "$hosteddomains" == *"$suspiciousdomain"* ]];then
    # it is hosted. we already have it in suspiciousdomainshosted file (already been reported)?
    if [[ "$(cat $thisscriptdir/$suspiciousdomainshosted)" != *"$suspiciousdomain"* ]];then
    # this suspicious domain was not on file so its new! lets add it to the file and send an email alert
    echo "$suspiciousdomain" >> suspiciousdomainshosted && sort -u suspiciousdomainshosted

    #echo "Suspicious domains hosted:"
    #cat suspiciousdomainshosted
    #echo ""
    echo "New suspicious domain hosted on $(hostname). Source: $webpage

    $suspiciousdomain

    Check this domain and suspend it from hosting billing system if needed. This is an email from "scamcheck" script." | mail -s "New suspicious domain at $(hostname)" $adminmail
    fi
    fi
    done
    done
    # echo "Done, if no output, it means no new suspicious domain that is hosted at this server."
    Then if you are sure above script is working, chmod the script file 700, run it if you really want to (maybe with "set -ex" on the first line to test it) and setup a conjob like:
    */10 * * * * /bin/sh /root/scamcheck/scamcheck

    the script will run every 10 minutes to check if you do not host any scam webpages.

  2. #2
    Junior Member Ders's Avatar
    Join Date
    Jan 2016
    Posts
    2
    Post Thanks / Like


    Is this useful / helpfull? Yes | No
    A great initiative. Thank you. Yes, and it beats monitoring a RSS and hoping you don't miss out in a busy period.

    For aa419, a more correct approach would be:
    Similarly, if you're a network owner, you could check for your ASN, say 22612:
    This would allow you the benefit of only checking for active websites, you don't need to disable disabled websites (or those on hold,but you could monitor for holds similarly)

    This should only be needed once or twice a day. Obviously the website admins get a bit testy if you monitor 5 times a minute of like

+ Post New Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  
 Protected by : ZB BLOCK  &  StopForumSpam