Bob,

When try to get the emerging-all.rules file at http://www.emergingthreats.net/rules/emerging-all.rules I get a 404 Not found error . Have they changed the paths and do you have an updated script?

Kevin

Yes they've had a reshuffle in line with recent changes (ET now offer a complete ruleset change over as an alternative to the Snort ruleset)...
http://rules.emergingthreats.net/

http://rules.emergingthreats.net/open/snort-2.8.4/emerging-all.rules

Thanks Tim.. Script downloads the rules now. Snort fails on restart with

FATAL ERROR: Rule-Threshold-Parse: could not create a threshold object -- only one per sid, sid = 2002383

I don't see anything in my Threshold.conf file. Where else do I need to look?

I also get the ClearSDN Intrusion Prevention Updates, will me using the ET rules affect the ClearSND updates or will it work with them?

Here's the latest script I've written. I just tested it on a freshly installed and patched ClearOS Virtual Machine and it appears to work correctly. Note the "DEBUG_ON" variable, if it is set to 1, lots of stuff will get printed to the console when the script is run, else not much at all.

Let me know how it works for you,

Bob

#!/bin/bash



# set DEBUG to 1 if you want debugging info

DEBUG_ON=1



# script variables

SNORT_HOME=/etc/snort/

SNORT_INIT_SCRIPT=/etc/init.d/snort

ET_RULES_FILE="emerging-all.rules"

RULES_ARCHIVE_DIR=/etc/snort/rules_archive/

ET_RULES_URL="http://rules.emergingthreats.net/open/snort-2.8.4/emerging-all.rules"

ERR="ERROR:"

DEBUG="DEBUG:"

INF="INFO:"



if [ -d $SNORT_HOME ]; then

	cd $SNORT_HOME

	if [ ! -d $RULES_ARCHIVE_DIR ]; then

		mkdir $RULES_ARCHIVE_DIR

	fi

else

	if [ $DEBUG_ON == 1 ]; then

	echo;echo "$DEBUG$ERR Target directory \"$SNORT_HOME\" does not exist exiting script"

	exit

	fi

fi



if [ $DEBUG_ON == 1 ]; then

	echo;echo "$DEBUG Current directory is: $PWD"

fi



if [ -f $SNORT_HOME$ET_RULES_FILE ]; then

	if [ $DEBUG_ON == 1 ]; then

		echo;echo "$DEBUG Ruleset file $ET_RULES_FILE found, moving to $RULES_ARCHIVE_DIR"

		mv $SNORT_HOME/$ET_RULES_FILE $RULES_ARCHIVE_DIR

	fi

else

	if [ $DEBUG_ON == 1 ]; then

	echo;echo "$DEBUG Existing rules file $ET_RULES_FILE does not exist"

	fi

fi



if [ $DEBUG_ON == 1 ]; then

	echo;wget $ET_RULES_URL --output-document $SNORT_HOME$ET_RULES_FILE

else

	echo;wget $ET_RULES_URL -q --output-document $SNORT_HOME$ET_RULES_FILE

fi



# MD5 verification TBD on next iteration of script

#wget http://rules.emergingthreats.net/open/snort-2.8.4/rules-md5.txt



if [ -f $SNORT_HOME$ET_RULES_FILE ]; then

	if [ $DEBUG_ON == 1 ]; then

        echo;echo "$DEBUG Download appears to have succeeded, $ET_RULES_FILE is present in $SNORT_HOME"

	fi

else

	if [ $DEBUG_ON == 1 ]; then

        echo;echo "$DEBUG$ERR $ET_RULES_FILE is not present in $SNORT_HOME, exiting script"

	exit

	fi

fi



if [ $DEBUG_ON == 1 ]; then

	echo;echo "$DEBUG Current contents of $SNORT_HOME which contains $ET_RULES_FILE:"

	ls -la $SNORT_HOME

fi



if [ -f $SNORT_INIT_SCRIPT ]; then

	echo;echo "$INF Rules downloaded successfully, restarting snort service:";$SNORT_INIT_SCRIPT restart

else

	echo;echo "$DEBUG$ERR $SNORT_INIT_SCRIPT does not exist exiting script"

	exit

fi



echo

Edit: It looks like my paste of the script is cutting off the end of some of the lines, and I am failing on attaching the script to this post, so here's a link to the script.

It looks interesting. I've tried it but I hit a glitch. I presume to include the file I need to add a line to /etc/snort.conf

include $RULE_PATH/emerging-all.rules

Restarting snort fails as there is now a duplicate sid (2002383) which also appears in the ftp.rules file. Is the emerging threats rule set meant to entirely replace the ClearOS one and if not what are you supposed to do about duplicate sid's?

Hi Nick, this has recently changed - Emerging Threats (ET) use to run a complementary rule set you can could safely add. Now they offer a complete replacement for Snort rules. This results in clashes with the existing ClearOS SID's

You can still download the complementary rules but as individual files from the Snort 2.8.4 folder
http://rules.emergingthreats.net/open/snort-2.8.4/rules/

The alternative is to comment out all the other #include lines from /etc/snort.conf and use only ET, but there are so many community added rules and others to the ClearOS set I don't know if you'd be missing out

Sorry here (containing the block fwsam addition so they get added to the IPS)
http://rules.emergingthreats.net/blockrules/

Nick,

Yeah, I found that too. Still learning about the overlap in rules. I need to think about how to deal with this in my script. Tim, thanks for taking the time to provide input on this. Looks like I'll need to work on the script some more...

Bob

What makes it more tricky is that ClearOS already contains many of the ET scan rules, although I have been able to add the trojan, virus, malware, botcc and compromised host rules without problems

It would be a bit of a hack which I would have to think about, but depending on the size allowed for the sid field, you add a fixed offset to the sid (say 9,000,000,000) to make the sid unique. You would need to test for the largest sid in the ClearOS rules. The largest sid in the EM rules is currently 100,000,908 (commas inserted to aid clarity!). This will shift the EM rules out of the range of the ClearOS rules.

You would end up with common rules duplicated and, Tim, I think you did something to hotlink the rules in the webconfig. I don't know if it would break that.

I would have to read up about sed and awk to see how to do it.

[edit]added reference to awk[/edit]

The webconfig rule hack just reads the SID from /etc/snort/sid-msg.map which is a long list of all existing SID's - if you add more you have to recreate this file using a script create-sidmap.pl but that's straight forward enough (script obtained from Oinkmaster sources I think)

Can we pick and choose which rules we want in the script? Perhaps have the capabilities to add multiple files at the ET_RULES_FILE line?

ET_RULES_FILE="emerging-all.rules" 

I've had this thread in the back of my mind for ages and I've been playing, learning a bit of sed and awk and cribbing and I've come up with this:

#!/bin/bash



# set DEBUG to 1 if you want debugging info

DEBUG_ON=0



# script variables

SNORT_HOME=/etc/snort/

SNORT_INIT_SCRIPT=/etc/init.d/snort

ET_RULES_FILE="emerging-all.rules"

RULES_ARCHIVE_DIR=/etc/snort/rules_archive/

ET_RULES_URL="http://rules.emergingthreats.net/open/snort-2.8.4/emerging-all.rules"

ERR="ERROR:"

DEBUG="DEBUG:"

INF="INFO:"



if [ -d $SNORT_HOME ]; then

	cd $SNORT_HOME

	if [ ! -d $RULES_ARCHIVE_DIR ]; then

		mkdir $RULES_ARCHIVE_DIR

	fi

else

	if [ $DEBUG_ON == 1 ]; then

		echo;echo "$DEBUG$ERR Target directory \"$SNORT_HOME\" does not exist exiting script"

		exit

	fi

fi



if [ $DEBUG_ON == 1 ]; then

	echo;echo "$DEBUG Current directory is: $PWD"

fi



cd $RULES_ARCHIVE_DIR

rm $ET_RULES_FILE



if [ $DEBUG_ON == 1 ]; then

	echo;wget $ET_RULES_URL --output-document $RULES_ARCHIVE_DIR$ET_RULES_FILE

else

	echo;wget $ET_RULES_URL -q --output-document $RULES_ARCHIVE_DIR$ET_RULES_FILE

fi



# MD5 verification TBD on next iteration of script

#wget http://rules.emergingthreats.net/open/snort-2.8.4/rules-md5.txt



if [ -f $RULES_ARCHIVE_DIR$ET_RULES_FILE ]; then

	if [ $DEBUG_ON == 1 ]; then

       	 echo;echo "$DEBUG Download appears to have succeeded, $ET_RULES_FILE is present in $RULES_ARCHIVE_DIR"

	fi

else

	if [ $DEBUG_ON == 1 ]; then

       	echo;echo "$DEBUG$ERR $ET_RULES_FILE is not present in $RULES_ARCHIVE_DIR, exiting script"

		exit

	fi

fi



if [ $DEBUG_ON == 1 ]; then

	echo;echo "$DEBUG Current contents of $RULES_ARCHIVE_DIR which contains $ET_RULES_FILE:"

	ls -la $SNORT_HOME

fi



sed '/^$/d; s/^[ \t]*//; /^#/d' $RULES_ARCHIVE_DIR$ET_RULES_FILE | awk '{ linetext=$0

	pos1 = index(linetext, "sid:")

	pos1 = pos1+4

	endtext = substr(linetext, pos1)

	pos2 = index(endtext, ";")

	sid = (substr(linetext, pos1, pos2 - 1) + 800000000)

	newtext = substr(linetext,1,pos1-1) sid substr(endtext, pos2)

	print newtext }' > $SNORT_HOME$ET_RULES_FILE



if [ -f $SNORT_INIT_SCRIPT ]; then

	echo;echo "$INF Rules downloaded successfully, restarting snort service:";$SNORT_INIT_SCRIPT restart

else

	echo;echo "$DEBUG$ERR $SNORT_INIT_SCRIPT does not exist exiting script"

	exit

fi

I've used Bob's original script and made some changes:
1 - Download the file straight into the archive directory.
2 - Use sed to remove blank lines, left align what is left and then remove all lines beginning "#" i.e. comments
3 - Piped the result into awk to identify the rule number and add 800,000,000 to it to force all rule numbers to be unique
4 - Put the resulting 4MB file into /etc/snort

You still need to add the line:

# Emerging threats

include $RULE_PATH/emerging-all.rules

I've no idea how good this is as many of the rules are now duplicated as some of the ET rules are used in the ClearOS base rule set and I don't know the effect of snort running with duplicate rules (apart from the rule number).

You could put the script in /etc/cron.daily or /etc/cron.weekly if you want it to run regularly.

Feedback would be welcome.

Great... Can't wait to give it a try.. Will let you know.

Thanks :cheer:

Because of the way the script has been written it will e-mail you a report every time it runs. This can be stopped by changing the following lines:

echo;wget $ET_RULES_URL -q --output-document $RULES_ARCHIVE_DIR$ET_RULES_FILE

to

wget $ET_RULES_URL -q --output-document $RULES_ARCHIVE_DIR$ET_RULES_FILE

and

echo;echo "$INF Rules downloaded successfully, restarting snort service:";$SNORT_INIT_SCRIPT restart

to

$SNORT_INIT_SCRIPT restart > /dev/null

It would also be safer to delete the downloaded file before you download the new one or you defeat some of the error checking. After

cd $RULES_ARCHIVE_DIR

add a line

rm $ET_RULES_FILE

I've edited my script to delete the downloaded file.

It's cool to see you guys working on making this script better.

I've chosen to use the ET rules exclusively on some of my systems as of late.

On these systems, I just have 2 rules files active in the snort.conf, the ET file, and local.rules so I can do site specific filters.

It would be cool if at some point we could get the ClearOS default rules and the ET rules to meld cleanly...

That is beyond me. When looking at the sed and awk documentation I saw a snippet to remove duplicates. Apparently it is very memory intensive and if Clear make a minor edit to some of the description then the lines become non-identical but share the same sid which will kill snort, or at least stop it from starting.

Perhaps the best approach would be to assemble all the rules into a single file, hack the bit of my script which picks out the sid and add an extra column to the beginning of the assembled file with the sid, sort on this column then drop any rule with a duplicate sid, then strip this prepended sid column. If you're really fancy you could test if the rules sharing the same sid are the same and if not change the sid rather than drop the line. To be honest though, I'm not going to try to do it.

Hello,

i also did some juggeling with the ET rules some time ago.
What i tried to do was add my own rules without breaking the Clearos given defaults.
I found that just adding a line like "include $RULE_PATH/emerging-all.rules" might be lost if you use the Webconfig to enable or disable rules later on, or an update gets installed.

So what i did is the following:
In the snort.conf file after the "var RULE_PATH xxxxxx" line i added a new line like "var PRIVATE_PATH yyyyyyyy".
Then before the section with the "include $RULE_PATH/xxxxxxx" i added a section with my own references consisting of any number of "include $PRIVATE_PATH/yyyyyy" statements.

This way even Clearos updates and modifications with webconfig will not break my own rules.

Cheers
Axel

...

####################################################################

# Step #6: Customize your rule set

#

# Up to date snort rules are available at http://www.snort.org

#

# The snort web site has documentation about how to write your own custom snort

# rules.



#=========================================

# Include all relevant rulesets here

#

# The following rulesets are disabled by default:

#

#   web-attacks, backdoor, shellcode, policy, porn, info, icmp-info, virus,

#   chat, multimedia, and p2p

#

# These rules are either site policy specific or require tuning in order to not

# generate false positive alerts in most enviornments.

#

# Please read the specific include file for more information and

# README.alert_order for how rule ordering affects how alerts are triggered.

#=========================================

# Local Rules

include $LOCAL_PATH/malware-user-agents.rules

include $LOCAL_PATH/emerging-drop-BLOCK.rules

#include $LOCAL_PATH/emerging-p2p.rules

include $LOCAL_PATH/emerging-tor-BLOCK.rules

include $LOCAL_PATH/blacklist.rules

include $LOCAL_PATH/botnet-cnc.rules

include $LOCAL_PATH/phishing-spam.rules

include $LOCAL_PATH/specific-threats.rules

include $LOCAL_PATH/spyware-put.rules

include $LOCAL_PATH/web-activex.rules

#include $LOCAL_PATH/policy.rules

#=========================================

include $RULE_PATH/attack-responses.rules

include $RULE_PATH/backdoor.rules

include $RULE_PATH/dns.rules

...

Does this get round the requirement for separate sid's? If not, could you not have just used the local.rules file?

It was some time ago that i did that, so i am not really sure, but i seem to remember that after some kind of update my local.rules file was suddenly empty.
So i decided not to use local.rules any more.
Also, using a second path variable, i can use as many personal files as i want to.
But you are right, it still requires seperate sid's.

Cheers
Axel

Nick Howitt wrote:

To be honest though, I'm not going to try to do it.

Famous last words and a challenge.

This is now obsolete. Please use this version of the script

I've come up with a script which can combine the ET rules with the ClearOS rules and remove any duplicates. As posted it is set up to combine the emerging-all.rules with all the rules which are enabled by default in ClearOS except the local.rules. To use it, paste the script below into a file in something like cron.weekly. In /etc/snort.conf, disable all include lines at the end except for the local.rules and add a line

include $RULE_PATH/combined.rules

This is the script:

#!/bin/bash



# The purpose of this script is to download any Emerging Threat files you want and combine them with

# any existing ClearOS snort rules you want. It removes any rules with duplicate sid's leaving the

# one with a higher revision number

#

# In order to use the script, you need to add a line "include $RULE_PATH/combined.rules" to /etc/snort.conf

# and disable any includes there where you are combinining the files in this script

#



# script variables

SNORT_HOME=/etc/snort/

RULES_DOWNLOAD_DIR=/etc/snort/ET-downloads/



# This function 

# - removes leading spaces, comments and blank lines

# - picks out the sid and revision number for each rule then it sticks the rule number, revision number

#   source file name and line number line number and puts them infront of the rule

# - It then output appends to rule to the temporory file



function indexrules {

	RULEFILE=$1

	sed '/^$/d; s/^[ \t]*//; /^#/d' $1 | awk -v FILE=$RULEFILE '{ linetext=$0

	pos1 = index(linetext, "sid:")

	pos1 = pos1+4

	endtext = substr(linetext, pos1)

	pos2 = index(endtext, ";")

	sid = substr(linetext, pos1, pos2 - 1)

	gsub(/ */, "", sid)



	pos3 = index(linetext, "rev:")

	pos3 = pos3+4

	if (pos3 < pos1)

		rev=0

	else

		{

		endtext = substr(linetext, pos3);

		pos4 = index(endtext, ";");

		rev = substr(linetext, pos3, pos4 - 1);

		}

	gsub(/ */, "", rev)

	newtext = sid " " rev " " FILE " " NR " " linetext

	print newtext }' >> $RULES_DOWNLOAD_DIR"indexed.tmp"

}



# This function

# - deletes the old rule file you are about to download

# - downloads the new file

# - calls the above function to build the temp file



function getrules {

	ET_RULES_URL="http://rules.emergingthreats.net/open/snort-2.8.4/"

	cd $RULES_DOWNLOAD_DIR

	rm -f $1

	wget -q $ET_RULES_URL$1

	if [ ! -e $1 ]; then

		echo;echo "$1 not downloaded. Exiting script"

		exit

	fi

	indexrules $1

}



# Create the download directory if it does not exist



if [ -d $SNORT_HOME ]; then

	cd $SNORT_HOME

	if [ ! -d $RULES_DOWNLOAD_DIR ]; then

		mkdir $RULES_DOWNLOAD_DIR

	fi

fi



# Clear the temp file



rm -f $RULES_DOWNLOAD_DIR"indexed.tmp"



# Download the rules from Emerging Threats

# Note that you can download some or all of the individual rule sets of you want instead.

# To do that change ET_RULES_URL to http://rules.emergingthreats.net/open/snort-2.8.4/rules/

# in the getrules function then repeat the line below as many times as you want changing the

# file name you want to download



getrules emerging-all.rules



# Append all the current rules to the temp file

# The list below is all the files in ClearOS 5.2 except local.rules

# I have also indicated which are the ones enabled in /etc/snort.cong by default

# I have not included local.rules as they are never downloaded from ET and if you add a local rule manually

# you would have to remember to run this script before the changes took effect instead of simply doing a

# service snort restart

###############################################################

#                                                             #

#                     IMPORTANT                               #

#                                                             #

# Any rules enabled below MUST be disabled in /etc/snort.conf #

#                                                             #

###############################################################



cd $SNORT_HOME

indexrules attack-responses.rules	# rule set used by default

indexrules backdoor.rules		# rule set used by default

indexrules bad-traffic.rules	# rule set used by default

#indexrules chat.rules

indexrules ddos.rules		# rule set used by default

#indexrules deleted.rules

indexrules dns.rules			# rule set used by default

indexrules dos.rules	# rule set used by default

#indexrules experimental.rules

#indexrules exploit.rules

indexrules finger.rules		# rule set used by default

indexrules ftp.rules	# rule set used by default

#indexrules icmp-info.rules

indexrules icmp.rules		# rule set used by default

indexrules imap.rules		# rule set used by default

#indexrules info.rules

#indexrules malware-user-agents.rules

indexrules misc.rules		# rule set used by default

#indexrules multimedia.rules

indexrules mysql.rules		# rule set used by default

#indexrules netbios.rules

indexrules nntp.rules		# rule set used by default

indexrules oracle.rules		# rule set used by default

#indexrules other-ids.rules

#indexrules p2p.rules

#indexrules policy.rules

indexrules pop2.rules		# rule set used by default

indexrules pop3.rules		# rule set used by default

#indexrules porn.rules

indexrules rpc.rules			# rule set used by default

indexrules rservices.rules		# rule set used by default

indexrules scan.rules		# rule set used by default

#indexrules shellcode.rules

#indexrules smtp.rules

indexrules snmp.rules		# rule set used by default

#indexrules sql-injection.rules

indexrules sql.rules			# rule set used by default

indexrules telnet.rules		# rule set used by default

indexrules tftp.rules		# rule set used by default

#indexrules virus.rules

indexrules web-attacks.rules	# rule set used by default

indexrules web-cgi.rules		# rule set used by default

indexrules web-client.rules		# rule set used by default

indexrules web-coldfusion.rules	# rule set used by default

indexrules web-frontpage.rules	# rule set used by default

indexrules web-iis.rules		# rule set used by default

indexrules web-misc.rules		# rule set used by default

indexrules web-php.rules		# rule set used by default

indexrules x11.rules			# rule set used by default



cd $RULES_DOWNLOAD_DIR



# This next bit:

# - sorts by sid and revision in descending order then

# - pipes it into awk which peints the first line of any sid only (therefore the one with the highest revision then

# - pipes it into sort which sorts by the original file name and original line number so the lines from any file

#   go back to the original order they were in then

# - pipes it into awk again which strips out the sid, revision number, original file name and original line number

#   fields which were added earlier to facilitate all the file sorting

# - the result is output to a file called combined.rules



sort -nr -k 1,2 indexed.tmp | \

	awk '{ if ($1 != oldtext) { line = $0; print line } oldtext = $1 }' | \

	sort -k 3,3 -k 4,4n | \

	awk '{ sub(/^[[:digit:]]* [[:digit:]]* .*\.rules [[:digit:]]* /, ""); print $0 }' > $SNORT_HOME"combined.rules"



# Remove the temp file



rm -f $RULES_DOWNLOAD_DIR"indexed.tmp"



# restart snort



service snort restart > /dev/null

It is easy to modify the code to pick and combine individual Emerging Threat files rather the emerging-all.rules but it is important that if the script is combining ClearOS files the same files are not loaded from the "include" lines in /etc/snort.conf.

I am pretty basic at script writing so any improvements/tips would be appreciated, especially with the awk stuff.

[edit] I wish this forum would display tabs in code blocks. It looks pants as posted but is is indented. [/edit]

[edit2]
Since posting this Emerging Threats has used an extra variable in their rules. To make the ET rules work with the ClearOS version of snort you need to make a one line change to snort.conf.
[/edit2]

Nice! Thanks Nick

Nick,

I would like to add the logging/email feature back in to the script. How would I do this?

Thanks

Looking at the script, I think you will get an e-mail when the wget completes if you remove the "-q" switch from the line with wget. You may get an e-mail anyway if the download fails from the line two down from the wget. Without testing again I can't remember. It is easy to test by giving it an invalid url to download from and put the script in cron.hourly. You will also get an e-mail when snort restarts if you change the line from "service snort restart > /dev/null" to "service snort restart".

What other logging would you like?

One way of adding logging into /var/log/messages is to add lines like:

logger -t My custom message

I have only used this once before and had problems if my message contained key words. There must be a way of quoting the message to avoid the issue.

[edit]
Note that you only get e-mail messages when cron runs the script. If you run it manually the messages only go on screen.
[/edit]

Hi Nick,

Decided to give this a go today. (Greast work BTW).

It worked flawlessly until I included other rules... see below

indexrules attack-responses.rules # rule set used by default
indexrules backdoor.rules # rule set used by default
indexrules bad-traffic.rules # rule set used by default
#indexrules chat.rules
indexrules ddos.rules # rule set used by default
#indexrules deleted.rules
indexrules dns.rules # rule set used by default
indexrules dos.rules # rule set used by default
#indexrules experimental.rules
indexrules exploit.rules
indexrules finger.rules # rule set used by default
indexrules ftp.rules # rule set used by default
indexrules icmp-info.rules
indexrules icmp.rules # rule set used by default
indexrules imap.rules # rule set used by default
indexrules info.rules
indexrules malware-user-agents.rules
indexrules misc.rules # rule set used by default
#indexrules multimedia.rules
indexrules mysql.rules # rule set used by default
indexrules netbios.rules
indexrules nntp.rules # rule set used by default
indexrules oracle.rules # rule set used by default
indexrules other-ids.rules
#indexrules p2p.rules
#indexrules policy.rules
indexrules pop2.rules # rule set used by default
indexrules pop3.rules # rule set used by default
#indexrules porn.rules
indexrules rpc.rules # rule set used by default
indexrules rservices.rules # rule set used by default
indexrules scan.rules # rule set used by default
indexrules shellcode.rules
indexrules smtp.rules
indexrules snmp.rules # rule set used by default
indexrules sql-injection.rules
indexrules sql.rules # rule set used by default
indexrules telnet.rules # rule set used by default
indexrules tftp.rules # rule set used by default
indexrules virus.rules
indexrules web-attacks.rules # rule set used by default
indexrules web-cgi.rules # rule set used by default
indexrules web-client.rules # rule set used by default
indexrules web-coldfusion.rules # rule set used by default
indexrules web-frontpage.rules # rule set used by default
indexrules web-iis.rules # rule set used by default
indexrules web-misc.rules # rule set used by default
indexrules web-php.rules # rule set used by default
indexrules x11.rules # rule set used by default

I was getting errors messages like HOME_NETany varible not defined... which was fixed by putting a space between HOME_NET and any... but I did encounter a few more errors as well... I just deleted them.

working now. but just in case it was not my end. letting you know.

edit: some that I remember included rcttp$HOME_NET, also one in the malware-user-agent.rules...happy to re-do and post

I don't know why you were getting the $HOME_NET error as it is not something I touched. Another error appeared when I updated ET over the weekend (I do it in cron.weekly) with an undefined variable $SSH_PORTS used in the new rule set. See here for my report and the solution.

To save my eyes, which were the extra rules you included?

If you get any other errors, if you post them I can look into them.

Sure.... I'll do that later tonight (as I am on my way out the door)...

Basically the things that I did included.

going into /etc/snort.conf and placing portvar SSH_PORTS 22 in there

I then commented out the existing .rules but placed include $RULE_PATH/combined.rules in there.

on first go of running your script I had these

indexrules backdoor.rules # rule set used by default
indexrules bad-traffic.rules # rule set used by default
#indexrules chat.rules
indexrules ddos.rules # rule set used by default
#indexrules deleted.rules
indexrules dns.rules # rule set used by default
indexrules dos.rules # rule set used by default
#indexrules experimental.rules
#indexrules exploit.rules
indexrules finger.rules # rule set used by default
indexrules ftp.rules # rule set used by default
#indexrules icmp-info.rules
indexrules icmp.rules # rule set used by default
indexrules imap.rules # rule set used by default
#indexrules info.rules
#indexrules malware-user-agents.rules
indexrules misc.rules # rule set used by default
#indexrules multimedia.rules
indexrules mysql.rules # rule set used by default
#indexrules netbios.rules
indexrules nntp.rules # rule set used by default
indexrules oracle.rules # rule set used by default
#indexrules other-ids.rules
#indexrules p2p.rules
#indexrules policy.rules
indexrules pop2.rules # rule set used by default
indexrules pop3.rules # rule set used by default
#indexrules porn.rules
indexrules rpc.rules # rule set used by default
indexrules rservices.rules # rule set used by default
indexrules scan.rules # rule set used by default
#indexrules shellcode.rules
#indexrules smtp.rules
indexrules snmp.rules # rule set used by default
#indexrules sql-injection.rules
indexrules sql.rules # rule set used by default
indexrules telnet.rules # rule set used by default
indexrules tftp.rules # rule set used by default
#indexrules virus.rules
indexrules web-attacks.rules # rule set used by default
indexrules web-cgi.rules # rule set used by default
indexrules web-client.rules # rule set used by default
indexrules web-coldfusion.rules # rule set used by default
indexrules web-frontpage.rules # rule set used by default
indexrules web-iis.rules # rule set used by default
indexrules web-misc.rules # rule set used by default
indexrules web-php.rules # rule set used by default
indexrules x11.rules # rule set used by default


it worked. I thought neat. so I then un commented some of the others out so it looked like

indexrules attack-responses.rules # rule set used by default
indexrules backdoor.rules # rule set used by default
indexrules bad-traffic.rules # rule set used by default
#indexrules chat.rules
indexrules ddos.rules # rule set used by default
#indexrules deleted.rules
indexrules dns.rules # rule set used by default
indexrules dos.rules # rule set used by default
#indexrules experimental.rules
indexrules exploit.rules
indexrules finger.rules # rule set used by default
indexrules ftp.rules # rule set used by default
indexrules icmp-info.rules
indexrules icmp.rules # rule set used by default
indexrules imap.rules # rule set used by default
indexrules info.rules
indexrules malware-user-agents.rules
indexrules misc.rules # rule set used by default
#indexrules multimedia.rules
indexrules mysql.rules # rule set used by default
indexrules netbios.rules
indexrules nntp.rules # rule set used by default
indexrules oracle.rules # rule set used by default
indexrules other-ids.rules
#indexrules p2p.rules
#indexrules policy.rules
indexrules pop2.rules # rule set used by default
indexrules pop3.rules # rule set used by default
#indexrules porn.rules
indexrules rpc.rules # rule set used by default
indexrules rservices.rules # rule set used by default
indexrules scan.rules # rule set used by default
indexrules shellcode.rules
indexrules smtp.rules
indexrules snmp.rules # rule set used by default
indexrules sql-injection.rules
indexrules sql.rules # rule set used by default
indexrules telnet.rules # rule set used by default
indexrules tftp.rules # rule set used by default
indexrules virus.rules
indexrules web-attacks.rules # rule set used by default
indexrules web-cgi.rules # rule set used by default
indexrules web-client.rules # rule set used by default
indexrules web-coldfusion.rules # rule set used by default
indexrules web-frontpage.rules # rule set used by default
indexrules web-iis.rules # rule set used by default
indexrules web-misc.rules # rule set used by default
indexrules web-php.rules # rule set used by default
indexrules x11.rules # rule set used by default

so I un commented out a few, first one was exploit.rules (sorry its hard to list them all right now) point is I just un commented out the ones I thought I'd like to use???

I deleted combined.rules from /etc/snort
and re-ran the script. (so it downloaded from ET into /etc/snort/ET-downloads) again. currently emerging-all.rules
is in that dir

I did a service snort status and it told me it was not running.
I did a snort start and then it spat out some errors from combined.rules file (which I edited as mentioned above)
until I successfully did a service snort start

Perhaps I have done something wrong.

follow up..

deleted combined.rules from /etc/snort
deleted emerging-all.rules from /etc/snort/ET-downloads

service snort stop
Stopping snort: [ OK ]

Ran script.

/etc/snort/ET-downloads has
6573683 Jul 8 12:23 emerging-all.rules

typed snort start.
INFO => [Alert_FWsam](FWsamCheckIn) Connected to host .
ERROR: Undefined variable name: (/etc/snort/combined.rules:8573): HOME_NETany
Fatal Error, Quitting..

nano /etc/snort/combined.rules (inserted space)

snort start
INFO => [Alert_FWsam](FWsamCheckIn) Connected to host .
ERROR: /etc/snort/combined.rules(8573) => Unknown rule type: rttcp$HOME_NET
Fatal Error, Quitting.

nano /etc/snort/combined.rules (deleted some lines)

snort start
INFO => [Alert_FWsam](FWsamCheckIn) Connected to host .
ERROR: /etc/snort/combined.rules(10306) => Unknown rule type: xbb\xbd-\xbf\xd9][^\x00][^\x00][^\x00][^\x00][\x31\x83][\x42\x43\x45-\x47\x50\x53\x55-\x58\x5a\x5d-\x5f\x68\x6a\x6b\x6e-\x70\x72\x73\x75\x77\x78\x7a\x7b\x7d\x7e\xc0\xc2\xc3\xc5-\xc7\xe8\xea\xeb\xed-\xef][\x04\x0e\x12\x17\xfc][\x03\x31\x83][\x42\x43\x45-\x47\x50\x53\x55-\x58\x5a\x5d-\x5f\x68\x6a\x6b\x6e-\x70\x72\x73\x75\x77\x78\x7a\x7b\x7d\x7e\xc0\xc2\xc3\xc5-\xc7\xe8\xea\xeb\xed-\xef][\x04\x0a\x0e\x12\x13\x17\xfc][\x03\x83]..[^\xe2][^\xf5])/sm";
Fatal Error, Quitting..

nano /etc/snort/combined.rules (deleted some lines)

snort start

Initializing Network Interface eth0
OpenPcap() device eth0 network lookup:
eth0: no IPv4 address assigned
ERROR: OpenPcap() FSM compilation failed:
syntax error
PCAP command: start
Fatal Error, Quitting..

service snort start
Starting snort: [ OK ]

Although the error occurs i the combined rules this is a rule set created by merging the ET rules with the uncommented ClearOS rules. As I am not getting the error I would assume there is an error in one of the ClearOS rules you have uncommented. I am not at home at the moment so I can't check myself but you could try from the /etc/snort directory:

grep -l rttcp$HOME_NET *.rules

This should locate the files with the bad lines.

Also when snort fails to start, have a look in /var/log/messages for the snort log. With luck the last line should be about the error. This is how I tracked down the SSH_PORTS error.


[edit]
At home now.

I cannot find the string rttcp$HOME_NET in any file. grep'ing the ClearOS rule numbers gives:
8573 - virus.rules but the line is disabled there. My string reads "#alert tcp $HOME_NET ....." with spaces so I think yours is corrupt. In any case my line is commented out by default so I am not sure how yours is enabled.
10306 - web-cgi.rules. This is not a rule number but just appears in two lines. It in the ClearOS default-enabled rule set but not the ET rules so I wonder if you have a corrupt set of rules files?

[/edit]

Jimmy,

There appears to be an error in your snort config file where you declare the HOME_NET variable:

typed snort start.

INFO => [Alert_FWsam](FWsamCheckIn) Connected to host .

ERROR: Undefined variable name: (/etc/snort/combined.rules:8573): HOME_NETany

Fatal Error, Quitting..

The variable name and value must be seperated by a space:

var HOME_NET any

Bob,

var HOME_NET should be set in /etc/snort.conf by the init script (see comments in snort.conf). The string rttcp$HOME_NET looks like it comes from line 2478 of the virus.rules lines but with the spacing changed (i.e it was "#alert tcp $HOME_NET any .....). The line is disabled in my (untouched) virus.rules but probably enabled in his. It also features in the ET rules so it could be that his download is corrupt, but Jimmy has re-downloaded. I have also just re-downloaded the ET rules and they check out OK.

Nick

Those of you who are using the ET signatures may want to consider getting on their mailing list: list signup

I have not ruled out 1) not knowing enough 2) problem at my end .


pwd
/etc/snort

grep -l rttcp$HOME_NET *.rules

nada... (as I deleted it from combined.rules already)

snort.conf=
.
.
.
.
# chat, multimedia, and p2p
#
# These rules are either site policy specific or require tuning in order to not
# generate false positive alerts in most enviornments.
#
# Please read the specific include file for more information and
# README.alert_order for how rule ordering affects how alerts are triggered.
#=========================================
include $RULE_PATH/combined.rules

everything beneath this line has been #'ed out.

rules in /etc/snort include

ls *.rules

attack-responses.rules icmp-info.rules p2p.rules sql.rules
backdoor.rules icmp.rules policy.rules telnet.rules
bad-traffic.rules imap.rules pop2.rules tftp.rules
chat.rules info.rules pop3.rules virus.rules
combined.rules local.rules porn.rules web-attacks.rules
ddos.rules malware-user-agents.rules rpc.rules web-cgi.rules
deleted.rules misc.rules rservices.rules web-client.rules
dns.rules multimedia.rules scan.rules web-coldfusion.rules
dos.rules mysql.rules shellcode.rules web-frontpage.rules
experimental.rules netbios.rules smtp.rules web-iis.rules
exploit.rules nntp.rules snmp.rules web-misc.rules
finger.rules oracle.rules spyware-put.rules web-php.rules
ftp.rules other-ids.rules sql-injection.rules x11.rules


the only thing I have every done to these is increase the duration of banning (ex in shellcode.rules it banned for 1 day (from memory), which I increased) note: truth be told I may have added the banning... ???(to this and other .rules)

in any case all I did was add fwsam: src, XYZ days XYZ=a number to the rules that others where triggering, I got sick of seeing people try and do system calls (and decided to just ban the ip for a period of time)


nano virus.rules and goto line 2478=
#alert tcp $HOME_NET any -> $EXTERNAL_NET 3306 (msg:"ET TROJAN Viruscatch.co.kr/Win32.Small.hvd Mysql Comm$...etc etc

any idea's? has my tinkering caused the prob... by adding things after the rev number


oh, I did use some other ET rules that sit in another dir. in snort.conf they have been #'ed out.

I'll have to try enabling the extra files you enabled and see if my combined file becomes corrupt. I'm not sure when I'll get the chance to do it.

Can you locate the line(s) in virus.rules which have the string 8573 in them and see if they read OK?

grep "8573" virus.rules=

#alert tcp $HOME_NET any -> $EXTERNAL_NET 3306 (msg:"ET TROJAN Viruscatch.co.kr/Win32.Small.hvd Mysql Command and Control Connection (user viruscatch)"; flow:established,to_server; dsize:<40; flowbits:isset,EText.mysql.greeting; content:"viruscatch|00|"; classtype:trojan-activity; reference:url,doc.emergingthreats.net/2008573; reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/VIRUS/TROJAN_Viruscatch.co.kr; sid:2008573; rev:2

grep "8573" *.rules=

combined.rules:alert tcp $HOME_NET any -> $EXTERNAL_NET 3306 (msg:"ET TROJAN Viruscatch.co.kr/Win32.Small.hvd Mysql Command and Control Connection (user viruscatch)"; flow:established,to_server; dsize:<40; content:"viruscatch|00|"; classtype:trojan-activity; reference:url,doc.emergingthreats.net/2008573; reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/VIRUS/TROJAN_Viruscatch.co.kr; sid:2008573; rev:3

virus.rules:#alert tcp $HOME_NET any -> $EXTERNAL_NET 3306 (msg:"ET TROJAN Viruscatch.co.kr/Win32.Small.hvd Mysql Command and Control Connection (user viruscatch)"; flow:established,to_server; dsize:<40; flowbits:isset,EText.mysql.greeting; content:"viruscatch|00|"; classtype:trojan-activity; reference:url,doc.emergingthreats.net/2008573; reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/VIRUS/TROJAN_Viruscatch.co.kr; sid:2008573; rev:2

The rules seem to read OK.In the combined rules you only have the ET rule which has a higher revision and in any case the virus.rules 8573 is commented out so it is not used. The ET rule should be fine as it works for me, so I have to assume it is the script going wrong somewhere. I will have a look at it some time at home but I am having kiddie issues at the moment so it is hard to get a block of time to concentrate on it.

OK, I've found the error(s) and it is not me!
In the extra rules you've enabled there are two errors

1 - malware-user-agents.rules lines 899/900. This should be one line. Delete the <cr> at the end of 899 and replace it with an "r" to correctly spell the word "reference"

2 - virus.rules lines 91/92 should be one line. Delete the <cr> at the end of 91 and replace it with a "/" or just delete the two lines as the rule is disabled anyway

BTW I've noticed that the number of the error reported in /var/log/messages is the line number of the error and not the rule number. That is the problem when not having the system to hand when posting.

nice.

will try from scratch and see what happens

ps. yeah I thought that when I posted the errors (from combined.rules) it was showing the line number.

I have another question, but will re-do from scratch and if it still is happening post again.

thanks for taking the time. I think it will help others

also a request (to any body) be great to be able to add a longer ban on any rule others trigger in a nice and easy way after updating rules/including other rules I did start a python script about 12 months ago.. but lost interest (did not have enough time to study snort rules etc/how everything worked, thought it best left to the experts)..

just as an example, if I see an IP triggering some rule many times, I wish it would auto block/ban for say 30 days (its all cool that it see's it, and then applies the appropriate ignore/drop in that instance) but I think it would be smarted to get rid of the source of the problem. some times for me a prioirty 3 warning deserve a longer block/ban personal preference (sorry for off topic)

example.
(ftp_telnet) FTP traffic encrypted[Priority: 3]: {TCP} MYIP:2660 -> SOME IP:21 (only common decency prevents me posting IP)
i have 3 days of logs of this over different times. I know this is not triggering a rule, but some inbuilt snort thing (sorry unclear of technical term, unclear of what is happening). but the only way I could remove this, was to use tims advance rule editor and block anything going out to port 21, and also to block incoming/outgoing to SOME IP which to me seems many more steps than need be. since doing this. have not seen this person again. (it was going both ways SOME IP:21 to me, and me :random port to SOME IP:21) I have been using clear for about 12 months , never have I seen this in my logs (only twice in the last week or so). AND it will not happen @ 3:00am (my time) nobody is on at that time at my end (home setup).

Are you running an ftp server with port 21 open to the public? There is little point in putting an outbound block. If you block the inbound ftp no outbound should happen unless your machine is compromised.

Nick, NO FTP service is running at all..... my initial thoughts was some software updates/ftp downloads..... but thats not the case.... the person is trying to use telnet commands on the ftp port... (without looking into it too much).

I am just hoping its not compromised. (not enough knowledge to say one way or the other in truth). I'll be honest the back and forth was a concern though.



open port finder:
Port 21 is closed on whateverMYIPIS...(and to the best of my knowledge has always been not available- pre this) even though they NEVER tried to connect to 21 my end.

I guess I really should trace it, and reverse dns it.... again this is my point snort detected it, did whatever....but my concern was it should have banned for 30 days (just for trying).... not allowed it to detect/drop for 3 days...I trust those in the know @ clear/snort and believe it did the right thing.