Posts in Category: Technology
Installing Snort++ in Ubuntu (Version 3.0 Alpha 4 build 245)
The instructions below show how to install Snort 3 alpha 4 build 245 on Ubuntu. This install has been tested on Ubuntu 14, 16, and 18, for the x64 architecture. For an outdated Ubuntu 12 version of these instructions, please go here. Note that Snort 3 is Alpha software, and therefore has bugs and issues, and should be installed for testing purposes only (not on production systems).
Snort 3 Alpha 4 Build 245 was released on May 24th, 2018, and this guide has been tested with that version (releases after this specific release may not follow the same steps). Generic build instructions, prerequisites, and detailed notes are available in the manual.
If you want a more in-depth explanation of the install steps, which are very similar to the 2.9.9.x version of Snort, as well as instructions on how to configure and enhance Snort’s functionality, see my series on installing Snort 2.9.9.x on Ubuntu.
So let’s get started. First, we will create a directory to save the downloaded tarball files:
mkdir ~/snort_src cd ~/snort_src
Next we need to install all the Snort pre-requisites from the Ubuntu repositories:
sudo apt-get install -y build-essential autotools-dev libdumbnet-dev libluajit-5.1-dev libpcap-dev libpcre3-dev zlib1g-dev pkg-config libhwloc-dev
For Ubuntu 16 and 18, you can install cmake from the default repository:
# Ubuntu 16 and 18 (and greater) only: sudo apt-get install -y cmake
For Ubuntu 14, you have to install cmake from source, because the version in the Ubuntu repository is too old to compile Snort:
# Ubuntu 14 only sudo apt-get remove -y cmake cd ~/snort_src wget https://cmake.org/files/v3.10/cmake-3.10.3.tar.gz tar -xzvf cmake-3.10.3.tar.gz cd cmake-3.10.3 ./bootstrap make sudo make install
Note: If you are running Ubuntu 14 x86, you must install a newer version of gcc, either from a ppa or from source. Snort will not compile properly on the x86 platform with the version of gcc in the Ubuntu repository (4.8.4). I do not cover these steps in this guide, but you can easily google this process. This does not apply to the x64 version of Ubuntu 14 which you can easily install using the steps below.
Install the optional (recommended) software:
sudo apt-get install -y liblzma-dev openssl libssl-dev cpputest libsqlite3-dev uuid-dev
Install tools required for compiling the source from github:
sudo apt-get install -y libtool git autoconf
Install the DAQ pre-requisites:
sudo apt-get install -y bison flex
If you want to build the documentation as well (not really needed, unless you want it, usually about 700 MB of libraries):
sudo apt-get install -y asciidoc dblatex source-highlight w3m
If you want to run Snort in inline mode using NFQ, install the required packages (not required for IDS mode or inline mode using afpacket). If you’re unsure, you should install this package.
sudo apt-get install -y libnetfilter-queue-dev
Next, compile and install safec for runtime bounds checks on certain legacy C-library calls (this is optional but recommended):
cd ~/snort_src wget http://downloads.sourceforge.net/project/safeclib/libsafec-10052013.tar.gz tar -xzvf libsafec-10052013.tar.gz cd libsafec-10052013 ./configure make sudo make install
Download and install gperftools 2.7, google’s thread-caching malloc (used in chrome). Tcmalloc is a memory allocator that’s optimized for high concurrency situations which will provide better speed for the trade-off of higher memory usage. We don’t want the version of tcmalloc from the repositories (version 2.5 in libgoogle-perftools-dev) as they don’t work with Snort. Tcmalloc is optional but recommended:
cd ~/snort_src wget https://github.com/gperftools/gperftools/releases/download/gperftools-2.7/gperftools-2.7.tar.gz tar xzvf gperftools-2.7.tar.gz cd gperftools-2.7 ./configure make sudo make install
One of the Snort recommended prerequisites is Hyperscan 4.4.0. From their webpage: “Hyperscan is a regular expression engine designed to offer high performance, the ability to match multiple expressions simultaneously and flexibility in scanning operation.” Hyperscan needs Ragel 6.9 and the Boost header libraries.
Install Ragel 6.10 from source:
cd ~/snort_src wget http://www.colm.net/files/ragel/ragel-6.10.tar.gz tar -xzvf ragel-6.10.tar.gz cd ragel-6.10 ./configure make sudo make install
Download the Boost 1.67.0 libraries, but do not install:
cd ~/snort_src wget https://dl.bintray.com/boostorg/release/1.67.0/source/boost_1_67_0.tar.gz tar -xvzf boost_1_67_0.tar.gz
Install Hyperscan 4.7.0 from source, referencing the location of the Boost source directory:
cd ~/snort_src wget https://github.com/intel/hyperscan/archive/v4.7.0.tar.gz tar -xvzf v4.7.0.tar.gz mkdir ~/snort_src/hyperscan-4.7.0-build cd hyperscan-4.7.0-build/ cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DBOOST_ROOT=~/snort_src/boost_1_67_0/ ../hyperscan-4.7.0 make sudo make install
If you want to test that Hyperscan works, from the build directory, run:
cd ~/snort_src/hyperscan-4.7.0-build/ ./bin/unit-hyperscan
The unit tests will run (this takes a few minutes).
Snort has an optional requirement for flatbuffers, A memory efficient serialization library:
cd ~/snort_src wget https://github.com/google/flatbuffers/archive/v1.9.0.tar.gz -O flatbuffers-v1.9.0.tar.gz tar -xzvf flatbuffers-1.9.0.tar.gz mkdir flatbuffers-build cd flatbuffers-build cmake ../flatbuffers-1.9.0 make sudo make install
Download and install Data AcQuisition library (DAQ) from the Snort website (note that DAQ for Snort 3 is a different DAQ than for the 2.9.9.x series of Snort):
cd ~/snort_src wget https://www.snort.org/downloads/snortplus/daq-2.2.2.tar.gz tar -xvzf daq-2.2.2.tar.gz cd daq-2.2.2 ./configure make sudo make install
Run the following command to update shared libraries:
sudo ldconfig
Now we are ready to install Snort from source. This command downloads and installs the latest version of Snort 3 (currently 3.0.0 Alpha 4, build 245, but as the codebase is updated, you’ll get a newer version).
If you want to install all the snort directories under a single directory, see the section at the bottom of this document titled Changing the install location of Snort. Here we choose to install Snort to the /usr/local directory (the standard location for additional software from the LSB)
cd ~/snort_src git clone git://github.com/snortadmin/snort3.git cd snort3 ./configure_cmake.sh --prefix=/usr/local cd build make sudo make install
Optional: If you are interested in seeing what additional options can be configured when building Snort, run ./configure cmake.sh ‐‐help for a full list. Some options you may be interested in is the Snort3 command line shell (‐‐enable-shell) or support for pcap files over 2 GB (‐‐enable-large-pcap).
Snort 3 requires a few environmental variables, we store them temporarily in the current session so we can continue working, and save them permanently to the ~/.bashrc file (you’ll need to do this for every user profile):
export LUA_PATH=/usr/local/include/snort/lua/\?.lua\;\; export SNORT_LUA_PATH=/usr/local/etc/snort sh -c "echo 'export LUA_PATH=/usr/local/include/snort/lua/\?.lua\;\;' >> ~/.bashrc" sh -c "echo 'export SNORT_LUA_PATH=/usr/local/etc/snort' >> ~/.bashrc"
to ensure that these two environmental variables are available when using sudo, we need to add them to the /etc/sudoers file:
sudo visudo
in the editor, add the following to to the bottom of the file:
Defaults env_keep += "LUA_PATH SNORT_LUA_PATH"
use ctrl-x to exit, save when prompted by pressing y, then press enter to save the file to /etc/sudoers.tmp (which will get copied automatically to /etc/sudoers).
The last step of our Snort installation is to test that the Snort Binary runs. Execute Snort with the -V flag, which causes Snort to print the current version. You should see output similar to the following:
user@snort3:~$ snort -V
,,_ -*> Snort++ <*-
o" )~ Version 3.0.0-a4 (Build 245) from 2.9.11
'''' By Martin Roesch & The Snort Team
http://snort.org/contact#team
Copyright (C) 2014-2017 Cisco and/or its affiliates. All rights reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using DAQ version 2.2.2
Using LuaJIT version 2.1.0-beta3
Using OpenSSL 1.1.0g 2 Nov 2017
Using libpcap version 1.8.1
Using PCRE version 8.39 2016-06-14
Using ZLIB version 1.2.11
Using FlatBuffers 1.9.0
Using Hyperscan version 4.7.0 2018-06-12
Using LZMA version 5.2.2
user@snort3:~$
A note on install locations:
When you install snort to /usr/local, you get the following folder structure:
user@snort3x86:/opt/snort$ tree /usr/local -L 3
/usr/local
├── bin
│ ├── fbstreamer
│ ├── snort
│ ├── snort2lua
│ ├── u2boat
│ └── u2spewfoo
├── etc
│ └── snort
│ ├── file_magic.lua
│ ├── snort_defaults.lua
│ └── snort.lua
├── include
│ └── snort
│ ├── actions
│ ├── codecs
│ ├── daqs
│ ├── decompress
│ ├── detection
│ ├── events
│ ├── file_api
│ ├── flow
│ ├── framework
│ ├── hash
│ ├── log
│ ├── lua
│ ├── main
│ ├── managers
│ ├── mime
│ ├── packet_io
│ ├── profiler
│ ├── protocols
│ ├── pub_sub
│ ├── search_engines
│ ├── sfip
│ ├── stream
│ ├── time
│ └── utils
├── lib
│ ├── pkgconfig
│ │ └── snort.pc
│ └── snort
│ └── daqs
└── share
└── doc
└── snort
38 directories, 8 files
The /usr/local/bin folder contains the following Snort binaries:
- fbstreamer : A utility for accessing the statistics generated in flatbuffer format.
- snort : The Snort binary.
- snort2lua : Tool to convert a Snort 2.9.8.x configuration file into a 3.x configuration file. More notes here.
- u2boat : U2boat is a tool for converting unified2 files into different formats.
- u2spewfoo: U2SpewFoo is a lightweight tool for dumping the contents of unified2 files to stdout.
Additionally, the following folders are created / used:
- /opt/snort/bin : Binaries for Snort and supporting software.
- /opt/snort/etc/snort : The configuration files for Snort.
- /opt/snort/include/snort : All include files for Snort.
- /opt/snort/lib/pkgconfig : The pkgconfig file for Snort (compilation details for Snort).
- /opt/snort/share/doc/snort : The documentation for the installed version of Snort.
Changing the install location of Snort
If you would rather have all these folders install to a single-folder location for testing (/opt/snort), add ‑‑prefix=/opt/snort to the ./configure command when preparing to build Snort. This will install all these folders under the path you choose. You also need to modify some of the other paths detailed above, so if you decide to install in that manner, you should follow the install instructions detailed in the Snort blog.
Snort Rules
Snort3 rules have more options than Snort 2 rules, and while the normal rules downloaded with PulledPork or manually will work, for testing you will probably want to download the set of community rules specificallycreated for snort3. You can manually download snort3 specific community rules from the snort website:
cd ~/snort_src/ wget https://www.snort.org/downloads/community/snort3-community-rules.tar.gz tar -xvzf snort3-community-rules.tar.gz cd snort3-community-rules sudo mkdir /usr/local/etc/snort/rules sudo mkdir /usr/local/etc/snort/builtin_rules sudo mkdir /usr/local/etc/snort/so_rules sudo mkdir /usr/local/etc/snort/lists sudo cp snort3-community.rules /usr/local/etc/snort/rules/ sudo cp sid-msg.map /usr/local/etc/snort/rules/
now test that snort can load these rules:
snort -c /usr/local/etc/snort/snort.lua -R /usr/local/etc/snort/rules/snort3-community.rules
your output should contain something similar:
Loading rules: Loading /opt/snort/etc/snort/rules/snort3-community.rules: Finished /opt/snort/etc/snort/rules/snort3-community.rules. Finished rules. -------------------------------------------------- rule counts total rules loaded: 829 text rules: 829 option chains: 829 chain headers: 46 -------------------------------------------------- ...
you may want to run Snort with the following flags to detect issues: the warn-all and pedantic flags. From the Snort3 manual:
Warnings are not emitted unless –warn-* is specified. –warn-all enables all warnings, and –pedantic makes such warnings fatal
Installing OpenAppID
OpenAppID allows for the identification of application layer traffic. The Snort team has put together a package of detectors, with assistance from the community that you can download and install, called the Application Detector Package which needs to be installed. First download the OpenAppID detector package:
cd ~/snort_src/ wget https://www.snort.org/downloads/openappid/7630 -O OpenAppId-7630.tar.gz tar -xzvf OpenAppId-7630.tar.gz sudo cp -R odp /usr/local/lib/
Now we need to edit our snort configuration file to point to this odp directory:
sudo vi /usr/local/etc/snort/snort.lua
At line 113 (yours line number may be slightly different) you will see the appid= entry. You will want to add the app detector dir option here, pointing to the parent folder of the odf folder. It should look like this:
appid =
{
app_detector_dir = '/usr/local/lib',
}
note that you must have four spaces (not a tab) for the indented line. Now we want to test the configuration file loads correctly:
snort -c /usr/local/etc/snort/snort.lua --warn-all
you should see output similar to:
... Snort successfully validated the configuration (with 0 warnings). o")~ Snort exiting
Now to load Snort with the OpenAppID detectors, as well as all rules (we omit the pedantic flag, since the rules will throw flowbit warnings that are non fatal and can be ignored:)
snort -c /usr/local/etc/snort/snort.lua -R /usr/local/etc/snort/rules/snort3-community.rules --warn-all
Create a simple rule to test that OpenAppID is working correctly:
sudo touch /usr/local/etc/snort/rules/local.rules sudo vi /usr/local/etc/snort/rules/local.rules
with the following content:
alert tcp any any -> any any ( msg:"Facebook trafic Seen"; appids:"Facebook";sid:10000001; )
test to make sure the rule loads correctly:
snort -c /usr/local/etc/snort/snort.lua -R /opt/snort/etc/snort/rules/local.rules --warn-all
You should see one rule loaded successfully. Now let’s run snort in detection mode on an interface (change eth0 below to match your interface name), printing alerts to the console:
sudo snort -c /usr/local/etc/snort/snort.lua -R /usr/local/etc/snort/rules/local.rules -i eth0 -A alert_fast -k none
the -k none flag tells Snort to ignore bad checksums. the Stream and Frag decoders will drop packets that have bad checksums, and the packets will not get processed by the OpenAppID detectors. By including this flag, we ensure that a packet with a bad checksum still gets processed. Now from another window on that computer (open a new terminal window or a second ssh session), use wget
to connect to facebook:
wget facebook.com
from the first console window you will see alerts output similar to the following:
08/05−19:13:45.451834 [∗∗] [1:10000001:0] ”Facebook trafic Seen” [∗∗] [Priority: 0] [AppID: Facebook] {TCP} 157.240.1.35:443 −> 10.0.0.104:33882
08/05−19:13:45.451842 [∗∗] [1:10000001:0] ”Facebook trafic Seen” [∗∗] [Priority: 0] [AppID: Facebook] {TCP} 10.0.0.104:33882 −> 157.240.1.35:443
use ctrl-c to stop Snort.
Note: if you are collecting packets with a larger MTU that the standard MTU for your adapter (VLAN tagged packets, MPLS Packets, packets from a different network type with a larger MTU), you may need to use the –snaplen flag to adjust snort to process larger packets)
Where to go from here
If you want to learn more about how to run the 2.9.9.x version of Snort, and how to install additional software to enhance a Snort system, see my series on installing Snort on Ubuntu. If you want to develop Snort plugins, please see my guide: Installing Snort++ Example Plugins.
I would love to get feedback from you about this guide. Recommendations, issues, or ideas, please contact me here.
Installing Snort++ in Ubuntu (Version 3.0 Alpha 4 build 240)
NOTE: this article is out of date and has been replaced with a newer article: Installing Snort++ (Snort 3 Alpha 4 build 245) in Ubuntu
The instructions below show how to install Snort 3 alpha 4 build 240 on Ubuntu. This install has been tested on Ubuntu 14 and 16, for both the x86 and x64 architectures. For an outdated Ubuntu 12 version of these instructions, please go here. Note that Snort 3 is Alpha software, and therefore has bugs and issues, and should be installed for testing purposes only (not on production systems).
Snort 3 Alpha 4 Build 240 was released on November 1st, 2017, and this guide has been tested with that version (releases after this specific release may not follow the same steps). Generic build instructions, prerequisites, and detailed notes are available in the manual.
If you want a more in-depth explanation of the install steps, which are very similar to the 2.9.9.x version of Snort, as well as instructions on how to configure and enhance Snort’s functionality, see my series on installing Snort 2.9.9.x on Ubuntu.
So let’s get started. First we need to install all the Snort pre-requisites from the Ubuntu repositories:
sudo apt-get install -y build-essential autotools-dev libdumbnet-dev libluajit-5.1-dev libpcap-dev libpcre3-dev zlib1g-dev pkg-config libhwloc-dev cmake
Install the optional (recommended) software:
sudo apt-get install -y liblzma-dev openssl libssl-dev cpputest libsqlite3-dev uuid-dev
Install tools required for compiling the source from github:
sudo apt-get install -y libtool git autoconf
Install the DAQ pre-requisites:
sudo apt-get install -y bison flex
If you want to build the documentation as well (not really needed, unless you want it, usually about 700 MB of libraries):
sudo apt-get install -y asciidoc dblatex source-highlight w3m
If you want to run Snort in inline mode using NFQ, install the required packages (not required for IDS mode or inline mode using afpacket). If you’re unsure, you should install this package.
sudo apt-get install -y libnetfilter-queue-dev
Next we will create a directory to save the downloaded tarball files:
mkdir ~/snort_src cd ~/snort_src
First and install safec for runtime bounds checks on certain legacy C-library calls (this is optional but recommended):
cd ~/snort_src wget http://downloads.sourceforge.net/project/safeclib/libsafec-10052013.tar.gz tar -xzvf libsafec-10052013.tar.gz cd libsafec-10052013 ./configure make sudo make install
One of the Snort recommended prerequisites is Hyperscan 4.4.0. From their webpage: “Hyperscan is a regular expression engine designed to offer high performance, the ability to match multiple expressions simultaneously and flexibility in scanning operation.” Hyperscan needs Ragel 6.9 and the Boost header libraries.
Install Ragel 6.10 from source:
cd ~/snort_src wget http://www.colm.net/files/ragel/ragel-6.10.tar.gz tar -xzvf ragel-6.10.tar.gz cd ragel-6.10 ./configure make sudo make install
Download the Boost 1.65.1 libraries, but do not install:
cd ~/snort_src wget https://dl.bintray.com/boostorg/release/1.65.1/source/boost_1_65_1.tar.gz tar -xvzf boost_1_65_1.tar.gz
Install Hyperscan 4.6.0 from source, referencing the location of the Boost source directory:
cd ~/snort_src wget https://github.com/intel/hyperscan/archive/v4.6.0.tar.gz tar -xvzf v4.6.0.tar.gz mkdir ~/snort_src/hyperscan-4.6.0-build cd hyperscan-4.6.0-build/ cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DBOOST_ROOT=~/snort_src/boost_1_65_1/ ../hyperscan-4.6.0 make sudo make install
If you want to test that Hyperscan works, from the build directory, run:
cd ~/snort_src/hyperscan-4.6.0-build/ ./bin/unit-hyperscan
The unit tests will run (this takes a few minutes).
Snort has an optional requirement for flatbuffers, A memory efficient serialization library:
cd ~/snort_src wget https://github.com/google/flatbuffers/archive/master.tar.gz -O flatbuffers-master.tar.gz tar -xvzf flatbuffers-master.tar.gz mkdir flatbuffers-build cd flatbuffers-build cmake ../flatbuffers-master make sudo make install
Download and install Data AcQuisition library (DAQ) from the Snort website (note that DAQ for Snort 3 is a different DAQ than for the 2.9.9.x series of Snort):
cd ~/snort_src wget https://www.snort.org/downloads/snortplus/daq-2.2.2.tar.gz tar -xvzf daq-2.2.2.tar.gz cd daq-2.2.2 ./configure make sudo make install
Run the following command to update shared libraries:
sudo ldconfig
Now we are ready to install Snort from source. This command downloads and installs the latest version of Snort 3 (currently 3.0.0 Alpha 4, build 239, but as the codebase is updated, you’ll get a newer version). If you want to specifically download the version used in this guide, use this URL instead with wget below: https://github.com/snortadmin/snort3/archive/BUILD_239.tar.gz.
If you want to install all the snort directories under a single directory, see the section at the bottom of this document titled Changing the install location of Snort. Here we choose to install the entire Snort directory structure to a single folder under /opt/:
cd ~/snort_src wget https://github.com/snortadmin/snort3/archive/master.tar.gz tar -xvzf master.tar.gz cd snort3-master/ autoreconf -isvf ./configure --prefix=/opt/snort make sudo make install
Since the Snort installation places the Snort binary at /opt/snort/bin/snort, it is common to create a symlink to /usr/sbin/snort:
sudo ln -s /opt/snort/bin/snort /usr/sbin/snort
Snort 3 requires a few environmental variables, we store them temporarily in the current session so we can continue working, and save them permanently to the ~/.bashrc file (you’ll need to do this for every user profile):
export LUA_PATH=/opt/snort/include/snort/lua/\?.lua\;\; export SNORT_LUA_PATH=/opt/snort/etc/snort sh -c "echo 'export LUA_PATH=/opt/snort/include/snort/lua/\?.lua\;\;' >> ~/.bashrc" sh -c "echo 'export SNORT_LUA_PATH=/opt/snort/etc/snort' >> ~/.bashrc"
to ensure that these two environmental variables are available when using sudo, we need to add them to the /etc/sudoers file:
sudo visudo
in the editor, add the following to to the bottom of the file:
Defaults env_keep += "LUA_PATH SNORT_LUA_PATH"
use ctrl-x to exit, save when prompted by pressing y, then press enter to save the file to /etc/sudoers.tmp (which will get copied automatically to /etc/sudoers).
The last step of our Snort installation is to test that the Snort Binary runs. Execute Snort with the -V flag, which causes Snort to print the current version. You should see output similar to the following:
user@snort3:~$ snort -V
,,_ -*> Snort++ <*-
o" )~ Version 3.0.0-a4 (Build 240) from 2.9.8-383
'''' By Martin Roesch & The Snort Team
http://snort.org/contact#team
Copyright (C) 2014-2017 Cisco and/or its affiliates. All rights reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using DAQ version 2.2.2
Using LuaJIT version 2.0.4
Using OpenSSL 1.0.2g 1 Mar 2016
Using libpcap version 1.7.4
Using PCRE version 8.38 2015-11-23
Using ZLIB version 1.2.8
Using FlatBuffers 1.7.0
Using Hyperscan version 4.6.0 2017-11-04
Using LZMA version 5.1.0alpha
user@snort3:~$
A note on install locations:
When you install snort to /opt/snort, you get the following folder structure:
user@snort3x86:/opt/snort$ tree /opt/snort -L 3
/opt/snort
├── bin
│ ├── fbstreamer
│ ├── snort
│ ├── snort2lua
│ ├── u2boat
│ └── u2spewfoo
├── etc
│ └── snort
│ ├── file_magic.lua
│ ├── snort_defaults.lua
│ └── snort.lua
├── include
│ └── snort
│ ├── actions
│ ├── codecs
│ ├── daqs
│ ├── decompress
│ ├── detection
│ ├── events
│ ├── file_api
│ ├── flow
│ ├── framework
│ ├── hash
│ ├── log
│ ├── lua
│ ├── main
│ ├── managers
│ ├── mime
│ ├── packet_io
│ ├── profiler
│ ├── protocols
│ ├── pub_sub
│ ├── search_engines
│ ├── sfip
│ ├── stream
│ ├── time
│ └── utils
├── lib
│ ├── pkgconfig
│ │ └── snort.pc
│ └── snort
│ └── daqs
└── share
└── doc
└── snort
38 directories, 8 files
The /opt/snort/bin folder contains the following Snort binaries:
- fbstreamer : A utility for accessing the statistics generated in flatbuffer format.
- snort : The Snort binary.
- snort2lua : Tool to convert a Snort 2.9.8.x configuration file into a 3.x configuration file. More notes here.
- u2boat : U2boat is a tool for converting unified2 files into different formats.
- u2spewfoo: U2SpewFoo is a lightweight tool for dumping the contents of unified2 files to stdout.
Additionally, the following folders are created / used:
- /opt/snort/bin : Binaries for Snort and supporting software.
- /opt/snort/etc/snort : The configuration files for Snort.
- /opt/snort/include/snort : All include files for Snort.
- /opt/snort/lib/pkgconfig : The pkgconfig file for Snort (compilation details for Snort).
- /opt/snort/share/doc/snort : The documentation for the installed version of Snort.
Changing the install location of Snort
If you would rather have all these folders install to a more normal location (/usr/local) , add ‑‑prefix=/usr/local/ to the ./configure command when preparing to build Snort. This will install all these folders under the path you choose. You also need to modify some of the other paths detailed above, so if you decide to install in that manner, you should follow the install instructions detailed in the Snort blog.
Snort Rules
Snort3 rules have more options than Snort 2 rules, and while the normal rules downloaded with PulledPork or manually will work, for testing you will probably want to download the set of community rules specificallycreated for snort3. You can manually download snort3 specific community rules from the snort website:
cd ~/snort_src/ wget https://www.snort.org/downloads/community/snort3-community-rules.tar.gz tar -xvzf snort3-community-rules.tar.gz cd snort3-community-rules sudo mkdir /opt/snort/etc/snort/rules sudo cp snort3-community.rules /opt/snort/etc/snort/rules/ sudo cp sid-msg.map /opt/snort/etc/snort/rules/
now test that snort can load these rules:
/opt/snort/bin/snort -c /opt/snort/etc/snort/snort.lua -R /opt/snort/etc/snort/rules/snort3-community.rules
your output should contain something similar:
Loading rules: Loading /opt/snort/etc/snort/rules/snort3-community.rules: Finished /opt/snort/etc/snort/rules/snort3-community.rules. Finished rules. -------------------------------------------------- rule counts total rules loaded: 3462 text rules: 3462 option chains: 3462 chain headers: 264 -------------------------------------------------- ...
you may want to run Snort with the following flags to detect issues: the warn-all and pedantic flags. From the Snort3 manual:
Warnings are not emitted unless –warn-* is specified. –warn-all enables all warnings, and –pedantic makes such warnings fatal
Installing OpenAppID
OpenAppID allows for the identification of application layer traffic. The Snort team has put together a package of detectors, with assistance from the community that you can download and install, called the Application Detector Package which needs to be installed. First download the OpenAppID detector package:
cd ~/snort_src/ wget https://www.snort.org/downloads/openappid/5759 -O OpenAppId-6329 tar -xzvf OpenAppId-6329 sudo cp -R odp /opt/snort/lib/
Now we need to edit our snort configuration file to point to this odp directory:
sudo vi /opt/snort/etc/snort/snort.lua
At line 113 (yours line number may be slightly different) you will see the appid= entry. You will want to add the app detector dir option here, pointing to the parent folder of the odf folder. It should look like this:
appid =
{
app_detector_dir = '/opt/snort/lib',
}
note that you must have four spaces (not a tab) for the indented line. Now we want to test the configuration file loads correctly:
/opt/snort/bin/snort -c /opt/snort/etc/snort/snort.lua --warn-all
you should see output similar to:
... Snort successfully validated the configuration (with 0 warnings). o")~ Snort exiting
Now to load Snort with the OpenAppID detectors, as well as all rules (we omit the pedantic flag, since the rules will throw flowbit warnings that are non fatal and can be ignored:)
/opt/snort/bin/snort -c /opt/snort/etc/snort/snort.lua -R /opt/snort/etc/snort/rules/snort3-community.rules --warn-all
Create a simple rule to test that OpenAppID is working correctly:
sudo touch /opt/snort/etc/snort/rules/local.rules sudo vi /opt/snort/etc/snort/rules/local.rules
with the following content:
alert tcp any any -> any any ( msg:"Facebook trafic Seen"; appids:"Facebook";sid:10000001; )
test to make sure the rule loads correctly:
/opt/snort/bin/snort -c /opt/snort/etc/snort/snort.lua -R /opt/snort/etc/snort/rules/local.rules --warn-all
You should see one rule loaded successfully. Now let’s run snort in detection mode on an interface (change eth0 below to match your interface name), printing alerts to the console:
sudo /opt/snort/bin/snort -c /opt/snort/etc/snort/snort.lua -R /opt/snort/etc/snort/rules/local.rules -i eth0 -A alert_fast -k none
the -k none flag tells Snort to ignore bad checksums. the Stream and Frag decoders will drop packets that have bad checksums, and the packets will not get processed by the OpenAppID detectors. By including this flag, we ensure that a packet with a bad checksum still gets processed. Now from another window on that computer (open a new terminal window or a second ssh session), use wget
to connect to facebook:
wget facebook.com
from the first console window you will see alerts output similar to the following:
08/05−19:13:45.451834 [∗∗] [1:10000001:0] ”Facebook trafic Seen” [∗∗] [Priority: 0] [AppID: Facebook] {TCP} 157.240.1.35:443 −> 10.0.0.104:33882
08/05−19:13:45.451842 [∗∗] [1:10000001:0] ”Facebook trafic Seen” [∗∗] [Priority: 0] [AppID: Facebook] {TCP} 10.0.0.104:33882 −> 157.240.1.35:443
use ctrl-c to stop Snort.
Note: if you are collecting packets with a larger MTU that the standard MTU for your adapter (VLAN tagged packets, MPLS Packets, packets from a different network type with a larger MTU), you may need to use the –snaplen flag to adjust snort to process larger packets)
Where to go from here
If you want to learn more about how to run the 2.9.9.x version of Snort, and how to install additional software to enhance a Snort system, see my series on installing Snort on Ubuntu. If you want to develop Snort plugins, please see my guide: Installing Snort++ Example Plugins.
I would love to get feedback from you about this guide. Recommendations, issues, or ideas, please contact me here.
Installing Snort++ Example Plugins
Updated June 15, 2018: Snort extras has moved to its own github repository. This guide has been updated to reflect that.
In this article, I will show you how to install and run the Example Plugins in Snort++. The example plugins are a series of additional plugins that the Snort team has made available for developers to use as examples. This guide assumes that you have cloned Snort++ (Snort 3.0 alpha) from github following my guide here.
These instructions are tested with Ubuntu 14, 16, and 18, with Snort 3.0 Alpha Build-245. If you have installed Snort++ from the github clone on another distribution or architecture, you should be able to modify the below instructions for your specific case. You should ensure that Snort is working correctly before you continue.
Assuming you’ve followed my Snort++ installation guide, we start by navigating to the ~/snort_src/ directory, git-clone the repository, compile, and install:
cd ~/snort_src/ git clone https://github.com/snort3/snort3_extra.git cd ./snort3_extra/ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig/ ./configure_cmake.sh --prefix=/usr/local cd build make sudo make install
We export PKG_CONFIG_PATH in the above instructions because that tells Snort where the pkgconfig file is for snort when we installed it. if you installed snort to another location, you’ll need to use that path. The above instructions compile and install the plugins, but you need to tell Snort where those plugin directories are when you run it. There are two types of plugins: compiled applications and Lua scripts, and you need to tell snort where to look for both types of extra plugins. To tell snort about these new directories containing the compiled plugins and lua plugins, you pass it the plugin path and/or the script path.
For example, to load the alert_ex plugin (compiled):
snort --plugin-path /opt/snort/lib/snort_extra -A alert_ex --warn-all
I’m using the the warn-all flag to warn of any errors, since snort will not display non-fatal errors by default.
If you want to test the lualert plugin (lua script):
snort --script-path /opt/snort/lib/snort_extra -A lualert --warn-all
To see all available plugins that snort is aware of when running, use the following command:
snort --list-plugins
this will not list the snort-extras plugins, since you haven’t given snort the plugin-path or script-path information. To have snort list all the new script and compiled plugins including those it can see in the extras directory:
snort --script-path /opt/snort/lib/snort_extra --plugin-path /opt/snort/lib/snort_extra --list-plugins
See the snort3 extras readme and the included source files for more information. To see the number of new plugins the extras folder makes available, let’s show all the enabled logging modules by default, then with the additional extras enabled:
noah@snort3:~$ snort --list-plugins | grep logger logger::alert_csv v0 static logger::alert_fast v0 static logger::alert_full v0 static logger::alert_json v0 static logger::alert_sfsocket v0 static logger::alert_syslog v0 static logger::alert_unixsock v0 static logger::log_codecs v0 static logger::log_hext v0 static logger::log_pcap v0 static logger::unified2 v0 static noah@snort3:~$ snort --script-path /usr/local/lib/snort\_extra \ --plugin-path /usr/local/lib/snort\_extra --list-plugins | grep logger logger::alert_csv v0 static logger::alert_ex v0 /usr/local/lib/snort_extra/loggers/alert_ex.so logger::alert_fast v0 static logger::alert_full v0 static logger::alert_json v0 static logger::alert_sfsocket v0 static logger::alert_syslog v0 static logger::alert_unixsock v0 static logger::log_codecs v0 static logger::log_hext v0 static logger::log_null v0 /usr/local/lib/snort_extra/loggers/log_null.so logger::log_pcap v0 static logger::lualert v0 static logger::unified2 v0 static
As you can see, when we tell snort to look in the extras directory, we now have additional plugins available for use.
I hope you found this article helpful. If you have issues, you can contact the snort-developers list for assistance. I would love to get feedback from you about this guide. Recommendations, issues, or ideas, please contact me here.
Installing Snort++ in Ubuntu (Version 3.0 Alpha 4 build 239)
NOTE: this article is out of date and has been replaced with a newer article: Installing Snort++ (Snort 3 Alpha 4 build 245) in Ubuntu
The instructions below show how to install Snort 3 alpha 4 build 239 on Ubuntu. This install has been tested on Ubuntu 14 and 16, for both the x86 and x64 architectures. For an outdated Ubuntu 12 version of these instructions, please go here. Note that Snort 3 is Alpha software, and therefore has bugs and issues, and should be installed for testing purposes only (not on production systems).
Snort 3 Alpha 4 Build 239 was released on July 28, 2017, and this guide has been tested with that version (releases after this specific release may not follow the same steps). Generic build instructions, prerequisites, and detailed notes are available in the manual.
If you want a more in-depth explanation of the install steps, which are very similar to the 2.9.9.x version of Snort, as well as instructions on how to configure and enhance Snort’s functionality, see my series on installing Snort 2.9.9.x on Ubuntu.
So let’s get started. First we need to install all the Snort pre-requisites from the Ubuntu repositories:
sudo apt-get install -y build-essential autotools-dev libdumbnet-dev libluajit-5.1-dev libpcap-dev libpcre3-dev zlib1g-dev pkg-config libhwloc-dev cmake
Install the optional (recommended) software:
sudo apt-get install -y liblzma-dev openssl libssl-dev cpputest libsqlite3-dev
Install tools required for compiling the source from github:
sudo apt-get install -y libtool git autoconf
Install the DAQ pre-requisites:
sudo apt-get install -y bison flex
If you want to build the documentation as well (not really needed, unless you want it, usually about 700 MB of libraries):
sudo apt-get install -y asciidoc dblatex source-highlight
Next we will create a directory to save the downloaded tarball files:
mkdir ~/snort_src cd ~/snort_src
First and install safec for runtime bounds checks on certain legacy C-library calls (this is optional but recommended):
cd ~/snort_src wget http://downloads.sourceforge.net/project/safeclib/libsafec-10052013.tar.gz tar -xzvf libsafec-10052013.tar.gz cd libsafec-10052013 ./configure make sudo make install
One of the Snort recommended prerequisites is Hyperscan 4.4.0. From their webpage: “Hyperscan is a regular expression engine designed to offer high performance, the ability to match multiple expressions simultaneously and flexibility in scanning operation.” Hyperscan needs Ragel 6.9 and the Boost header libraries.
Install Ragel 6.10 from source:
cd ~/snort_src wget http://www.colm.net/files/ragel/ragel-6.10.tar.gz tar -xzvf ragel-6.10.tar.gz cd ragel-6.10 ./configure make sudo make install
Download the Boost 1.64 libraries, but do not install:
cd ~/snort_src wget https://dl.bintray.com/boostorg/release/1.64.0/source/boost_1_64_0.tar.gz tar -xvzf boost_1_64_0.tar.gz
Install Hyperscan 4.5.1 from source, referencing the location of the Boost source directory:
cd ~/snort_src wget https://github.com/01org/hyperscan/archive/v4.5.2.tar.gz tar -xvzf v4.5.2.tar.gz mkdir ~/snort_src/hyperscan-4.5.2-build cd hyperscan-4.5.2-build/ cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DBOOST_ROOT=~/snort_src/boost_1_64_0/ ../hyperscan-4.5.2 make sudo make install
If you want to test that Hyperscan works, from the build directory, run:
cd ~/snort_src/hyperscan-4.5.2-build/ ./bin/unit-hyperscan
The unit tests will run (this takes a few minutes).
Download and install Data AcQuisition library (DAQ) from the Snort website (note that DAQ for Snort 3 is a different DAQ than for the 2.9.9.x series of Snort):
cd ~/snort_src wget https://www.snort.org/downloads/snortplus/daq-2.2.2.tar.gz tar -xvzf daq-2.2.2.tar.gz cd daq-2.2.2 ./configure make sudo make install
Run the following command to update shared libraries:
sudo ldconfig
Now we are ready to install Snort from source. This command downloads and installs the latest version of Snort 3 (currently 3.0.0 Alpha 4, build 239, but as the codebase is updated, you’ll get a newer version). If you want to specifically download the version used in this guide, use this URL instead with wget below: https://github.com/snortadmin/snort3/archive/BUILD_239.tar.gz.
If you want to install all the snort directories under a single directory, see the section at the bottom of this document titled Changing the install location of Snort. Here we choose to install the entire Snort directory structure to a single folder under /opt/:
cd ~/snort_src wget https://github.com/snortadmin/snort3/archive/master.tar.gz tar -xvzf master.tar.gz cd snort3-master/ autoreconf -isvf ./configure --prefix=/opt/snort make sudo make install
Since the Snort installation places the Snort binary at /opt/snort/bin/snort, it is common to create a symlink to /usr/sbin/snort:
sudo ln -s /opt/snort/bin/snort /usr/sbin/snort
Snort 3 requires a few environmental variables, we store them temporarily in the current session so we can continue working, and save them permanently to the ~/.bashrc file (you’ll need to do this for every user profile):
export LUA_PATH=/opt/snort/include/snort/lua/\?.lua\;\; export SNORT_LUA_PATH=/opt/snort/etc/snort sh -c "echo 'export LUA_PATH=/opt/snort/include/snort/lua/\?.lua\;\;' >> ~/.bashrc" sh -c "echo 'export SNORT_LUA_PATH=/opt/snort/etc/snort' >> ~/.bashrc"
to ensure that these two environmental variables are available when using sudo, we need to add them to the /etc/sudoers file:
sudo visudo
in the editor, add the following to to the bottom of the file:
Defaults env_keep += "LUA_PATH SNORT_LUA_PATH"
use ctrl-x to exit, save when prompted by pressing y, then press enter to save the file to /etc/sudoers.tmp (which will get copied automatically to /etc/sudoers).
The last step of our Snort installation is to test that the Snort Binary runs. Execute Snort with the -V flag, which causes Snort to print the current version. You should see output similar to the following:
user@snort3:~$ snort -V
,,_ -*> Snort++ <*-
o" )~ Version 3.0.0-a4 (Build 239) from 2.9.8-383
'''' By Martin Roesch & The Snort Team
http://snort.org/contact#team
Copyright (C) 2014-2017 Cisco and/or its affiliates. All rights reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using DAQ version 2.2.2
Using libpcap version 1.7.4
Using LuaJIT version 2.0.4
Using PCRE version 8.38 2015-11-23
Using ZLIB version 1.2.8
Using LZMA version 5.1.0alpha
Using OpenSSL 1.0.2g 1 Mar 2016
Using Hyperscan version 4.5.2 2017-08-05
user@snort3:~$
A note on install locations:
When you install snort to /opt/snort, you get the following folder structure:
user@snort3x86:/opt/snort$ tree /opt/snort -L 3
/opt/snort
├── bin
│ ├── snort
│ ├── snort2lua
│ ├── u2boat
│ └── u2spewfoo
├── etc
│ └── snort
│ ├── file_magic.lua
│ ├── snort_defaults.lua
│ └── snort.lua
├── include
│ └── snort
│ ├── actions
│ ├── codecs
│ ├── daqs
│ ├── decompress
│ ├── detection
│ ├── events
│ ├── file_api
│ ├── flow
│ ├── framework
│ ├── hash
│ ├── log
│ ├── lua
│ ├── main
│ ├── managers
│ ├── mime
│ ├── packet_io
│ ├── profiler
│ ├── protocols
│ ├── pub_sub
│ ├── search_engines
│ ├── sfip
│ ├── stream
│ ├── time
│ └── utils
├── lib
│ ├── pkgconfig
│ │ └── snort.pc
│ └── snort
│ └── daqs
└── share
└── doc
└── snort
36 directories, 8 files
The /opt/snort/bin folder contains the following Snort binaries:
- snort : The Snort binary.
- snort2lua : Tool to convert a Snort 2.9.8.x configuration file into a 3.x configuration file. More notes here.
- u2boat : U2boat is a tool for converting unified2 files into different formats.
- u2spewfoo: U2SpewFoo is a lightweight tool for dumping the contents of unified2 files to stdout.
Additionally, the following folders are created / used:
- /opt/snort/bin : Binaries for Snort and supporting software.
- /opt/snort/etc/snort : The configuration files for Snort.
- /opt/snort/include/snort : All include files for Snort.
- /opt/snort/lib/pkgconfig : The pkgconfig file for Snort (compilation details for Snort).
- /opt/snort/share/doc/snort : The documentation for the installed version of Snort.
Changing the install location of Snort
If you would rather have all these folders install to a more normal location (/usr/local) , add ‑‑prefix=/usr/local/ to the ./configure command when preparing to build Snort. This will install all these folders under the path you choose. You also need to modify some of the other paths detailed above, so if you decide to install in that manner, you should follow the install instructions detailed in the Snort blog.
Snort Rules
Snort3 rules have more options than Snort 2 rules, and while the normal rules downloaded with PulledPork or manually will work, for testing you will probably want to download the set of community rules specificallycreated for snort3. You can manually download snort3 specific community rules from the snort website:
cd ~/snort_src/ wget https://www.snort.org/downloads/community/snort3-community-rules.tar.gz tar -xvzf snort3-community-rules.tar.gz cd snort3-community-rules sudo mkdir /opt/snort/etc/snort/rules sudo cp snort3-community.rules /opt/snort/etc/snort/rules/ sudo cp sid-msg.map /opt/snort/etc/snort/rules/
now test that snort can load these rules:
/opt/snort/bin/snort -c /opt/snort/etc/snort/snort.lua -R /opt/snort/etc/snort/rules/snort3-community.rules
your output should contain something similar:
Loading rules: Loading /opt/snort/etc/snort/rules/snort3-community.rules: Finished /opt/snort/etc/snort/rules/snort3-community.rules. Finished rules. -------------------------------------------------- rule counts total rules loaded: 3462 text rules: 3462 option chains: 3462 chain headers: 264 -------------------------------------------------- ...
you may want to run Snort with the following flags to detect issues: the warn-all and pedantic flags. From the Snort3 manual:
Warnings are not emitted unless –warn-* is specified. –warn-all enables all warnings, and –pedantic makes such warnings fatal
Installing OpenAppID
OpenAppID allows for the identification of application layer traffic. The Snort team has put together a package of detectors, with assistance from the community that you can download and install, called the Application Detector Package which needs to be installed. First download the OpenAppID detector package:
cd ~/snort_src/ wget https://www.snort.org/downloads/openappid/5759 -O OpenAppId-5759 tar -xzvf OpenAppId-5759 sudo cp -R odp /opt/snort/lib/
Now we need to edit our snort configuration file to point to this odp directory:
sudo vi /opt/snort/etc/snort/snort.lua
At line 113 (yours line number may be slightly different) you will see the appid= entry. You will want to add the app detector dir option here, pointing to the parent folder of the odf folder. It should look like this:
appid =
{
app_detector_dir = '/opt/snort/lib',
}
note that you must have four spaces (not a tab) for the indented line. Now we want to test the configuration file loads correctly:
/opt/snort/bin/snort -c /opt/snort/etc/snort/snort.lua --warn-all
you should see output similar to:
... Snort successfully validated the configuration (with 0 warnings). o")~ Snort exiting
Now to load Snort with the OpenAppID detectors, as well as all rules (we omit the pedantic flag, since the rules will throw flowbit warnings that are non fatal and can be ignored:)
/opt/snort/bin/snort -c /opt/snort/etc/snort/snort.lua -R /opt/snort/etc/snort/rules/snort3-community.rules --warn-all
Create a simple rule to test that OpenAppID is working correctly:
sudo touch /opt/snort/etc/snort/rules/local.rules sudo vi /opt/snort/etc/snort/rules/local.rules
with the following content:
alert tcp any any -> any any ( msg:"Facebook trafic Seen"; appids:"Facebook";sid:10000001; )
test to make sure the rule loads correctly:
/opt/snort/bin/snort -c /opt/snort/etc/snort/snort.lua -R /opt/snort/etc/snort/rules/local.rules --warn-all
You should see one rule loaded successfully. Now let’s run snort in detection mode on an interface (change eth0 below to match your interface name), printing alerts to the console:
sudo /opt/snort/bin/snort -c /opt/snort/etc/snort/snort.lua -R /opt/snort/etc/snort/rules/local.rules -i eth0 -A alert_fast -k none
the -k none flag tells Snort to ignore bad checksums. the Stream and Frag decoders will drop packets that have bad checksums, and the packets will not get processed by the OpenAppID detectors. By including this flag, we ensure that a packet with a bad checksum still gets processed. Now from another window on that computer (open a new terminal window or a second ssh session), use wget
to connect to facebook:
wget facebook.com
from the first console window you will see alerts output similar to the following:
08/05−19:13:45.451834 [∗∗] [1:10000001:0] ”Facebook trafic Seen” [∗∗] [Priority: 0] [AppID: Facebook] {TCP} 157.240.1.35:443 −> 10.0.0.104:33882
08/05−19:13:45.451842 [∗∗] [1:10000001:0] ”Facebook trafic Seen” [∗∗] [Priority: 0] [AppID: Facebook] {TCP} 10.0.0.104:33882 −> 157.240.1.35:443
use ctrl-c to stop Snort.
Note: if you are collecting packets with a larger MTU that the standard MTU for your adapter (VLAN tagged packets, MPLS Packets, packets from a different network type with a larger MTU), you may need to use the –snaplen flag to adjust snort to process larger packets)
Where to go from here
If you want to learn more about how to run the 2.9.9.x version of Snort, and how to install additional software to enhance a Snort system, see my series on installing Snort on Ubuntu. If you want to develop Snort plugins, please see my guide: Installing Snort++ Example Plugins.
I would love to get feedback from you about this guide. Recommendations, issues, or ideas, please contact me here.
Installing Snort++ in Ubuntu (Version 3.0 Alpha 4 build 237)
NOTE: this article is out of date and has been replaced with a newer article: Installing Snort++ (Snort 3 Alpha 4 build 240) in Ubuntu
The instructions below show how to install Snort 3 alpha 4 build 237 on Ubuntu. This install has been tested on Ubuntu 14 and 16, for both the x86 and x64 architectures. For an outdated Ubuntu 12 version of these instructions, please go here. Note that Snort 3 is Alpha software, and therefore has bugs and issues, and should be installed for testing purposes only (not on production systems).
Snort 3 Alpha 4 Build 237 was released on July 13, 2017, and this guide has been tested with that version (releases after this specific release may not follow the same steps). Generic build instructions, prerequisites, and detailed notes are available in the manual.
If you want a more in-depth explanation of the install steps, which are very similar to the 2.9.9.x version of Snort, as well as instructions on how to configure and enhance Snort’s functionality, see my series on installing Snort 2.9.9.x on Ubuntu.
So let’s get started. First we need to install all the Snort pre-requisites from the Ubuntu repositories:
sudo apt-get install -y build-essential autotools-dev libdumbnet-dev libluajit-5.1-dev libpcap-dev libpcre3-dev zlib1g-dev pkg-config libhwloc-dev cmake
Install the optional (recommended) software:
sudo apt-get install -y liblzma-dev openssl libssl-dev cpputest cmake libsqlite3-dev
Install tools required for compiling the source from github:
sudo apt-get install -y libtool git autoconf
Install the DAQ pre-requisites:
sudo apt-get install -y bison flex
If you want to build the documentation as well (not really needed, unless you want it, usually about 700 MB of libraries):
sudo apt-get install -y asciidoc dblatex source-highlight
Next we will create a directory to save the downloaded tarball files:
mkdir ~/snort_src cd ~/snort_src
First and install safec for runtime bounds checks on certain legacy C-library calls (this is optional but recommended):
cd ~/snort_src wget http://downloads.sourceforge.net/project/safeclib/libsafec-10052013.tar.gz tar -xzvf libsafec-10052013.tar.gz cd libsafec-10052013 ./configure make sudo make install
One of the Snort recommended prerequisites is Hyperscan 4.4.0. From their webpage: “Hyperscan is a regular expression engine designed to offer high performance, the ability to match multiple expressions simultaneously and flexibility in scanning operation.” Hyperscan needs Ragel 6.9 and the Boost header libraries.
Install Ragel 6.10 from source:
cd ~/snort_src wget http://www.colm.net/files/ragel/ragel-6.10.tar.gz tar -xzvf ragel-6.10.tar.gz cd ragel-6.10 ./configure make sudo make install
Download the Boost 1.64 libraries, but do not install:
cd ~/snort_src wget https://dl.bintray.com/boostorg/release/1.64.0/source/boost_1_64_0.tar.gz tar -xvzf boost_1_64_0.tar.gz
Install Hyperscan 4.5.1 from source, referencing the location of the Boost source directory:
cd ~/snort_src wget https://github.com/01org/hyperscan/archive/v4.5.1.tar.gz tar -xvzf v4.5.1.tar.gz mkdir ~/snort_src/hyperscan-4.5.1-build cd hyperscan-4.5.1-build/ cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DBOOST_ROOT=~/snort_src/boost_1_64_0/ ../hyperscan-4.5.1 make sudo make install
If you want to test that Hyperscan works, from the build directory, run:
cd ~/snort_src/hyperscan-4.5.1-build/ ./bin/unit-hyperscan
The unit tests will run (this takes a few minutes).
Download and install Data AcQuisition library (DAQ) from the Snort website (note that DAQ for Snort 3 is a different DAQ than for the 2.9.9.x series of Snort):
cd ~/snort_src wget https://www.snort.org/downloads/snortplus/daq-2.2.1.tar.gz tar -xvzf daq-2.2.1.tar.gz cd daq-2.2.1 ./configure make sudo make install
Run the following command to update shared libraries:
sudo ldconfig
Now we are ready to install Snort from source. This command downloads and installs the latest version of Snort 3 (currently 3.0.0 Alpha 4, build 237, but as the codebase is updated, you’ll get a newer version). If you want to specifically download the version used in this guide, use this URL instead with wget below: https://github.com/snortadmin/snort3/archive/3376324350b3ef6228c4e30799a22779413789c2.tar.gz.
If you want to install all the snort directories under a single directory, see the section at the bottom of this document titled Changing the install location of Snort. Here we choose to install the entire Snort directory structure to a single folder under /opt/:
cd ~/snort_src wget https://github.com/snortadmin/snort3/archive/master.tar.gz tar -xvzf master.tar.gz cd snort3-master/ autoreconf -isvf ./configure --prefix=/opt/snort make sudo make install
Since the Snort installation places the Snort binary at /opt/snort/bin/snort, it is common to create a symlink to /usr/sbin/snort:
sudo ln -s /opt/snort/bin/snort /usr/sbin/snort
Snort 3 requires a few environmental variables, we store them temporarily in the current session so we can continue working, and save them permanently to the ~/.bashrc file (you’ll need to do this for every user profile):
export LUA_PATH=/opt/snort/include/snort/lua/\?.lua\;\; export SNORT_LUA_PATH=/opt/snort/etc/snort sh -c "echo 'export LUA_PATH=/opt/snort/include/snort/lua/\?.lua\;\;' >> ~/.bashrc" sh -c "echo 'export SNORT_LUA_PATH=/opt/snort/etc/snort' >> ~/.bashrc"
to ensure that these two environmental variables are available when using sudo, we need to add them to the /etc/sudoers file:
sudo visudo
in the editor, add the following to to the bottom of the file:
Defaults env_keep += "LUA_PATH SNORT_LUA_PATH"
use ctrl-x to exit, save when prompted by pressing y, then press enter to save the file to /etc/sudoers.tmp (which will get copied automatically to /etc/sudoers).
The last step of our Snort installation is to test that the Snort Binary runs. Execute Snort with the -V flag, which causes Snort to print the current version. You should see output similar to the following:
user@snort3:~$ snort -V
,,_ -*> Snort++ <*-
o" )~ Version 3.0.0-a4 (Build 237) from 2.9.8-383
'''' By Martin Roesch & The Snort Team
http://snort.org/contact#team
Copyright (C) 2014-2017 Cisco and/or its affiliates. All rights reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using DAQ version 2.2.1
Using libpcap version 1.7.4
Using LuaJIT version 2.0.4
Using PCRE version 8.38 2015-11-23
Using ZLIB version 1.2.8
Using LZMA version 5.1.0alpha
Using OpenSSL 1.0.2g 1 Mar 2016
Using Hyperscan version 4.5.1 2017-07-18
user@snort3:~$
Now let’s test snort with the default configuration file and ruleset:
user@snort3:~$ /opt/snort/bin/snort -c /opt/snort/etc/snort/snort.lua -R /opt/snort/etc/snort/sample.rules
--------------------------------------------------
o")~ Snort++ 3.0.0-a4-237
--------------------------------------------------
Loading /opt/snort/etc/snort/snort.lua:
ssh
pop
stream_tcp
gtp_inspect
stream_icmp
ftp_server
stream_udp
ips
http_inspect
wizard
file_id
ftp_data
smtp
back_orifice
port_scan
telnet
ssl
sip
rpc_decode
reputation
classifications
arp_spoof
appid
stream_user
stream_ip
stream
dnp3
ftp_client
references
dns
imap
stream_file
Finished /opt/snort/etc/snort/snort.lua.
Loading rules:
Loading /opt/snort/etc/snort/sample.rules:
Finished /opt/snort/etc/snort/sample.rules.
Finished rules.
--------------------------------------------------
rule counts
total rules loaded: 3974
text rules: 3974
option chains: 3974
chain headers: 187
--------------------------------------------------
port rule counts
tcp udp icmp ip
any 119 31 29 26
src 1685 4 0 0
dst 1927 232 0 0
both 0 1 0 0
total 3731 268 29 26
--------------------------------------------------
flowbits
defined: 124
not checked: 9
not set: 2
--------------------------------------------------
service rule counts - tcp to-srv to-cli
dcerpc: 3 0
drda: 3 0
ftp: 8 2
http: 1161 1553
ident: 1 0
imap: 29 1044
ircd: 1 1
ldap: 5 0
mysql: 29 1
netbios-ns: 4 0
netbios-ssn: 25 7
pop3: 12 1043
rtsp: 2 0
smtp: 550 3
ssl: 5 1
sunrpc: 9 0
telnet: 5 1
vnc-server: 1 3
total: 1853 3659
--------------------------------------------------
service rule counts - udp to-srv to-cli
dcerpc: 2 0
dns: 170 2
kerberos: 4 4
netbios-dgm: 7 7
netbios-ns: 1 1
netbios-ssn: 1 1
ntp: 1 0
snmp: 1 1
ssdp: 8 0
sunrpc: 7 3
tftp: 1 0
total: 203 19
--------------------------------------------------
fast pattern port groups src dst any
packet: 27 125 4
key: 23 107 1
header: 23 107 1
body: 0 2 0
file: 23 107 1
--------------------------------------------------
fast pattern service groups to-srv to-cli
packet: 29 17
key: 3 0
header: 1 3
body: 1 0
file: 1 4
--------------------------------------------------
search engine
instances: 565
patterns: 24659
pattern chars: 515569
num states: 409078
num match states: 23799
memory scale: MB
total memory: 10.6222
pattern memory: 1.43181
match list memory: 3.91914
transition memory: 5.20227
--------------------------------------------------
pcap DAQ configured to passive.
Snort successfully validated the configuration.
o")~ Snort exiting
user@snort3:~$
If you have output similar to the above, then Snort 3.0.0 Alpha 4 is installed and works.
A note on install locations:
When you install snort to /opt/snort, you get the following folder structure:
user@snort3x86:/opt/snort$ tree /opt/snort -L 3
/opt/snort
├── bin
│ ├── snort
│ ├── snort2lua
│ ├── u2boat
│ └── u2spewfoo
├── etc
│ └── snort
│ ├── file_magic.lua
│ ├── sample.rules
│ ├── snort_defaults.lua
│ └── snort.lua
├── include
│ └── snort
│ ├── actions
│ ├── codecs
│ ├── daqs
│ ├── decompress
│ ├── detection
│ ├── events
│ ├── file_api
│ ├── flow
│ ├── framework
│ ├── hash
│ ├── log
│ ├── lua
│ ├── main
│ ├── managers
│ ├── mime
│ ├── packet_io
│ ├── profiler
│ ├── protocols
│ ├── search_engines
│ ├── sfip
│ ├── stream
│ ├── time
│ └── utils
├── lib
│ ├── pkgconfig
│ │ └── snort.pc
│ └── snort
│ └── daqs
└── share
└── doc
└── snort
35 directories, 9 files
The /opt/snort/bin folder contains the following Snort binaries:
- snort : The Snort binary.
- snort2lua : Tool to convert a Snort 2.9.8.x configuration file into a 3.x configuration file. More notes here.
- u2boat : U2boat is a tool for converting unified2 files into different formats.
- u2spewfoo: U2SpewFoo is a lightweight tool for dumping the contents of unified2 files to stdout.
Additionally, the following folders are created / used:
- /opt/snort/bin : Binaries for Snort and supporting software.
- /opt/snort/etc/snort : The configuration files for Snort.
- /opt/snort/include/snort : All include files for Snort.
- /opt/snort/lib/pkgconfig : The pkgconfig file for Snort (compilation details for Snort).
- /opt/snort/share/doc/snort : The documentation for the installed version of Snort.
Changing the install location of Snort
If you would rather have all these folders install to a more normal location (/usr/local) , add ‑‑prefix=/usr/local/ to the ./configure command when preparing to build Snort. This will install all these folders under the path you choose. You also need to modify some of the other paths detailed above, so if you decide to install in that manner, you should follow the install instructions detailed in the Snort blog.
Where to go from here
If you want to learn more about how to run the 2.9.9.x version of Snort, and how to install additional software to enhance a Snort system, see my series on installing Snort on Ubuntu. If you want to work with protocol (layer 7) detection, please see my article on OpenAppID.
I would love to get feedback from you about this guide. Recommendations, issues, or ideas, please contact me here.
Snort IPS With NFQ (nfqueue) Routing on Ubuntu
Overview
This guide will show you how to configure Snort to run inline using the NFQUEUE DAQ (referred to as NFQ). This allows your Snort server to use iptables to route traffic between any number of subnets, with Snort evaluating all traffic passing through the system. This guide will assume some knowledge of routing and IP addressing, especially as it is implemented under Linux, as well as some experience setting up Snort. Configuring Snort as an inline NIPS with NFQ is more complicated than setting snort up as a NIDS, and is more complicated than setting up Snort as a NIPS using the AFPACKET DAQ. The complexity is due to the use of iptables and the need to understand IP routing. I will assume that you have installed Snort before, so I may gloss over some concepts and explanations in this guide. If you find yourself having difficult with installing and configuring Snort, I recommend you first work through my complete guide on installing Snort which should give you a good foundation on understanding how to install and configure Snort.
This guide has been written and tested on Ubuntu 16 x64. It should work on most currently supported versions of Ubuntu and Debian derivatives, but your mileage may vary.
NFQUEUE versus AFPACKET
On Ubuntu, you can run Snort two different ways in inline mode, with AFPACKET or with NFQ. AFPACKET is simpler to setup (see my guide here), but only lets you bridge sets of paired interfaces. This means you can bridge eth0 with eth1 (pass traffic between them), and also bridge eth2 with eth3, but you can not pass traffic between eth0 and eth4. A benefit of AFPACKET is that you can install an inline Snort machine without any changes to IP addressing, routing, or networking changes. NFQ on the other hand lets you leverage the power of iptables to make routing decisions. It’s more complicated to setup (it requires you as a systems administrator to understand Linux routing), but is more powerful for network security. You can also use iptables to blottck traffic before Snort even sees it (to filter out noisy traffic sources), or use additional tools that interface with iptables to increase security (like psad or fwsnort).
IPv6 Notes
Enabling NFQ with IPv6 is very similar to IPv4, the only thing to note is that you will have to run a separate instance of Snort to process IPv6 packets, and setup another iptables rule to forward IPv6 packets to a second NFQUEUE using the ip6tables command. Essentially you follow this guide twice, once for IPv4, and once for IPv6, making changes as necessary.
Steps
This guide will go through the following steps:
- Network Configuration
- Route Configuration
- Enable Kernel IP forwarding on the System
- Software Installation
- Software configuration
- Enabling NFQUEUE
- Enabling additional firewall rules
One thing to note: for simplicity, we will enable routing for all traffic passing through the system before we lock down the firewall rules. Because of this, I highly recommend that you test this on a development network to understand how it works before you implement on a production network. In a real environment, you would want your firewall rules in place before you enabled routing of traffic on a device that bridged networks with different security levels. For this guide, it’s much simpler to apply firewall rules at the end for the purpose of learning. I have noted a number of excellent online resources and books at the end of this guide to help understand networking, iptables, and network security.
Network Configuration
For this guide, we will use the following network, with our Snort router having three interfaces on three different networks:
The 172.16.0.0/24 network is our internal admin or management network, and will not be routed by Snort. No traffic should pass between the 172.16.0.0 network and the other networks. This is how we will SSH into this snort system for management purposes, and if configured: Snort could send log data back to a logging server on this network. It is good practice to keep a separate network (through VLANS or separate hardware) for management purposes. We won’t lock down this network at first, but will apply firewall rules at the end of this article to prevent traffic from entering this network across our Snort router.
The 10.0.0.0/24 network is our outbound network. All traffic not destined for a locally connected network will be directed out this network to the next hop gateway (in this case the ISP router). Note that the computer on this network has the Snort router as it’s default gateway. While this works, normally you’d configure the computer to have the actual external gateway (10.0.0.1 in this case) as the gateway. It’s set this way to simplify testing.
The 192.168.0.0/24 network is our client network (screened subnet). Hosts will be configured on this network to use the Snort server as it’s gateway. Traffic passing between this network and the 10.0.0.0 network will be scanned by Snort, and will be dropped and/or logged if suspicious.
Because this server is running Ubuntu 16, the interface names no longer follow the ethX standard (eth0, eth1, …). Instead, interfaces names are assigned as Predictable Network Interface Names. This means you need to check the names of your interfaces using ifconfig -a. In my case, what was originally eth0 is now ens160.
Since you are using your Snort system as a router, you’ll want static interfaces on each address. You don’t want dynamic (DCHP) addresses for the interfaces on this system because any clients on that subnet will use the IP address of the Snort system as their gateway. For any interface that Snort will process traffic on, you need to disable LRO and GRO (there’s an explanation of this in my complete guide on installing Snort). For any interface that sends traffic to an external network, you’ll need a gateway configured.
Edit /etc/network/interfaces as an admin:
sudo vi /etc/network/interfaces
Configure your network interfaces:
# Routed subnet auto ens160 iface ens160 inet static address 10.0.0.2 netmask 255.255.255.0 network 10.0.0.0 broadcast 10.0.0.255 gateway 10.0.0.1 post-up ethtool -K ens160 gro off post-up ethtool -K ens160 lro off # Host Subnet (internal network / screened subnet) auto ens224 iface ens224 inet static address 192.168.0.2 netmask 255.255.255.0 network 192.168.0.0 broadcast 192.168.0.255 # no gateway required post-up ethtool -K ens160 gro off post-up ethtool -K ens160 lro off # Management Subnet (internal network, not routed) auto ens192 iface ens192 inet static address 172.16.0.2 netmask 255.255.255.0 network 172.16.0.0 broadcast 172.16.0.255 # no gateway # snort not processing traffic on this interface, so no need to disable LRO or GRO
Setting up Routes
Because we don’t have additional subnets connected to the 192.1668.0.0 or 172.16.0.0 networks, we don’t need to specify any routes. We only configured one gateway on this system (10.0.0.1), so all traffic not destined to a local subnet will be sent to that gateway. To verify, use the ip route command:
noah@snort-router:~ $ ip route default via 10.0.0.1 dev ens160 onlink 10.0.0.0/24 dev ens160 proto kernel scope link src 10.0.0.2 172.16.0.0/24 dev ens192 proto kernel scope link src 172.16.0.2 192.168.0.0/24 dev ens224 proto kernel scope link src 192.168.0.2
If you have multiple subnets configured off of one of those other two subnets, you would need to add a gateway for that subnet in your interfaces file, and you would need to add a static route.
Sidebar: A more complicated example (don’t follow these steps unless you have more complicated routing needs): if you had the 192.168.99.0/24 subnet connected to your 192.168.0.0/24 subnet, reachable through 192.168.0.3, as shown highlighted in red in the image below:
you would need to add two commands to the network interface for ens224: One to specify a gateway, and one to add this route. Since you now have multiple gateways on your Snort server, you want to tell the system which gateway is the default gateway (the gateway for all traffic that it does not have an explicit route for). Interface ens224 on the snort server would now look like:
# Host Subnet (internal network / screened subnet) # Additional routing example auto ens224 iface ens224 inet static address 192.168.0.2 netmask 255.255.255.0 network 192.168.0.0 broadcast 192.168.0.255 gateway 192.168.0.3 post-up ethtool -K ens160 gro off post-up ethtool -K ens160 lro off up ip route add 192.168.99.0/24 via 192.168.0.3 || true
While ens160 with the default route will look like:
# Routed subnet # Additional routing example auto ens160 iface ens160 inet static address 10.0.0.2 netmask 255.255.255.0 network 10.0.0.0 broadcast 10.0.0.255 gateway 10.0.0.1 post-up ethtool -K ens160 gro off post-up ethtool -K ens160 lro off up ip route add default via ens160 || yes
At this time, you should verify that you can ping hosts in every network from your Snort server, as well as external hosts via your gateway. Use the ip route commmand as well as tracepath to verify this is setup right. Do not continue until all this is working.
Firewalls
At this point, even though the system can reach all connected networks, the system will not pass traffic between the networks. Traffic from a host on the management network can not reach any hosts on the screened network, and vice versa. This is by design. Enabling this is easy, however you may want to restrict traffic between certain networks before enabling this option, for security purposes. For simplicity, we will not enable firewall rules at this time, so if your system is a live system connected to both internal sensitive networks and public networks, you may want to read through the entire article first to understand how to add firewall rules to protect the system before enabling routing.
Enable Kernel IP forwarding
Use vi to edit /etc/sysctl.conf
Un-comment the line:
net.ipv4.ip_forward=1
Reboot the system. At this time, you should now be able to ping between two devices connected to different networks. If your routing is setup correctly, you should be able to ping out the default route as well. If pinging between locally connected subnets work, but you can’t ping to external subnets, you probably need to look at the routing further down the line, do those devices have a path back to your locally connected subnets? If you have a large network, you may want to look at enabling a routing protocol on your Snort server to exchange routing information (probably OSPF or RIP, depending on you network configuration and level of comfort).
Installing DAQ and Snort With NFQ support
If you have a Snort system up and running, you will probably need to re-compile DAQ and Snort for this guide. The reason is that to enable NFQ, you need to install libraries prior to compiling DAQ.
Start by installing all pre-requisites for Snort:
sudo apt-get install -y build-essential libpcap-dev libpcre3-dev libdumbnet-dev bison flex zlib1g-dev liblzma-dev openssl libssl-dev
We need the development libraries for Nghttp2. On Ubuntu 16 this is simple:
# Ubuntu 16 only (not Ubuntu 14) sudo apt-get install -y libnghttp2-dev
On Ubuntu 14, we do this from scratch:
# Ubuntu 14 only (not Ubuntu 16) sudo apt-get install -y autoconf libtool pkg-config cd ~/snort_src wget https://github.com/nghttp2/nghttp2/releases/download/v1.17.0/nghttp2-1.17.0.tar.gz tar -xzvf nghttp2-1.17.0.tar.gz cd nghttp2-1.17.0 autoreconf -i --force automake autoconf ./configure --enable-lib-only make sudo make install
Finally we need the NFQ specific libraries:
sudo apt-get install libnetfilter-queue-dev
We will make a directory for all the source tarball files:
mkdir ~/snort_src cd ~/snort_src
Download and install Data Acquisition library (DAQ) from the Snort website:
cd ~/snort_src wget https://www.snort.org/downloads/snort/daq-2.0.6.tar.gz tar -xvzf daq-2.0.6.tar.gz cd daq-2.0.6 ./configure make sudo make install
after the ./configure option, you will see the DAQ modules that are enabled. You must have NFQ enabled here, as seen below:
Build AFPacket DAQ module.. : yes Build Dump DAQ module...... : yes Build IPFW DAQ module...... : yes Build IPQ DAQ module....... : no Build NFQ DAQ module....... : yes <<<<< MUST BE YES Build PCAP DAQ module...... : yes Build netmap DAQ module.... : no
Now we are ready to install Snort from source:
cd ~/snort_src wget https://snort.org/downloads/snort/snort-2.9.9.0.tar.gz tar -xvzf snort-2.9.9.0.tar.gz cd snort-2.9.9.0 ./configure --enable-sourcefire make sudo make install
Run the following command to update shared libraries:
sudo ldconfig
Since the Snort installation places the Snort binary at /usr/local/bin/snort, it is common to create a symlink to /usr/sbin/snort:
sudo ln -s /usr/local/bin/snort /usr/sbin/snort
The last step of our Snort installation is to test that the Snort binary runs. Execute Snort with the -V flag, which causes Snort to print the current version. You should see output similar to the following:
noah@snort-router:~ $ snort -V
,,_ -*> Snort! <*-
o" )~ Version 2.9.9.0 GRE (Build 56)
'''' By Martin Roesch & The Snort Team: http://www.snort.org/contact#team
Copyright (C) 2014-2016 Cisco and/or its affiliates. All rights reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using libpcap version 1.7.4
Using PCRE version: 8.38 2015-11-23
Using ZLIB version: 1.2.8
noah@snort-router:~ $
Now we want to verify that Snort can see the NFQ DAQ. We do this by running snort with the –daq-list flag:
noah@snort-router:~ $ snort --daq-list Available DAQ modules: pcap(v3): readback live multi unpriv nfq(v7): live inline multi <<<<<< Need this line ipfw(v3): live inline multi unpriv dump(v3): readback live inline multi unpriv afpacket(v5): live inline multi unpriv
At this point, Snort has been compiled and installed with the NFQ DAQ. Next is configuration.
Configuring Snort
Most of this configuration comes from my longer articles on installing Snort. I’ll skip over some of the details of certain steps, but more in-depth information is available in my full Snort install guide. First off, for security reasons we want Snort to run as an unprivileged user. We create a snort user and group for this purpose:
sudo groupadd snort sudo useradd snort -r -s /sbin/nologin -c SNORT_IDS -g snort
Next, we need to create a number of files and folders that Snort expects when running in NIDS mode. We will then change the ownership of those files to our new snort user:
# Create the Snort directories: sudo mkdir /etc/snort sudo mkdir /etc/snort/rules sudo mkdir /etc/snort/rules/iplists sudo mkdir /etc/snort/preproc_rules sudo mkdir /usr/local/lib/snort_dynamicrules sudo mkdir /etc/snort/so_rules # Create some files that stores rules and ip lists sudo touch /etc/snort/rules/iplists/black_list.rules sudo touch /etc/snort/rules/iplists/white_list.rules sudo touch /etc/snort/rules/local.rules sudo touch /etc/snort/sid-msg.map # Create our logging directories: sudo mkdir /var/log/snort sudo mkdir /var/log/snort/archived_logs # Adjust permissions: sudo chmod -R 5775 /etc/snort sudo chmod -R 5775 /var/log/snort sudo chmod -R 5775 /var/log/snort/archived_logs sudo chmod -R 5775 /etc/snort/so_rules sudo chmod -R 5775 /usr/local/lib/snort_dynamicrules # Change Ownership on folders: sudo chown -R snort:snort /etc/snort sudo chown -R snort:snort /var/log/snort sudo chown -R snort:snort /usr/local/lib/snort_dynamicrules
We now need to move the following files from the extracted Snort tarball to the snort configuration folder:
cd ~/snort_src/snort-2.9.9.0/etc/ sudo cp *.conf* /etc/snort sudo cp *.map /etc/snort sudo cp *.dtd /etc/snort cd ~/snort_src/snort-2.9.9.0/src/dynamic-preprocessors/build/usr/local/lib/snort_dynamicpreprocessor/ sudo cp * /usr/local/lib/snort_dynamicpreprocessor/
Disable all rule files:
sudo sed -i 's/include \$RULE\_PATH/#include \$RULE\_PATH/' /etc/snort/snort.conf
Now We need to edit our snort configuration file:
sudo vi /etc/snort/snort.conf
Set your internal networks here. I am choosing to have Snort consider the 192.168.0.0 and 172.16.0.0 networks as internal networks. This maters for rule processing, where many rules look for attacks against computers in your HOME_NET subnet ranges:
# line 45 (no spaces between IP addresses) ipvar HOME_NET [192.168.0.0/24,176.16.0.0/24]
Next we need to tell Snort about the locations of all the folders we created earlier. I have included the line numbers after the hash so you can more easily find the setting (do not write the line number, just change the path to match what is below):
var RULE_PATH /etc/snort/rules # line 104 var SO_RULE_PATH /etc/snort/so_rules # line 105 var PREPROC_RULE_PATH /etc/snort/preproc_rules # line 106 var WHITE_LIST_PATH /etc/snort/rules/iplists # line 113 var BLACK_LIST_PATH /etc/snort/rules/iplists # line 114
Enable the Local rules file. Un-comment the following line (line 545) by deleting the hash from the beginning of the line:
include $RULE_PATH/local.rules
Now we need to configure Snort to use the NFQ DAQ an run inline mode. You can choose to pass these options from the command line, but it’s neater to do it here in the configuration file. Add the following three lines to enable the NFQ DAQ in inline mode, looking at NFQ queue number 4 for packets (we will configure iptables later to pass all routed packets to this same numbered queue):
# at line 168, add the following new lines: config daq: nfq config daq_mode: inline config daq_var: queue=4
Save the file, and now we need to test the configuration. To test: we run the following command. The T flag indicates we want to test, the c flag specifies the snort.conf file, and the Q flag tells snort that we’re working in inline mode (required later for Drop rules to actually drop, instead of just alerting):
$ sudo snort -T -c /etc/snort/snort.conf -Q
You should have similar output to the following (truncated for clarity):
Enabling inline operation
Running in Test mode
...
nfq DAQ configured to inline.
Decoding Raw IP4
--== Initialization Complete ==--
,,_ -*> Snort! <*-
o" )~ Version 2.9.9.0 GRE (Build 56)
'''' By Martin Roesch & The Snort Team: http://www.snort.org/contact#team
Copyright (C) 2014-2016 Cisco and/or its affiliates. All rights reserved.
Copyright (C) 1998-2013 Sourcefire, Inc., et al.
Using libpcap version 1.7.4
Using PCRE version: 8.38 2015-11-23
Using ZLIB version: 1.2.8
Rules Engine: SF_SNORT_DETECTION_ENGINE Version 3.0 <Build 1>
Preprocessor Object: SF_SIP Version 1.1 <Build 1>
Preprocessor Object: SF_SSLPP Version 1.1 <Build 4>
Preprocessor Object: SF_SDF Version 1.1 <Build 1>
Preprocessor Object: SF_SMTP Version 1.1 <Build 9>
Preprocessor Object: SF_GTP Version 1.1 <Build 1>
Preprocessor Object: SF_DNP3 Version 1.1 <Build 1>
Preprocessor Object: SF_DCERPC2 Version 1.0 <Build 3>
Preprocessor Object: SF_DNS Version 1.1 <Build 4>
Preprocessor Object: SF_POP Version 1.0 <Build 1>
Preprocessor Object: SF_IMAP Version 1.0 <Build 1>
Preprocessor Object: SF_MODBUS Version 1.1 <Build 1>
Preprocessor Object: SF_FTPTELNET Version 1.2 <Build 13>
Preprocessor Object: SF_SSH Version 1.1 <Build 3>
Preprocessor Object: SF_REPUTATION Version 1.1 <Build 1>
Snort successfully validated the configuration!
Snort exiting
If you are seeing similar output, then you’re in a good place. If not, troubleshoot the errors (usually Snort will output line number where the error was found) before continuing.
Testing a Rule
We will add a simple rule to detect ICMP traffic to verify that we’re detecting and passing traffic correctly. Add the following to our local.rules file:
/etc/snort/rules/local.rules:
alert icmp any any -> $HOME_NET any (msg:"ICMP test detected"; GID:1; sid:10000001; rev:001; classtype:icmp-event;)
Since we have made changes to the file that snort loads (local.rules), it is a good idea to test the configuration file again:
sudo snort -T -c /etc/snort/snort.conf -Q
If successful, you should be able to scroll up through the output and see that Snort has loaded our one rule:
+++++++++++++++++++++++++++++++++++++++++++++++++++ Initializing rule chains... 1 Snort rules read 1 detection rules 0 decoder rules 0 preprocessor rules 1 Option Chains linked into 1 Chain Headers 0 Dynamic rules +++++++++++++++++++++++++++++++++++++++++++++++++++ +-------------------[Rule Port Counts]--------------------------------------- | tcp udp icmp ip | src 0 0 0 0 | dst 0 0 0 0 | any 0 0 1 0 | nc 0 0 1 0 | s+d 0 0 0 0 +----------------------------------------------------------------------------
Configuring NFQUEUE
I will assume you don’t have any entries in your iptables at this time. You can verify with the following command:
$ sudo iptables -L
which should show the following output:
Chain INPUT (policy ACCEPT) target prot opt source destination Chain FORWARD (policy ACCEPT) target prot opt source destination Chain OUTPUT (policy ACCEPT) target prot opt source destination
We want to add a rule to our iptables that makes every packet that is going to be forwarded (not destined for local delivery) to be scanned by Snort. To do this, we add a single rule that moves all forwarded traffic (from the FORWARD queue) to NFQUEUE queue number 4, which is the same queue we specified in our snort.conf configuration above, where Snort is listening for it (I chose queue 4 arbitrarily, you could use any number, as long as it matched the snort.conf queue). To add this rule to the FORWARD chain, we run the following:
sudo iptables -I FORWARD -j NFQUEUE --queue-num=4
Since Snort is not running right now, the system will deliver traffic to that queue, but it will not be processed and will drop (we can change this later to a fail-open option if wanted). Let’s run Snort with the following flags to see traffic being processed:
-A console the console option prints fast mode alerts to stdout -q Quiet. Don't show banner and status report. -u snort run snort as the following user after startup -g snort run snort as the following group after startup -c /etc/snort/snort.conf the path to our snort.conf file -Q Run inline mode (required to drop packets)
Run Snort with the command below:
sudo /usr/local/bin/snort -A console -q -u snort -g snort -c /etc/snort/snort.conf -Q
If you ping from one machine to another machine, you should see alerts show on your snort machine. You can’t ping to or from the snort machine directly, because this rule only looks at traffic not destined for the local machine.
06/07-14:38:16.275283 [**] [1:10000001:1] ICMP test detected [**] [Classification: Generic ICMP event] [Priority: 3] {ICMP} 10.0.0.88 -> 192.168.0.10
06/07-14:38:17.274174 [**] [1:10000001:1] ICMP test detected [**] [Classification: Generic ICMP event] [Priority: 3] {ICMP} 10.0.0.88 -> 192.168.0.10
06/07-14:38:18.273215 [**] [1:10000001:1] ICMP test detected [**] [Classification: Generic ICMP event] [Priority: 3] {ICMP} 10.0.0.88 -> 192.168.0.10
You have to use ctrl-c to stop snort from running after the above output.
To test that snort will Drop rather than just alert on your rules, edit your local.rules and replace the first word “alert” with “drop”. Re-run Snort as above and try pinging again. You should see alerts, and your host’s should not be able to ping. Note that all other traffic is permitted, our rule is only blocking ICMP traffic. You should see alerts specifying the Drop action:
06/07-14:42:23.935433 [Drop] [**] [1:10000001:1] ICMP test detected [**] [Classification: Generic ICMP event] [Priority: 3] {ICMP} 10.0.0.88 -> 192.168.0.10
06/07-14:42:24.942280 [Drop] [**] [1:10000001:1] ICMP test detected [**] [Classification: Generic ICMP event] [Priority: 3] {ICMP} 10.0.0.88 -> 192.168.0.10
If you were to reboot your computer, you would loose all your iptables configurations. To make them persistent, we need to export the current configuration to a text file, and tell the system to load that text file each time the network starts up. To Export the current settings, we use the iptables-save tool as follows:
sudo sh -c "iptables-save > /etc/network/iptables.rules"
To load the settings, we use the iptables-restore tool in our network interfaces file (/etc/network/interfaces) for our primary network interface, so now the configuration for ens160 looks like this:
auto ens160 iface ens160 inet static address 10.0.0.2 netmask 255.255.255.0 network 10.0.0.0 broadcast 10.0.0.255 gateway 10.0.0.1 pre-up iptables-restore < /etc/network/iptables.rules post-up ethtool -K ens160 gro off post-up ethtool -K ens160 lro off
Reboot and test. You’ll need to manually re-start Snort for any traffic to be passed at this time. Remember that iptables is placing all routed traffic in NFQUEUE number 4, but if there is no application listening to that queue, traffic will not pass.
Fail open
The following is optional based on your security needs. If you want traffic to pass between networks when Snort is not running (fail-open mode), if Snort crashes or stops running for some reason, you’ll want to enable the –queue-bypass flag for the iptables rule. We will need to modify the current rule. First let’s check the state of all chains:
$ sudo iptables -L --line-numbers Chain INPUT (policy ACCEPT) num target prot opt source destination Chain FORWARD (policy ACCEPT) num target prot opt source destination 1 NFQUEUE all -- anywhere anywhere NFQUEUE num 4 Chain OUTPUT (policy ACCEPT) num target prot opt source destination
we want to modify rule number 1 in the FORWARD chain. To do this, we run the following command, which Replaces (the -R flag) this rule, adding the queue-bypass option:
sudo iptables -R FORWARD 1 -j NFQUEUE --queue-num=4 --queue-bypass
you’ll want to re-save the running configuration as above (overwrite the text file), reboot, and test.
Additional Firewall Rules
Rather than try to fit a small guide for iptables on production systems into this guide, I will refer you to a couple of other excellent short guides on how to use iptables to secure your system. You should at a minimum block all access to the system (the INPUT queue) that isn’t management related or from the management network. Configuring Drop by default rather than accept is recommended.
Basic iptables lock-downs for a system.
More in-depth iptables lock-downs for a server.
At a minimum, to block traffic from going to the management network, you would want to insert the following rule:
sudo iptables -I FORWARD -o ens192 -j DROP
this blocks all routed traffic out the management interface. You have to insert this above the NFQUEUE rule, because once Snort permits the traffic, it does not go through other chains.
Final Steps
-
You now need to setup Snort as a daemon, so it runs every time you reboot. This has been covered in my other articles:
- Creating Upstart Scripts for Snort on Ubuntu 14
- Creating systemD Scripts for Snort on Ubuntu 16
You will probably also want to setup PulledPork to download rules automatically, install barnyard2 for log processing, and BASE to view alerts in a GUI. All these steps can be found in my guide on snort, here: Installing Snort
Where to go From here
If you found yourself struggling to get routing working even before you got to the Snort portion of this guide, I highly recommend that you look into taking a networking course. I personally took the Cisco CCNA course offered by the local community college (I think it was only $45 USD a unit). I learned an amazing amount about how networks work, and it made me a much better systems administrator. Don’t think that because you’re a computer administrator that Cisco training isn’t for you, it will compliment your knowledge and abilities.
Books:
The CCNA Books offered by Cisco are excellent:
The CCNA official Cert Guide is what I used to pass my CCNA years ago, and are very well written textbooks. While it’s Cisco specific, most of the basic knowledge in these books are general enough to be useful to anyone working with networking.
The CCNA Routing and Switching Portable Command Guide is one of the few hard-copy books I keep on my shelf and refer to whenever i’m working on Cisco eqipment.
I found the O’reily Network Warior book to be quite satisfying, it gave a broad overview of a number of technologies related to network technology to really broaden my knowledge.
If you’re really serious about TCP/IP, the gold standard (and really heavy) book to have is the The TCP/IP Guide: A Comprehensive, Illustrated Internet Protocols Reference.
Links:
How to Enable IP Forwarding in Linux
Understanding the ip route command and IP routing in Linux.
Iptables Tutorial 1.2.2. If you want to know how it works at the architecture level (not the user level, and not the programming level), this is the guide you want to read. I found Chapters three through nine to be most helpful, but if you need some more info on TCP/IP, the first few chapters are quite good as well.
The second flowchart at this page is awesome for understanding how packets flow through iptables. Use it in conjunction with the iptables tutorial above.
the NFQUEUE options for iptables.
This article:
A Deep Dive into Iptables and Netfilter Architecture, may get a little deep into the internal workings of the kernel IP stack, but I found it fascinating. Compliments this SANS article as well.
A few relevant books that get good reviews, but which i haven’t read yet:
Linux iptables Pocket Reference
Linux Firewalls: Attack Detection and Response with iptables, psad, and fwsnort
How Linux Works: What Every Superuser Should Know
Feedback
I hope this article has been helpful to you. Please feel free to provide feedback, both issues you experienced and recommendations that you have. The goal of this guide was not just for you to create a Snort NIPS with NFQ, but to understand how all the parts work together, and get a deeper understanding of all the components, so that you can troubleshoot and modify your Snort NIDS with confidence.
Simple Splunk Scripted Input Example
Overview
In this article, I will walk you through the process of creating a scripted input in Splunk. With a scripted input, you configure your Splunk server or Universal Forwarder (UF) to run a script and capture the output of that script as events to be indexed by Splunk. This article will assume that you have some understanding of Splunk, running python and shell scripts on your system, and understand the difference between a Universal Forwarder and a Splunk Indexer. This article has been tested on Ubuntu 14, running Splunk 6.5. With minor modifications it should work for most Linux and Unix-based systems.
Getting Started
We will assume that our initial goal is to have Splunk run a python script, capturing the output as events. We will configure Splunk to run this python script as a scripted input by creating a new add-on on the Splunk system. To do this, we will create our add-on folder in the apps directory of the Splunk system. The apps directory is located under the etc folder in the $SPLUNK_HOME directory. $SPLUNK_HOME is the location where Splunk was installed. On an indexer, this will often be /opt/splunk, while on a Universal Forwarder, this will often be /opt/splunkforwarder. This guide will assume that you are working on a Universal Forwarder, but all steps can be easily modified for a Splunk Indexer.
Following Splunk’s naming conventions for applications, we will name this application TA-SimpleApp. The TA stands for “Technology Add-on” (Which is different from a Splunk App, which has a GUI). First we will create all the necessary folder and files:
# The path below is the apps folder for a Universal Forwarder # Change if you are on an indexer (/opt/splunk) or used a different install location cd /opt/splunkforwarder/etc/apps/ sudo mkdir TA-SimpleApp sudo mkdir TA-SimpleApp/bin sudo mkdir TA-SimpleApp/default sudo touch TA-SimpleApp/bin/TA-SimpleApp.py sudo touch TA-SimpleApp/default/inputs.conf
Next we need to adjust permissions:
cd /opt/splunkforwarder/etc/apps/ sudo chown -R splunk:splunk TA-SimpleApp
When you install Splunk, a Splunk user is created. We want these folders to be owned by the Splunk user, to ensure that it can access these files.
you should now have the following files and folders:
noah@thor:/opt/splunkforwarder/etc/apps$ tree TA-SimpleApp/
TA-SimpleApp/
├── bin
│ └── TA-SimpleApp.py
└── default
└── inputs.conf
Now we need to add the content of these two files. First we’ll setup our default inputs.conf (located in the default folder):
[script://./bin/TA-SimpleApp.py] interval = 10 sourcetype = my_sourcetype disabled = False index = main
A breakdown of this file:
- The first line tells Splunk that we are creating a scripted input. We reference the path to the file we want executed (note that we can not pass parameters to the script here, to do that you need a wrapper script, shown below). Rather than referencing the relative path to the script, you can also reference the full path of the script, relative to $SPLUNK_HOME. an example of this would be: [script://$SPLUNK_HOME/etc/apps/TA-SimpleApp/bin/TA-SimpleApp.py]. I prefer the relative path. If you want to reference a location outside of the $SPLUNK_HOME folder heirarchy, you need to use the .path option.
- The second line tells Splunk to run this script every 10 seconds. You can also set the script to run on a schedule.
- The third line is optional, and tells Splunk they sourcetype of the events (Why this matters).
- The fourth line tells Splunk to run this script. Set to True or 1 to prevent Splunk from running this script.
- The fifth line is optional, and tells Splunk which index events from this script should be written to. The index must exist on the Indexer, and the default index is main.
Next we add the following content to our python script (TA-SimpleApp.py):
# So we can run this scipt under python 2 or 3
from __future__ import print_function
import sys # for sys.stderr.write()
import time # for strftime
from datetime import datetime # for datetime.utcnow()
import random # to provide random data for this example
sys.stderr.write("TA-SimpleApp python script is starting up\n")
# output a single event
print (str(time.time()) + ", username=\"agent smith\", status=\"mediocre\", admin=noah, money=" + str(random.randint(1, 1000)))
# output three events, each one separated by a newline (each line will be a unique event)
for x in range(0, 3):
strEvent = str(time.time()) + ", "
strEvent += "username=\"" + random.choice(["Stryker", "Valkerie", "Disco Stu"]) + "\", "
strEvent += "status=\"" + random.choice(["groovy", "hungry", "rage quit"]) + "\", "
strEvent += "admin=" + random.choice(["lenny", "carl", "moe"]) + ", "
strEvent += "money=" + str(random.randint(1, 1000))
print (strEvent)
Line 9 is an example of how we write information to the Splunk event log: $SPLUNK_HOME/var/log/splunk/splunkd.log. See the section below on logging.
Line 12 is where we generate a single event for Splunk to consume. We start by printing the UNIX epoch time, followed by a comma-separated list of keys and values, ending with a newline (the print function ends each line with a newline). Epoch time is preferred because it is easy for Splunk to identify, and it is highly accurate. Splunk will automatically identify the key-value pairs in this instance. If your output is less structured, you would need to configure a props.conf and transforms.conf (a simple example of these files).
Beginning with line 15, we loop thee times to create three random events. Each event will be written to stdout (using the print function at line 21, like above), and will terminate with a newline, which Splunk interprets as the end of the event. You can have Splunk ingest multiple-line events, by configuring Line Breaking in props.conf.
Now we want to test that the python script works correctly. We can do this by simply running it, and making sure we are seeing the stderr and stdout being written correctly to the screen (this is how Splunk will ingest this information). Run the script manually, and look for similar output:
noah@thor:/opt/splunkforwarder/etc/apps/TA-SimpleApp/bin$ python ./TA-SimpleAppy TA-SimpleApp python script is starting up 1484997424.92, username="agent smith", status="mediocre", admin=noah, money=627 1484997424.92, username="Valkerie", status="hungry", admin=carl, money=800 1484997424.92, username="Disco Stu", status="rage quit", admin=lenny, money=663 1484997424.92, username="Stryker", status="groovy", admin=moe, money=483
If you have the following output, then your script is correct. Now you need to restart Splunk to have it load your Technology Add-on, and run your script. Reboot Splunk:
noah@thor:~$ cd /opt/splunkforwarder/bin/ noah@thor:/opt/splunkforwarder/bin$ sudo ./splunk restart
Check your log files (see the next section on error logging) and the SplunkWeb search for these events. In the SplunkWeb search app, you should see something similar to:
Congratulations, if you have similar output to above, you now have a simple scripted input for Splunk. On a UF, you have to ensure that python is available. On an indexer, Splunk will use it’s own version of python (2.7.5), and on a UF, you’ll use the system’s version of python.
Error Logging
One challenge is that you have to be an administrator to view this log file, or even browse the log folder. To view this folder, I usually use sudo bash in a separate terminal window to work with log files. Because Splunk captures all output from your script, we need to differentiate between events and log information. We do this by writing events to stdout, which is ingested by Splunk, becomes an event, and is indexed. Anything written to stderr is captured by Splunk and is written to the splunkd.log file. This event will look like the following entry in the splunkd.log:
01-21-2017 09:43:09.216 +0200 ERROR ExecProcessor - message from "/opt/splunkforwarder/etc/apps/TA-SimpleApp/bin/myAppLauncher.sh" TA-SimpleApp myAppLauncher.sh is starting
You will notice that Splunk marks this event as an ERROR in the log, there doesn’t seem to be a way to modify this event to change the severity.
A good way to follow these events on a Universal Forwarder is as follows:
root@thor:/opt/splunkforwarder/var/log/splunk# sudo tail -f splunkd.log | grep ExecProcessor 01-21-2017 09:49:32.684 +0200 INFO ExecProcessor - New scheduled exec process: python /opt/splunkforwarder/etc/apps/TA-SimpleApp/bin/TA-SimpleApp.py 01-21-2017 09:49:32.684 +0200 INFO ExecProcessor - interval: 10 ms 01-21-2017 09:49:35.195 +0200 ERROR ExecProcessor - message from "python /opt/splunkforwarder/etc/apps/TA-SimpleApp/bin/TA-SimpleApp.py" TA-SimpleApp python script is starting up
On an indexer (rather than a UF), this command generates too much information (there are a number of apps that are started by Splunk on an Indexer, where on a UF there is only the one we created). On an indexer, you may want to grep for the name of the app: TA-SimpleApp instead.
Wrapper Script
If you need to pass parameters to your script, or you want to execute an application, you can have Splunk call a shell script, where you have more options in launching your script. Let’s create this script:
cd /opt/splunk/etc/apps sudo touch TA-SimpleApp/bin/myAppLauncher.sh sudo chmod a+x TA-SimpleApp/bin/myAppLauncher.sh sudo chown splunk:splunk TA-SimpleApp/bin/myAppLauncher.sh
We need to modify our inputs.conf to call this new shell script, rather than the python script directly. Modify the first line of inputs.conf to look like this (everything else is the same):
[script://./bin/myAppLauncher.sh]
And enter the following content for this shell script (myAppLauncher.sh):
#!/bin/bash
# Write a line to the splunk log file
echo TA-SimpleApp myAppLauncher.sh is starting >&2
# Check if we can run a python interpreter, exit otherwise
command -v python >/dev/null 2>&1 || { echo No python interpreter found. >&2; exit 1; }
# execute our python script, located in the same folder as this script
cd $( dirname "${BASH_SOURCE[0]}" )
python ./TA-SimpleApp.py
On line 4, we are writing to stderr, which adds the output to an event to the splunkd.log event log (this would probably be removed in a final release, but it is good for testing).
On line 7 we are checking that we can find a python interpreter to run our script. If we can’t find one, we write an error to the splunkd.log log file and quit.
on line 10 we are setting the current directory to the directory of this shell script.
on line 11 we are calling our python script, which will execute and generate output just like the example above.
This example is very similar to launching a python script directly, however we now have the ability to pass parameters, do additional setup, change environmental variables (if needed), test for a python interpreter, and other housekeeping. You could also call a binary executable, output to your own log files, and other required setup or testing for your script. Basically, if you can do it from a shell script, and it generates output to stdout and stderr, you can use it as a scripted input. For any scripts or applications you execute here (including all child applications), all data written to stdout will become an event on your Splunk indexer. All errors written to stderr will become entries in the splunkd.log log file.
Testing your script as Splunk will run it
When Splunk runs your script, it is doing it within it’s own environment, with it’s own environmental variables. To test your script within that environment (without setting it up as an App), you can run ./splunk cmd from the bin folder:
noah@thor:/opt/splunkforwarder/bin$ ./splunk cmd ../etc/apps/TA-SimpleApp/bin/myAppLauncher.sh
you’ll see the following output:
TA-SimpleApp myAppLauncher.sh is starting TA-SimpleApp python script is starting up 1484994520.21, username="agent smith", status="mediocre", admin=noah, money=736 1484994520.21, username="Valkerie", status="hungry", admin=carl, money=141 1484994520.21, username="Stryker", status="rage quit", admin=carl, money=941 1484994520.21, username="Disco Stu", status="groovy", admin=moe, money=168
All data written to stdout and sdterr will show on the screen as your application writes it. Splunk is not running your app and processing the output, this tool merely lets you test your script to see if it would run in the environment that Splunk would run it. If you want to see the environmental variables for Splunk’s environment, just add the printevn command to the end of your shell script and run the ./splunk cmd command again. A simple example of the difference in environmental variables:
noah@thor:/opt/splunkforwarder/bin$ printenv | grep "splunk" PWD=/opt/splunkforwarder/bin noah@thor:/opt/splunkforwarder/bin$ ./splunk cmd /usr/bin/printenv | grep "splunk" PATH=/opt/splunkforwarder/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin PWD=/opt/splunkforwarder/bin _=./splunk SPLUNK_HOME=/opt/splunkforwarder SPLUNK_DB=/opt/splunkforwarder/var/lib/splunk SPLUNK_ETC=/opt/splunkforwarder/etc SPLUNK_WEB_NAME=splunkweb LD_LIBRARY_PATH=/opt/splunkforwarder/lib OPENSSL_CONF=/opt/splunkforwarder/openssl/openssl.cnf LDAPCONF=/opt/splunkforwarder/etc/openldap/ldap.conf
Where To Go From Here
If you want to see some examples of Splunk Tecnology Add-ons, just download them from the Splunkbase and browse through their files, or look at the links below.
Feedback is welcomed, especially if there are errors in this guide or recommendations you have from your own experience. Please contact me here.
Resources
Setting up a scripted input
Add a scripted input with inputs.conf
Writing reliable scripts
Anatomy of an app
Advanced Python Script Testing
A good Scripted Inputs Tutorial
Building Splunk Technology Add-ons From the Splunk blog.
Package and publish a Splunk app
Apps and add-ons: an introduction
Creating eBooks from AsciiDoc
Overview
This guide will show you how to convert AsciiDoc files into EPUB and Kindle’s .mobi format using open source software on Linux. This guide will assume you have some experience with Linux, and a general understanding of html, css, and XML will be very helpful. I will walk you through all the steps required to create these files, explaining how each tool works so that you can troubleshoot and adapt the workflow to make it work for you. My goal is for you to really understand how this process works, rather than just following rote steps without really understanding them. I will include links to related resources as we go through that will help round out your understanding. At the end of this article, you will have a Makefile that will allow you to easily build and modify eBooks.
This guide is tested on Ubuntu 14 x64, although it should work on most Debian-based systems without much effort. It could easily be used on any Linux-based system with a minimum amount of modification.
Overview of the Workflow
AsciiDoc is a simple document format that allows you to easily mark up text in an easy to read format, which can also be easily converted to a specific type of XML that is used for EPUBs. From Wikipedia: “AsciiDoc is a human-readable document format, semantically equivalent to DocBook XML, but using plain-text mark-up conventions.” A good overview of AsciiDoc can be found on the AsciiDoctor website.
Documents written in AsciiDoc will be converted to DocBook XML, a semantic language made for technical documentation. This DocBook XML is converted to EPUB3 html files using DocBook XSL stylesheets. XSL Stylesheets are used to convert XML to another format, in this case html files (eBooks, including .mobi and EPUB are merely an archive of html files and cascading style sheets, essentially a web page wrapped into a single archive). We will then turn those html files into an eBook using a few different tools.
Tools
All tools and files here are open source or free. We will use the following tools:
- asciidoctor: Used to convert AsciiDoc to DocBook XML.
- jing: An XML docbook schema validator.
- xsltproc: Used to convert Docbook XML to epub files and folders
- epubcheck: Validate and convert epub files & folders into single file.
- calibre: Calibre is a eBook viewer.
- KindleGen: Amazon tool to convert epub to mobi (kf8 and the older .mobi file).
We will also use the following files (download instructions are below):
The DocBook schema (in Relax NG schema Language).
The DocBook XSL Stylesheets, used to convert DocBook XML to HTML files from the DocBook Project.
Enviornment
This guide will have you setup a folder to hold the files for your eBook and all required files. It will assume that the filename of your eBook in adoc format is myBook.adoc. In this guide, we will setup all the required files, as well provide sample content for each file, so you will be able to create a full eBook from the example, and can then modify it after everything is working to match your workflow.
First, let’s install the required software:
sudo apt-get install -y asciidoctor jing xsltproc epubcheck calibre
Next let’s create the folder that will hold all the files for this eBook. We’re using a folder called ebook on the Desktop. We’ll create a number of necessary folders here as well:
mkdir ~/Desktop/ebook/ mkdir ~/Desktop/ebook/build-resources/ mkdir ~/Desktop/ebook/ebook-resources/ mkdir ~/Desktop/ebook/ebook-resources/graphics/ mkdir ~/Desktop/ebook/output/ cd ~/Desktop/ebook/
The build-resources folder will hold the DocBook XSL Stylesheets and the DocBook Schema file that we will download. This folder holds files that can be used for any ebook. The ebook-resources folder will store files that are specific to this one eBook, including css files, graphics (like the cover of the ebook), and any other files you want to include in your ebook. The output file is where our final products will be stored (the .mobi and EPUB files).
Now let’s create a simple AsciiDoc file. This will be the source material for our ebook. This format is text-based, and is simple to read and create. Since it’s text, it can also be added to your favorite version control tool (SVN, subversion, or the like). Here we will edit the adoc file using your favorite editor:
touch ~/Desktop/ebook/myBook.adoc xdg-open ~/Desktop/ebook/myBook.adoc
with the following content:
= Witty Book Title :doctype: book :backend: docbook :docinfo: :!numbered: :imagesdir: graphics [dedication] == My Dedications This book is dedicated to..... I'd also like to thank.... == This is the First Chapter Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas vehicula congue dolor, vel commodo magna viverra ac. Morbi ullamcorper, est eu egestas semper, velit elit bibendum orci, ut tristique tellus nulla sit amet ipsum. Sed fringilla, lacus sed viverra dictum, nulla augue placerat lectus, in efficitur magna risus non nibh. Ut laoreet, tortor at tempus mollis, magna risus ullamcorper dolor, quis rutrum ex augue a risus. Vivamus pellentesque accumsan est aliquam fringilla. Quisque eleifend ac eros in volutpat. Quisque eu euismod metus, at blandit diam. Phasellus in magna eget erat finibus lacinia quis at metus. Cras ut hendrerit sem. Vivamus ligula est, volutpat nec convallis eget, efficitur at orci. Proin non aliquet nunc. Mauris dui odio, bibendum consectetur ligula at, faucibus dapibus est. Praesent porttitor, nisi sit amet accumsan euismod, sem felis semper turpis, ut fermentum leo orci sit amet tortor. Nulla eros leo, eleifend vitae ornare quis, mollis tristique eros. In quis accumsan arcu. Ut hendrerit vitae sem ut consectetur. Nunc enim massa, tempus id orci vitae, rhoncus laoreet nulla. Pellentesque elementum purus rutrum, condimentum elit vitae, sagittis magna. Maecenas ornare justo et arcu consequat, nec volutpat risus fermentum. == This is the Second Chapter Duis cursus ac augue id blandit. Nulla varius accumsan odio, sed vestibulum odio lobortis quis. Nunc vitae ipsum tortor. Ut ut eros dignissim est luctus finibus ac quis nulla. Etiam consequat, neque sit amet laoreet laoreet, magna odio ornare justo, et ornare sapien nunc quis dolor. Praesent felis metus, facilisis a quam id, euismod faucibus nibh. Mauris venenatis dui erat, vel auctor felis tempus eget. Pellentesque tellus metus, pretium aliquam tristique eget, bibendum ut sapien. Curabitur magna augue, feugiat id enim congue, ullamcorper iaculis arcu. Integer pulvinar elit nulla, at gravida velit sodales eget. In quis leo ac mauris mollis facilisis fermentum non ex. Ut a lorem lacinia, egestas sem eu, tincidunt risus. Proin non ornare lacus, vitae imperdiet erat. == This is the Third Chapter Lorem ipsum dolor sit amet, consectetur adipiscing elit. Maecenas vehicula congue dolor, vel commodo magna viverra ac. Morbi ullamcorper, est eu egestas semper, velit elit bibendum orci, ut tristique tellus nulla sit amet ipsum. Sed fringilla, lacus sed viverra dictum, nulla augue placerat lectus, in efficitur magna risus non nibh. Ut laoreet, tortor at tempus mollis, magna risus ullamcorper dolor, quis rutrum ex augue a risus. Vivamus pellentesque accumsan est aliquam fringilla. Quisque eleifend ac eros in volutpat. Quisque eu euismod metus, at blandit diam. Phasellus in magna eget erat finibus lacinia quis at metus. Cras ut hendrerit sem. Vivamus ut feugiat neque, sed varius tortor. Phasellus sit amet ante ut tortor pulvinar efficitur non non massa. Curabitur imperdiet justo nec urna cursus, sit amet dapibus lectus posuere. Aliquam hendrerit nisi eget nunc aliquet, gravida aliquam elit volutpat. Donec semper tincidunt neque in aliquet. Curabitur lobortis rutrum felis quis tempus. In bibendum neque vitae ipsum tempus aliquet. Maecenas euismod consequat pellentesque. Maecenas in tincidunt nibh. Aliquam tempus libero non augue finibus fermentum. Sed massa leo, tempus sollicitudin consequat vel, ornare nec ligula. Donec et tellus bibendum, blandit nunc sed, porttitor magna. Integer eget pulvinar lorem. Sed in bibendum quam. Donec eu molestie ipsum, at maximus ipsum. Nunc vel mauris vulputate, faucibus dui quis, imperdiet enim. Phasellus sodales turpis quis velit egestas, in rhoncus diam pellentesque.
There are a number of interesting things happening in this file. The text we are using is Lorem ipsum, common filler text used by typesetters to allow you see how text looks on the screen or in print without getting caught up in the content of the text. The header of the adoc file begins with the Title of the book on the first line, underlined on the next line with a number of equals signs. The third line says that when we convert this file, we want the final document type to be a book (more on different document types here.) The backend actually gets overwridden at the command line when we convert this file later, but is good to have. The docinfo file command indicates that there is a docinfo.xml file (the one we will create below) that contains further header information. The Numbered line indicates that we don’t want page numbers. The first section we have is the dedication. The two equals signs indicate a chapter heading.
Now create the css file. This css file tells our ebook (EPUB or .mobi) how to be displayed, including font size, color, any anyting else that can be configured with css).
touch ~/Desktop/ebook/ebook-resources/master.css xdg-open ~/Desktop/ebook/ebook-resources/master.css
and enter the following information:
html, body { height: 100%; margin: 0; padding:0; border-width: 0; }
@page { margin: 5pt; }
/* indent paragraph */
h2 + p {
text-indent:0;
}
p {
text-indent:1em;
margin: 0;
}
/* Set the minimum amount of lines to show up on a separate page. (There is not much support for this at the moment.)
https://github.com/reitermarkus/epub3-boilerplate/blob/master/Ebook/OPS/css/main.css*/
p,
blockquote {
orphans: 2;
widows: 2;
}
/* page break for dedication (xslst keeps on same page as the copyright */
div.dedication {
page-break-before:always;
}
/* Move the legal notice from the title page to its own page */
div.legalnotice{
page-break-before:always;
}
/* Tile Page formatting */
div.book div.titlepage h1{
font-family: Helvetica,Arial,sans-serif;
text-align: center;
color: blue;
}
The docinfo file is an xml file that holds information about the author and the copyright information. This file needs to have the same name as your AsciiDoc file with -docinfo.xml appended, and be in the same folder as your adoc file. In this example, our AsciiDoc book is named myBook.adoc, so the docinfo file is named myBook-docinfo.xml. Create this file:
touch ~/Desktop/ebook/myBook-docinfo.xml
with the following content (note that there can’t be any blank space at the beginning of this file):
Important note: use a text editor that will automatically recognize that you’re working in an XML file, so that it will format this file correctly (replacing the spaces with tabs). It is important that this file be formatted correctly, or you’ll get errors.
<author>
<personname>
<honorific>Mr</honorific>
<firstname>Noah</firstname>
<surname>Dietrich</surname>
</personname>
</author>
<copyright>
<year>2017</year>
<holder>SublimeRobots Intl.</holder>
</copyright>
<legalnotice>
<para>
Copyright © 2017 by Noah Dietrich
</para>
<para>
All rights reserved. This book or any portion thereof may not be reproduced or used in any manner whatsoever without the express written permission of the publisher except for the use of brief quotations in a book review.
</para>
<para>
Printed in the United States of America
</para>
<para>
First Printing, 2017
</para>
<para>
ISBN 0-9000000-0-0
</para>
<para>
Jim & Joe Publishers, LLC
</para>
<para>
www.SublimeRobots.com
</para>
</legalnotice>
<cover>
<mediaobject>
<imageobject>
<imagedata fileref="graphics/cover.jpg">
</imagedata>
</imageobject>
</mediaobject>
</cover>
This information will be added to your ebook when it is processed, but is not stored in the adoc file. A good example of a docinfo file can be found here.
Next, we need to get the DocBook XML Schema file (docbookxi.rng) and the DocBook XSLT stylesheets. We will store them in our build-resources folder:
cd ~/Desktop/ebook/build-resources wget http://docbook.org/xml/5.2b01/rng/docbookxi.rng wget http://downloads.sourceforge.net/project/docbook/docbook-xsl-ns/1.79.1/docbook-xsl-ns-1.79.1.tar.bz2 tar -xvjf docbook-xsl-ns-1.79.1.tar.bz2
Finally, we need to download the KindleGen application from Amazon. Navigate to the KindleGen homepage, download the linux version, and extract the kindleGen binary to the build-resources folder.
cd ~/Desktop/ebook/build-resources wget http://kindlegen.s3.amazonaws.com/kindlegen_linux_2.6_i386_v2_9.tar.gz tar -xzvf kindlegen_linux_2.6_i386_v2_9.tar.gz kindlegen
The Book Cover: You’ll need to put a jpeg for the cover into the graphics folder named cover.jpg. If you don’t do this, you’ll need to remove the cover section in your myBook-docinfo.xml file. You can get information on recommended jpeg sizing here.
Begin Processing
The first step is converting the adoc file into DocBook XML format. This part can be a little frustrating sometimes, as many small semantic issues can cause errors to show up at this stage. Some issues I have encountered (and quick fixes for them if I have an answer):
- Double Spaces. This can be fixed with: sed -i.bak ‘/^$/d;G’ myBook.adoc
- Spaces at the begining of lines: This can be fixed with sed -i.bak ‘s/^[ \t]*//’ myBook.adoc
- Empty Chapters.
from your ebooks directory, assuming you have all the above files setup correctly, run the following command:
cd ~/Desktop/ebook/ asciidoctor --backend docbook5 --doctype book --verbose --destination-dir ./output/ myBook.adoc
here we are using the asciidoctor application to convert the adoc file into an AsciiDoc XML file. This XML file will have all the same content as our original adoc file, only formatted (and marked up semantically) to meet the DocBook schema (docbookxi.rng). The schema describes the legal layout of all valid files. We are using these options:
- --backend docbook5: Remember when I said above we’d override the backend command in the header of the adoc file? this is the line that does that. This line says that we want to use the docbook5 backend, which will generate DocBook 5.0 XML.
- --doctype book: We also have this information in the header, but it’s better to be sure. The command line overrides the setting in the header as well, if there is a difference.
- --verbose: We want as much output as possible, good for troubleshooting.
- --destination-dir ./output/: we want the xml file that is created to be put into the output folder
- myBook.adoc: this is the filename of the adoc file we want converted to DocBook XML.
you should see output similar to:
noah@thor:~/Desktop/ebook$ asciidoctor --backend docbook5 --doctype book --verbose --destination-dir ./output/ myBook.adoc Input file: myBook.adoc Time to read and parse source: 0.00527 Time to render document: 0.00944 Total time to read, parse and render: 0.01476 noah@thor:~/Desktop/ebook$
If you you have errors, you may see output similar to:
asciidoctor: WARNING: myBook.adoc: line 9: invalid style for paragraph: dedication
This usually means that there is either an error in your header, or you docinfo xml file has an error (spaces instead of tabs, extra spaces between elements, and similar issues). You must fix these issues before continuing.
Now that we have our ebook in the DocBook XML file format, we want to validate that it is semantically correct. We want to check to see that its format matches the schema defined in the docbookxi.rng file (the schema in Relax NG schema language). For this, we use a tool called jing. There is another tool called xmllint, that also does validation, but I encountered issues with it, and found jing to be much more reliable. An excellent resource for understanding the details can be found in the Processing DocBook5 section in the DocBook XSL: The Complete Guide (you’ll be referencing this online ebook a lot if you want to do any configuration of your ebook).
So the content of our adoc file and the docinfo file have been combined into a single xml file in the output directory (you can open it to see what it looks like), and we need to validate it to make sure it’s formatted correctly (sometimes asciidoctor makes mistakes). To do this, we run the following command from the same directory as before (not the output directory):
cd ~/Desktop/ebook/ jing -i ./build-resources/docbookxi.rng output/myBook.xml
This command is simple, it takes the docbookxi.rng schema file as the first input (-i), and our book in xml format as our second input, and will tell us if it’s valid (properly formatted) DocBook XML. If you have issues, try to figure out what line of the xml file is causing the issue, and try to track it back to the original asciidoc or docinfo file. This can be a challenge to do, sometimes searching the internet for your error can help.
If you see no output, then there are no errors.
Next, we are going to use xsltproc to convert our DocBook XML file into a series of html files (HTML 5 files actually), copy in our css and images to create a folder that represents our entire ebook, including all required resources.
It helps here to understand how ebook file systems are laid out before they are zipped into an archive we consider an EPUB or mobi file. A basic EPUB has the following files and folder heirarchy stored in a zipped container:
mimetype
META-INF/
container.xml
OEBPS/
content.opf
chapter1.xhtml
chapter2.xhtml
css/
style.css
toc.ncx
graphics/
cover.jpg
Good explanations of these files can be found on Wikipedia, as well as here and here.
We need to convert our valid DocBook XML into the above folder structure. Do do this, we use xsltproc, whic applies XSLT stylesheets to XML documents. XSLT stylesheets are a language for converting XML into other formats (in our case, HTML documents). The XSLT stylesheets are provided by the DocBook project. The following command converts our DocBook XML into an EPUB folder hierarchy:
cd ~/Desktop/ebook/ xsltproc --stringparam base.dir ./output/epub3-book/OEBPS/ --stringparam chapter.autolabel 0 --stringparam chunker.output.indent yes ./build-resources/docbook-xsl-ns-1.79.1/epub3/chunk.xsl ./output/myBook.xml
Let’s break this down:
- xsltproc: The application that applies the XSLT stylesheets.
- –stringparam base.dir ./output/epub3-book/OEBPS/: Where will the EPUB folders be output. The epub3-book folder is the location of the EBOOK. this option requires the location of the OEBPS folder.
- –stringparam chapter.autolabel 0: Do not number chapter headings (I feel it looks better without numbers on chapters).
- –stringparam chunker.output.indent yes : Make the html output pretty (helps for troubleshooting)
- ./build-resources/docbook-xsl-ns-1.79.1/epub3/chunk.xsl: These are the XSLT stylesheets.
- ./output/myBook.xml: The source DocBook XML file.
the stringparam options above are specific to the XSLT files that we are working with. To find other options that are available, read through Chapter 7. HTML output options of DocBook XSL: The Complete Guide.
You should see output similar to:
noah@thor:~/Desktop/ebook$ xsltproc --stringparam base.dir ./output/epub3-book/OEBPS/ --stringparam chapter.autolabel 0 --stringparam chunker.output.indent yes ./build-resources/docbook-xsl-ns-1.79.1/epub3/chunk.xsl ./output/myBook.xml Writing ./output/epub3-book/OEBPS/bk01-toc.xhtml for book Writing ./output/epub3-book/OEBPS/ch01.xhtml for chapter(_this_is_the_first_chapter) Writing ./output/epub3-book/OEBPS/ch02.xhtml for chapter(_this_is_the_second_chapter) Writing ./output/epub3-book/OEBPS/ch03.xhtml for chapter(_this_is_the_third_chapter) Writing ./output/epub3-book/OEBPS/index.xhtml for book Writing ./output/epub3-book/OEBPS/docbook-epub.css for book Generating EPUB package files. Writing ./output/epub3-book/OEBPS/cover.xhtml for mediaobject Generating image list ... Writing ./output/epub3-book/OEBPS/package.opf for book Writing ./output/epub3-book/OEBPS/../META-INF/container.xml for book Writing ./output/epub3-book/OEBPS/../mimetype for book Generating NCX file ... Writing ./output/epub3-book/OEBPS/toc.ncx for book noah@thor:~/Desktop/ebook$
We also need to manually move our css file and images into the EPUB folder hierarchy (add any additional graphics you need at this stage):
cd ~/Desktop/ebook/ cp ./ebook-resources/master.css ./output/epub3-book/OEBPS/docbook-epub.css cp -r ./ebook-resources/graphics/ ./output/epub3-book/OEBPS/
The next step is to convert our EPUB folders into a single file (our actual EPUB). To do this we use epubcheck, then rename the file:
epubcheck ./output/epub3-book/ -mode exp -v 3.0 -save mv ./output/epub3-book.epub ./output/myBook.epub
Here we are using -mode exp to have epubcheck validate the expanded EPUB archives, version 3.0, and save it to a single file: ./output/myBook.epub.
This epub file is the first final product. You can view this epub on any epub compatible reader (including calibre, which we installed earlier).
The final step is to convert out epub into the mobi format, for use on Amazon Kindle devices. This is done with KindleGen. This tool is simple, it takes the name of the epub folder to convert, and the name of the .mobi to create:
cd ~/Desktop/ebook/build-resources ./kindlegen ../output/myBook.epub
if you look in your ./output folder, you will now see your epub and .mobi files. If you have an amazon device, you can email the .mobi file to yourself and have it automatically download to your device. All kindle devices support this .mobi format. More information can be found here and here.
Automating the Build Process
You will quickly find that as you are modifying your files, it becomes a hassle to constantly run these commands. The solution to this is to use a Makefile. This tool was originally designed to compile software, but can be easily modified to simplify your ebook workflow.
in your ebook folder, create a new file called Makefile:
cd ~/Desktop/ebook/ touch Makefile
enter the following text (as with the docbook file above, replace spaces at the beginning of lines with tabs if needed):
mobi : epub #ebook-convert ./output/myBook.epub ./output/myBook.mobi ./build-resources/kindlegen ./output/myBook.epub epub : ebook epubcheck ./output/epub3-book/ -mode exp -v 3.0 -save mv ./output/epub3-book.epub ./output/myBook.epub ebook : docbook xsltproc --stringparam base.dir ./output/epub3-book/OEBPS/ \ --stringparam chapter.autolabel 0 \ --stringparam chunker.output.indent yes \ ./build-resources/docbook-xsl-ns-1.79.1/epub3/chunk.xsl ./output/myBook.xml cp ./ebook-resources/master.css ./output/epub3-book/OEBPS/docbook-epub.css cp -r ./ebook-resources/graphics/ ./output/epub3-book/OEBPS/ docbook : asciidoctor --backend docbook5 --doctype book --verbose --destination-dir ./output/ myBook.adoc jing -i ./build-resources/docbookxi.rng output/myBook.xml .PHONY: clean clean : -rm -rf ./output/*
Open a command prompt, navigate to the ebook folder, and you can now build your ebook by issuing the command make complete. If you want to delete all old versions of the ebook, you can run make clean. if you get an error: Makefile:3: *** missing separator. Stop, then you need to replace all spaces with tabs at the beginning of lines (there are issues pasting tabs from a website into a document).
Some of the options you have here:
- make docbook: Convert your adoc to DocBook XML and validate.
- make ebook: This runs make docbook, and then creates the ebook folders.
- make epub: This runs the above two commands, then converts the ebook folders into a single epub
- make mobi: This runs the above commands, and finally converts the epub into a mobi
- make clean: delete all files in the output folder.
You can modify this makefile to match your workflow, such as adding options to xsltproc (new lines are broken up with a backslash to improve readability), or having more files added to your ebook directory.
Conclusion
This guide has given you a simple framework for creating an ebook workflow. There are a number of things that can be improved or modified in this process to suit your needs, but hopeful you have learned enough to make these modifications yourself. You’ll probably want to improve the css files for your ebook (there are a number of websites that can better discuss epub css options, some of them are linked below). You may want to look at embedding images into your book, using specific fonts, adding different parameters to the XSL transforms, and many other options.
Feedback is welcomed, especially if there are errors in this guide or recommendations you have from your own experience: please contact me here.
Helpful Links
DocInfo.xml example.
Oreily Publications docinfo.xml example for erlang book.
publishing with iBooks example docinfo.xml.
another Oreily docbook.xml example.
Amazon Kindle Publishing Guidelines
CSS Boilerplate for eBooks.
Basic css styles for Kindle html.
The eBook Design and Development Guide on Amazon.
These two guides below use a2x from the asciidoc package, rather than asciidoctor to generate the xml from docbook. I prefer asciidoctor, as i find that it worked better for my workflow.
A good guide on converting docbook to epub and mobi.
Another good guide.
Snort 2.9.9.x on Ubuntu – Part 8: Conclusion
- Installing Snort
- Configuring Snort to Run as a NIDS
- Writing and Testing a Single Rule With Snort
- Installing Barnyard2
- Installing PulledPork
- Creating Upstart Scripts for Snort on Ubuntu 14
- Creating systemD Scripts for Snort on Ubuntu 16
- Installing BASE
- Conclusion
Where to Go From Here
I hope this series of articles has been helpful to you. Please feel free to provide feedback, both issues you experienced and recommendations that you have. The goal of this guide was not just for you to create a Snort NIDS, but to understand how all the parts work together, and get a deeper understanding of all the components, so that you can troubleshoot and modify your Snort NIDS with confidence.
Capturing More Traffic With Snort
You will probably want to configure your network infrastructure to mirror traffic meant for other hosts to your Snort sensor. This configuration is dependent on what network equipment you are using. If you are running Snort as a Virtual Machine on a VMware ESXi server, you can configure promiscuous mode for ESXi by following my instructions in this article: configure promiscuous mode for ESXi.
For different network infrastrucutre, you will need to do a little research to configure network mirroring for your Snort server. Cisco calls this a span port, but most other vendors call this Port Mirroring. Instructions for Mikrotik (a linux based switch and router product that I like). If you run DD-WRT, it can be configured with iptables, like any linux based system. If you have network equipment not listed above, any search engine should point you towards a solution, if one exists. Note that many consumer switches will not have the ability to mirror ports.
You can also purchase devices specifically made to mirror data (called taps). Some products that have been recommended on the Snort-Users list are:
More Advanced Snort Configuration
Snort has the ability to do much more than we’ve covered in this set of articles. Hopefully you’ve learned enough through this setup that you will be able to implement more advanced configurations and make Snort work for you. Some things that Snort is capable of:
- Multiple remote Snort sensors, for example on different subnets.
- The documentation section of the Snort website has a number of useful articles about more advanced things you can do with Snort.
Some other related articles I have written:
- Snort as a Network Intrusion Prevention System (NIPS).
- Installing Snort++ (Snort 3.0 Alpha).
- OpenAppID for Layer 7 protocol detection.
Recommended Reading
- Snort IDS and IPS Toolkit (Jay Beale’s Open Source Security)
(Kindle Version
) – This is a good book for understanding how Snort works under the hood. It is a little old, but is still relevent and very detailed.
- Snort Cookbook
– This book is very helpful in showing how Snort can be run to meet specific needs (using recipes that describe specific situations).
- Applied Network Security Monitoring: Collection, Detection, and Analysis
– I haven’t read this book, but it is well reviewed, and covers NIDS at a much higher level than the other two books.
Feedback
I would love to get feedback from you about this guide. Recommendations, issues, or ideas, please contact me here.
Snort 2.9.9.x on Ubuntu – Part 7: Installing BASE
- Installing Snort
- Configuring Snort to Run as a NIDS
- Writing and Testing a Single Rule With Snort
- Installing Barnyard2
- Installing PulledPork
- Creating Upstart Scripts for Snort on Ubuntu 14
- Creating systemD Scripts for Snort on Ubuntu 16
- Installing BASE
- Conclusion
Installing BASE On Ubuntu
BASE is a simple web GUI for Snort. Alternate products include Snorby, Splunk, Sguil, AlienVault OSSIM, and any syslog server.
Splunk is a fantastic product, great for ingesting, collating, and parsing large data sets. Splunk is free to use (limited to 500 MB of data per day, which is a lot for a small shop). Sguil client is an application written in tcl/tk. Snorby is abandoned, and relies on old versions of many Ruby packages that makes documenting the installation difficult, and a constantly changing target.
I’ve chosen to use BASE in this guide because it’s simple to setup, simple to use, and works well for what it does. Both BASE and Snorby are abandoned projects, and while Snorby gives a nice web-2.0 interface, since it is written in Ruby-on-Rails, the Ruby packages it relies on are constantly upgrading, which causes compatibility issues with other required Snorby packages, which causes too many installation problems. If you want to try installing Snorby, please see these unsupported out of date guides for Ubuntu 14 or Ubuntu 16.
There is a slight difference between BASE on Ubuntu 14 versus 16: BASE requires PHP 5, which isn’t available in the Ubuntu 16 archives (Ubuntu has moved on to PHP 7 in this release), so we have to use a PPA on Ubuntu 16 to install the php 5 packages:
# Ubuntu 16 only: sudo add-apt-repository ppa:ondrej/php sudo apt-get update sudo apt-get install -y apache2 libapache2-mod-php5.6 php5.6-mysql php5.6-cli php5.6 php5.6-common php5.6-gd php5.6-cli php-pear php5.6-xml
in Ubuntu 14, we can just install the necessary libraries:
# Ubuntu 14 only: sudo apt-get install -y apache2 libapache2-mod-php5 php5 php5-mysql php5-common php5-gd php5-cli php-pear
next install Pear image Graph:
sudo pear install -f --alldeps Image_Graph
Download and install ADODB:
cd ~/snort_src wget https://sourceforge.net/projects/adodb/files/adodb-php5-only/adodb-520-for-php5/adodb-5.20.8.tar.gz tar -xvzf adodb-5.20.8.tar.gz sudo mv adodb5 /var/adodb sudo chmod -R 755 /var/adodb
Download BASE and copy to apache root
cd ~/snort_src wget http://sourceforge.net/projects/secureideas/files/BASE/base-1.4.5/base-1.4.5.tar.gz tar xzvf base-1.4.5.tar.gz sudo mv base-1.4.5 /var/www/html/base/
Create the BASE configuration file:
cd /var/www/html/base sudo cp base_conf.php.dist base_conf.php
Now edit the config file:
sudo vi /var/www/html/base/base_conf.php
with the following settings (note that the trailing slash on line 80 is required, despite the instructions in the configuration file):
$BASE_urlpath = '/base'; # line 50 $DBlib_path = '/var/adodb/'; #line 80 $alert_dbname = 'snort'; # line 102 $alert_host = 'localhost'; $alert_port = ''; $alert_user = 'snort'; $alert_password = 'MySqlSNORTpassword'; # line 106
While in the base conf.php file, you will also want to comment out line 457 (we don’t want the DejaVuSans font), and un-comment (remove the two backslashes) from line 459, enabling a blank font. The section for fonts (begining at line 456) should look like this:
//$graph_font_name = "Verdana"; //$graph_font_name = "DejaVuSans"; //$graph_font_name = "Image_Graph_Font"; $graph_font_name = "";
Set permissions on the BASE folder, and since the password is in the base conf.php file, we should prevent other users from reading it:
sudo chown -R www-data:www-data /var/www/html/base sudo chmod o-r /var/www/html/base/base_conf.php
restart Apache:
sudo service apache2 restart
The last step to configure BASE is done via http:
- Browse to http://ServerIP/base/index.php and click on the setup page link (replace ServerIP with the IP of your Snort Server).
- Click on the Create BASE AG button on the upper right of the page.
- Click on the Main page link.
Note: If you read through the BASE configuration file, there are a number of other options you can implement if you like. A few options are SMTP Email alerts, IP Address to Country Support, and user authentication.
Congratulations, if you’ve made it this far, you have a fully-functioning Snort system. Please continue on to the Conclusion for more things you can do with Snort.