Setting up flow-inspector can be a bit tricky, as you first need to have a data source that generates your flow data. We can import data using a variety of ways, but most of them require you to set up our network monitoring toolkit VERMONT. Furthermore, there is a certain lack of documentation ...
This blog posts will try to walk you through the process of setting up VERMONT and flow-inspector on your system. After following this blog post, you should have a working installation of VERMONT that monitors the interfaces of your system, generates flow data from these observations, and pushes the data into the visualization system.
Preparing Your System
In order to install any of the components, you need to prepare your local system by installing some packages. As we currently do not ship any binaries for either VERMONT or flow-inspector, you need to build them from the sources.
In order to do this, you need to be have a unix-based system such as Linux (any modern distribution should work), FreeBSD, or Mac OS X. Install a development environment on your system and make sure the development packages for the dependencies that are documented in the READMEs of VERMONT and Flow-Inspector are installed.
This is the current list for VERMONT:
- build-essentials (XCode on Mac OS X)
For flow-inspector you need
- python (>= 2.6)
- Jinja2 (tested with v2.6) pip install Jinja2
- ujson (tested with 1.18) (optional but recommended) pip install ujson
- Redis (tested with v2.4.3) pip install redis
- MySQLdb (tested with v1.2.3) pip install mysql-python
For this example, we will use MySQL as our backend for storing the flow data. You can choose between the MySQL, Oracle and MongoDB backends. Currently, I recommend using the MySQL backend. However, you might want to check the flow-inspector page as this recommendation might change over time.
Create a database called "flowinspector" in your DB, create a MySQL user and grant him all permissions on the flowinspector DB.
Installing and Configuring Flow-Inspector
After you installed all base components into your system, you are now ready to get the latest flow-inspector version:
After you retrieved the source code, you need to adopt the configuration for your system. In order to do this, you need to create a file in the directory "flow-inspector/config/":
cp config.default.py config.py
Then use your favorite editor to change the configuration. You need to change the destination flow backend part of the configuration in order to make sure flow-inspector can connect to your data base:
db_backend = "mysql"
db_host = "127.0.0.1"
db_port = 3306
db_user = ""
db_password = ""
db_name = "flowinspector"
Make sure you enter the username and password of the MySQL user that you created and granted the rights on the database flowinspector.
After you did this, you should try to start flow-inspector by running the command
from the flow-inspector base directory. You should receive a message that looks like:
willet:flow-inspector braun$ ./app/app.py
Bottle v0.11.3 server starting up (using WSGIRefServer())...
Listening on http://0.0.0.0:8080/
Hit Ctrl-C to quit.
Running this script starts a webserver on port 8080 that you can use to connect your browser to. At the time of this writeup, only Chrome and Safari have all the features implemented that are required to run flow-inspector.
If you successfully managed to perform all the steps above, your browser should show you a screen that looks like this:
As you can see, there is some kind of user interface, but it doesn't show any data. This is what we would expect because we did not yet import any data into flow-inspector. Please note that the application script will, at this moment, throw some exceptions whenever you point your web browser at the system. This is flow-inspector's way to complain about missing data. Ignore these messages until we actually have some data to display.
In order to make flow-inspector insert data into the system, you need to start the preprocessor that you can find in the directory preprocess/. You can start the preprocessor by changing into the directory preprocess and running:
It is necessary to keep this script running whenever you want to import data. If the script is not running, no new data will be imported into flow-inspector.
If you want to end the importing process, you can finish the script using the key combination "STRG+C". The script will not finish immediately, but it will perform some cleanups and commit some cached data to the database. Please be patient while the script finishes and do not abort the processing. If you abort the script by pressing "STRG+C" for a second time, you will certainly see some data loss. So please be patient while the script finishes :)
Now that we have the importing process running, we need to make sure that we generate some data that can be imported.
Compiling and Configuring VERMONT
Make sure you installed all the packages that are required for VERMONT. Then get the latest development version from the repository:
git clone https://github.com/constcast/vermont.git
VERMONT can directly insert flows into the preprocessing system of flow-inspector. However, the module that is able to perform this task is still under active development at the time this blog post was written. As we aim at keeping our master development branch stable (since we only rarely create releases), this feature will only be merged into the main line after it received some more testing.
As I'm not sure that I'll update this blog post when we merge the feature into the main branch, the flow-inspector connection might already be included into the master branch. You can check this by having a look at the documentation at https://github.com/constcast/vermont/wiki/Moduleconfiguration.
If you can find documentation for the module ipfixFlowInspectorExporter on this site, then you can safely choose to build the master branch. Otherwise, you should checkout the branch "merge-features"
git branch merge-features origin/merge-features
git checkout merge-features
If you checked out the proper branch, then you need to configure the system:
cmake -DSUPPORT_SCTP=OFF -DSUPPORT_REDIS=ON -DWITH_TOOLS=OFF .
If you successfully installed all the dependencies, you should get some output that looks similar to this one:
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE
-- Found hiredis: /usr/local/include, /usr/local/lib/libhiredis.dylib
-- Boost version: 1.53.0
-- Found boost libraries
-- Found LibXml2: /usr/lib/libxml2.dylib (found version "2.7.8")
-- Found libxml2 libraries
-- Found libxml2 libraries
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/braun/code/vermont
If you don't get the message that the build files have been written, then you should check the error messages and install the missing packages. In case you encounter any problems that you cannot fix, just open an issue at our tracker and someone (probably me) will have a look at your problems.
Once you successfully ran cmake, try to build the system by running
This process might take a while. After it finished, you should see a binary file named "vermont" in the base source directory. This file is the monitoring probe that can be run in order to generate the flow data. It requires a configuration file which defines the flow generation and export process. A sample file is shipped with vermont:
You need to change the configuration file using your favorite text editor. Open the file, search for the tag inside the module, and change the default interface to the main interface of your machine (e.g. eth0, eth1, wlan0, ...). You can keep all other configuration options as they are supplied in the file. Please note that we don't yet support IPv6 :(
Afterwards, you need to start vermont. As vermont needs to have the right to set your interface into promiscuous mode, you probably need to run it as root:
sudo ./vermont -f configs/flowinspector_exporter.xml
Make sure you have your preprocessor running. If VERMONT did not give you any error message, then it's time to do some browsing or any other kind of network-related activities that generate network traffic.
Waiting for Data ...
It might be a good idea to get some coffee at this time, as it will take at least 10 minutes until some data will be imported into the database with the default configuration.
The default configuration uses an inactive timeout of 5 minutes and an active timeout of 10 minutes for the flow generation process. In addition, flow-inspector will cache data for five minutes before committing it to the database.
You can change this behavior, but I'd not recommend to do this unless you really know what you're doing. You can monitor the preprocessor in order to determine when the first data was inserted into the database. The preprocessor will output the number of flows it has processed every ten seconds. This number will (and should) be zero throughout the first five minutes, as VERMONT will not produce any flow data unless a flow timeout occurs (and this will not happen before at least 5 minutes have elapsed).
Afterwards, you should see some flows being processed (if you generated some traffic in the meantime). Flow-inspector will cache the incoming data for 5 additional minutes before committing it to the database. Flow-inspector will also commit data if at least 100,000 flows have been seen (whatever happens first). It will inform you that about this commit process by printing the line
Live import. Flushing caches ...
After you have seen this line, you can start looking at your flow data in the web frontend.
You can also force flow-inspector to commit the cached data by terminating the script (press STRG+C once). Wait for the preprocessor to finish, and you should see some flows in the frontend. You can afterwards start the preprocessor again to consume more flows.