Context and Vision



While local (city and county) government is acknowledged as being integral to critical infrastructure protection, no reasonable effort has been made to address cyber defenses on the local scale. The Public Regional Information Security Event Management (PRISEM) project, funded by the US Department of Homeland Security Science and Technology Directorate, is designed to address several of the fundamental issues (e.g., lack of clue, limited collaboration) hampering critical infrastructure protection activities at the local level. PRISEM's vision is to create an open-source and maintained repository of scalable, extensible botnet detection scripts along with legal agreements and procedural documents that facilitate the creation of regional response centers.

Our initial efforts for PRISEM include a package of individual Python-based detectors that listen to already readily available live data streams (e.g., netflow and syslog) and send out alerts when suspicious activity is detected. While initially driven by the needs of regional efforts, we see parallels in the challenges experienced by other groups (e.g., universities, enterprises) and we believe such a system and associated procedural and legal documentation will be a useful addition to many a network warrior's tool chest.

The goal of this website is to provide a centralized location for the sharing of these maintained free, open-source tools useful in the detection of botnets and other malicious traffic, as well as the above mentioned legal and procedural documents that can be used by regional network security response centers.

Initial Set of Tools

The initial set of botnet/malware detectors is refered to as the "Botnets System" in the documentation that follows and the downloadable code documents.

There will also be a set of contributed detectors that we will post here, but which will not necessarily be maintained by the project staff.

Botnets System Summary:

Note: Much of this text is taken verbatim from the README in the core botnets distribution package.

Botnets is a system for the automated detection of botnets and other malicious traffic.

Note the use of the singular proper noun 'Botnets,' which is the name of this system and should be distinguished from the plural common noun 'botnets,' which are the networks of malware this system tries to detect. :-)

Botnets consists of a set of detectors. Detectors vary, but generally each is designed so that it listens on a UDP port for an incoming stream of netflow records and generates alerts when it discovers traffic represented by those records that it deems suspicious. Detectors can also run on an input of ASCII CSV flow records instead of listening for netflow traffic on a port, but this is a less common usage scenario.

Since each detector is different, and even the above may not apply to all future detectors, it is best to leave the details for the documentation accompanying each detector. However, to get started, the Detector Types section below gives a quick overview of the detectors that constitute this release, including example usage. Remember that the code of each detector includes more details and that each detector supports a --help command that lists its options and required arguments.

Each detector currently generates one alert for a given source,dest IP pair per purge period. The purge period defaults to 24 hours.

Current Detector Set

Detector Function
Blacklist (and shadow, cymru, bogon, darknet derivatives) Initialized with one or more blacklist or specific IP/prefix; detects traffic destined to addresses in blacklist; Can also help build list of all active IPs on a network
Port Scan Alerts when more than threshold connections to well-known ports are attempted in threshold seconds by a single source
Syn Flood Alerts when too many syns are seen within threshold time window to the same destination
SMTP Spam Flags SMTP server-like behavior; often indicating a compromised host
Service/Server Flags server services (well known ports) or traffic volume indicative of likely server
IRC Reports traffic destined to one of the several well-known IRC ports; ignores likely innocuous traffic
Feature Alerts when a flow matching commandline-specified features is detected (e.g. src/dest ip, src/dest port, protocol, packet count)

Bug reports, patches, and contributions

Report bugs to the email address at the bottom of the page. As this is an open-source project, we encourage you to submit a patch along with your bug report. :-)

Questions and Help

If, after reading all attached documentation and consulting a web search engine, you have unanswered questions, feel free to try to reach us via the email address at the bottom of the page. Please put the uppercase word BOTNETS at the start of your email subject line.

Dependencies

Botnets requires the following infrastructure and software to function. Please see the INSTALL document for more details.

  • One or more Netflow v5 streams from your network to your Botnets host. Typically, netflow is either exported by a router, or generated by one of several widely available free and commercial software tools, such as nprobe, frobe, and others. If you are a network operator or an administrator of a large enterprise network, chances are that you already have a netflow collection infrastructure set up for accounting and law enforcement compliance. Otherwise, setting up a sniffer and running one of the free tools is not too difficult to do. Botnets currently only supports netflow v5, the most widely used version.

    IMPORTANT NOTE: If your environment includes sampled netflow, wherein the netflow stream coming to the detectors only represents a subset of the actual flows present on your network, some of the detectors may not be able to pick up suspicious behavior with any degree of accuracy. This is especially true for detectors that rely on capturing a single host's many-to-one behavior, such as port_scan_detector and syn_flood_detector.
  • Python 2.6.x or later; Py-Radix, tested with ver 0.5; PySNMP, tested with ver 4.1.12a; pyasn1, tested with ver. 0.0.10a; See INSTALL for details. The scripts currently all expect a python2.6 executable to be available, so if you have 2.6 installed or simlinked to 'python', but not python2.6, you might have to add a symlink from python2.6 to python.

Features Common Across Detectors:

  • Alert output to: syslog, SNMP, stdout (terminal)
  • Input from Netflow v5 Stream or ASCII CSV flow record file
  • Whitelist feature
  • Caching ("penalty box") of repeat alerts. The default is to purge cache every 24 hours.
  • Email list of alerts on purge of cache
  • Naming networks in alerts

Features Available in Several Detectors:

  • Rate-based detection: A certain number of violations must occur in a certain number of seconds in order for an alert to be generated. See detector --help.

The Team

The Botnets core code is developed and tended to by folks from the Research and Development team at Merit Network, Inc. and by security researchers from the University of Michigan, department of Computer Science and Engineering. The PRISEM pilot is supported by the efforts of folks with the City of Seattle and the University of Washington.

Michael Bailey

University of Michigan, Computer Science and Engineering.

Jake Czyz

Merit Network, Inc., Research and Development

Dave Dittrich

University of Washington, Applied Physics Lab

Michael Hamilton

City of Seattle, Department of Information Technology

Manish Karir

Merit Network, Inc., Research and Development

Downloads

Note that the below code is licensed under the GPLv2 License.