FLAIM
Encyclopedia
FLAIM is a modular tool designed to allow computer and network log sharing through application of complex data sanitization policies.
FLAIM is aimed at 3 different user communities. First, FLAIM can be used by the security engineer who is investigating a broad incident spanning multiple organizations. Because of the sensitivity inherent in security relevant logs, many organizations are reluctant to share them. However, this reluctance inhibits the sharing necessary to investigate intrusions that commonly span organizational boundaries. Second, anyone designing log analysis or computer forensics
tools needs data with which they can test their tools. The larger and more diverse the data set, the more robust they can make their tools. For many, this means they must gather many logs from outside sources, not just what they can generate in-house. Again, this requires log sharing. Third, researchers in many computer science
disciplines (e.g., network measurements
, computer security
, etc.) need large and diverse data sets to study. Having data sanitization tools available makes organizations more willing to share with these researchers their own logs.
FLAIM is available under the Open Source Initiative
approved University of Illinois/NCSA Open Source License. This is BSD-style license. It runs on Unix
and Unix-like
systems, including Linux
, FreeBSD
, NetBSD
, OpenBSD
and Mac OS X
.
While FLAIM is not the only log anonymizer, it is unique in its flexibility to create complex XML
policies and its support for multiple log types. More specifically, it is the only such tool to meet the following 4 goals. (1) FLAIM provides a diverse set of anonymization primitives. (2) FLAIM supports multiple log type, including linux process accounting logs, netfilter alerts, tcpdump
traces and NFDUMP NetFlows
. (3) With a flexible anonymization policy language, complex policies that make trade-offs between information loss and security can be made. (4) FLAIM is modular and easily extensible to new types of logs and data. The anonymization engine is agnostic to the syntax of the actual log.
. At first this was for anonymizing logs in-house to share with the SIFT group. Soon there was a need for more powerful anonymization and anonymization of different types of logs. CANINE was created to anonymize and convert between multiple formats of NetFlows
. This was a Java GUI based tool. Later, Scrub-PA was created to anonymize Process Accounting logs. Scrub-PA was based on the Java code used for CANINE. The development of both of these tools were funded under the Office of Naval Research
NCASSR research center through the SLAGEL project.
It was quickly realized that building one-off tools for each new log format was not the way to go. Also, the earlier tools were limited in that they could not be scripted from the command line. It was decided that a new, modular command line based UNIX
tool was needed. Because speed was also a concern, this tool need to be written in C++
. With the successful acquisition of a Cyber Trust grant from the National Science Foundation
, the LAIM
Working Group was formed at the NCSA
. From this project headed by the PI, Adam Slagell, FLAIM was developed to overcome these limitations of CANINE and Scrub-PA. The first public version of FLAIM, 0.4., was released on July 23, 2006.
FLAIM is aimed at 3 different user communities. First, FLAIM can be used by the security engineer who is investigating a broad incident spanning multiple organizations. Because of the sensitivity inherent in security relevant logs, many organizations are reluctant to share them. However, this reluctance inhibits the sharing necessary to investigate intrusions that commonly span organizational boundaries. Second, anyone designing log analysis or computer forensics
Computer forensics
Computer forensics is a branch of digital forensic science pertaining to legal evidence found in computers and digital storage media...
tools needs data with which they can test their tools. The larger and more diverse the data set, the more robust they can make their tools. For many, this means they must gather many logs from outside sources, not just what they can generate in-house. Again, this requires log sharing. Third, researchers in many computer science
Computer science
Computer science or computing science is the study of the theoretical foundations of information and computation and of practical techniques for their implementation and application in computer systems...
disciplines (e.g., network measurements
Network traffic measurement
In computer networks, network traffic measurement is the process of measuring the amount and type of traffic on a particular network. This is especially important with regard to effective bandwidth management.- Tools :...
, computer security
Computer security
Computer security is a branch of computer technology known as information security as applied to computers and networks. The objective of computer security includes protection of information and property from theft, corruption, or natural disaster, while allowing the information and property to...
, etc.) need large and diverse data sets to study. Having data sanitization tools available makes organizations more willing to share with these researchers their own logs.
FLAIM is available under the Open Source Initiative
Open Source Initiative
The Open Source Initiative is an organization dedicated to promoting open source software.The organization was founded in February 1998, by Bruce Perens and Eric S. Raymond, prompted by Netscape Communications Corporation publishing the source code for its flagship Netscape Communicator product...
approved University of Illinois/NCSA Open Source License. This is BSD-style license. It runs on Unix
Unix
Unix is a multitasking, multi-user computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe Ossanna...
and Unix-like
Unix-like
A Unix-like operating system is one that behaves in a manner similar to a Unix system, while not necessarily conforming to or being certified to any version of the Single UNIX Specification....
systems, including Linux
Linux
Linux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
, FreeBSD
FreeBSD
FreeBSD is a free Unix-like operating system descended from AT&T UNIX via BSD UNIX. Although for legal reasons FreeBSD cannot be called “UNIX”, as the direct descendant of BSD UNIX , FreeBSD’s internals and system APIs are UNIX-compliant...
, NetBSD
NetBSD
NetBSD is a freely available open source version of the Berkeley Software Distribution Unix operating system. It was the second open source BSD descendant to be formally released, after 386BSD, and continues to be actively developed. The NetBSD project is primarily focused on high quality design,...
, OpenBSD
OpenBSD
OpenBSD is a Unix-like computer operating system descended from Berkeley Software Distribution , a Unix derivative developed at the University of California, Berkeley. It was forked from NetBSD by project leader Theo de Raadt in late 1995...
and Mac OS X
Mac OS X
Mac OS X is a series of Unix-based operating systems and graphical user interfaces developed, marketed, and sold by Apple Inc. Since 2002, has been included with all new Macintosh computer systems...
.
While FLAIM is not the only log anonymizer, it is unique in its flexibility to create complex XML
XML
Extensible Markup Language is a set of rules for encoding documents in machine-readable form. It is defined in the XML 1.0 Specification produced by the W3C, and several other related specifications, all gratis open standards....
policies and its support for multiple log types. More specifically, it is the only such tool to meet the following 4 goals. (1) FLAIM provides a diverse set of anonymization primitives. (2) FLAIM supports multiple log type, including linux process accounting logs, netfilter alerts, tcpdump
Tcpdump
tcpdump is a common packet analyzer that runs under the command line. It allows the user to intercept and display TCP/IP and other packets being transmitted or received over a network to which the computer is attached...
traces and NFDUMP NetFlows
Netflow
NetFlow is a network protocol developed by Cisco Systems for collecting IP traffic information. NetFlow has become an industry standard for traffic monitoring and is supported by platforms other than Cisco IOS and NXOS such as Juniper routers, Enterasys Switches, vNetworking in version 5 of...
. (3) With a flexible anonymization policy language, complex policies that make trade-offs between information loss and security can be made. (4) FLAIM is modular and easily extensible to new types of logs and data. The anonymization engine is agnostic to the syntax of the actual log.
History
Work on log anonymization began in 2004 at the NCSANational Center for Supercomputing Applications
The National Center for Supercomputing Applications is an American state-federal partnership to develop and deploy national-scale cyberinfrastructure that advances science and engineering. NCSA operates as a unit of the University of Illinois at Urbana-Champaign but it provides high-performance...
. At first this was for anonymizing logs in-house to share with the SIFT group. Soon there was a need for more powerful anonymization and anonymization of different types of logs. CANINE was created to anonymize and convert between multiple formats of NetFlows
Netflow
NetFlow is a network protocol developed by Cisco Systems for collecting IP traffic information. NetFlow has become an industry standard for traffic monitoring and is supported by platforms other than Cisco IOS and NXOS such as Juniper routers, Enterasys Switches, vNetworking in version 5 of...
. This was a Java GUI based tool. Later, Scrub-PA was created to anonymize Process Accounting logs. Scrub-PA was based on the Java code used for CANINE. The development of both of these tools were funded under the Office of Naval Research
Office of Naval Research
The Office of Naval Research , headquartered in Arlington, Virginia , is the office within the United States Department of the Navy that coordinates, executes, and promotes the science and technology programs of the U.S...
NCASSR research center through the SLAGEL project.
It was quickly realized that building one-off tools for each new log format was not the way to go. Also, the earlier tools were limited in that they could not be scripted from the command line. It was decided that a new, modular command line based UNIX
Unix
Unix is a multitasking, multi-user computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe Ossanna...
tool was needed. Because speed was also a concern, this tool need to be written in C++
C++
C++ is a statically typed, free-form, multi-paradigm, compiled, general-purpose programming language. It is regarded as an intermediate-level language, as it comprises a combination of both high-level and low-level language features. It was developed by Bjarne Stroustrup starting in 1979 at Bell...
. With the successful acquisition of a Cyber Trust grant from the National Science Foundation
National Science Foundation
The National Science Foundation is a United States government agency that supports fundamental research and education in all the non-medical fields of science and engineering. Its medical counterpart is the National Institutes of Health...
, the LAIM
LAIM Working Group
The LAIM Working Group is a NSF and ONR funded research group at the National Center for Supercomputing Applications under the direction of . Work from this group focuses upon log anonymization and Internet privacy. The LAIM group, established in 2005, has released 3 different log anonymization...
Working Group was formed at the NCSA
National Center for Supercomputing Applications
The National Center for Supercomputing Applications is an American state-federal partnership to develop and deploy national-scale cyberinfrastructure that advances science and engineering. NCSA operates as a unit of the University of Illinois at Urbana-Champaign but it provides high-performance...
. From this project headed by the PI, Adam Slagell, FLAIM was developed to overcome these limitations of CANINE and Scrub-PA. The first public version of FLAIM, 0.4., was released on July 23, 2006.
Features
- Flexible XMLXMLExtensible Markup Language is a set of rules for encoding documents in machine-readable form. It is defined in the XML 1.0 Specification produced by the W3C, and several other related specifications, all gratis open standards....
policy language - Modular to support simple plugins for new log types
- Support for major UNIX-likeUnix-likeA Unix-like operating system is one that behaves in a manner similar to a Unix system, while not necessarily conforming to or being certified to any version of the Single UNIX Specification....
Operating Systems - Built-in support for several anonymization primitives
- Plugin for NFDUMP format NetFlowsNetflowNetFlow is a network protocol developed by Cisco Systems for collecting IP traffic information. NetFlow has become an industry standard for traffic monitoring and is supported by platforms other than Cisco IOS and NXOS such as Juniper routers, Enterasys Switches, vNetworking in version 5 of...
- Plugin for netfilter firewall logs
- Plugin for pcapPcapIn the field of computer network administration, pcap consists of an application programming interface for capturing network traffic...
traces form tcpdumpTcpdumptcpdump is a common packet analyzer that runs under the command line. It allows the user to intercept and display TCP/IP and other packets being transmitted or received over a network to which the computer is attached... - Plugin for linux process accounting logs