Syslog-ng
Encyclopedia
syslog-ng is an open source
implementation of the Syslog
protocol for Unix
and Unix-like
systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP
for transport. As of today syslog-ng is developed by Balabit
IT Security Ltd. It has two editions with a common codebase. The first is called syslog-ng OSE (with the license LGPL) and have additional plugins (modules) under proprietary
license. This edition is called Premium Edition (PE).
protocol, specified in RFC
3164. As the text of RFC 3164 is vague and is just an informational description and not a standard, various incompatible extensions of it emerged. Since version 3.0 also supports the standard syslog protocol specified in RFC 5424 which was released in 2009. syslog-ng tries hard to interoperate with a wide variety of devices, and the format of relayed messages can be customized.
The most important extensions of the original protocol endorsed by syslog-ng are:
Right after the release of syslog-ng 1.0.x, a reimplementation of the code base started to address some of the shortcomings of nsyslogd and to address the licensing concerns of Darren Reed, the original nsyslogd author. This reimplementation was named stable in the October of 1999 with the release of 1.2.0. This time around, syslog-ng depended on some code originally developed for lsh by Niels Möller.
Three major releases (1.2, 1.4 and 1.6) were using this code base, the last release of the 1.6.x branch in February 2007. In this period of about 8 years, syslog-ng became one of the most popular alternative syslog implementations.
In a volunteer based effort, yet another rewrite
was started back in 2001, dropping lsh code and using the more widely available GLib library. This rewrite of the codebase took its time, the first stable release of 2.0.0 happened in October 2006.
Development efforts were focused on improving the 2.0.x branch; support for 1.6.x was dropped at the end of 2007. Support for 2.X was dropped at the end of 2009, but it is still used in some Linux distributions. Balabit
, the company behind syslog-ng, started a parallel, commercial fork of syslog-ng, called syslog-ng Premium Edition. Portions of the commercial income are used to sponsor development of the free version.
Syslog-ng version 3.0 was released in the fourth quarter of 2008.
Starting with the 3.0 version developments efforts were parallel on the Premium and on the Open Source Editions. PE efforts were focused on quality, transport reliability, performance and encrypted log storage. The Open Source Edition efforts focused on improving the flexibility of the core infrastructure to allow more and more different, non-syslog message sources.
Both the OSE & PE forks produced two releases (3.1 and 3.2) in 2010.
Among others:
The list above is based on BalaBit's current first hand experience, other platforms may also work, but your mileage may vary.
Open source
The term open source describes practices in production and development that promote access to the end product's source materials. Some consider open source a philosophy, others consider it a pragmatic methodology...
implementation of the Syslog
Syslog
Syslog is a standard for computer data logging. It allows separation of the software that generates messages from the system that stores them and the software that reports and analyzes them...
protocol for Unix
Unix
Unix is a multitasking, multi-user computer operating system originally developed in 1969 by a group of AT&T employees at Bell Labs, including Ken Thompson, Dennis Ritchie, Brian Kernighan, Douglas McIlroy, and Joe Ossanna...
and Unix-like
Unix-like
A Unix-like operating system is one that behaves in a manner similar to a Unix system, while not necessarily conforming to or being certified to any version of the Single UNIX Specification....
systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP
Transmission Control Protocol
The Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
for transport. As of today syslog-ng is developed by Balabit
Balabit
BalaBit IT Security, founded in 1996, is a software company specializing in the development of IT security systems and related services.-History:...
IT Security Ltd. It has two editions with a common codebase. The first is called syslog-ng OSE (with the license LGPL) and have additional plugins (modules) under proprietary
Proprietary software
Proprietary software is computer software licensed under exclusive legal right of the copyright holder. The licensee is given the right to use the software under certain conditions, while restricted from other uses, such as modification, further distribution, or reverse engineering.Complementary...
license. This edition is called Premium Edition (PE).
Protocol
syslog-ng uses the quasi-standard BSD syslogSyslog
Syslog is a standard for computer data logging. It allows separation of the software that generates messages from the system that stores them and the software that reports and analyzes them...
protocol, specified in RFC
Request for Comments
In computer network engineering, a Request for Comments is a memorandum published by the Internet Engineering Task Force describing methods, behaviors, research, or innovations applicable to the working of the Internet and Internet-connected systems.Through the Internet Society, engineers and...
3164. As the text of RFC 3164 is vague and is just an informational description and not a standard, various incompatible extensions of it emerged. Since version 3.0 also supports the standard syslog protocol specified in RFC 5424 which was released in 2009. syslog-ng tries hard to interoperate with a wide variety of devices, and the format of relayed messages can be customized.
The most important extensions of the original protocol endorsed by syslog-ng are:
- ISO 8601ISO 8601ISO 8601 Data elements and interchange formats – Information interchange – Representation of dates and times is an international standard covering the exchange of date and time-related data. It was issued by the International Organization for Standardization and was first published in 1988...
timestamp with millisecond granularity and timezone information - the addition of the name of relays in the host fields to make it possible to track the path a given message has traversed
- reliable transport using TCPTransmission Control ProtocolThe Transmission Control Protocol is one of the core protocols of the Internet Protocol Suite. TCP is one of the two original components of the suite, complementing the Internet Protocol , and therefore the entire suite is commonly referred to as TCP/IP...
- TLSTransport Layer SecurityTransport Layer Security and its predecessor, Secure Sockets Layer , are cryptographic protocols that provide communication security over the Internet...
encryption (Since 3.0.1 in OSE )
History
The syslog-ng project began in 1998, when Balázs Scheidler, the primary author of syslog-ng, ported the existing nsyslogd code to Linux. The 1.0.x branch of syslog-ng was still based on the nsyslogd sources and are available in the syslog-ng source archive.Right after the release of syslog-ng 1.0.x, a reimplementation of the code base started to address some of the shortcomings of nsyslogd and to address the licensing concerns of Darren Reed, the original nsyslogd author. This reimplementation was named stable in the October of 1999 with the release of 1.2.0. This time around, syslog-ng depended on some code originally developed for lsh by Niels Möller.
Three major releases (1.2, 1.4 and 1.6) were using this code base, the last release of the 1.6.x branch in February 2007. In this period of about 8 years, syslog-ng became one of the most popular alternative syslog implementations.
In a volunteer based effort, yet another rewrite
Rewrite (programming)
A rewrite in computer programming is the act or result of re-implementing a large portion of existing functionality without re-use of its source code. When the rewrite is not using existing code at all, it is common to speak of a rewrite from scratch...
was started back in 2001, dropping lsh code and using the more widely available GLib library. This rewrite of the codebase took its time, the first stable release of 2.0.0 happened in October 2006.
Development efforts were focused on improving the 2.0.x branch; support for 1.6.x was dropped at the end of 2007. Support for 2.X was dropped at the end of 2009, but it is still used in some Linux distributions. Balabit
Balabit
BalaBit IT Security, founded in 1996, is a software company specializing in the development of IT security systems and related services.-History:...
, the company behind syslog-ng, started a parallel, commercial fork of syslog-ng, called syslog-ng Premium Edition. Portions of the commercial income are used to sponsor development of the free version.
Syslog-ng version 3.0 was released in the fourth quarter of 2008.
Starting with the 3.0 version developments efforts were parallel on the Premium and on the Open Source Editions. PE efforts were focused on quality, transport reliability, performance and encrypted log storage. The Open Source Edition efforts focused on improving the flexibility of the core infrastructure to allow more and more different, non-syslog message sources.
Both the OSE & PE forks produced two releases (3.1 and 3.2) in 2010.
Features
syslog-ng has a much larger scope than merely transporting syslog messages and storing them to plain text log files:- the ability to format log messages using UNIX shell-like variable expansion;
- the use of this shell-like variable expansion when naming files, thus covering thousands of destination files with a single statement;
- the ability to send log messages to local applications;
- ability to message flow-control in network transport;
- logging directly into a database (since syslog-ng OSE 2.1);
- rewrite portions of the syslog message with set and substitute primitives (since syslog-ng OSE 3.0);
- classify incoming log messages and at the same time extract structured information from the unstructured syslog message (since syslog-ng OSE 3.0);
- generic name-value support: each message is just a set of name-value pairs, which can be used to store extra information (since syslog-ng OSE 3.0);
- the ability to process structured message formats transmitted over syslog, like extract columns from CSV formatted lines (since syslog-ng OSE 3.0);
- the ability to correlate multiple incoming messages to form a more complex, correlated event (since syslog-ng OSE 3.2);
Distributions
syslog-ng is part of a number of different GNU/Linux and Unix distributions. Some distributions install it as the default system logger, others only provide a package and an upgrade path from the standard syslogd.Among others:
- openSUSEOpenSUSEopenSUSE is a general purpose operating system built on top of the Linux kernel, developed by the community-supported openSUSE Project and sponsored by SUSE...
used it as default prior to openSUSE 11.2, and is still available - SLESSLESSLES is an initialism for:* Sodium lauryl ether sulfate* SUSE Linux Enterprise Server, a server-oriented Linux distribution supplied by Novell, Inc. and targeted at the business market....
uses it as default - Debian GNU/Linux used before version 5.0 syslogd and klogd (Lenny (5.0) uses RsyslogRsyslogRsyslog is an open source software utility used on UNIX and Unix-like computer systems for forwarding log messages in an IP network. It implements the basic syslog protocol, extends it with content-based filtering, rich filtering capabilities, flexible configuration options and adds important...
)
- Gentoo LinuxGentoo LinuxGentoo Linux is a computer operating system built on top of the Linux kernel and based on the Portage package management system. It is distributed as free and open source software. Unlike a conventional software distribution, the user compiles the source code locally according to their chosen...
- FedoraFedora (operating system)Fedora is a RPM-based, general purpose collection of software, including an operating system based on the Linux kernel, developed by the community-supported Fedora Project and sponsored by Red Hat...
used it prior to Fedora 10 - Arch LinuxArch LinuxArch Linux is an independently developed, Linux-based operating system for i686 and x86-64 computers. It is composed predominantly of free and open source software, and supports community involvement....
- Hewlett-PackardHewlett-PackardHewlett-Packard Company or HP is an American multinational information technology corporation headquartered in Palo Alto, California, USA that provides products, technologies, softwares, solutions and services to consumers, small- and medium-sized businesses and large enterprises, including...
's HP-UXHP-UXHP-UX is Hewlett-Packard's proprietary implementation of the Unix operating system, based on UNIX System V and first released in 1984... - FreeBSDFreeBSDFreeBSD is a free Unix-like operating system descended from AT&T UNIX via BSD UNIX. Although for legal reasons FreeBSD cannot be called “UNIX”, as the direct descendant of BSD UNIX , FreeBSD’s internals and system APIs are UNIX-compliant...
- A CygwinCygwinCygwin is a Unix-like environment and command-line interface for Microsoft Windows. Cygwin provides native integration of Windows-based applications, data, and other system resources with applications, software tools, and data of the Unix-like environment...
port is available for Microsoft WindowsMicrosoft WindowsMicrosoft Windows is a series of operating systems produced by Microsoft.Microsoft introduced an operating environment named Windows on November 20, 1985 as an add-on to MS-DOS in response to the growing interest in graphical user interfaces . Microsoft Windows came to dominate the world's personal...
Portability
syslog-ng is highly portable to many Unix systems, old and new alike. A list of the currently known to work Unix versions are found below:- LinuxLinuxLinux is a Unix-like computer operating system assembled under the model of free and open source software development and distribution. The defining component of any Linux system is the Linux kernel, an operating system kernel first released October 5, 1991 by Linus Torvalds...
on i386, ARMARMAn arm is an upper limb of the body.Arm may also refer to:-Geography:* Arm , a narrow stretch of a larger body of water** Canal arm, a subsidiary branch of a canal or inland waterway** Distributary or arm, a subsidiary branch of a river...
, PowerPC, SPARCSPARCSPARC is a RISC instruction set architecture developed by Sun Microsystems and introduced in mid-1987....
and x86-64X86-64x86-64 is an extension of the x86 instruction set. It supports vastly larger virtual and physical address spaces than are possible on x86, thereby allowing programmers to conveniently work with much larger data sets. x86-64 also provides 64-bit general purpose registers and numerous other...
CPUs - FreeBSDFreeBSDFreeBSD is a free Unix-like operating system descended from AT&T UNIX via BSD UNIX. Although for legal reasons FreeBSD cannot be called “UNIX”, as the direct descendant of BSD UNIX , FreeBSD’s internals and system APIs are UNIX-compliant...
7.x - 9.x on i386 and x86-64X86-64x86-64 is an extension of the x86 instruction set. It supports vastly larger virtual and physical address spaces than are possible on x86, thereby allowing programmers to conveniently work with much larger data sets. x86-64 also provides 64-bit general purpose registers and numerous other...
CPUs - AIX 5 on IBM POWERIBM POWERPOWER is a reduced instruction set computer instruction set architecture developed by IBM. The name is an acronym for Performance Optimization With Enhanced RISC....
CPUs - HP-UXHP-UXHP-UX is Hewlett-Packard's proprietary implementation of the Unix operating system, based on UNIX System V and first released in 1984...
11iv1, 11iv2 on PA-RISCPA-RISCPA-RISC is an instruction set architecture developed by Hewlett-Packard. As the name implies, it is a reduced instruction set computer architecture, where the PA stands for Precision Architecture...
and ItaniumItaniumItanium is a family of 64-bit Intel microprocessors that implement the Intel Itanium architecture . Intel markets the processors for enterprise servers and high-performance computing systems...
CPUs - Solaris 8, 9, 10 on SPARC, x86-64 and i386 CPUs
- Tru64 5.1b on AlphaAlphaAlpha is the first letter of the Greek alphabet. Alpha or ALPHA may also refer to:-Science:*Alpha , the highest ranking individuals in a community of social animals...
CPUs
The list above is based on BalaBit's current first hand experience, other platforms may also work, but your mileage may vary.
Related RFCs & working groups
- RFC 3164 - The BSD syslog protocol
- RFC 5424 - The Syslog Protocol
- RFC 5425 - Transport Layer Security (TLS) Transport Mapping for Syslog
- RFC 5426 - Transmission of Syslog Messages over UDP
Official links
External links
- Michael D. Bauer: Linux Server Security, Second Edition published 2005 at O'Reilly: System Log Management and Monitoring (Chapter 12)
- XLog-Solution is a centralized multi-platform log manager for systems and software.
- Php-syslog-ng - a web interface and reporting tool for syslog-ng data
- LogZilla is a scalable, multi-vendor, Enterprise Class syslog event viewer for centralized network event and problem management.
- syslog-ng FAQ
- syslog-ng support wiki & forum
- Syslog-ng and vlogger meet