Log management and intelligence
Encyclopedia
Log management comprises an approach to dealing with large volumes of computer
-generated log messages (also known as audit records, audit trail
s, event-logs, etc.). LM covers log collection, centralized aggregation, long-term retention, log analysis
(in real-time and in bulk after storage) as well as log search and reporting.
Log management is driven by reasons of security
, system and network operations (such as system
or network administration
) and regulatory compliance.
Effectively analyzing large volumes of diverse logs can pose many challenges — such as huge log-volumes (reaching hundreds of gigabytes of data per day for a large organization
), log-format diversity, undocumented proprietary log-formats (that resist analysis) as well as the presence of false log records in some types of logs (such as intrusion-detection
logs).
Users and potential users of LM can build their own log management and intelligence tools, assemble the functionality from various open-source
components, or acquire (sub-)systems from commercial vendors. Log management is a complicated process and organizations often make mistakes while approaching it .
Computer
A computer is a programmable machine designed to sequentially and automatically carry out a sequence of arithmetic or logical operations. The particular sequence of operations can be changed readily, allowing the computer to solve more than one kind of problem...
-generated log messages (also known as audit records, audit trail
Audit trail
Audit trail is a sequence of steps supported by proof documenting the real processing of a transaction flow through an organization, a process or a system.....
s, event-logs, etc.). LM covers log collection, centralized aggregation, long-term retention, log analysis
Log analysis
Log analysis is an art and science seeking to make sense out of computer-generated records...
(in real-time and in bulk after storage) as well as log search and reporting.
Log management is driven by reasons of security
Computer security
Computer security is a branch of computer technology known as information security as applied to computers and networks. The objective of computer security includes protection of information and property from theft, corruption, or natural disaster, while allowing the information and property to...
, system and network operations (such as system
System administrator
A system administrator, IT systems administrator, systems administrator, or sysadmin is a person employed to maintain and operate a computer system and/or network...
or network administration
Network administrator
A network administrator, network analyst or network engineer is a person responsible for the maintenance of computer hardware and software that comprises a computer network...
) and regulatory compliance.
Effectively analyzing large volumes of diverse logs can pose many challenges — such as huge log-volumes (reaching hundreds of gigabytes of data per day for a large organization
Organization
An organization is a social group which distributes tasks for a collective goal. The word itself is derived from the Greek word organon, itself derived from the better-known word ergon - as we know `organ` - and it means a compartment for a particular job.There are a variety of legal types of...
), log-format diversity, undocumented proprietary log-formats (that resist analysis) as well as the presence of false log records in some types of logs (such as intrusion-detection
Intrusion detection
In Information Security, intrusion detection is the act of detecting actions that attempt to compromise the confidentiality, integrity or availability of a resource. When Intrusion detection takes a preventive measure without direct human intervention, then it becomes an Intrusion-prevention...
logs).
Users and potential users of LM can build their own log management and intelligence tools, assemble the functionality from various open-source
Open source
The term open source describes practices in production and development that promote access to the end product's source materials. Some consider open source a philosophy, others consider it a pragmatic methodology...
components, or acquire (sub-)systems from commercial vendors. Log management is a complicated process and organizations often make mistakes while approaching it .
Log Management Key Features and Technology
The deployment of a Log Management architecture generally involves the following steps:- Step 1: Define the requirement and goals. Needs can be security log analysisLog analysisLog analysis is an art and science seeking to make sense out of computer-generated records...
, application problem analysis, or reporting for the purposes of regulatory compliance.
- Step 2: Define the logging framework, log types, and system specification where logs are generated.
- Step 3: Determine what you’re going to use log management for according to your goals. Are you going to collect the logs? Maybe you need to analyze or even report and monitor the logs on remote machine. If you plan on collecting log data, how long will it need to be archived? Is it going to be encrypted? Regulatory compliance may provide specification for such needs.
- Step 4: What information and intelligence are you planning to extract out of your log? End user patterns reports, application problems and more can be taken.
- Step 5: Evaluate technology and vendors solution to select the best fit to your needs. You may also select to build a log management solution internally, leveraging open source solutions. Add a reporting and analysis layer later on for intelligence.
Deployment life-cycle
One view of assessing the maturity of an organization in terms of the deployment of log-management tools might use successive categories such as:- Level 1: in the initial stages, organizations use different log-analyzers for analyzing the logs in the devices on the security-perimeter. They aim to identify the patterns of attack on the perimeter infrastructure of the organization.
- Level 2: with increased use of integrated computing, organizations mandate logs to identify the access and usage of confidential data within the security-perimeter.
- Level 3: at the next level of maturity, the log analyzer can track and monitor the performance and availability of systems at the level of the enterpriseBusinessA business is an organization engaged in the trade of goods, services, or both to consumers. Businesses are predominant in capitalist economies, where most of them are privately owned and administered to earn profit to increase the wealth of their owners. Businesses may also be not-for-profit...
— especially of those information-assets whose availability organizations regard as vital.
- Level 4: organizations integrate the logs of various businessBusinessA business is an organization engaged in the trade of goods, services, or both to consumers. Businesses are predominant in capitalist economies, where most of them are privately owned and administered to earn profit to increase the wealth of their owners. Businesses may also be not-for-profit...
-applications into an enterprise log manager for better value propositionValue propositionA value proposition is a promise of value to be delivered and a belief from the customer of value that will be experienced. A value proposition can apply to an entire organization, or parts thereof, or customer accounts, or products or services....
.
- Level 5: organizations merge the physical-access monitoring and the logical-access monitoring into a single view.
See also
- Log management knowledge baseLog management knowledge baseThe Log Management Knowledge Base is a free database of detailed descriptions on over 20,000 event logs generated by Windows systems, syslog devices and applications...
- Audit trailAudit trailAudit trail is a sequence of steps supported by proof documenting the real processing of a transaction flow through an organization, a process or a system.....
- Server logServer logA server log is a log file automatically created and maintained by a server of activity performed by it.A typical example is a web server log which maintains a history of page requests. The W3C maintains a standard format for web server log files, but other proprietary formats exist...
- Log analysisLog analysisLog analysis is an art and science seeking to make sense out of computer-generated records...
- Web log analysis softwareWeb log analysis softwareWeb log analysis software is a simple kind of Web analytics software that parses a log file from a web server, and based on the values contained in the log file, derives indicators about who, when, and how a web server is visited...
- Web counterWeb counterA web counter or hit counter is a computer software program that indicates the number of visitors, or hits, a particular webpage has received...
- Data logging
- Common Log FormatCommon Log FormatThe Common Log Format is a standardised text file format used by web servers when generating log files. Because the format is standardised, the files may be analysed by a variety of analysis programs....
- SyslogSyslogSyslog is a standard for computer data logging. It allows separation of the software that generates messages from the system that stores them and the software that reports and analyzes them...
- Common Base EventCommon Base EventCommon Base Event is an IBM implementation of the Web Services Distributed Management Web Event Format standard. IBM also implemented the Common Event Infrastructure, a unified set of APIs and infrastructure for the creation, transmission, persistence and distribution of a wide range of business,...