Logentries Docs

Find comprehensive guides and documentation to help you start and continue to work with Logentries.

    

Search results for "{{ search.query }}"

No results found for "{{search.query}}".

Linux Agent

Logentries Linux Agent

Logentries agent

A command line utility for a convenient access to Logentries logging
infrastructure.

su -
echo 'deb http://rep.logentries.com/ xenial main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys A5270289C43C79AD && gpg -a --export A5270289C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon
sudo -sH
echo 'deb http://rep.logentries.com/ trusty main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys A5270289C43C79AD && gpg -a --export A5270289C43C79AD | apt-key add -
apt-get update
apt-get install logentries
le register
apt-get install logentries-daemon
sudo -sH
echo 'deb http://rep.logentries.com/ saucy main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys A5270289C43C79AD && gpg -a --export A5270289C43C79AD | apt-key add -
apt-get update
apt-get install logentries
le register
apt-get install logentries-daemon
sudo -sH
echo 'deb http://rep.logentries.com/ raring main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys A5270289C43C79AD && gpg -a --export A5270289C43C79AD | apt-key add -
apt-get update
apt-get install logentries
le register
apt-get install logentries-daemon
sudo -sH
echo 'deb http://rep.logentries.com/ quantal main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys A5270289C43C79AD && gpg -a --export A5270289C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon
sudo -sH
echo 'deb http://rep.logentries.com/ precise main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys A5270289C43C79AD && gpg -a --export A5270289C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon
sudo -sH
echo 'deb http://rep.logentries.com/ oneiric main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys A5270289C43C79AD && gpg -a --export A5270289C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon
sudo -sH
echo 'deb http://rep.logentries.com/ natty main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys A5270289C43C79AD && gpg -a --export A5270289C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon
sudo -sH
echo 'deb http://rep.logentries.com/ lucid main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys A5270289C43C79AD && gpg -a --export A5270289C43C79AD | apt-key add -
apt-get update
apt-get install logentries
le register
apt-get install logentries-daemon
su -
tee /etc/yum.repos.d/logentries.repo <<EOF
[logentries]
name=Logentries repo
enabled=1
metadata_expire=1d
baseurl=http://rep.logentries.com/rh/\$basearch
gpgkey=http://rep.logentries.com/RPM-GPG-KEY-logentries
EOF
yum update
yum install logentries
le register
yum install logentries-daemon
su -
yum-config-manager --add-repo http://rep.logentries.com/helpers/fedora/logentries.repo
yum install logentries
le register
yum install logentries-daemon
su -
yum-config-manager --add-repo http://rep.logentries.com/helpers/fedora20/logentries.repo
yum install logentries
le register
yum install logentries-daemon
su -
tee /etc/yum.repos.d/logentries.repo <<EOF
[logentries]
name=Logentries repo
enabled=1
metadata_expire=1d
baseurl=http://rep.logentries.com/centos6/\$basearch
gpgkey=http://rep.logentries.com/RPM-GPG-KEY-logentries
EOF
yum update
yum install logentries
le register
yum install logentries-daemon
su -
tee /etc/yum.repos.d/logentries.repo <<EOF
[logentries]
name=Logentries repo
enabled=1
metadata_expire=1d
gpgcheck=0
baseurl=http://rep.logentries.com/rh/\$basearch
gpgkey=http://rep.logentries.com/RPM-GPG-KEY-logentries
EOF
yum update
yum install logentries
le register
yum install logentries-daemon
sudo -s
tee /etc/yum.repos.d/logentries.repo <<EOF
[logentries]
name=Logentries repo
enabled=1
metadata_expire=1d
baseurl=http://rep.logentries.com/amazonlatest/\$basearch
gpgkey=http://rep.logentries.com/RPM-GPG-KEY-logentries
EOF
yum update
yum install logentries
le register
yum install logentries-daemon
su -
echo 'deb http://rep.logentries.com/ wheezy main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys A5270289C43C79AD && gpg -a --export A5270289C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon
su -
echo 'deb http://rep.logentries.com/ jessie main' > /etc/apt/sources.list.d/logentries.list
gpg --keyserver pgp.mit.edu --recv-keys A5270289C43C79AD && gpg -a --export A5270289C43C79AD | apt-key add -
apt-get update
apt-get install python-setproctitle logentries
le register
apt-get install logentries-daemon

Command Line Arguments

usage: le COMMAND [ARGS]

Where COMMAND is one of:
init      Write local configuration file
reinit    As init but does not reset undefined parameters
register  Register this host
--name=  name of the host
--hostname=  hostname of the host
whoami    Displays settings for this host
monitor   Monitor this host
follow <filename>  Follow the given log
--name=  name of the log
--type=  type of the log
followed <filename>  Check if the file is followed
clean     Removes configuration file
ls        List internal filesystem and settings: <path>
ls ips    List IP addresses used by the agent
rm        Remove entity: <path>
pull      Pull log file: <path> <when> <filter> <limit>

Where ARGS are:
--help            show usage help and exit
--version         display version number and exit
--account-key=    set account key and exit
--host-key=       set local host key and exit, generate key if key is empty
--no-timestamps   no timestamps in agent reportings
--force           force given operation
--datahub         send logs to the specified data hub address
                  the format is address:port with port being optional
--suppress-ssl    do not use SSL with API server
--yes              always respond yes
--pull-server-side-config=False do not use server-side config for following files

Installation

There are three ways to install the LE Agent.

  1. Using the Configuration based setup. This setup is the recommend way of getting setup with the Agent.
  2. Interactive - Simply run sudo bash logentries_install.sh. This will download and install the LE Agent on your machine and prompt you for your Logentries account email and Logentries account password.
  3. Automated, using your Logentries' account key - Run the Linux installer using your Logentries Account Key as the first command line arguemnt as in sudo bash logentries_install.sh <account_key> for example sudo bash logentries_install.sh xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. This will bypass the prompts for your Email or password and simply download and install the LE Agent adding this Host and its Logs to your Account.

To automatically run the install script run the following command.

wget https://raw.github.com/logentries/le/master/install/linux/logentries_install.sh && sudo bash logentries_install.sh

To attain your Logentries Account Key from the Logentries web UI see: https://logentries.com/doc/accountkey/

Uninstall the Agent

To remove the agent run the following commands:

sudo service logentries stop
sudo apt-get remove logentries-daemon
sudo le clean
sudo apt-get remove logentries
sudo rm -rf /root/.cache/logentries/

Configuration file

The agent stores configuration in ~/.le/config for ordinary users and in
/etc/le/config for root (daemon). It is created with init or reinit
commands and can be created or modified manually.

The agent supports loading multiple configuration files from a configuration
directory. By default the configuration directory is ~/.le/conf.d/ for
ordinary users and /etc/le/conf.d for root (daemon). Only files with .conf
extension are recognized as configuration files.

The structure of a configuration file follows standard similar to what you
find in .git/config or Windows INI files. For example:

[Main]
user-key = e720a1e8-a7d5-4f8b-8879-854e51c9290d
agent-key = 428b888a-29ab-4079-99ec-9cb7aa2ffea7

[cassandra]
metrics-process = org.apache.cassandra.service.CassandraDaemon
path = /var/log/cassandra/system.log
token = a846bd59-a674-4088-b9fd-e72da1df5946

Main section [Main] contains agent-wide general configuration. Any other
section defines per-application settings such as log filenames and metrics.

In the main section, user-key (account key) which identifies account, and
agent-key which identifies host (host key).

Note the monitor command requires both user-key and agent-key defined.

User-specified configuration directory

Additionaly to the default configuration directory, the agent allows to specify
a secondary configuration directory. Just specify the include parameter in
the main section as follows:

[Main]
user-key = e720a1e8-a7d5-4f8b-8879-854e51c9290d
include = /opt/configurations/logentries/conf.d

Again, the agent recognizes all files with .conf extension.

Follow log files through server-side configuration

After registering the host (via register command or specifying agent-key in
configuration) you can add a file to follow via follow command:

sudo le follow /srv/log/cassandra/system.out [--name Cassandra]

You can repeat the command for additional logs. The agent creates a new log
entry in Logentries under the host specified. It will also enable the file to
be followed by the agent.

Note --name is optional to specify log name as it will appear in UI and log
listing. If not specified, plain file name is used.

You need to restart the agent to pick up the new configuration:

sudo service logentries restart

Follow log files through your configuration file

In an auto scaling environment you may not want to create a Host each time you
install the agent.

To disable pulling server-side configuration (and thus avoiding communication
with Logentries API) add this line in the [Main] section of the configuration:

pull-server-side-config=False

Or specify --pull-server-side-config=False on the command like for the init
or reinit commands:

sudo le reinit --pull-server-side-config=False

By default, locally configured logs are sent to Logentries in Syslog format RFC 5424 which prepends a timestamp and other useful information. If you wish to disable this, you can set the formatter to 'plain' in the [Main] section of the configuration.

formatter = plain

You can reuse the same configuration file multiple times without creating new hosts.
Each log to follow has a separate section in the configuration of the form:

[name]
path = /path/to/log/file
token = MY_TOKEN

Where:

  • name is an identifier of the application that is added to your log entries
  • path is an absolute path to the file you wish to follow
  • token is the token for destination log created in Logentries

Alternatively, instead of token specify destination parameter in the format
of `host name/log name'. The agent will search for the host and log identified
by their name and retrieve the token automatically. If the host or log does not
exist, it is created.

Note: When using the destination parameter it is advised not to initialize multiple agents
with the same configuration file at the same time. This is to prevent a race condition where
duplicate Log Sets may be created.

Example:

[name]
path = /path/to/log/file
destination = MyHost/MyLog

Follow logs that change their names

Certain log rollover policies do not allow to specify destination file name.
That is typical when log files are timestamped or assigned a sequential number.
The Logentries agent can handle there cases for you. The Logentries agent can
be pointed at particular folders to gather any active logs from that directory
or its subdirectories using wildcards in file names. For example, the
following patterns can be used with the follow command to gather logs from the
given directories:

/var/log/mysystem/mylog-*.log

Using wildcards when specifying the log to follow allows for situations where
you need to follow the most recent log in a particular folder. The agent looks
for any active log in the folder and will monitor the events in that log.

Note wildcards are NOT needed in typical syslog log rotation scheme, where
log file named xxx.log is renamed to xxx.log.0 and a new xxx.log file is
created. In this situation follow the xxx.log file only, do not specify
wildcards.

Manipulate your data in transit

If you want to modify log entries before they are sent to Logentries, the agent
enables you to do so via filters. Filters are useful for filtering sensitive
information, obfuscating, or explicit parsing (adding key-value pairs).

Specify a Python module directory in your configuration by adding a line in the form of:

filters = /opt/le/le_filters

Create empty __init__.py to set up a module. Then add filters.py file which
contains filters dictionary. The dictionary informs the agent that for the
given log name, log ID, or token, the specified filtering function should be
used. For example the following dictionary:

filters = {
    "example.log": filter_logname,
    "7e518e54-40e4-4c5a-88df-4559d03126e6": filter_logid,
}

Where filter_logname and filter_loguuid are functions which filters events
for the respective log. Filtering functions receive a single string containing
log entries terminated with a new line. Function can modify input entry in any way
and return is back for sending to Logentries servers.
The following skeleton displays typical structure of the
filtering function:

def filter_example(entry):
    # Do something with entry
    new_entry = entry # XXX
    # Return modified output
    return new_entry

Typical filtering function is much simpler though. For example the following
filtering function removes all occurrences of credit card numbers:

import re

# Credit card number matcher
CREDIT_CARD = re.compile( r'\d{4}-\d{4}-\d{4}-\d{4}')
# Credit card number replacement
CC_REPLACEMENT = 'xxxx-xxxx-xxxx-xxxx'  # '-'.join( ['x'*4]*4) if you prefer

def filter_credit_card( events):
    return CREDIT_CARD.sub( CC_REPLACEMENT, events)

Format output entries

The agent allows to format log entries on transit. By default the agent formats entries according to syslog format specification RFC 5424 for locally configured log and plain format for server-side logs.

Default formatter can be overriden by global formatter specified in [Main] section. Formatter can be also specified for each application which has a precedence. Lastly, formatters can be implemented by a user-specified code.

Standard formatters

Standard formatter are plain which does not format entries at all, and syslog which formats log entries according to syslog format RFC 5424:

formatter = syslog

Custom formatters

Format of log entry can be specified directly as a simple substitution template. Selected $-prepended variables are substituted with real values. Recognized variables are:

$isodatetime ISO date time, for example `2015-08-31T23:05:34.159291`
$appname application's name
$hostname local hostname
$line log entry

For example, the following specification:

formatter = $isodatetime $appname[$hostname]: $line

Will result in the following log entry:

2015-08-31T23:05:34.159350 web[myhost]: GET /

User-implemented formatters

If the standard set of formatters or custom templates do not satisfy your needs, you may provide your own Python implementation.

Specify a Python module directory in your configuration by adding a line in the form of:

    formatters = /opt/le/le_formatters

Create empty __init__.py to set up a module. The add formatters.py file which contains the formatters dictionary. The dictionary informs the agent that for the given log name, log ID, or token, the specified formatting function should be used. For example the following dictionary:

formatters: {
    'apache': setup_apache_filter,
    'ba543c25-844c-4505-be10-b5aa0b678328': setup_cassandra_filter,
}

will assign a filter crated by setup_apache_filter function to any log with apache name. Similarly, log with the token ba543c25-844c-4505-be10-b5aa0b678328 will be formatted by output of the setup_cassandra_filter function.

Setup functions must accept hostname, log name, and token and return a function that accepts log entry. For example:

class Form(object):
def init(self, hostname, log_name, token):
self.hostname = hostname
self.log_name = log_name
self.token = token

def format_line(self, line):
    return '%sapache on %s: %s'%(self.token, self.hostname, line)

formatters = {
'apache' : lambda hostname, log_name, token: Form( hostname, log_name, token).format_line,
}

Filtering file names

If you want to explicitly restrict which files can the agent follow, create the
filters module as described in the previous section and define the
filter_filenames function. The filter_filenames function accepts full path to a
file which is about to be followed. The function returns True if the file name
is acceptable or False otherwise. The agent will ignore files which does not
pass this test. The following example defines filter which allows the agent to
follow log files only:

def filter_filenames( filename):
    return filename.endswith( '.log')

Alternatively, the following example defines filter which denies to follow any
file outside /var/log/ directory:

def filter_filenames( filename):
    return filename.startswith( '/var/log/')

Note the examples above do not take into account symbolic links.

Multiline log entries

If you want to merge multiple lines into one entry, use the entry_identifier parameter. It defines via regular expression the beginning of a new entry. In most cases it is a timestamp.

The entry_identifier can be used in the global Main section to be applied for all logs, or in individual sections.

For example, the following pattern identifies entries based on timestamp encoded in regular expression:

[cassandra]
entry_identifier = \d{4}-\d\d-\d\d \d\d:\d\d:\d\d,\d{3}

Unfollow Log Files

If you haven’t set the Agent to use its local configuration only, use the le rm command as shown below:

  1. Get a list of logs the agent is following on your host by running:
    sudo le ls hosts/<HOST_MACHINE_NAME>/
    
    For example, you would run the following for a host named “linuxbox”:
    sudo le ls hosts/linuxbox/
    
    If you’re not sure what your machine’s hostname is, you can find it by copying and pasting the following into your terminal:
    echo $HOSTNAME2.
    
    Instruct the agent to unfollow the log:
    Run the le rm command using the log you obtained in Step 1:
    sudo le rm /hosts/linuxbox/myLog.log`
    
    You will then receive confirmation that the log has been removed:
    Log myLog.log removed
    

Linux Agent

Logentries Linux Agent