Logentries Docs

Find comprehensive guides and documentation to help you start and continue to work with Logentries.

    

Search results for "{{ search.query }}"

No results found for "{{search.query}}".
View All Results

JSON & KVP Parsing

JSON structured logs are regularly produced by software components and are promoted as a logging best practice. JSON formatted logs are easily produced and consumed by a variety of software components and are human readable. If you are not ready to make the move to JSON, using Key-Value Pairs into your logs provides similar benefits. Both formats will allow you to take advantage of our advanced search and analytic capabilities.

Automatic Parsing

Logentries will automatically identify and index fields within your JSON and KVP log events. Keys are automatically highlighted and clickable. Clicking on a field will populate the search bar to allow for quick searching of across your data (Key Highlighting is temporarily disabled). The specification for which log formats will be parsed automatically is detailed in the JSON and KVP Specification Section of this document.

JSON Parsing Specification

The system will automatically parse JSON formatted log events formatted in compliance with the JSON Specification (IETF RFC 4627). To take advantage of the prettification option, the entry must be a single object ({ … }) or array of JSON objects. JSON objects that appear embedded in unstructured data will still have their keys recognized but you will not be able to expand the JSON hierarchy.

KVP Parsing Specification

Logentries is able to automatically parse a wide variety of KVP formats in compliance with the specification detailed below.

Specification

A key value pair consists of an identifier called a KEY, followed by a SEPARATOR, a VALUE and has a prefix and suffix termed the KVDELIMITER

KVDELIMITER KEY {whitespace} SEPARATOR {whitespace} VALUE{UNIT} KVDELIMITER

Term Definitions

KVDELIMITER

*can be space (” “), comma (“,”), tab “\t” or “\r” or “\n” or “\r\n”
KEY

  • the allowed list of characters is any Alphanumeric plus “_” “.” “$” “@”
  • must have at least one non-numeric character from the allowed list of characters
    *a ” or ‘ either side of a key will result in anything between the quotations marks being interpreted as the KEY, e.g. “key[1]” will give a KEY of key[1]
    {whitespace}

can be 0 or 1 spaces (Ascii 0x20 or ” “) SEPARATOR*

  • is “=” or “:”
  • can be preceded by and/or followed by a space
  • in order to appear in a key or value, the separator must be between a pair of quotes or escaped
    VALUE

  • consists of any alphanumeric plus “_” “.” “$” “@”

  • a ” or ‘ either side of a value will result in anything between the quotations marks being interpreted as the VALUE, e.g. key1=”value description” will give a KEY of key1 and a VALUE of “value description”
    • Placing quotes around the VALUE is required for parsing URLs or other items that contain slashes (/)
      *a SEPARATOR in a value will be treated such that this value (and the possible key preceding it) will not be treated as a key value pair, e.g. so timestamps not broken up

UNIT

  • This is the alphabetical text or “%” which follows the last number in a Numeric only VALUE up to the next space or first non alphabetical character or up to 2 characters (whichever occurs first) – those 2 characters will not be considered as part of value for querying, i.e. 100ms is treated as 100.

CHARACTER ESCAPING

  • this is used to allow a KEY or VALUE to contain a character that would not otherwise be allowed, e.g. a space or quote or comma or bracket is not allowed in a KEY or VALUE unless escaped.
  • this is supported in both a KEY and a VALUE by adding a backslash (“\”) in front of the character to be escaped or by including the entire KEY or VALUE in quotes (pairs of ” or pairs of ‘). As per the definition of KEY and VALUE, the \ is not an allowed character and only appears to support escaping, e.g a space can be included by either using quotes as in name=”David Tracey” or escaping as in name=David\ Tracey. Regarding quotes being in pairs of ” or pairs of ‘, this means that a ‘ does not finish an escaped sequence started with ” and it is possible for one type of quote to be appear within quotes of the other type, e.g. somekey = “David’s One” will include the ‘ in the value.
  • note that the \ is only treated as escaping a character in the process of determining a name or a value

Notes

  • “Key=value” “Key :value” “Key = value” “Key: value” are valid key value pairs
  • “-“, “+” not allowed in KEY or VALUE (unless escaped) to avoid confusion with use in query language
  • this definition means a Key Value Pair must have both a key and a value the inclusion of “.” and “_” means that “cpu.time” and “disk_size” are both valid as a KEY or VALUE
  • UTF-8 Encoding is the supported encoding

JSON & KVP Parsing