Installing and configuring the DataHub is an easy straight forward process that should take you no more than 20 total minutes. The DataHub is designed to be run on a server instance inside your cloud or in an on-premise environment. Setup requires downloading the DataHub, editing the configuration file with your credentials, generating keys for your S3 bucket encryption, and finally configuring your clients to send data to your Logentries account via the DataHub.
The DataHub currently supports both Linux and Windows. Please click the logo below of the OS you wish to deploy. You’ll be taken to a separate page detailing instructions for the version you selected. This page will remain open underneath the new page to allow you to come back.
Step 2: Configuration
Login to your Logentries account and access the DataHub tab and click the Settings button. There are two main options on the Settings page for DataHub, S3 Settings and Local Storage Settings.
The DataHub has the ability to send directly to S3 from your on-premise or cloud environment installed DataHub. If this is something that you desire, the only requirements are an S3 Bucket and an IAM user. The four items that will be necessary to configure S3 storage are as follows:
S3 Bucket Name: This is the name of the bucket created in your AWS environment where you want to store log data.
S3 Account ID: The Account ID of the IAM user created for S3 access.
S3 Secret Key: The Secret Key of the IAM user created for S3 access.
S3 Encryption Key: This location varies based on whether the Windows or Linux version of the DataHub was installed. For Linux, it will be /etc/dathub/public_key.txt. For Windows, it will be “C:\Program Files(x86)\Logentries\Datahub\public.key.txt”. Please make sure to type this in manually and not copy and paste from this website. The website formatting may cause issues with the forms on the site.
For a quick tutorial on how to setup an S3 Bucket and IAM user, click here.
Local Storage settings allow DataHub customers to maintain a local copy of log data. This can be a NFS mount point on Linux, a mapped network drive on Windows, or local disk. Keep in mind that the path provided here must be absolute – meaning it must start with a / in Linux or a drive letter in Windows. UNC paths are not supported.
It’s important that when configuring the routing rules that you select either S3 or Local storage if you want functionality turned on.
After running the Datahub, you should now be able to go into the Logentries UI and select the Datahub Tab. By default, there will be a rule called “Default” which will route all log data collected by the Datahub to the log Datahub/Default.
To create rules, click the Add Connection button. This will present you with the following screen:
From here, you can create routing rules. The fields are as follows:
Connection Name: Can be any generic string notating the description of the rule.
Syslog Hostname: If desired, you can break out routes to specific logs based on hostname. This is where that hostname is supplied (e.g., mail server)
Syslog Tag: If desired, you can break out routes to specific logs based on process. This is where that process is supplied (e.g., sshd)
Matching Pattern (Required): This field tells the Datahub which log entries to place into the log file defined. To forward all logs from a specific host, for instance, “.*” would be used.
Data Destination: This where the route will load log entries.
If S3 Archiving or Local Storage is required, please check the appropriate box on the Routing Table:
Both Logentries agents as well as syslog clients can communicate through the Datahub. For the agent, you need to supply the IP/hostname of the Datahub and the port, for syslog clients, you need to specify the IP/host within the syslog.conf file:
- Obtain the IP Address of the DataHub server:
- Add the following line to the /etc/rsyslog.conf of each client:
- Restart rsyslog daemon
sudo service rsyslog restart
- Test integration
logger -t test from logentries
The Logentries Windows Agent can forward it’s logs to DataHub instead of the Logentries API server. To do this open the Windows Agent from the Windows Tray. From here you can select the Agent Mode tab and select the Syslog Mode radio button. Now fill out the fields required to send to DataHub by entering in the IP of machine that DataHub is hosted on and the port it is listening to, by default DataHub will be configured to listen on port 10000. Windows-Agent-DataHub-Integration