InsightOps - Configure the Insight Agent to Send Logs

The Insight Agent sends asset log data to the Insight platform using a special configuration file called logging.json. By default, the logging.json file instructs the agent to send logs directly to the data.logs.insight.rapid7.com endpoint according to your region, but you can configure a proxy destination if necessary.

Prerequisites

To use log forwarding through the logging.json file, you must have access to an API key that allows you to create logs and logsets.

Ensure you have the correct permissions:

  1. Create a new user account.
  2. Assign the user to only InsightOps.
  3. Restrict access to Log Search operations by selecting the Log Search Insight Agent Service Role permission.
  4. Generate an API key.

Do not use an organization key or an administrator’s API key

If you do so, the selected API key inherits that user's permissions and will provide access to other APIs such as user management.

Proxy Configuration

If you prefer to send asset log data through a proxy instead of directly to the Insight platform, you can do so by adding the following additional key-value pairs in the overall config object of your logging.json file:

  • proxy-type - Specifies the application protocol. At this time, only HTTP is supported.
  • proxy-url - Specifies the address of your proxy server. This can be a hostname or IP address.
  • proxy-port - Specifies the port that the Insight Agent will use to communicate with your proxy server.

You can specify a Collector as your proxy

If you’ve already deployed one or more Collectors in your environment, you can target one of them to use as a proxy for this use case. The Insight Agent’s collection tasks related to InsightOps do not distinguish between a Collector or another proxy type. Specifying either achieves the same goal of reaching the internet.

Object configuration in logging.json for a Collector address remains the same, except that proxy-port must be set to 8037 in accordance with Collector requirements.

This example shows how you should format your proxy definition in logging.json:

json
1
{
2
  "config": {
3
    "proxy-type": "HTTP",
4
    "proxy-url": "10.7.1.219",
5
    "proxy-port": "3128",
6
    ...followed by the rest of your objects
7
   }
8
}

logging.json Creation Wizard

Use the Insight Agent data source wizard in InsightOps to automatically create and format a logging.json file that includes your selected API key and data region. You can download the complete file directly from the wizard afterwards.

You must create proxy objects manually

The logging.json creation wizard in InsightOps does not currently include the proxy-related objects described previously. If you want the agent to send logs through a proxy, you must edit your logging.json file after downloading it and add the proxy objects manually.

To create your logging.json file with the wizard:

  1. In InsightOps, click the Add Data button.
    • Alternatively, you can click the Add Data link located on either the Log Search or Data Collection tabs.
  2. In the "Install the Insight Agent" section, click the operating system of the asset that you intend to install the agent on.

TIP

The wizard will assume that you have not yet installed the Insight Agent on your assets, so feel free to skip the installation step if you have done so already.

  1. On the "Send Logs via Insight Agent" panel, choose to use your selected API key.
  2. Paste your selected API key in the field and click Apply. Mac Agent Logging
  3. Verify that your formatted logging.json file includes your selected API key, your data region endpoint, and your region code. Note that the hostnames in the 'destination' fields can be changed as per requiremnents. Click Download to retrieve your file.
  4. Place your logging.json file in one of the following directories on your asset according to its operating system:
    • Windows - C:\Program Files\Rapid7\Insight Agent\components\insight_agent\common\config
    • Mac and Linux - /opt/rapid7/ir_agent/components/insight_agent/common/config

Protect the logging.json file

Ensure your selected API key is hidden from those who should not have access to the Insight Platform:

  • Windows - Verify that only the LocalSystem has read and write access to the file.
  • Mac and Linux - Restrict file system permissions to read access by using chmod 0600.

  1. Finally, restart the Insight Agent service.
    • On Windows, this service is called "Rapid7 Insight Agent" in your service manager.
    • On Mac and Linux, you can restart the service by running the sudo service ir_agent restart command in a terminal.

logging.json Formatting Templates

If you prefer to create your logging.json file manually, see the following templates.

Mac and Linux

The Mac and Linux version of logging.json has the following structure:

logging.json
1
{
2
  "config": {
3
		"name": "A label for your log following configuration",
4
		"endpoint": "eu.data.logs.insight.rapid7.com",
5
		"region": "eu",
6
		"api-key": "Your API key",
7
		"state-file": "/opt/rapid7/ir_agent/components/insight_agent/common/config/logs.state",
8
		"formatter" : "plain",
9
		"metrics": {
10
			"system-stat-token": "your log token here",
11
			"system-stat-enabled": true,
12
			"metrics-interval": "60s",
13
			"metrics-cpu": "system",
14
			"metrics-vcpu": "core",
15
			"metrics-mem": "system",
16
			"metrics-swap": "system",
17
			"metrics-net": "sum eth0",
18
			"metrics-disk": "sum sda4 sda5",
19
			"metrics-space": "/"
20
		},
21
    "logs": [
22
        {
23
          "name": "System log",
24
          "destination": "your-server-name/syslog",
25
          "path": "/var/log/syslog",
26
          "enabled": true
27
        },
28
        {
29
          "name": "Kernel",
30
          "destination": "your-server-name/kernel",
31
          "path": "/var/log/kern.log",
32
          "enabled": true
33
        },
34
        {
35
          "name": "Authentication",
36
          "destination": "your-server-name/auth",
37
          "path": "/var/log/auth.log",
38
          "enabled": true
39
        },
40
        {
41
          "name": "Insight Agent log",
42
          "token": "your log token here",
43
          "path": "/opt/rapid7/ir_agent/components/insight_agent/common/agent.log",
44
          "enabled": false
45
        }]
46
  }
47
}

Windows

NOTE - Event log collection from domain controllers is not supported

The InsightOps component of the Insight Agent does not currently support the collection of event logs from assets acting as domain controllers. If you need to deploy a collection method for this use case, consider implementing one of the following alternatives:

The Windows version of logging.json has the following structure:

logging.json
1
{
2
  "config": {
3
    "name": "A label for your log following configuration",
4
    "endpoint": "eu.data.logs.insight.rapid7.com",
5
    "region": "eu",
6
    "api-key": "Your API key",
7
    "state-file": "C:\\Program Files\\Rapid7\\Insight Agent\\components\\insight_agent\\common\\state.file",
8
    "formatter" : "plain",
9
    "metrics": {
10
      "system-stat-token": "your log token here",
11
      "system-stat-enabled": true,
12
      "metrics-interval": "60s",
13
      "metrics-cpu": "system",
14
      "metrics-vcpu": "core",
15
      "metrics-mem": "system",
16
      "metrics-swap": "system",
17
      "metrics-net": "sum eth0",
18
      "metrics-disk": "sum sda4 sda5",
19
      "metrics-space": "/"
20
    },
21
    "windows-eventlog": {
22
      "enabled": true,
23
      "token": "cfe50cc8-7651-42f3-ac23-866bf87a6aad"
24
      },
25
     "logs": [
26
     {
27
       "name": "Custom log",
28
       "destination": "your-server-name/custom log",
29
       "path": "c:\\logs\\mylogfile*.log",
30
       "enabled": true
31
     },
32
     {
33
       "name": "Insight Agent log",
34
       "token": "your log token here",
35
       "path": "C:\\Program Files\\Rapid7\\Insight Agent\\components\\insight_agent\\common\\agent.log",
36
       "enabled": false
37
     }]
38
  }
39
}

Object legend

Object

Description

name

The label for the configuration for your own reference.

endpoint

The log data endpoint that the agent should forward log data to. For example:
AP - ap.data.logs.insight.rapid7.com,
AU - au.data.logs.insight.rapid7.com,
CA - ca.data.logs.insight.rapid7.com,
EU - eu.data.logs.insight.rapid7.com,
US-1 - us.data.logs.insight.rapid7.com,
US-2 - us2.data.logs.insight.rapid7.com,
US-3 - us3.data.logs.insight.rapid7.com

region

The specific region where your deployment of InsightOps is based. For example:
ap, au, ca, eu, us, us2, us3

api-key

The API key that's generated and fetched from Settings > API Key management.

Do not use an organization key or an administrator’s API key. To use log forwarding through the logging.json file, create a new user account and generate an API key.

state-file

This is the path to a local file that tracks the last entry that was sent to InsightOps. When the agent restarts, it will send logs from that point to minimize missing log data while the agent is offline.

formatter

Setting formatter to "plain" will send the entries to InsightOps exactly as they were written to in the local log file.

Remove this field if you want the log lines to be sent in Syslog format RFC 5424 - adding a timestamp, host and other information.

metrics

This section allows you to send the host's resource utilization metrics to InsightOps.

system-stat-token

Enter a log token in this field to define which log will receive metrics data in InsightOps. You can generate a token using Add Data > Quick Add in the application.

windows-eventlog

This section defines if you want to automatically collect entries from the Windows Event log. These entries will be converted to JSON for easy search analysis.

This does not apply to Domain Controllers.

token

Enter a log token in this field to define which log will receive Windows events in InsightOps. You can generate a token using Add Data > Quick Add in the application.

logs

This section is a list of log files on the host that you want to follow. As new lines are written to these logs, updates will be sent to InsightOps in real time.

name

The label for the configuration for your own reference.

destination

When used, this field will automatically create a Log set and a log if a matching one does not exist.

Replace this field with a token field if you want to write an existing log in InsightOps or aggregate log data in one location in InsightOps.

path

This is the path to the file you wish to follow on the host running the agent. For files on Windows machines, you will need to escape the backslash. For example:
"path": "c:\\logs\\mylogfile.log".

InsightOps supports log file rotation for certain log rollover policies that do not allow you to specify an absolute destination file name. This is typical when log files are timestamped or assigned a sequential number. You can specify a wildcard in the filename so the contents of any active log file matching the pattern in the specified directory will be forwarded to the same log in InsightOps. Be aware that the Insight Agent is not capable of watching a directory itself. Rather, it's designed to watch logs that rotate on dates and sequential numbers, or a syslog rotation.

For example:

"path": "c:\\logs\\mylogfile-*.log"


When specifying a wildcard rule, make sure that your wildcard is anchored to a specific log filename base. Do not specify unanchored wildcards, such as "path": "c:\\logs\\*.log", because this will pull any and all log files being written to your specified directory and result in an inordinate amount of data transmission.


Note that wildcards are not necessary in typical syslog rotation schemes. The typical scheme will perform this rotation by renaming the original logname.log to logname.log.0 before creating a new logname.log to replace it. In this situation, you only need to specify logname.log by itself without any wildcard character.

enabled

This is a boolean value of true or false and is used to specify if the file should be followed or not.

proxy-type

Specifies the application protocol. At this time, only HTTP is supported.

proxy-url

Specifies the address of your proxy server. This can be a hostname or IP address.

proxy-port

Specifies the port that the Insight Agent will use to communicate with your proxy server. If you're specifying a Collector in your proxy definition, this must be set to 8037.