Graylog2 Collector

The new approach

Server side controlled configuration

Staring with Graylog 2 the a new collector approach (Graylog Collector Sidecar) is proposed, based on wrapping NXLog.

Installation is straightforward, as indicated in online documents (sample for Windows, a later post might share Ubuntu experience).

Install NXLog:

• Download NXLog Windows installation kit NXLog downloads
• Run the installer
• Run (as an administrator) "C:\Program Files (x86)\nxlog\nxlog.exe" -u; this will un-install nxlog as a service but will keep binaries

Now download and install latest collector-sidecar binaries (Collector-Sidecar releases)

Before installing/starting the collector as a service:

• On the server administration site go to System->Collectors->Manage Configurations
• Add a configuration with a meaningful name
  • Add one or more relevant tags
  • Add an Output (by choosing from already configured Inputs)
  • Add one or more inputs; for file type inputs you can check the multi-line option and provide a RegEx for the start pattern, like /^\d{2}\.\d{2}\.\d{4}/ (notice enclosing slashes!)

Go back to the monitored computer and edit C:\Program Files (x86)\graylog\collector-sidecar\collector-sidecar.yaml:

• update server_url
• update to a meaningful name the node_id
• add one of the tags previously defined in the tags list
• install and start the service (run as administrator):
  • "C:\Program Files (x86)\graylog\collector-sidecar\graylog-collector-sidecar.exe" -service install
  • "C:\Program Files (x86)\graylog\collector-sidecar\graylog-collector-sidecar.exe" -service start

*Service can be started/stopped also from Task manager (Services tab, collector-sidecar Name) or from Services Manager (Graylog collector sidecar Display name).

Now we should have automatically created, based on the previously defined configuration, the nxlog configuration file C:\Program Files (x86)\graylog\collector-sidecar\generated\nxlog.conf and in administration site the collector should be listed in System->Collectors section.

Please notice that if there is some error in the yaml configuration file the service will not start and no error message will be logged (it happened to me if I commented the tags list, see next section)

Locally controlled configuration

Under certain conditions you might need to use advanced NXLog features which are not supported by the server side configuration. You can do the following trick:

• Start with the approach described in the previous section and check that everything goes fine; at this moment we should have a valid NXLog configuration file
• Move the configuration file in another directory and change the configuration_path value to that file
• Remove all valid tags and add a dummy one that does not exists in server side configuration; do not remove the tags entry or leave the list empty since the collector will not start and no error message will be logged
• Restart the collector-sidecar service
• If everything works as expected, read carefully the NXLog manual and do your own enhancements to the configuration
• Restart the collector-sidecar service

I applied the approach because I needed an aditional message field (Exec $source_id = 'WildFly10';) source_id for one of the inputs, like shown below.

<Input 576d1d3d659fb107f722db38>
    Module im_file
    File "C:\Java\wildfly-10.0.0.Final\standalone\log\server.log"
    PollInterval 1
    SavePos True
    ReadFromLast True
    Recursive False
    RenameCheck False
    Exec $FileName = file_name(); # Send file name with each message
    Exec $source_id = 'WildFly10';
    InputType 576d1d3d659fb107f722db38-multiline
</Input>

The full yaml configuration file content:

server_url: http://graylog.mycompany.local:12900
tls_skip_verify: false
node_id: graylog-collector-sidecar-devsrv
collector_id: file:C:\Program Files (x86)\graylog\collector-sidecar\collector-id
tags:
    - undefined
log_path: C:\Program Files (x86)\graylog\collector-sidecar
update_interval: 10
backends:
    - name: nxlog
      enabled: true
      binary_path: C:\Program Files (x86)\nxlog\nxlog.exe
#      configuration_path: C:\Program Files (x86)\graylog\collector-sidecar\generated\nxlog.conf
      configuration_path: C:\Program Files (x86)\graylog\collector-sidecar\conf\nxlog.conf

As an alternate approach (I haven't tested but it should work) is to go for the first option (server side controlled configuration) but instead of defining Outputs/Inputs just place all nxlog.conf content in the default nxlog snippet, in the Windows section. Be sure you have in sync the collector side client_id with the one in the nxlog.conf content, Output section (see Exec $gl2_source_collector = '0324bd79-b9a9-458e-9977-d0eecbd2d347';):

<Output 576cf426659fb107f722b011>
    Module om_tcp
    Host graylog.mycompany.local
    Port 12201
    OutputType  GELF_TCP
    Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
    Exec $gl2_source_collector = '0324bd79-b9a9-458e-9977-d0eecbd2d347';
    Exec $Hostname = hostname_fqdn();
</Output>

Duplicate configuration on another server

Once we have a collector configured we can duplicate easily the installation on another one:

• Perform the NXLog installation steps
• Install the collector-sidecar (just run the setup)
• Copy yaml and nxlog.conf files on the target server
• Change the yaml to reflect the right node_id
• Change the content of the collector-id with some unique string (a GUID?)
• Change the nxlog.conf Output tag to reflect the collector-id value

IyBUaGUgbmV3IGFwcHJvYWNoDQoNCiMjIFNlcnZlciBzaWRlIGNvbnRyb2xsZWQgY29uZmlndXJhdGlvbg0KDQpTdGFyaW5nIHdpdGggR3JheWxvZyAyIHRoZSBhIG5ldyBjb2xsZWN0b3IgYXBwcm9hY2ggKGBHcmF5bG9nIENvbGxlY3RvciBTaWRlY2FyYCkgaXMgcHJvcG9zZWQsIGJhc2VkIG9uIHdyYXBwaW5nIGBOWExvZ2AuDQoNCkluc3RhbGxhdGlvbiBpcyBzdHJhaWdodGZvcndhcmQsIGFzIGluZGljYXRlZCBpbiBvbmxpbmUgZG9jdW1lbnRzIChzYW1wbGUgZm9yIFdpbmRvd3MsIGEgbGF0ZXIgcG9zdCBtaWdodCBzaGFyZSBVYnVudHUgZXhwZXJpZW5jZSkuDQoNCkluc3RhbGwgTlhMb2c6DQoNCi0gRG93bmxvYWQgTlhMb2cgV2luZG93cyBpbnN0YWxsYXRpb24ga2l0IFtOWExvZyBkb3dubG9hZHNdKGh0dHBzOi8vbnhsb2cub3JnL3Byb2R1Y3RzL254bG9nLWNvbW11bml0eS1lZGl0aW9uL2Rvd25sb2FkICJOWExvZyBkb3dubG9hZHMiKQ0KLSBSdW4gdGhlIGluc3RhbGxlcg0KLSBSdW4gKGFzIGFuIGFkbWluaXN0cmF0b3IpIGAiQzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxueGxvZ1xueGxvZy5leGUiIC11YDsgdGhpcyB3aWxsIHVuLWluc3RhbGwgYG54bG9nYCBhcyBhIHNlcnZpY2UgYnV0IHdpbGwga2VlcCBiaW5hcmllcw0KDQpOb3cgZG93bmxvYWQgYW5kIGluc3RhbGwgbGF0ZXN0IGNvbGxlY3Rvci1zaWRlY2FyIGJpbmFyaWVzIChbQ29sbGVjdG9yLVNpZGVjYXIgcmVsZWFzZXNdKGh0dHBzOi8vZ2l0aHViLmNvbS9HcmF5bG9nMi9jb2xsZWN0b3Itc2lkZWNhci9yZWxlYXNlcykpDQoNCkJlZm9yZSBpbnN0YWxsaW5nL3N0YXJ0aW5nIHRoZSBjb2xsZWN0b3IgYXMgYSBzZXJ2aWNlOg0KDQotIE9uIHRoZSBzZXJ2ZXIgYWRtaW5pc3RyYXRpb24gc2l0ZSBnbyB0byBTeXN0ZW0tPkNvbGxlY3RvcnMtPk1hbmFnZSBDb25maWd1cmF0aW9ucw0KLSBBZGQgYSBjb25maWd1cmF0aW9uIHdpdGggYSBtZWFuaW5nZnVsIG5hbWUNCgktIEFkZCBvbmUgb3IgbW9yZSByZWxldmFudCB0YWdzDQoJLSBBZGQgYW4gT3V0cHV0IChieSBjaG9vc2luZyBmcm9tIGFscmVhZHkgY29uZmlndXJlZCBJbnB1dHMpDQoJLSBBZGQgb25lIG9yIG1vcmUgaW5wdXRzOyBmb3IgZmlsZSB0eXBlIGlucHV0cyB5b3UgY2FuIGNoZWNrIHRoZSBtdWx0aS1saW5lIG9wdGlvbiBhbmQgcHJvdmlkZSBhIFJlZ0V4IGZvciB0aGUgc3RhcnQgcGF0dGVybiwgbGlrZSBgL15cZHsyfVwuXGR7Mn1cLlxkezR9L2AgKG5vdGljZSBlbmNsb3Npbmcgc2xhc2hlcyEpDQoNCkdvIGJhY2sgdG8gdGhlIG1vbml0b3JlZCBjb21wdXRlciBhbmQgZWRpdCBgQzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxncmF5bG9nXGNvbGxlY3Rvci1zaWRlY2FyXGNvbGxlY3Rvci1zaWRlY2FyLnlhbWxgOg0KDQotIHVwZGF0ZSBzZXJ2ZXJfdXJsDQotIHVwZGF0ZSB0byBhIG1lYW5pbmdmdWwgbmFtZSB0aGUgbm9kZV9pZA0KLSBhZGQgb25lIG9mIHRoZSB0YWdzIHByZXZpb3VzbHkgZGVmaW5lZCBpbiB0aGUgdGFncyBsaXN0DQotIGluc3RhbGwgYW5kIHN0YXJ0IHRoZSBzZXJ2aWNlIChydW4gYXMgYWRtaW5pc3RyYXRvcik6DQoJLSBgIkM6XFByb2dyYW0gRmlsZXMgKHg4NilcZ3JheWxvZ1xjb2xsZWN0b3Itc2lkZWNhclxncmF5bG9nLWNvbGxlY3Rvci1zaWRlY2FyLmV4ZSIgLXNlcnZpY2UgaW5zdGFsbGANCgktIGAiQzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxncmF5bG9nXGNvbGxlY3Rvci1zaWRlY2FyXGdyYXlsb2ctY29sbGVjdG9yLXNpZGVjYXIuZXhlIiAtc2VydmljZSBzdGFydGANCg0KKlNlcnZpY2UgY2FuIGJlIHN0YXJ0ZWQvc3RvcHBlZCBhbHNvIGZyb20gVGFzayBtYW5hZ2VyIChTZXJ2aWNlcyB0YWIsIGBjb2xsZWN0b3Itc2lkZWNhcmAgTmFtZSkgb3IgZnJvbSBTZXJ2aWNlcyBNYW5hZ2VyIChgR3JheWxvZyBjb2xsZWN0b3Igc2lkZWNhcmAgRGlzcGxheSBuYW1lKS4NCg0KTm93IHdlIHNob3VsZCBoYXZlIGF1dG9tYXRpY2FsbHkgY3JlYXRlZCwgYmFzZWQgb24gdGhlIHByZXZpb3VzbHkgZGVmaW5lZCBjb25maWd1cmF0aW9uLCB0aGUgbnhsb2cgY29uZmlndXJhdGlvbiBmaWxlIGBDOlxQcm9ncmFtIEZpbGVzICh4ODYpXGdyYXlsb2dcY29sbGVjdG9yLXNpZGVjYXJcZ2VuZXJhdGVkXG54bG9nLmNvbmZgIGFuZCBpbiBhZG1pbmlzdHJhdGlvbiBzaXRlIHRoZSBjb2xsZWN0b3Igc2hvdWxkIGJlIGxpc3RlZCBpbiBTeXN0ZW0tPkNvbGxlY3RvcnMgc2VjdGlvbi4NCg0KUGxlYXNlIG5vdGljZSB0aGF0IGlmIHRoZXJlIGlzIHNvbWUgZXJyb3IgaW4gdGhlIHlhbWwgY29uZmlndXJhdGlvbiBmaWxlIHRoZSBzZXJ2aWNlIHdpbGwgbm90IHN0YXJ0IGFuZCBubyBlcnJvciBtZXNzYWdlIHdpbGwgYmUgbG9nZ2VkIChpdCBoYXBwZW5lZCB0byBtZSBpZiBJIGNvbW1lbnRlZCB0aGUgdGFncyBsaXN0LCBzZWUgbmV4dCBzZWN0aW9uKQ0KDQojIyBMb2NhbGx5IGNvbnRyb2xsZWQgY29uZmlndXJhdGlvbg0KDQpVbmRlciBjZXJ0YWluIGNvbmRpdGlvbnMgeW91IG1pZ2h0IG5lZWQgdG8gdXNlIGFkdmFuY2VkIE5YTG9nIGZlYXR1cmVzIHdoaWNoIGFyZSBub3Qgc3VwcG9ydGVkIGJ5IHRoZSBzZXJ2ZXIgc2lkZSBjb25maWd1cmF0aW9uLiBZb3UgY2FuIGRvIHRoZSBmb2xsb3dpbmcgdHJpY2s6DQotIFN0YXJ0IHdpdGggdGhlIGFwcHJvYWNoIGRlc2NyaWJlZCBpbiB0aGUgcHJldmlvdXMgc2VjdGlvbiBhbmQgY2hlY2sgdGhhdCBldmVyeXRoaW5nIGdvZXMgZmluZTsgYXQgdGhpcyBtb21lbnQgd2Ugc2hvdWxkIGhhdmUgYSB2YWxpZCBOWExvZyBjb25maWd1cmF0aW9uIGZpbGUNCi0gTW92ZSB0aGUgY29uZmlndXJhdGlvbiBmaWxlIGluIGFub3RoZXIgZGlyZWN0b3J5IGFuZCBjaGFuZ2UgdGhlIGNvbmZpZ3VyYXRpb25fcGF0aCB2YWx1ZSB0byB0aGF0IGZpbGUNCi0gUmVtb3ZlIGFsbCB2YWxpZCB0YWdzIGFuZCBhZGQgYSBkdW1teSBvbmUgdGhhdCBkb2VzIG5vdCBleGlzdHMgaW4gc2VydmVyIHNpZGUgY29uZmlndXJhdGlvbjsgZG8gbm90IHJlbW92ZSB0aGUgdGFncyBlbnRyeSBvciBsZWF2ZSB0aGUgbGlzdCBlbXB0eSBzaW5jZSB0aGUgY29sbGVjdG9yIHdpbGwgbm90IHN0YXJ0IGFuZCBubyBlcnJvciBtZXNzYWdlIHdpbGwgYmUgbG9nZ2VkDQotIFJlc3RhcnQgdGhlIGNvbGxlY3Rvci1zaWRlY2FyIHNlcnZpY2UNCi0gSWYgZXZlcnl0aGluZyB3b3JrcyBhcyBleHBlY3RlZCwgcmVhZCBjYXJlZnVsbHkgdGhlIE5YTG9nIG1hbnVhbCBhbmQgZG8geW91ciBvd24gZW5oYW5jZW1lbnRzIHRvIHRoZSBjb25maWd1cmF0aW9uDQotIFJlc3RhcnQgdGhlIGNvbGxlY3Rvci1zaWRlY2FyIHNlcnZpY2UNCg0KSSBhcHBsaWVkIHRoZSBhcHByb2FjaCBiZWNhdXNlIEkgbmVlZGVkIGFuIGFkaXRpb25hbCBtZXNzYWdlIGZpZWxkIChgRXhlYyAkc291cmNlX2lkID0gJ1dpbGRGbHkxMCc7YCkgc291cmNlX2lkIGZvciBvbmUgb2YgdGhlIGlucHV0cywgbGlrZSBzaG93biBiZWxvdy4NCmBgYA0KPElucHV0IDU3NmQxZDNkNjU5ZmIxMDdmNzIyZGIzOD4NCglNb2R1bGUgaW1fZmlsZQ0KCUZpbGUgIkM6XEphdmFcd2lsZGZseS0xMC4wLjAuRmluYWxcc3RhbmRhbG9uZVxsb2dcc2VydmVyLmxvZyINCglQb2xsSW50ZXJ2YWwgMQ0KCVNhdmVQb3MJVHJ1ZQ0KCVJlYWRGcm9tTGFzdCBUcnVlDQoJUmVjdXJzaXZlIEZhbHNlDQoJUmVuYW1lQ2hlY2sgRmFsc2UNCglFeGVjICRGaWxlTmFtZSA9IGZpbGVfbmFtZSgpOyAjIFNlbmQgZmlsZSBuYW1lIHdpdGggZWFjaCBtZXNzYWdlDQoJRXhlYyAkc291cmNlX2lkID0gJ1dpbGRGbHkxMCc7DQoJSW5wdXRUeXBlIDU3NmQxZDNkNjU5ZmIxMDdmNzIyZGIzOC1tdWx0aWxpbmUNCjwvSW5wdXQ+DQoNCmBgYA0KDQpUaGUgZnVsbCB5YW1sIGNvbmZpZ3VyYXRpb24gZmlsZSBjb250ZW50Og0KYGBgDQpzZXJ2ZXJfdXJsOiBodHRwOi8vZ3JheWxvZy5teWNvbXBhbnkubG9jYWw6MTI5MDANCnRsc19za2lwX3ZlcmlmeTogZmFsc2UNCm5vZGVfaWQ6IGdyYXlsb2ctY29sbGVjdG9yLXNpZGVjYXItZGV2c3J2DQpjb2xsZWN0b3JfaWQ6IGZpbGU6QzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxncmF5bG9nXGNvbGxlY3Rvci1zaWRlY2FyXGNvbGxlY3Rvci1pZA0KdGFnczoNCiAgICAtIHVuZGVmaW5lZA0KbG9nX3BhdGg6IEM6XFByb2dyYW0gRmlsZXMgKHg4NilcZ3JheWxvZ1xjb2xsZWN0b3Itc2lkZWNhcg0KdXBkYXRlX2ludGVydmFsOiAxMA0KYmFja2VuZHM6DQogICAgLSBuYW1lOiBueGxvZw0KICAgICAgZW5hYmxlZDogdHJ1ZQ0KICAgICAgYmluYXJ5X3BhdGg6IEM6XFByb2dyYW0gRmlsZXMgKHg4Nilcbnhsb2dcbnhsb2cuZXhlDQojICAgICAgY29uZmlndXJhdGlvbl9wYXRoOiBDOlxQcm9ncmFtIEZpbGVzICh4ODYpXGdyYXlsb2dcY29sbGVjdG9yLXNpZGVjYXJcZ2VuZXJhdGVkXG54bG9nLmNvbmYNCiAgICAgIGNvbmZpZ3VyYXRpb25fcGF0aDogQzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxncmF5bG9nXGNvbGxlY3Rvci1zaWRlY2FyXGNvbmZcbnhsb2cuY29uZg0KYGBgDQoNCkFzIGFuIGFsdGVybmF0ZSBhcHByb2FjaCAoSSBoYXZlbid0IHRlc3RlZCBidXQgaXQgc2hvdWxkIHdvcmspIGlzIHRvIGdvIGZvciB0aGUgZmlyc3Qgb3B0aW9uIChzZXJ2ZXIgc2lkZSBjb250cm9sbGVkIGNvbmZpZ3VyYXRpb24pIGJ1dCBpbnN0ZWFkIG9mIGRlZmluaW5nIE91dHB1dHMvSW5wdXRzIGp1c3QgcGxhY2UgYWxsIG54bG9nLmNvbmYgY29udGVudCBpbiB0aGUgZGVmYXVsdCBueGxvZyBzbmlwcGV0LCBpbiB0aGUgV2luZG93cyBzZWN0aW9uLiBCZSBzdXJlIHlvdSBoYXZlIGluIHN5bmMgdGhlIGNvbGxlY3RvciBzaWRlIGNsaWVudF9pZCB3aXRoIHRoZSBvbmUgaW4gdGhlIG54bG9nLmNvbmYgY29udGVudCwgT3V0cHV0IHNlY3Rpb24gKHNlZSBgRXhlYyAkZ2wyX3NvdXJjZV9jb2xsZWN0b3IgPSAnMDMyNGJkNzktYjlhOS00NThlLTk5NzctZDBlZWNiZDJkMzQ3JztgKToNCg0KYGBgDQo8T3V0cHV0IDU3NmNmNDI2NjU5ZmIxMDdmNzIyYjAxMT4NCglNb2R1bGUgb21fdGNwDQoJSG9zdCBncmF5bG9nLm15Y29tcGFueS5sb2NhbA0KCVBvcnQgMTIyMDENCglPdXRwdXRUeXBlICBHRUxGX1RDUA0KCUV4ZWMgJHNob3J0X21lc3NhZ2UgPSAkcmF3X2V2ZW50OyAjIEF2b2lkcyB0cnVuY2F0aW9uIG9mIHRoZSBzaG9ydF9tZXNzYWdlIGZpZWxkLg0KCUV4ZWMgJGdsMl9zb3VyY2VfY29sbGVjdG9yID0gJzAzMjRiZDc5LWI5YTktNDU4ZS05OTc3LWQwZWVjYmQyZDM0Nyc7DQoJRXhlYyAkSG9zdG5hbWUgPSBob3N0bmFtZV9mcWRuKCk7DQo8L091dHB1dD4NCmBgYA0KDQojIyMgRHVwbGljYXRlIGNvbmZpZ3VyYXRpb24gb24gYW5vdGhlciBzZXJ2ZXINCg0KT25jZSB3ZSBoYXZlIGEgY29sbGVjdG9yIGNvbmZpZ3VyZWQgd2UgY2FuIGR1cGxpY2F0ZSBlYXNpbHkgdGhlIGluc3RhbGxhdGlvbiBvbiBhbm90aGVyIG9uZToNCg0KDQotIFBlcmZvcm0gdGhlIE5YTG9nIGluc3RhbGxhdGlvbiBzdGVwcw0KLSBJbnN0YWxsIHRoZSBjb2xsZWN0b3Itc2lkZWNhciAoanVzdCBydW4gdGhlIHNldHVwKQ0KLSBDb3B5IHlhbWwgYW5kIG54bG9nLmNvbmYgZmlsZXMgb24gdGhlIHRhcmdldCBzZXJ2ZXINCi0gQ2hhbmdlIHRoZSB5YW1sIHRvIHJlZmxlY3QgdGhlIHJpZ2h0IG5vZGVfaWQNCi0gQ2hhbmdlIHRoZSBjb250ZW50IG9mIHRoZSBjb2xsZWN0b3ItaWQgd2l0aCBzb21lIHVuaXF1ZSBzdHJpbmcgKGEgR1VJRD8pDQotIENoYW5nZSB0aGUgbnhsb2cuY29uZiBPdXRwdXQgdGFnIHRvIHJlZmxlY3QgdGhlIGNvbGxlY3Rvci1pZCB2YWx1ZSAg