The new approach
Server side controlled configuration
Staring with Graylog 2 the a new collector approach (Graylog Collector Sidecar) is proposed, based on wrapping NXLog.
Installation is straightforward, as indicated in online documents (sample for Windows, a later post might share Ubuntu experience).
Install NXLog:
- Download NXLog Windows installation kit NXLog downloads
- Run the installer
- Run (as an administrator)
"C:\Program Files (x86)\nxlog\nxlog.exe" -u; this will un-install nxlog as a service but will keep binaries
Now download and install latest collector-sidecar binaries (Collector-Sidecar releases)
Before installing/starting the collector as a service:
- On the server administration site go to System->Collectors->Manage Configurations
- Add a configuration with a meaningful name
- Add one or more relevant tags
- Add an Output (by choosing from already configured Inputs)
- Add one or more inputs; for file type inputs you can check the multi-line option and provide a RegEx for the start pattern, like
/^\d{2}\.\d{2}\.\d{4}/ (notice enclosing slashes!)
Go back to the monitored computer and edit C:\Program Files (x86)\graylog\collector-sidecar\collector-sidecar.yaml:
- update server_url
- update to a meaningful name the node_id
- add one of the tags previously defined in the tags list
- install and start the service (run as administrator):
"C:\Program Files (x86)\graylog\collector-sidecar\graylog-collector-sidecar.exe" -service install"C:\Program Files (x86)\graylog\collector-sidecar\graylog-collector-sidecar.exe" -service start
*Service can be started/stopped also from Task manager (Services tab, collector-sidecar Name) or from Services Manager (Graylog collector sidecar Display name).
Now we should have automatically created, based on the previously defined configuration, the nxlog configuration file C:\Program Files (x86)\graylog\collector-sidecar\generated\nxlog.conf and in administration site the collector should be listed in System->Collectors section.
Please notice that if there is some error in the yaml configuration file the service will not start and no error message will be logged (it happened to me if I commented the tags list, see next section)
Locally controlled configuration
Under certain conditions you might need to use advanced NXLog features which are not supported by the server side configuration. You can do the following trick:
- Start with the approach described in the previous section and check that everything goes fine; at this moment we should have a valid NXLog configuration file
- Move the configuration file in another directory and change the configuration_path value to that file
- Remove all valid tags and add a dummy one that does not exists in server side configuration; do not remove the tags entry or leave the list empty since the collector will not start and no error message will be logged
- Restart the collector-sidecar service
- If everything works as expected, read carefully the NXLog manual and do your own enhancements to the configuration
- Restart the collector-sidecar service
I applied the approach because I needed an aditional message field (Exec $source_id = 'WildFly10';) source_id for one of the inputs, like shown below.
<Input 576d1d3d659fb107f722db38>
Module im_file
File "C:\Java\wildfly-10.0.0.Final\standalone\log\server.log"
PollInterval 1
SavePos True
ReadFromLast True
Recursive False
RenameCheck False
Exec $FileName = file_name(); # Send file name with each message
Exec $source_id = 'WildFly10';
InputType 576d1d3d659fb107f722db38-multiline
</Input>
The full yaml configuration file content:
server_url: http://graylog.mycompany.local:12900
tls_skip_verify: false
node_id: graylog-collector-sidecar-devsrv
collector_id: file:C:\Program Files (x86)\graylog\collector-sidecar\collector-id
tags:
- undefined
log_path: C:\Program Files (x86)\graylog\collector-sidecar
update_interval: 10
backends:
- name: nxlog
enabled: true
binary_path: C:\Program Files (x86)\nxlog\nxlog.exe
# configuration_path: C:\Program Files (x86)\graylog\collector-sidecar\generated\nxlog.conf
configuration_path: C:\Program Files (x86)\graylog\collector-sidecar\conf\nxlog.conf
As an alternate approach (I haven't tested but it should work) is to go for the first option (server side controlled configuration) but instead of defining Outputs/Inputs just place all nxlog.conf content in the default nxlog snippet, in the Windows section. Be sure you have in sync the collector side client_id with the one in the nxlog.conf content, Output section (see Exec $gl2_source_collector = '0324bd79-b9a9-458e-9977-d0eecbd2d347';):
<Output 576cf426659fb107f722b011>
Module om_tcp
Host graylog.mycompany.local
Port 12201
OutputType GELF_TCP
Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
Exec $gl2_source_collector = '0324bd79-b9a9-458e-9977-d0eecbd2d347';
Exec $Hostname = hostname_fqdn();
</Output>
Duplicate configuration on another server
Once we have a collector configured we can duplicate easily the installation on another one:
- Perform the NXLog installation steps
- Install the collector-sidecar (just run the setup)
- Copy yaml and nxlog.conf files on the target server
- Change the yaml to reflect the right node_id
- Change the content of the collector-id with some unique string (a GUID?)
- Change the nxlog.conf Output tag to reflect the collector-id value
IyBUaGUgbmV3IGFwcHJvYWNoDQoNCiMjIFNlcnZlciBzaWRlIGNvbnRyb2xsZWQgY29uZmlndXJhdGlvbg0KDQpTdGFyaW5nIHdpdGggR3JheWxvZyAyIHRoZSBhIG5ldyBjb2xsZWN0b3IgYXBwcm9hY2ggKGBHcmF5bG9nIENvbGxlY3RvciBTaWRlY2FyYCkgaXMgcHJvcG9zZWQsIGJhc2VkIG9uIHdyYXBwaW5nIGBOWExvZ2AuDQoNCkluc3RhbGxhdGlvbiBpcyBzdHJhaWdodGZvcndhcmQsIGFzIGluZGljYXRlZCBpbiBvbmxpbmUgZG9jdW1lbnRzIChzYW1wbGUgZm9yIFdpbmRvd3MsIGEgbGF0ZXIgcG9zdCBtaWdodCBzaGFyZSBVYnVudHUgZXhwZXJpZW5jZSkuDQoNCkluc3RhbGwgTlhMb2c6DQoNCi0gRG93bmxvYWQgTlhMb2cgV2luZG93cyBpbnN0YWxsYXRpb24ga2l0IFtOWExvZyBkb3dubG9hZHNdKGh0dHBzOi8vbnhsb2cub3JnL3Byb2R1Y3RzL254bG9nLWNvbW11bml0eS1lZGl0aW9uL2Rvd25sb2FkICJOWExvZyBkb3dubG9hZHMiKQ0KLSBSdW4gdGhlIGluc3RhbGxlcg0KLSBSdW4gKGFzIGFuIGFkbWluaXN0cmF0b3IpIGAiQzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxueGxvZ1xueGxvZy5leGUiIC11YDsgdGhpcyB3aWxsIHVuLWluc3RhbGwgYG54bG9nYCBhcyBhIHNlcnZpY2UgYnV0IHdpbGwga2VlcCBiaW5hcmllcw0KDQpOb3cgZG93bmxvYWQgYW5kIGluc3RhbGwgbGF0ZXN0IGNvbGxlY3Rvci1zaWRlY2FyIGJpbmFyaWVzIChbQ29sbGVjdG9yLVNpZGVjYXIgcmVsZWFzZXNdKGh0dHBzOi8vZ2l0aHViLmNvbS9HcmF5bG9nMi9jb2xsZWN0b3Itc2lkZWNhci9yZWxlYXNlcykpDQoNCkJlZm9yZSBpbnN0YWxsaW5nL3N0YXJ0aW5nIHRoZSBjb2xsZWN0b3IgYXMgYSBzZXJ2aWNlOg0KDQotIE9uIHRoZSBzZXJ2ZXIgYWRtaW5pc3RyYXRpb24gc2l0ZSBnbyB0byBTeXN0ZW0tPkNvbGxlY3RvcnMtPk1hbmFnZSBDb25maWd1cmF0aW9ucw0KLSBBZGQgYSBjb25maWd1cmF0aW9uIHdpdGggYSBtZWFuaW5nZnVsIG5hbWUNCgktIEFkZCBvbmUgb3IgbW9yZSByZWxldmFudCB0YWdzDQoJLSBBZGQgYW4gT3V0cHV0IChieSBjaG9vc2luZyBmcm9tIGFscmVhZHkgY29uZmlndXJlZCBJbnB1dHMpDQoJLSBBZGQgb25lIG9yIG1vcmUgaW5wdXRzOyBmb3IgZmlsZSB0eXBlIGlucHV0cyB5b3UgY2FuIGNoZWNrIHRoZSBtdWx0aS1saW5lIG9wdGlvbiBhbmQgcHJvdmlkZSBhIFJlZ0V4IGZvciB0aGUgc3RhcnQgcGF0dGVybiwgbGlrZSBgL15cZHsyfVwuXGR7Mn1cLlxkezR9L2AgKG5vdGljZSBlbmNsb3Npbmcgc2xhc2hlcyEpDQoNCkdvIGJhY2sgdG8gdGhlIG1vbml0b3JlZCBjb21wdXRlciBhbmQgZWRpdCBgQzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxncmF5bG9nXGNvbGxlY3Rvci1zaWRlY2FyXGNvbGxlY3Rvci1zaWRlY2FyLnlhbWxgOg0KDQotIHVwZGF0ZSBzZXJ2ZXJfdXJsDQotIHVwZGF0ZSB0byBhIG1lYW5pbmdmdWwgbmFtZSB0aGUgbm9kZV9pZA0KLSBhZGQgb25lIG9mIHRoZSB0YWdzIHByZXZpb3VzbHkgZGVmaW5lZCBpbiB0aGUgdGFncyBsaXN0DQotIGluc3RhbGwgYW5kIHN0YXJ0IHRoZSBzZXJ2aWNlIChydW4gYXMgYWRtaW5pc3RyYXRvcik6DQoJLSBgIkM6XFByb2dyYW0gRmlsZXMgKHg4NilcZ3JheWxvZ1xjb2xsZWN0b3Itc2lkZWNhclxncmF5bG9nLWNvbGxlY3Rvci1zaWRlY2FyLmV4ZSIgLXNlcnZpY2UgaW5zdGFsbGANCgktIGAiQzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxncmF5bG9nXGNvbGxlY3Rvci1zaWRlY2FyXGdyYXlsb2ctY29sbGVjdG9yLXNpZGVjYXIuZXhlIiAtc2VydmljZSBzdGFydGANCg0KKlNlcnZpY2UgY2FuIGJlIHN0YXJ0ZWQvc3RvcHBlZCBhbHNvIGZyb20gVGFzayBtYW5hZ2VyIChTZXJ2aWNlcyB0YWIsIGBjb2xsZWN0b3Itc2lkZWNhcmAgTmFtZSkgb3IgZnJvbSBTZXJ2aWNlcyBNYW5hZ2VyIChgR3JheWxvZyBjb2xsZWN0b3Igc2lkZWNhcmAgRGlzcGxheSBuYW1lKS4NCg0KTm93IHdlIHNob3VsZCBoYXZlIGF1dG9tYXRpY2FsbHkgY3JlYXRlZCwgYmFzZWQgb24gdGhlIHByZXZpb3VzbHkgZGVmaW5lZCBjb25maWd1cmF0aW9uLCB0aGUgbnhsb2cgY29uZmlndXJhdGlvbiBmaWxlIGBDOlxQcm9ncmFtIEZpbGVzICh4ODYpXGdyYXlsb2dcY29sbGVjdG9yLXNpZGVjYXJcZ2VuZXJhdGVkXG54bG9nLmNvbmZgIGFuZCBpbiBhZG1pbmlzdHJhdGlvbiBzaXRlIHRoZSBjb2xsZWN0b3Igc2hvdWxkIGJlIGxpc3RlZCBpbiBTeXN0ZW0tPkNvbGxlY3RvcnMgc2VjdGlvbi4NCg0KUGxlYXNlIG5vdGljZSB0aGF0IGlmIHRoZXJlIGlzIHNvbWUgZXJyb3IgaW4gdGhlIHlhbWwgY29uZmlndXJhdGlvbiBmaWxlIHRoZSBzZXJ2aWNlIHdpbGwgbm90IHN0YXJ0IGFuZCBubyBlcnJvciBtZXNzYWdlIHdpbGwgYmUgbG9nZ2VkIChpdCBoYXBwZW5lZCB0byBtZSBpZiBJIGNvbW1lbnRlZCB0aGUgdGFncyBsaXN0LCBzZWUgbmV4dCBzZWN0aW9uKQ0KDQojIyBMb2NhbGx5IGNvbnRyb2xsZWQgY29uZmlndXJhdGlvbg0KDQpVbmRlciBjZXJ0YWluIGNvbmRpdGlvbnMgeW91IG1pZ2h0IG5lZWQgdG8gdXNlIGFkdmFuY2VkIE5YTG9nIGZlYXR1cmVzIHdoaWNoIGFyZSBub3Qgc3VwcG9ydGVkIGJ5IHRoZSBzZXJ2ZXIgc2lkZSBjb25maWd1cmF0aW9uLiBZb3UgY2FuIGRvIHRoZSBmb2xsb3dpbmcgdHJpY2s6DQotIFN0YXJ0IHdpdGggdGhlIGFwcHJvYWNoIGRlc2NyaWJlZCBpbiB0aGUgcHJldmlvdXMgc2VjdGlvbiBhbmQgY2hlY2sgdGhhdCBldmVyeXRoaW5nIGdvZXMgZmluZTsgYXQgdGhpcyBtb21lbnQgd2Ugc2hvdWxkIGhhdmUgYSB2YWxpZCBOWExvZyBjb25maWd1cmF0aW9uIGZpbGUNCi0gTW92ZSB0aGUgY29uZmlndXJhdGlvbiBmaWxlIGluIGFub3RoZXIgZGlyZWN0b3J5IGFuZCBjaGFuZ2UgdGhlIGNvbmZpZ3VyYXRpb25fcGF0aCB2YWx1ZSB0byB0aGF0IGZpbGUNCi0gUmVtb3ZlIGFsbCB2YWxpZCB0YWdzIGFuZCBhZGQgYSBkdW1teSBvbmUgdGhhdCBkb2VzIG5vdCBleGlzdHMgaW4gc2VydmVyIHNpZGUgY29uZmlndXJhdGlvbjsgZG8gbm90IHJlbW92ZSB0aGUgdGFncyBlbnRyeSBvciBsZWF2ZSB0aGUgbGlzdCBlbXB0eSBzaW5jZSB0aGUgY29sbGVjdG9yIHdpbGwgbm90IHN0YXJ0IGFuZCBubyBlcnJvciBtZXNzYWdlIHdpbGwgYmUgbG9nZ2VkDQotIFJlc3RhcnQgdGhlIGNvbGxlY3Rvci1zaWRlY2FyIHNlcnZpY2UNCi0gSWYgZXZlcnl0aGluZyB3b3JrcyBhcyBleHBlY3RlZCwgcmVhZCBjYXJlZnVsbHkgdGhlIE5YTG9nIG1hbnVhbCBhbmQgZG8geW91ciBvd24gZW5oYW5jZW1lbnRzIHRvIHRoZSBjb25maWd1cmF0aW9uDQotIFJlc3RhcnQgdGhlIGNvbGxlY3Rvci1zaWRlY2FyIHNlcnZpY2UNCg0KSSBhcHBsaWVkIHRoZSBhcHByb2FjaCBiZWNhdXNlIEkgbmVlZGVkIGFuIGFkaXRpb25hbCBtZXNzYWdlIGZpZWxkIChgRXhlYyAkc291cmNlX2lkID0gJ1dpbGRGbHkxMCc7YCkgc291cmNlX2lkIGZvciBvbmUgb2YgdGhlIGlucHV0cywgbGlrZSBzaG93biBiZWxvdy4NCmBgYA0KPElucHV0IDU3NmQxZDNkNjU5ZmIxMDdmNzIyZGIzOD4NCglNb2R1bGUgaW1fZmlsZQ0KCUZpbGUgIkM6XEphdmFcd2lsZGZseS0xMC4wLjAuRmluYWxcc3RhbmRhbG9uZVxsb2dcc2VydmVyLmxvZyINCglQb2xsSW50ZXJ2YWwgMQ0KCVNhdmVQb3MJVHJ1ZQ0KCVJlYWRGcm9tTGFzdCBUcnVlDQoJUmVjdXJzaXZlIEZhbHNlDQoJUmVuYW1lQ2hlY2sgRmFsc2UNCglFeGVjICRGaWxlTmFtZSA9IGZpbGVfbmFtZSgpOyAjIFNlbmQgZmlsZSBuYW1lIHdpdGggZWFjaCBtZXNzYWdlDQoJRXhlYyAkc291cmNlX2lkID0gJ1dpbGRGbHkxMCc7DQoJSW5wdXRUeXBlIDU3NmQxZDNkNjU5ZmIxMDdmNzIyZGIzOC1tdWx0aWxpbmUNCjwvSW5wdXQ+DQoNCmBgYA0KDQpUaGUgZnVsbCB5YW1sIGNvbmZpZ3VyYXRpb24gZmlsZSBjb250ZW50Og0KYGBgDQpzZXJ2ZXJfdXJsOiBodHRwOi8vZ3JheWxvZy5teWNvbXBhbnkubG9jYWw6MTI5MDANCnRsc19za2lwX3ZlcmlmeTogZmFsc2UNCm5vZGVfaWQ6IGdyYXlsb2ctY29sbGVjdG9yLXNpZGVjYXItZGV2c3J2DQpjb2xsZWN0b3JfaWQ6IGZpbGU6QzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxncmF5bG9nXGNvbGxlY3Rvci1zaWRlY2FyXGNvbGxlY3Rvci1pZA0KdGFnczoNCiAgICAtIHVuZGVmaW5lZA0KbG9nX3BhdGg6IEM6XFByb2dyYW0gRmlsZXMgKHg4NilcZ3JheWxvZ1xjb2xsZWN0b3Itc2lkZWNhcg0KdXBkYXRlX2ludGVydmFsOiAxMA0KYmFja2VuZHM6DQogICAgLSBuYW1lOiBueGxvZw0KICAgICAgZW5hYmxlZDogdHJ1ZQ0KICAgICAgYmluYXJ5X3BhdGg6IEM6XFByb2dyYW0gRmlsZXMgKHg4Nilcbnhsb2dcbnhsb2cuZXhlDQojICAgICAgY29uZmlndXJhdGlvbl9wYXRoOiBDOlxQcm9ncmFtIEZpbGVzICh4ODYpXGdyYXlsb2dcY29sbGVjdG9yLXNpZGVjYXJcZ2VuZXJhdGVkXG54bG9nLmNvbmYNCiAgICAgIGNvbmZpZ3VyYXRpb25fcGF0aDogQzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxncmF5bG9nXGNvbGxlY3Rvci1zaWRlY2FyXGNvbmZcbnhsb2cuY29uZg0KYGBgDQoNCkFzIGFuIGFsdGVybmF0ZSBhcHByb2FjaCAoSSBoYXZlbid0IHRlc3RlZCBidXQgaXQgc2hvdWxkIHdvcmspIGlzIHRvIGdvIGZvciB0aGUgZmlyc3Qgb3B0aW9uIChzZXJ2ZXIgc2lkZSBjb250cm9sbGVkIGNvbmZpZ3VyYXRpb24pIGJ1dCBpbnN0ZWFkIG9mIGRlZmluaW5nIE91dHB1dHMvSW5wdXRzIGp1c3QgcGxhY2UgYWxsIG54bG9nLmNvbmYgY29udGVudCBpbiB0aGUgZGVmYXVsdCBueGxvZyBzbmlwcGV0LCBpbiB0aGUgV2luZG93cyBzZWN0aW9uLiBCZSBzdXJlIHlvdSBoYXZlIGluIHN5bmMgdGhlIGNvbGxlY3RvciBzaWRlIGNsaWVudF9pZCB3aXRoIHRoZSBvbmUgaW4gdGhlIG54bG9nLmNvbmYgY29udGVudCwgT3V0cHV0IHNlY3Rpb24gKHNlZSBgRXhlYyAkZ2wyX3NvdXJjZV9jb2xsZWN0b3IgPSAnMDMyNGJkNzktYjlhOS00NThlLTk5NzctZDBlZWNiZDJkMzQ3JztgKToNCg0KYGBgDQo8T3V0cHV0IDU3NmNmNDI2NjU5ZmIxMDdmNzIyYjAxMT4NCglNb2R1bGUgb21fdGNwDQoJSG9zdCBncmF5bG9nLm15Y29tcGFueS5sb2NhbA0KCVBvcnQgMTIyMDENCglPdXRwdXRUeXBlICBHRUxGX1RDUA0KCUV4ZWMgJHNob3J0X21lc3NhZ2UgPSAkcmF3X2V2ZW50OyAjIEF2b2lkcyB0cnVuY2F0aW9uIG9mIHRoZSBzaG9ydF9tZXNzYWdlIGZpZWxkLg0KCUV4ZWMgJGdsMl9zb3VyY2VfY29sbGVjdG9yID0gJzAzMjRiZDc5LWI5YTktNDU4ZS05OTc3LWQwZWVjYmQyZDM0Nyc7DQoJRXhlYyAkSG9zdG5hbWUgPSBob3N0bmFtZV9mcWRuKCk7DQo8L091dHB1dD4NCmBgYA0KDQojIyMgRHVwbGljYXRlIGNvbmZpZ3VyYXRpb24gb24gYW5vdGhlciBzZXJ2ZXINCg0KT25jZSB3ZSBoYXZlIGEgY29sbGVjdG9yIGNvbmZpZ3VyZWQgd2UgY2FuIGR1cGxpY2F0ZSBlYXNpbHkgdGhlIGluc3RhbGxhdGlvbiBvbiBhbm90aGVyIG9uZToNCg0KDQotIFBlcmZvcm0gdGhlIE5YTG9nIGluc3RhbGxhdGlvbiBzdGVwcw0KLSBJbnN0YWxsIHRoZSBjb2xsZWN0b3Itc2lkZWNhciAoanVzdCBydW4gdGhlIHNldHVwKQ0KLSBDb3B5IHlhbWwgYW5kIG54bG9nLmNvbmYgZmlsZXMgb24gdGhlIHRhcmdldCBzZXJ2ZXINCi0gQ2hhbmdlIHRoZSB5YW1sIHRvIHJlZmxlY3QgdGhlIHJpZ2h0IG5vZGVfaWQNCi0gQ2hhbmdlIHRoZSBjb250ZW50IG9mIHRoZSBjb2xsZWN0b3ItaWQgd2l0aCBzb21lIHVuaXF1ZSBzdHJpbmcgKGEgR1VJRD8pDQotIENoYW5nZSB0aGUgbnhsb2cuY29uZiBPdXRwdXQgdGFnIHRvIHJlZmxlY3QgdGhlIGNvbGxlY3Rvci1pZCB2YWx1ZSAg
