Install PostgeSQL on Ubuntu 16.04

Based on https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-16-04

Install PosgreSQL:

sudo apt-get update
sudo apt-get install postgresql postgresql-contrib

• Find out where configuration files are stored:

sudo -u postgres psql # use \q to exit

show config_file; # somethis like /etc/postgresql/9.5/main...
\q

• Cchange in posgresql.config

  • #listen_addresses = 'localhost' # what IP address(es) to listen on;
  • to listen_addresses = '*' # what IP address(es) to listen on;

• Add in pg_hba.conf

  • host all all 10.0.0.0/8 md5 Change through psql the password for database login postgres:
  • \password postgres

• Restart postgresql:

  • exit (to logout from postgres user session)
  • systemctl restart postgreql

IyBJbnN0YWxsIFBvc3RnZVNRTCBvbiBVYnVudHUgMTYuMDQNCg0KQmFzZWQgb24gW2h0dHBzOi8vd3d3LmRpZ2l0YWxvY2Vhbi5jb20vY29tbXVuaXR5L3R1dG9yaWFscy9ob3ctdG8taW5zdGFsbC1hbmQtdXNlLXBvc3RncmVzcWwtb24tdWJ1bnR1LTE2LTA0XShodHRwczovL3d3dy5kaWdpdGFsb2NlYW4uY29tL2NvbW11bml0eS90dXRvcmlhbHMvaG93LXRvLWluc3RhbGwtYW5kLXVzZS1wb3N0Z3Jlc3FsLW9uLXVidW50dS0xNi0wNCAiSW5zdGFsbCIpDQoNCkluc3RhbGwgUG9zZ3JlU1FMOg0KYGBgDQpzdWRvIGFwdC1nZXQgdXBkYXRlDQpzdWRvIGFwdC1nZXQgaW5zdGFsbCBwb3N0Z3Jlc3FsIHBvc3RncmVzcWwtY29udHJpYg0KYGBgDQoNCi0gRmluZCBvdXQgd2hlcmUgY29uZmlndXJhdGlvbiBmaWxlcyBhcmUgc3RvcmVkOg0KDQpzdWRvIC11IHBvc3RncmVzIHBzcWwgIyB1c2UgXHEgdG8gZXhpdA0KDQpgYGANCnNob3cgY29uZmlnX2ZpbGU7ICMgc29tZXRoaXMgbGlrZSAvZXRjL3Bvc3RncmVzcWwvOS41L21haW4uLi4NClxxDQpgYGANCg0KLSBDY2hhbmdlIGluIHBvc2dyZXNxbC5jb25maWcNCgktIGAjbGlzdGVuX2FkZHJlc3NlcyA9ICdsb2NhbGhvc3QnCQkjIHdoYXQgSVAgYWRkcmVzcyhlcykgdG8gbGlzdGVuIG9uO2ANCgktIHRvIGBsaXN0ZW5fYWRkcmVzc2VzID0gJyonCQkjIHdoYXQgSVAgYWRkcmVzcyhlcykgdG8gbGlzdGVuIG9uO2ANCg0KLSBBZGQgaW4gcGdfaGJhLmNvbmYNCgktIGhvc3QgICAgYWxsICAgICAgICAgICAgIGFsbCAgICAgICAgICAgICAxMC4wLjAuMC84ICAgICAgICAgICAgICBtZDUNCkNoYW5nZSB0aHJvdWdoIGBwc3FsYCB0aGUgcGFzc3dvcmQgZm9yIGRhdGFiYXNlIGxvZ2luIHBvc3RncmVzOg0KCS0gXHBhc3N3b3JkIHBvc3RncmVzDQoNCi0gUmVzdGFydCBwb3N0Z3Jlc3FsOg0KCS0gZXhpdCAodG8gbG9nb3V0IGZyb20gcG9zdGdyZXMgdXNlciBzZXNzaW9uKQ0KCS0gc3lzdGVtY3RsIHJlc3RhcnQgcG9zdGdyZXFs

Install Wildfly 10.1 on Ubuntu Server 16.04

Install Wildfly 10.1 on Ubuntu 16.04

Be sure to use the same Java version as the one used as compilation target of generated WAR.

Run the following script:

sudo apt-get update
apt install software-properties-common
add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

cd /tmp
wget http://download.jboss.org/wildfly/10.1.0.Final/wildfly-10.1.0.Final.tar.gz
tar -xzvf wildfly-10.1.0.Final.tar.gz
mv wildfly-10.1.0.Final wildfly
cp -R  wildfly /opt/

groupadd -r wildfly
useradd -r -g wildfly -d /opt/wildfly -s /sbin/nologin wildfly
chown -R wildfly:wildfly /opt/wildfly

cd wildfly/docs/contrib/scripts/systemd

mkdir /etc/wildfly
cp wildfly.conf /etc/wildfly/
cp wildfly.service /etc/systemd/system/
cp launch.sh /opt/wildfly/bin/
chmod +x /opt/wildfly/bin/launch.sh

# do not launch wildfly through standalone.sh->it will screw up file permissions

systemctl start wildfly.service
systemctl enable wildfly.service

• Watch out for CRLF EndOfLine (should be just LFs)
• Check that Wildfly is running through systemctl status wildfly

In order to check from outside current VM/physical machine that Wildfly was started change standalone.xml content so that:

        <interface name="public">
            <any-address/>
        </interface>

There might be required some other changes to the default distribution. Besure that wildfly user is owner all added items (chown -R wildfly:wildfly /opt/wildfly/modules)

Add EclipseLink Support

• Download eclipselink.jar to modules\system\layers\base\org\eclipse\persistence\main
• Change module.xml so that:

    <resources>
        <resource-root path="jipijapa-eclipselink-10.1.0.Final.jar"/>
        <resource-root path="eclipselink.jar">
            <filter>
                <exclude path="javax/**" />
            </filter>
        </resource-root>
    </resources>

Add SQL Server support

• Create modules\system\layers\base\com\microsoft\sqlserver\jdbc\main
• Download sqljdbc4.jar
• Create module.xml:

<?xml version="1.0" encoding="UTF-8"?>
<module xmlns="urn:jboss:module:1.3" name="com.microsoft.sqlserver.jdbc">
    <resources>
        <resource-root path="sqljdbc4.jar"/>
    </resources>
    <dependencies>
        <module name="javax.api"/>
        <module name="javax.transaction.api"/>
    </dependencies>
</module>

Add PostgreSQL Support

• Create modules\system\layers\base\org\postgresql\main
• Download latest JDBC PostgreSQL driver
• Create module.xml(change driver name acordingly):

<?xml version="1.0" encoding="UTF-8"?>
<module xmlns="urn:jboss:module:1.3" name="org.postgresql">
    <resources>
        <!-- Make sure this matches the name of the JAR you are installing -->
        <resource-root path="postgresql-9.4-1202.jdbc4.jar"/>
    </resources>
    <dependencies>
        <module name="javax.api"/>
        <module name="javax.transaction.api"/>
        <module name="javax.servlet.api" optional="true"/>
    </dependencies>
</module>

Restart Wildfly...

Velocity Note

If some of the used components in generated WAR use Velocity you will encounter the following error in server.log (during hot deployment or Wildfly restart): Error configuring AvalonLogSystem : java.io.FileNotFoundException: /velocity.log (Permission denied). This is due to the fact that velocity.properties (as found in velocity jar, path org\apache\velocity\runtime\defaults\ states:

runtime.log = velocity.log

The quick solution is to create an empty velocity.log file in the root of your file system with full access (?) for wildfly user.

IyBJbnN0YWxsIFdpbGRmbHkgMTAuMSBvbiBVYnVudHUgMTYuMDQNCg0KKipCZSBzdXJlIHRvIHVzZSB0aGUgc2FtZSBKYXZhIHZlcnNpb24gYXMgdGhlIG9uZSB1c2VkIGFzIGNvbXBpbGF0aW9uIHRhcmdldCBvZiBnZW5lcmF0ZWQgV0FSLioqDQoNClJ1biB0aGUgZm9sbG93aW5nIHNjcmlwdDoNCmBgYA0Kc3VkbyBhcHQtZ2V0IHVwZGF0ZQ0KYXB0IGluc3RhbGwgc29mdHdhcmUtcHJvcGVydGllcy1jb21tb24NCmFkZC1hcHQtcmVwb3NpdG9yeSBwcGE6d2VidXBkOHRlYW0vamF2YQ0Kc3VkbyBhcHQtZ2V0IHVwZGF0ZQ0Kc3VkbyBhcHQtZ2V0IGluc3RhbGwgb3JhY2xlLWphdmE4LWluc3RhbGxlcg0KDQpjZCAvdG1wDQp3Z2V0IGh0dHA6Ly9kb3dubG9hZC5qYm9zcy5vcmcvd2lsZGZseS8xMC4xLjAuRmluYWwvd2lsZGZseS0xMC4xLjAuRmluYWwudGFyLmd6DQp0YXIgLXh6dmYgd2lsZGZseS0xMC4xLjAuRmluYWwudGFyLmd6DQptdiB3aWxkZmx5LTEwLjEuMC5GaW5hbCB3aWxkZmx5DQpjcCAtUiAgd2lsZGZseSAvb3B0Lw0KDQpncm91cGFkZCAtciB3aWxkZmx5DQp1c2VyYWRkIC1yIC1nIHdpbGRmbHkgLWQgL29wdC93aWxkZmx5IC1zIC9zYmluL25vbG9naW4gd2lsZGZseQ0KY2hvd24gLVIgd2lsZGZseTp3aWxkZmx5IC9vcHQvd2lsZGZseQ0KDQpjZCB3aWxkZmx5L2RvY3MvY29udHJpYi9zY3JpcHRzL3N5c3RlbWQNCg0KbWtkaXIgL2V0Yy93aWxkZmx5DQpjcCB3aWxkZmx5LmNvbmYgL2V0Yy93aWxkZmx5Lw0KY3Agd2lsZGZseS5zZXJ2aWNlIC9ldGMvc3lzdGVtZC9zeXN0ZW0vDQpjcCBsYXVuY2guc2ggL29wdC93aWxkZmx5L2Jpbi8NCmNobW9kICt4IC9vcHQvd2lsZGZseS9iaW4vbGF1bmNoLnNoDQoNCiMgZG8gbm90IGxhdW5jaCB3aWxkZmx5IHRocm91Z2ggc3RhbmRhbG9uZS5zaC0+aXQgd2lsbCBzY3JldyB1cCBmaWxlIHBlcm1pc3Npb25zDQoNCnN5c3RlbWN0bCBzdGFydCB3aWxkZmx5LnNlcnZpY2UNCnN5c3RlbWN0bCBlbmFibGUgd2lsZGZseS5zZXJ2aWNlDQpgYGANCg0KLSBXYXRjaCBvdXQgZm9yIENSTEYgRW5kT2ZMaW5lIChzaG91bGQgYmUganVzdCBMRnMpDQotIENoZWNrIHRoYXQgV2lsZGZseSBpcyBydW5uaW5nIHRocm91Z2ggYHN5c3RlbWN0bCBzdGF0dXMgd2lsZGZseWANCg0KSW4gb3JkZXIgdG8gY2hlY2sgZnJvbSBvdXRzaWRlIGN1cnJlbnQgVk0vcGh5c2ljYWwgbWFjaGluZSB0aGF0IFdpbGRmbHkgd2FzIHN0YXJ0ZWQgY2hhbmdlIHN0YW5kYWxvbmUueG1sIGNvbnRlbnQgc28gdGhhdDoNCmBgYHhtbA0KICAgICAgICA8aW50ZXJmYWNlIG5hbWU9InB1YmxpYyI+DQogICAgICAgICAgICA8YW55LWFkZHJlc3MvPg0KICAgICAgICA8L2ludGVyZmFjZT4NCmBgYA0KVGhlcmUgbWlnaHQgYmUgcmVxdWlyZWQgc29tZSBvdGhlciBjaGFuZ2VzIHRvIHRoZSBkZWZhdWx0IGRpc3RyaWJ1dGlvbi4gQmVzdXJlIHRoYXQgd2lsZGZseSB1c2VyIGlzIG93bmVyIGFsbCBhZGRlZCBpdGVtcyAoYGNob3duIC1SIHdpbGRmbHk6d2lsZGZseSAvb3B0L3dpbGRmbHkvbW9kdWxlc2ApDQoNCiMjIEFkZCBFY2xpcHNlTGluayBTdXBwb3J0DQoNCi0gRG93bmxvYWQgZWNsaXBzZWxpbmsuamFyIHRvIGBtb2R1bGVzXHN5c3RlbVxsYXllcnNcYmFzZVxvcmdcZWNsaXBzZVxwZXJzaXN0ZW5jZVxtYWluYA0KLSBDaGFuZ2UgYG1vZHVsZS54bWxgIHNvIHRoYXQ6DQpgYGB4bWwNCiAgICA8cmVzb3VyY2VzPg0KICAgICAgICA8cmVzb3VyY2Utcm9vdCBwYXRoPSJqaXBpamFwYS1lY2xpcHNlbGluay0xMC4xLjAuRmluYWwuamFyIi8+DQoJCTxyZXNvdXJjZS1yb290IHBhdGg9ImVjbGlwc2VsaW5rLmphciI+DQoJCQk8ZmlsdGVyPg0KCQkJCTxleGNsdWRlIHBhdGg9ImphdmF4LyoqIiAvPg0KCQkJPC9maWx0ZXI+DQoJCTwvcmVzb3VyY2Utcm9vdD4NCiAgICA8L3Jlc291cmNlcz4NCmBgYA0KDQojIyBBZGQgU1FMIFNlcnZlciBzdXBwb3J0DQoNCi0gQ3JlYXRlIGBtb2R1bGVzXHN5c3RlbVxsYXllcnNcYmFzZVxjb21cbWljcm9zb2Z0XHNxbHNlcnZlclxqZGJjXG1haW5gDQotIERvd25sb2FkIGBzcWxqZGJjNC5qYXJgDQotIENyZWF0ZSBgbW9kdWxlLnhtbGA6DQpgYGB4bWwNCjw/eG1sIHZlcnNpb249IjEuMCIgZW5jb2Rpbmc9IlVURi04Ij8+DQo8bW9kdWxlIHhtbG5zPSJ1cm46amJvc3M6bW9kdWxlOjEuMyIgbmFtZT0iY29tLm1pY3Jvc29mdC5zcWxzZXJ2ZXIuamRiYyI+DQogICAgPHJlc291cmNlcz4NCiAgICAgICAgPHJlc291cmNlLXJvb3QgcGF0aD0ic3FsamRiYzQuamFyIi8+DQogICAgPC9yZXNvdXJjZXM+DQogICAgPGRlcGVuZGVuY2llcz4NCiAgICAgICAgPG1vZHVsZSBuYW1lPSJqYXZheC5hcGkiLz4NCiAgICAgICAgPG1vZHVsZSBuYW1lPSJqYXZheC50cmFuc2FjdGlvbi5hcGkiLz4NCiAgICA8L2RlcGVuZGVuY2llcz4NCjwvbW9kdWxlPg0KYGBgIA0KDQojIyBBZGQgUG9zdGdyZVNRTCBTdXBwb3J0DQoNCg0KLSBDcmVhdGUgYG1vZHVsZXNcc3lzdGVtXGxheWVyc1xiYXNlXG9yZ1xwb3N0Z3Jlc3FsXG1haW5gDQotIERvd25sb2FkIGxhdGVzdCBKREJDIFBvc3RncmVTUUwgZHJpdmVyDQotIENyZWF0ZSBgbW9kdWxlLnhtbGAoY2hhbmdlIGRyaXZlciBuYW1lIGFjb3JkaW5nbHkpOg0KYGBgeG1sDQo8P3htbCB2ZXJzaW9uPSIxLjAiIGVuY29kaW5nPSJVVEYtOCI/Pg0KPG1vZHVsZSB4bWxucz0idXJuOmpib3NzOm1vZHVsZToxLjMiIG5hbWU9Im9yZy5wb3N0Z3Jlc3FsIj4NCiAgICA8cmVzb3VyY2VzPg0KICAgICAgICA8IS0tIE1ha2Ugc3VyZSB0aGlzIG1hdGNoZXMgdGhlIG5hbWUgb2YgdGhlIEpBUiB5b3UgYXJlIGluc3RhbGxpbmcgLS0+DQogICAgICAgIDxyZXNvdXJjZS1yb290IHBhdGg9InBvc3RncmVzcWwtOS40LTEyMDIuamRiYzQuamFyIi8+DQogICAgPC9yZXNvdXJjZXM+DQogICAgPGRlcGVuZGVuY2llcz4NCiAgICAgICAgPG1vZHVsZSBuYW1lPSJqYXZheC5hcGkiLz4NCiAgICAgICAgPG1vZHVsZSBuYW1lPSJqYXZheC50cmFuc2FjdGlvbi5hcGkiLz4NCiAgICAgICAgPG1vZHVsZSBuYW1lPSJqYXZheC5zZXJ2bGV0LmFwaSIgb3B0aW9uYWw9InRydWUiLz4NCiAgICA8L2RlcGVuZGVuY2llcz4NCjwvbW9kdWxlPg0KYGBgDQoNCiMjIFJlc3RhcnQgV2lsZGZseS4uLg0KDQojIyBWZWxvY2l0eSBOb3RlDQoNCklmIHNvbWUgb2YgdGhlIHVzZWQgY29tcG9uZW50cyBpbiBnZW5lcmF0ZWQgV0FSIHVzZSBWZWxvY2l0eSB5b3Ugd2lsbCBlbmNvdW50ZXIgdGhlIGZvbGxvd2luZyBlcnJvciBpbiBzZXJ2ZXIubG9nIChkdXJpbmcgaG90IGRlcGxveW1lbnQgb3IgV2lsZGZseSByZXN0YXJ0KTogYEVycm9yIGNvbmZpZ3VyaW5nIEF2YWxvbkxvZ1N5c3RlbSA6IGphdmEuaW8uRmlsZU5vdEZvdW5kRXhjZXB0aW9uOiAvdmVsb2NpdHkubG9nIChQZXJtaXNzaW9uIGRlbmllZClgLiBUaGlzIGlzIGR1ZSB0byB0aGUgZmFjdCB0aGF0IHZlbG9jaXR5LnByb3BlcnRpZXMgKGFzIGZvdW5kIGluIHZlbG9jaXR5IGphciwgcGF0aCBgb3JnXGFwYWNoZVx2ZWxvY2l0eVxydW50aW1lXGRlZmF1bHRzXGAgc3RhdGVzOg0KYGBgDQpydW50aW1lLmxvZyA9IHZlbG9jaXR5LmxvZw0KYGBgDQpUaGUgcXVpY2sgc29sdXRpb24gaXMgdG8gY3JlYXRlIGFuIGVtcHR5IGB2ZWxvY2l0eS5sb2dgIGZpbGUgaW4gdGhlIHJvb3Qgb2YgeW91ciBmaWxlIHN5c3RlbSB3aXRoIGZ1bGwgYWNjZXNzICg/KSBmb3Igd2lsZGZseSB1c2VyLg0KIA==

Graylog2 & MongoDB

Graylog uses an instance of MongoDB as a configuration repository. Browsing data can be done (for instance) by using Robomango:

• Install Robomango, free edition (Robomango)
• Configure a connection, providing Address (IP or FQDN) and Port (27017)
• Happy browsing!

R3JheWxvZyB1c2VzIGFuIGluc3RhbmNlIG9mIE1vbmdvREIgYXMgYSBjb25maWd1cmF0aW9uIHJlcG9zaXRvcnkuIEJyb3dzaW5nIGRhdGEgY2FuIGJlIGRvbmUgKGZvciBpbnN0YW5jZSkgYnkgdXNpbmcgUm9ib21hbmdvOg0KDQotIEluc3RhbGwgUm9ib21hbmdvLCBmcmVlIGVkaXRpb24gKFtSb2JvbWFuZ29dKGh0dHBzOi8vcm9ib21vbmdvLm9yZy9idXkpKQ0KLSBDb25maWd1cmUgYSBjb25uZWN0aW9uLCBwcm92aWRpbmcgQWRkcmVzcyAoSVAgb3IgRlFETikgYW5kIFBvcnQgKDI3MDE3KQ0KLSBIYXBweSBicm93c2luZyE=

Graylog2 Rules

Running server side rules that drop/enrich messages is quite simple:

• edit /opt/graylog/conf/graylog.conf by adding the following line rules_file = /opt/graylog/conf/rules.drl (there is special empty section starting with the comment # Drools Rule File (Use to rewrite incoming log messages)
• add /opt/graylog/conf/rules.drl file with the appropriate content
• restart graylog (sudo graylog-ctl start, etc.)

Sample rules file

import org.graylog2.plugin.Message
import java.util.regex.Matcher
import java.util.regex.Pattern

rule "WildFly10"
  when
      m : Message( getField("source_id") == "WildFly10" )
  then
      Matcher matcher = Pattern.compile("^\\d{4}-\\d{2}-\\d{2}\\s\\d{2}:\\d{2}:\\d{2},\\d{3}\\s(\\w+)").matcher(m.getMessage());
      if (matcher.find()) {
         m.addField("event-level", matcher.group(1));
      }
end

UnVubmluZyBzZXJ2ZXIgc2lkZSBydWxlcyB0aGF0IGRyb3AvZW5yaWNoIG1lc3NhZ2VzIGlzIHF1aXRlIHNpbXBsZToNCg0KLSBlZGl0IGAvb3B0L2dyYXlsb2cvY29uZi9ncmF5bG9nLmNvbmZgIGJ5IGFkZGluZyB0aGUgZm9sbG93aW5nIGxpbmUgYHJ1bGVzX2ZpbGUgPSAvb3B0L2dyYXlsb2cvY29uZi9ydWxlcy5kcmxgICh0aGVyZSBpcyBzcGVjaWFsIGVtcHR5IHNlY3Rpb24gc3RhcnRpbmcgd2l0aCB0aGUgY29tbWVudCBgIyBEcm9vbHMgUnVsZSBGaWxlIChVc2UgdG8gcmV3cml0ZSBpbmNvbWluZyBsb2cgbWVzc2FnZXMpYA0KLSBhZGQgYC9vcHQvZ3JheWxvZy9jb25mL3J1bGVzLmRybGAgZmlsZSB3aXRoIHRoZSBhcHByb3ByaWF0ZSBjb250ZW50DQotIHJlc3RhcnQgZ3JheWxvZyAoYHN1ZG8gZ3JheWxvZy1jdGwgc3RhcnRgLCBldGMuKQ0KDQpTYW1wbGUgcnVsZXMgZmlsZQ0KYGBgDQppbXBvcnQgb3JnLmdyYXlsb2cyLnBsdWdpbi5NZXNzYWdlDQppbXBvcnQgamF2YS51dGlsLnJlZ2V4Lk1hdGNoZXINCmltcG9ydCBqYXZhLnV0aWwucmVnZXguUGF0dGVybg0KDQpydWxlICJXaWxkRmx5MTAiDQogIHdoZW4NCiAgICAgIG0gOiBNZXNzYWdlKCBnZXRGaWVsZCgic291cmNlX2lkIikgPT0gIldpbGRGbHkxMCIgKQ0KICB0aGVuDQogICAgICBNYXRjaGVyIG1hdGNoZXIgPSBQYXR0ZXJuLmNvbXBpbGUoIl5cXGR7NH0tXFxkezJ9LVxcZHsyfVxcc1xcZHsyfTpcXGR7Mn06XFxkezJ9LFxcZHszfVxccyhcXHcrKSIpLm1hdGNoZXIobS5nZXRNZXNzYWdlKCkpOw0KICAgICAgaWYgKG1hdGNoZXIuZmluZCgpKSB7DQogICAgICAgICBtLmFkZEZpZWxkKCJldmVudC1sZXZlbCIsIG1hdGNoZXIuZ3JvdXAoMSkpOw0KICAgICAgfQ0KZW5kDQpgYGANCg==

Graylog2 Collector

The new approach

Server side controlled configuration

Staring with Graylog 2 the a new collector approach (Graylog Collector Sidecar) is proposed, based on wrapping NXLog.

Installation is straightforward, as indicated in online documents (sample for Windows, a later post might share Ubuntu experience).

Install NXLog:

• Download NXLog Windows installation kit NXLog downloads
• Run the installer
• Run (as an administrator) "C:\Program Files (x86)\nxlog\nxlog.exe" -u; this will un-install nxlog as a service but will keep binaries

Now download and install latest collector-sidecar binaries (Collector-Sidecar releases)

Before installing/starting the collector as a service:

• On the server administration site go to System->Collectors->Manage Configurations
• Add a configuration with a meaningful name
  • Add one or more relevant tags
  • Add an Output (by choosing from already configured Inputs)
  • Add one or more inputs; for file type inputs you can check the multi-line option and provide a RegEx for the start pattern, like /^\d{2}\.\d{2}\.\d{4}/ (notice enclosing slashes!)

Go back to the monitored computer and edit C:\Program Files (x86)\graylog\collector-sidecar\collector-sidecar.yaml:

• update server_url
• update to a meaningful name the node_id
• add one of the tags previously defined in the tags list
• install and start the service (run as administrator):
  • "C:\Program Files (x86)\graylog\collector-sidecar\graylog-collector-sidecar.exe" -service install
  • "C:\Program Files (x86)\graylog\collector-sidecar\graylog-collector-sidecar.exe" -service start

*Service can be started/stopped also from Task manager (Services tab, collector-sidecar Name) or from Services Manager (Graylog collector sidecar Display name).

Now we should have automatically created, based on the previously defined configuration, the nxlog configuration file C:\Program Files (x86)\graylog\collector-sidecar\generated\nxlog.conf and in administration site the collector should be listed in System->Collectors section.

Please notice that if there is some error in the yaml configuration file the service will not start and no error message will be logged (it happened to me if I commented the tags list, see next section)

Locally controlled configuration

Under certain conditions you might need to use advanced NXLog features which are not supported by the server side configuration. You can do the following trick:

• Start with the approach described in the previous section and check that everything goes fine; at this moment we should have a valid NXLog configuration file
• Move the configuration file in another directory and change the configuration_path value to that file
• Remove all valid tags and add a dummy one that does not exists in server side configuration; do not remove the tags entry or leave the list empty since the collector will not start and no error message will be logged
• Restart the collector-sidecar service
• If everything works as expected, read carefully the NXLog manual and do your own enhancements to the configuration
• Restart the collector-sidecar service

I applied the approach because I needed an aditional message field (Exec $source_id = 'WildFly10';) source_id for one of the inputs, like shown below.

<Input 576d1d3d659fb107f722db38>
    Module im_file
    File "C:\Java\wildfly-10.0.0.Final\standalone\log\server.log"
    PollInterval 1
    SavePos True
    ReadFromLast True
    Recursive False
    RenameCheck False
    Exec $FileName = file_name(); # Send file name with each message
    Exec $source_id = 'WildFly10';
    InputType 576d1d3d659fb107f722db38-multiline
</Input>

The full yaml configuration file content:

server_url: http://graylog.mycompany.local:12900
tls_skip_verify: false
node_id: graylog-collector-sidecar-devsrv
collector_id: file:C:\Program Files (x86)\graylog\collector-sidecar\collector-id
tags:
    - undefined
log_path: C:\Program Files (x86)\graylog\collector-sidecar
update_interval: 10
backends:
    - name: nxlog
      enabled: true
      binary_path: C:\Program Files (x86)\nxlog\nxlog.exe
#      configuration_path: C:\Program Files (x86)\graylog\collector-sidecar\generated\nxlog.conf
      configuration_path: C:\Program Files (x86)\graylog\collector-sidecar\conf\nxlog.conf

As an alternate approach (I haven't tested but it should work) is to go for the first option (server side controlled configuration) but instead of defining Outputs/Inputs just place all nxlog.conf content in the default nxlog snippet, in the Windows section. Be sure you have in sync the collector side client_id with the one in the nxlog.conf content, Output section (see Exec $gl2_source_collector = '0324bd79-b9a9-458e-9977-d0eecbd2d347';):

<Output 576cf426659fb107f722b011>
    Module om_tcp
    Host graylog.mycompany.local
    Port 12201
    OutputType  GELF_TCP
    Exec $short_message = $raw_event; # Avoids truncation of the short_message field.
    Exec $gl2_source_collector = '0324bd79-b9a9-458e-9977-d0eecbd2d347';
    Exec $Hostname = hostname_fqdn();
</Output>

Duplicate configuration on another server

Once we have a collector configured we can duplicate easily the installation on another one:

• Perform the NXLog installation steps
• Install the collector-sidecar (just run the setup)
• Copy yaml and nxlog.conf files on the target server
• Change the yaml to reflect the right node_id
• Change the content of the collector-id with some unique string (a GUID?)
• Change the nxlog.conf Output tag to reflect the collector-id value

IyBUaGUgbmV3IGFwcHJvYWNoDQoNCiMjIFNlcnZlciBzaWRlIGNvbnRyb2xsZWQgY29uZmlndXJhdGlvbg0KDQpTdGFyaW5nIHdpdGggR3JheWxvZyAyIHRoZSBhIG5ldyBjb2xsZWN0b3IgYXBwcm9hY2ggKGBHcmF5bG9nIENvbGxlY3RvciBTaWRlY2FyYCkgaXMgcHJvcG9zZWQsIGJhc2VkIG9uIHdyYXBwaW5nIGBOWExvZ2AuDQoNCkluc3RhbGxhdGlvbiBpcyBzdHJhaWdodGZvcndhcmQsIGFzIGluZGljYXRlZCBpbiBvbmxpbmUgZG9jdW1lbnRzIChzYW1wbGUgZm9yIFdpbmRvd3MsIGEgbGF0ZXIgcG9zdCBtaWdodCBzaGFyZSBVYnVudHUgZXhwZXJpZW5jZSkuDQoNCkluc3RhbGwgTlhMb2c6DQoNCi0gRG93bmxvYWQgTlhMb2cgV2luZG93cyBpbnN0YWxsYXRpb24ga2l0IFtOWExvZyBkb3dubG9hZHNdKGh0dHBzOi8vbnhsb2cub3JnL3Byb2R1Y3RzL254bG9nLWNvbW11bml0eS1lZGl0aW9uL2Rvd25sb2FkICJOWExvZyBkb3dubG9hZHMiKQ0KLSBSdW4gdGhlIGluc3RhbGxlcg0KLSBSdW4gKGFzIGFuIGFkbWluaXN0cmF0b3IpIGAiQzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxueGxvZ1xueGxvZy5leGUiIC11YDsgdGhpcyB3aWxsIHVuLWluc3RhbGwgYG54bG9nYCBhcyBhIHNlcnZpY2UgYnV0IHdpbGwga2VlcCBiaW5hcmllcw0KDQpOb3cgZG93bmxvYWQgYW5kIGluc3RhbGwgbGF0ZXN0IGNvbGxlY3Rvci1zaWRlY2FyIGJpbmFyaWVzIChbQ29sbGVjdG9yLVNpZGVjYXIgcmVsZWFzZXNdKGh0dHBzOi8vZ2l0aHViLmNvbS9HcmF5bG9nMi9jb2xsZWN0b3Itc2lkZWNhci9yZWxlYXNlcykpDQoNCkJlZm9yZSBpbnN0YWxsaW5nL3N0YXJ0aW5nIHRoZSBjb2xsZWN0b3IgYXMgYSBzZXJ2aWNlOg0KDQotIE9uIHRoZSBzZXJ2ZXIgYWRtaW5pc3RyYXRpb24gc2l0ZSBnbyB0byBTeXN0ZW0tPkNvbGxlY3RvcnMtPk1hbmFnZSBDb25maWd1cmF0aW9ucw0KLSBBZGQgYSBjb25maWd1cmF0aW9uIHdpdGggYSBtZWFuaW5nZnVsIG5hbWUNCgktIEFkZCBvbmUgb3IgbW9yZSByZWxldmFudCB0YWdzDQoJLSBBZGQgYW4gT3V0cHV0IChieSBjaG9vc2luZyBmcm9tIGFscmVhZHkgY29uZmlndXJlZCBJbnB1dHMpDQoJLSBBZGQgb25lIG9yIG1vcmUgaW5wdXRzOyBmb3IgZmlsZSB0eXBlIGlucHV0cyB5b3UgY2FuIGNoZWNrIHRoZSBtdWx0aS1saW5lIG9wdGlvbiBhbmQgcHJvdmlkZSBhIFJlZ0V4IGZvciB0aGUgc3RhcnQgcGF0dGVybiwgbGlrZSBgL15cZHsyfVwuXGR7Mn1cLlxkezR9L2AgKG5vdGljZSBlbmNsb3Npbmcgc2xhc2hlcyEpDQoNCkdvIGJhY2sgdG8gdGhlIG1vbml0b3JlZCBjb21wdXRlciBhbmQgZWRpdCBgQzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxncmF5bG9nXGNvbGxlY3Rvci1zaWRlY2FyXGNvbGxlY3Rvci1zaWRlY2FyLnlhbWxgOg0KDQotIHVwZGF0ZSBzZXJ2ZXJfdXJsDQotIHVwZGF0ZSB0byBhIG1lYW5pbmdmdWwgbmFtZSB0aGUgbm9kZV9pZA0KLSBhZGQgb25lIG9mIHRoZSB0YWdzIHByZXZpb3VzbHkgZGVmaW5lZCBpbiB0aGUgdGFncyBsaXN0DQotIGluc3RhbGwgYW5kIHN0YXJ0IHRoZSBzZXJ2aWNlIChydW4gYXMgYWRtaW5pc3RyYXRvcik6DQoJLSBgIkM6XFByb2dyYW0gRmlsZXMgKHg4NilcZ3JheWxvZ1xjb2xsZWN0b3Itc2lkZWNhclxncmF5bG9nLWNvbGxlY3Rvci1zaWRlY2FyLmV4ZSIgLXNlcnZpY2UgaW5zdGFsbGANCgktIGAiQzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxncmF5bG9nXGNvbGxlY3Rvci1zaWRlY2FyXGdyYXlsb2ctY29sbGVjdG9yLXNpZGVjYXIuZXhlIiAtc2VydmljZSBzdGFydGANCg0KKlNlcnZpY2UgY2FuIGJlIHN0YXJ0ZWQvc3RvcHBlZCBhbHNvIGZyb20gVGFzayBtYW5hZ2VyIChTZXJ2aWNlcyB0YWIsIGBjb2xsZWN0b3Itc2lkZWNhcmAgTmFtZSkgb3IgZnJvbSBTZXJ2aWNlcyBNYW5hZ2VyIChgR3JheWxvZyBjb2xsZWN0b3Igc2lkZWNhcmAgRGlzcGxheSBuYW1lKS4NCg0KTm93IHdlIHNob3VsZCBoYXZlIGF1dG9tYXRpY2FsbHkgY3JlYXRlZCwgYmFzZWQgb24gdGhlIHByZXZpb3VzbHkgZGVmaW5lZCBjb25maWd1cmF0aW9uLCB0aGUgbnhsb2cgY29uZmlndXJhdGlvbiBmaWxlIGBDOlxQcm9ncmFtIEZpbGVzICh4ODYpXGdyYXlsb2dcY29sbGVjdG9yLXNpZGVjYXJcZ2VuZXJhdGVkXG54bG9nLmNvbmZgIGFuZCBpbiBhZG1pbmlzdHJhdGlvbiBzaXRlIHRoZSBjb2xsZWN0b3Igc2hvdWxkIGJlIGxpc3RlZCBpbiBTeXN0ZW0tPkNvbGxlY3RvcnMgc2VjdGlvbi4NCg0KUGxlYXNlIG5vdGljZSB0aGF0IGlmIHRoZXJlIGlzIHNvbWUgZXJyb3IgaW4gdGhlIHlhbWwgY29uZmlndXJhdGlvbiBmaWxlIHRoZSBzZXJ2aWNlIHdpbGwgbm90IHN0YXJ0IGFuZCBubyBlcnJvciBtZXNzYWdlIHdpbGwgYmUgbG9nZ2VkIChpdCBoYXBwZW5lZCB0byBtZSBpZiBJIGNvbW1lbnRlZCB0aGUgdGFncyBsaXN0LCBzZWUgbmV4dCBzZWN0aW9uKQ0KDQojIyBMb2NhbGx5IGNvbnRyb2xsZWQgY29uZmlndXJhdGlvbg0KDQpVbmRlciBjZXJ0YWluIGNvbmRpdGlvbnMgeW91IG1pZ2h0IG5lZWQgdG8gdXNlIGFkdmFuY2VkIE5YTG9nIGZlYXR1cmVzIHdoaWNoIGFyZSBub3Qgc3VwcG9ydGVkIGJ5IHRoZSBzZXJ2ZXIgc2lkZSBjb25maWd1cmF0aW9uLiBZb3UgY2FuIGRvIHRoZSBmb2xsb3dpbmcgdHJpY2s6DQotIFN0YXJ0IHdpdGggdGhlIGFwcHJvYWNoIGRlc2NyaWJlZCBpbiB0aGUgcHJldmlvdXMgc2VjdGlvbiBhbmQgY2hlY2sgdGhhdCBldmVyeXRoaW5nIGdvZXMgZmluZTsgYXQgdGhpcyBtb21lbnQgd2Ugc2hvdWxkIGhhdmUgYSB2YWxpZCBOWExvZyBjb25maWd1cmF0aW9uIGZpbGUNCi0gTW92ZSB0aGUgY29uZmlndXJhdGlvbiBmaWxlIGluIGFub3RoZXIgZGlyZWN0b3J5IGFuZCBjaGFuZ2UgdGhlIGNvbmZpZ3VyYXRpb25fcGF0aCB2YWx1ZSB0byB0aGF0IGZpbGUNCi0gUmVtb3ZlIGFsbCB2YWxpZCB0YWdzIGFuZCBhZGQgYSBkdW1teSBvbmUgdGhhdCBkb2VzIG5vdCBleGlzdHMgaW4gc2VydmVyIHNpZGUgY29uZmlndXJhdGlvbjsgZG8gbm90IHJlbW92ZSB0aGUgdGFncyBlbnRyeSBvciBsZWF2ZSB0aGUgbGlzdCBlbXB0eSBzaW5jZSB0aGUgY29sbGVjdG9yIHdpbGwgbm90IHN0YXJ0IGFuZCBubyBlcnJvciBtZXNzYWdlIHdpbGwgYmUgbG9nZ2VkDQotIFJlc3RhcnQgdGhlIGNvbGxlY3Rvci1zaWRlY2FyIHNlcnZpY2UNCi0gSWYgZXZlcnl0aGluZyB3b3JrcyBhcyBleHBlY3RlZCwgcmVhZCBjYXJlZnVsbHkgdGhlIE5YTG9nIG1hbnVhbCBhbmQgZG8geW91ciBvd24gZW5oYW5jZW1lbnRzIHRvIHRoZSBjb25maWd1cmF0aW9uDQotIFJlc3RhcnQgdGhlIGNvbGxlY3Rvci1zaWRlY2FyIHNlcnZpY2UNCg0KSSBhcHBsaWVkIHRoZSBhcHByb2FjaCBiZWNhdXNlIEkgbmVlZGVkIGFuIGFkaXRpb25hbCBtZXNzYWdlIGZpZWxkIChgRXhlYyAkc291cmNlX2lkID0gJ1dpbGRGbHkxMCc7YCkgc291cmNlX2lkIGZvciBvbmUgb2YgdGhlIGlucHV0cywgbGlrZSBzaG93biBiZWxvdy4NCmBgYA0KPElucHV0IDU3NmQxZDNkNjU5ZmIxMDdmNzIyZGIzOD4NCglNb2R1bGUgaW1fZmlsZQ0KCUZpbGUgIkM6XEphdmFcd2lsZGZseS0xMC4wLjAuRmluYWxcc3RhbmRhbG9uZVxsb2dcc2VydmVyLmxvZyINCglQb2xsSW50ZXJ2YWwgMQ0KCVNhdmVQb3MJVHJ1ZQ0KCVJlYWRGcm9tTGFzdCBUcnVlDQoJUmVjdXJzaXZlIEZhbHNlDQoJUmVuYW1lQ2hlY2sgRmFsc2UNCglFeGVjICRGaWxlTmFtZSA9IGZpbGVfbmFtZSgpOyAjIFNlbmQgZmlsZSBuYW1lIHdpdGggZWFjaCBtZXNzYWdlDQoJRXhlYyAkc291cmNlX2lkID0gJ1dpbGRGbHkxMCc7DQoJSW5wdXRUeXBlIDU3NmQxZDNkNjU5ZmIxMDdmNzIyZGIzOC1tdWx0aWxpbmUNCjwvSW5wdXQ+DQoNCmBgYA0KDQpUaGUgZnVsbCB5YW1sIGNvbmZpZ3VyYXRpb24gZmlsZSBjb250ZW50Og0KYGBgDQpzZXJ2ZXJfdXJsOiBodHRwOi8vZ3JheWxvZy5teWNvbXBhbnkubG9jYWw6MTI5MDANCnRsc19za2lwX3ZlcmlmeTogZmFsc2UNCm5vZGVfaWQ6IGdyYXlsb2ctY29sbGVjdG9yLXNpZGVjYXItZGV2c3J2DQpjb2xsZWN0b3JfaWQ6IGZpbGU6QzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxncmF5bG9nXGNvbGxlY3Rvci1zaWRlY2FyXGNvbGxlY3Rvci1pZA0KdGFnczoNCiAgICAtIHVuZGVmaW5lZA0KbG9nX3BhdGg6IEM6XFByb2dyYW0gRmlsZXMgKHg4NilcZ3JheWxvZ1xjb2xsZWN0b3Itc2lkZWNhcg0KdXBkYXRlX2ludGVydmFsOiAxMA0KYmFja2VuZHM6DQogICAgLSBuYW1lOiBueGxvZw0KICAgICAgZW5hYmxlZDogdHJ1ZQ0KICAgICAgYmluYXJ5X3BhdGg6IEM6XFByb2dyYW0gRmlsZXMgKHg4Nilcbnhsb2dcbnhsb2cuZXhlDQojICAgICAgY29uZmlndXJhdGlvbl9wYXRoOiBDOlxQcm9ncmFtIEZpbGVzICh4ODYpXGdyYXlsb2dcY29sbGVjdG9yLXNpZGVjYXJcZ2VuZXJhdGVkXG54bG9nLmNvbmYNCiAgICAgIGNvbmZpZ3VyYXRpb25fcGF0aDogQzpcUHJvZ3JhbSBGaWxlcyAoeDg2KVxncmF5bG9nXGNvbGxlY3Rvci1zaWRlY2FyXGNvbmZcbnhsb2cuY29uZg0KYGBgDQoNCkFzIGFuIGFsdGVybmF0ZSBhcHByb2FjaCAoSSBoYXZlbid0IHRlc3RlZCBidXQgaXQgc2hvdWxkIHdvcmspIGlzIHRvIGdvIGZvciB0aGUgZmlyc3Qgb3B0aW9uIChzZXJ2ZXIgc2lkZSBjb250cm9sbGVkIGNvbmZpZ3VyYXRpb24pIGJ1dCBpbnN0ZWFkIG9mIGRlZmluaW5nIE91dHB1dHMvSW5wdXRzIGp1c3QgcGxhY2UgYWxsIG54bG9nLmNvbmYgY29udGVudCBpbiB0aGUgZGVmYXVsdCBueGxvZyBzbmlwcGV0LCBpbiB0aGUgV2luZG93cyBzZWN0aW9uLiBCZSBzdXJlIHlvdSBoYXZlIGluIHN5bmMgdGhlIGNvbGxlY3RvciBzaWRlIGNsaWVudF9pZCB3aXRoIHRoZSBvbmUgaW4gdGhlIG54bG9nLmNvbmYgY29udGVudCwgT3V0cHV0IHNlY3Rpb24gKHNlZSBgRXhlYyAkZ2wyX3NvdXJjZV9jb2xsZWN0b3IgPSAnMDMyNGJkNzktYjlhOS00NThlLTk5NzctZDBlZWNiZDJkMzQ3JztgKToNCg0KYGBgDQo8T3V0cHV0IDU3NmNmNDI2NjU5ZmIxMDdmNzIyYjAxMT4NCglNb2R1bGUgb21fdGNwDQoJSG9zdCBncmF5bG9nLm15Y29tcGFueS5sb2NhbA0KCVBvcnQgMTIyMDENCglPdXRwdXRUeXBlICBHRUxGX1RDUA0KCUV4ZWMgJHNob3J0X21lc3NhZ2UgPSAkcmF3X2V2ZW50OyAjIEF2b2lkcyB0cnVuY2F0aW9uIG9mIHRoZSBzaG9ydF9tZXNzYWdlIGZpZWxkLg0KCUV4ZWMgJGdsMl9zb3VyY2VfY29sbGVjdG9yID0gJzAzMjRiZDc5LWI5YTktNDU4ZS05OTc3LWQwZWVjYmQyZDM0Nyc7DQoJRXhlYyAkSG9zdG5hbWUgPSBob3N0bmFtZV9mcWRuKCk7DQo8L091dHB1dD4NCmBgYA0KDQojIyMgRHVwbGljYXRlIGNvbmZpZ3VyYXRpb24gb24gYW5vdGhlciBzZXJ2ZXINCg0KT25jZSB3ZSBoYXZlIGEgY29sbGVjdG9yIGNvbmZpZ3VyZWQgd2UgY2FuIGR1cGxpY2F0ZSBlYXNpbHkgdGhlIGluc3RhbGxhdGlvbiBvbiBhbm90aGVyIG9uZToNCg0KDQotIFBlcmZvcm0gdGhlIE5YTG9nIGluc3RhbGxhdGlvbiBzdGVwcw0KLSBJbnN0YWxsIHRoZSBjb2xsZWN0b3Itc2lkZWNhciAoanVzdCBydW4gdGhlIHNldHVwKQ0KLSBDb3B5IHlhbWwgYW5kIG54bG9nLmNvbmYgZmlsZXMgb24gdGhlIHRhcmdldCBzZXJ2ZXINCi0gQ2hhbmdlIHRoZSB5YW1sIHRvIHJlZmxlY3QgdGhlIHJpZ2h0IG5vZGVfaWQNCi0gQ2hhbmdlIHRoZSBjb250ZW50IG9mIHRoZSBjb2xsZWN0b3ItaWQgd2l0aCBzb21lIHVuaXF1ZSBzdHJpbmcgKGEgR1VJRD8pDQotIENoYW5nZSB0aGUgbnhsb2cuY29uZiBPdXRwdXQgdGFnIHRvIHJlZmxlY3QgdGhlIGNvbGxlY3Rvci1pZCB2YWx1ZSAg

Cracking Passwords

The problem

We have to face the following situation: we want to migrate our customers from one system (legacy) to another. Customer data (emails,, address, login) are the easy part. What about passwords; wouldn't be nice that the users login into the new system with their actual passwords?

Technological we weren't lucky:

• the old system stores passwords as crypt(3) over MD5 (algorithm ID 1); see https://quickhash.com/crypt3-md5-online if you want generate some test data.
• the new system accepts SHA-256

so we can't just copy also the hashed password

Remark: by no means assertions in this blog should be considered the final truth: while facts and samples were tested I might missed some options that might further enhance your experience.

The solution

As I hope you understand this is just used for documenting some cracking password tools not a recommendation for a best practice: imagine what customers would thing about your ethics if realizing what you have done...

So, coming back to the topic: try some cracking tools, do a reverse engineering of passwords and then compute the target SHA-256 hash.

Problem solved! Well, not exactly. Having a technical solution is not always equivalent with having the problem actually solved, as we will discover soon.

Tools investigated:

• john the ripper (http://www.openwall.com/john/)
• hashcat (https://hashcat.net/wiki/)

John

Basic usage is straight forward:

• Export hashed password in accepted text format: <username>:<crypt(3)> per line; something like florin:$1$florin$1Y2H3TC.KoSWWN7v9hUZ0.
• run john: john passwords.txt

Several variations

Single mode

Run just rules from the configuration file section [List.Rules:Single] (see john.ini)

john --single passwords.txt

Incremental

Based on the Incremental:XXX section definition john will try (brute force) all combinations. See below a customized MyAlpha section:

[Incremental:MyAlpha]
File = $JOHN/alpha.chr
MinLen = 1
MaxLen = 6
CharCount = 26

rem john -i=MyAlpha ../Passwords.txt

Remarks

The good news

Not so good for customers actually: the single mode will be able to quick (in a matter of seconds for the 1000 samples I used) solve 10% of the users by using common patterns applied to their user names. This might not be valid for other samples

The bad news

john is single threaded, so I had to split the passwords.txt into four files and run it in parallel. After several days just some additional tens of passwords were discovered, (1, 2 or 3 letters/digits) so I cad say good by to the rest

Hashcat

It comes into two flavours:

• CPU based (https://hashcat.net/hashcat/)
• GPU based (https://hashcat.net/oclhashcat/)

If you are lucky enough to have one of the two supported GPU' (AMD or Nvidia) go for the second (several magnitude times faster in my samples!).

The usage looked more complicated than for john (might worth to explore dipper); what I used sort of extensively was the mask attack (https://hashcat.net/wiki/doku.php?id=mask_attack).

Password text file should contain (per line) just the ctrypto(3) part, no user names

Samples (should be the same for CPU based hashcat (hashcat-cli64.exe)):

cudaHashcat64.exe --hash-type=500 -a 3 CryptHashes.txt ?l?l?l?l?l
cudaHashcat64.exe --hash-type=500 -a 3 CryptHashes.txt ?l?l?l?d
cudaHashcat64.exe --hash-type=500 -a 3 CryptHashes.txt ?l?l?l?l?d
cudaHashcat64.exe --hash-type=500 -a 3 CryptHashes.txt ?l?l?l?l?l?d
cudaHashcat64.exe --hash-type=500 -a 3 CryptHashes.txt ?l?l?l?l?l?l?d
cudaHashcat64.exe --increment --increment-min=4 --hash-type=500 -a 3 CryptHashes.txt -1 ?l?d ?1?1?1?1?1?1

You can use up to 4 custom char set definitions, something like:

cudaHashcat64.exe --hash-type=500 -a 3 CryptHashes.txt -1 ?u -2 ?l -3 ?d ?1?2?2?2?3?3

This will mean try all combinations that start with an upper letter, contains 3 lower case letters and ends up into two figures.

Best guess: there is a tool that based on mask can generate word files on which you could run hashcat by applying rules. Just putting (once again, as best guess) together what I remember from reading chaotically the documentation.

The Good News

If passwords are short enough (5~6 in length) and you will be able to guess the right mask the GPU based version will might do the job (like 20 hours for ?l?l?l?l?l mask). CPU load will be minimal.

The Bad News

If the users were clever enough to use longer passwords, combination of Upper/lower/digits/special character spread all over the string you are stuck in months/years of waiting

Common Remarks

Found passwords are stored in john.pot, respectively *hashcat.pot Both tools are re-start-able:

• cold restart: just restart the process and found passwords are skipped (but the process is started from the beginning on a smaller list of hashes)
• warm restart, based on a saved session which will actually restart were from it was interrupted:
  • tested tor cudaHashcat: provide a session name at first run and use that session name for restore

cudaHashcat64.exe --hash-type=500 -a 3 --session=Session CryptHashes10.txt  -1 abcdefghijlmnoprstuv ?1?1?1?1?1?1
...
cudaHashcat64.exe --session=Session --restore

Be aware: Do not quit immediately after restarting in restore mode; wait till the current restore point changes from zero to something near the actual progress value otherwise you will loose all work done so far. This is normal behaviour as stated on some answers on forum

IyBUaGUgcHJvYmxlbQ0KDQpXZSBoYXZlIHRvIGZhY2UgdGhlIGZvbGxvd2luZyBzaXR1YXRpb246IHdlIHdhbnQgdG8gbWlncmF0ZSBvdXIgY3VzdG9tZXJzIGZyb20gb25lIHN5c3RlbSAobGVnYWN5KSB0byBhbm90aGVyLiBDdXN0b21lciBkYXRhIChlbWFpbHMsLCBhZGRyZXNzLCBsb2dpbikgYXJlIHRoZSBlYXN5IHBhcnQuIFdoYXQgYWJvdXQgcGFzc3dvcmRzOyB3b3VsZG4ndCBiZSBuaWNlIHRoYXQgdGhlIHVzZXJzIGxvZ2luIGludG8gdGhlIG5ldyBzeXN0ZW0gd2l0aCB0aGVpciBhY3R1YWwgcGFzc3dvcmRzPw0KDQpUZWNobm9sb2dpY2FsIHdlIHdlcmVuJ3QgbHVja3k6DQotIHRoZSBvbGQgc3lzdGVtIHN0b3JlcyBwYXNzd29yZHMgYXMgY3J5cHQoMykgb3ZlciBNRDUgKGFsZ29yaXRobSBJRCAxKTsgc2VlIFtodHRwczovL3F1aWNraGFzaC5jb20vY3J5cHQzLW1kNS1vbmxpbmVdKGh0dHBzOi8vcXVpY2toYXNoLmNvbS9jcnlwdDMtbWQ1LW9ubGluZSkgaWYgeW91IHdhbnQgZ2VuZXJhdGUgc29tZSB0ZXN0IGRhdGEuDQotIHRoZSBuZXcgc3lzdGVtIGFjY2VwdHMgU0hBLTI1Ng0KDQpzbyB3ZSBjYW4ndCBqdXN0IGNvcHkgYWxzbyB0aGUgaGFzaGVkIHBhc3N3b3JkDQoNCioqUmVtYXJrKio6IGJ5IG5vIG1lYW5zIGFzc2VydGlvbnMgaW4gdGhpcyBibG9nIHNob3VsZCBiZSBjb25zaWRlcmVkIHRoZSBmaW5hbCB0cnV0aDogd2hpbGUgZmFjdHMgYW5kIHNhbXBsZXMgd2VyZSB0ZXN0ZWQgSSBtaWdodCBtaXNzZWQgc29tZSBvcHRpb25zIHRoYXQgbWlnaHQgZnVydGhlciBlbmhhbmNlIHlvdXIgZXhwZXJpZW5jZS4NCg0KIyBUaGUgc29sdXRpb24NCg0KKkFzIEkgaG9wZSB5b3UgdW5kZXJzdGFuZCB0aGlzIGlzIGp1c3QgdXNlZCBmb3IgZG9jdW1lbnRpbmcgc29tZSBjcmFja2luZyBwYXNzd29yZCB0b29scyBub3QgYSByZWNvbW1lbmRhdGlvbiBmb3IgYSBiZXN0IHByYWN0aWNlOiBpbWFnaW5lIHdoYXQgY3VzdG9tZXJzIHdvdWxkIHRoaW5nIGFib3V0IHlvdXIgZXRoaWNzIGlmIHJlYWxpemluZyB3aGF0IHlvdSBoYXZlIGRvbmUuLi4qDQoNClNvLCBjb21pbmcgYmFjayB0byB0aGUgdG9waWM6IHRyeSBzb21lIGNyYWNraW5nIHRvb2xzLCBkbyBhIHJldmVyc2UgZW5naW5lZXJpbmcgb2YgcGFzc3dvcmRzIGFuZCB0aGVuIGNvbXB1dGUgdGhlIHRhcmdldCBTSEEtMjU2IGhhc2guDQoNClByb2JsZW0gc29sdmVkISBXZWxsLCBub3QgZXhhY3RseS4gSGF2aW5nIGEgdGVjaG5pY2FsIHNvbHV0aW9uIGlzIG5vdCBhbHdheXMgZXF1aXZhbGVudCB3aXRoIGhhdmluZyB0aGUgcHJvYmxlbSBhY3R1YWxseSBzb2x2ZWQsIGFzIHdlIHdpbGwgZGlzY292ZXIgc29vbi4NCg0KVG9vbHMgaW52ZXN0aWdhdGVkOg0KLSBqb2huIHRoZSByaXBwZXIgKFtodHRwOi8vd3d3Lm9wZW53YWxsLmNvbS9qb2huL10oaHR0cDovL3d3dy5vcGVud2FsbC5jb20vam9obi8pKQ0KLSBoYXNoY2F0IChbaHR0cHM6Ly9oYXNoY2F0Lm5ldC93aWtpL10oaHR0cHM6Ly9oYXNoY2F0Lm5ldC93aWtpLykpDQoNCiMjIEpvaG4NCg0KQmFzaWMgdXNhZ2UgaXMgc3RyYWlnaHQgZm9yd2FyZDoNCi0gRXhwb3J0IGhhc2hlZCBwYXNzd29yZCBpbiBhY2NlcHRlZCB0ZXh0IGZvcm1hdDogYDx1c2VybmFtZT46PGNyeXB0KDMpPmAgcGVyIGxpbmU7IHNvbWV0aGluZyBsaWtlIGBmbG9yaW46JDEkZmxvcmluJDFZMkgzVEMuS29TV1dON3Y5aFVaMC5gDQotIHJ1biBqb2huOiBgam9obiBwYXNzd29yZHMudHh0YA0KDQojIyMgU2V2ZXJhbCB2YXJpYXRpb25zDQoNCiMjIyMgU2luZ2xlIG1vZGUNCg0KUnVuIGp1c3QgcnVsZXMgZnJvbSB0aGUgY29uZmlndXJhdGlvbiBmaWxlIHNlY3Rpb24gW0xpc3QuUnVsZXM6U2luZ2xlXSAoc2VlIGpvaG4uaW5pKQ0KDQpgam9obiAtLXNpbmdsZSBwYXNzd29yZHMudHh0YA0KDQojIyMjIEluY3JlbWVudGFsDQoNCkJhc2VkIG9uIHRoZSBgSW5jcmVtZW50YWw6WFhYYCBzZWN0aW9uIGRlZmluaXRpb24gam9obiB3aWxsIHRyeSAoYnJ1dGUgZm9yY2UpIGFsbCBjb21iaW5hdGlvbnMuIFNlZSBiZWxvdyBhIGN1c3RvbWl6ZWQgTXlBbHBoYSBzZWN0aW9uOg0KDQpgYGANCltJbmNyZW1lbnRhbDpNeUFscGhhXQ0KRmlsZSA9ICRKT0hOL2FscGhhLmNocg0KTWluTGVuID0gMQ0KTWF4TGVuID0gNg0KQ2hhckNvdW50ID0gMjYNCmBgYA0KYHJlbSBqb2huIC1pPU15QWxwaGEgLi4vUGFzc3dvcmRzLnR4dGANCg0KIyMjIFJlbWFya3MNCg0KIyMjIyBUaGUgZ29vZCBuZXdzDQoNCk5vdCBzbyBnb29kIGZvciBjdXN0b21lcnMgYWN0dWFsbHk6IHRoZSBzaW5nbGUgbW9kZSB3aWxsIGJlIGFibGUgdG8gcXVpY2sgKGluIGEgbWF0dGVyIG9mIHNlY29uZHMgZm9yIHRoZSAxMDAwIHNhbXBsZXMgSSB1c2VkKSBzb2x2ZSAxMCUgb2YgdGhlIHVzZXJzIGJ5IHVzaW5nIGNvbW1vbiBwYXR0ZXJucyBhcHBsaWVkIHRvIHRoZWlyIHVzZXIgbmFtZXMuIFRoaXMgbWlnaHQgbm90IGJlIHZhbGlkIGZvciBvdGhlciBzYW1wbGVzDQoNCiMjIyMgVGhlIGJhZCBuZXdzDQoNCioqam9obioqIGlzIHNpbmdsZSB0aHJlYWRlZCwgc28gSSBoYWQgdG8gc3BsaXQgdGhlIHBhc3N3b3Jkcy50eHQgaW50byBmb3VyIGZpbGVzIGFuZCBydW4gaXQgaW4gcGFyYWxsZWwuIEFmdGVyIHNldmVyYWwgZGF5cyBqdXN0IHNvbWUgYWRkaXRpb25hbCB0ZW5zIG9mIHBhc3N3b3JkcyB3ZXJlIGRpc2NvdmVyZWQsICgxLCAyIG9yIDMgbGV0dGVycy9kaWdpdHMpIHNvIEkgY2FkIHNheSBnb29kIGJ5IHRvIHRoZSByZXN0DQoNCiMjIEhhc2hjYXQNCg0KSXQgY29tZXMgaW50byB0d28gZmxhdm91cnM6DQotIENQVSBiYXNlZCAoW2h0dHBzOi8vaGFzaGNhdC5uZXQvaGFzaGNhdC9dKGh0dHBzOi8vaGFzaGNhdC5uZXQvaGFzaGNhdC8pKQ0KLSBHUFUgYmFzZWQgKFtodHRwczovL2hhc2hjYXQubmV0L29jbGhhc2hjYXQvXShodHRwczovL2hhc2hjYXQubmV0L29jbGhhc2hjYXQvKSkNCg0KSWYgeW91IGFyZSBsdWNreSBlbm91Z2ggdG8gaGF2ZSBvbmUgb2YgdGhlIHR3byBzdXBwb3J0ZWQgR1BVJyAoQU1EIG9yIE52aWRpYSkgZ28gZm9yIHRoZSBzZWNvbmQgKHNldmVyYWwgbWFnbml0dWRlIHRpbWVzIGZhc3RlciBpbiBteSBzYW1wbGVzISkuDQoNClRoZSB1c2FnZSBsb29rZWQgbW9yZSBjb21wbGljYXRlZCB0aGFuIGZvciBqb2huIChtaWdodCB3b3J0aCB0byBleHBsb3JlIGRpcHBlcik7IHdoYXQgSSB1c2VkIHNvcnQgb2YgZXh0ZW5zaXZlbHkgd2FzIHRoZSBgbWFzayBhdHRhY2tgIChbaHR0cHM6Ly9oYXNoY2F0Lm5ldC93aWtpL2Rva3UucGhwP2lkPW1hc2tfYXR0YWNrXShodHRwczovL2hhc2hjYXQubmV0L3dpa2kvZG9rdS5waHA/aWQ9bWFza19hdHRhY2spKS4NCg0KUGFzc3dvcmQgdGV4dCBmaWxlIHNob3VsZCBjb250YWluIChwZXIgbGluZSkganVzdCB0aGUgY3RyeXB0bygzKSBwYXJ0LCBubyB1c2VyIG5hbWVzDQoNClNhbXBsZXMgKHNob3VsZCBiZSB0aGUgc2FtZSBmb3IgQ1BVIGJhc2VkIGhhc2hjYXQgKGBoYXNoY2F0LWNsaTY0LmV4ZWApKToNCmBgYA0KY3VkYUhhc2hjYXQ2NC5leGUgLS1oYXNoLXR5cGU9NTAwIC1hIDMgQ3J5cHRIYXNoZXMudHh0ID9sP2w/bD9sP2wNCmN1ZGFIYXNoY2F0NjQuZXhlIC0taGFzaC10eXBlPTUwMCAtYSAzIENyeXB0SGFzaGVzLnR4dCA/bD9sP2w/ZA0KY3VkYUhhc2hjYXQ2NC5leGUgLS1oYXNoLXR5cGU9NTAwIC1hIDMgQ3J5cHRIYXNoZXMudHh0ID9sP2w/bD9sP2QNCmN1ZGFIYXNoY2F0NjQuZXhlIC0taGFzaC10eXBlPTUwMCAtYSAzIENyeXB0SGFzaGVzLnR4dCA/bD9sP2w/bD9sP2QNCmN1ZGFIYXNoY2F0NjQuZXhlIC0taGFzaC10eXBlPTUwMCAtYSAzIENyeXB0SGFzaGVzLnR4dCA/bD9sP2w/bD9sP2w/ZA0KY3VkYUhhc2hjYXQ2NC5leGUgLS1pbmNyZW1lbnQgLS1pbmNyZW1lbnQtbWluPTQgLS1oYXNoLXR5cGU9NTAwIC1hIDMgQ3J5cHRIYXNoZXMudHh0IC0xID9sP2QgPzE/MT8xPzE/MT8xDQoNCmBgYA0KWW91IGNhbiB1c2UgdXAgdG8gNCBjdXN0b20gY2hhciBzZXQgZGVmaW5pdGlvbnMsIHNvbWV0aGluZyBsaWtlOg0KYGBgDQpjdWRhSGFzaGNhdDY0LmV4ZSAtLWhhc2gtdHlwZT01MDAgLWEgMyBDcnlwdEhhc2hlcy50eHQgLTEgP3UgLTIgP2wgLTMgP2QgPzE/Mj8yPzI/Mz8zDQpgYGANClRoaXMgd2lsbCBtZWFuIHRyeSBhbGwgY29tYmluYXRpb25zIHRoYXQgc3RhcnQgd2l0aCBhbiB1cHBlciBsZXR0ZXIsIGNvbnRhaW5zIDMgbG93ZXIgY2FzZSBsZXR0ZXJzIGFuZCBlbmRzIHVwIGludG8gdHdvIGZpZ3VyZXMuDQoNCg0KQmVzdCBndWVzczogdGhlcmUgaXMgYSB0b29sIHRoYXQgYmFzZWQgb24gbWFzayBjYW4gZ2VuZXJhdGUgd29yZCBmaWxlcyBvbiB3aGljaCB5b3UgY291bGQgcnVuIGhhc2hjYXQgYnkgYXBwbHlpbmcgcnVsZXMuIEp1c3QgcHV0dGluZyAob25jZSBhZ2FpbiwgYXMgYmVzdCBndWVzcykgdG9nZXRoZXIgd2hhdCBJIHJlbWVtYmVyIGZyb20gcmVhZGluZyBjaGFvdGljYWxseSB0aGUgZG9jdW1lbnRhdGlvbi4NCg0KIyMjIFRoZSBHb29kIE5ld3MNCg0KSWYgcGFzc3dvcmRzIGFyZSBzaG9ydCBlbm91Z2ggKDV+NiBpbiBsZW5ndGgpIGFuZCB5b3Ugd2lsbCBiZSBhYmxlIHRvIGd1ZXNzIHRoZSByaWdodCBtYXNrIHRoZSBHUFUgYmFzZWQgdmVyc2lvbiB3aWxsIG1pZ2h0IGRvIHRoZSBqb2IgKGxpa2UgMjAgaG91cnMgZm9yID9sP2w/bD9sP2wgbWFzaykuIENQVSBsb2FkIHdpbGwgYmUgbWluaW1hbC4NCg0KDQojIyMgVGhlIEJhZCBOZXdzDQpJZiB0aGUgdXNlcnMgd2VyZSBjbGV2ZXIgZW5vdWdoIHRvIHVzZSBsb25nZXIgcGFzc3dvcmRzLCBjb21iaW5hdGlvbiBvZiBVcHBlci9sb3dlci9kaWdpdHMvc3BlY2lhbCBjaGFyYWN0ZXIgc3ByZWFkIGFsbCBvdmVyIHRoZSBzdHJpbmcgeW91IGFyZSBzdHVjayBpbiBtb250aHMveWVhcnMgb2Ygd2FpdGluZw0KDQojIENvbW1vbiBSZW1hcmtzDQoNCkZvdW5kIHBhc3N3b3JkcyBhcmUgc3RvcmVkIGluIGpvaG4ucG90LCByZXNwZWN0aXZlbHkgKmhhc2hjYXQucG90DQpCb3RoIHRvb2xzIGFyZSByZS1zdGFydC1hYmxlOg0KDQotIGNvbGQgcmVzdGFydDoganVzdCByZXN0YXJ0IHRoZSBwcm9jZXNzIGFuZCBmb3VuZCBwYXNzd29yZHMgYXJlIHNraXBwZWQgKGJ1dCB0aGUgcHJvY2VzcyBpcyBzdGFydGVkIGZyb20gdGhlIGJlZ2lubmluZyBvbiBhIHNtYWxsZXIgbGlzdCBvZiBoYXNoZXMpDQotIHdhcm0gcmVzdGFydCwgYmFzZWQgb24gYSBzYXZlZCBzZXNzaW9uIHdoaWNoIHdpbGwgYWN0dWFsbHkgcmVzdGFydCB3ZXJlIGZyb20gaXQgd2FzIGludGVycnVwdGVkOg0KCS0gdGVzdGVkIHRvciBjdWRhSGFzaGNhdDogcHJvdmlkZSBhIHNlc3Npb24gbmFtZSBhdCBmaXJzdCBydW4gYW5kIHVzZSB0aGF0IHNlc3Npb24gbmFtZSBmb3IgcmVzdG9yZQ0KYGBgDQpjdWRhSGFzaGNhdDY0LmV4ZSAtLWhhc2gtdHlwZT01MDAgLWEgMyAtLXNlc3Npb249U2Vzc2lvbiBDcnlwdEhhc2hlczEwLnR4dCAgLTEgYWJjZGVmZ2hpamxtbm9wcnN0dXYgPzE/MT8xPzE/MT8xDQouLi4NCmN1ZGFIYXNoY2F0NjQuZXhlIC0tc2Vzc2lvbj1TZXNzaW9uIC0tcmVzdG9yZQ0KYGBgDQoqKkJlIGF3YXJlOioqIERvIG5vdCBxdWl0IGltbWVkaWF0ZWx5IGFmdGVyIHJlc3RhcnRpbmcgaW4gcmVzdG9yZSBtb2RlOyB3YWl0IHRpbGwgdGhlIGN1cnJlbnQgcmVzdG9yZSBwb2ludCBjaGFuZ2VzIGZyb20gemVybyB0byBzb21ldGhpbmcgbmVhciB0aGUgYWN0dWFsIHByb2dyZXNzIHZhbHVlIG90aGVyd2lzZSB5b3Ugd2lsbCBsb29zZSBhbGwgd29yayBkb25lIHNvIGZhci4gVGhpcyBpcyBub3JtYWwgYmVoYXZpb3VyIGFzIHN0YXRlZCBvbiBzb21lIGFuc3dlcnMgb24gZm9ydW0s

MySQL Linked Server from SQL Server 2014

It might work from other SQL Server versions...

Install MySQL ODBC connector

• Download and install latest x64 MySQL ODBC connector; version 5.3.6 at the moment of this post.
• Create an x64 ODBC System DSN; suppose we named it MySQLDS

Create the Linked Server

• Add a Linked Server
  • Name: at your will, suppose MySQLLS
  • Provider: Microsoft OLEDB Provider for ODBC Drivers
  • Product Name: MySQL
  • Data Source: MySQLDS
  • Provider String:ODBC;DSN=MySQLDS
  • Location: (empty)
  • Catalog:<name of desired catalog>
• On Security tab:
  • Check Be made using this security context:
    • root (not really very secure)
    • <password>

Change Provider Options (Linked Servers->Providesr->MSDASQL)

• Check Level zero only; this will allow sentences like SELECT * FROM MySQLLS...Table

IyBNeVNRTCBMaW5rZWQgU2VydmVyIGZyb20gU1FMIFNlcnZlciAyMDE0DQoNCkl0IG1pZ2h0IHdvcmsgZnJvbSBvdGhlciBTUUwgU2VydmVyIHZlcnNpb25zLi4uDQoNCiMjIEluc3RhbGwgTXlTUUwgT0RCQyBjb25uZWN0b3INCg0KKiBEb3dubG9hZCBhbmQgaW5zdGFsbCBsYXRlc3QgeDY0IE15U1FMIE9EQkMgY29ubmVjdG9yOyB2ZXJzaW9uIDUuMy42IGF0IHRoZSBtb21lbnQgb2YgdGhpcyBwb3N0Lg0KKiBDcmVhdGUgYW4geDY0IE9EQkMgU3lzdGVtIERTTjsgc3VwcG9zZSB3ZSBuYW1lZCBpdCBgTXlTUUxEU2ANCg0KIyMgQ3JlYXRlIHRoZSBMaW5rZWQgU2VydmVyDQoNCiogQWRkIGEgTGlua2VkIFNlcnZlcg0KCSogTmFtZTogYXQgeW91ciB3aWxsLCBzdXBwb3NlIGBNeVNRTExTYA0KCSogUHJvdmlkZXI6IGBNaWNyb3NvZnQgT0xFREIgUHJvdmlkZXIgZm9yIE9EQkMgRHJpdmVyc2ANCgkqIFByb2R1Y3QgTmFtZTogTXlTUUwNCgkqIERhdGEgU291cmNlOiBgTXlTUUxEU2ANCgkqIFByb3ZpZGVyIFN0cmluZzpgT0RCQztEU049TXlTUUxEU2ANCgkqIExvY2F0aW9uOiAoZW1wdHkpDQoJKiBDYXRhbG9nOmA8bmFtZSBvZiBkZXNpcmVkIGNhdGFsb2c+YA0KKiBPbiBTZWN1cml0eSB0YWI6DQoJKiBDaGVjayAqQmUgbWFkZSB1c2luZyB0aGlzIHNlY3VyaXR5IGNvbnRleHQqOg0KCQkqIGByb290YCAobm90IHJlYWxseSB2ZXJ5IHNlY3VyZSkNCgkJKiBcPHBhc3N3b3JkXD4NCg0KIyMgQ2hhbmdlIFByb3ZpZGVyIE9wdGlvbnMgKExpbmtlZCBTZXJ2ZXJzLT5Qcm92aWRlc3ItPk1TREFTUUwpDQoNCiogQ2hlY2sgYExldmVsIHplcm8gb25seWA7IHRoaXMgd2lsbCBhbGxvdyBzZW50ZW5jZXMgbGlrZSBgU0VMRUNUICogRlJPTSBNeVNRTExTLi4uVGFibGVg