Posts

Not too long ago I became obsessed with Prometheus. I'd heard about it for a while, knew it was powerful, and couldn't quite understand how everything fit together. The documentation is extremely verbose for good reason but it took playing with it for a while for everything to click. This post is a rather concise and extensive overview that goes a long way in expressing the basic concepts to my developer brain. In their simplest form, exporters expose an HTTP endpoint of /metrics with the output being statistics in Prometheus' format. The real power of Prometheus comes when you expose your own /metrics endpoint and have Prometheus consume the statistics you generate. This post is also a very good introduction with the section Building your own exporter being extremely valuable in describing just some of the possibilities.

After getting my bearings I started with a prototype with a simple premise "Why look at the usage graphs in Digital Ocean for each server independently? Why not have it in one location?" How To Install Prometheus on Ubuntu 16.04 is a very good primer to get everything up and running quickly.

I've made a few modifications since working through the article:

• Prometheus version 2.3.1

  • There have been massive perf improvements in v2.3.x.

• node_exporter version 0.16.0

  • There are significant changes to the metrics naming conventions.
  • This exporter typically has the most coupling with Grafana dashboards and often requires altering them to work correctly.

• Use prometheus:prometheus for ownership of core prometheus processes like prometheus or alertmanager.

  • sudo useradd --no-create-home --shell /bin/false prometheus

• Use prometheus-exporter:prometheus-exporter for ownership of exporters. Exporters should possibly be more isolated but I feel it may be a case of YAGNI.

  • sudo useradd --no-create-home --shell /bin/false prometheus-exporter

• Set scrape_interval to 1 minute: scrape_interval: 1m.

  • 15 seconds is still doable but I'm currently not concerned with very granular detail.
  • This reduces the load of making 4 calls per minute to just 1, reducing some overhead required for Prometheus and every exporter.

At $dayJob we've moved to provisioning servers using Laravel Forge, which has the possibility of utilizing exporters for mysqld, mariadb, postgres, memcached, redis, beanstalkd, nginx, php-fpm, and sendmail. I've opted to use node_exporter, mysqld, nginx-vts-exporter, php-fpm, and redis respectively. To put the original premise into perspective, replicating the newer monitoring agent graphs in Digital Ocean only require node_exporter. A few of the exporters require very little setup, only setting a few configuration variables systemd service definitions. Other exporters like nginx-vts-exporter require building nginx from source.

I plan to introduce a series of posts that should aid in getting a very rudimentary implementation running. There is an abundant usage of Kubernetes in the Prometheus ecosystem, to the point that it almost seems required but fortunately it also just works(tm) in a traditional virtual machine without any real fuss.

In this blog post on January 26 2015, Code School became part of Pluralsight. The website codeschool.com continued to operate normally until earlier this year when a banner showed the site would shut down and transition to Pluralsight June 1st. The banner pointed to this url, which gives a great overview of the changes but was sparse on what would take place during the transition.

It wasn't until June 1st that I finally understood the full breadth of the transition and stumbled upon the integration faqs. The important bit of information is this snippet:

Will I be able to access my Code School invoices or course history?

No. Your invoices and course history will not carry over or be accessible as of 6/1.

Code School customers were instructed to generate a PDF of their profile before the migration. Due to finding the integration FAQs after June 1st, sadly I wasn't able to do that in time.

What particularly impacts me the most is a belief that pointing potential employers to a reputable website as a source of truth carries far more weight than a PDF that can be altered. As a web developer in an industry where employers seem to assume a resume is partially or wholly embellished, this seems like a step backwards.

In spite of the transition pains, I do find Pluralsight's Skill IQ to be a fresh way to measure competency with multiple choice questions that cover broad aspects of a given topic. You're shown what is marked wrong so you can learn from your mistake and the equivalent of the old Code School subscription I believe allows unlimited retests. The integration with Stack Overflow's developer story is compelling enough to use it and I did gain quite a sense of accomplishment when I scored in the very low expert level range.

As I finished typing this up I noticed Pluralsight seems to have a fair number of the Code School courses by searching for the keyword "Code School". There are newer interactive courses like the one titled HTML 5 and CSS 3: Overview of Tag, Attribute and Selector Additions but the introductory video includes the Front End Formations title that it was called on Code School. It appears that some of the content is migrating over but things aren't 1:1 so we may never get credit for courses we've essentially completed. I plan on going through the course shortly as I hope at least the challenges have been updated but it would be a terrible experience to go through all of this realizing I've accomplished it recently.

I don't quite know how I feel about the transition a month in and now after noticing at least some of the content was moved over. It's hard to lose the accomplishments but the outcome would've been no different if Code School closed completely. It does have me pause to make sure the course accomplishments I share are worth the investment and that's likely an important thing to remember whenever similar services catch my attention.

I've been bitten by this issue so many times that I have a form of amnesia where I forget that it happened all over again. This github issue highlights the problem but I'm more of a visual learner.

The problem can be traced back to configuring the redirect_uri parameter incorrectly. OAuth2 highly requires that the callbacks are identical between the server and consumer(s). For consumers that are external to the app, this is almost never a problem. For first-party consumers like Swagger(vel), this is extremely easy to configure incorrectly.

After putting the solution in my previous post through its paces for a few weeks, I realized the less intrusive approach is to patch Homestead v2's scripts/create-mysql.sh with the following snippet:

#!/usr/bin/env bash

cat > /etc/mysql/conf.d/password_expiration.cnf << EOF
[mysqld]
default_password_lifetime = 0
EOF

service mysql restart

DB=$1;

mysql -e "CREATE DATABASE IF NOT EXISTS \`$DB\` DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_unicode_ci";

This change pipes the default_password_lifetime setting into the file /etc/mysql/conf.d/password_expiration.cnf and restarts the mysql service. The provisioning process then can proceed as normal.

This approach requires no updated vagrant virtualbox image or other similar adjustments and allows us to keep using version 0.3.3 indefinitely.

I'm likely going to abandon my settler and homestead forks as I couldn't adequately maintain them moving forward. I'll work to push this upstream as I feel it should be implemented there.

On November 7th 2016, I was hit with a peculiar issue I've never seen before working in a provisioned Homestead box. The exception:

PDOException in Connector.php line 55:
SQLSTATE[HY000] [1862] Your password has expired. To log in you must change it using a client that supports expired passwords.

Firing up a different vagrant machine, I was greeted with the same problem. This seemed to affect all of the vagrant boxes using version laravel/homestead (virtualbox, 0.3.3).

On the machine, the MySQL version displayed by mysql --version is

mysql  Ver 14.14 Distrib 5.7.9, for Linux (x86_64) using  EditLine wrapper

MySQL 5.7's password expiration policy seemed to point to the culprit.

From MySQL 5.7.4 to 5.7.10, the default default_password_lifetime value is 360 (passwords must be changed approximately once per year). For those versions, be aware that, if you make no changes to the default_password_lifetime variable or to individual user accounts, all user passwords will expire after 360 days, and all user accounts will start running in restricted mode when this happens.

Looking at the list of users with relevant columns shown that the password for the user homestead was set on 2015-11-13 03:50:18.

mysql> select host, user, authentication_string, password_expired, password_last_changed, password_lifetime from mysql.user;
+-----------+-----------+-------------------------------------------+------------------+-----------------------+-------------------+
| host      | user      | authentication_string                     | password_expired | password_last_changed | password_lifetime |
+-----------+-----------+-------------------------------------------+------------------+-----------------------+-------------------+
| localhost | root      | *14E65567ABDB5135D0CFD9A70B3032C179A49EE7 | N                | 2016-11-08 22:28:11   |              NULL |
| localhost | mysql.sys | *THISISNOTAVALIDPASSWORDTHATCANBEUSEDHERE | N                | 2015-11-13 03:50:10   |              NULL |
| 0.0.0.0   | root      | *14E65567ABDB5135D0CFD9A70B3032C179A49EE7 | N                | 2015-11-13 03:50:15   |              NULL |
| 0.0.0.0   | homestead | *14E65567ABDB5135D0CFD9A70B3032C179A49EE7 | N                | 2015-11-13 03:50:18   |              NULL |
| %         | homestead | *14E65567ABDB5135D0CFD9A70B3032C179A49EE7 | N                | 2015-11-13 03:50:18   |              NULL |
+-----------+-----------+-------------------------------------------+------------------+-----------------------+-------------------+
5 rows in set (0.00 sec)

Date manipulation in PHP showed that 360 days from 2015-11-13 03:50:18 is 2016-11-07 03:50:18, about the time this started occurring.

It was later that I discovered this pull request didn't make it into branch revert-56-master used to build the 0.3.3 box. It succinctly described the problem at hand.

I saw 4 possible choices for a permanent solution:

1. Set default_password_lifetime=0 explicitly in /etc/mysql/my.cnf.
2. Upgrade MySQL to 5.7.11 or higher as the default was changed from 360 to 0.
3. Create the homestead user with the keywords PASSWORD EXPIRE NEVER to disable password expiration for that user.
4. All of the above.

Solution

In looking to correct upstream, the pull request was denied with very good reason. It was a ton of work to seemingly get the 5.6 branch up to master and I have absolutely no guarantee that something wasn't broken in the process.

Not being content with abandoning that work, I pushed a vagrant virtualbox image that should continue the 5.6 branch forward for the foreseeable future. There is one major caveat, it requires a patch to Homestead v2 to accommodate the changes introduced.

Steps required to use the image:

1. Add the new vagrant image: vagrant box add w0rd-driven/homestead.
2. Add the line box: "w0rd-driven/homestead" in Homestead.yaml to specify a different vagrant box than the default of laravel/homestead.
3. Add the line "laravel/homestead": "2.0.x-dev" to the require-dev section of composer.json.
4. Add or update the repositories section of composer.json:

"repositories": [
    {
        "type": "git",
        "url": "https://github.com/w0rd-driven/homestead.git"
    }
],

5. Run composer update to change to the new composer package.
6. Rebuild your box using vagrant destroy -f then vagrant up.
7. Test everything.

I've enabled issues on both forks of settler and homestead. Unfortunately, I don't have VMWare Fusion to build the vmware provider image. If anyone has the capabilities, I would gladly grant the access to push the image.