Pump

Seeing that I've had a whole string of StatusNet posts, I figure that it's time to start some pump.io posts. I've had a pump running since mid January and it's been much fun watching the early development happen. As of this writing, the current stable version (which Evan has his various pumps running) is v0.2.1. My pump is running git master because I like to live on the edge. Although, for the most part, it has been mostly painless. I have tried my damnedest to put any issues that I come across on the issue list. With this post, I will be going through my setup. I should note that my setup may not work for everyone, YMMV.

I am running my pump on my mostly recently unused EeePC 1000HE w/2GB RAM which is running good ol' ArchLinux. Seeing that my tablet has become my goto mobile device (and even got a keyboard dock for it as well), my netbook just seems bulky and underpowered anymore. (bulky? what?) It was time to find something else for it to do. Enter the pump.

Before starting the rest, here are the versions of various things that I'm currently using as of this writing:

  • nodejs: v0.8.21
  • npm: v1.2.11
  • redis: v2.6.10
  • nginx: v1.3.14
  • pumpio: v0.3.0-alpha1 Also, I grabbed a free SSL cert from StartSSL as I do for all of my domains/subdomains.

Following the Local install directions, I installed nodejs and Redis and cloned pump out of git and installed the dependencies with npm. Along the way of running the pump, I've also installed a couple of other tools which have helped things out, forever and bunyan. Both of these I've installed globally as root with:

# npm install -g bunyan
# npm install -g forever

For Redis, I've left it with the default settings. To setup the pump, I copied the pump.io.json.sample to /etc/pump.io.json and edited it similar to this:

{
 "driver": "redis",
 "params": {"host":"localhost","port":6379},
 "secret": "secretkeygoeshere",
 "noweb": false,
 "site": "pump.jpope.org",
 "owner": "jpope dot org",
 "ownerURL": "http://jpope.org/",
 "port": 4443,
 "urlPort":443,
 "hostname": "pump.jpope.org",
 "address": "int.er.nal.ip",
 "nologger": false,
 "serverUser": "pumpio",
 "key": "/path/to/pump.key",
 "cert": "/path/to/pump.crt",
 "uploaddir": "/srv/http/pump.jpope.org/uploads",
 "debugClient": false,
 "firehose": "ofirehose.com",
 "noCDN":true,
 "requireEmail": false,
 "disableRegistration": false,
 "canUpload": true,
 "sockjs": true
 }

A few of these setting here I have added from finding them in the code here and there. After this was setup, I created the pumpio user that I wanted the pump to run as with:

# useradd -s /bin/bash -d /srv/http/pump.jpope.org

And then as root, started the pump with

# ./bin/pump

Since my main webserver runs nginx, I needed to setup a proxy so that requests would pass through to the EeePC. Here is an example of my nginx conf:

upstream pumpbackend {
server eee.pc.ip.add:4443 max_fails=3 fail_timeout=30s;
server eee.pc.ip.add:4443 max_fails=3 fail_timeout=60s;
server eee.pc.ip.add:4443 max_fails=3 fail_timeout=90s;
}
server {
listen 80;
server_name pump.jpope.org;
rewrite ^ https://pump.jpope.org$request_uri?;
}
server {
listen 443 ssl;
server_name pump.jpope.org;

access_log /var/log/nginx/pump.access.log;
error_log /var/log/nginx/pump.error.log debug;

ssl_stapling on;
ssl_trusted_certificate /path/to/startssl.pem;
ssl_certificate /path/to/pump.crt;
ssl_certificate_key /path/to/pump.key;

client_max_body_size 6m;

keepalive_timeout 75 75;
gzip_vary off;

location / {
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_buffers 16 32k;
proxy_cache off;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_pass https://pumpbackend;
}
}

And there we go. My pump is a-pumping away.

One thing I'd like to note. Pumpio is designed to run as it is, a stand alone webapp that doesn't require a standard LAMP/LEMP setup. It isn't meant to run behind a proxy. It uses websockets via socket.io so, (and this is just my braindump, it could be way off) it could be made work behind a websocket proxy. Which with nginx 1.3.14 it should be possible however, I haven't been able to get it to work. None-the-less, if you are trying to run it behind a proxy and things don't work right, it's likely due to the proxy.

And those other items that I installed globally? It's possible that they will be included in a standard pump install in the future. Forever will restart the pump in case it craps out. I wrote an init script to start the pump that includes it as well. (and yes, my Arch webserver(s) are still using init scripts and not that damn systemd)

#!/bin/bash

. /etc/rc.conf
. /etc/rc.d/functions

daemon_name=pumpd
logfile=/var/log/pumpio/pumpd.log

case "$1" in
  start)
stat_busy "Starting pump.io Daemon"
forever /srv/http/pump.jpope.org/bin/pump &>$logfile &
if [ $? -gt 0 ]; then
  stat_fail
else
  add_daemon $daemon_name
  stat_done
fi
;;
  stop)
stat_busy "Stopping pump.io Daemon"
pkill -f /srv/http/pump.jpope.org/bin/pump &> /dev/null
if [ $? -gt 0 ]; then
  stat_fail
else
  rm_daemon $daemon_name
  stat_done
fi
;;
  restart)
$0 stop
sleep 5
$0 start
;;
  *)
echo "usage: $0 {start|stop|restart}"
esac
exit 0

The above could be adapted for other distros as needed. Having Forever installed globally allows for the init script to access it.

And for Bunyan, the logging generated can be piped through it for readability. I find myself using

$ tail -f /var/log/pumpio/pumpd.log|bunyan

quite a bit when I'm watching the logs.

There we have it, the current pump.jpope.org setup.


Pumped up comments:

Kyriakos Brastianos

Thank you for the great guide Jeremy! Bookmarked.

2013-04-03 22:24:38

LeSpocky

followed your guide to get pump.io running on our server, thanks for that! The address is https://pump.n39.eu/ and someone on IRC said, it’s not possible to log on with another server account. Could you confirm this for your installation?

2013-05-12 03:12:42

jpope

Yep. I can't remotely login to your pump. Nginx is giving me a "411 Length Required" error. I remember seeing this error (not pumpio related) a long time ago but, I don't remember how I fixed it. If you can update your nginx install to 1.4.1 (stable branch and has websocket support), you can base your nginx config on my current config for pump. I need to write an update to this blog post actually regarding this new config. :)

2013-05-13 00:02:49

Olivier Mehani

The “411 Length Required” is due to chunked encoding not being supported [0]. I had the same problem with an old Nginx as the frontend. I ended up grabbing the stable Debian package straight from Nginx (more recent than Debian), and recompile it to enable the HttpChunkinModule [1]. Doing this solved that issue.

[0] http://forum.nginx.org/read.php?2,5015,5025
[1] http://wiki.nginx.org/HttpChunkinModule

2013-05-16 07:56:26

jpope

Thanks Olivier. Seems this is how I fixed this issue when it came up for me.

2013-05-16 19:59:39

LeSpocky

We use Debian Wheezy which has nginx 1.2.1. I’ll have to talk to some guys in our hackerspace and then decide if I use a self compiled nginx or try the ChunkInModule. Both require building some stuff by ourselves. Maybe I ask the Debian guys if there will be a backport or maybe I use a proxy on another virtual machine. This will take some time, but thanks for your support. :-)

2013-05-17 03:19:18

Olivier Mehani

LeSpocky: You can grab Nginx’s own stable Debian packages [0] from

deb http://nginx.org/packages/debian/ codename nginx
deb-src http://nginx.org/packages/debian/ codename nginx

Then do an apt-get source nginx, and slightly modify the ./configure line in the debian/rules files to activate this module. The main difference between this package and Debian’s is that it doesn’t support the site-enabled thing by default, but you can easily enable it by adding include /etc/nginx/sites-enabled/*; in the configuration file.

[0] http://nginx.org/en/linux_packages.html#stable

2013-05-16 07:56:26


Have a response to this post? Please use this link.