StatusNet Setup Notes

So now that I've established my StatusNet instance and have it running pretty well, I figured that it'd be a good time to document some of my experiences on setting it up. If you've read my previous entry when I actually started my instance, you may have noticed that I pretty well started it up quite hastily. I did have some experience in setting an instance up but, hadn't really messed with the configuration much aside from disabling some of the plugins. And that instance was (and is still) a private instance where the majority of it's activity is from my various machines checking in with uptime and update stats. For my public instance, I really didn't put much thought into how heavy, or how light, I was going to have it. Initially, I just wanted a SN instance that would work.

StatusNet Logo

In the end, my instance is probably overkill, especially considering it is just a single user instance. If you look through the active plugins on my instance, between having the Realtime w/ a Meteor server, Memcache, Twitterbridge, InfinateScroll, etc. some may say that it's even overkill for some smaller multiuser sites. However, I find the experience and performance to be quite good and my machine seems to be handling it just fine.

One other thing that I've found interesting is that on the day I decided to start my instance, I found out that quite a few other identicats had been or were in the process of setting up their own. Seems that there was some sort of line that was crossed sometime in October that pushed quite a few of us over. ...

Anyhoo.

Theme

I intitially stuck with the default install as far as plugins and whatnot. My main focus was initially on getting rid of the default theme. I needed (or maybe just wanted) the theme to match my current Drupal theme. So I started hacking on the default 'Neo' theme, changing the background & colors. I quickly found out that it was quite the pain in the ass to get the css to do what I wanted. The default theme is (at least with my limited wev dev experience) quite complicated with many pieces pulling from various files, seemingly from both the 'Neo' theme and the 'Base' theme. I ended up killing off my first attempt, made a copy of the 'Base' theme folder, pointed my config.php at my new theme and started hacking on that instead. I found that it was much easier to find all the pieces that I wanted to change going this route. And after an hour or so, I pretty well had the majority of my new theme lined out.

The majority of my theme work, turned into my porting of the Drupal Dartik theme as seen here.

Performance

Before I continue, I want to mention this post by @jbfavre on StatusNet optimization. There is a great amount of info there that I'll not repeat here so be sure to go read it. One note, any time you make a change to your config.php file, be sure to run the checkschema script. The script will go through the config and make sure all the required db entries are present.

php scripts/checkschema.php

Daemons

The next thing on my list was to improve the performance. The default install leaves you with a working but poorly performing site. This is due to all processes being done one after the other, or synchronous. There is a way to background some of these processes and when doing so, the performance difference is night and day. These daemons can be controlled by the two scripts in the scripts folder, appropriately named startdaemons.sh and stopdaemons.sh. The primary daemon that needs to be running to enable the asynchronous daemons, is the queuedaemon. Prior to starting the daemons, a couple of the scripts need to be checked first. Open the file scripts/getvaliddaemons.php in a file editor and verify that this line either exists or is not commented out:

$daemons[] = INSTALLDIR.'/scripts/queuedaemon.php';

At the same time, if you are not planning on using XMPP, you may want to comment out the imdaemon line.

The startdaemons.sh script will check for which daemons need to be running via the getvaliddaemons script. However, the stopdaemons.sh script only stops daemons that are listed within the stopdaemons script. So, in order to make sure that the queuedaemon is stopped properly, it'll need to be edited. Open scripts/stopdaemons.sh and add queuedaemon</code> (and <code>imdaemon` as well) to the list starting around line 26. Mine currently looks similar to this:

for f in ombhandler smshandler pinghandler \
    queuedaemon imdaemon \ 
    twitterhandler facebookhandler \ 
    twitterstatusfetcher synctwitterfriends \
    pluginhandler rsscloudhandler; do

Be sure to check @jbfavre's article for how to add the daemon settings in config.php. You should now be able to start/stop the daemons properly. I find that the daemons need to be kicked every now and then so I have a crontab entry to run the stopdaemons and startdaemons scripts overnight.

Logs

As with anything, having some logs to go through to find issues can be quite helpful. By adding the first line below to your config.php, you can add some of the debugging logging to your StatusNet log. Also, in order to keep the StatusNet log seperate from your syslog, add the second line, being sure to change the path where you want the log file to reside.

$config['site']['logdebug'] = true;
$config['site']['logfile'] = '/var/log/statusnet.log';

One thing to note though, while there is plenty of good info to find in the logs, having all of the debug entries leads to quite a few repeat entries. And all the noise sure makes it difficult to find the useful information. Once you get setup and running well, you may want to scale back on the entries to the logs. There is a LogFilter plugin that is fantastic for this. My current settings in config.php as follows, these will filter out the majority of the noise, leaving me with a useful log.

addPlugin('LogFilter', array( 'priority' => array(LOG_ERR => true, 
    LOG_INFO => true, 
    LOG_DEBUG => false),
    'regex' => array('/About to push/i' => false,
      '/twitter/i' => false,
      '/Successfully handled item/i' => false)
    ));

Subscriptions

Somewhere during the initial weeks of starting my instance, one of the other !feds was having issues subscribing to another. It mostly seemed to be OStatus related so there were quite a few of us looking at the different scripts within the plugins/OStatus/scripts/ directory as there are a few handy scripts located there. After much playing around with the various scripts, I now have a two-step that will scan the statusnet log for subscribed feeds with bad signatures. In my experience, whenever I'm not seeing notices from someone, it's due to this bad 'SHA-1' signature. My 'Bad Feed Two-Step' script can be found here. Edit the few path options and run.

I've found that the best way to keep on top of these for me is to rotate my SN logs daily and set a cronjob for the Two-Step script to run just prior to the rotation. To setup a rotate script, I just copied and edited /etc/logrotate.d/apache2. I also start/stop the daemons at this point as well. My rotate script currently looks like this.

So, what makes a feed go bad you ask? Good question. For me, the majority of the feeds that go bad are identi.ca based and they mostly seem to happen when identi.ca has some caching issues. That said, I've hardly had any bad feeds since identi.ca's datacenter move, maybe 1 or 2 a week.

Realtime

Another item that I really missed from my identi.ca days was the realtime updates. To the dismay of many, in order to have realtime, you need another server/process running to queue the notices. The StatusNet wiki has a list of options that can be used, located here. I ended up using Meteor and I've found it to work flawlessly so far. The initial setup was somewhat a pain but, once I got it working, it's been great. For a nice writeup on getting Meteor running, read @ryanweal's post on it link.

I have found that for my low traffic instance (compared to identi.ca), Meteor doesn't take up much resources when running. I initially wasn't going to mess with it, thinking that it'd be too heavy for a single user site. I'm glad I said to hell with it and gave it a shot.

A few other notes, observations and hacks

On initial setup, getting your URLs set how you want before you start subscribing/being subscribed too is quite important. Majority of people prefer to have the 'Fancy URLs' option on their instance. If you are running apache, be sure to have mod_rewrite active before setting up your instance. Also, prior to running install.php, copy the htaccess.sample to .htaccess and edit appropriately. Also, be sure to decide whether or not you want your instance running SSL or not. Any change in your feed URLs will cause some (or alot) of headache later on.

If you are running mysql with it's default settings, it could probably use some tweaks so that it runs better. For that, I have been using mysqltuner to provide me with some suggestions on what settings to tweak. Check your repos, it may be in there already (I know it is in Debian).

Enabling the queue daemons provides the greatest performance boost for a SN instance. The second greatest boost was setting the schemacheck to script (as noted in @jbfavre's post). Once you do this, it is even more important to run php scripts/checkschema.php after every change of the config.php file.

I've changed a few things in the layout of my instance by playing with a few of the core files. This really isn't recommended due to the fact that I'll most likely loose these changes whenever I update my install.

I increased the number of avatars in the 'following' - 'followers' - 'groups' sections on my profile page by editing lib/groupminilist.php and lib/profileminilist.php, line 36 in both files.

I've changed the primary navigation links on the top of the page so that it is consistent across all aspects of my site. I did this by editing lib/primarynav.php to include what links I wanted. I also moved the 'login' - 'admin' - 'settings' links to the bottom of the page by adding them to lib/secondarynav.php. Examples of my edits of each of these can be found here and here. Both of these are a little tricky only due to making certain that you nest the links properly in the if statements. These if statements determine whether or not someone can see them logged in or not.

A sample of my config.php can be found here. If you want to see all of the possible config options available run:

php scripts/setconfig.php -a

It may seem like quite a bit to run a StatusNet instance but, it really isn't. Above all, it's extremely fun to be able to tweak all these settings and put it to good use.

One last thing, I keep a page with some StatusNet links that I've found to be handy here.

!VivaLaFederation

Flag of the United Federation of StatusNet


Have a response to this post? Please use this link.