$config['site']['logperf'] = true; // to record & dump total hits of each type and the runtime to syslog
$config['site']['logperf_detail'] = true; // very verbose -- dump the individual cache keys and queries as they get used (may contain private info in some queries)
Seeing 180 cache gets on a timeline page seems not unusual currently; since these run in serial, even relatively small roundtrip times can add up heavily.
We should consider ways to reduce the number of round trips, such as more frequently storing compound objects or the output of processing in memcached.
Doing parallel multi-key lookups could also help by collapsing round-trip times, but might not be easy to fit into SN's object model. (For things like streams this should actually work pretty well -- grab the list, then when it's returned go grab all the individual items in parallel and return the list)
We can make a lot of HTTP requests from the server side. This change
adds some configuration options for using an HTTP proxy, which can
cache hits from multiple sites (good for status.net-like services, for example).
If a cache entry is dependent on the code that's running, upgrading
(or enabling/disabling plugins) can generate hard-to-track
inconsistencies.
This change adds a close-to-unique fingerprint of the running code to
some cache keys, so that if the fingerprint changes, the old values
are ignored and new values are used.
If the automated uniqueness fails, an administrator can add an extra
config value, $config['site']['build'], that's thrown into the key also.
If a cache entry is dependent on the code that's running, upgrading
(or enabling/disabling plugins) can generate hard-to-track
inconsistencies.
This change adds a close-to-unique fingerprint of the running code to
some cache keys, so that if the fingerprint changes, the old values
are ignored and new values are used.
If the automated uniqueness fails, an administrator can add an extra
config value, $config['site']['build'], that's thrown into the key also.
This option may be useful for intranet sites that don't have direct access to the internet, as they may be unable to successfully fetch those resources.
This will apply to *ALL* plugins in *ALL* languages, so should probably only be used when doing site customization...
You'd probably do:
$config['site']['locale_path'] = '/srv/awesome/data/locale';
$config['plugins']['locale_path'] = '/srv/awesome/data/locale';
with a structure like:
srv/
awesome/
data/
locale/
en/
LC_MESSAGES/
statusnet.po
OpenID.po
AnonymousFave.po
etc, all alongside each other. You could separate plugins from the core if you like.
Where locale files have not already been generated, you can build one for a plugin like so:
php scripts/update_po_templates.php --plugin=MyPlugin
and pull out the template file:
plugins/MyPlugin/locale/MyPlugin.pot
Edit that (make sure you at least set the CHARSET, probably to UTF-8) and save your customized .po
files into the structure as above, and use msgfmt to generate .mo files for final output.
Two prongs here:
* We attempt to enable SNI on the SSL stream context with the appropriate hostname... This requires PHP 5.3.2 and OpenSSL that supports the TLS extensions. Unfortunately this doesn't seem to be working in my testing.
* If set $config['http']['curl'] = true, we'll use the CURL backend if available. In my testing on Ubuntu 10.04, this works. No guarantees on other systems.
I'm not enabling CURL mode by default just yet; want to make sure there's no other surprises.
Currently only one custom theme may be uploaded per site, saved with the name 'custom' and stored into the local/themes subdirectory.
Administrators can upload a .ZIP archive containing a theme through the design admin panel; its contents are validated to ensure that only legit files are saved, and a 5M size quota is enforced.
Theme upload requires the zip extension for PHP; if not present, theme uploading is disabled by default.
Uploading and the custom CSS can be controlled via $config['theme_upload']['enabled'] and $config['custom_css']['enabled'].
Configurable directory/path/server for 'local' subdirectory (currently only as used for themes; local plugins not yet switched over)
Can set $config['local']['dir'] etc; not currently exposed in the admin panels.
Per-site directories on a separate themes server could be set up such as:
$config['local']['dir'] = '/path/to/themes/local/' . $_nickname;
$config['local']['server'] = 'themes.example.com';
$config['local']['path'] = '/local/' . $_nickname;
$config['local']['ssl'] = 'never';
Added a 2-second default timeout for XMLRPC/extended pings, configurable as [ping,timeout].
No longer repeating the entire ping section if we had an HTTP error during a submission.
For now, dropping the bad item and continuing on with others. (Todo: individual retry and cleaner discards of blacklisted broken-for-now sites.)
Users and administrators can set how long an URL can be before it's
shortened, and how long a notice can be before all its URLs are
shortened. They can also turn off shortening altogether.
Squashed commit of the following:
commit d136b39011
Author: Evan Prodromou <evan@status.net>
Date: Mon Apr 26 02:39:00 2010 -0400
use site and user settings to determine when to shorten URLs
commit 1e1c851ff3
Author: Evan Prodromou <evan@status.net>
Date: Mon Apr 26 02:38:40 2010 -0400
add a method to force shortening URLs
commit 4d29ca0b91
Author: Evan Prodromou <evan@status.net>
Date: Mon Apr 26 02:37:41 2010 -0400
static method for getting best URL shortening service
commit a9c6a3bace
Author: Evan Prodromou <evan@status.net>
Date: Mon Apr 26 02:37:11 2010 -0400
allow 0 in numeric entries in othersettings
commit 767ff2f7ec
Author: Evan Prodromou <evan@status.net>
Date: Mon Apr 26 02:36:46 2010 -0400
allow 0 or blank string in inputs
commit 1e21af42a6
Author: Evan Prodromou <evan@status.net>
Date: Mon Apr 26 02:01:11 2010 -0400
add more URL-shortening options to othersettings
commit 869a6be0f5
Author: Evan Prodromou <evan@status.net>
Date: Sat Apr 24 14:22:51 2010 -0400
move url shortener superclass to lib from plugin
commit 9c0c9863d5
Author: Evan Prodromou <evan@status.net>
Date: Sat Apr 24 14:20:28 2010 -0400
documentation and whitespace on UrlShortenerPlugin
commit 7a1dd5798f
Author: Evan Prodromou <evan@status.net>
Date: Sat Apr 24 14:05:46 2010 -0400
add defaults for URL shortening
commit d259c37ad2
Author: Evan Prodromou <evan@status.net>
Date: Sat Apr 24 13:40:10 2010 -0400
Add User_urlshortener_prefs
Add a table for URL shortener prefs, a corresponding class, and the
correct mumbo-jumbo in statusnet.ini to make everything work.
Conflicts:
actions/imsettings.php
lib/jabber.php
Made a quick attempt to merge the new JID validation into the XmppPlugin, have not had a chance to test that version live yet.
Should also move over the test cases.
I swapped the settings from negative to positive ($config['queue']['stomp_transactions'] = false rather than $config['queue']['stomp_no_transactions'] = true), gave them defaults (both on for best ActiveMQ experience), and added notes to the README about configuring them.
- switch 'en_US' to 'en', fixes the "admin panel switches to Arabic" bug
- tweak setting descriptions to clarify that most of the time we'll be using browser language
- add a backend switch to disable language detection (should this be exposed to ui?)
We've been making pretty crummy tag: URIs for a while. We should
continue to favor HTTP URIs, since it's nice to be able to discover
things about an object you've shared the ID of. Where that's not
possible, this makes nicer tag URIs.
All breakout queues that we're going to need to listen to now need to be explicitly listed in $config['queue']['breakout'].
Until XMPP is moved to component model, this setting will let the individual processes work with their own queues:
$config['queue']['breakout'][] = 'xmpp/xmppout/' . $config['site']['nickname'];
- Multiplexing queues into groups and for multiple sites.
- Sharing vs breakout configurable per site and per queue via $config['queue']['breakout']
- Detect how many times a message is redelivered, discard if it's killed too many daemons
- count configurable with $config['queue']['max_retries']
- can dump the items to files in $config['queue']['dead_letter_dir']
Queue daemon memory & resource leak fixes:
- avoid unnecessary reconnections to memcached server (switch persistent connections back in on second initialization, assuming it's child process)
- monkey-patch for leaky .ini loads in DB_DataObject::databaseStructure() - was leaking 200k per active switch
- applied leak fixes to Status_network as well, using intermediate base Safe_DataObject for both it and Memcache_DataObject
Misc queue fixes:
- correct handling of child processes exiting due to signal termination instead of regular exit
- shutdown instead of infinite respawn loop if we're already past the soft memory limit at startup
- Added --all option for xmppdaemon... still opens one xmpp connection per site that has xmpp active
Cache updates:
- add Cache::increment() method with native support for memcached atomic increment
Defaulting to only looking at last 90 days of activity, can be adjusted up or down.
$config['tag']['cutoff'] = 86400 * 90;
$config['popular']['cutoff'] = 86400 * 90;
Per-user and per-group tag clouds do not use the cutoff (and it doesn't help with indexing on them).
Adds a robots.txt file to the site root. Defaults defined by
'robotstxt' section of config. New events StartRobotsTxt and
EndRobotsTxt to let plugins add information. Probably not
useful if path is not /, but won't hurt anything, either.
If an array of multiple servers is put in $config['queue']['stomp_server'], enqueues will pick a random server to send to (failing over automatically if any are down).
Queue handling daemons connect all servers so they get events no matter where they were delivered.
In case of disconnection, daemons should now handle it gracefully and attempt to reconnect every 60 seconds or so, automatically resubscribing to all queues once it's back up.
Can put to 'native' failover for reads as well by disabling $config['stomp']['manual_failover'] = false; but this is untested and may explode in addition to requiring that your ActiveMQ cluster actually be set up to handle its own data distribution.
Additionally, can choose which queues to mark as persistent by setting $config['stomp']['persistent'] to an array of queue names.
* testing: (130 commits)
HTTP auth provided is evaluated even if it's not required
Rename rc3to09.sql to rc3torc4.sql to avoid confusion if we add a last-minute change after this!
Add new oauth tables and modifications to 'consumer' table for rc4
Centred leaderboard ad
camelcase the uap param names
move leaderboard to after the header
Moved rectangle ad into aside and leaderboard to the right in header.
Aligning wide skyscraper to the right instead of left
CSS ids and classes fixed in UAPPlugin
wrong height for rectangle in BlankAd
Add the moved BlankAdPlugin
make BlankAd dir and change to use a 1x1 image
move BlankAdPlugin to its own dir
Add BlankAdPlugin to test ad layout in different themes
make uapplugin an abstract class
move UAP plugin to core
Lowercased switch cases in UAP Plugin
Plugin for Universal Ad Package. Outputs four most widely used ad types.
Add persistent:true property to Stomp messages so ActiveMQ doesn't decide to discard them even though persistence is enabled on the broker. :) (Thanks Aric!)
quick fix: use common_path() on realtime update JS so it works with the new JS path code (will pull from main server for now)
...
Conflicts:
actions/apioauthaccesstoken.php
actions/apioauthauthorize.php
actions/apioauthrequesttoken.php
actions/editapplication.php
actions/newapplication.php
lib/apiauth.php
lib/queuemanager.php
lib/router.php
We have about 10-12 JavaScript pages per Web page. They usually
are based on the same server as the Web pages, but since they're
static files, it makes sense to offload them to a lite server that
handles static files well.
This commit lets you set a separate Javascript server and path for the
default Javascript code in StatusNet.
Squashed commit of the following:
commit 139d1622fdafe5ad00c820224416d9021efc3234
Author: Evan Prodromou <evan@status.net>
Date: Wed Jan 27 11:30:24 2010 -0500
modules that call htmloutputter::script() don't prescribe js/ path
commit c6ca3174af73efed55eaed5ff1e2a3bdc77d2d87
Author: Evan Prodromou <evan@status.net>
Date: Wed Jan 27 11:28:07 2010 -0500
configurable server and path for javascript files
queuectl.php --update -s<site>
queuectl.php --stop
queuectl.php --restart
Default control channel is /topic/statusnet-control. For external utilities to send a site update ping direct to the queue server, connect via Stomp and send a message formatted thus:
update:<nickname>
(Nickname here, *not* server hostname! The rest of the queues will be updated to use nicknames later.)
Note that all currently-connected queue daemons will get these notifications, including both queuedaemon.php and xmppdaemon.php. (XMPP will ignore site update requests for sites that it's not handling.)
Limitations:
* only implemented for stomp queue manager so far
* --update may not yet handle a changed server name properly
* --restart won't reload PHP code files that were already loaded at startup. Still need to stop and restart the daemons from 'outside' when updating code base.
New configuration options to define a single-user mode. This hides
most of the "community" pages, like the public timeline and groups.
The main user's timeline becomes the main page, and most other URLs
are changed.
Switching back and forth between 1-user and multi-user mode is
probably hazardous.
Squashed commit of the following:
commit d814aa5c92d14a27a12baba7893f3f8bf63f1d08
Author: Evan Prodromou <evan@status.net>
Date: Tue Jan 26 00:17:27 2010 -0500
don't show inbox and outbox in single-user mode
commit 47f19b9523a7015d4c6e460b73ea32c839e00aa1
Author: Evan Prodromou <evan@status.net>
Date: Tue Jan 26 00:15:22 2010 -0500
show correct URL for logo in single-user mode
commit 552010cffc33eadbc512ec5a67619dbc2015239a
Author: Evan Prodromou <evan@status.net>
Date: Tue Jan 26 00:15:06 2010 -0500
make singleuser its own config section
commit 786ab260a3ca172e57b555c75ca10946d8f258a1
Author: Evan Prodromou <evan@status.net>
Date: Tue Jan 26 00:05:19 2010 -0500
make single-user mode work
commit 5b21d7309b3a8dd5a4e0f29aea76f7897f1818b1
Author: Evan Prodromou <evan@status.net>
Date: Mon Jan 25 23:45:55 2010 -0500
add single-user mode
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.
This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues
Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
This is based on Craig Andrews' work branch to generalize queues to take any
object, but conservatively leaving out the serialization for now.
Table updater (preserves any existing queued items) in db/rc3to09.sql
Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
each run through the event loop to watch for memory leaks
To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
to queues.
* Find a way to improve interactive performance of the database-backed queue
handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
grouping used to split out the XMPP output queue is a bit awkward.
Conflicts:
scripts/xmppdaemon.php
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.
This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues
Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
This is based on Craig Andrews' work branch to generalize queues to take any
object, but conservatively leaving out the serialization for now.
Table updater (preserves any existing queued items) in db/rc3to09.sql
Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
each run through the event loop to watch for memory leaks
To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
to queues.
* Find a way to improve interactive performance of the database-backed queue
handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
grouping used to split out the XMPP output queue is a bit awkward.
Key changes:
* Initialization code moved from common.php to StatusNet class;
can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
the run loop is now managed by IoMaster run via scripts/queuedaemon.php
IoManager subclasses are woken to handle socket input or polling, and may
cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
QueueHandler class; no more need to add a daemon.
The new QueueDaemon runs from scripts/queuedaemon.php:
* This replaces most of the old *handler.php scripts; they've been refactored
to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
(defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon
Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
QueueManager [listen or poll] -> QueueHandler
XmppManager [ping & keepalive]
XmppConfirmManager [poll updates]
Todo:
* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases
Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
Upgrade notes:
* Index names have changed from hardcoded 'Identica_people' and 'Identica_notices' to use the database name and actual table names. Must reindex.
New events:
* GetSearchEngine to override default search engine class selection from plugins
New scripts:
* gen_config.php generates a sphinx.conf from database configuration (with theoretical support for status_network table, but it doesn't seem to be cleanly queriable right now without knowing the db setup info for that. Needs generalized support.)
* Replaced old sphinx-indexer.sh and sphinx-cron.sh with index_update.php
Other fixes:
* sphinx.conf.sample better matches our live config, skipping unused stopword list and using a more realistic indexer memory limit
Further notes:
* Probably doesn't work right with PostgreSQL yet; Sphinx can pull from PG but the extraction queries currently look like they use some MySQL-specific functions.
Caching support will be added in future work after unit tests have been added.
* extlib: add PEAR HTTP_Request2 0.4.1 alpha
* extlib: update PEAR Net_URL2 to 0.3.0 beta for HTTP_Request2 compatibility
* moved direct usage of CURL and file_get_contents to HTTPClient class, excluding external-sourced libraries
* adapted GeonamesPlugin for new HTTPResponse interface
Note some plugins haven't been fully tested yet.
Caching support will be added in future work after unit tests have been added.
* extlib: add PEAR HTTP_Request2 0.4.1 alpha
* extlib: update PEAR Net_URL2 to 0.3.0 beta for HTTP_Request2 compatibility
* moved direct usage of CURL and file_get_contents to HTTPClient class, excluding external-sourced libraries
Note some plugins haven't been tested yet.
