Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.
This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues
Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
This is based on Craig Andrews' work branch to generalize queues to take any
object, but conservatively leaving out the serialization for now.
Table updater (preserves any existing queued items) in db/rc3to09.sql
Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
each run through the event loop to watch for memory leaks
To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
to queues.
* Find a way to improve interactive performance of the database-backed queue
handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
grouping used to split out the XMPP output queue is a bit awkward.
* need lower case in some places
* update target path
Not in production yet at translatewiki.net - need some Translate extension code changes to get YAML config support for Gettext first.
Key changes:
* Initialization code moved from common.php to StatusNet class;
can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
the run loop is now managed by IoMaster run via scripts/queuedaemon.php
IoManager subclasses are woken to handle socket input or polling, and may
cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
QueueHandler class; no more need to add a daemon.
The new QueueDaemon runs from scripts/queuedaemon.php:
* This replaces most of the old *handler.php scripts; they've been refactored
to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
(defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon
Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
QueueManager [listen or poll] -> QueueHandler
XmppManager [ping & keepalive]
XmppConfirmManager [poll updates]
Todo:
* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases
Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
* Mostly punctuation updates so that the same message is used consistently in all of StatusNet.
* Some cases of "Title Case" removed, because that does not appear to be used consistently.
Since we added locations to the database, some users may have
location strings in their profiles but not structured locations. This
script updates the locations for single users or for all users.
Used to query user's emails.
Mostly used for administration, to see if a user requesting something is who they say.
Also, some people assume that the admin knows this data, and says things like:
"If you could do _____ with the account connected to this email".
It'd be nice if we could do that without raw SQL.
* master: (67 commits)
Ticket 2038: fix bad bug tracker link
Fix regression in group posting: bug introduced in commit 1319002e15. Need to use actual profile object rather than an id on a variable that doesn't exist when checking blocks :D
Log database errors when saving notice_inbox entries
Drop the username from the log id for now; seems to trigger an error loop in some circumstances
request id on logs... pid + random id per web request + username + method + url
Add OpenID ini info back into statusnet.ini as a stopgap until we can
Some changes to the OpenID DataObjects to make them emit the exact same
OpenID plugin should set 'user_openid.display' as unique key
Remove relationship: user_openid.user_id -> user.id. I don't think this
Have OpenID plugin DataObjects emit their own .ini info
Revert "Allow plugin DB_DataObject classes to not have to use the .ini file by overriding keys(), table(), and sequenceKey() for them"
Catch and report exceptions from notice_to_omb_notice() instead of letting the OMB queue handler die.
Fix regression in remote subscription; added hasRole() shadow method on Remote_profile.
Fix fatal error on OMB subscription for first-timers
Remove annoying log msg
Drop error message on setlocale() failure; this is harmless, since we actually have a working locale set up.
Catch uncaught exception
Fixed bug where reply-sync bit wasn't getting saved
Forgot to render the nav menu when on FB Connect login tab
Facebook plugin no longer takes over Login and Connect settings nav menus
...
Conflicts:
db/08to09_pg.sql
db/statusnet_pg.sql
locale/pt_BR/LC_MESSAGES/statusnet.mo
plugins/Mapstraction/MapstractionPlugin.php
* 0.9-release: (874 commits)
Removed call to NewDirectMessage() until IE return is fixed i.e.,
Don't show flag user button your own profile
Fixed HXR response for flag user
Using the right form class name
Using common_redirect
Left a form_data class of a <ul> in the user admin panel
Added validation to fields in user admin panel
Added a user admin panel
Added mobile logos for default and identica themes
Changed gif to png
Changed this to action. THANKS zach!
Doing content negotiation only once
Add execute bit to pingqueuehandler
Localisation updates for !StatusNet from !translatewiki.net
Use the browser's geolocation API to set the location on the notice form
Add geometa library, and include it.
Add location form elements to the noticeform, and save their values on submission
Use the $user object nickname, as login name doesnt have to == nickname anymore with plugins such as ldap/etc
Revert "Re added NICKNAME_FMT constant to router.php."
Moved most path and server settings to a new paths admin panel
...
Conflicts:
js/util.js
locale/it_IT/LC_MESSAGES/statusnet.mo
locale/mk_MK/LC_MESSAGES/statusnet.mo
locale/mk_MK/LC_MESSAGES/statusnet.po
locale/pt_BR/LC_MESSAGES/statusnet.mo
locale/vi_VN/LC_MESSAGES/statusnet.mo
plugins/InfiniteScroll/infinitescroll.js
plugins/Realtime/realtimeupdate.js
Upgrade notes:
* Index names have changed from hardcoded 'Identica_people' and 'Identica_notices' to use the database name and actual table names. Must reindex.
New events:
* GetSearchEngine to override default search engine class selection from plugins
New scripts:
* gen_config.php generates a sphinx.conf from database configuration (with theoretical support for status_network table, but it doesn't seem to be cleanly queriable right now without knowing the db setup info for that. Needs generalized support.)
* Replaced old sphinx-indexer.sh and sphinx-cron.sh with index_update.php
Other fixes:
* sphinx.conf.sample better matches our live config, skipping unused stopword list and using a more realistic indexer memory limit
Further notes:
* Probably doesn't work right with PostgreSQL yet; Sphinx can pull from PG but the extraction queries currently look like they use some MySQL-specific functions.
Uses readline for line input editing if available; falls back to bash+readline if not native, and takes fgets() in worst case.
Currently a bit awkward in that each input line is parsed separately, so loops and function defs have to be squished to one line.
Uses readline for line input editing if available; falls back to bash+readline if not native, and takes fgets() in worst case.
Currently a bit awkward in that each input line is parsed separately, so loops and function defs have to be squished to one line.
scripts/update_translations.php now pulls updated files from TranslateWiki and merges them to an updated master.
Note that the .po files exported from TranslateWiki do lose some of the old manual comments for now.
Caching support will be added in future work after unit tests have been added.
* extlib: add PEAR HTTP_Request2 0.4.1 alpha
* extlib: update PEAR Net_URL2 to 0.3.0 beta for HTTP_Request2 compatibility
* moved direct usage of CURL and file_get_contents to HTTPClient class, excluding external-sourced libraries
* adapted GeonamesPlugin for new HTTPResponse interface
Note some plugins haven't been fully tested yet.
Caching support will be added in future work after unit tests have been added.
* extlib: add PEAR HTTP_Request2 0.4.1 alpha
* extlib: update PEAR Net_URL2 to 0.3.0 beta for HTTP_Request2 compatibility
* moved direct usage of CURL and file_get_contents to HTTPClient class, excluding external-sourced libraries
Note some plugins haven't been tested yet.
* 0.9.x: (247 commits)
Added in credits.
Use site's name for basic auth realm
Make apigroupcreate.php pass phpcs
Took out some unnecessary intializations
Implemented create group api
CamelCase all function names in the API code
These same params are used in most API actions; moved to base API class
Missed some of the references to the old TwitterApiAction - removed
Remove more redundant $formats
Remove dead code
Move all basic auth output and processing to base classes
$format is used by every API action. Set it in the base class.
Delete action/api.php and rename lib/twitterapi.php to lib/api.php
New actions for blocks via API
fix FBConnect so it doesn't muffle EndPrimaryNav
don't write session if it's unchanged
Fixed facebook connect primary nav to hide search option when site is private and user is not logged in
Fixed facebook connect primary nav to obey sms/twitter/openid settings
Fixed facebook connect login nav to obey openid settings
Fixed facebook connect nav to obey sms/twitter disabled
...
