* respect count instead of listing all groups
* respect page parameter
* don't spew notice on undefined $id
* don't spew notice on undefined $group->homepage_url (dropped the element since there's nothing to go in it)
* respect count instead of listing all groups
* respect page parameter
* don't spew notice on undefined $id
* don't spew notice on undefined $group->homepage_url (dropped the element since there's nothing to go in it)
I swapped the settings from negative to positive ($config['queue']['stomp_transactions'] = false rather than $config['queue']['stomp_no_transactions'] = true), gave them defaults (both on for best ActiveMQ experience), and added notes to the README about configuring them.
Conflicts:
lib/attachmentlist.php
plugins/OStatus/classes/Ostatus_profile.php
Merge tried to delete things that it seems it shouldn't, very confusing order. Hope rest of the cherry-picking isn't a problem.
Moved the various classes used by the Activity class to their own
files. There were >10 classes in the same file, with around 1500 lines
in the file. Just too big.
This change makes autoloading work for these classes, so also removed
the hard require in lib/common.php.
RSS feeds have the format
<rss><channel><item/><item/><item/></channel></rss>. The element named
$rss was actually the <channel> element, so I renamed the variable so
I wouldn't hurt my head.
First steps to parsing RSS items as activities. RSS feeds don't seem
to have enough data to make good remote profiles, but this may work
with some "hints".
* Subscription::start was sometimes passing users instead of profiles to hooks, which broke OStatus subscription notifications; now normalizing to profiles for processing.
* H-card parsing would trigger a lot of PHP warnings and notices in hKit. Now suppressing warnings and notices for the duration of the call to keep them out of output when display_errors is on.
* H-card parsing would trigger a PHP fatal error if the source page was not well-formed XML and Tidy was not present on the system. Switched normalization to use the PHP DOM module which is always present, as we have no need for Tidy's extra features here.
* Trying to fetch avatars from Google profiles failed and triggered a PHP warning due to the relative URL not being resolved during h-card parsing. Now passing profile page URL into hKit by sneaking a <base> tag in while we normalize the HTML source.
* Profile pages without a "Link" header could trigger PHP notices due to a bad NULL -> array(NULL) conversion in LinkHeader::getLink(). Now checking that there was a return value before converting single return value into array.
We were double-unescaping for <content type="html">, turning <b> escaped chars into literal tags (which then may get removed entirely by the HTML scrubber).
For instance this was throwing an exception for DB_DataObject::staticGet when there's no match... definitely not what we want when all our code expects to get a nice null.
Example of this causing trouble: http://gitorious.org/statusnet/mainline/merge_requests/131
Revert "Don't attempt to retrieve the current user from the DB while processing a DB error"
This reverts commit 68347691b0.
Revert "Use PHP exceptions for PEAR error handling."
This reverts commit d8212977ce.
For instance this was throwing an exception for DB_DataObject::staticGet when there's no match... definitely not what we want when all our code expects to get a nice null.
Example of this causing trouble: http://gitorious.org/statusnet/mainline/merge_requests/131
Revert "Don't attempt to retrieve the current user from the DB while processing a DB error"
This reverts commit 68347691b0.
Revert "Use PHP exceptions for PEAR error handling."
This reverts commit d8212977ce.
While deletion is in progress, the account is locked with the 'deleted' role, which disables all actions with rights control.
Todo:
* Pretty up the notice on the profile page about the pending delete. Show status?
* Possibly more thorough account disabling, such as disallowing all use for login and access.
* Improve error recovery; worst case is that an account gets left locked in 'deleted' state but the queue jobs have gotten dropped out. This would leave the username in use and any undeleted notices in place.
Under MySQL, new tables will be created as InnoDB with UTF-8 (utf8/utf8_bin) same as core tables.
Existing plugin tables will have table engine and default charset/collation updated, and string columns will have charset updated, at checkschema time.
Switched from 'DESCRIBE' to INFORMATION_SCHEMA for pulling column information in order to get charset. A second hit to INFORMATION_SCHEMA is also needed to get table properties.
Indices were only being created at table creation time, which ain't so hot. Now also adding/dropping indices when they change.
Fixed up some schema defs in OStatus plugin that were a bit flaky, causing extra alter tables to be run.
TODO: Generalize this infrastructure a bit more up to base schema & pg schema classes.
I think this is a bug in Youtube's web server (sending chunked encoding of an empty body with a HEAD response, leaving the connection out of sync when it doesn't attempt to read a body) but the HTTP_Request2 library may need to be adjusted to watch out for that.
Keeps stray daemon subprocesses from floating around when we kill the parents via a signal!
Accomplished by opening a bidirectional pipe in the parent process; the children close out the writer end and keep the reader in their open sockets list. When the parent dies, the children see that the socket's been closed out and can perform an orderly shutdown.
The final whitespace should be dropped from the source messages after we've stabilized; trailing space is pretty unreliable to keep through translation tools and should be avoided. Use separator strings outside the messages!
OStatus plugin overrides these to allow using webfinger (user@example.com), profile URL (http://example.com/user) and bare profile URL (example.com/user) as arguments.
Gets Spanish, French, Russian etc UI localization working on Debian Lenny fresh installation set up in Spanish (so es_ES.UTF-8 is available but en_US.UTF-8 isn't).
The upstream class sets $this->basejid with host unconditionally, which wasn't previously an issue as the fulljid would always be filled in by the server at connect time before sending messages. With the new queued messaging, we need to make sure we've filled out $this->fulljid correctly without making a connection.
Now using $server if provided to build $this->basejid and $this->fulljid in the queued XMPP proxy class, so queued messages are sent correctly.
- switch 'en_US' to 'en', fixes the "admin panel switches to Arabic" bug
- tweak setting descriptions to clarify that most of the time we'll be using browser language
- add a backend switch to disable language detection (should this be exposed to ui?)
* 'testing' of gitorious.org:statusnet/mainline:
Using position relative only for the remote subscription in section
Added group subscription button to groups mini list
Added event hooks at the start and end of groups mini list
All 'connect' menu panels used to be optional, so Action tried to
figure out what the first item on the 'connect' menu should be.
This is no longer necessary because we have the non-optional OAuth
client connections panel now, which is not optional and can't be
turned off.
Also stripping id from foreign HTML messages (could interfere with UI) and disabled failing attachment popup for a.attachment links that don't have a proper id, so you can click through instead of getting an error.
Issues:
* any other links aren't marked and saved
* inconsistent behavior between local and remote attachments (local displays in lightbox, remote doesn't)
* if the enclosure'd object isn't referenced in the content, you won't be offered a link to it in our UI
We only need one author for user feeds: the user themselves. So, show
the user as the activity:subject, and don't repeat the same
activity:actor for every notice unnecessarily.
* 'testing' of gitorious.org:statusnet/mainline: (25 commits)
Fix a bunch of notice & warning-level messages that were breaking my inter-instance communications
more output in updateostatus.php
lost important fields when switching queries
show service debug info
pass listener URI into consumer for OMB
remove strict check on OMB exception strings
return correct HTTP status code for OMB errors
send smaller error pages for OMB API endpoints
Remove check for secret in token deletion on Subscription::cancel()
Better logging on bad token in subscription
Return empty array when no subscriptions to remote
drop tokens for OMB on unsubscribe
fix path for updateostatus.php
Script to convert OMB subscriptions to OStatus subscriptions
show service debug info
pass listener URI into consumer for OMB
remove strict check on OMB exception strings
return correct HTTP status code for OMB errors
send smaller error pages for OMB API endpoints
Remove check for secret in token deletion on Subscription::cancel()
...
In a federated system, "@nickname" is insufficient to uniquely
identify a user. However, it's a very convenient idiom. We need to
guess from context who 'nickname' refers to.
Previously, we were using the sender's profile (or what we knew about
them) as the only context. So, we assumed that they'd be mentioning to
someone they followed, or someone who followed them, or someone on
their own server.
Now, we include the notice information for context. We check to see if
the notice is a reply to another notice, and if the author of the
original notice has the nickname 'nickname', then the mention is
probably for them. Alternately, if the original notice mentions someone
with nickname 'nickname', then this notice is probably referring to
_them_.
Doing this kind of context sleuthing means we have to render the
content very late in the notice-saving process.
* 'testing' of gitorious.org:statusnet/mainline:
init_conversation.php script to copy old notice conversations into the conversation table
cache results of webfinger lookups
Parse an hcard for hints, if available
use new hcard method for webfinger
add hkit for hCard parsing
use new dedicated hcard method for Webfinger profile
Add an hcard action
* 'testing' of gitorious.org:statusnet/mainline:
OStatus: if no <link> available and no profileurl hint passed in, use object id if it's an HTTP(S) URL. Fixes profile link for Google accounts.
Tweak common_url_to_nickname to take the last path component; fixes pulling nicks from Google profile pages (path is "/profile/<nickname>")
Fix bug on subscribe/unsubscribe in profile lists. Bogus call to nonexisting profile->getProfile() was masked by DB_DataObject
A dedicated hcard action for users. Our profile page includes
an hcard, but it's so full of other hcards that it's ambiguous which
one is the "real" one. So, this one make sense for meaning, "This is
my hcard."
* 'testing' of gitorious.org:statusnet/mainline:
Not sure how this ended up in wrong-cased dir...
OStatus: fix remote groups to work with new user_groups/local_groups split.
Conflicts:
classes/User_group.php
- fix <activity:subject> generation so we get the profile info (what's available so far)
- use id instead of nickname for group join/leave forms so we can join/leave remote groups
while the rest of the groups UI remains limited to local groups
(plugins are responsible for making sure remote notifications and permission checks are done)
- fix remote notification when joining group through OStatus's remote subscribe form
* 'testing' of gitorious.org:statusnet/mainline:
Merge StatusNet core localization updates from 0.9.x branch
Fix update_po_templates.php to support the plural and context variants of _m() in plugins
Drop HTMLPurifier; we don't need its extra capabilities and we're already using htmLawed which is lighter-weight.
OStatus: handle update-profile Salmon pings
Revert "Updated jQuery Form Plugin from v2.17 to v2.36"
OStatus: disable HTMLPurify cache unless we've configured a writable path for it.
- add event hooks to profile update pings
- send Salmon pings with custom update-profile event to OStatus subscribees and groups (subscribers will see it on your next post)
- fix OStatus queues with overlong transport names, should work on DB queues now
- Ostatus_profile::notifyActivity() and ::notifyDeferred() now can take XML, Notice, or Activity for convenience
the shorthand that I've proposed at
http://microformats.org/wiki/geo-brainstorming#latitude_longitude_shorthand_and_geo_link
If anyone wants to pick up on where the discussion was left off or
get more implementation support by other sites and software, and be
recognized by parsers, I'd be happy to go back to the shorthand.
Because you know, it actually makes a lot of sense.
- added rel="ostatus:attention" links for group delivery
- added events for plugins to override group profile/permalink pages
- pulled Notice::saveGroups up to save-time so we can override;
it's relatively cheap and gives us a clean list of target
groups for distrib time even with customized delivery.
- fixed notice::getGroups to return group objects as expected
- added some doc on new parameters to Notice::saveNew
- 'groups' list of group IDs to push to in place of parsing
- messages that come in via PuSH and contain local group targets
are delivered to local group members
- messages that come in via PuSH and contain remote group targets
are delivered to local members of the remote group
Todo:
- handle group posts that only come through Salmon
- handle conflicts in case something comes in both through Salmon and PuSH
- better source verification
- need a cleaner interface to look up groups by URI
- need a way to handle remote groups with conflicting names
Combined the code that finds mentions of other profiles into one place.
common_find_mentions() finds mentions and calls hooks to allow
supplemental syntax for mentions (like OStatus).
common_linkify_mentions() links mentions.
common_linkify_mention() links a mention.
Notice::saveReplies() now uses common_find_mentions() instead of
trying to parse everything again.
I changed the way that tag: URIs are minted, so we now use the right
base. Ideally most of these would use HTTP URIs instead, but for
now at least they use the right base.
We've been making pretty crummy tag: URIs for a while. We should
continue to favor HTTP URIs, since it's nice to be able to discover
things about an object you've shared the ID of. Where that's not
possible, this makes nicer tag URIs.
The subs_* functions in subs.php have made a lot of assumptions
about users versus profiles. I've refactored the functions to
be methods of the Subscription class instead, and to use Profile
objects throughout.
Some of the checks for blocks or existing subscriptions depended
on users or profiles, so I've moved those methods around a bit.
I've left stubs for the subs_* functions until we get time to replace
them.
All breakout queues that we're going to need to listen to now need to be explicitly listed in $config['queue']['breakout'].
Until XMPP is moved to component model, this setting will let the individual processes work with their own queues:
$config['queue']['breakout'][] = 'xmpp/xmppout/' . $config['site']['nickname'];
- Multiplexing queues into groups and for multiple sites.
- Sharing vs breakout configurable per site and per queue via $config['queue']['breakout']
- Detect how many times a message is redelivered, discard if it's killed too many daemons
- count configurable with $config['queue']['max_retries']
- can dump the items to files in $config['queue']['dead_letter_dir']
Queue daemon memory & resource leak fixes:
- avoid unnecessary reconnections to memcached server (switch persistent connections back in on second initialization, assuming it's child process)
- monkey-patch for leaky .ini loads in DB_DataObject::databaseStructure() - was leaking 200k per active switch
- applied leak fixes to Status_network as well, using intermediate base Safe_DataObject for both it and Memcache_DataObject
Misc queue fixes:
- correct handling of child processes exiting due to signal termination instead of regular exit
- shutdown instead of infinite respawn loop if we're already past the soft memory limit at startup
- Added --all option for xmppdaemon... still opens one xmpp connection per site that has xmpp active
Cache updates:
- add Cache::increment() method with native support for memcached atomic increment
* skip unnecessary unsubscribes on graceful shutdown -- takes a long time for many queues, slows down our restarts when hitting graceful mem limit
* fix control channel (was broken when we switched to support multiple queue servers)
* detection of group feeds is currently a nasty hack based on presence of '/groups/' in URL -- should use some property on the feed?
* listing for the remote group is kinda cruddy; needs to be named more cleanly
* still need to establish per-author profiles (easier once we have the updated Atom code in)
* group delivery probably not right yet
* saving of group messages still triggering some weird behavior
Added support for since_id and max_id on group timeline feeds as a free extra. Enjoy!
* Treat linkless feed posts as status updates; drop the "New post:" prefix and quotes on them.
* Use stable user IDs for atom/rss2 feed links instead of unstable nicknames
* Pull Atom feed preferentially when subscribing -- can now put the remote user's profile page straight into the feed subscription form and get to the right place.
* Clean up naming for push endpoints
* renamed FeedSub plugin to OStatus
* now setting avatar on subscriptions
* general fixes for subscription
* integrated PuSH hub to handle only user timelines on canonical ID url; sends updates directly
* set $config['feedsub']['nohub'] = true to test w/ foreign feeds that don't have hubs (won't actually receive updates though)
* a few bits of code documentation
* HMAC support for verified distributions (safest if sub setup is on HTTPS)
And a couple core changes:
* minimizing HTML output for exceptions in API requests to aid in debugging
* fix for rel=self link in apitimelineuser when id given
This does not not yet include any of the individual subscription management (Salmon notifications for sub/unsub, etc) nor a nice UI for user subscriptions.
Needs some further cleanup to treat posts as status updates instead of link references.
Defaulting to only looking at last 90 days of activity, can be adjusted up or down.
$config['tag']['cutoff'] = 86400 * 90;
$config['popular']['cutoff'] = 86400 * 90;
Per-user and per-group tag clouds do not use the cutoff (and it doesn't help with indexing on them).
Adds a robots.txt file to the site root. Defaults defined by
'robotstxt' section of config. New events StartRobotsTxt and
EndRobotsTxt to let plugins add information. Probably not
useful if path is not /, but won't hurt anything, either.
If an array of multiple servers is put in $config['queue']['stomp_server'], enqueues will pick a random server to send to (failing over automatically if any are down).
Queue handling daemons connect all servers so they get events no matter where they were delivered.
In case of disconnection, daemons should now handle it gracefully and attempt to reconnect every 60 seconds or so, automatically resubscribing to all queues once it's back up.
Can put to 'native' failover for reads as well by disabling $config['stomp']['manual_failover'] = false; but this is untested and may explode in addition to requiring that your ActiveMQ cluster actually be set up to handle its own data distribution.
Additionally, can choose which queues to mark as persistent by setting $config['stomp']['persistent'] to an array of queue names.
Move definition of NICKNAME_FMT above plugin initialization but below loading of Validate package.
A merge error when refactoring setup lead to this not being defined yet when plugins were initialized, causing the router setup in MapstractionPlugin which tried to use this constant to fail.
Result was bogus links and if you hit the URL directly the action would be "unrecognized".
* testing: (130 commits)
HTTP auth provided is evaluated even if it's not required
Rename rc3to09.sql to rc3torc4.sql to avoid confusion if we add a last-minute change after this!
Add new oauth tables and modifications to 'consumer' table for rc4
Centred leaderboard ad
camelcase the uap param names
move leaderboard to after the header
Moved rectangle ad into aside and leaderboard to the right in header.
Aligning wide skyscraper to the right instead of left
CSS ids and classes fixed in UAPPlugin
wrong height for rectangle in BlankAd
Add the moved BlankAdPlugin
make BlankAd dir and change to use a 1x1 image
move BlankAdPlugin to its own dir
Add BlankAdPlugin to test ad layout in different themes
make uapplugin an abstract class
move UAP plugin to core
Lowercased switch cases in UAP Plugin
Plugin for Universal Ad Package. Outputs four most widely used ad types.
Add persistent:true property to Stomp messages so ActiveMQ doesn't decide to discard them even though persistence is enabled on the broker. :) (Thanks Aric!)
quick fix: use common_path() on realtime update JS so it works with the new JS path code (will pull from main server for now)
...
Conflicts:
actions/apioauthaccesstoken.php
actions/apioauthauthorize.php
actions/apioauthrequesttoken.php
actions/editapplication.php
actions/newapplication.php
lib/apiauth.php
lib/queuemanager.php
lib/router.php
We have about 10-12 JavaScript pages per Web page. They usually
are based on the same server as the Web pages, but since they're
static files, it makes sense to offload them to a lite server that
handles static files well.
This commit lets you set a separate Javascript server and path for the
default Javascript code in StatusNet.
Squashed commit of the following:
commit 139d1622fdafe5ad00c820224416d9021efc3234
Author: Evan Prodromou <evan@status.net>
Date: Wed Jan 27 11:30:24 2010 -0500
modules that call htmloutputter::script() don't prescribe js/ path
commit c6ca3174af73efed55eaed5ff1e2a3bdc77d2d87
Author: Evan Prodromou <evan@status.net>
Date: Wed Jan 27 11:28:07 2010 -0500
configurable server and path for javascript files
queuectl.php --update -s<site>
queuectl.php --stop
queuectl.php --restart
Default control channel is /topic/statusnet-control. For external utilities to send a site update ping direct to the queue server, connect via Stomp and send a message formatted thus:
update:<nickname>
(Nickname here, *not* server hostname! The rest of the queues will be updated to use nicknames later.)
Note that all currently-connected queue daemons will get these notifications, including both queuedaemon.php and xmppdaemon.php. (XMPP will ignore site update requests for sites that it's not handling.)
Limitations:
* only implemented for stomp queue manager so far
* --update may not yet handle a changed server name properly
* --restart won't reload PHP code files that were already loaded at startup. Still need to stop and restart the daemons from 'outside' when updating code base.
New configuration options to define a single-user mode. This hides
most of the "community" pages, like the public timeline and groups.
The main user's timeline becomes the main page, and most other URLs
are changed.
Switching back and forth between 1-user and multi-user mode is
probably hazardous.
Squashed commit of the following:
commit d814aa5c92d14a27a12baba7893f3f8bf63f1d08
Author: Evan Prodromou <evan@status.net>
Date: Tue Jan 26 00:17:27 2010 -0500
don't show inbox and outbox in single-user mode
commit 47f19b9523a7015d4c6e460b73ea32c839e00aa1
Author: Evan Prodromou <evan@status.net>
Date: Tue Jan 26 00:15:22 2010 -0500
show correct URL for logo in single-user mode
commit 552010cffc33eadbc512ec5a67619dbc2015239a
Author: Evan Prodromou <evan@status.net>
Date: Tue Jan 26 00:15:06 2010 -0500
make singleuser its own config section
commit 786ab260a3ca172e57b555c75ca10946d8f258a1
Author: Evan Prodromou <evan@status.net>
Date: Tue Jan 26 00:05:19 2010 -0500
make single-user mode work
commit 5b21d7309b3a8dd5a4e0f29aea76f7897f1818b1
Author: Evan Prodromou <evan@status.net>
Date: Mon Jan 25 23:45:55 2010 -0500
add single-user mode
Moved much of the writing that happens when posting a notice to a new
queuehandler, distribqueuehandler. This updates tags, groups, replies
and inboxes at queue time (or at Web time, if queues are disabled).
To make this work well, I had to break up the monolithic
Notice::blowCaches() and make cache blowing happen closer to where
data is updated.
Squashed commit of the following:
commit 5257626c62750ac4ac1db0ce2b71410c5711cfa3
Author: Evan Prodromou <evan@status.net>
Date: Mon Jan 25 14:56:41 2010 -0500
slightly better handling of blowing tag memory cache
commit 8a22a3cdf6ec28685da129a0313e7b2a0837c9ef
Author: Evan Prodromou <evan@status.net>
Date: Mon Jan 25 01:42:56 2010 -0500
change 'distribute' to 'distrib' so not too long for dbqueue
commit 7a063315b0f7fad27cb6fbd2bdd74e253af83e4f
Author: Evan Prodromou <evan@status.net>
Date: Mon Jan 25 01:39:15 2010 -0500
change handle_notice() to handle() in distributqueuehandler
commit 1a39ccd28b9994137d7bfd21bb4f230546938e77
Author: Evan Prodromou <evan@status.net>
Date: Mon Jan 25 16:05:25 2010 -0500
error with queuemanager
commit e6b3bb93f305cfd2de71a6340b8aa6fb890049b7
Author: Evan Prodromou <evan@status.net>
Date: Mon Jan 25 01:11:34 2010 -0500
Blow memcache at different point rather than one big function for Notice class
commit 94d557cdc016187d1d0647ae1794cd94d6fb8ac8
Author: Evan Prodromou <evan@status.net>
Date: Mon Jan 25 00:48:44 2010 -0500
Blow memcache at different point rather than one big function for Notice class
commit 1c781dd08c88a35dafc5c01230b4872fd6b95182
Author: Evan Prodromou <evan@status.net>
Date: Wed Jan 20 08:54:18 2010 -0500
move broadcasting and distributing to new queuehandler
commit da3e46d26b84e4f028f34a13fd2ee373e4c1b954
Author: Evan Prodromou <evan@status.net>
Date: Wed Jan 20 08:53:12 2010 -0500
Move distribution of notices to new distribute queue handler
Previously, messages once delivered would just get stuck in the queue seemingly forever if they never got ACKed.
Note this could lead to partial duplication, for instance if the OMB or Twitter queue handlers die after 1/2 of the outgoing sends.
Recommendations:
* catch exceptions more aggressively within queue handlers (so only PHP fatal errors are likely to kill in the middle)
* for processing that involves sending to multiple clients, consider a second queue similar to the XMPP output, eg for OMB:
- first queue gets delivery list and builds message data, enqueueing it for each target address
- second queue can handle each individual outgoing message (and attempt redelivery etc separately)
This would also protect better against a recurring error preventing delivery in the second part, and could spread out any slow sends over multiple threads.
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.
This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues
Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
This is based on Craig Andrews' work branch to generalize queues to take any
object, but conservatively leaving out the serialization for now.
Table updater (preserves any existing queued items) in db/rc3to09.sql
Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
each run through the event loop to watch for memory leaks
To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
to queues.
* Find a way to improve interactive performance of the database-backed queue
handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
grouping used to split out the XMPP output queue is a bit awkward.
Conflicts:
scripts/xmppdaemon.php
Previously, messages once delivered would just get stuck in the queue seemingly forever if they never got ACKed.
Note this could lead to partial duplication, for instance if the OMB or Twitter queue handlers die after 1/2 of the outgoing sends.
Recommendations:
* catch exceptions more aggressively within queue handlers (so only PHP fatal errors are likely to kill in the middle)
* for processing that involves sending to multiple clients, consider a second queue similar to the XMPP output, eg for OMB:
- first queue gets delivery list and builds message data, enqueueing it for each target address
- second queue can handle each individual outgoing message (and attempt redelivery etc separately)
This would also protect better against a recurring error preventing delivery in the second part, and could spread out any slow sends over multiple threads.
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.
This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues
Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
This is based on Craig Andrews' work branch to generalize queues to take any
object, but conservatively leaving out the serialization for now.
Table updater (preserves any existing queued items) in db/rc3to09.sql
Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
each run through the event loop to watch for memory leaks
To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
to queues.
* Find a way to improve interactive performance of the database-backed queue
handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
grouping used to split out the XMPP output queue is a bit awkward.
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.
This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues
Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
This is based on Craig Andrews' work branch to generalize queues to take any
object, but conservatively leaving out the serialization for now.
Table updater (preserves any existing queued items) in db/rc3to09.sql
Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
each run through the event loop to watch for memory leaks
To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
to queues.
* Find a way to improve interactive performance of the database-backed queue
handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
grouping used to split out the XMPP output queue is a bit awkward.
- NOTICE_INBOX_SOURCE_* constants moved to common.php since Notice_inbox.php not always loaded
- fixed typo in User::staticGet() call which caused user #1 to receive messages once for each subscriber instead of for him/herself
- 'continue' -> 'continue 2' inside switch() statement to fix loop escape (PHP considers switch() a looping construct for break & continue)
Key changes:
* Initialization code moved from common.php to StatusNet class;
can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
the run loop is now managed by IoMaster run via scripts/queuedaemon.php
IoManager subclasses are woken to handle socket input or polling, and may
cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
QueueHandler class; no more need to add a daemon.
The new QueueDaemon runs from scripts/queuedaemon.php:
* This replaces most of the old *handler.php scripts; they've been refactored
to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
(defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon
Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
QueueManager [listen or poll] -> QueueHandler
XmppManager [ping & keepalive]
XmppConfirmManager [poll updates]
Todo:
* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases
Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
