Commit Graph

48 Commits

Author SHA1 Message Date
Mikael Nordfeldth
7e638cb8da OpportunisticQM can't run as subsystem qm 2013-11-20 21:46:16 +01:00
Mikael Nordfeldth
9400795a5f Cron split into Cronish and OpportunisticQM
/main/cron changed to /main/runqueue

The key-required functionality is not throughly tested yet.
2013-11-20 21:20:42 +01:00
Mikael Nordfeldth
0cd93c2761 Cron plugin added and now default queue handler
Generally the Cron plugin will run if there's still execution time for
1 second since starting the Action processing. If you want to change
this (such as disabling, 0 seconds, or maybe running bigger chunks,
for like 4 seconds) you can do this, where 'n' is time in seconds.

   addPlugin('Cron', array('secs_per_action', n));

Add 'rel_to_pageload'=>false to the array if you want to run the queue
for a certain amount of seconds _despite_ maybe already having run that
long in the previous parts of Action processing.

Perhaps you want to run the cron script remotely, using a machine capable
of background processing (or locally, to avoid running daemon processes),
simply do an HTTP GET request to the route /main/cron of your GNU social.
Setting secs_per_action to 0 in the plugin config will imply that you run
all your queue handling by calling /main/cron (which runs as long as it can).

/main/cron will output "0" if it has finished processing, "1" if it should
be called again to complete processing (because it ran out of time due to
PHP's max_execution_time INI setting).

The Cron plugin also runs events as close to hourly, daily and weekly
as you get, based on the opportunistic method of running whenever a user
visits the site. This means of course that the cron events should be as
fast as possible, not only to avoid delaying page load for users but
also to minimize the risk of running into PHP's max_execution_time. One
suggestion is to only use the events to add new queue items for later processing.

These events are called CronHourly, CronDaily, CronWeekly - however there
is no guarantee that all events will execute, so some kind of failsafe,
transaction-ish method must be implemented in the future.
2013-11-19 14:13:33 +01:00
Mikael Nordfeldth
5998825458 We haven't added the cron plugin yet 2013-11-19 14:02:53 +01:00
Mikael Nordfeldth
1a0e17fea6 Minor queue fixes, probably not very important. Preparing for cron 2013-11-13 18:15:21 +01:00
Mikael Nordfeldth
e7a4ccb7b5 Enable json-specified objects in queue_item frames 2013-10-28 12:44:06 +01:00
Mikael Nordfeldth
1744fec89f Array to string conversion in queuemanager logger 2013-09-24 00:40:23 +02:00
Zach Copley
6f0bd73e6c Squashed commit of the following:
Move OMB to a plugin

commit 75d21f00246bcc56d7f854936be1e28395e079a2
Merge: cea0199 d594d07
Author: Zach Copley <zach@status.net>
Date:   Fri Jul 15 11:16:54 2011 -0700

    Merge branch 'kill-omb2' of gitorious.org:~zcopley/statusnet/zcopleys-clone into kill-omb2

    * 'kill-omb2' of gitorious.org:~zcopley/statusnet/zcopleys-clone:
      Fix paths
      Oops, I left out the ability to authorize a token in ApiStatusNetOAuthDataStore
      Some odds and ends
      Remove omb stuff from queuemanager defaults
      Add check to make sure we're not untagging an OMB profile to OMB plugin
      Move some more subscription stuff and peopletag checks to OMB plugin
      Move some OMB-specific unsubscribe stuff to OMB plugin
      Finish removing libomb from core extlibs
      Fix more conflicts
      Fix queuing/queuehandling
      Move some stuff around; fix references
      Fix conflicts
      Move OMB-specific files to OMB plugin
      Move some stuff around; fix references
      Add OMB plugin README and rm references to OMB in mail StatusNet README
      Update paths
      Fix define
      Basic plugin finished
      Move OMB-specific files to OMB plugin
      Remove OMB stuff from router

commit cea019967f343042ebaea14b7bbb0d54289bcc1a
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 13 14:38:40 2011 -0700

    Fix paths

commit d412aa3c0ea0e21e65a72a16c7b9edd64ff373e1
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 12 18:17:06 2011 -0700

    Oops, I left out the ability to authorize a token in ApiStatusNetOAuthDataStore

commit b459c9f10ac283d6e774ef13f3293fc8a6948143
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 19:02:08 2011 -0700

    Some odds and ends

commit 895cfbfce58ffb3a05beebf48a90c549e00f1cce
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 19:01:23 2011 -0700

    Remove omb stuff from queuemanager defaults

commit b41b9e994f291ff83afb2460d9b37aee8ec1ec2b
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 18:46:44 2011 -0700

    Add check to make sure we're not untagging an OMB profile to OMB plugin

commit 94374d26ddd428dac8e4cd4541fd56db748c248b
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 18:34:20 2011 -0700

    Move some more subscription stuff and peopletag checks to OMB plugin

commit b91043b7820d5cd8b0ba4e9ee2a9d03c99248f11
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 16:37:25 2011 -0700

    Move some OMB-specific unsubscribe stuff to OMB plugin

commit d9430fe52975d9497b4a0d3d54da35b222e207ad
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 15:26:30 2011 -0700

    Finish removing libomb from core extlibs

commit bb6257eb85cc7ba392e91468c01503f51faeb989
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 13 12:47:32 2011 -0700

    Fix more conflicts

commit 3c760d0a4b4a083ae5fca2530d22aad5f4a9fdae
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:49:22 2011 -0700

    Fix queuing/queuehandling

commit ed635fa0c20e150673709c04ecc7f285d12e0ce2
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:29:35 2011 -0700

    Move some stuff around; fix references

commit cbc553a147941cad16e205a6b66ab4b32a5e3d3d
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 13 12:46:05 2011 -0700

    Fix conflicts

commit 5d77c81f75b57f5d5357d6b46d503650a4b3225d
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 19:10:38 2011 -0700

    Move OMB-specific files to OMB plugin

commit 2ed051dbce0ce9b44723b14922026849c39ed603
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:29:35 2011 -0700

    Move some stuff around; fix references

commit 8809b5e35b1aacb67d70ae3e55a43003b6f591b7
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:28:59 2011 -0700

    Add OMB plugin README and rm references to OMB in mail StatusNet README

commit 35ced4067c1915baca0b3e184f9533a91a951d2d
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 23:50:09 2011 -0700

    Update paths

commit 0ee5bafbce95fc9b8db98c1e828d33d26d08bc73
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 23:38:03 2011 -0700

    Fix define

commit e309dd22ffb9087d7fcf9180ede4f531dbd88c3c
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 23:30:43 2011 -0700

    Basic plugin finished

commit 00f1e930f27e080b04d1e82952f7886c84e01d97
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 19:10:38 2011 -0700

    Move OMB-specific files to OMB plugin

commit 39dcd031a79b49da0b4fe25f1594d2e406b5eb65
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 19:10:01 2011 -0700

    Remove OMB stuff from router

commit d594d071be1ec42518dd5465db61e01e7e8ec036
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 13 14:38:40 2011 -0700

    Fix paths

commit 48c1064b4b50e89cf51d2cab388f708f60601247
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 12 18:17:06 2011 -0700

    Oops, I left out the ability to authorize a token in ApiStatusNetOAuthDataStore

commit 1e1168978f38c31dbf0206b3493b2b6dcbe61589
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 19:02:08 2011 -0700

    Some odds and ends

commit ac43af2b497d8b9286c49a9469a1dff950e41650
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 19:01:23 2011 -0700

    Remove omb stuff from queuemanager defaults

commit 2471af2f8800515a3db544b3a186a18f3e8a43af
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 18:46:44 2011 -0700

    Add check to make sure we're not untagging an OMB profile to OMB plugin

commit df974646459ac6d5d97a40d008f1aab66f998226
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 18:34:20 2011 -0700

    Move some more subscription stuff and peopletag checks to OMB plugin

commit 8a1427b759e791c14a7a7a22128ba05f0b4b6d12
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 16:37:25 2011 -0700

    Move some OMB-specific unsubscribe stuff to OMB plugin

commit bd24220dbb5170af22ea0dea8a3062e6d1aeb6a2
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 6 15:26:30 2011 -0700

    Finish removing libomb from core extlibs

commit 4c3c6f1fabb0f2c92635ccc5e8f38db2293f5456
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 13 12:47:32 2011 -0700

    Fix more conflicts

commit db44deefd731a412685c5669c4c6fa69833de922
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:49:22 2011 -0700

    Fix queuing/queuehandling

commit ea2d84d2f3d518950d3aa1956ddc8f3a25ca55f3
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:29:35 2011 -0700

    Move some stuff around; fix references

commit 8ac3e010444b41bd9a78766f5e37e49dff023b45
Author: Zach Copley <zach@status.net>
Date:   Wed Jul 13 12:46:05 2011 -0700

    Fix conflicts

commit 0aad6e10e3637b3189a87b42c24c1d6de1b346bc
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 19:10:38 2011 -0700

    Move OMB-specific files to OMB plugin

commit d982d7076c5cb28c7b8e4b1dde8d07d7e58e278f
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:29:35 2011 -0700

    Move some stuff around; fix references

commit 4b9d39c93562ff4c45c37c940013e8b78197dec1
Author: Zach Copley <zach@status.net>
Date:   Tue Jul 5 15:28:59 2011 -0700

    Add OMB plugin README and rm references to OMB in mail StatusNet README

commit dab0fb6647a85e6835298496d7127a398b6b9293
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 23:50:09 2011 -0700

    Update paths

commit 2cb73dac8ad971f1545dcf6ba57746c777e232ef
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 23:38:03 2011 -0700

    Fix define

commit 6f226b18a059f175b1bdd3abcb8cb95eedc22ee7
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 23:30:43 2011 -0700

    Basic plugin finished

commit 7be304beaa0f39755c3978e0b852fde768950da4
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 19:10:38 2011 -0700

    Move OMB-specific files to OMB plugin

commit 5b30da01cfa2802d6e7a4a4a4f39b8043c54f472
Author: Zach Copley <zach@status.net>
Date:   Thu Jun 30 19:10:01 2011 -0700

    Remove OMB stuff from router
2011-07-15 12:13:57 -07:00
Zach Copley
b7d0746694 Merge branch '0.9.x' into 1.0.x
Conflicts:
	actions/confirmaddress.php
	actions/emailsettings.php
	actions/hostmeta.php
	actions/imsettings.php
	actions/login.php
	actions/profilesettings.php
	actions/showgroup.php
	actions/smssettings.php
	actions/urlsettings.php
	actions/userauthorization.php
	actions/userdesignsettings.php
	classes/Memcached_DataObject.php
	index.php
	lib/accountsettingsaction.php
	lib/action.php
	lib/common.php
	lib/connectsettingsaction.php
	lib/designsettings.php
	lib/personalgroupnav.php
	lib/profileaction.php
	lib/userprofile.php
	plugins/ClientSideShorten/ClientSideShortenPlugin.php
	plugins/Facebook/FBConnectSettings.php
	plugins/Facebook/FacebookPlugin.php
	plugins/NewMenu/NewMenuPlugin.php
	plugins/NewMenu/newmenu.css
2011-02-28 15:39:43 -08:00
Evan Prodromou
fb681990d9 Account moving is a background activity 2011-01-17 17:31:56 -05:00
Brion Vibber
26baad63f2 Merge branch '0.9.x' into 1.0.x 2010-12-22 15:25:38 -08:00
Evan Prodromou
044763cf06 move activity importing code to two different queuehandler classes 2010-12-17 13:12:17 -05:00
Brion Vibber
e89908f261 Merge branch '0.9.x' of git@gitorious.org:statusnet/mainline into 1.0.x
Conflicts:
	lib/channel.php
	scripts/imdaemon.php
2010-03-22 13:56:16 -07:00
Brion Vibber
441e52718e Background deletion of user accounts. Notices are deleted in chunks, then the user itself when they're all gone.
While deletion is in progress, the account is locked with the 'deleted' role, which disables all actions with rights control.

Todo:
* Pretty up the notice on the profile page about the pending delete. Show status?
* Possibly more thorough account disabling, such as disallowing all use for login and access.
* Improve error recovery; worst case is that an account gets left locked in 'deleted' state but the queue jobs have gotten dropped out. This would leave the username in use and any undeleted notices in place.
2010-03-15 16:08:00 -07:00
Craig Andrews
c187bf5597 Merge branch '0.9.x' into 1.0.x
Conflicts:
	EVENTS.txt
	db/statusnet.sql
	lib/queuemanager.php
2010-02-24 20:52:45 -05:00
Brion Vibber
59be4b8cae Merge branch 'testing' of gitorious.org:statusnet/mainline into 0.9.x 2010-02-24 15:47:51 -08:00
Brion Vibber
c36bdc1ba5 - break OMB profile update pings to a background queue
- add event hooks to profile update pings
- send Salmon pings with custom update-profile event to OStatus subscribees and groups (subscribers will see it on your next post)
- fix OStatus queues with overlong transport names, should work on DB queues now
- Ostatus_profile::notifyActivity() and ::notifyDeferred() now can take XML, Notice, or Activity for convenience
2010-02-24 20:36:36 +00:00
Craig Andrews
20d6a7caed Merge branch '0.9.x' into 1.0.x
Conflicts:
	lib/queuemanager.php
	lib/xmppmanager.php
	plugins/Xmpp/Fake_XMPP.php
	scripts/imdaemon.php
2010-02-16 13:15:09 -05:00
Brion Vibber
d5cbfe8071 Merge branch 'testing' into 0.9.x
Conflicts:
	lib/iomaster.php
2010-02-16 09:25:09 -08:00
Brion Vibber
c74aea589d Stomp queue restructuring for mass scalability:
- Multiplexing queues into groups and for multiple sites.
- Sharing vs breakout configurable per site and per queue via $config['queue']['breakout']
- Detect how many times a message is redelivered, discard if it's killed too many daemons
 - count configurable with $config['queue']['max_retries']
 - can dump the items to files in $config['queue']['dead_letter_dir']

Queue daemon memory & resource leak fixes:
- avoid unnecessary reconnections to memcached server (switch persistent connections back in on second initialization, assuming it's child process)
- monkey-patch for leaky .ini loads in DB_DataObject::databaseStructure() - was leaking 200k per active switch
- applied leak fixes to Status_network as well, using intermediate base Safe_DataObject for both it and Memcache_DataObject

Misc queue fixes:
- correct handling of child processes exiting due to signal termination instead of regular exit
- shutdown instead of infinite respawn loop if we're already past the soft memory limit at startup
- Added --all option for xmppdaemon... still opens one xmpp connection per site that has xmpp active

Cache updates:
- add Cache::increment() method with native support for memcached atomic increment
2010-02-16 09:16:51 -08:00
Brion Vibber
3d0c3f0577 Pull fix from testing branch: use new encoding funcs w/ stomp queues 2010-02-16 09:15:29 -08:00
Craig Andrews
32084e33a2 Merge branch '0.9.x' into 1.0.x
Conflicts:
	lib/queuemanager.php
2010-02-16 10:25:57 -05:00
Brion Vibber
122c8677b7 Merge branch 'testing' of gitorious.org:statusnet/mainline into 0.9.x 2010-02-12 11:18:35 -08:00
Brion Vibber
045797331c fix up hub queueing to work w/ stomp queues 2010-02-10 12:27:41 -08:00
Craig Andrews
914bc9f9c5 Merge branch '0.9.x' into 1.0.x
Conflicts:
	lib/queuemanager.php
2010-01-27 22:07:04 -05:00
Zach Copley
78079f34e2 Merge branch 'testing' into -1.9.x
* testing: (130 commits)
  HTTP auth provided is evaluated even if it's not required
  Rename rc3to09.sql to rc3torc4.sql to avoid confusion if we add a last-minute change after this!
  Add new oauth tables and modifications to 'consumer' table for rc4
  Centred leaderboard ad
  camelcase the uap param names
  move leaderboard to after the header
  Moved rectangle ad into aside and leaderboard to the right in header.
  Aligning wide skyscraper to the right instead of left
  CSS ids and classes fixed in UAPPlugin
  wrong height for rectangle in BlankAd
  Add the moved BlankAdPlugin
  make BlankAd dir and change to use a 1x1 image
  move BlankAdPlugin to its own dir
  Add BlankAdPlugin to test ad layout in different themes
  make uapplugin an abstract class
  move UAP plugin to core
  Lowercased switch cases in UAP Plugin
  Plugin for Universal Ad Package. Outputs four most widely used ad types.
  Add persistent:true property to Stomp messages so ActiveMQ doesn't decide to discard them even though persistence is enabled on the broker. :) (Thanks Aric!)
  quick fix: use common_path() on realtime update JS so it works with the new JS path code (will pull from main server for now)
  ...

Conflicts:
	actions/apioauthaccesstoken.php
	actions/apioauthauthorize.php
	actions/apioauthrequesttoken.php
	actions/editapplication.php
	actions/newapplication.php
	lib/apiauth.php
	lib/queuemanager.php
	lib/router.php
2010-01-27 14:27:22 -08:00
Craig Andrews
e9995b0f6a Create IM plugin, Pluginize XMPP, Create AIM plugin 2010-01-27 13:48:35 -05:00
Craig Andrews
5224c7d6c2 Merge branch '0.9.x' into 1.0.x 2010-01-27 13:41:02 -05:00
Brion Vibber
58be61b641 Control channel for queue daemons to request graceful shutdown, restart, or update to listen to a newly added or reconfigured site.
queuectl.php --update -s<site>
  queuectl.php --stop
  queuectl.php --restart

Default control channel is /topic/statusnet-control. For external utilities to send a site update ping direct to the queue server, connect via Stomp and send a message formatted thus:

  update:<nickname>

(Nickname here, *not* server hostname! The rest of the queues will be updated to use nicknames later.)

Note that all currently-connected queue daemons will get these notifications, including both queuedaemon.php and xmppdaemon.php. (XMPP will ignore site update requests for sites that it's not handling.)

Limitations:
* only implemented for stomp queue manager so far
* --update may not yet handle a changed server name properly
* --restart won't reload PHP code files that were already loaded at startup. Still need to stop and restart the daemons from 'outside' when updating code base.
2010-01-26 11:49:49 -08:00
Evan Prodromou
663e4e02a1 Merge branch 'master' into testing
Conflicts:
	lib/queuemanager.php
2010-01-25 18:13:09 -05:00
Evan Prodromou
b6aa1511eb Merge branch 'master' into 0.9.x 2010-01-25 18:10:59 -05:00
Evan Prodromou
e26a843caf Offload inbox updates to a queue handler to speed up posting online
Moved much of the writing that happens when posting a notice to a new
queuehandler, distribqueuehandler. This updates tags, groups, replies
and inboxes at queue time (or at Web time, if queues are disabled).

To make this work well, I had to break up the monolithic
Notice::blowCaches() and make cache blowing happen closer to where
data is updated.

Squashed commit of the following:

commit 5257626c62750ac4ac1db0ce2b71410c5711cfa3
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 14:56:41 2010 -0500

    slightly better handling of blowing tag memory cache

commit 8a22a3cdf6ec28685da129a0313e7b2a0837c9ef
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 01:42:56 2010 -0500

    change 'distribute' to 'distrib' so not too long for dbqueue

commit 7a063315b0f7fad27cb6fbd2bdd74e253af83e4f
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 01:39:15 2010 -0500

    change handle_notice() to handle() in distributqueuehandler

commit 1a39ccd28b9994137d7bfd21bb4f230546938e77
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 16:05:25 2010 -0500

    error with queuemanager

commit e6b3bb93f305cfd2de71a6340b8aa6fb890049b7
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 01:11:34 2010 -0500

    Blow memcache at different point rather than one big function for Notice class

commit 94d557cdc016187d1d0647ae1794cd94d6fb8ac8
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 00:48:44 2010 -0500

    Blow memcache at different point rather than one big function for Notice class

commit 1c781dd08c88a35dafc5c01230b4872fd6b95182
Author: Evan Prodromou <evan@status.net>
Date:   Wed Jan 20 08:54:18 2010 -0500

    move broadcasting and distributing to new queuehandler

commit da3e46d26b84e4f028f34a13fd2ee373e4c1b954
Author: Evan Prodromou <evan@status.net>
Date:   Wed Jan 20 08:53:12 2010 -0500

    Move distribution of notices to new distribute queue handler
2010-01-25 18:08:21 -05:00
Craig Andrews
23c0d663d6 Allow for instances as well as class names to be passed as queue handlers and iomanagers. 2010-01-23 01:03:41 -05:00
Craig Andrews
b34bbb0e80 Store serialized representations of queue items in the queue 2010-01-22 17:55:26 -05:00
Brion Vibber
c7507e7e9d XMPP queued output & initial retooling of DB queue manager to support non-Notice objects.
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues

Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
  This is based on Craig Andrews' work branch to generalize queues to take any
  object, but conservatively leaving out the serialization for now.
  Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
  thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
  directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
  each run through the event loop to watch for memory leaks

To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
  notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
  to queues.
* Find a way to improve interactive performance of the database-backed queue
  handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
  grouping used to split out the XMPP output queue is a bit awkward.

Conflicts:

	scripts/xmppdaemon.php
2010-01-22 12:52:36 -08:00
Brion Vibber
0e852def6a XMPP queued output & initial retooling of DB queue manager to support non-Notice objects.
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues

Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
  This is based on Craig Andrews' work branch to generalize queues to take any
  object, but conservatively leaving out the serialization for now.
  Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
  thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
  directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
  each run through the event loop to watch for memory leaks

To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
  notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
  to queues.
* Find a way to improve interactive performance of the database-backed queue
  handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
  grouping used to split out the XMPP output queue is a bit awkward.
2010-01-21 22:40:35 -08:00
Brion Vibber
598072468c --xmpp-only hack for queuedaemon.php to run separate queue daemon with only xmpp threads 2010-01-15 11:13:06 -08:00
Brion Vibber
58bc33850a temporary --skip-xmpp flag on queuedaemon.php, allows to run queue daemons but skip subscription to xmpp-based queues
(still working on making these behave gracefully when server is down)
2010-01-14 15:32:37 -08:00
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Evan Prodromou
ae883ceb9b change controlyourself.ca to status.net 2009-08-25 18:19:04 -04:00
Evan Prodromou
d35b2d3f3c change laconi.ca to status.net 2009-08-25 18:16:46 -04:00
Evan Prodromou
c8b8f07af1 change Laconica and Control Yourself to StatusNet in PHP files 2009-08-25 18:12:20 -04:00
Evan Prodromou
49eaa04b50 return singleton if initialized 2009-07-04 01:42:42 -04:00
Evan Prodromou
49c5c6f92b move handling code into queuemanager 2009-07-04 00:31:28 -04:00
Evan Prodromou
887d35cfc8 better queue manager detection, new method fail() 2009-07-01 12:09:18 -04:00
Evan Prodromou
e0bf8ad95b Add UnQueueManager for immediate handling
Perhaps it's a little precious, but I took out the switches in
util.php to determine what's supposed to be sent when, and made
a queuemanager class that will just do things when they're supposed to
be done.
2009-07-01 11:34:12 -04:00
Evan Prodromou
4c256a6d7e better hook variables for StartQueueManager 2009-07-01 11:09:58 -04:00
Evan Prodromou
e5b758dbbe start of queuemanager code 2009-06-28 14:38:31 -04:00