gnu-social/lib/dbqueuemanager.php
Brion Vibber 26fdf0c9d2 XMPP queued output & initial retooling of DB queue manager to support non-Notice objects.
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues

Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
  This is based on Craig Andrews' work branch to generalize queues to take any
  object, but conservatively leaving out the serialization for now.
  Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
  thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
  directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
  each run through the event loop to watch for memory leaks

To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
  notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
  to queues.
* Find a way to improve interactive performance of the database-backed queue
  handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
  grouping used to split out the XMPP output queue is a bit awkward.
2010-01-21 16:42:50 -08:00

151 lines
4.5 KiB
PHP

<?php
/**
* StatusNet, the distributed open-source microblogging tool
*
* Simple-minded queue manager for storing items in the database
*
* PHP version 5
*
* LICENCE: This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as published by
* the Free Software Foundation, either version 3 of the License, or
* (at your option) any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*
* @category QueueManager
* @package StatusNet
* @author Evan Prodromou <evan@status.net>
* @author Brion Vibber <brion@status.net>
* @copyright 2009-2010 StatusNet, Inc.
* @license http://www.fsf.org/licensing/licenses/agpl-3.0.html GNU Affero General Public License version 3.0
* @link http://status.net/
*/
class DBQueueManager extends QueueManager
{
/**
* Saves an object reference into the queue item table.
* @return boolean true on success
* @throws ServerException on failure
*/
public function enqueue($object, $queue)
{
$qi = new Queue_item();
$qi->frame = $this->encode($object);
$qi->transport = $queue;
$qi->created = common_sql_now();
$result = $qi->insert();
if (!$result) {
common_log_db_error($qi, 'INSERT', __FILE__);
throw new ServerException('DB error inserting queue item');
}
$this->stats('enqueued', $queue);
return true;
}
/**
* Poll every 10 seconds for new events during idle periods.
* We'll look in more often when there's data available.
*
* @return int seconds
*/
public function pollInterval()
{
return 10;
}
/**
* Run a polling cycle during idle processing in the input loop.
* @return boolean true if we should poll again for more data immediately
*/
public function poll()
{
$this->_log(LOG_DEBUG, 'Checking for notices...');
$qi = Queue_item::top($this->getQueues());
if (empty($qi)) {
$this->_log(LOG_DEBUG, 'No notices waiting; idling.');
return false;
}
$queue = $qi->transport;
$item = $this->decode($qi->frame);
if ($item) {
$rep = $this->logrep($item);
$this->_log(LOG_INFO, "Got $rep for transport $queue");
$handler = $this->getHandler($queue);
if ($handler) {
if ($handler->handle($item)) {
$this->_log(LOG_INFO, "[$queue:$rep] Successfully handled item");
$this->_done($qi);
} else {
$this->_log(LOG_INFO, "[$queue:$rep] Failed to handle item");
$this->_fail($qi);
}
} else {
$this->_log(LOG_INFO, "[$queue:$rep] No handler for queue $queue; discarding.");
$this->_done($qi);
}
} else {
$this->_log(LOG_INFO, "[$queue] Got empty/deleted item, discarding");
$this->_fail($qi);
}
return true;
}
/**
* Delete our claimed item from the queue after successful processing.
*
* @param QueueItem $qi
*/
protected function _done($qi)
{
$queue = $qi->transport;
if (empty($qi->claimed)) {
$this->_log(LOG_WARNING, "Reluctantly releasing unclaimed queue item $qi->id from $qi->queue");
}
$qi->delete();
$this->stats('handled', $queue);
}
/**
* Free our claimed queue item for later reprocessing in case of
* temporary failure.
*
* @param QueueItem $qi
*/
protected function _fail($qi)
{
$queue = $qi->transport;
if (empty($qi->claimed)) {
$this->_log(LOG_WARNING, "[$queue:item $qi->id] Ignoring failure for unclaimed queue item");
} else {
$orig = clone($qi);
$qi->claimed = null;
$qi->update($orig);
}
$this->stats('error', $queue);
}
protected function _log($level, $msg)
{
common_log($level, 'DBQueueManager: '.$msg);
}
}