Commit Graph

26 Commits

Author SHA1 Message Date
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Evan Prodromou
e2dee5fedb always set site/server to hostname if it exists 2010-01-10 00:20:08 -08:00
Evan Prodromou
3d723ed1ed allow hostname with SSL 2010-01-09 22:49:26 -08:00
Evan Prodromou
e8d85a1ef5 use nickname, not sitename, in domain for SSL 2010-01-09 22:48:05 -08:00
Evan Prodromou
deb5ee6154 correct superglobal variable name 2010-01-09 16:31:25 -08:00
Evan Prodromou
5ca41b6870 redirect to sitename.wildcard for SSL 2010-01-09 16:19:45 -08:00
Brion Vibber
53c86c43c4 Bringing Sphinx search support up to code: broken out to a plugin, now supports multiple sites on a single server.
Upgrade notes:
* Index names have changed from hardcoded 'Identica_people' and 'Identica_notices' to use the database name and actual table names. Must reindex.

New events:
* GetSearchEngine to override default search engine class selection from plugins

New scripts:
* gen_config.php generates a sphinx.conf from database configuration (with theoretical support for status_network table, but it doesn't seem to be cleanly queriable right now without knowing the db setup info for that. Needs generalized support.)
* Replaced old sphinx-indexer.sh and sphinx-cron.sh with index_update.php

Other fixes:
* sphinx.conf.sample better matches our live config, skipping unused stopword list and using a more realistic indexer memory limit

Further notes:
* Probably doesn't work right with PostgreSQL yet; Sphinx can pull from PG but the extraction queries currently look like they use some MySQL-specific functions.
2009-11-10 13:44:40 -08:00
Evan Prodromou
c628029ef1 Status_network had wrong ini file 2009-08-28 10:42:34 -07:00
Evan Prodromou
df86aa7214 define LACONICA and accept LACONICA for backwards compatibility 2009-08-26 10:41:36 -04:00
Evan Prodromou
3567b9d708 global search and replace for laconica -> statusnet 2009-08-25 18:53:24 -04:00
Evan Prodromou
865b716f09 change LACONICA to STATUSNET 2009-08-25 18:42:34 -04:00
Evan Prodromou
4737563b95 a distributed -> the distributed 2009-08-25 18:14:12 -04:00
Evan Prodromou
c8b8f07af1 change Laconica and Control Yourself to StatusNet in PHP files 2009-08-25 18:12:20 -04:00
Evan Prodromou
166e4a4b58 bad string compare in Status_network 2009-06-28 14:48:19 -04:00
Evan Prodromou
d4f1637f8a fallback for www. addresses 2009-06-28 14:45:33 -04:00
Evan Prodromou
39117bf88c forgot to disinherit Memcached_DataObject in Status_network 2009-06-15 21:55:56 -07:00
Evan Prodromou
0a886868fb Add some basic memcached handling to status_network
Status_network can't be a subclass of Memcached_DataObject -- the
latter is too entrenched in Laconica's memc handling functions, which
aren't loaded when Status_network is running! But the importance of
caching these values can't be overstated. So, a considerably
slimmed-down version of the Memcached_DataObject code is transcribed
into Status_network.
2009-06-15 21:33:56 -07:00
Evan Prodromou
8c24a3bc92 a little better query handling in redirect code 2009-06-15 16:55:01 -07:00
Evan Prodromou
f809663245 a little better query handling in redirect code 2009-06-15 16:51:49 -07:00
Evan Prodromou
2f82a3d44c forgot some functions aren't available at status time 2009-06-15 16:43:39 -07:00
Evan Prodromou
587b7a8b2a redirect on non-canonical server name 2009-06-15 16:40:53 -07:00
Evan Prodromou
6532f0dff6 strncmp -> strcasecmp 2009-06-15 14:29:25 -07:00
Evan Prodromou
d6ff702d7f Return network from network setup function
Return the network from the network setup function. Also, special-case
for when we get a server name the same as the wildcard.
2009-06-15 13:34:52 -07:00
Evan Prodromou
eb6a60ef88 updates to Status_network 2009-06-15 08:54:52 -07:00
Evan Prodromou
68d90bcab0 some class files had x bit set 2009-05-22 21:11:46 -04:00
Evan Prodromou
60861760fc Host multiple sites with the same codebase
This is the beginning of the code for status.net and related status
farms. It will read basic information about a site from a shared,
central database and use the data stored there to switch on the
hostname.
2009-04-07 17:10:54 -04:00