Commit Graph

35 Commits

Author SHA1 Message Date
James Walker
5688c635a6 backwards compatibility for old tags format in hasTag 2010-07-28 12:13:53 -04:00
James Walker
29b8a6a18f don't try to save empty tags 2010-07-28 11:57:54 -04:00
James Walker
7065450f03 normalizing tags for status_network 2010-07-20 17:34:58 -07:00
James Walker
b667092153 adding a Status_network::updateKeys() method, since DB_DataObject update doesn't do keys. 2010-06-17 13:44:17 -04:00
Brion Vibber
8b9436e8ae Option to divert PuSH items directly to the target site's queue when local 2010-06-03 18:06:55 -07:00
Brion Vibber
c74aea589d Stomp queue restructuring for mass scalability:
- Multiplexing queues into groups and for multiple sites.
- Sharing vs breakout configurable per site and per queue via $config['queue']['breakout']
- Detect how many times a message is redelivered, discard if it's killed too many daemons
 - count configurable with $config['queue']['max_retries']
 - can dump the items to files in $config['queue']['dead_letter_dir']

Queue daemon memory & resource leak fixes:
- avoid unnecessary reconnections to memcached server (switch persistent connections back in on second initialization, assuming it's child process)
- monkey-patch for leaky .ini loads in DB_DataObject::databaseStructure() - was leaking 200k per active switch
- applied leak fixes to Status_network as well, using intermediate base Safe_DataObject for both it and Memcache_DataObject

Misc queue fixes:
- correct handling of child processes exiting due to signal termination instead of regular exit
- shutdown instead of infinite respawn loop if we're already past the soft memory limit at startup
- Added --all option for xmppdaemon... still opens one xmpp connection per site that has xmpp active

Cache updates:
- add Cache::increment() method with native support for memcached atomic increment
2010-02-16 09:16:51 -08:00
Brion Vibber
58be61b641 Control channel for queue daemons to request graceful shutdown, restart, or update to listen to a newly added or reconfigured site.
queuectl.php --update -s<site>
  queuectl.php --stop
  queuectl.php --restart

Default control channel is /topic/statusnet-control. For external utilities to send a site update ping direct to the queue server, connect via Stomp and send a message formatted thus:

  update:<nickname>

(Nickname here, *not* server hostname! The rest of the queues will be updated to use nicknames later.)

Note that all currently-connected queue daemons will get these notifications, including both queuedaemon.php and xmppdaemon.php. (XMPP will ignore site update requests for sites that it's not handling.)

Limitations:
* only implemented for stomp queue manager so far
* --update may not yet handle a changed server name properly
* --restart won't reload PHP code files that were already loaded at startup. Still need to stop and restart the daemons from 'outside' when updating code base.
2010-01-26 11:49:49 -08:00
Brion Vibber
ad6f0501ff Site metadata tags in status_network: single 'tags' field, pipe-separated.
$sn->tags() returns tag list as array; $sn->hasTag('blah') to check for a particular tag only

Could be used to control things in config file:

  $sn = Status_network::setupSite($_server, $_path, $_wildcard);
  if (!$sn) { die("No such site"); }
  if ($sn->hasTag('individual')) { /* blah */ }

Note memcached keys are unchanged; if tags are changed from an external tool clear:
  statusnet:<dbname>:status_network:<key>:<val>
  for <key>s 'nickname', 'hostname', and 'pathname'
2010-01-26 10:33:20 -08:00
Evan Prodromou
238998eca0 save nickname and wildcard when setting up status network 2010-01-24 18:18:24 -05:00
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Evan Prodromou
e2dee5fedb always set site/server to hostname if it exists 2010-01-10 00:20:08 -08:00
Evan Prodromou
3d723ed1ed allow hostname with SSL 2010-01-09 22:49:26 -08:00
Evan Prodromou
e8d85a1ef5 use nickname, not sitename, in domain for SSL 2010-01-09 22:48:05 -08:00
Evan Prodromou
deb5ee6154 correct superglobal variable name 2010-01-09 16:31:25 -08:00
Evan Prodromou
5ca41b6870 redirect to sitename.wildcard for SSL 2010-01-09 16:19:45 -08:00
Brion Vibber
53c86c43c4 Bringing Sphinx search support up to code: broken out to a plugin, now supports multiple sites on a single server.
Upgrade notes:
* Index names have changed from hardcoded 'Identica_people' and 'Identica_notices' to use the database name and actual table names. Must reindex.

New events:
* GetSearchEngine to override default search engine class selection from plugins

New scripts:
* gen_config.php generates a sphinx.conf from database configuration (with theoretical support for status_network table, but it doesn't seem to be cleanly queriable right now without knowing the db setup info for that. Needs generalized support.)
* Replaced old sphinx-indexer.sh and sphinx-cron.sh with index_update.php

Other fixes:
* sphinx.conf.sample better matches our live config, skipping unused stopword list and using a more realistic indexer memory limit

Further notes:
* Probably doesn't work right with PostgreSQL yet; Sphinx can pull from PG but the extraction queries currently look like they use some MySQL-specific functions.
2009-11-10 13:44:40 -08:00
Evan Prodromou
c628029ef1 Status_network had wrong ini file 2009-08-28 10:42:34 -07:00
Evan Prodromou
df86aa7214 define LACONICA and accept LACONICA for backwards compatibility 2009-08-26 10:41:36 -04:00
Evan Prodromou
3567b9d708 global search and replace for laconica -> statusnet 2009-08-25 18:53:24 -04:00
Evan Prodromou
865b716f09 change LACONICA to STATUSNET 2009-08-25 18:42:34 -04:00
Evan Prodromou
4737563b95 a distributed -> the distributed 2009-08-25 18:14:12 -04:00
Evan Prodromou
c8b8f07af1 change Laconica and Control Yourself to StatusNet in PHP files 2009-08-25 18:12:20 -04:00
Evan Prodromou
166e4a4b58 bad string compare in Status_network 2009-06-28 14:48:19 -04:00
Evan Prodromou
d4f1637f8a fallback for www. addresses 2009-06-28 14:45:33 -04:00
Evan Prodromou
39117bf88c forgot to disinherit Memcached_DataObject in Status_network 2009-06-15 21:55:56 -07:00
Evan Prodromou
0a886868fb Add some basic memcached handling to status_network
Status_network can't be a subclass of Memcached_DataObject -- the
latter is too entrenched in Laconica's memc handling functions, which
aren't loaded when Status_network is running! But the importance of
caching these values can't be overstated. So, a considerably
slimmed-down version of the Memcached_DataObject code is transcribed
into Status_network.
2009-06-15 21:33:56 -07:00
Evan Prodromou
8c24a3bc92 a little better query handling in redirect code 2009-06-15 16:55:01 -07:00
Evan Prodromou
f809663245 a little better query handling in redirect code 2009-06-15 16:51:49 -07:00
Evan Prodromou
2f82a3d44c forgot some functions aren't available at status time 2009-06-15 16:43:39 -07:00
Evan Prodromou
587b7a8b2a redirect on non-canonical server name 2009-06-15 16:40:53 -07:00
Evan Prodromou
6532f0dff6 strncmp -> strcasecmp 2009-06-15 14:29:25 -07:00
Evan Prodromou
d6ff702d7f Return network from network setup function
Return the network from the network setup function. Also, special-case
for when we get a server name the same as the wildcard.
2009-06-15 13:34:52 -07:00
Evan Prodromou
eb6a60ef88 updates to Status_network 2009-06-15 08:54:52 -07:00
Evan Prodromou
68d90bcab0 some class files had x bit set 2009-05-22 21:11:46 -04:00
Evan Prodromou
60861760fc Host multiple sites with the same codebase
This is the beginning of the code for status.net and related status
farms. It will read basic information about a site from a shared,
central database and use the data stored there to switch on the
hostname.
2009-04-07 17:10:54 -04:00