Commit Graph

707 Commits

Author SHA1 Message Date
Brion Vibber
0e852def6a XMPP queued output & initial retooling of DB queue manager to support non-Notice objects.
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues

Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
  This is based on Craig Andrews' work branch to generalize queues to take any
  object, but conservatively leaving out the serialization for now.
  Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
  thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
  directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
  each run through the event loop to watch for memory leaks

To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
  notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
  to queues.
* Find a way to improve interactive performance of the database-backed queue
  handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
  grouping used to split out the XMPP output queue is a bit awkward.
2010-01-21 22:40:35 -08:00
Evan Prodromou
464ce741a4 remove session info in syslog 2010-01-13 16:19:20 -08:00
Evan Prodromou
155e4ef4f3 remove some debugging statements from util.php 2010-01-12 23:58:25 -08:00
Evan Prodromou
e34ece8b06 Merge branch '0.9.x' of git@gitorious.org:statusnet/mainline into 0.9.x 2010-01-12 23:25:25 -08:00
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Evan Prodromou
5d676352c3 strip out session ID from root URL 2010-01-11 15:09:46 -08:00
Evan Prodromou
8809e577b2 Merge branch 'sessionidparam' into 0.9.x
Conflicts:
	lib/command.php
2010-01-11 00:45:26 -08:00
Evan Prodromou
ae7469a127 accept session from 2010-01-11 08:39:02 +00:00
Evan Prodromou
e0eb51e4bb add session ID to local URL when server parts differ 2010-01-10 23:51:57 -08:00
Evan Prodromou
dd7195346c Sever -> server in error message 2010-01-10 22:59:50 -08:00
Evan Prodromou
ad63a9518c Sever -> server in error message 2010-01-10 22:59:32 -08:00
Evan Prodromou
866dfa6822 Merge branch 'master' into 0.9.x
Conflicts:
	classes/Memcached_DataObject.php
2010-01-10 14:18:19 -08:00
Evan Prodromou
30409f7bad debugging code to find passed-in objects in munge_password 2010-01-10 12:07:49 -08:00
Brion Vibber
af95005bc4 Ticket 2141: bugs with weighted popularity lists across year boundary.
Consolidated several separate implementations of the same weighting algorithm into common_sql_weight() and fixed some bugs...
For MySQL, now using timestampdiff() instead of subtraction for the comparison, so we get sane results when the year doesn't match, and utc_timestamp() rather than now() so we don't get negative ages for recent items with local server timezone.
Unknown whether the same problems affect PostgreSQL, but note that it lacks the timestampdiff() SQL function.
2010-01-04 13:01:17 -08:00
Brion Vibber
6911e1c797 Ticket 2141: bugs with weighted popularity lists across year boundary.
Consolidated several separate implementations of the same weighting algorithm into common_sql_weight() and fixed some bugs...
For MySQL, now using timestampdiff() instead of subtraction for the comparison, so we get sane results when the year doesn't match, and utc_timestamp() rather than now() so we don't get negative ages for recent items with local server timezone.
Unknown whether the same problems affect PostgreSQL, but note that it lacks the timestampdiff() SQL function.
2010-01-04 11:55:27 -08:00
Evan Prodromou
c5d23e27a6 Make caching a plugin system 2010-01-02 20:26:33 -10:00
Evan Prodromou
3b912ac97e fixup memcache functions 2010-01-02 16:27:04 -10:00
Craig Andrews
1e9c03e199 Enable memcache automatic compression, starting at 20k and only if compression gain is greater than 20%.
Allows storage of larger objects (over 1mb in size uncompressed), such as huge LDAP schemas.
Should also improve cache efficiency (allows more stuff to be stored in same memory) and reduce network latency (less data transfer)
2009-12-30 19:29:38 -05:00
Evan Prodromou
f6bf952980 Merge branch 'testing' 2009-12-22 16:44:19 -08:00
Brion Vibber
f987273f11 Ignore user language settings that aren't listed in language config; we'll then fall back to current autodetection. This prevents the surprises where your profile suddenly switches to Arabic because it was selected by default due to lack of a match in the drop-down box. 2009-12-21 06:57:10 -08:00
Brion Vibber
b244ac6462 Ignore user language settings that aren't listed in language config; we'll then fall back to current autodetection. This prevents the surprises where your profile suddenly switches to Arabic because it was selected by default due to lack of a match in the drop-down box. 2009-12-19 15:29:53 -05:00
Brion Vibber
dc4bedd25a Add some doc comments and fixmes in util.php 2009-12-16 09:27:48 -05:00
Brion Vibber
0158f4f73d PHP 5.3 closure-based implementation of curry(); old implementation used as fallback for older PHP versions. Added unit tests to confirm they both work! 2009-12-16 09:27:48 -05:00
Craig Andrews
2ab01e040e Add 2 new events to enable logger pluginization: StartLog and EndLog 2009-12-05 02:11:27 -05:00
Brion Vibber
c89b10ffe4 Code style cleanup: dropped some unnecessary =& reference assignments where they're used only out of habit for PHP 4-style object semantics 2009-12-03 12:58:48 -08:00
Brion Vibber
914242e224 Drop the username from the log id for now; seems to trigger an error loop in some circumstances 2009-11-27 13:30:21 -08:00
Brion Vibber
9f075a6898 Drop the username from the log id for now; seems to trigger an error loop in some circumstances 2009-11-27 13:29:33 -08:00
Brion Vibber
b933f5bb15 request id on logs... pid + random id per web request + username + method + url 2009-11-27 12:32:55 -08:00
Brion Vibber
5bacd98905 request id on logs... pid + random id per web request + username + method + url 2009-11-27 12:28:15 -08:00
Brion Vibber
6d5d63636a Drop error message on setlocale() failure; this is harmless, since we actually have a working locale set up. 2009-11-25 11:23:11 -08:00
Brion Vibber
4ef515973a Drop error message on setlocale() failure; this is harmless, since we actually have a working locale set up. 2009-11-25 11:20:41 -08:00
Craig Andrews
d07df8a796 Added Authorization plugin
Added LDAPAuthorization plugin
2009-11-18 14:19:43 -05:00
Craig Andrews
f60a55d9ec blasted, that should not be there 2009-11-12 20:15:10 -05:00
Craig Andrews
ed690615de Added a User_username table that links the external username with a StatusNet user_id
Added EmailAuthenticationPlugin
Added ReverseUsernameAuthenticationPlugin
Changed the StartChangePassword and EndChangePassword events to take a user, instead of a nickname
User::allowed_nickname was declared non-static, but used as if it was static, so I made the declaration static
2009-11-12 20:12:36 -05:00
Craig Andrews
014d6b1d19 Redid how URL shorteners work. This way is much more like how Evan wants events to work (and more like how the rest of SN works). 2009-11-11 14:04:44 -05:00
Brion Vibber
7f8dbb8e45 Fix bug 1963: Web UI throws warnings during previously open login session after user account is deleted
common_logged_in() returned bogus results because it checks against null specifically, but common_current_user() was sticking 'false' into $_cur because that's what User::staticGet() returned from a failed lookup. Now we skip over a failed lookup here, so we keep null and all is well.
2009-11-11 10:38:11 -08:00
Evan Prodromou
923fa068a6 change credential check to work more like other events 2009-11-10 14:45:19 -05:00
Evan Prodromou
7ae10c27b0 Merge branch '0.9.x' of git@gitorious.org:statusnet/mainline into 0.9.x
Conflicts:
	lib/util.php
2009-11-10 14:26:49 -05:00
Evan Prodromou
8d5c2b3129 fixup output of object attributes in db error code 2009-11-10 14:06:30 -05:00
Brion Vibber
088081675f Revert "Remove more contractions"
This reverts commit 5ab709b739.

Missed this one yesterday...
2009-11-09 20:01:46 +01:00
Siebrand Mazeland
b10f362ede Merge branch '0.9.x' of git://gitorious.org/statusnet/mainline into 0.9.x 2009-11-08 23:33:58 +01:00
Siebrand Mazeland
5ab709b739 Remove more contractions
* doesn't
* won't
* isn't
* don't
2009-11-08 23:32:15 +01:00
Brion Vibber
fc5002015b Revert "* [Cc]an't -> [Cc]annot"
This reverts commit 0ab17f382b.
2009-11-08 23:28:51 +01:00
Siebrand Mazeland
0ab17f382b * [Cc]an't -> [Cc]annot
* [Cc]ould't -> [Cc]ould not
2009-11-08 23:22:38 +01:00
Craig Andrews
1845c8c773 Added an AutoRegister event
LDAP plugin can do autoregistration
2009-11-05 16:40:41 -05:00
Craig Andrews
a82df5fae8 Added a CheckPassword event 2009-11-04 13:06:55 -05:00
Brion Vibber
86560eeb3d Merge commit 'refs/merge-requests/1900' of git://gitorious.org/statusnet/mainline into integration 2009-11-02 10:16:06 -08:00
Craig Andrews
1cf67f4f71 allow <> to surround the url (like () or []) 2009-11-02 12:59:11 -05:00
Craig Andrews
8782f5fedf do not allow " in URLs - they are not legal URL characters 2009-11-02 12:53:02 -05:00
Craig Andrews
b179ab650a do not allow [ and ] - they are not legal URL characters 2009-11-02 12:53:02 -05:00