Commit Graph

438 Commits

Author SHA1 Message Date
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Brion Vibber
be1ac6678d fix long options on deleteuser.php 2010-01-11 13:24:40 -08:00
Siebrand Mazeland
4af6b7f5c3 Lots of tiny message changes.
* Mostly punctuation updates so that the same message is used consistently in all of StatusNet.
* Some cases of "Title Case" removed, because that does not appear to be used consistently.
2010-01-10 12:26:24 +01:00
Craig Andrews
055f3fdddb Add an IMAP daemon so StatusNet can process incoming user posts via catch-all mailbox (in addition to the pre-existing script alias method) 2010-01-08 18:52:43 -05:00
Brion Vibber
6f5b765c97 suppress notice for undefined prompt variable when console.php is used from non-interactive terminal 2010-01-06 13:08:56 -08:00
Zach Copley
ff26b8d88b Add an RSSCloud queue handler daemon 2010-01-05 23:19:13 -08:00
Evan Prodromou
0c31c3d80c free some memory in createsim.php 2010-01-04 22:49:09 -10:00
Evan Prodromou
31b62bf781 Merge branch 'batchlocation' 2010-01-04 15:10:43 -10:00
Evan Prodromou
55ba858e8c Script to update the location ID for users
Since we added locations to the database, some users may have
location strings in their profiles but not structured locations. This
script updates the locations for single users or for all users.
2009-12-31 12:38:58 -10:00
Brion Vibber
45c9d3d729 Add progress output and optional --sleep-time parameter to triminboxes.php 2009-12-29 14:17:03 -08:00
Evan Prodromou
7160e11395 add setconfig.php script to set configuration options 2009-12-24 15:13:30 -08:00
Evan Prodromou
d40f0931aa remove dead code for handling direct messages, now done with commands 2009-12-15 10:12:36 -05:00
Brion Vibber
b93068083e Clean up console output for non-interactive mode (handy for batch setup scripts) 2009-12-14 11:51:38 -08:00
Brion Vibber
e2f0fc7b3f Make useremail.php executable 2009-12-11 13:14:40 -08:00
Christopher Vollick
280b0b500f Added UserEmail script.
Used to query user's emails.

Mostly used for administration, to see if a user requesting something is who they say.

Also, some people assume that the admin knows this data, and says things like:
"If you could do _____ with the account connected to this email".

It'd be nice if we could do that without raw SQL.
2009-12-11 10:34:57 -05:00
Brion Vibber
4b5e977a7b New _m() gettext wrapper with smart detection of plugin domains. Plugin base class registers your gettext files if present at initialization.
update_pot.sh replaced with update_po_templates.php which can do core, plugins, or all (default).
Top-level Makefile added to build .mo files for plugins as well as core.

As described on list:
http://lists.status.net/pipermail/statusnet-dev/2009-December/002869.html
2009-12-08 12:17:11 -08:00
Brion Vibber
5affa49820 Tweak updateavatarurl.php: emit a newline whether we're on verbose or non-quiet, emit help when no users specified. 2009-12-08 08:30:30 -08:00
Zach Copley
4eceef0dbe Grand ALL permissions to the DB user when setting up a new status.net site. 2009-12-07 11:34:03 -08:00
Brion Vibber
c89b10ffe4 Code style cleanup: dropped some unnecessary =& reference assignments where they're used only out of habit for PHP 4-style object semantics 2009-12-03 12:58:48 -08:00
Siebrand Mazeland
5edc27be6e Do not rebuild/add .mo files by default
FIXME: should be made a command line parameter.
2009-12-03 00:28:00 +01:00
Evan Prodromou
ba4fd15abd add pluginhandler to list of daemons to shut down 2009-12-02 11:07:44 -05:00
Craig Andrews
b5451353e0 make checkschema.php executable (which makes sense, cause the README says to run it :-) ) 2009-11-30 17:00:00 -05:00
Zach Copley
9dc888894b Merge branch 'master' into 0.9.x
* master: (67 commits)
  Ticket 2038: fix bad bug tracker link
  Fix regression in group posting: bug introduced in commit 1319002e15. Need to use actual profile object rather than an id on a variable that doesn't exist when checking blocks :D
  Log database errors when saving notice_inbox entries
  Drop the username from the log id for now; seems to trigger an error loop in some circumstances
  request id on logs... pid + random id per web request + username + method + url
  Add OpenID ini info back into statusnet.ini as a stopgap until we can
  Some changes to the OpenID DataObjects to make them emit the exact same
  OpenID plugin should set 'user_openid.display' as unique key
  Remove relationship: user_openid.user_id -> user.id. I don't think this
  Have OpenID plugin DataObjects emit their own .ini info
  Revert "Allow plugin DB_DataObject classes to not have to use the .ini file by overriding keys(), table(), and sequenceKey() for them"
  Catch and report exceptions from notice_to_omb_notice() instead of letting the OMB queue handler die.
  Fix regression in remote subscription; added hasRole() shadow method on Remote_profile.
  Fix fatal error on OMB subscription for first-timers
  Remove annoying log msg
  Drop error message on setlocale() failure; this is harmless, since we actually have a working locale set up.
  Catch uncaught exception
  Fixed bug where reply-sync bit wasn't getting saved
  Forgot to render the nav menu when on FB Connect login tab
  Facebook plugin no longer takes over Login and Connect settings nav menus
  ...

Conflicts:
	db/08to09_pg.sql
	db/statusnet_pg.sql
	locale/pt_BR/LC_MESSAGES/statusnet.mo
	plugins/Mapstraction/MapstractionPlugin.php
2009-11-30 10:28:58 -08:00
Siebrand Mazeland
99604267d0 Do not use fuzzy in the compiled messages files. 2009-11-21 18:40:34 +01:00
Brion Vibber
09a021cefe start for pluginqueuehandler 2009-11-20 11:20:02 -08:00
Brion Vibber
92453936c2 start for pluginqueuehandler 2009-11-20 11:18:12 -08:00
Zach Copley
4b98edf75f Merge branch '0.9-release'
* 0.9-release: (874 commits)
  Removed call to NewDirectMessage() until IE return is fixed i.e.,
  Don't show flag user button your own profile
  Fixed HXR response for flag user
  Using the right form class name
  Using common_redirect
  Left a form_data class of a <ul> in the user admin panel
  Added validation to fields in user admin panel
  Added a user admin panel
  Added mobile logos for default and identica themes
  Changed gif to png
  Changed this to action. THANKS zach!
  Doing content negotiation only once
  Add execute bit to pingqueuehandler
  Localisation updates for !StatusNet from !translatewiki.net
  Use the browser's geolocation API to set the location on the notice form
  Add geometa library, and include it.
  Add location form elements to the noticeform, and save their values on submission
  Use the $user object nickname, as login name doesnt have to == nickname anymore with plugins such as ldap/etc
  Revert "Re added NICKNAME_FMT constant to router.php."
  Moved most path and server settings to a new paths admin panel
  ...

Conflicts:
	js/util.js
	locale/it_IT/LC_MESSAGES/statusnet.mo
	locale/mk_MK/LC_MESSAGES/statusnet.mo
	locale/mk_MK/LC_MESSAGES/statusnet.po
	locale/pt_BR/LC_MESSAGES/statusnet.mo
	locale/vi_VN/LC_MESSAGES/statusnet.mo
	plugins/InfiniteScroll/infinitescroll.js
	plugins/Realtime/realtimeupdate.js
2009-11-19 20:12:46 -08:00
Zach Copley
ad56ebbb97 Add execute bit to pingqueuehandler 2009-11-19 12:45:45 -08:00
Siebrand Mazeland
69abde6e0c Removing "join-existing". Do not add on update, just rebuild complete pot. The pot file gets really dirty, really quickly. 2009-11-19 01:04:47 +01:00
Brion Vibber
1827256d0e Added support for pgettext() and npgettext() to separate contexts for translatable messages that are going to be ambiguous in English original. 2009-11-18 14:57:18 -08:00
Brion Vibber
b7660b3d99 A little cleanup on console.php; save readline history more aggressively; avoid some notice sloppiness) 2009-11-17 17:09:31 -08:00
Eric Helgeson
b7a08fdacc +x deleteuser.php 2009-11-16 15:00:36 -05:00
Ciaran Gultnieks
8109d39a56 Minor typo correction in log message. Seems trivial, unless you are trying to search the log for it. 2009-11-12 09:18:15 +00:00
CiaranG
41944f36e4 CLAIM_TIMEOUT is already defined elsewhere - causes a PHP warning (minor perf. issue) and confusion if someone tries to change one or the other 2009-11-11 20:31:58 +00:00
Brion Vibber
53c86c43c4 Bringing Sphinx search support up to code: broken out to a plugin, now supports multiple sites on a single server.
Upgrade notes:
* Index names have changed from hardcoded 'Identica_people' and 'Identica_notices' to use the database name and actual table names. Must reindex.

New events:
* GetSearchEngine to override default search engine class selection from plugins

New scripts:
* gen_config.php generates a sphinx.conf from database configuration (with theoretical support for status_network table, but it doesn't seem to be cleanly queriable right now without knowing the db setup info for that. Needs generalized support.)
* Replaced old sphinx-indexer.sh and sphinx-cron.sh with index_update.php

Other fixes:
* sphinx.conf.sample better matches our live config, skipping unused stopword list and using a more realistic indexer memory limit

Further notes:
* Probably doesn't work right with PostgreSQL yet; Sphinx can pull from PG but the extraction queries currently look like they use some MySQL-specific functions.
2009-11-10 13:44:40 -08:00
Evan Prodromou
8bcc58a989 Merge branch 'master' into 0.8.x
Conflicts:
	scripts/updateavatarurl.php
2009-11-09 17:43:45 -05:00
Evan Prodromou
499b3555df broadcast profile changes from updateavatarurl.php 2009-11-09 17:41:51 -05:00
Evan Prodromou
a6312cb79c script to update avatar URLs on server 2009-11-09 17:26:52 -05:00
Evan Prodromou
a0a9c7bcbf script to update avatar URLs on server 2009-11-09 17:26:36 -05:00
Brion Vibber
088081675f Revert "Remove more contractions"
This reverts commit 5ab709b739.

Missed this one yesterday...
2009-11-09 20:01:46 +01:00
Siebrand Mazeland
b10f362ede Merge branch '0.9.x' of git://gitorious.org/statusnet/mainline into 0.9.x 2009-11-08 23:33:58 +01:00
Siebrand Mazeland
5ab709b739 Remove more contractions
* doesn't
* won't
* isn't
* don't
2009-11-08 23:32:15 +01:00
Brion Vibber
fc5002015b Revert "* [Cc]an't -> [Cc]annot"
This reverts commit 0ab17f382b.
2009-11-08 23:28:51 +01:00
Siebrand Mazeland
104a47e4a2 Merge branch '0.9.x' of git://gitorious.org/statusnet/mainline into 0.9.x 2009-11-08 23:22:50 +01:00
Siebrand Mazeland
0ab17f382b * [Cc]an't -> [Cc]annot
* [Cc]ould't -> [Cc]ould not
2009-11-08 23:22:38 +01:00
Evan Prodromou
321ac38884 script for granting/revoking user roles 2009-11-07 22:35:35 -05:00
Siebrand Mazeland
f7b0017f21 Do not export codes twice 2009-11-06 18:49:42 +01:00
Siebrand Mazeland
023d5c17ab More bug fixes. 2009-11-06 17:16:43 +01:00
Siebrand Mazeland
814b513aa5 Fix a few bugs I added in previous revision 2009-11-06 16:53:07 +01:00
Siebrand Mazeland
8beef3eab3 * add language code conversion for translatewiki.net where needed
* skip update of code 'en' (as English is the source language)
2009-11-06 16:40:54 +01:00