Commit Graph

168 Commits

Author SHA1 Message Date
Brion Vibber
07f145049e Merge branch 'master' into testing 2010-02-17 16:52:13 -08:00
Brion Vibber
ce6be4f836 Queues: redid the breakout control model so we can start up and subscribe to queues without running through the complete site list, which is ok at 1k sites but too slow at 10k.
All breakout queues that we're going to need to listen to now need to be explicitly listed in $config['queue']['breakout'].

Until XMPP is moved to component model, this setting will let the individual processes work with their own queues:
$config['queue']['breakout'][] = 'xmpp/xmppout/' . $config['site']['nickname'];
2010-02-17 16:49:00 -08:00
Brion Vibber
d5cbfe8071 Merge branch 'testing' into 0.9.x
Conflicts:
	lib/iomaster.php
2010-02-16 09:25:09 -08:00
Brion Vibber
81b6b58e33 Merge branch 'master' into testing
Conflicts:
	lib/stompqueuemanager.php
2010-02-16 09:22:02 -08:00
Brion Vibber
c74aea589d Stomp queue restructuring for mass scalability:
- Multiplexing queues into groups and for multiple sites.
- Sharing vs breakout configurable per site and per queue via $config['queue']['breakout']
- Detect how many times a message is redelivered, discard if it's killed too many daemons
 - count configurable with $config['queue']['max_retries']
 - can dump the items to files in $config['queue']['dead_letter_dir']

Queue daemon memory & resource leak fixes:
- avoid unnecessary reconnections to memcached server (switch persistent connections back in on second initialization, assuming it's child process)
- monkey-patch for leaky .ini loads in DB_DataObject::databaseStructure() - was leaking 200k per active switch
- applied leak fixes to Status_network as well, using intermediate base Safe_DataObject for both it and Memcache_DataObject

Misc queue fixes:
- correct handling of child processes exiting due to signal termination instead of regular exit
- shutdown instead of infinite respawn loop if we're already past the soft memory limit at startup
- Added --all option for xmppdaemon... still opens one xmpp connection per site that has xmpp active

Cache updates:
- add Cache::increment() method with native support for memcached atomic increment
2010-02-16 09:16:51 -08:00
Brion Vibber
122c8677b7 Merge branch 'testing' of gitorious.org:statusnet/mainline into 0.9.x 2010-02-12 11:18:35 -08:00
Evan Prodromou
31461e120f let files go to SSL dir too 2010-02-11 17:06:57 -05:00
Evan Prodromou
3018683718 let backgrounds be put under SSL 2010-02-11 17:03:31 -05:00
Evan Prodromou
b96af33d97 put Javascript files under SSL 2010-02-11 16:59:39 -05:00
Evan Prodromou
d6869cde7b let avatars be served over SSL 2010-02-11 16:51:15 -05:00
Evan Prodromou
5175b5062e default theme ssl to null 2010-02-11 16:48:15 -05:00
Evan Prodromou
a6ab9c4a3e Themes can be served from an SSL server 2010-02-11 16:42:58 -05:00
Brion Vibber
e856af34c3 Configurable delay between queuedaemon.php spawns/respawns to help stagger out startups and subscriptions. Defaults to 1 second.
$config['queue']['spawndelay'] = 1;
2010-02-09 14:27:33 -08:00
Brion Vibber
cd0f288fa7 Configurable delay between queuedaemon.php spawns/respawns to help stagger out startups and subscriptions. Defaults to 1 second.
$config['queue']['spawndelay'] = 1;
2010-02-09 12:39:31 -08:00
Brion Vibber
c59eee72a8 Merge branch 'testing' of gitorious.org:statusnet/mainline into 0.9.x 2010-02-04 12:23:56 -08:00
Brion Vibber
5a1cbdc6f1 Add time-based cutoffs for public tag cloud, favorited lists to speed up those queries.
Defaulting to only looking at last 90 days of activity, can be adjusted up or down.

$config['tag']['cutoff'] = 86400 * 90;
$config['popular']['cutoff'] = 86400 * 90;

Per-user and per-group tag clouds do not use the cutoff (and it doesn't help with indexing on them).
2010-02-04 11:06:01 -08:00
Evan Prodromou
779204b194 Merge branch 'testing' into 0.9.x
Conflicts:
	actions/apioauthauthorize.php
2010-01-31 15:27:58 -05:00
Evan Prodromou
dc62246443 Add a robots.txt URL to the site root
Adds a robots.txt file to the site root. Defaults defined by
'robotstxt' section of config. New events StartRobotsTxt and
EndRobotsTxt to let plugins add information. Probably not
useful if path is not /, but won't hurt anything, either.
2010-01-31 10:12:26 -05:00
Zach Copley
5868511716 Move sessions settings to its own panel 2010-01-28 22:06:01 -08:00
Zach Copley
440ee00b1e Move sessions settings to its own panel 2010-01-28 22:04:14 -08:00
Brion Vibber
155a5d446f Manual failover for stomp queues.
If an array of multiple servers is put in $config['queue']['stomp_server'], enqueues will pick a random server to send to (failing over automatically if any are down).
Queue handling daemons connect all servers so they get events no matter where they were delivered.
In case of disconnection, daemons should now handle it gracefully and attempt to reconnect every 60 seconds or so, automatically resubscribing to all queues once it's back up.

Can put to 'native' failover for reads as well by disabling $config['stomp']['manual_failover'] = false; but this is untested and may explode in addition to requiring that your ActiveMQ cluster actually be set up to handle its own data distribution.

Additionally, can choose which queues to mark as persistent by setting $config['stomp']['persistent'] to an array of queue names.
2010-01-28 16:49:32 -08:00
Brion Vibber
a868a523a5 Can now set $config['queue']['stomp_persistent'] = false; to explicitly disable persistence when we queue items 2010-01-28 09:52:35 -08:00
Brion Vibber
3abfb454a3 Adds an emergency switch so we can run inbox distribution at save time (bypassing 'distrib' queue)
Set $config['queue']['inboxes'] = false to do so
2010-01-27 18:39:17 -08:00
Zach Copley
78079f34e2 Merge branch 'testing' into -1.9.x
* testing: (130 commits)
  HTTP auth provided is evaluated even if it's not required
  Rename rc3to09.sql to rc3torc4.sql to avoid confusion if we add a last-minute change after this!
  Add new oauth tables and modifications to 'consumer' table for rc4
  Centred leaderboard ad
  camelcase the uap param names
  move leaderboard to after the header
  Moved rectangle ad into aside and leaderboard to the right in header.
  Aligning wide skyscraper to the right instead of left
  CSS ids and classes fixed in UAPPlugin
  wrong height for rectangle in BlankAd
  Add the moved BlankAdPlugin
  make BlankAd dir and change to use a 1x1 image
  move BlankAdPlugin to its own dir
  Add BlankAdPlugin to test ad layout in different themes
  make uapplugin an abstract class
  move UAP plugin to core
  Lowercased switch cases in UAP Plugin
  Plugin for Universal Ad Package. Outputs four most widely used ad types.
  Add persistent:true property to Stomp messages so ActiveMQ doesn't decide to discard them even though persistence is enabled on the broker. :) (Thanks Aric!)
  quick fix: use common_path() on realtime update JS so it works with the new JS path code (will pull from main server for now)
  ...

Conflicts:
	actions/apioauthaccesstoken.php
	actions/apioauthauthorize.php
	actions/apioauthrequesttoken.php
	actions/editapplication.php
	actions/newapplication.php
	lib/apiauth.php
	lib/queuemanager.php
	lib/router.php
2010-01-27 14:27:22 -08:00
Evan Prodromou
c52951cef5 Optionally set a separate Javascript server and path
We have about 10-12 JavaScript pages per Web page. They usually
are based on the same server as the Web pages, but since they're
static files, it makes sense to offload them to a lite server that
handles static files well.

This commit lets you set a separate Javascript server and path for the
default Javascript code in StatusNet.

Squashed commit of the following:

commit 139d1622fdafe5ad00c820224416d9021efc3234
Author: Evan Prodromou <evan@status.net>
Date:   Wed Jan 27 11:30:24 2010 -0500

    modules that call htmloutputter::script() don't prescribe js/ path

commit c6ca3174af73efed55eaed5ff1e2a3bdc77d2d87
Author: Evan Prodromou <evan@status.net>
Date:   Wed Jan 27 11:28:07 2010 -0500

    configurable server and path for javascript files
2010-01-27 11:37:22 -05:00
Zach Copley
aad42427cc New access admin panel for site registration settings 2010-01-26 15:56:19 -08:00
Brion Vibber
58be61b641 Control channel for queue daemons to request graceful shutdown, restart, or update to listen to a newly added or reconfigured site.
queuectl.php --update -s<site>
  queuectl.php --stop
  queuectl.php --restart

Default control channel is /topic/statusnet-control. For external utilities to send a site update ping direct to the queue server, connect via Stomp and send a message formatted thus:

  update:<nickname>

(Nickname here, *not* server hostname! The rest of the queues will be updated to use nicknames later.)

Note that all currently-connected queue daemons will get these notifications, including both queuedaemon.php and xmppdaemon.php. (XMPP will ignore site update requests for sites that it's not handling.)

Limitations:
* only implemented for stomp queue manager so far
* --update may not yet handle a changed server name properly
* --restart won't reload PHP code files that were already loaded at startup. Still need to stop and restart the daemons from 'outside' when updating code base.
2010-01-26 11:49:49 -08:00
Evan Prodromou
655573c213 Single-user mode
New configuration options to define a single-user mode. This hides
most of the "community" pages, like the public timeline and groups.
The main user's timeline becomes the main page, and most other URLs
are changed.

Switching back and forth between 1-user and multi-user mode is
probably hazardous.

Squashed commit of the following:

commit d814aa5c92d14a27a12baba7893f3f8bf63f1d08
Author: Evan Prodromou <evan@status.net>
Date:   Tue Jan 26 00:17:27 2010 -0500

    don't show inbox and outbox in single-user mode

commit 47f19b9523a7015d4c6e460b73ea32c839e00aa1
Author: Evan Prodromou <evan@status.net>
Date:   Tue Jan 26 00:15:22 2010 -0500

    show correct URL for logo in single-user mode

commit 552010cffc33eadbc512ec5a67619dbc2015239a
Author: Evan Prodromou <evan@status.net>
Date:   Tue Jan 26 00:15:06 2010 -0500

    make singleuser its own config section

commit 786ab260a3ca172e57b555c75ca10946d8f258a1
Author: Evan Prodromou <evan@status.net>
Date:   Tue Jan 26 00:05:19 2010 -0500

    make single-user mode work

commit 5b21d7309b3a8dd5a4e0f29aea76f7897f1818b1
Author: Evan Prodromou <evan@status.net>
Date:   Mon Jan 25 23:45:55 2010 -0500

    add single-user mode
2010-01-26 00:21:05 -05:00
Evan Prodromou
b6aa1511eb Merge branch 'master' into 0.9.x 2010-01-25 18:10:59 -05:00
Zach Copley
3c2b05d222 Workflow for registering new OAuth apps pretty much done. 2010-01-24 16:36:02 -08:00
Evan Prodromou
49d500e3e1 defaults for nickname and wildcard 2010-01-24 16:01:08 -08:00
Evan Prodromou
7cd666ae92 defaults for nickname and wildcard 2010-01-24 18:19:13 -05:00
Brion Vibber
c7507e7e9d XMPP queued output & initial retooling of DB queue manager to support non-Notice objects.
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues

Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
  This is based on Craig Andrews' work branch to generalize queues to take any
  object, but conservatively leaving out the serialization for now.
  Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
  thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
  directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
  each run through the event loop to watch for memory leaks

To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
  notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
  to queues.
* Find a way to improve interactive performance of the database-backed queue
  handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
  grouping used to split out the XMPP output queue is a bit awkward.

Conflicts:

	scripts/xmppdaemon.php
2010-01-22 12:52:36 -08:00
Evan Prodromou
c8bc598cfd Merge branch 'testing' into 0.9.x 2010-01-22 14:18:43 -05:00
Brion Vibber
0e852def6a XMPP queued output & initial retooling of DB queue manager to support non-Notice objects.
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues

Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
  This is based on Craig Andrews' work branch to generalize queues to take any
  object, but conservatively leaving out the serialization for now.
  Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
  thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
  directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
  each run through the event loop to watch for memory leaks

To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
  notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
  to queues.
* Find a way to improve interactive performance of the database-backed queue
  handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
  grouping used to split out the XMPP output queue is a bit awkward.
2010-01-21 22:40:35 -08:00
Brion Vibber
8dafe09ab2 Option to log slow db queries or all db queries
$config['db']['log_queries'] = true; // all
$config['db']['log_slow_queries'] = 10; // queries taking > 10 seconds
2010-01-21 11:07:52 -08:00
Evan Prodromou
c63832f7bf add PubSubHubBub and RSSCloud to list of default plugins 2010-01-20 17:58:23 -05:00
Evan Prodromou
01dc77c2fd Merge branch 'master' into 0.9.x 2010-01-19 00:31:42 -05:00
Evan Prodromou
c7f14cd777 allow all rights reserved, private data 2010-01-19 00:04:58 -05:00
Zach Copley
dad67b030f Workflow for registering new OAuth apps pretty much done. 2010-01-14 02:41:04 +00:00
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Evan Prodromou
2aa0ab9777 let system administrators disallow certain admin panels 2010-01-08 01:00:29 -08:00
Evan Prodromou
505cd382f3 ptiturl.com correct name 2010-01-07 23:38:19 -08:00
Evan Prodromou
2c33e61b94 add default plugins and load them 2010-01-07 16:13:36 -08:00
Craig Andrews
250bcfa8dc Require users to login to view attachments on private sites
Thank you jeff-themovie for this implementation!
2010-01-05 17:49:28 -05:00
Evan Prodromou
cc5534d180 First version of Memcache plugin 2010-01-02 21:16:59 -10:00
Evan Prodromou
39bdda9c7e More configuration options for location sharing 2009-12-28 14:43:34 -08:00
Evan Prodromou
bb93d6b1c7 remove namespace setting from location; it's unused 2009-12-28 14:21:07 -08:00
Evan Prodromou
43ee2ec42f default for login command is disabled 2009-12-14 18:09:30 -05:00
Brion Vibber
3536f01258 Make default & sample val for $config['queue']['queue_basename'] actually work! Now matches the recommendation in README which is in fact correct. Must include the /queue/ prefix (unless somthing else actually makes sense) 2009-12-08 09:57:37 -08:00
Craig Andrews
51f0dd5e37 Add configuration option to toggle the indenting of the output HTML. Defaults to indent enabled. 2009-12-05 01:03:04 -05:00
Brion Vibber
15a9c86979 Ticket 2048: make OMB posting HTTP timeout configurable as $config['omb']['timeout']; defaults to 5 seconds instead of 20-second default in Yadis library 2009-12-01 14:50:18 -08:00
Evan Prodromou
fe18063bd2 Revert "Ask users if they wish to send statistics to SNI, default is off."
This reverts commit 7efea1115f.

Conflicts:

	install.php
2009-11-14 17:38:10 +01:00
Eric Helgeson
7efea1115f Ask users if they wish to send statistics to SNI, default is off.
Users may not know about this setting and are unaware they are sending stats.
This allows them to make that decision.
2009-11-11 18:45:21 -05:00
Brion Vibber
53c86c43c4 Bringing Sphinx search support up to code: broken out to a plugin, now supports multiple sites on a single server.
Upgrade notes:
* Index names have changed from hardcoded 'Identica_people' and 'Identica_notices' to use the database name and actual table names. Must reindex.

New events:
* GetSearchEngine to override default search engine class selection from plugins

New scripts:
* gen_config.php generates a sphinx.conf from database configuration (with theoretical support for status_network table, but it doesn't seem to be cleanly queriable right now without knowing the db setup info for that. Needs generalized support.)
* Replaced old sphinx-indexer.sh and sphinx-cron.sh with index_update.php

Other fixes:
* sphinx.conf.sample better matches our live config, skipping unused stopword list and using a more realistic indexer memory limit

Further notes:
* Probably doesn't work right with PostgreSQL yet; Sphinx can pull from PG but the extraction queries currently look like they use some MySQL-specific functions.
2009-11-10 13:44:40 -08:00
Brion Vibber
5581143bee Rebuilt HTTPClient class as an extension of PEAR HTTP_Request2 package, adding redirect handling and convenience functions.
Caching support will be added in future work after unit tests have been added.

* extlib: add PEAR HTTP_Request2 0.4.1 alpha
* extlib: update PEAR Net_URL2 to 0.3.0 beta for HTTP_Request2 compatibility
* moved direct usage of CURL and file_get_contents to HTTPClient class, excluding external-sourced libraries
* adapted GeonamesPlugin for new HTTPResponse interface

Note some plugins haven't been fully tested yet.
2009-11-02 09:14:15 -08:00
Brion Vibber
b22fc5b74a Revert "Rebuilt HTTPClient class as an extension of PEAR HTTP_Request2 package, adding redirect handling and convenience functions."
Going to restructure a little more before finalizing this...

This reverts commit fa37967858.
2009-11-02 07:51:29 -08:00
Brion Vibber
fa37967858 Rebuilt HTTPClient class as an extension of PEAR HTTP_Request2 package, adding redirect handling and convenience functions.
Caching support will be added in future work after unit tests have been added.

* extlib: add PEAR HTTP_Request2 0.4.1 alpha
* extlib: update PEAR Net_URL2 to 0.3.0 beta for HTTP_Request2 compatibility
* moved direct usage of CURL and file_get_contents to HTTPClient class, excluding external-sourced libraries

Note some plugins haven't been tested yet.
2009-11-02 06:31:28 -08:00
Evan Prodromou
b73d6d18ba Merge branch '0.9.x' into location 2009-10-28 17:24:53 -04:00
Craig Andrews
0b4390e7f2 Make email domain checking optional, as some statusnet installations (such as those behind restrictive corporate firewalls, or on home systems on restrictive connections) cannot connect to any mail
systems, and this check will always fail.
2009-10-26 10:31:12 -04:00
Evan Prodromou
7b227fd1c0 start getting locations from remote services 2009-10-21 22:47:19 -04:00
Evan Prodromou
4e639b8769 Merge branch '0.9.x' of git@gitorious.org:statusnet/mainline into 0.9.x 2009-10-21 22:41:42 -04:00
Evan Prodromou
0df85711d6 reformat default.php 2009-10-21 22:41:23 -04:00
Zach Copley
109a54c4f0 Changed config flag for importing friends' timeline and added some comments 2009-10-17 00:32:02 +00:00
Evan Prodromou
4f5badda94 remove inboxes option 2009-10-13 17:38:27 -04:00
Craig Andrews
f719720e8a Merge remote branch 'statusnet/0.8.x' into 0.9.x
Conflicts:
	lib/common.php
2009-10-11 13:39:10 -04:00
Evan Prodromou
d103522ff3 check the schema 2009-10-01 15:11:12 -04:00
Evan Prodromou
ecb75561af Move configuration defaults to new module and variable
Moved the default values of $config to $default. The code for
setting up $default is moved to lib/default.php.
2009-09-24 22:25:30 -04:00