Commit Graph

578 Commits

Author SHA1 Message Date
Brion Vibber
ecb009bcf5 Merge branch 'testing' of git@gitorious.org:statusnet/mainline into 0.9.x 2010-03-24 13:57:20 -07:00
Brion Vibber
13d59e0c76 fixup_deletions.php script to look for notices posted by now-deleted profiles and remove them. 2010-03-23 17:24:01 -07:00
Brion Vibber
a5c78a6f1a Merge branch 'testing' of git@gitorious.org:statusnet/mainline into 0.9.x 2010-03-23 08:21:32 -07:00
Zach Copley
b8e97ac709 Some initial media parsing
- Activity now returns a list of activity objects
- Processing of photo objects
2010-03-22 18:55:17 -07:00
Brion Vibber
3e2e88b0df Merge branch 'testing' of git@gitorious.org:statusnet/mainline into 0.9.x 2010-03-18 17:19:28 -07:00
Brion Vibber
0fe82bd0c9 Runtime check for known bad PHP versions with 64-bit stream_select() bug in xmppdaemon.php. 2010-03-18 11:08:27 -07:00
Brion Vibber
67f2f01c5e Merge branch 'testing' into 0.9.x 2010-03-17 12:15:40 -07:00
Brion Vibber
ca5612c451 Merge branch 'master' into testing 2010-03-17 12:15:12 -07:00
Brion Vibber
3e2b806e75 Add scripts/docgen.php to build basic doxygen HTML docs from doc comments, either for core or a given plugin.
Nothing too fancy yet; style and layout needs some loving!
2010-03-15 09:42:25 -07:00
Brion Vibber
6828567ed0 Add forgotten scripts/fixup_files.php to clean up "the h bug" 2010-03-11 11:52:19 -08:00
Brion Vibber
6d73fc1d54 Merge branch 'testing' of git@gitorious.org:statusnet/mainline into 0.9.x 2010-03-10 11:57:38 -08:00
Brion Vibber
54ff17010b Merge branch 'master' of git@gitorious.org:statusnet/mainline into testing 2010-03-10 11:56:51 -08:00
Brion Vibber
532e486a93 Detect when queuedaemon/xmppdaemon parent processes die and kill the child processes.
Keeps stray daemon subprocesses from floating around when we kill the parents via a signal!

Accomplished by opening a bidirectional pipe in the parent process; the children close out the writer end and keep the reader in their open sockets list. When the parent dies, the children see that the socket's been closed out and can perform an orderly shutdown.
2010-03-10 11:54:00 -08:00
Brion Vibber
6ea7e1c06e Merge branch 'testing' into 0.9.x 2010-03-09 14:20:50 -08:00
Brion Vibber
971f1f64f1 Added scripts/command.php, can be used to run commands such as subscription on behalf of users.
This includes whatever support for extended command parsing plugins may have added.

Example:
  ./scripts/command.php -nbrionv sub update@status.net
2010-03-09 13:41:05 -08:00
Brion Vibber
6c4ade4251 Merge branch 'testing' into 0.9.x 2010-03-09 11:27:53 -08:00
Evan Prodromou
61434ebaa2 a script to flush site 2010-03-09 10:55:48 -05:00
Evan Prodromou
7faf6ec755 add a script to import Twitter atom feed as notices 2010-03-07 23:42:22 -05:00
Rasmus Lerdorf
086d517b87 Fix a few typos 2010-03-04 23:30:04 -05:00
Brion Vibber
f969d6349c Merge branch 'testing' into 0.9.x
Conflicts:
	db/08to09.sql
2010-03-04 10:16:59 -08:00
James Walker
02f49193d5 adding checkschema to setup script 2010-03-04 11:49:19 -05:00
Brion Vibber
3c4ead4996 Merge branch 'testing' of gitorious.org:statusnet/mainline into 0.9.x
Conflicts:
	db/08to09.sql
	locale/statusnet.po
	scripts/update_po_templates.php
2010-03-01 14:59:26 -08:00
Brion Vibber
00f724d0e6 Localization tweak: include doc comments marked as 'TRANS' in .po file output; these should now get automatically pulled through to the TranslateWiki interface as translator help hints. 2010-03-01 14:10:13 -08:00
Siebrand Mazeland
87c17bc0f0 Also extract _m() used in core.
Signed-off-by: Siebrand Mazeland <s.mazeland@xs4all.nl>
2010-03-01 15:47:43 +01:00
Brion Vibber
9b366547d7 Merge branch 'testing' of gitorious.org:statusnet/mainline into 0.9.x 2010-02-26 12:33:50 -08:00
Evan Prodromou
6781f95c73 Merge branch 'blacklistplus' into testing
Conflicts:
	EVENTS.txt
2010-02-26 12:40:30 -05:00
Brion Vibber
8dfc8f1635 Merge branch 'testing' into 0.9.x 2010-02-26 09:35:28 -08:00
Evan Prodromou
200a32b922 don't choke on dupelimit in createsim.php 2010-02-26 12:19:33 -05:00
Brion Vibber
c8ddcbe0d3 init_conversation.php script to copy old notice conversations into the conversation table 2010-02-25 17:56:01 -08:00
Brion Vibber
4a71753f20 Merge branch 'master' of gitorious.org:statusnet/mainline into testing 2010-02-25 09:28:19 -08:00
Brion Vibber
99c0e16844 Merge commit 'refs/merge-requests/121' of git://gitorious.org/statusnet/mainline into integration 2010-02-25 09:12:45 -08:00
Brion Vibber
66373f8e5a Merge commit 'refs/merge-requests/2220' of git://gitorious.org/statusnet/mainline into integration 2010-02-25 09:11:07 -08:00
Brion Vibber
8914b69d50 Merge branch 'testing' of gitorious.org:statusnet/mainline into 0.9.x 2010-02-24 16:35:57 -08:00
Brion Vibber
3b7084c6dd Fix update_po_templates.php to support the plural and context variants of _m() in plugins 2010-02-24 16:34:45 -08:00
Brion Vibber
85528ccb1f Merge branch 'testing' of gitorious.org:statusnet/mainline into 0.9.x 2010-02-21 11:12:56 -08:00
Christopher Vollick
e717cba19c Add Script To Update Group Avatar URLs
Similar to scripts/updateavatarurl.php.

Works for groups.
Again, I had to do some weird thing because using clone screwed up the find() iteration.
2010-02-18 13:44:16 -05:00
Christopher Vollick
ddef800ec9 Update Avatar URL Did Weird Stuff.
It was only finding the first two avatars and then thinking it was done.

I'm not entirely sure why it was doing that.
I think maybe all the cloning made it forget where it was or something.

Either way, it seems to work now, and really uses less memory.
2010-02-18 12:14:34 -05:00
Brion Vibber
07f145049e Merge branch 'master' into testing 2010-02-17 16:52:13 -08:00
Brion Vibber
ce6be4f836 Queues: redid the breakout control model so we can start up and subscribe to queues without running through the complete site list, which is ok at 1k sites but too slow at 10k.
All breakout queues that we're going to need to listen to now need to be explicitly listed in $config['queue']['breakout'].

Until XMPP is moved to component model, this setting will let the individual processes work with their own queues:
$config['queue']['breakout'][] = 'xmpp/xmppout/' . $config['site']['nickname'];
2010-02-17 16:49:00 -08:00
Brion Vibber
d5cbfe8071 Merge branch 'testing' into 0.9.x
Conflicts:
	lib/iomaster.php
2010-02-16 09:25:09 -08:00
Brion Vibber
81b6b58e33 Merge branch 'master' into testing
Conflicts:
	lib/stompqueuemanager.php
2010-02-16 09:22:02 -08:00
Brion Vibber
c74aea589d Stomp queue restructuring for mass scalability:
- Multiplexing queues into groups and for multiple sites.
- Sharing vs breakout configurable per site and per queue via $config['queue']['breakout']
- Detect how many times a message is redelivered, discard if it's killed too many daemons
 - count configurable with $config['queue']['max_retries']
 - can dump the items to files in $config['queue']['dead_letter_dir']

Queue daemon memory & resource leak fixes:
- avoid unnecessary reconnections to memcached server (switch persistent connections back in on second initialization, assuming it's child process)
- monkey-patch for leaky .ini loads in DB_DataObject::databaseStructure() - was leaking 200k per active switch
- applied leak fixes to Status_network as well, using intermediate base Safe_DataObject for both it and Memcache_DataObject

Misc queue fixes:
- correct handling of child processes exiting due to signal termination instead of regular exit
- shutdown instead of infinite respawn loop if we're already past the soft memory limit at startup
- Added --all option for xmppdaemon... still opens one xmpp connection per site that has xmpp active

Cache updates:
- add Cache::increment() method with native support for memcached atomic increment
2010-02-16 09:16:51 -08:00
Brion Vibber
b2e8d8407c Merge branch 'testing' of git@gitorious.org:statusnet/mainline into 0.9.x 2010-02-08 15:48:52 -08:00
Brion Vibber
4e6f587f86 Merge branch 'master' of git@gitorious.org:statusnet/mainline into testing 2010-02-08 15:47:55 -08:00
Brion Vibber
96ef4435b6 Allow scripts/decache.php to blow out cache for objects that don't exist (anymore).
May miss keys other than the given or primary key, but should work for a lot of common cases where a bad entry's been removed from DB but lingers in cache.
2010-02-08 15:32:20 -08:00
Brion Vibber
c59eee72a8 Merge branch 'testing' of gitorious.org:statusnet/mainline into 0.9.x 2010-02-04 12:23:56 -08:00
Brion Vibber
a020e23086 Merge branch 'master' of gitorious.org:statusnet/mainline into testing 2010-02-04 11:56:07 -08:00
Evan Prodromou
7a7e2162dd Script to update profile URLs 2010-02-03 14:58:29 -05:00
Evan Prodromou
0e0beea5aa clearcache.php column flag was conflicting with default flag 2010-02-03 12:13:20 -05:00
Evan Prodromou
aee73d87ee showcache.php column flag was conflicting with default flag 2010-02-03 12:13:07 -05:00
Evan Prodromou
644a916383 change 'sitetype' to 'siteplan' to use consistent language 2010-02-03 11:55:54 -05:00
Brion Vibber
f4c81fa1e1 Merge branch 'testing' of git@gitorious.org:statusnet/mainline into 0.9.x 2010-02-02 09:57:50 -08:00
Brion Vibber
5f14301c55 setconfig.php: list all current settings if no parameters given 2010-02-02 06:15:01 -08:00
Brion Vibber
4f6052d8d8 Apply xopher's fix to add 'sitetype' parameter to setup_status_network.sh, exposed to the email 2010-02-02 05:53:05 -08:00
Brion Vibber
c14ac57b19 Merge branch 'testing' of git@gitorious.org:statusnet/mainline into 0.9.x 2010-02-01 10:30:45 -08:00
Evan Prodromou
5d2c083899 Add a script to clear the cache for a given key
Like showcache.php, clearcache.php clears the cache for a given key.
2010-01-31 22:58:29 -05:00
Evan Prodromou
779204b194 Merge branch 'testing' into 0.9.x
Conflicts:
	actions/apioauthauthorize.php
2010-01-31 15:27:58 -05:00
Evan Prodromou
2a054a50fb live fast, die young in bash scripts 2010-01-29 15:33:35 -05:00
Brion Vibber
288dc3452f Log exceptions from queuedaemon.php if they're not already caught 2010-01-28 22:05:14 -08:00
Brion Vibber
864ce8e276 Fixes for status_network db object .ini and tag setter script 2010-01-28 20:09:17 -08:00
Evan Prodromou
4a0413c027 Add a script to set tags for sites 2010-01-28 20:00:33 -08:00
Brion Vibber
fe531dfc6c Shuffle params on setup_status_network; adding fullname and pushing tags up 2010-01-27 16:03:51 -08:00
Brion Vibber
e7a5471287 add additional post-install shell script option for setup_status_network.sh to do any other site-specific setup 2010-01-27 15:37:18 -08:00
Brion Vibber
06cd335897 Add scripts/sendemail.php to send email to a user's address. Updated setup_status_network.sh to create a user with the site's nick, accept site tags, and send a mail to the user (if a template is set)
Email and tag params added to the end:
setup_status_net.sh mysite 'My Site' 'owner@example.com' '1user'

(If multiple tags are needed, separate them with a pipe "|". Be sure to quote properly!)

New parameters for setup.cfg need to be set:
export PHPBASE=/var/www/statusnet
export WILDCARD=example.net
export MAILTEMPLATE=/etc/statusnet/newsite-mail.txt
export MAILSUBJECT="Your new StatusNet site"

$PHPBASE is the base dir for a callable StatusNet install, used to run command-line scripts for user setup.
$WILDCARD is the wildcard domain, needed to build a full server name to pass into command-line scripts.
$MAILTEMPLATE points to a file containing an e-mail message template. '$nickname', '$sitename', and '$userpass' can be used in the template for substitution.
$MAILSUBJECT is the subject line for said email.

To skip sending an email on creation, leave $MAILTEMPLATE blank or point to a non-existing file.
2010-01-27 15:12:19 -08:00
Zach Copley
78079f34e2 Merge branch 'testing' into -1.9.x
* testing: (130 commits)
  HTTP auth provided is evaluated even if it's not required
  Rename rc3to09.sql to rc3torc4.sql to avoid confusion if we add a last-minute change after this!
  Add new oauth tables and modifications to 'consumer' table for rc4
  Centred leaderboard ad
  camelcase the uap param names
  move leaderboard to after the header
  Moved rectangle ad into aside and leaderboard to the right in header.
  Aligning wide skyscraper to the right instead of left
  CSS ids and classes fixed in UAPPlugin
  wrong height for rectangle in BlankAd
  Add the moved BlankAdPlugin
  make BlankAd dir and change to use a 1x1 image
  move BlankAdPlugin to its own dir
  Add BlankAdPlugin to test ad layout in different themes
  make uapplugin an abstract class
  move UAP plugin to core
  Lowercased switch cases in UAP Plugin
  Plugin for Universal Ad Package. Outputs four most widely used ad types.
  Add persistent:true property to Stomp messages so ActiveMQ doesn't decide to discard them even though persistence is enabled on the broker. :) (Thanks Aric!)
  quick fix: use common_path() on realtime update JS so it works with the new JS path code (will pull from main server for now)
  ...

Conflicts:
	actions/apioauthaccesstoken.php
	actions/apioauthauthorize.php
	actions/apioauthrequesttoken.php
	actions/editapplication.php
	actions/newapplication.php
	lib/apiauth.php
	lib/queuemanager.php
	lib/router.php
2010-01-27 14:27:22 -08:00
Brion Vibber
58be61b641 Control channel for queue daemons to request graceful shutdown, restart, or update to listen to a newly added or reconfigured site.
queuectl.php --update -s<site>
  queuectl.php --stop
  queuectl.php --restart

Default control channel is /topic/statusnet-control. For external utilities to send a site update ping direct to the queue server, connect via Stomp and send a message formatted thus:

  update:<nickname>

(Nickname here, *not* server hostname! The rest of the queues will be updated to use nicknames later.)

Note that all currently-connected queue daemons will get these notifications, including both queuedaemon.php and xmppdaemon.php. (XMPP will ignore site update requests for sites that it's not handling.)

Limitations:
* only implemented for stomp queue manager so far
* --update may not yet handle a changed server name properly
* --restart won't reload PHP code files that were already loaded at startup. Still need to stop and restart the daemons from 'outside' when updating code base.
2010-01-26 11:49:49 -08:00
Brion Vibber
ee4ca8f260 quick fix to console.php: don't save blank lines into readline history 2010-01-25 09:41:40 -08:00
Brion Vibber
055a00bcae drop now-unused --skip-xmpp and --xmpp-only options from queuedaemon.php 2010-01-25 09:36:20 -08:00
Evan Prodromou
c8bc598cfd Merge branch 'testing' into 0.9.x 2010-01-22 14:18:43 -05:00
Evan Prodromou
e666433eb4 Merge branch 'master' into 0.9.x 2010-01-22 14:18:40 -05:00
Brion Vibber
0e852def6a XMPP queued output & initial retooling of DB queue manager to support non-Notice objects.
Queue handlers for XMPP individual & firehose output now send their XML stanzas
to another output queue instead of connecting directly to the chat server. This
lets us have as many general processing threads as we need, while all actual
XMPP input and output go through a single daemon with a single connection open.

This avoids problems with multiple connected resources:
* multiple windows shown in some chat clients (psi, gajim, kopete)
* extra load on server
* incoming message delivery forwarding issues

Database changes:
* queue_item drops 'notice_id' in favor of a 'frame' blob.
  This is based on Craig Andrews' work branch to generalize queues to take any
  object, but conservatively leaving out the serialization for now.
  Table updater (preserves any existing queued items) in db/rc3to09.sql

Code changes to watch out for:
* Queue handlers should now define a handle() method instead of handle_notice()
* QueueDaemon and XmppDaemon now share common i/o (IoMaster) and respawning
  thread management (RespawningDaemon) infrastructure.
* The polling XmppConfirmManager has been dropped, as the message is queued
  directly when saving IM settings.
* Enable $config['queue']['debug_memory'] to output current memory usage at
  each run through the event loop to watch for memory leaks

To do:
* Adapt XMPP i/o to component connection mode for multi-site support.
* XMPP input can also be broken out to a queue, which would allow the actual
  notice save etc to be handled by general queue threads.
* Make sure there are no problems with simply pushing serialized Notice objects
  to queues.
* Find a way to improve interactive performance of the database-backed queue
  handler; polling is pretty painful to XMPP.
* Possibly redo the way QueueHandlers are injected into a QueueManager. The
  grouping used to split out the XMPP output queue is a bit awkward.
2010-01-21 22:40:35 -08:00
Brion Vibber
0bb23e6fd7 drop debug line from xmppdaemon.php, we're done debugging that 2010-01-21 16:34:26 -08:00
Brion Vibber
71df23642c Extra debug instrumentation for xmppdaemon 2010-01-19 12:26:59 -08:00
Eric Helgeson
e3dc45d103 Merge commit 'refs/merge-requests/117' of git://gitorious.org/statusnet/mainline into integration 2010-01-16 17:42:53 -05:00
Brion Vibber
598072468c --xmpp-only hack for queuedaemon.php to run separate queue daemon with only xmpp threads 2010-01-15 11:13:06 -08:00
Brion Vibber
58bc33850a temporary --skip-xmpp flag on queuedaemon.php, allows to run queue daemons but skip subscription to xmpp-based queues
(still working on making these behave gracefully when server is down)
2010-01-14 15:32:37 -08:00
Brion Vibber
35a2f8de4d actually use the user id instead of 0 ;) 2010-01-13 20:46:44 -08:00
Brion Vibber
82f2fe8cca in case we have cached 'fake' inboxes, make sure we rebuild them running initializeinbox.php 2010-01-13 20:30:42 -08:00
Brion Vibber
ec873922e3 Gracefully skip missing user entries when initializing inboxes in bulk 2010-01-13 20:11:00 -08:00
Evan Prodromou
aa23698553 accept file for initializeinbox.php 2010-01-13 18:28:08 -08:00
Siebrand Mazeland
519e1e6250 Update generation of YAML files
* need lower case in some places
* update target path

Not in production yet at translatewiki.net - need some Translate extension code changes to get YAML config support for Gettext first.
2010-01-13 23:33:05 +01:00
Christopher Vollick
1e8707d29a Include Unconfirmed Addresses Too.
Looks like there are other places in the db where email addresses can go.
Found them now!
2010-01-13 10:27:50 -05:00
Evan Prodromou
b25e59a11f add a script for initializing inboxes 2010-01-13 00:25:43 -08:00
Evan Prodromou
436b8c845e Merge branch '0.9.x' into inblob 2010-01-12 23:53:52 -08:00
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Evan Prodromou
ecb3abf84c Merge branch '0.9.x' into inblob 2010-01-11 16:28:27 -08:00
Brion Vibber
be1ac6678d fix long options on deleteuser.php 2010-01-11 13:24:40 -08:00
Siebrand Mazeland
4af6b7f5c3 Lots of tiny message changes.
* Mostly punctuation updates so that the same message is used consistently in all of StatusNet.
* Some cases of "Title Case" removed, because that does not appear to be used consistently.
2010-01-10 12:26:24 +01:00
Evan Prodromou
a180658a3b Merge branch 'inblob' of git@gitorious.org:~evan/statusnet/evans-mainline into inblob
Conflicts:
	classes/Inbox.php
	classes/Notice.php
	classes/Notice_inbox.php
2010-01-09 14:19:59 -08:00
Evan Prodromou
2272438698 remove triminboxes.php; it's no longer used 2010-01-09 10:02:07 -08:00
Evan Prodromou
72934e9f50 Revert "Replace Notice_inbox with Inbox"
We use Notice_inbox to transition to Inbox.

This reverts commit 7640d3f07b.
2010-01-09 10:02:07 -08:00
Evan Prodromou
7ec27b657e Replace Notice_inbox with Inbox 2010-01-09 10:02:07 -08:00
Craig Andrews
055f3fdddb Add an IMAP daemon so StatusNet can process incoming user posts via catch-all mailbox (in addition to the pre-existing script alias method) 2010-01-08 18:52:43 -05:00
Brion Vibber
6f5b765c97 suppress notice for undefined prompt variable when console.php is used from non-interactive terminal 2010-01-06 13:08:56 -08:00
Zach Copley
ff26b8d88b Add an RSSCloud queue handler daemon 2010-01-05 23:19:13 -08:00
Evan Prodromou
0c31c3d80c free some memory in createsim.php 2010-01-04 22:49:09 -10:00
Evan Prodromou
31b62bf781 Merge branch 'batchlocation' 2010-01-04 15:10:43 -10:00
Evan Prodromou
55ba858e8c Script to update the location ID for users
Since we added locations to the database, some users may have
location strings in their profiles but not structured locations. This
script updates the locations for single users or for all users.
2009-12-31 12:38:58 -10:00
Evan Prodromou
7640d3f07b Replace Notice_inbox with Inbox 2009-12-30 09:03:06 -10:00
Brion Vibber
45c9d3d729 Add progress output and optional --sleep-time parameter to triminboxes.php 2009-12-29 14:17:03 -08:00