wamo/gnu-social
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
83d1997f59
gnu-social/scripts
History
Brion Vibber 55b1f3d84c Scalability work on user backup stream generation. 
UserActivityStream -- used to create a full activity stream including subscriptions, favorites, notices, etc -- normally buffers everything into memory at once. This is infeasible for accounts with long histories of serious usage; it can take tens of seconds just to pull all records from the database, and working with them all in memory is very likely to hit resource limits.
This commit adds an alternate mode for this class which avoids pulling notices until during the actual output. Instead of pre-sorting and buffering all the notices, empty spaces between the other activities are filled in with notices as we're making output. This means more smaller queries spread out during operations, and less stuff kept in memory.

Callers (backupaccount action, and backupuser.php) which can stream their output pass an $outputMode param of UserActivityStream::OUTPUT_RAW, and during getString() it'll send straight to output as well as slurping the notices in this extra funky fashion.
Other callers will let it default to the OUTPUT_STRING mode, which keeps the previous behavior.

There should be a better way to do this, swapping out the stringer output for raw output more consitently.
2011-02-25 12:15:38 -08:00
..
allsites.php Revert "Open tags should have closing tags" 2009-10-08 11:41:39 -04:00
apidocs.config Some initial Doxygen-based documentation for the API 2010-04-22 13:11:49 -07:00
backupuser.php Scalability work on user backup stream generation. 2011-02-25 12:15:38 -08:00
checkschema.php make checkschema.php executable (which makes sense, cause the README says to run it :-) ) 2009-11-30 17:00:00 -05:00
clear_jabber.php clear_jabber.php script to clear confirmed jabber/xmpp addresses from one or more accounts 2010-11-15 13:26:42 -08:00
clearcache.php clearcache.php column flag was conflicting with default flag 2010-02-03 12:13:20 -05:00
cleardb.sh clear and rebuild db scripts 2008-09-22 15:58:24 -04:00
command.php add an option to run a command as the site owner 2010-03-28 14:38:07 -04:00
commandline.inc printfv and printfq functions in commandline.inc 2010-09-21 14:06:39 -04:00
console.php quick fix to console.php: don't save blank lines into readline history 2010-01-25 09:41:40 -08:00
createsim.php don't choke on dupelimit in createsim.php 2010-02-26 12:19:33 -05:00
decache.php Allow scripts/decache.php to blow out cache for objects that don't exist (anymore). 2010-02-08 15:32:20 -08:00
delete_status_network.sh Fixes for delete_status_network.sh: 2010-11-22 11:10:10 -08:00
deletegroup.php scripts/deletegroup.php -- basic CLI script to delete a group by id or local nickname. Like deleteuser.php, this can be used in batch runs by providing the -y override. 2010-11-18 14:40:05 -08:00
deleteuser.php fix long options on deleteuser.php 2010-01-11 13:24:40 -08:00
docgen.php docgen.php no longer depends on having a working installation; uses Console_GetOpt directly and regexes the STATUSNET_VERSION from common.php 2010-04-21 20:20:43 +02:00
doxygen.tmpl Add scripts/docgen.php to build basic doxygen HTML docs from doc comments, either for core or a given plugin. 2010-03-15 09:42:25 -07:00
fixup_blocks.php fixup_blocks.php: finds any stray subscriptions in violation of blocks, and removes them. 2010-04-07 11:40:16 -07:00
fixup_conversations.php define LACONICA for compatibility on scripts 2009-10-13 13:34:28 -04:00
fixup_deletions.php fixup_deletions.php script to look for notices posted by now-deleted profiles and remove them. 2010-03-23 17:24:01 -07:00
fixup_files.php Add forgotten scripts/fixup_files.php to clean up "the h bug" 2010-03-11 11:52:19 -08:00
fixup_group_uri.php scripts/fixup_group_uri.php to fill in empty user_group.uri entries; needed before changing domain names on sites that are hosting groups for remote users 2010-10-04 15:13:27 -07:00
fixup_hashtags.php Revert "Open tags should have closing tags" 2009-10-08 11:41:39 -04:00
fixup_inboxes.php Revert "Open tags should have closing tags" 2009-10-08 11:41:39 -04:00
fixup_notices_rendered.php define LACONICA and accept LACONICA for backwards compatibility 2009-08-26 10:41:36 -04:00
fixup_replies.php Revert "Open tags should have closing tags" 2009-10-08 11:41:39 -04:00
fixup_status_network.php try/catch just in case.. 2010-07-28 12:04:37 -04:00
fixup_utf8.php Fix a few typos 2010-03-04 23:30:04 -05:00
flushrouter.php a quick script to flush the router from cache if needed 2010-11-30 10:27:32 -05:00
flushsite.php a script to flush site 2010-03-09 10:55:48 -05:00
getpiddir.php Revert "Open tags should have closing tags" 2009-10-08 11:41:39 -04:00
getvaliddaemons.php Major refactoring of queue handlers to support running multiple sites in one daemon. 2010-01-12 20:45:09 -08:00
handlequeued.php XMPP queued output & initial retooling of DB queue manager to support non-Notice objects. 2010-01-21 22:40:35 -08:00
importtwitteratom.php Always specify UTF-8 targt charset for html_entity_decode(); default is 8-bit ISO-8859-1 which causes things to break when we later pass them through things that expect to work with UTF-8. For instance, running through preg_replace() with the /u option results in NULL, leading to problems with OStatus and SubMirror generating their plaintext versions and doing length-cropping. 2010-09-30 11:29:31 -07:00
init_conversation.php init_conversation.php script to copy old notice conversations into the conversation table 2010-02-25 17:56:01 -08:00
initializeinbox.php actually use the user id instead of 0 ;) 2010-01-13 20:46:44 -08:00
install_cli.php Always specify UTF-8 targt charset for html_entity_decode(); default is 8-bit ISO-8859-1 which causes things to break when we later pass them through things that expect to work with UTF-8. For instance, running through preg_replace() with the /u option results in NULL, leading to problems with OStatus and SubMirror generating their plaintext versions and doing length-cropping. 2010-09-30 11:29:31 -07:00
maildaemon.php Add an IMAP daemon so StatusNet can process incoming user posts via catch-all mailbox (in addition to the pre-existing script alias method) 2010-01-08 18:52:43 -05:00
make-release.php Quickie script to aid in building release tarballs -- encapsulates the archive-i18n gen-changelog gen-retar steps. 2010-10-29 14:34:27 -07:00
makegroupadmin.php Revert "* [Cc]an't -> [Cc]annot" 2009-11-08 23:28:51 +01:00
moveuser.php Add a scary 'experimental feture' warning & are-you-sure prompt on moveuser.php 2011-01-19 15:13:46 -08:00
queuectl.php Control channel for queue daemons to request graceful shutdown, restart, or update to listen to a newly added or reconfigured site. 2010-01-26 11:49:49 -08:00
queuedaemon.php Detect when queuedaemon/xmppdaemon parent processes die and kill the child processes. 2010-03-10 11:54:00 -08:00
rebuilddb_psql.sh global search and replace for laconica -> statusnet 2009-08-25 18:53:24 -04:00
rebuilddb.sh clear and rebuild db scripts 2008-09-22 15:58:24 -04:00
registeruser.php Revert "* [Cc]an't -> [Cc]annot" 2009-11-08 23:28:51 +01:00
reportsnapshot.php Revert "Open tags should have closing tags" 2009-10-08 11:41:39 -04:00
restoreuser.php Make restoreuser use new FeedImporter queue handler 2010-12-17 16:27:20 -05:00
SearchMonkey-MrH.0.txt Updated URL patterns for identica Profiles for YahooSearchMonkey app. 2009-03-09 19:43:46 +00:00
SearchMonkey-Om3.0.txt Updated URL patterns for identica Profiles for YahooSearchMonkey app. 2009-03-09 19:43:46 +00:00
SearchMonkey-yQP.0.txt Updated URL patterns for identica Profiles for YahooSearchMonkey app. 2009-03-09 19:43:46 +00:00
sendemail.php Add scripts/sendemail.php to send email to a user's address. Updated setup_status_network.sh to create a user with the site's nick, accept site tags, and send a mail to the user (if a template is set) 2010-01-27 15:12:19 -08:00
sessiongc.php Revert "Open tags should have closing tags" 2009-10-08 11:41:39 -04:00
setconfig.php Allow setconfig.php to Print Entire Config 2010-04-15 23:02:29 +02:00
setpassword.php Revert "Open tags should have closing tags" 2009-10-08 11:41:39 -04:00
settag.php normalizing tags for status_network 2010-07-20 17:34:58 -07:00
setup_status_network.sh fix typo in the comment (for better copy & pasting) 2010-09-09 13:14:52 -04:00
setup.cfg.sample add additional post-install shell script option for setup_status_network.sh to do any other site-specific setup 2010-01-27 15:37:18 -08:00
showcache.php showcache.php column flag was conflicting with default flag 2010-02-03 12:13:07 -05:00
showplugins.php showplugins.php: dump list of activated plugins and their settings on this site 2010-04-12 11:50:42 -07:00
showtable.php statusize schema-related modules 2009-09-23 09:20:04 -04:00
sitemap.php Revert "* [Cc]an't -> [Cc]annot" 2009-11-08 23:28:51 +01:00
startdaemons.sh have to provide full path for daemons 2009-09-21 14:42:20 -04:00
statusnet.spec last scrub of Laconica -> StatusNet 2009-08-26 03:33:43 -04:00
stopdaemons.sh Add an RSSCloud queue handler daemon 2010-01-05 23:19:13 -08:00
strip_geo.php Fix for strip_geo's decaching; also added --all option to run over all notices by given profile to help in fixing up cache inconsistencies 2010-04-09 11:37:29 -07:00
uncache_users.php Revert "Open tags should have closing tags" 2009-10-08 11:41:39 -04:00
update_po_templates.php Remove do_translatewiki_plugin(). No longer needed. 2010-09-18 00:23:38 +02:00
update_translations.php Update group ID for translations export. 2010-09-18 23:50:36 +02:00
updateavatarurl_group.php Add Script To Update Group Avatar URLs 2010-02-18 13:44:16 -05:00
updateavatarurl.php Update Avatar URL Did Weird Stuff. 2010-02-18 12:14:34 -05:00
updatelocation.php Script to update the location ID for users 2009-12-31 12:38:58 -10:00
updateprofileurl.php Script to update profile URLs 2010-02-03 14:58:29 -05:00
useremail.php Include Unconfirmed Addresses Too. 2010-01-13 10:27:50 -05:00
userrole.php userrole.php will take a profile id for remote profiles 2010-10-26 11:19:47 -04:00
xmppdaemon.php Runtime check for known bad PHP versions with 64-bit stream_select() bug in xmppdaemon.php. 2010-03-18 11:08:27 -07:00