Commit Graph

50 Commits

Author SHA1 Message Date
Brion Vibber
e76028b629 Work in progress: starting on new TwitterDaemon using the Site Streams API -- code is incomplete, pulling bits from streamtest.php pending a chance to test the actual site-streams mode 2010-10-28 18:26:48 -07:00
Brion Vibber
0eaa26476c Split the guts of Twitter status -> notice import from twitterstatusfetcher daemon into TwitterImport class which can be called from other places, letting us reuse code for the streaming API. 2010-10-05 13:57:32 -07:00
Brion Vibber
1acc7d66c6 Always specify UTF-8 targt charset for html_entity_decode(); default is 8-bit ISO-8859-1 which causes things to break when we later pass them through things that expect to work with UTF-8. For instance, running through preg_replace() with the /u option results in NULL, leading to problems with OStatus and SubMirror generating their plaintext versions and doing length-cropping. 2010-09-30 11:29:31 -07:00
Evan Prodromou
78ed0348b0 Merge remote branch 'gitorious/0.9.x' into 0.9.x 2010-09-21 14:07:48 -04:00
Siebrand Mazeland
ba6538ed78 * i18n/L10n updates
* whitespace updates.
2010-09-18 17:45:18 +02:00
Evan Prodromou
398df867de use capturing parens for matching at-mentions in twitterstatusfetcher 2010-09-10 09:21:01 -04:00
Evan Prodromou
deb8f2a38b make our own twitter links if Twitter doesn't give us entities 2010-09-10 09:10:53 -04:00
Evan Prodromou
5c98dfd58c text substitution is Unicode-aware for twitterstatusfetcher 2010-09-07 15:54:06 -04:00
Evan Prodromou
d37f0da077 debug statements for setting/getting last id 2010-09-07 11:53:26 -04:00
Evan Prodromou
9793e4ceb4 Store foreign ID for synch info, not user ID 2010-09-07 10:53:11 -04:00
Evan Prodromou
21bea14140 remove data dump of retrieved statuses 2010-09-07 04:42:23 -04:00
Evan Prodromou
e53e152d6f use and save since_id parameter for home_timeline 2010-09-07 04:22:55 -04:00
Evan Prodromou
a46ba0b548 save Twitter mentions as StatusNet replies 2010-09-07 04:00:59 -04:00
Evan Prodromou
c2c6acbfd2 add back in # and @ for links 2010-09-07 04:00:59 -04:00
Evan Prodromou
2de6d0b03d add Twitter-approved links to Twitter statuses 2010-09-07 04:00:59 -04:00
Evan Prodromou
702d36257c truncate retweeted stuff if it's too long 2010-09-07 04:00:58 -04:00
Evan Prodromou
9847bffe99 correctly mark repeats from Twitter as non-local 2010-09-07 04:00:58 -04:00
Evan Prodromou
e20dc9fec3 correctly check for local notice to status mappings when notice originated here 2010-09-07 04:00:58 -04:00
Evan Prodromou
e9e06879bb use Notice_to_status to check for duplicates 2010-09-07 04:00:58 -04:00
Evan Prodromou
a1981770ce Save notice-to-status mapping in its own table
Introduce a table mapping notices to Twitter statuses. Initialize
this table at checkSchema() time. Save the mapping when we push
or pull statuses. Use the table to determine if a notice has a
Twitter equivalent.
2010-09-07 04:00:58 -04:00
Evan Prodromou
928239ecf8 do our own repeating so we can pass in a uri 2010-09-07 04:00:58 -04:00
Evan Prodromou
1bf4972cc0 remove require_once for disappeared TwitterBasicAuthClient 2010-09-07 04:00:57 -04:00
Evan Prodromou
23dfadd9a6 remove basic auth code for Twitter since it's no longer supported 2010-09-07 04:00:57 -04:00
Evan Prodromou
a8c90d0243 switch twitterstatusfetcher from friends_timeline to home_timeline 2010-09-07 04:00:57 -04:00
Evan Prodromou
1a54291cc7 debug code to dump new status data 2010-09-07 04:00:57 -04:00
Evan Prodromou
184ee7fc1f move inbox insert code to status saver main loop 2010-09-07 04:00:57 -04:00
Evan Prodromou
20f6ea4840 some info notices in twitter status fetch for repeats and replies 2010-09-07 04:00:57 -04:00
Evan Prodromou
5651f79466 if something's a retweet, save it as a repeat in bridge 2010-09-07 04:00:57 -04:00
Evan Prodromou
4d70ba9597 save reply status for Twitter notices 2010-09-07 04:00:57 -04:00
Evan Prodromou
ed99fd51e8 don't save statuses from protected users 2010-09-07 04:00:57 -04:00
Zach Copley
5b7b38ab66 Some fixes to make the twitterstatusfetcher behave better in a multi-site configuration 2010-03-27 23:43:05 +00:00
root
982edc653f Another typo 2010-03-04 23:30:36 -05:00
Zach Copley
72f72d17db - Fix cache handling in TwitterStatusFetcher
- Other stability fixes
2010-02-04 01:53:08 +00:00
Evan Prodromou
436b8c845e Merge branch '0.9.x' into inblob 2010-01-12 23:53:52 -08:00
Brion Vibber
ec145b73fc Major refactoring of queue handlers to support running multiple sites in one daemon.
Key changes:
* Initialization code moved from common.php to StatusNet class;
  can now switch configurations during runtime.
* As a consequence, configuration files must now be idempotent...
  Be careful with constant, function or class definitions.
* Control structure for daemons/QueueManager/QueueHandler has been refactored;
  the run loop is now managed by IoMaster run via scripts/queuedaemon.php
  IoManager subclasses are woken to handle socket input or polling, and may
  cover multiple sites.
* Plugins can implement notice queue handlers more easily by registering a
  QueueHandler class; no more need to add a daemon.

The new QueueDaemon runs from scripts/queuedaemon.php:

* This replaces most of the old *handler.php scripts; they've been refactored
  to the bare handler classes.
* Spawns multiple child processes to spread load; defaults to CPU count on
  Linux and Mac OS X systems, or override with --threads=N
* When multithreaded, child processes are automatically respawned on failure.
* Threads gracefully shut down and restart when passing a soft memory limit
  (defaults to 90% of memory_limit), limiting damage from memory leaks.
* Support for UDP-based monitoring: http://www.gitorious.org/snqmon

Rough control flow diagram:
QueueDaemon -> IoMaster -> IoManager
                           QueueManager [listen or poll] -> QueueHandler
                           XmppManager [ping & keepalive]
                           XmppConfirmManager [poll updates]

Todo:

* Respawning features not currently available running single-threaded.
* When running single-site, configuration changes aren't picked up.
* New sites or config changes affecting queue subscriptions are not yet
  handled without a daemon restart.
* SNMP monitoring output to integrate with general tools (nagios, ganglia)
* Convert XMPP confirmation message sends to use stomp queue instead of polling
* Convert xmppdaemon.php to IoManager?
* Convert Twitter status, friends import polling daemons to IoManager
* Clean up some error reporting and failure modes
* May need to adjust queue priorities for best perf in backlog/flood cases

Detailed code history available in my daemon-work branch:
http://www.gitorious.org/~brion/statusnet/brion-fixes/commits/daemon-work
2010-01-12 20:45:09 -08:00
Evan Prodromou
2db840b2f5 use Inbox in twitterstatusfetcher 2010-01-09 14:02:01 -08:00
Zach Copley
eb0a5977ff Don't try to broadcast the notice if something went wrong. 2009-12-04 20:39:50 +00:00
Zach Copley
cff12603cf Make imported Twitter notices show up via real time plugins. 2009-12-04 20:17:42 +00:00
Zach Copley
fcf1c4e53f Remove useless debugging statement 2009-12-02 19:33:08 +00:00
Brion Vibber
088081675f Revert "Remove more contractions"
This reverts commit 5ab709b739.

Missed this one yesterday...
2009-11-09 20:01:46 +01:00
Siebrand Mazeland
b10f362ede Merge branch '0.9.x' of git://gitorious.org/statusnet/mainline into 0.9.x 2009-11-08 23:33:58 +01:00
Siebrand Mazeland
5ab709b739 Remove more contractions
* doesn't
* won't
* isn't
* don't
2009-11-08 23:32:15 +01:00
Brion Vibber
fc5002015b Revert "* [Cc]an't -> [Cc]annot"
This reverts commit 0ab17f382b.
2009-11-08 23:28:51 +01:00
Siebrand Mazeland
0ab17f382b * [Cc]an't -> [Cc]annot
* [Cc]ould't -> [Cc]ould not
2009-11-08 23:22:38 +01:00
Brion Vibber
5581143bee Rebuilt HTTPClient class as an extension of PEAR HTTP_Request2 package, adding redirect handling and convenience functions.
Caching support will be added in future work after unit tests have been added.

* extlib: add PEAR HTTP_Request2 0.4.1 alpha
* extlib: update PEAR Net_URL2 to 0.3.0 beta for HTTP_Request2 compatibility
* moved direct usage of CURL and file_get_contents to HTTPClient class, excluding external-sourced libraries
* adapted GeonamesPlugin for new HTTPResponse interface

Note some plugins haven't been fully tested yet.
2009-11-02 09:14:15 -08:00
Brion Vibber
b22fc5b74a Revert "Rebuilt HTTPClient class as an extension of PEAR HTTP_Request2 package, adding redirect handling and convenience functions."
Going to restructure a little more before finalizing this...

This reverts commit fa37967858.
2009-11-02 07:51:29 -08:00
Brion Vibber
fa37967858 Rebuilt HTTPClient class as an extension of PEAR HTTP_Request2 package, adding redirect handling and convenience functions.
Caching support will be added in future work after unit tests have been added.

* extlib: add PEAR HTTP_Request2 0.4.1 alpha
* extlib: update PEAR Net_URL2 to 0.3.0 beta for HTTP_Request2 compatibility
* moved direct usage of CURL and file_get_contents to HTTPClient class, excluding external-sourced libraries

Note some plugins haven't been tested yet.
2009-11-02 06:31:28 -08:00
Zach Copley
0fd8e758ad Make queuing and daemons work via events 2009-10-14 04:50:16 +00:00
Zach Copley
9d87313eae Merge branch '0.9.x' into pluginize-twitter-bridge
Conflicts:
	plugins/TwitterBridge/twitterauthorization.php
2009-09-08 16:02:57 -07:00
Zach Copley
5efe588174 Moved the rest of the Twitter stuff into the TwitterBridge plugin 2009-08-26 00:59:06 +00:00