Conversation trees works pretty bad with the current layout, javascript
etc. So it's best if we separate it and work on it as a side-project. The
oldschool settings are currently being deprecated (or broken out like this).
I'll wait with removing User preferences for oldschool conversation tree,
since that might be reusable data. But I guess it will go in the near future.
The File object now stores width and height of files that can
supply this kind of information. Formats which we can not read
natively in PHP do not currently benefit from this. However an
event hook will be introduced later.
The CreateFileImageThumbnail event is renamed to:
CreateFileImageThumbnailSource to clarify that the hooks should not
generate their own thumbnails but only the source image. Also it now
accepts File objects, not MediaFile objects.
The thumbnail generation is documented in the source code. For
developers, call 'getThumbnail' on a File object and hope for the best.
Default thumbnail sizes have increased to be more appealing.
The only reason it worked was because DB fetches calls to get$varname if
the dataobject has a variable with the specific name. However, it started
blurting out errors that the case must be correct (which would require
'geturl' to be the function name).
Since we probably want to replace DB sometime, we'll just override this
auto-fetching mechanism and use more explicitly defined functions.
Avoiding collisions with date (shorter than before) and 4 character
random alphanumeric string. I bet someone could mass-upload files
and generate all combinations of aaaa-zzzz during the course of a
day, but then maybe that user should be disabled anyway :)
(filling the collision space will cause a never-ending loop).
The exception thrown from MediaFile will be caught and simply result in
no thumbnail at all right now. In the future we might use a catch-all
and have a "cannot generate preview"-icon or something.
VideoThumbnails requires php5-ffmpeg and php5-gd.
Added the following FIXME:
How should a Twitter user get their Inbox filled with foreign tweets?
Every imported Twitter user has a profile in the Profile table, so we
could setup a Subscription entry for each of those, meaning they get
collected in the InboxNoticeStream... But this would mean a lot of
unnecessary entries and listings that generally just point to the
locked down Twitter service.
Let's figure out a good relation so we can connect any profile to any
imported foreign notice, so it shows up in the "all" feed.
This will work without much extra effort because there will always be
more notices (higher value) than conversations (so no collisions).
But please run upgrade.php to avoid having an autoincrement id on
conversation table.
Installations using code after 2014-03-01 will have identical
conversation IDs to the initial (conversation root) notice IDs. This
will not affect older installations, which will have very different
values.
Also removed the entirely unused saveGroups function.
Now avoiding multiGet and using listFind in Profile->getGroups()
so we don't have to deal with ArrayWrapper.
NoResultException was the wrong choice in this case, because it was
not a DB_DataObject instance that performed the search, but a static
call to the Notice class.
This reverts commit 8cc4660bd9.
This seems like something Evan only did to make pump.io import notices easier,
or maybe he just wanted to get rid of the identi.ca URLs?
This is the beginning of getting notice URI info via WebFinger
*XrdActionLinks is renamed *WebFingerProfileLinks, check EVENTS.txt
in WebFinger plugin for new events.
_flow_ reported on IRC that install.php had stopped working. This was
because default plugins had been put into two separate lists, and the
list with AuthCrypt was never loaded when performing an installation.
Core plugins cannot be disabled.
I also removed the Memcache autodetection thing since it should be
solved in a more elegant manner.
...no need to make a separate call to Local_group's setNickname all the time,
or a bunch of redundant code for the Profile table.
Next up is User->update()...
I implemented changes from quitter.se's new API that their front-end qvitter
uses, https://github.com/hannesmannerheim/qvitter/blob/master/api-changes-1.1.1/CHANGES
However I left out the URL shortening commens, since I believe whatever behaviour
they experienced that caused them to implement this was a bug (or many) and should
be fixed in their proper areas and that shortening should not be entirely left
out in API calls.
Profile_prefs aims to consolidate all the profile preferences into a
single table. Otherwise we end up with a bajillion *_prefs classes, like
User_urlshortener_prefs, or new fields in existing User/Profile classes,
like 'urlshorteningservice', 'homepage', 'phone_number', 'pet_name' etc.
Eventually we should migrate as many user-settable preferences as we can
into this system.
The data in Profile_prefs is organized by:
* profile_id Identify the current Profile.
* namespace Which plugin/section the preference is for.
* topic Preference name (like 'homepage')
* data Preference data (like 'https://gnu.org/')
The names 'topic' and 'data' are because 'key' and 'value' may be rather
ambigous when dealing with our DB_DataObject classes etc.
We're also now using $config['image']['jpegquality'] to determine the
quality setting for resized images.
To set Avatar max size, adjust $config['avatar']['maxsize']
The getAvatar call now throws exceptions too. Related changes applied.
Now let's move Profile->avatarUrl to the Avatar class!
* getOriginal added to Avatar class
This is a static function that retrieves the original avatar in a leaner
way than Profile->getOriginalAvatar() did (see below).
This will throw an Exception if there was none to be found.
* getProfileAvatars added to Avatar class
This gets all Avatars from a profile and returns them in an array.
* newSize added to Avatar class
This will scale an original avatar or throw an Exception (originally from
Avatar::getOriginal) if one wasn't found.
* deleteFromProfile added to Avatar class
Deletes all avatars for a Profile. This makes the code much smarter when
removing all avatars from a user.
Previously only specific, hardcoded (through constants) sizes would be
deleted. If you ever changed lib/framework.php then many oddsized avatars
would remain with the old method.
* Migrated Profile class to new Avatar::getOriginal support
Profile class now uses Avatar::getOriginal through its own
$this->getOriginalAvatar and thus remains backwards compatible.
* Updating stock GNU Social to use Avatar::getOriginal
All places where core StatusNet code used the
$profile->getOriginalAvatar, it will now useAvatar::getOriginal with
proper error handling.
* Updated Profile class to use Avatar::newSize
When doing setOriginal, the scaling will be done with the new method
introduced in this merge.
This also edits the _fillAvatar function to avoid adding NULL values to
the array (which causes errors when attempting to access array entries as
objects). See issue #3478 at http://status.net/open-source/issues/3478
New plugins:
* LRDD
LRDD implements client-side RFC6415 and RFC7033 resource descriptor
discovery procedures. I.e. LRDD, host-meta and WebFinger stuff.
OStatus and OpenID now depend on the LRDD plugin (XML_XRD).
* WebFinger
This plugin implements the server-side of RFC6415 and RFC7033. Note:
WebFinger technically doesn't handle XRD, but we serve both that and
JRD (JSON Resource Descriptor), depending on Accept header and one
ugly hack to check for old StatusNet installations.
WebFinger depends on LRDD.
We might make this even prettier by using Net_WebFinger, but it is not
currently RFC7033 compliant (no /.well-known/webfinger resource GETs).
Disabling the WebFinger plugin would effectively render your site non-
federated (which might be desired on a private site).
Disabling the LRDD plugin would make your site unable to do modern web
URI lookups (making life just a little bit harder).
maxContent==0 implies that a notice text can be infinitely long, but
this value was directly transferred to maxNoticeLength, where 0 was
tested if it was longer than the notice length - which of course always
was false.
This commit fixes the problem for infinite length notices that always
got shortened.
getUser calls are much more strict, and one place where this was found was
in the (un)subscribe start/end event handlers, which resulted in making the
Subscription class a bit stricter, regarding ::start and ::cancel at least.
Several minor fixes in many files were made due to this.
This does NOT touch the Foreign_link function, which should also have a more
strict getUser call. That is a future project.
Lots of the Memcached_DataObject classes stopped working when upgraded to
Managed_DataObject because they lacked schemaDef().
I have _hopefully_ made it so that all the references to the table uses
each class' schemaDef, rather than the more manual ColumnDef stuff. Not
all plugins have been tested thoroughly yet.
NOTE: This is applied with getKV calls instead of staticGet, as it was
important for PHP Strict Standards compliance to avoid calling the non-
static functions statically. (unfortunately DB and DB_DataObject still do
this within themselves...)
Not all instances of this has been fixed, but at least the ones
in the base class of Memcached_DataObject.
Avatar fix (in classes/Profile.php) requires a pkeyGet function
in the Avatar class (or as in this tree, the parent class of
Managed_DataObject)
Memcached_DataObject now defines
* pkeyGetClass to avoid collision with Managed_DataObject pkeyGet
* getClassKV to avoid collision with Managed_DataObject getKV
I used this hacky sed-command (run it from your GNU Social root, or change the first grep's path to where it actually lies) to do a rough fix on all ::staticGet calls and rename them to ::getKV
sed -i -s -e '/DataObject::staticGet/I!s/::staticGet/::getKV/Ig' $(grep -R ::staticGet `pwd`/* | grep -v -e '^extlib' | grep -v DataObject:: |grep -v "function staticGet"|cut -d: -f1 |sort |uniq)
If you're applying this, remember to change the Managed_DataObject and Memcached_DataObject function definitions of staticGet to getKV!
This might of course take some getting used to, or modification fo StatusNet plugins, but the result is that all the static calls (to staticGet) are now properly made without breaking PHP Strict Standards. Standards are there to be followed (and they caused some very bad confusion when used with get_called_class)
Reasonably any plugin or code that tests for the definition of 'GNUSOCIAL' or similar will take this change into consideration.
In some brief tests, this causes no problems.
In this state however, you would need to modify DB_DataObject to have a static declaration of staticget (and probably pkeyGet). The next commit will change the staticGet overload to a unique function name (like getKV for getKeyValue), which means we can properly call the function by PHP Strict Standards.
The parent class for our database objects, Managed_DataObject, has a
dynamically assigned class in staticGet which objects get put into,
leaving us with less code to do the same thing.
We will probably have to move away from the DB_DataObject 'staticGet'
call as it is nowadays deprecated.
We always call staticGet statically, so we define it statically. Next
step is to remove a bunch of definitions of 'staticGet' from classes
that can instead fall back to a parent class in Managed_DataObject.
The ampersand is removed as we're returning a class anyway, which does
not need a reference (and when we return false, it means nothing).
It may be a bad experience for new users to immediately when trying
out the service be asked for their geographical position. Instead,
let them opt-in for this behaviour.
If you look at classes/User_group.php on line 412 in the current code, you can see that a call to $profile->getGroups() is made. This implies getGroups($offset=0, $limit=PROFILES_PER_PAGE) only giving a limited amount of groups.
This means only the first 20 groups in an ascending numerical order by locally stored User_group->id will be addressable with the bangtag syntax.
I solved this by making the getGroups() call to the same one made in Profile->isMember(), i.e. $profile->getGroups(0, null);
User::getTaggedSubscriptions()
This change escapes the $tag argument to prevent a SQL injection
attack in User::getTaggedSubscriptions(). The parameter was not
escaped higher up the stack, so this vulnerability could be exploited.
This change escapes the argument to User::getTaggedSubscribers() to
prevent SQL injection attacks.
Both code paths up the stack fail to escape this parameter, so this is
a potential SQL injection attack.
This patch escapes query parameters in Profile_tag::getTagged(). This
is an extra security step; since these parameters come out of the
database, it's unlikely that they would have dangerous data in them.
This change adds additional escapes for arguments to
Profile_tag::moveTag(). The arguments are canonicalized in the API and
Web UI paths higher up the stack, but this change makes sure that no
other paths can introduce SQL injection errors.
This patch escapes the $tag parameter in
Profile::getTaggedSubscribers(). The parameter is not escaped either
in actions/subscriptions.php or in actions/apiuserfollowers.php. So
there is a potential for SQL injection here.
This change escapes a parameter in Local_group::setNickname(). Review
of the code paths that call this function sanitize the parameter
higher up the stack, but it's escaped here to prevent mistakes later.
Note that nickname parameters are normally alphanum strings, so
there's not much danger in double-escaping them.
This change escapes a parameter in Local_group::setNickname(). Review
of the code paths that call this function sanitize the parameter
higher up the stack, but it's escaped here to prevent mistakes later.
Note that nickname parameters are normally alphanum strings, so
there's not much danger in double-escaping them.
commit 74c5e4cce42ae601c07b447e100f097c15ebf9d2
Author: Evan Prodromou <evan@status.net>
Date: Thu Oct 20 12:48:52 2011 -0400
Add back in some optimization indices lost in schema conversion
commit ef5c2acfcd123b25910a1c8bb4ae01a3f9608e5e
Author: Evan Prodromou <evan@status.net>
Date: Thu Oct 20 12:29:57 2011 -0400
restore some of the lost optimized indices on notice table
commit fb1dfa9e98ded23fb5bdebae6465424a8cb8acd6
Author: Evan Prodromou <evan@status.net>
Date: Thu Oct 20 10:40:07 2011 -0400
Use popular notice stream for favorited page
commit e1d409ff738e39061ad35589d546ce9bed456975
Author: Evan Prodromou <evan@status.net>
Date: Thu Oct 20 10:32:23 2011 -0400
Use a caching stream for popular notice section
Instead of a big cached query, we now use a caching notice stream for
the popular notice section. It uses a single-table query at the
bottom, then scopes the notices and filters for silenced users. This
should be much nicer to our database servers.
Also clears the popular cache when someone favors or disfavors
something. A nice optimization would be to save the last weights and
re-calculate them at invalidation time, adding the new notice (or not)
depending on its own score. That will have to wait for another day,
though.
commit e9b7ab4c26c95e755adaff53c3957dcfca31c16b
Author: Evan Prodromou <evan@status.net>
Date: Thu Oct 20 10:31:14 2011 -0400
Let CachingNoticeStream users skip the ';last' optimization
This should resolve the issues darkip was reporting with user_im_prefs entries returning null immediately after insertion (seen with memcached off, so it was happening even with the built-in in-process cache in the Cache base class).
What was happening was that the initial pkeyGet() would end up saving a negative cache entry under the form with the fields sorted in the key, as via multicacheKey():
'statusnet:blaguette:user_im_prefs:screenname,transport:brionv,sms' => 'N;'
then we'd do an insert() on the new entry, saving cache entries for the non-sorted key names returned by _allCacheKeys():
'statusnet:blaguette:user_im_prefs:transport,screenname:sms,brionv' => 'O...'
'statusnet:blaguette:user_im_prefs:user_id,transport:1234,sms' => 'O...'
but the next query via pkeyGet() still saw the negative lookup cache from before, and came back with null.
Now, _allCacheKeys() sorts the fields in the keys by using the same key-builder function, and queries pick up the same thing you just inserted. :)
Memcached_DataObject doesn't quite fully understand unique indexes, and can't properly build cache keys for compound unique or primary keys.
Managed_DataObject has more information in its schema data, so we can build a proper list.
* '1.0.x' of gitorious.org:statusnet/mainline:
Issue #546: enable case-insensitive searching in MySQL
remove more groupnav
remove group nav
shorter title for tag cloud section on groups
remove group nav
move pending queue to sidebar
Move group logo edit from object nav to block actions
Show blocked users from group in section
MySQL needs the collation utf8_general_ci to support case-insensitive
searching. lib/mysqlschema.php already supports a 'collate' attribute on
a per-column basis, so we just need to take advantage of that attribute
on the columns we want to search and Bob (and BOB, and bOB) is your
uncle.
Signed-off-by: Dan Scott <dan@coffeecode.net>
I was storing the full objects in the cache for the listGet()
function. I've changed it to store only pkeys, and use pivotGet() to
get all the corresponding values.
This also required changing pivotGet() so it can get objects with
multi-column pkeys, which complicated the whole thing quite a bit. But
it seems to work OK.
This method lets you get all the objects with a given variable key and
another set of "fixed" keys. A good example is getting all the avatars
for a notice list; the avatar size stays the same, but the IDs change.
Since it's very similar to multiGet(), I refactored that function to
use pivotGet().
And, yes, I realize these are kind of hard to follow.
L10n/i18n updates.
Superfluous whitespace removed.
Add FIXME for a few i18n issues I couldn't solve this quickly.
Takes care of documentation for all core code added in merge of "people tags" feature (Commit:e75c9988ebe33822e493ac225859bc593ff9b855).
For groups that require a private scope, we force every notice to be limited to group scope.
Changed the group-discovery code so we only get groups once -- regardless if they were provided or not.
Notice::saveNew() now does these checks directly when making a repeat:
* make sure the original is valid and existing
* stop you from repeating your own message
* stop you from repeating something you've previously repeated
* prevent repeats of any non-public messages
* explicit inScope() check to make sure you can read the original too (just in case there's a funky extension at play that changes scoping rules)
These error conditions throw exceptions, which the caller either uses as an error message or passes on up the stack, without having to duplicate the checks in each i/o channel.
Numbered parameters when more than one used in a message.
L10n updates for consistency.
i18n for non-translatable exception.
Updated translator documentation.
Removed superfluous whitespace.
We disallow repeating a notice (or whatever) if the scope of the
notice is too private. So, only notices that are public scope
(available to everyone in the world) or site scope (available to
everyone on the site) can be repeated.
Enforce this rule at a low level in Notice.php, and in the API,
commands, and Web UI. Repeat button doesn't appear on tightly-scoped
notices in the Web UI.
We've been muddling through with 6- or 8-argument functions for managing streams. I'd
like to start thinking of streams as their own thing, and give them some more value.
So, the new NoticeStream class takes over the Notice::stream() function and Notice::getStreamByIds().
There's probably some fine-tuning to do on the object interface.
like leprous boils in our code. So, I've replaced all of them with //
comments instead. It's a massive, meaningless, and potentially buggy
change -- great one for the middle of a release cycle, eh?
As a hack this removes the mysql_timestamp bit from the field settings on reply.modified so that our value actually gets saved. This *should* work ok as long as system timezone is set correctly, which we now set to UTC to match when connecting.
I've extended the rights framework (centering on the Right class and Profile::hasRight()) to cover
Web login and API use. This will make it possible to prevent login and API use by users.
I added two new Right constants to the Right class: WEBLOGIN and API. I check these rights using
Profile::hasRight() when initializing users. If the rights check fails, I throw an exception.
I created a new AuthorizationException class for this particular
exception, in order to allow a different UI for these kinds of exceptions (or whatever).
Workaround for deleted profiles still appearing in cached subscriptions/subscribers lists: if we couldn't fetch them, don't include them in the ArrayWrapper.
ArrayWrapper doesn't deal well with null entries, which aren't meant to happen in how it works. This code has recently changed from dying directly with a PHP fatal error in that case to throwing an exception, which allows tracking down the caller.
It looks like there might be some cases where profiles and their matching subscriptions get deleted, but the subscription entries don't get properly cleared from cache... that still bears further investigation. The regular code path looks ok; calls Subscription::cancel() from code called in Profile::delete(); but if they're batch-deleted instead of one row at a time, that could fail to trigger.
$config['site']['logperf'] = true; // to record & dump total hits of each type and the runtime to syslog
$config['site']['logperf_detail'] = true; // very verbose -- dump the individual cache keys and queries as they get used (may contain private info in some queries)
Seeing 180 cache gets on a timeline page seems not unusual currently; since these run in serial, even relatively small roundtrip times can add up heavily.
We should consider ways to reduce the number of round trips, such as more frequently storing compound objects or the output of processing in memcached.
Doing parallel multi-key lookups could also help by collapsing round-trip times, but might not be easy to fit into SN's object model. (For things like streams this should actually work pretty well -- grab the list, then when it's returned go grab all the individual items in parallel and return the list)
* dropped unnecessary join on notice table
* made the function actually static, since it makes no sense as an instance variable. The only caller (in AttachmentList) is updated.
MySQL stores TIMESTAMP columns as UTC, but with a local time interface. (SRSLY?!) DATETIME columns are always bare and assumed to be local time, but we keep only UTC in them.
Forcing the session time_zone to UTC means we won't have to worry as much about what we're sending/receiving in there.
Also will let us remove the hack in master commit a7abb2323e for session tweaks
Had to tweak statusnet.ini to remove the DB_DATAOBJECT_MYSQLTIMESTAMP bitfield constant on session.modified; while it sounds like a useful and legit setting, it actually just means that DB_DataObject silently fails to pass through any attempts to explicitly set the value. As a result, MySQL does its default behavior which is to insert the current *LOCAL* time, which is useless.
This was leading to early GC west of GMT, or late GC east of it. Early GC could at worst destroy all live sessions (whoever's session *triggered* GC is fine, as the session then gets saved right back.)
Previously, if someone you subscribe to repeats a notice by someone you've blocked, you got the message and had to just roll your eyes.
Now blocks are checked against both the current notice's posting profile, and the poster of the original if it's a repeat.
It seems to have actually been saving correctly, but the update of the colors on the form success page wasn't working properly.
When a design object is pulled out of the database, the numeric fields are read in as strings, so black comes back as "0".
But, when we populate the new object and then stick it live, we've populated it with actual integers; with memcache on these might live for a while in the cache...
The fallback code in Design::toWebColor() did a check ($color == null) which would be false for the string "0", but counts as true for the *integer* 0.
Thus, the display code would initially interpret the correctly-saved black color as "use default".
Changing the check to === against null and "" empty string avoids the false positive on integers, and lets us see our nice black text immediately after save.
* adds Right::CREATEGROUP
* logic in Profile::hasRight() checks for silencing
* NewgroupAction checks for the permission before letting you see or process the form in the UI
* User_group::register() logic does a low-level check on the specified initial group admin, and rejects creation if that user doesn't have the right; guaranteeing that API methods etc will also have this restriction applied sensibly.
Made two new functions, Subscription::bySubscriber() and
Subscription::bySubscribed(), to get streams of Subscription objects.
Converted Profile::getSubscribers() and Profile::getSubscriptions() to
use these functions.
Big thanks to the folks at http://pecl.php.net/bugs/bug.php?id=16745 for the secret juju!
Classes were being torn down before session save handlers got called at the end of the request, which exploded with complaints about being unable to find various classes.
Registering a shutdown function lets us explicitly close out the session before everything gets torn down.
Big thanks to the folks at http://pecl.php.net/bugs/bug.php?id=16745 for the secret juju!
Classes were being torn down before session save handlers got called at the end of the request, which exploded with complaints about being unable to find various classes.
Registering a shutdown function lets us explicitly close out the session before everything gets torn down.
We had two ways to generate an activity entry from a notice; one through
Notice::asAtomEntry() and one through Notice::asActivity() and
Activity::asString(). The code paths had already diverged somewhat. I
took the conditions that were in Notice::asAtomEntry() and made sure
they were replicated in the other two functions. Then, I rewrote
Notice::asAtomEntry() to use the other two functions instead.
This change passes the ActivityGenerationTests unit tests, but there
may be some other stuff that's not getting covered.
common_shorten_links() can only access the web session's logged-in user, so never properly took user options into effect for posting via XMPP, API, mail, etc.
Adds an optional $user parameter on common_shorten_links(), and a $user->shortenLinks() as a clearer interface for that.
Tweaked some lower-level functions so $user gets passed down -- making the $notice_id param previously there for saving URLs at notice save time generalized a little.
Note also ticket #2919: there's a lot of duplicate code calling the shortening, checking the length, and reporting near-identical error messages. These should be consolidated to aid in code and translation maintenance.
Code was doing a batch call to $avatar->delete() which fails to properly engage the file deletion code. Calling the existing profile->delete_avatars() function deletes them individually, which makes it all work nice again.
This option may be useful for intranet sites that don't have direct access to the internet, as they may be unable to successfully fetch those resources.
Newly supported:
- TwitPic: added a local function using TwitPic's API, since the oohembed implementation for TwitPic produced invalid output which Services_oEmbed rejects. (bug filed upstream)
Tweaked...
- Flickr: works, now using whitelist to use their endpoint directly instead of going through oohembed
- Youtube: worked around a bug in Services_oEmbed which broke the direct use of API discovery info, so we don't have to use oohembed.
Not currently working...
- YFrog: whitelisting their endpoint directly as the oohembed output is broken, but this doesn't appear to work currently as I think things are confused by YFrog's servers giving a '204 No Content' response on our HEAD checks on the original link.
The old code attempted to compare the value of the notice.created field against now() directly, which tends to explode in our current systems. now() comes up as the server/connection local timezone generally, while the created field is currently set as hardcoded UTC from the web servers. This would lead to breakage when we got a difference in seconds that's several hours off in either direction (depending on the local timezone). New code calculates a threshold by subtracting the number of seconds from the current UNIX timestamp and passing that in in correct format for a simple comparison. As a bonus, this should also be more efficient, as it should be able to follow the index on profile_id and created.
