This was introduced in 11ebb98919f56f7dcf888adfbebf9e8826f995b4#diff-96141878409d7418ea5a9eefbde509b43482c01R70
The arg number should have been one, as there's only one...
If the sysadmin decides that StoreRemoteMedia plugin should store
original, then its thumbs will be regenerated as well, making it safe to
delete them if needed. Beware that Embed plugin never stores the original tho.
This bug was introduced with the commit "[Media] Fix issues with database file storage"
due to the API change "File::getByHash now returns a yield of files".
I had updated this function on that commit but had missed this little detail.
[StoreRemoteMedia] Upgrade plugin to use the new Media system
API Changes:
- Added getters to File to better formalize the ideas of the commit "[Media] Fix issues with database file storage"
UI Changes:
- Now presented thumbnails are actual thumbnails (bug fix)
- Attachment actions have a slightly more extended behaviour
Many other minor bug fixes...
Fixed file quota as well.
There can be more than one file for the same filehash IF the url are different.
Possible states:
- A file with no url and with filename is a local file.
- A file with an url but no filename is a remote file that wasn't fetched,
not even the thumbnail.
- A file with an url and filename is a fetched remote file (maybe just a
thumbnail of it).
- A file with no filename nor url is a redirect.
Routes:
Given these states, updated routes so that an attachment can only be
retrieved by id and a file by filehash.
Major API changes:
File::getByHash now returns a yield of files
Major UI changes:
- Now remote non stored files are presented.
- /view became preferred
- Redirects to remote originals are preferred.
Many other minor bug fixes...
Fix OAuth and Realtime issues introduced in 9a515b9234
[DATABASE] Fix an empty default value mistake introduced in
fde929b151
[DATABASE][PostgreSQL] Avoid use of pg_constraint.consrc, which was removed in
PostgreSQL 12.
[DATABASE][MariaDB] Fix a typo introduced in aed2344bd4
[DAEMON] Wrap an assignment inside "switch":
a follow-up to adc689cb15
Ideally the character set should be set with the connection, and so this is
exactly what's being done now.
And now the character set code is attempted to be generalised.
Changing `character_set_server` requires root permissions and rebooting
the server.
Which is impossible on shared web hosting services.
So use `character_set_database`. This variable can be changed with
user permissions using `ALTER DATABASE`.
"Magic quotes" were removed in PHP 5.4, no need to mitigate it anymore.
Avoid implode() with the join()-like order of arguments which was deprecated
since PHP 7.4 and implicitly since PHP 5.3.
Also avoid implode() with an implicit separator for stylistic reasons.
mktime() with no arguments has been deprecated since PHP 5.1.
The previous approach sent the key values twice, which for large sets is
twice as bad.
As an optional feature of this approach multiGet now allows retrieving tuples
in exact order and amount of the requested key values.
Before now table definitions could define collations only for MariaDB using the
MariaDB's collation names directly.
Now instead definitions get a slightly more abstract collation name syntax, but
only supporting the collations utf8mb4_bin and utf8mb4_unicode_(cs|ci) (wrapped
as utf8_bin, utf8_general_(cs|ci)), because those are the ones that have
practical use for GNU social.
Which also means that on MariaDB the formerly used utf8mb4_general_(cs|ci) have
been superseded by utf8mb4_unicode_(cs|ci), as they are the more modern
replacement.
Introduce collation support on PostgreSQL which results in use of the C (POSIX)
collation as utf8_bin and the und-x-icu collation as utf8_general_cs.
utf8_general_ci is also mapped to und-x-icu, which makes it case-sensitive,
unfortunately.
Now analogous to the simple and safe PostgreSQL's websearch_to_tsquery syntax.
MariaDB's full-text boolean search queries will fail on input such as "@user"
(unquoted) which is particularly noticeable to the user.
WebP format is now the default image format for anything that
is not an animated GIF. Image Intervention doesn't support
animated WebPs so we don't convert animated GIFs.
ImageFile:
MediaFile:
default:
- Add WebP support
ImageMagickPlugin:
- Remove animated thumbnail setting, we'll be able to use FFmpeg for performance
- Remove onFillImageFileMetadata and onCreateFileImageThumbnailSource(), these
are handled just fine by ImageFile
- Bump minor version number
README:
- Update
"blob" is practically used with the expectation of unlimited length, which is
true with PostgreSQL's bytea, but not with MariaDB's BLOB, which is limited to
64KiB.
So instead use LONGBLOB, which has a maximum of 4GiB, effectively unlimited.
This check made registration impossible when welcomeuser didn't have validation
as well.
And rename the "grandfatherCutoff" option to "exemptBefore".
"Grandfathering" is a relatively obscure term linked to the history of the
United States of America, so replace that with something self-descriptive.
Give priority to cookies over GET.
Make sure session ids have only expected characters
(PHP file session handler's limitation).
Replace a mostly useless log warning with a debug message.
On PostgreSQL:
- Parse defaults for strings and booleans properly.
- Parse the "serial" definition type properly.
- Get information on the "enum" definition type.
- Re-work getting information about keys/indices.
On MariaDB:
- Get information about lengths in indices.
- Get foreign key information separately from the rest as they can have
colliding names.
This adds a requirement for all definitions that have foreign keys to also
require indices for all source (local) attributes mentioned in foreign keys.
MariaDB/MySQL creates indices for source attributes automatically, so this
serves as a way to get rid of those automatic indices and create clean explicit
ones instead.
In PostgreSQL, most of the time, indices on the source are necessary to
decrease performance penalty of foreign keys (like in MariaDB), but they aren't
created automatically, so this serves to remove that difference between
PostgreSQL and MariaDB.
- getMessages() is now fetching from the Notice table as supposed
- every show{format}* method is properly updated to use Notice objects
- json and xml responses retrieve multi-recipients without compromising
backwards compatibility
Instead of relying on the MariaDB's ON UPDATE CURRENT_TIMESTAMP trigger update
"modified" attributes in Managed_DataObject. Every raw query that needs
adjusting is adjusted, as they won't update "modified" automatically anymore.
The main goal behind this change is to fix "modified" updates on PostgreSQL.
Password hashes are now stored in a TEXT attribute, not limited to 199 symbols.
That limitation makes no sense as password hashes are not the kind of
information to be indexed.
Actually replace crypt() with password_verify() for password checking, current
code left password_verify() unused.
Only update passwords when they use a different algorithm from the current
default. Previously "overwrite" meant rehashing every login.
Replace the "argon" boolean option with "algorithm" and "algorithm_options" for
better configurability.
The default remains whichever is default for PHP's password_hash.
This way UNKNOWN (NULL) explicitly turns to FALSE when three-valued logic is
reduced to binary.
In pgsqlschema, however, use "IS FALSE" as boolean attributes in pg_index are
non-nullable, there is no outer join and there's no clear preference for NULL
reduction.
Over-complicated constructions in TagCloud queries have been simplified, which
should not affect their performance.
Additionally, in TagCloud's lib/subscriptionspeopleselftagcloudsection.php
a typing mistake in an equi-join of "profile_tag" and "profile_list" on
"tagger" was fixed.
That regression was introduced in f446db8e2a
Argument 3 passed to htmloutputter::input() must be of the type string or null, array given, called in /srv/gnusocial/plugins/Bookmark/forms/bookmark.php on line 166
Argument 1 passed to xmloutputter::text() must be of the type string, null given, called in /srv/gnusocial/plugins/ExtendedProfile/lib/extendedprofilewidget.php on line 556
After adding a verb condition there, MariaDB now prefers the
("created", "id", "is_local") and ("profile_id", "verb", "created", "id")
indices for that query, even though they are slow for the job.
So replace them with ("is_local", "created", "id") and
("profile_id", "verb", "created", "id") respectively.
Also fix the naming of the ("profile_id", "created", "id") index.
There are only three possible visible notice verbs: POST, SHARE and DELETE.
What including all verbs does is it makes limiting (FETCH FIRST) unreliable as
the query will fetch invisible stuff (like favourites) and count it in, but
nothing will be displayed.
NoticeStream only allows POST and SHARE, so this effectively removes tombstones
from the profile page like in 78a111b57d
Introduce a new property in CachingNoticeStream for always checking if there
are any new elements in the stream.
It would be extremely hard to blow InboxNoticeStream, so instead the database
hit will still occur, but it is be much faster than starting fresh.
This fixes a regression introduced in 36a55d8436
There was no checking of attributedTo, actors and referent object IDs to make
sure they exist in the same domain. Therefore, one could spoof messages from
people by doing attributedTo: whoever-i-want-to-spoof
The notice.created sort forced the notice_tag by notice join plain to employ
materialisation, which can have a serious performance penalty depending on the
size of the database.
Sort by notice_tag.created instead, which should be exactly the same.
All verbs for not visible notices are filtered out, so this should not
break the timeline.
Additionally, filter by profile outside of the derived relation as that shows
better performance in PostgreSQL and MariaDB both.
It appears this was added to display "tombstones" of deleted notices.
However, it has other side-effects and the concept of keeping them visible has
not been adopted by the wider fediverse.
Avoid the use of deprecated MariaDB "zero dates" globally. If they're present
as attribute defaults somewhere, they will be replaced with NULL implicitly.
The existing "zero dates" in MariaDB storage will be left intact and this
should not present any issues.
The "timestamp" type in table definitions now corresponds to DATETIME in
MariaDB with "DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP", which
should be close enough to the original behaviour for compatibility purposes.
It is now the recommended type for "modified" attributes, because of the
update trigger on MariaDB. But there is no such trigger implemented on
PostgreSQL as of this moment.
Make common_sql_weight employ standard SQL functions for the timestamp
difference in seconds.
Also replace UTC_TIMESTAMP in the MariaDB-specific part with CURRENT_TIMESTAMP
as it is the only occurence and GNU social sets UTC as a default timezone.
In a delete_orphan_files.php script simplify the main query considerably.
In clean_profiles.php stop using COUNT as if it is ANY, that is unnecessary
punishment for the database. Instead implement the anti-join with a
left outer join.
In Autocomplete and Activitypub_profile use joins instead of a WHERE OR
anti-pattern for the semi-joins.
In lib/ui/galleryaction.php replace a CROSS JOIN with an INNER JOIN.
In actions/sup.php remove a redundant subquery: WHERE is applied before
grouping either way.
The Census event is now replaced with module settings for populating the
protocols array. With this we can shutdown some plugins and still make
them be checked by TFN.
The performance:high config is now added when deciding whether or not to
do online lookup after the offline lookup fails.
default:
- Add default values for the TFN protocol setting
EVENTS:
- Remove Cencus event
TheFreeNetworkModule
- Remove Census event handler, update protocols array to use module's settings
- Use performance:high config when deciding to do online lookup
Activitypub_profile:
- Update do_insert to trigger TFN's assistance in inserting the profile
explorer:
- Use the new LRDD's method for grabbing profile aliases
Remove redundant setting retrival code.
An example config.php entry to activate the new config:
$config['site']['sensitivecontent']['hideforvisitors'] = true;
install.php:
installer.php:
- add option field and handle logic
display.css:
- hardcode ssl's label margin-top value so all options are correctly displayed
This was to keep the following two commits readable
- actions/showstream.php
- actions/userbyid.php
- lib/modules/ActivityHandlerModule.php
- lib/modules/ActivityHandlerPlugin.php
- lib/notices/conversationnoticestream.php
- lib/notices/noticelistitem.php
- lib/notices/noticestream.php
- lib/notices/threadednoticelistitem.php
This should fix nulls on explorer lookups inputed by postman after generate_followers/getSubscribers, that I think were caused by calling common_profile_uri that, curiously, only handles local profiles
On big databases these queries from the Nodeinfo plugin choked up:
SELECT profile_id FROM notice
WHERE notice.created >= (CURRENT_DATE - INTERVAL '180' DAY)
AND notice.is_local = 1;
SELECT id FROM "user"
WHERE "user".created >= (CURRENT_DATE - INTERVAL '180' DAY);
Use $object->sqlValue('NULL') (identical to DataObject_Cast'ing) instead and
fix related issues like (email|sms)settings considering these NULLs as a
false positive for the E-Mail address still being set when it's been removed.
There could also be security implications to the now-disabled approach of
considering 'NULL' strings as SQL NULLs.
ActivityPubPlugin:
- Rework onProfileDeleteRelated to account for the tables _rsa and _pending_follow_requests
- Update onEndShowAccountProfileBlock to stop creating the ap_profile if it doesn't exist (we'll handle this in a different manner)
Activitypub_profile:
- Remove unnecessary code from from_profile method and add return type information
Explorer:
- Update travel_collection to call itself instead of _lookup, that was wrong
Trying to enable the RedisCache with the latest nightly, getting this with the daemon:
sep 25 11:40:18 friedrich startdaemons.sh[21428]: PHP Fatal error: Uncaught Error: Call to a member function getPayload() on null in /var/www/social/plugins/RedisCache/RedisCachePlugin.php:96
sep 25 11:40:18 friedrich startdaemons.sh[21428]: Stack trace:
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #0 /var/www/social/lib/util/event.php(89): RedisCachePlugin->onStartCacheSet('gnusocial:herds...', Object(HubSub), NULL, 86400, false)
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #1 /var/www/social/lib/cache/cache.php(202): Event::handle('StartCacheSet', Array)
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #2 /var/www/social/classes/Memcached_DataObject.php(520): Cache->set('gnusocial:herds...', Object(HubSub))
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #3 /var/www/social/classes/Memcached_DataObject.php(52): Memcached_DataObject->encache()
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #4 /var/www/social/classes/Managed_DataObject.php(50): Memcached_DataObject::getClassKV('HubSub', 'hashkey', 'a38b9dc516371af...')
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #5 /var/www/social/plugins/OStatus/classes/HubSub.php(47): Managed_DataObject::getKV('hashkey', 'a38b9dc516371af...')
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #6 /var/www/social/plugins/OStatus/lib/hubprepqueuehandler.php(68): HubSub::getByHashkey('https://herds.e...', 'https://raki.so...')
sep 25 11:40:18 friedrich startdaemons.sh[21428]: #7 /var/www/social/plugins/RedisQueue/classes/RedisQueueManager.php(58): HubPrepQueueHandl in /var/www/social/plugins/RedisCache/RedisCachePlugin.php on line 96
Sign in to j
Refactored Activitypub_profile::ensure_web_finger to Activitypub_profile::ensure_webfinger
Do not throw exceptions in the handling of this event because we don't
want to stop the regular search just because we were unable to find
ActivityPub actors or notes.
The code used to operate under the assumption that MariaDB doesn't support
quoting identifiers. Not only is that not exactly true, but MariaDB has
reserved keywords that cannot be used as table or column names unquoted.
Issue description as reported by aab:
2019-08-28 17:34:48 LOG_ERR: [khp.ignorelist.com:61055.f5f7f51c GET /api/statuses/show/178372.atom] ServerErrorAction: 500 Class 'OAuthRequest' not found
2019-08-28 17:39:50 LOG_ERR: [khp.ignorelist.com:65390.4483ff85 GET /api/statusnet/conversation/104672.rss] Handled serverError (500) but cannot output into desired format ('rss'): 'Class \'OAuthRequest\' not found'
2019-08-28 17:39:50 LOG_ERR: [khp.ignorelist.com:65390.4483ff85 GET /api/statusnet/conversation/104672.rss] ServerErrorAction: 500 Class 'OAuthRequest' not found
2019-08-28 17:40:49 LOG_ERR: [khp.ignorelist.com:65390.4c745f68 GET /api/statuses/show/18132.atom] Handled serverError (500) but cannot output into desired format ('atom'): 'Class \'OAuthRequest\' not found'
2019-08-28 17:40:49 LOG_ERR: [khp.ignorelist.com:65390.4c745f68 GET /api/statuses/show/18132.atom] ServerErrorAction: 500 Class 'OAuthRequest' not found
2019-08-28 17:47:41 LOG_ERR: [khp.ignorelist.com:65390.42ee9fd4 GET /api/statusnet/conversation/133023.as] Handled serverError (500) but cannot output into desired format ('as'): 'Class \'OAuthRequest\' not found'
Issue description as reported by aab:
About one day after enabling redis plugin:
PHP Fatal error: Uncaught Error: Call to a member function getPayload() on int in /var/www/html/plugins/RedisCache/RedisCachePlugin.php:96
Stack trace:
0 /var/www/html/lib/event.php(89): RedisCachePlugin->onStartCacheSet('gnusocial:gatea...', Object(Queue_item), NULL, 86400, false)
1 /var/www/html/lib/cache.php(202): Event::handle('StartCacheSet', Array)
2 /var/www/html/classes/Memcached_DataObject.php(496): Cache->set('gnusocial:gatea...', Object(Queue_item))
3 /var/www/html/classes/Memcached_DataObject.php(416): Memcached_DataObject->encache()
4 /var/www/html/classes/Managed_DataObject.php(620): Memcached_DataObject->update(Object(Queue_item))
5 /var/www/html/classes/Queue_item.php(74): Managed_DataObject->update(Object(Queue_item))
6 /var/www/html/lib/dbqueuemanager.php(75): Queue_item::top(Array, Array)
7 /var/www/html/lib/iomaster.php(287): DBQueueManager->poll()
8 /var/www/html/lib/iomaster.php(161): IoMaster->poll()
9 /var/www/html/scripts/queuedaemon.php(112): IoMaster->service()
10 /var/www/html/lib/spawni in /var/www/html/plugins/RedisCache/RedisCachePlugin.php on line 96
NoticeSearchAction:
- Add new event before finding query matches
ActivityPubPlugin:
- Subscribe new searchNotice event
- Bump minor version number
Activitypub_explorer:
- Update lookup to make remote-grabbing optional
ActivityPubPlugin:
- update grab_notice_from_url to make online grab optional
- subscribe events of user and profile deletion
- bump minor version number
Activitypub_inbox_handler:
- separate handle_delete for delete-note and delete-person
Activitypub_postman:
- add delete-person logic
Activitypub_delete:
- update validation method to check for the "Person" type
- update to_array method to target the activity
Both StartSubscribe and StartUnsubscribe had a wrong initial if-condition.
Furthermore, this events were calling Activitypub_profile::from_profile()
which is wrong because it creates the Activitypub_profile object when
the goal is only to check if it exists already.
OStatusPlugin:
- Stop adding the remote-follow button
- Subscribe to required RemoteFollow plugin events
- Drop main/ostatussub route and update urls to the main/RemoteFollowSub route
- Bump plugin minor version number
actions/ostatusgroup,
actions/ostatuspeopletag:
- Update urls to the main/RemoteFollowSub route
lib/util:
- Port required functions from OStatusSubAction and adapt to be used with the new events
lib/default.php
- Add RemoteFollow to the list of default plugins
RemoteFollowPlugin:
- Subscribe events to add the remote-follow button
RemoteFollowInitAction:
- Handles the remote-follow form and getting the redirection url for follow completion
RemoteFollowSubAction:
- Handles the remote profile pulling and actual following
ActivityPubPlugin:
- Subscribe DirectMessage events
Activitypub_inbox_handler:
- Update handle_create_note to create private messages
Activitypub_postman:
- Add create_direct_note for sending private messages
Activitypub_create:
- Update create_to_array to support the 'directMessage' attribute
- Add isPrivateNote to verify private activities
Activitypub_notice:
- Update create_note to support the 'directMessage' attribute
- Remove isPrivateNote
lib/models:
- Add Activitypub_message, the model in charge of private notes
Activitypub_profile:
- Fix subscription-counter getter functions, invalid profiles were being counted
apActorFollowingAction:
- Small rewrite of generate_following, didn't make sense to not use try-catch block
apActorFollowersAction:
- Small rewrite of generate_followers, didn't make sense to not use try-catch block
Note that this commit isn't intended to add support for sending such notes
in GS. Instead, we handle the reception, storage and direct reply to this
type of notices, in AP.
ActivityPubPlugin:
- Subscribe the event StartNoticeSave to hack answering non-public notes
Activitypub_create:
- Add 'directMessage' attribute to the Create activity, defaulting to false for now
- Update validation method: validate 'directMessage' and add debug
Activitypub_notice:
- Handle incoming unlisted/followers-only notes
- Add support for unlisted-replies
- Add method to verify private (direct) notices
inbox_handler:
- Add handler for CREATE Note
- Prepare logic for private-messaging
- Overall refactor: Class members were continuously being passed as function arguments without need
SharePlugin:
- Stop showing the announce button in non public posts
For reference (raised by rozzin in IRC):
* http://foldoc.org/module
* http://foldoc.org/library
* http://foldoc.org/plugin
As noted by XRevan86, modules are not necessarily non-essential.
As we will keep the modules directory in GS root [therefore, near to
plugins/], it is evidenced the difference between both.
This is a simple yet fundamental structural change. It doesn't change
functionality but makes clearer the way we understand GNU social's
internals.
This commit does the necessary rework to store private messages
as Notices and to support Federation. The plugin's README presents
some more detail about the changes and future work that is still
required to do.
This solves the problem of routes that differ only in having
or not $_GET params. The ones not having params (static) were
being matched first during URL generation.
The way this problem was solved was by separating the $reverse
array in both $reverse_statics and $reverse_dynamics and explicitly
traversing this last one first in the generation function. Note that
maintaining the $reverse array and unshifting dynamic routes to its
head ( and therefore to the front of the static ones ) doesn't work
since even among dynamic routes the order of arrival should be kept.
ActivityPubPlugin:
- Change event-based notice distribution to queues logic
ActivityPub/lib:
- Add queue handler class activitypubqueuehandler.php
Misc:
- Add documentation for the (Start/End)InitializeQueueManager events
OStatusPlugin:
- Enqueue in the last position, as it should be. No need to worry about
the OMB comment, this protocol no longer have queue handlers that could
cause a conflict.
ActivityPubPlugin:
- Prevent sending a Delete for an Announce
Activitypub_announce:
- Update announce_to_array to add id, to and cc information to the retrieved object
Activitypub_follow:
- Add id to the arguments of follow_to_array, useful for Accept-Follow activities
Activitypub_notice:
- Fix notice validation, url isn't a MUST
Activitypub_inbox_handler:
- Make handle_follow use the received activity id for the later Accept-Follow
Activitypub_postman:
- Fix call to the updated announce_to_array
- Fix successive unnecessary calls to ActivityPubPlugin::actor_uri()
ActivityPubPlugin:
- Minor onDeleteOwnNotice rewrite
Activitypub_inbox_handler:
- Add deletion check to incoming notice
Activitypub_postman:
- Call the correct getUrl function
ActivityPubPlugin:
- Minor re-write of favor/disfavor event handlers
Activitypub_postman:
like/undo-like:
- fix proper getUrl() call
misc:
- make all activities accumulate errors (may be needed later) and log some information about it
ActivityPubPlugin:
- Fix of accepted activity verbs to include SHARES
- Add attention profiles to delivery when announcing
Activitypub_notice:
- New local function to retrieve original URL
- Removal of unnecessary 'Atom*' attributes
- Small fix to the ensuring of actor profile
Activitypub_profile:
- New local function to fetch AP profiles from a collection
Activitypub_postman:
- Fix url passed in the announce activity
Follow interaction:
- Fixed mini-bug where the subscriber profile was being used as the subscribed
- Updated cache subscription-related values in both instances
- Tested and working with local GS instances
Unfollow interaction:
- Updated cache subscription-related values in both instances
- Tested and working with local GS instances
Followers/Following collections:
- Now returning ActivityPub profiles only
- Stored collections in cache
Misc:
- Fix bug concerning the retrieval of public/private-key after in-function generation
This is not the same as the one in https://notabug.org/diogo/gnu-social-activitypub-plugin
Differences to the first "release"
-> Doesn't use guzzle nor has any composer dependencies
-> Supports HTTP Signatures
-> Has basic l10n/i18n
-> Some minor bug fixes
wamo
SENOO, Ken
Diogo Peralta Cordeiro
Maiyannah Bishop
Alexei Sorokin
someonewithpc
Andrew Tokarskiy
SENOO, Ken
Diogo Cordeiro
tenma
Hugo Sales
susdiv
rainydaysavings
Hugo Sales
nee
Miguel Dantas