Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Send only updated/changed torrents to the client #6

Open
wants to merge 6 commits into
base: develop
Choose a base branch
from

Conversation

bendikro
Copy link
Member

@bendikro bendikro commented Aug 3, 2014

This branch contains the following modifications:

  • Client can request to get only the torrents that were updated since last time
    • This reduces the amount of network data that needs to be transfered between the client and server greatly. With thousands of torrents, the overhead of processing the entire torrent list to be sent and received regularly causes the major part of the CPU usage as seen by profiling.
  • The filtering of the torrents has been split between the server and client. With thousands of torrents, the workload of filtering the torrents each time the list is sent becomes a performance bottleneck. This has been changed so that the server only filters on the owner to make sure only the torrents the user is allowed to fetch are returned. The rest of the filtering is performed on the client side.
    • Much faster to change the list on the client side as it no longer has to ask the server.
    • The filtering functionality is implemented with PyDbLite.
      • This should make it possible to implement advanced filtering quite easily.
      • The filtering is much more efficient than the current code
  • Write more unit tests
  • The code changes for the webui should work but the code should be reviewed

Feedback is very appreciated. 😉 @gazpachoking

Note: PyDbLite is a separate project: https://pypi.python.org/pypi/PyDbLite

Changes on the server side

deluge/core/core.py

Changed functions:

  • get_torrents_status:
    • Added keyword argument only_updated which makes get_torrents_status only return the torrents that have been updated since the last call.
    • Argument filter_dict is now obsolete.
    • Argument diff is now obsolete.
      Removed functions:
    • get_filter_tree - The filtertree is now created on the client side

deluge/core/filtermanager.py

Most of the functionality of filtermanager.py has been moved to the client side (torrentfilter.py). filtermanager.py now only filters on the owner so the client will only receive the torrents it is authorized to see.
It filters on owner using the new databases in deluge/include/pydblite.

deluge/core/rpcserver.py

Changed functions:

  • connectionLost
    • Now emits ClientDisconnectedEvent when client disconnects

deluge/core/torrent.py

Changed functions:

  • set_tracker_status
    Now emits TorrentTrackerStatusChangedEvent when torrent status is set

deluge/core/torrentmanager.py

Added status_dicts which contains all the torrents for each session. When a torrent is updated (either by libtorrent or a plugin) it is removed from the cache to mark it as updated.

Changed functions:

  • handle_torrents_status_callback
    • Creates a status based on the only_updated argument, so if only_updated is True, only the torrents that do not exist in the cache (for this session_id) are returned.
  • on_alert_state_update
    • Removes the updated torrents from the cache, and emits TorrentsUpdatedEvent

New Functions:

  • update_torrent_cache(torrent_ids)
    • Removes the torrent_ids from the cache

Changes on the client side

deluge/ui/gtkui/torrentview.py

Changed to now only update the necessary rows.
The main difference is that self.status now always contains all the torrents, not only the torrents that are displayed. Which torrents that are currently displayed is stored in the set self.visible_torrents which is updated in _on_get_torrents_status.

Changed functions:

  • on_get_torrents_status
    • Updates self.visible_torrents no contain the currently visible rows.
  • update_view
    • Added arguments:
    • only_updated_ids
      • The torrent_ids that have been updated with new data. If None, all have been updated.
    • to_show
      • The torrent_ids to show (change filter column value to True)
    • to_hide
      • The torrent_ids to hide (change filter column value to False)

Removed functions:

  • on_torrentstatechanged_event
    • No longer needed as this is handled by sessionproxy.
  • mark_dirty
    • The dirty functionality was not used anymore (since the last major torrentview changes) so it was completely removed.

deluge/ui/sessionproxy.py

Has been completely changed. It no longer keeps track of cache times for each value. This was extremely expensive, and is no longer needed as the torrents that have been updated on the server side are now updated automatically.
self.updated_ids contains the torrent ids that have been updated from core and not yet updated in the ui.

All of the filtering previously performed in filtermanager is now performed by self.torrentfilter. This reduces the workload for the daemon tremendously and makes it much faster on the client side to get the filtered results.

Changed functions:

  • get_torrents_status
  • The biggest change is that it now takes the status dictionary as input and updates the status with the torrents as needed. It returns which torrents were updated and which torrents that should be shown or hidden in the view.

New arguments:

  • torrents_state
    • Contains state info for the client
  • only_updated=False
    • Give only the updated torrents
  • from_cache=True
    • If it should wait for the data from core, or return data from the cache.

deluge/ui/torrentfilter.py

This does all the filtering on the client side which was previously done in filtermanager.py on the server side.
torrentfilter.py uses PyDbLite (which uses SQLite if available, else pure Python) to perform all the filtering. The filtering code should be cleaner and simpler than the old code, and also much more efficient with many torrents.

deluge/plugins/Label/deluge/plugins/label/{core.py,gktui/init.py}

Changed to work with the new filtering

Changed to handle “hide owners” in sidebar:
deluge/ui/gtkui/filtertreeview.py
deluge/ui/gtkui/glade/main_window.ui
deluge/ui/gtkui/menubar.py
deluge/ui/web/server.py
deluge/ui/gtkui/gtkui.py

Changes for both client and server

deluge/event.py

  • Added events
    • TorrentTrackerStatusChangedEvent
    • ClientDisconnectedEvent

PyDbLite for efficient filtering

Added files:

  • deluge/filterdb.py
    • Contains Classes to create and setup the PyDbLite database

log.info("Finished loading %d torrents.", len(state.torrents))

finished = datetime.datetime.now()
log.info("Finished loading %d torrents in %s", (len(state.torrents), str(finished-start)))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log takes *args for the string replacement, so there are some extra parens here

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: 3fe458d

@bendikro bendikro force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch from 3fe458d to df79207 Compare September 10, 2014 21:35
@bendikro bendikro force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch from df79207 to c356ae7 Compare September 30, 2014 13:42
@bendikro bendikro force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch 3 times, most recently from 9a8f491 to 74ddd8e Compare October 10, 2014 14:37
@bendikro bendikro force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch from 74ddd8e to 3a61089 Compare December 1, 2014 14:32
@bendikro bendikro force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch from 3a61089 to 57b00b7 Compare January 26, 2015 10:51
@bendikro bendikro force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch from bdb2934 to 63f52e1 Compare April 3, 2016 16:23
@bendikro bendikro force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch 3 times, most recently from 889b2bb to 04f9b5e Compare April 12, 2016 00:38
@bendikro bendikro force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch 3 times, most recently from f20513d to 3ac6c86 Compare April 13, 2016 01:21
@bendikro bendikro force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch from 3ac6c86 to 30138e9 Compare April 19, 2016 23:02
@bendikro bendikro force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch 2 times, most recently from 1d55cd7 to a0347a9 Compare April 21, 2016 12:35
@cas-- cas-- removed the blocked label Apr 25, 2016
@bendikro bendikro force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch 2 times, most recently from 81cfb23 to 6fb33ca Compare November 3, 2016 12:22
@cas-- cas-- force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch 2 times, most recently from 0b7fd1f to 735eb98 Compare November 4, 2016 18:10
@cas--
Copy link
Member

cas-- commented Nov 4, 2016

Rebased and squashed fixups upon develop

bendikro and others added 6 commits November 4, 2016 23:55
All the event handle management is done by either EventManager or
EventManagerClient. A client can now emit an event that will be
triggered by an event handler on the same client using
client.emit(...)
…changed

Clients can now request a list of torrents that have changed since
last request. This reduces the amount of data transferred with the RPC
protocol, making it much faster with large amounts of torrents.

The UIs can more easily update only the necessary torrent list entires
making the UIs render faster.

* GKTUI (torrentview) now asks only for changed torrents
* Added torrentfilter.py to sessionproxy which handles filtering
* Implemented filtering in filtermanager.py and torrentfilter.py using
  PyDbLite. The new filtering allows for more advanced filter queries
  to be implemented.
@cas-- cas-- force-pushed the develop-only-updated-torrents-torrentfilter-on-client-side-dev branch from 735eb98 to 2fdae2d Compare November 4, 2016 23:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants