[WIP] Partially Abstract Usenet Logic and Fix Statuses

[qualman]

This work is ~95% done and runs without error, but is still WIP. Need some advice from @maxdorninger, please see comments

Summary

This PR aims to accomplish two main goals:

1. Repair broken status functions for Usenet/Sabnzbd downloads (addressing [BUG] Sabnzbd download status unknown #209)
2. Do so cleanly by partially abstracting core Usenet modules, and then converting their objects to/from Torrent objects as needed.

Change Details

Goal 1

• all possible Sabnzbd download statuses were codified as Schema
• More simple statuses in the vein of the TorrentStatus class were also codified
• a mapping function handles converting between the two
• as mentioned in [BUG] Sabnzbd download status unknown #209, the download status function now checks both the Sabnzbd queue, and history, as downloads will exist in one or the other depending on status

Goal 2

This was more of a bonus on top of what I promised to do. But if I am reading between the lines correctly, it seems MediaManager is headed in a direction that might more fully support Usenet in the future. But I also didn’t want to burden you (or me) by chopping your whole app in half without permission. With that in mind…

• The core logic of the Sabnzbd download client, and the Usenet schema, now exist in media_manager/usenet/
• Not desiring to suddenly split your code into a million pieces, the usenet module has a convenience class for simplifying this partial abstraction: TorrentNzbAdapter
• This handles converting Torrent objects to Nzb objects on their way into the module, and back to Torrents on their way out.
• It’s implemented via torrent/sabnzbd.py, which now acts as the abstraction layer between abstractDownloadClient (another core thing I did not want to blow up), and the actual Sabnzbd client, now in usenet/sabnzbd.py
  • All of this was done so MediaManager could have a foot in the door to a future where the code can be written in a way that is agnostic to type of download pipeline.
  • But it also just made it a heck of a lot easier to grok the functionality, being able to read and write things in the proper Usenet-first terms.

Final Thoughts

So I started just trying to fix the status bug, but I thought my work in goal 2 might help you get a bit ahead on future Usenet work. But I have no issues at all if you are not ready for this level of abstraction, or just want me to change it, or remove it. Your feedback in any way obviously welcome since this is your code!

And thank you for this project! An alternative was desperately needed, and your foundation here is quite strong.

[qualman]

Hey @maxdorninger, I could use your advice. As mentioned, everything is working properly, and I see download statues for Sabnzbd track properly and reach finished. However from there, nothing else seems to happen—the downloads just remain in the show’s download section, not imported.

Can you tell me how the operations for moving files to their home once complete is supposed to occur, so I can track this down? Because by all accounts it seems the original caller is getting the finished status properly and should be triggering some kind of move. Thank you!!

[maxdorninger]

So basically in movie and tv service.py, there is a function which looks like this, which runs periodically (scheduled by ap_scheduler):

def import_all_show_torrents() -> None:
    with next(get_session()) as db:
        tv_repository = TvRepository(db=db)
        torrent_service = TorrentService(torrent_repository=TorrentRepository(db=db))
        indexer_service = IndexerService(indexer_repository=IndexerRepository(db=db))
        tv_service = TvService(
            tv_repository=tv_repository,
            torrent_service=torrent_service,
            indexer_service=indexer_service,
        )
        log.info("Importing all torrents")
        torrents = torrent_service.get_all_torrents()
        log.info("Found %d torrents to import", len(torrents))
        for t in torrents:
            try:
                if not t.imported and t.status == TorrentStatus.finished:
                    show = torrent_service.get_show_of_torrent(torrent=t)
                    if show is None:
                        log.warning(
                            f"torrent {t.title} is not a tv torrent, skipping import."
                        )
                        continue
                    tv_service.import_torrent_files(torrent=t, show=show)
            except RuntimeError as e:
                log.error(
                    f"Error importing torrent {t.title} for show {show.name}: {e}"
                )
        log.info("Finished importing all torrents")
        db.commit()

this function calls this one for each torrent:

 def import_torrent_files(self, torrent: Torrent, show: Show) -> None:
        """
        Organizes files from a torrent into the TV directory structure, mapping them to seasons and episodes.
        :param torrent: The Torrent object
        :param show: The Show object
        """

        video_files, subtitle_files, all_files = get_files_for_import(torrent=torrent)

        success: bool = True  # determines if the import was successful, if true, the Imported flag will be set to True after the import

        log.debug(
            f"Importing these {len(video_files)} files:\n" + pprint.pformat(video_files)
        )

        season_files = self.torrent_service.get_season_files_of_torrent(torrent=torrent)
        log.info(
            f"Found {len(season_files)} season files associated with torrent {torrent.title}"
        )

        for season_file in season_files:
            season = self.get_season(season_id=season_file.season_id)
            season_import_success, imported_episodes_count = self.import_season(
                show=show,
                season=season,
                video_files=video_files,
                subtitle_files=subtitle_files,
                file_path_suffix=season_file.file_path_suffix,
            )
            if season_import_success:
                log.info(
                    f"Season {season.number} successfully imported from torrent {torrent.title}"
                )
            else:
                log.warning(
                    f"Season {season.number} failed to import from torrent {torrent.title}"
                )
                success = False

        log.info(
            f"Finished importing files for torrent {torrent.title} {'without' if success else 'with'} errors"
        )

        if success:
            torrent.imported = True
            self.torrent_service.torrent_repository.save_torrent(torrent=torrent)

            # Send successful season download notification
            if self.notification_service:
                self.notification_service.send_notification_to_all_providers(
                    title="TV Season Downloaded",
                    message=f"Successfully imported {show.name} ({show.year}) from torrent {torrent.title}.",
                )

and the import_season method then imports the files.
Movies are imported similarly.

[qualman]

Thank you!! Will try to wrap this up soon!

[hellow554]

Here are my two cents, since this is a feature I want :)

[hellow554]

What's the reason for introducing a copy of the torrent.AbstractDownloadClient here?

[hellow554]

not deriving from the abstract class?

[hellow554]

I would rewrite it as something like this:

def get_nzb_status(self, nzb: Nzb) -> NzbStatus:
        """
        Get the status of a specific download from SABnzbd.

        :param nzb: The nzb to get the status of.
        :return: The status of the nzb.
        """
        # Check the queue for in progress downloads
        log.debug("Getting Status for %s", nzb)
        if res := self._get_nzb_status(nzb):
            status = res["status"]
            log.debug("Status for nzb %s: %s", nzb.title, status)
            return sabnzbd_to_nzb_status(status)

        log.info("Didn't find any downloads for nzb %s", nzb.title)
        return NzbStatus.unknown

def _get_nzb_status(self, nzb: Nzb) -> dict | None:
        """
        Tries to get the nzb in either the current download or history from SABnzbd.

        :param nzb: The nzb to search for
        :return: A dictionary with the result.
        """
        log.debug("Trying to get nzb from progress")
        response = self.client.get_downloads(nzo_ids=nzb.hash)
        log.debug("SABnzbd downloads response: %s", response)
        nzb_slots = response["queue"]["slots"]
        if len(nzb_slots):
            return nzb_slots[0]

        log.debug("Trying to get nzb from history")
        response = self.client.get_history(nzo_ids=nzb.hash)
        log.debug("SABnzbd history response: %s", response)
        nzb_slots = response["history"]["slots"]
        if len(nzb_slots):
            return nzb_slots[0]

        log.warning("Didn't find any nzb that matches %s", nzb)

[hellow554]

I was wondering why there is no such paused state, but that's for another day, I guess?

[hellow554]

str | SabnzbdStatus not or

[qualman]

ty, will change!