I improved community golang cli tool, please take a look, might be worth making it official

[GrigoryEvko]

Hi, I made a fork of Golang CLI tool and improved it substantially, java app didn't start on my server and I decided to tweak rest api downloader, please take a look:
https://github.com/GrigoryEvko/NBIA_data_retriever_CLI

it works nicely now with many features I lacked from the original go tool, didn't have a chance to launch yours though

../nbia-data-retriever-cli -i PROSTATEx-v1-doiJNLP.tcia -p 20 --max-connections 20 --max-retries 5 -o ./mri
Found 18321 series to fetch metadata for
[18321/18321] 100.0% | Fetched: 0 | Cached: 18321 | Failed: 0 | Current: 1.3.6.1.4.1.14519.5.2.1.7310.5...
Successfully fetched metadata for 18321 files

Downloading 18321 series with 20 workers...

[18321/18321] 100.0% | Downloaded: 10181 | Skipped: 8140 | Failed: 0 | Current: 1.3.6.1.4.1.14519.5.2.1.7311.5...

=== Download Summary ===
Total files: 18321
Downloaded: 10181
Skipped: 8140
Failed: 0
Total time: 8m34s
Average rate: 35.7 files/second

I hope you can link it somewhere or fork the code and continue working on the cli tool on your own!

[GrigoryEvko]

Btw sorry if it was me crashing your auth server today! It stopped responding when i was running test suite, so that's likely.

[kirbyju]

Hi @GrigoryEvko sorry for the delayed reply, and thanks so much for giving us a heads up about this! Could we get together for a chat sometime soon to discuss? Please drop me a line at justin.kirby at nih.gov and we can find a time.

[kirbyju]

In the meantime, can you update your github repo with the Topic tag "tcia-dac" so that it automatically appears in our list of community developed Github tools? I've added your tool to our Data Analysis Center registry as well (https://wiki.cancerimagingarchive.net/x/x49XAQ).

[GrigoryEvko]

In the meantime, can you update your github repo with the Topic tag "tcia-dac" so that it automatically appears in our list of community developed Github tools? I've added your tool to our Data Analysis Center registry as well (https://wiki.cancerimagingarchive.net/x/x49XAQ).

Cool, added it!

[kirbyju]

Hi @GrigoryEvko -- just wanted to let you know we've decided to run with this! We're currently working on building a GUI layer to sit on top of it and are extending it with additional features such as the ability to download other data types that we host outside of NBIA using other tools. Thanks again!

[GrigoryEvko]

Oh great! Sorry i didn't reply to your email, once I'll have some time I'll try to improve it and make more things clear and consistent. Regarding threadings and number of connections, i think that was not related to your service, but to my network at the moment.
I'd like to help with another option to download what's stored on IBM Aspera as I tried everything they offer (cli, browser + gui app) and it just didn't work at all. And why downloading 500MB of annotations even require 50MB gui app to be installed with browser extension...

[kirbyju]

Nobody on our side has really started investigating a way to tie this to Aspera Faspex packages yet, so it would be phenomenal if you wanted to investigate that! I think https://github.com/laurent-martin/aspera-api-examples/tree/main might be helpful if you haven't already found that repo. I know Martin is on the IBM team that manages Aspera. Unfortunately if you look in https://github.com/laurent-martin/aspera-api-examples/tree/main/app the Go example says it's a work in progress, but one of the files in there was modified last week so maybe more is coming? I also found https://github.com/tera-insights/go-aspera this morning but haven't had any time to review it yet.

Other items of note:

1. https://github.com/aregesteruams/NBIA_data_retriever_gui is our unofficial primary fork of what you did at the moment.
2. I've been playing with ways to add spreadsheet-based manifest files that apply different logic based on column header names, but be warned, this is largely a vibe coded effort as I'm the project manager (not one of our real devs) 😆 https://github.com/kirbyju/TCIA_data_retriever is where that's happening.