Why does Tagstudio take an eternity to convert json to SQL

[Thecreatre]

I'm mainly asking this cuz well i just want to test how well it work with my current library's how ever it takes an eternity to convert just for a not very well tagged library of 6k entries

especially cause its barely using my pc's resources (like 1% usage of cpu most of the time and marginal usage of the drive)

just kinda asking cuz I'm inclined to make enhancement req to uh make it use more of the speed it could be using

ig this is more of a short term problem cause once everyone is converted to this new version the conversion process taking 2 Saturn orbits to occur isn't really a problem anymore

but yaknow its annoying rn so bleh

(oh and ye am referring to 9.5 prerelease for context)

[Thecreatre]

i also had my friend try it and theirs also took forever but when it finished had uh discrepancies on all the things where it was feasible for it to say discrepancies (except entries, tags, file extensions, and extensions list type)

[Thecreatre]

yeah.. my 6k test thing i s uh

its been like an hour+ and its still hasnt converted

[CyanVoxel]

Wait so it hasn't actually finished yet? You left that out. It may be stuck on something, I had no idea otherwise from your post

[CyanVoxel]

Why does Tagstudio take an eternity to convert json to SQL?
i also had my friend try it and theirs also took forever
the conversion process taking 2 Saturn orbits to occur

Terms like "take an eternity" or "took forever" are subjective and not helpful for determining if there's a problem with the conversion process. Do you consider 5 minutes an eternity? Or did it take 5 hours? 20 Hours? I have no idea from this post. Giving the actual time these library conversion took along with the library stats displayed on the conversion screen along with your system specs would be very helpful and necessary to determining if there's an actual issue or not.

just kinda asking cuz I'm inclined to make enhancement req to uh make it use more of the speed it could be using

ig this is more of a short term problem cause once everyone is converted to this new version the conversion process taking 2 Saturn orbits to occur isn't really a problem anymore

The conversion process is designed to use the same systems as the normal SQL library while taking a step to ensure that all of your data has converted properly and will report back any discrepancies its found. There currently isn't much more performance to squeeze out imo, and we take handling user's data very seriously above all else.

i also had my friend try it and theirs also took forever but when it finished had uh discrepancies on all the things where it was feasible for it to say discrepancies (except entries, tags, file extensions, and extensions list type)

This should not happen most of the time after a library has converted. This would be incredibly useful as a bug report instead of an off-hand comment. Please ask your friend to open a bug report with the discrepancies found if possible.

[Thecreatre]

neither for the onee i was trying right now, im just estimating

[Thecreatre]

on the data integrity thing: i was thinking that it was to keep it sound and stuff, but the fact it was barely hitting my cpu (amd ryzen 5 3600 6 core proccesssor) made me think that uh something was going wrong

[Thecreatre]

Now. im not sure for what all the steps are but tag studio's conversion it seems to be stuck on making the Data base tables (as its been saying "creating database tables" for as long as i renmber the process going)

[CyanVoxel]

on the data integrity thing: i was thinking that it was to keep it sound and stuff, but the fact it was barely hitting my cpu (amd ryzen 5 3600 6 core proccesssor) made me think that uh something was going wrong

The library conversion process is mostly tied to your disk speed, though the processor speed helps.

Now. im not sure for what all the steps are but tag studio's conversion it seems to be stuck on making the Data base tables (as its been saying "creating database tables" for as long as i renmber the process going)

That does in fact seem like something's stuck. That's the first step and normally is so quick that you may not even see it. Could you try closing the program and restarting the conversion process to see if it gets past that step?

[Thecreatre]

oh yeah in both of the times ive tried to convert on my own machine (yes i forgot to tell i tried todo it with an other library, that one w as way bigger tho)

it both times got stuck on the table generation i beleave but ill give an other go for the sake uh fixing it

[Thecreatre]

altho ig i should ask weather i should just shut it down and make it stop converting

[Thecreatre]

(sense dunno if you can extract any useful info from it)

[Thecreatre]

other than wrting smth to the disk supposidly

[CyanVoxel]

Yes, it seems like the conversion process is stuck not being able to start. It could be related to now having write permissions on your disk, but I'm just taking a guess there. Could you try closing the program and restarting the conversion process to see if it gets past that step?

[Thecreatre]

oh.. and i forgot that it sends this err when i force it to stop (maybe an other useful comment is that im using the portable version of the app (cuz i unno))

[CyanVoxel]

And for your friend, are you sure that their conversion still listed discrepancies after the conversion finished? And it gave a message box with a list of all the specific issues it found? Otherwise it's normal to see readouts for in-progress checks to say that there are discrepancies found before the actual checks pass. It hasn't finished the checks, therefore it defaults to saying that something could be wrong.

[Thecreatre]

also update trying it again, it got thru the tables in under a second

[Thecreatre]

and it says it migrated the test thing fullly allready (confusion)

[CyanVoxel]

"Already?" I think you mean that it just finished quickly? It doesn't pick up from previous migrations, it starts over

[Thecreatre]

uh "already" as in i didnt expect it to finish before i put a message in here

[Thecreatre]

hm.. i think the reason i thought my friend's conversion when arry is due to the fact the update to the save file format is that it shuffles all the files compared to how they where sorted in the old version

[CyanVoxel]

yaknow these two arnt the same, in stuff

There's also nothing saying that these are the same files, either. And in one window you have a search going while you don't in the other. This is just showing different files between these windows. If the same files exist in both libraries, you're good.

[Thecreatre]

although now i do think it was red when they clicked finish conversion (er mainly cause i now renmber that i told them to click finish conversion in the first place)

[CyanVoxel]

You had originally said:

i also had my friend try it and theirs also took forever but when it finished had uh discrepancies on all the things where it was feasible for it to say discrepancies (except entries, tags, file extensions, and extensions list type)

So were these actual discrepancies listed by the conversion window (after the conversion finished) or were you just using the same term for the fact that the files showed up in a different order after a successful conversion?

i believe so yes, the reason i backed away abit was i couldn't renmber ever seeing them click on the "finish conversion" button in the stream so was maybe thinking that they may have clicked it while i was away or smth and the stuff also turned green at the time

The internal conversion process finishes when the progress bar goes dormant, the text says "Migration Complete!" and the "Finish Migration" button becomes actually available to press. If they reached that point and there were real discrepancies, the red text would've remained and they also would've gotten a popup listing out the specific files and data discrepancies that were found. If they never made it to that point, then yeah it seems like everything is normal since the process was never allowed to finish.

[Thecreatre]

yaknow these two arnt the same, in stuff

There's also nothing saying that these are the same files, either. And in one window you have a search going while you don't in the other. This is just showing different files between these windows. If the same files exist in both libraries, you're good.

hm, well in that comment was more just bringing up that the way things are sorted seemed to be different (so was like " maybe this is why i thought it was a messed up conversion" instead of me renumbering that it was throwing red discrepancies in the menu before this)

[Thecreatre]

You had originally said:

i also had my friend try it and theirs also took forever but when it finished had uh discrepancies on all the things where it was feasible for it to say discrepancies (except entries, tags, file extensions, and extensions list type)

So were these actual discrepancies listed by the conversion window (after the conversion finished) or were you just using the same term for the fact that the files showed up in a different order after a successful conversion?

i believe so yes, the reason i backed away abit was i couldn't renmber ever seeing them click on the "finish conversion" button in the stream so was maybe thinking that they may have clicked it while i was away or smth and the stuff also turned green at the time

The internal conversion process finishes when the progress bar goes dormant, the text says "Migration Complete!" and the "Finish Migration" button becomes actually available to press. If they reached that point and there were real discrepancies, the red text would've remained and they also would've gotten a popup listing out the specific files and data discrepancies that were found. If they never made it to that point, then yeah it seems like everything is normal since the process was never allowed to finish.

hm... yeah i dunno

[Thecreatre]

ig this comment chain is morso just me getting embarrassed from me not explain things very well

, unrelated, but why is the stuff reshuffled after conversion anyway

[CyanVoxel]

There was no true "order" in the JSON library. So when switching to SQL where we're actually keeping track of things like the date you added a file to your library, there's now actual proper types of orders starting from this update onwards.

You may also just be used to ascending / descending order and have it flipped.

[Thecreatre]

no.. its definitely different (i tried both accending and decending)

[Thecreatre]

hm.. hold on.. it might jsut be that the conversion also uh renmbered to check for new files and the old save uh doesnt have the new files

[Thecreatre]

so now that ive updated the old file lemme try converting again hold on

[Thecreatre]

oh. it was the thing set to accending, bleh

[Thecreatre]

(oh right the 3rd isssue to made was the tag searching autocorrect being nonfunctional as in it only finds what i can only describe as the "default tagss")

[CyanVoxel]

like the back and forward button not working correctly

This would be appreciated as a new bug report.

collage button being missing

As the README has always said, this was an unfinished feature. It's presence was further removed in 9.5.

tag searching autocorrect being nonfunctional

This would also be appreciated as a new bug report, at least to see if there's anything else going on.

[Thecreatre]

hm new one to add to t hee list im making here (cuz im too lazy to use discord ig) is that the "recent library's" and or the sister function that auto opens the last library have been removed and or never been re added

[CyanVoxel]

If you're experiencing an issue with the "Recent Library" list that's been moved to the File menu or an issue with the "Open Library on Start" setting, please open relevant bug reports for them.

[Thecreatre]

oh, cool that it hasnt been removed altho, the "open library on start" setting was allready enabled so, ig it isnt working