Switch tov2 items #154
Switch tov2 items #154
Conversation
The puppet stage of the provisioning by vagrant now also installs node.js and grunt. This is needed for the following call of bin/update.sh to make sure all js and css files are available.
This prevents us from using any public Type/Subtype/Rarity IDs. We create our own, if we encounter a new type, but that means, we can't make sure, IDs are stable in dev environments. We should probably check in a SQL-File with "our" ID->Title tables. Disable instance pooling of Propel, because we will get memory problems otherwise...
Puppet will now: - import known types, sub-types and disciplines. That way we keep our IDs stable - clear our caches and set the request slots up - run tools/update-items-from-api.php to get the initial batch of items. Due to the amount of items, the provisioning will take quite some time and php will be swapping a lot in the end...
These classes are only slightly changed copies from the v1 API. Using the v2 API shortens the update-items-from-api.php by about a third.
This commit finishes the move to the v2/items-API of GW. We can now: - nix the ItemDB-Queue (which happened, the workers will output a message and exit with code 1) - use tools/update-items-from-api.php for item updates. The script runs about 5 minutes and creates new (Sub)Types if needed. I also adapted the Readme.
|
Hmmm... I've been playing around withthose changes and the way the priority queue is currently written makes adding new types dynamically a little tricky... For example currently the type "back" gets created. If you want to fill the item listing queue however, it does not know which priority this type should get. Since the commerce API allows us to fetch up to 200 (I think?) items at once, I am also not sure if the priority stuff is still needed at all. |
This is needed to be able to put the back items into the item listings queue. More adaptions to the Readme and disable instance pooling for propel on importing the recipe map... That prevents swapping and makes the import faster by a HUGE amount.
|
hmm, we can change the worker to do up to 200 items in 1 requests at least to speed things up. I'd like to keep the priority system in place though, over the past years I've gone back and forth between different ways of fetching the prices and the priority system has always been a way of mitigating the pain when the method of fetching the prices was slow. |
|
I took some time to look through the diff and the PR looks good btw, I just need to test and deploy it, I'm trying really hard to make some time for that, hopefully this evening! |
|
Take your time. I won't have much time this weekend to set the default |
|
it's working better than the currently broken scripts so an improvement at the very least. if it has to do with failing requests, I got a different library to do the multicurl a bit better if necesary |
|
Well... I noticed that sometimes I got single But both happened when I queried by IDs instead of using paging. I forgot to |
This branch completely switches Item updates (not listing updates) to the official GW2 API v2.
I also took the liberty to remove the classes associated with the ItemDB-Queue as it is not needed anymore. I hope I did not overlook any references.
The scripts called in the
daemonsdirectory now output an error message and exit with code 1 if they are called. That way no cron job or other automated call should slip past.