Dev #52
Merged
Dev #52
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* #62 upload logs every n seconds. * trim some code cruft (aka old requests based calls)
* Fix for bad serialization Fix for bad timeout setting * remove generated outputs * gather stats * run configuration updates * Configuration updates for a couple missing env variables
* Gleaner issue: ecrr_examples not harvesting. use gh_pages reference nabu issue: no ecrr_submitted release * update ecrr repo name * add some run cli doco
* update sitemapcheck README * graph types csv asset * Stats from missing_graph * update to dagster 1.5.6 * weekly task examples * fix compose files with correct variables * fix compose file. * address a few other locations. --------- Co-authored-by: Douglas Fils <[email protected]> Co-authored-by: Jeff McKenna <[email protected]>
update eco_override container
* add SCHED_HOSTNAME * update envfile.env, and colorize hint
* added local docker config * mounted generated codes to the container * changes for local run (#25) * Update compose_project.yaml restart no. * changes for local run * fix bash logic * specify platform * update script to remind user to update the generated code. add couple runConfigurations to help with that --------- Co-authored-by: Ya-Lan Yang <[email protected]> * try to use the docker.py * fix get_archive of logs add configs to the Dockerfile_local * Add some * Change the logic of setting up the network * Initial Resource Configuration * Start rework of the uploading tasks. * reworking * Config working. test asset underway * Config working. test asset working * Config working. test asset working * Starting on a sensor * Starting on a sensor * Starting on a sensor. change namspace location * NOTES.md * Merge and remove top level gleanerio directory * add protype tennant file * WOrking.. need to move to the minios3... the boto is inconsistent in how to handle Keys * reworked a couple of thins to do the source partitioning * reworked a couple of thins to do the source partitioning * summon and release job working * Creates dynmaic partitions. * Move some stuff to utilities. * start to add summarize * start to add summarize * ponder a partition config for a release sensor * added pangaea datasets * added report_stats to weekly_sch.py * moved report to asset graph * update earthcube-utilities version * added 3 communities summary reports * removed unnecessary returns * added dagster logger * refined codes * Added TODO --------- Co-authored-by: David Valentine <[email protected]>
* see why deployment fails * see why deployment fails * see why deployment fails * If there are duplicate names in the gleanerconfig.yaml, two definitions generated. * add missing variaibles * REmove variables... add not to use aws_ variables for s3 keys
summary
spelling
* Quick overview of how dagster is deployed by ECO * added local docker config * mounted generated codes to the container * Quick overview of how dagster is deployed by ECO * Add a note about refactoring * changes for local run (#25) * Update compose_project.yaml restart no. * changes for local run * fix bash logic * specify platform * update script to remind user to update the generated code. add couple runConfigurations to help with that --------- Co-authored-by: Ya-Lan Yang <[email protected]> * try to use the docker.py * fix get_archive of logs add configs to the Dockerfile_local * Add some * Change the logic of setting up the network * updated dagster local deployment * added dagster deployment * fix * fixed numbered points * typo * updates * Initial Resource Configuration * Start rework of the uploading tasks. * reworking * Config working. test asset underway * Config working. test asset working * Config working. test asset working * Starting on a sensor * Starting on a sensor * Starting on a sensor. change namspace location * NOTES.md * Merge and remove top level gleanerio directory * add protype tennant file * WOrking.. need to move to the minios3... the boto is inconsistent in how to handle Keys * summon and release job working * Creates dynmaic partitions. * Move some stuff to utilities. * add tennants from s3 * Update dagster to 1.7.6 Paritions are working to be generated from a sources listed in a gleaner config in s3. * Update dagster to 1.7.6 Paritions are working to be generated from a sources listed in a gleaner config in s3. fix issues with graph reports * #101 Update dagster to 1.7.6 Paritions are working to be generated from a sources listed in a gleaner config in s3. fix issues with graph reports * Sensors, tenant load releease files (#103) * remove extra compse env from local * catch and just log to dagster uploaded gleanerio log errors * #102 tennant loading prototyping. Setup to for loading releases and creating tenants. Create graph namespaces using EC Utils use the tenants.yaml graph namespaces Uploading releases slack #19 fixing spelling tenant in tasks, too * Fix upload release to use partitions (#105) * remove extra compse env from local * catch and just log to dagster uploaded gleanerio log errors * #102 tennant loading prototyping. Setup to for loading releases and creating tenants. Create graph namespaces using EC Utils slack #19 fixing spelling tenant in tasks, too schedule... Fix upload release. Set to run automatically * begin documentation update * dcos and clean up tenant sensor * dcos and clean up tenant sensor * try to do a multiple jobs for tenant_sensor... can't without more coding * Improve docs * Improve docs * Improve docs fix envFile.env * Improve docs fix envFile.env * Improve docs * Improve docs * Updates to get docker stack working for tasks and ingest workflows * merge with dev. Fix community sensor * fix ecrr * cleaning up some compose env variables * cleaning up some compose env variables * update readme, and compose_project.yaml * trigger github action for dev_101_ingest_sources_wf * trigger github action for dev_101_ingest_sources_wf * trigger github action for dev_101_ingest_sources_wf * trigger github action for dev_101_ingest_sources_wf * trigger github action for dev_101_ingest_sources_wf * trigger github action for dev_101_ingest_sources_wf * trigger github action for dev_101_ingest_sources_wf * fix issue in stats code * rename task_tenant to task_tenant_names * fix issues with resource configuraiton * fix issues with resource configuraiton * fix issues load stats * fix issues load stats * some runConfigs * add volume so materialized persist across rebuilds * slack sensor in every workflow * remove hardcoded @Weekly schedule * remove hardcoded @Weekly schedule. dont forget to add to compose files * add note about present meaning of PROJECT * disable all_report_stats * Use key_prefix to separate workflows. * doc changes * #111, and create volumes * sync python version in docker * fix an asset key * update requirements * fix missing non-ssl s3 update runConfigs to use prefix'd asset names * #116 * #119 --------- Co-authored-by: Ya-Lan Yang <[email protected]>
Lots of work needed
* check sitemaps #132, #133 * works better if you return an error * concurrency by tags
* add stats reports to tenant * Update contanerize.yaml * add reports * update graph summary endpoint * add csv * Update tenants.py * Update tenants.py * Update tenants.py * Update tenants.py * update yamls * update url * add csv config
Merge in Dave's work in earthcube scheduler, dev to dev
Point earthcubeutilities to dev
Update requirements.txt (#154)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.