From a353d3a2a73f2140578cbf4aba81cf35187fc046 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Thu, 16 Jun 2022 13:30:36 -0700 Subject: [PATCH 01/30] first crack at commands --- docs/nul | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 docs/nul diff --git a/docs/nul b/docs/nul new file mode 100644 index 000000000..e69de29bb From 8e810850746033c8fc71482b2f4cb16556e393c3 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Thu, 16 Jun 2022 13:31:45 -0700 Subject: [PATCH 02/30] Revert "first crack at commands" This reverts commit a353d3a2a73f2140578cbf4aba81cf35187fc046. --- docs/nul | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 docs/nul diff --git a/docs/nul b/docs/nul deleted file mode 100644 index e69de29bb..000000000 From 742434ad27e90d5958b23f10e41e9fea25330005 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Thu, 16 Jun 2022 13:43:32 -0700 Subject: [PATCH 03/30] first crack at commands --- docs/cli-tutorial.md | 196 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 196 insertions(+) create mode 100644 docs/cli-tutorial.md diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md new file mode 100644 index 000000000..c6ec71ce6 --- /dev/null +++ b/docs/cli-tutorial.md @@ -0,0 +1,196 @@ + + +## Commands + +### Auth + +Initialize with user name and password + +```console +$ planet auth init +``` + +See API key that is being used for requests + +```console +$ planet auth value +``` + +*NOTE: Right now you must init before you get help, until https://github.com/planetlabs/planet-client-python/issues/463 is resolved* + +### Get Help + +Most commands are pretty self-documenting, can use `--help` after any command to get usage info. And many will give you help +if you don't form a complete command. + +```console +$ planet --help +$ planet +$ planet orders +$ planet orders list --help +``` + +### Orders + +See recent orders you've made + +```console +$ planet orders list +``` + +Print them more nicely +(both are options, jq is a bit more advanced, but prints a bit nicer and is more powerful) + +```console +$ planet orders list | jq +$ planet orders list --pretty +``` + +Get the number of your recent orders. (-s collects objects into a single array) + +```console +$ planet orders list | jq -s length +``` + +Get more info on a single order + +```console +$ planet orders get 782b414e-4e34-4f31-86f4-5b757bd062d7 +``` + +Download the order + +```console +$ planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 +``` + +Download it to a different directory (relative to where the command is run) + + +```console +$ mkdir psscene +$ planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --directory psscene +``` + +Download to a directory specified absolutely (in this case to my desktop) + +```console +$ planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --directory /Users/cholmes/Desktop/ +``` + +Verify the checksum to make sure the file I got wasn't corrupted along the way (during download, etc) + +```console +$ planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --checksum MD5 +``` + +Create a new order request from a scene ID found in Explorer + +```console +$ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b --item-type PSScene +``` + +Save to a file in the directory where the command was run + +```console +$ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b --item-type PSScene > request-1.json +``` + +Make a new order with that request + +```console +$ planet orders create request-1.json +``` + +Create the request and make a new order with it all in one call + +```console +$ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b --item-type PSScene | planet orders create +``` + +Download - only works when the order is ready: + +```console +$ planet orders download 65df4eb0-e416-4243-a4d2-38afcf382c30 +``` + +Wait until the order is ready for download, then order it: +(Note that \ just tells the console to treat the next line as the same one - used here to be a bit easier to read) + +```console +$ planet orders wait 65df4eb0-e416-4243-a4d2-38afcf382c30 \ +&& planet orders download 65df4eb0-e416-4243-a4d2-38afcf382c30 +``` + +Create an order from a request, wait for it and download when ready +(TODO: don't quite have this working yet) + +```console +$ planet orders create request-1.json | jq -r '.id' | planet orders wait - | jq -r '.id' | planet orders download - +``` + + +planet orders list + +` --pretty or | jq` + +``` + +Order and download a scene id found from explorer + +```console +20220605_124027_64_242b +``` + +```console +$ +``` + +```console +$ +``` + +```console +$ +``` + +As surface reflectance +Order and download the results of 3-4 of the first set of searches above +Order and download the above as STAC to: +Add to an existing STAC catalog with stactools +Browse resulting catalog with STAC Browser +Visualize results in Unfolded Studio +Do this with both surface reflectance data as well as clipped, composite, harmonized data. +Run search of geojson of 7 moderate-sized fields, download clipped, cloud-free imagery of each. +Do clip & composite of them +Clip, composite & harmonize +Clip, composite and NDVI +COG’s for each of the outputs + + +Run a search for planetscope data on a bounding box in Iowa +Run search for planetscope with cloud cover less than 20 +Run search with permission filter to just be data that can be downloaded +Run search with just good sun angles +Run search for sentinel 2 images +Run search for landsat8 images +Run search for sentinel 1 images +Run search for all data from Planet & public satellites for the last 5 grow seasons in Iowa (April 15 - October 15th) +Get it into a shapefile for use by naive GIS people +Get stats of number of images captured by satellite, show in chart +Get stats of number of cloud free images by satellite +Run search with a geojson of 7 small to moderate sized fields, try out a few filters +Run search and turn results into STAC output, to: +Understand results with stac-terminal +Add to an existing stac catalog with stactools + + + +Download 3 months of imagery over a small field stored in a shapefile, but only download the imagery that is 90% cloud free over the field. +https://hello.planet.com/code/benjamin/cloud_free_order can help us +Set up a subscription of planetscope data over a bound box in iowa +With cloud cover less than 25% +Set up a subscription clipped to the 7 small to moderate-sized fields +And then do it with harmonization & composites +Only use those that are 90% cloud-free over the actual AOI +Backfill a subscription for the previous 6 months From f69775fc2fa69605176e042935156bcbc5f77f09 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Tue, 21 Jun 2022 15:46:16 -0700 Subject: [PATCH 04/30] more examples --- docs/cli-tutorial.md | 255 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 247 insertions(+), 8 deletions(-) diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md index c6ec71ce6..cd2dc22fc 100644 --- a/docs/cli-tutorial.md +++ b/docs/cli-tutorial.md @@ -85,18 +85,22 @@ $ planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --checksum MD5 ``` Create a new order request from a scene ID found in Explorer +(Note that \ just tells the console to treat the next line as the same one - used here to be a bit easier to read) ```console -$ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b --item-type PSScene +$ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ + --item-type PSScene ``` Save to a file in the directory where the command was run ```console -$ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b --item-type PSScene > request-1.json +$ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ + --item-type PSScene > request-1.json ``` -Make a new order with that request +Make a new order with that request (should be the same as +[this one](https://gist.githubusercontent.com/cholmes/892f851d5c55f7cf93e210595750ecfe/raw/4c255dcdf0973e0f72d43d82c776d8251fb7545e/request-1.json)) ```console $ planet orders create request-1.json @@ -105,17 +109,17 @@ $ planet orders create request-1.json Create the request and make a new order with it all in one call ```console -$ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b --item-type PSScene | planet orders create +$ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ + --item-type PSScene | planet orders create ``` -Download - only works when the order is ready: +Download the order. Note this only works when the order is ready: ```console $ planet orders download 65df4eb0-e416-4243-a4d2-38afcf382c30 ``` Wait until the order is ready for download, then order it: -(Note that \ just tells the console to treat the next line as the same one - used here to be a bit easier to read) ```console $ planet orders wait 65df4eb0-e416-4243-a4d2-38afcf382c30 \ @@ -123,12 +127,247 @@ $ planet orders wait 65df4eb0-e416-4243-a4d2-38afcf382c30 \ ``` Create an order from a request, wait for it and download when ready -(TODO: don't quite have this working yet) +(TODO: this works, but doesn't match usage examples, so should align to that.) + +```console +$ id=`planet orders create request-1.json | jq -r '.id'` && planet orders wait $id && planet orders download $id +``` + +Create an order clipped to a geometry + +(download [raw json](https://gist.githubusercontent.com/cholmes/c7736ac5241d77605524d01ed2dc57a1/raw/7d24e02ba894e64c4c737c253a0cce4cac54167c/geometry.geojson) +from github, or [preview](https://gist.github.com/cholmes/c7736ac5241d77605524d01ed2dc57a1)) + +```console +$ planet orders request --item-type PSScene --clip geometry.geojson --name clipped-geom \ + --bundle analytic_sr_udm2 --id 20220605_124027_64_242b | planet orders create +``` + +Order Two Scenes in one order + +```console +$ planet orders request --item-type PSScene --name "Two Scenes" \ + --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b | planet orders create +``` + +Composite them together + +Use this, save as tools-composite.json + +```json +[ + { + "composite": { + } + } +] +``` + +Then create command and order + +```console +$ planet orders request --item-type PSScene --name "Two Scenes Composited" \ + --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b --tools tools-composite.json | planet orders create +``` + +Get the output as a cloud-optimized GeoTIFF: +(save as tools-cog.json) + +```json +[ + { + "composite": { + } + }, + { + "file_format": { + "format": "COG" + } + } +] +``` + +The following command just shows the output, you can pipe into orders create if you'd like: + +```console +$ planet orders request --item-type PSScene --name "Two Scenes Composited" \ + --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b --tools tools-cog.json +``` + +To clip and composite you need to specify the clip in the tools (instead of `--clip`). +Can get it from [this gist](https://gist.github.com/cholmes/378d050a263ae433ddbbb91c3439994b) + +```console +$ planet orders request --item-type PSScene --name "Two Scenes Clipped and Composited" \ + --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b --tools tools-clip-composite.json +``` + +Or can even stream that in: + +```console +$ url -s https://gist.githubusercontent.com/cholmes/378d050a263ae433ddbbb91c3439994b/raw/ebcaa54cacdc6f696a0506705785f8ff8dae9af1/ \ +tools-clip-composite.json | planet orders request --item-type PSScene --name "Streaming Clip & Composite" \ + --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b --tools - | planet orders create +``` + +TODO: STAC Output + +### Data + +Run a search for all the latest planetscope scenes that you have download access to: + +```console +$ planet data filter > filter.json +$ planet data search-quick PSScene filter.json +``` + +Or using piping (recommended) + +```console +$ planet data filter | planet data search-quick PSScene - +``` + +Run a search for all the latest PS Scenes in a bounding box in Iowa: + +Save this as geometry.json + +```json +{ + "type": "FeatureCollection", + "features": [ + { + "type": "Feature", + "properties": {}, + "geometry": { + "type": "Polygon", + "coordinates": [ + [ + [ + -93.7353, + 41.6236 + ], + [ + -92.2741, + 41.6236 + ], + [ + -92.2741, + 42.3747 + ], + [ + -93.7353, + 42.3747 + ], + [ + -93.7353, + 41.6236 + ] + ] + ] + } + } + ] +} +``` + +```console +$ planet data filter --geom geometry.json | planet data search-quick PSScene - +``` + +And turn into GeoJSON ```console -$ planet orders create request-1.json | jq -r '.id' | planet orders wait - | jq -r '.id' | planet orders download - +$ planet data filter --geom geometry.json | planet data search-quick PSScene - | planet collect ``` +```console +$ +``` + + +```console +$ +``` + +```console +$ +``` + +```console +$ +``` + + +```console +$ +``` + +```console +$ +``` + +```console +$ +``` + + +Run a search for planetscope data on a bounding box in Iowa + + + +```json +{ + "type": "FeatureCollection", + "features": [ + { + "type": "Feature", + "properties": {}, + "geometry": { + "type": "Polygon", + "coordinates": [ + [ + [ + -93.7353, + 41.6236 + ], + [ + -92.2741, + 41.6236 + ], + [ + -92.2741, + 42.3747 + ], + [ + -93.7353, + 42.3747 + ], + [ + -93.7353, + 41.6236 + ] + ] + ] + } + } + ] +} +``` + +Run search for planetscope with cloud cover less than 20 +Run search with permission filter to just be data that can be downloaded +Run search with just good sun angles +Run search for sentinel 2 images +Run search for landsat8 images +Run search for sentinel 1 images +Run search for all data from Planet & public satellites for the last 5 grow seasons in Iowa (April 15 - October 15th) +Get it into a shapefile for use by naive GIS people +Get stats of number of images captured by satellite, show in chart +Get stats of number of cloud free images by satellite +Run search with a geojson of 7 small to moderate sized fields, try out a few filters +Run search and turn results into STAC output, to: +Understand results with stac-terminal +Add to an existing stac catalog with stactools planet orders list From 1ea9408fb437f1c4b428010546a361853a26e446 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Tue, 21 Jun 2022 15:46:33 -0700 Subject: [PATCH 05/30] first commit, mostly placeholder --- docs/cli-concepts.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 docs/cli-concepts.md diff --git a/docs/cli-concepts.md b/docs/cli-concepts.md new file mode 100644 index 000000000..a41bc9922 --- /dev/null +++ b/docs/cli-concepts.md @@ -0,0 +1,30 @@ +# Command-line Interface (CLI) Concepts + +Planet's command-line interface is built of composable pieces that combine in powerful ways. +To take full advantage of all that Planet's CLI offers there are a few concepts and tools +that are worth explaining. + +## Core Unix Concepts + +These are core functions from Unix + +### Piping + +incuding reading from a pipe with - + +stringing several together + +### Output to file + +> + +### head & tail + +### Running a command within another + +## JQ + +## curl + +## sed + From 28dcb7799a978ddab21802538475c5040ec7eb65 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Tue, 21 Jun 2022 22:18:42 -0700 Subject: [PATCH 06/30] more additions --- docs/cli-tutorial.md | 55 +++++++++++++++++++++++++++++++++++------ docs/order-request.json | 1 + 2 files changed, 49 insertions(+), 7 deletions(-) create mode 100644 docs/order-request.json diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md index cd2dc22fc..72f093782 100644 --- a/docs/cli-tutorial.md +++ b/docs/cli-tutorial.md @@ -274,42 +274,83 @@ Save this as geometry.json $ planet data filter --geom geometry.json | planet data search-quick PSScene - ``` -And turn into GeoJSON +And turn into GeoJSON (many gis programs do better with 'real' geojson, the default output is newline delimited geojson) ```console $ planet data filter --geom geometry.json | planet data search-quick PSScene - | planet collect ``` +Search just for 8-band assets + ```console -$ +$ planet data filter --asset ortho_analytic_8b_sr | planet data search-quick PSScene - ``` +Search for 8-band assets that also have a UDM ```console -$ +$ planet data filter --asset ortho_analytic_8b_sr --asset udm2 | planet data search-quick PSScene - ``` +Search for any 8 band data, not just ones that you have download access to (default is only those you have access to) + ```console -$ +$ planet data filter --permission false --asset ortho_analytic_8b_sr | planet data search-quick PSScene - ``` +Make a filter for data acquired in July 2021 + ```console -$ +$ planet data filter --date-range acquired gte 2021-07-01 --date-range acquired lt 2021-08-01 ``` +Make a filter for data with clear pixels greater than 90% ```console -$ +$ planet data filter --range clear_percent gt 90 ``` +Search for all landsat 8, sentinel 2 and planetscope images in an area of interest + ```console -$ +$ planet data filter --geom geometry.json | planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G +``` + +Get the id of the most recent skysat image taken (that you have download access to) + +```console +$ planet data filter | planet data search-quick SkySatCollect --limit 1 - | jq -r .id +``` + +Order the most recent skysat image taken. + +```console +$ planet orders request --name "SkySat Latest" --item-type SkySatCollect --bundle analytic \ +--id `planet data filter | planet data search-quick SkySatCollect --limit 1 - | jq -r .id` \ +| planet orders create +``` + +Get the 5 latest cloud free images in an area and create an order that clips to that area, using +[geometry.geojson](https://gist.github.com/cholmes/378d050a263ae433ddbbb91c3439994b) from above: +(this one uses variables in unix) +(note that you need the tr and sd to format the output of jq into the comma delimited list we need as input. There is likely some better way to do this...) +(currently lists two options, but only because range isn't yet working, so once that is we'll take it away. May also switch to PSScene) + + +```console +$ ids=`planet data filter --geom geometry.geojson | planet data search-quick SkySatCollect --limit 5 - | jq -r .id | tr '\n' , | sed 's/.$//'` +$ ids=`planet data filter --geom geometry.geojson --range clear_percent gt 90 | planet data search-quick SkySatCollect --limit 5 - | jq -r .id | tr '\n' , | sed 's/.$//'` +$ planet orders request -- "Clipped Scenes" --item-type SkySatCollect --bundle analytic --id $ids | planet orders create --clip geometry.geojson + ``` ```console $ ``` +```console +$ +``` Run a search for planetscope data on a bounding box in Iowa diff --git a/docs/order-request.json b/docs/order-request.json new file mode 100644 index 000000000..fa07c4be8 --- /dev/null +++ b/docs/order-request.json @@ -0,0 +1 @@ +{"name": "clipped-geom", "products": [{"item_ids": ["20220605_124027_64_242b"], "item_type": "psscene", "product_bundle": "analytic_sr_udm2"}]} From b62b0b5476da571eae076275b01366409cff4221 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Tue, 21 Jun 2022 22:26:55 -0700 Subject: [PATCH 07/30] another thing to add --- docs/cli-concepts.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/cli-concepts.md b/docs/cli-concepts.md index a41bc9922..80b49b9dd 100644 --- a/docs/cli-concepts.md +++ b/docs/cli-concepts.md @@ -22,6 +22,8 @@ stringing several together ### Running a command within another +### variables + ## JQ ## curl From 8a291c4950a0dee16ee0241566d86bd562002d96 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Mon, 27 Jun 2022 22:22:22 -0700 Subject: [PATCH 08/30] more improvements --- docs/cli-tutorial.md | 155 ++++++------------------------------------- 1 file changed, 22 insertions(+), 133 deletions(-) diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md index 72f093782..ba7a6a03f 100644 --- a/docs/cli-tutorial.md +++ b/docs/cli-tutorial.md @@ -1,4 +1,8 @@ +## About + +This tutorial is designed to show off the core Planet command-line interface capabilities, without +requiring ## Commands @@ -127,7 +131,6 @@ $ planet orders wait 65df4eb0-e416-4243-a4d2-38afcf382c30 \ ``` Create an order from a request, wait for it and download when ready -(TODO: this works, but doesn't match usage examples, so should align to that.) ```console $ id=`planet orders create request-1.json | jq -r '.id'` && planet orders wait $id && planet orders download $id @@ -310,6 +313,12 @@ Make a filter for data with clear pixels greater than 90% $ planet data filter --range clear_percent gt 90 ``` +Make a filter for all data in a single strip: + +```console +$ planet data filter --string-in strip_id 5743640 +``` + Search for all landsat 8, sentinel 2 and planetscope images in an area of interest ```console @@ -322,7 +331,8 @@ Get the id of the most recent skysat image taken (that you have download access $ planet data filter | planet data search-quick SkySatCollect --limit 1 - | jq -r .id ``` -Order the most recent skysat image taken. +Order the most recent skysat image published +(TODO: we need 'sort' to be able to show most recent acquired) ```console $ planet orders request --name "SkySat Latest" --item-type SkySatCollect --bundle analytic \ @@ -334,143 +344,22 @@ Get the 5 latest cloud free images in an area and create an order that clips to [geometry.geojson](https://gist.github.com/cholmes/378d050a263ae433ddbbb91c3439994b) from above: (this one uses variables in unix) (note that you need the tr and sd to format the output of jq into the comma delimited list we need as input. There is likely some better way to do this...) -(currently lists two options, but only because range isn't yet working, so once that is we'll take it away. May also switch to PSScene) ```console -$ ids=`planet data filter --geom geometry.geojson | planet data search-quick SkySatCollect --limit 5 - | jq -r .id | tr '\n' , | sed 's/.$//'` -$ ids=`planet data filter --geom geometry.geojson --range clear_percent gt 90 | planet data search-quick SkySatCollect --limit 5 - | jq -r .id | tr '\n' , | sed 's/.$//'` -$ planet orders request -- "Clipped Scenes" --item-type SkySatCollect --bundle analytic --id $ids | planet orders create --clip geometry.geojson - +$ ids=`planet data filter --geom geometry.geojson --range clear_percent gt 90 | planet data \ +search-quick PSScene --limit 5 - | jq -r .id | tr '\n' , | sed 's/.$//'` +$ planet orders request --name "Clipped Scenes" --item-type PSScene --bundle analytic_sr_udm2 \ +--id $ids --clip geometry.geojson | planet orders create - ``` -```console -$ -``` - -```console -$ -``` - -Run a search for planetscope data on a bounding box in Iowa - - - -```json -{ - "type": "FeatureCollection", - "features": [ - { - "type": "Feature", - "properties": {}, - "geometry": { - "type": "Polygon", - "coordinates": [ - [ - [ - -93.7353, - 41.6236 - ], - [ - -92.2741, - 41.6236 - ], - [ - -92.2741, - 42.3747 - ], - [ - -93.7353, - 42.3747 - ], - [ - -93.7353, - 41.6236 - ] - ] - ] - } - } - ] -} -``` - -Run search for planetscope with cloud cover less than 20 -Run search with permission filter to just be data that can be downloaded -Run search with just good sun angles -Run search for sentinel 2 images -Run search for landsat8 images -Run search for sentinel 1 images -Run search for all data from Planet & public satellites for the last 5 grow seasons in Iowa (April 15 - October 15th) -Get it into a shapefile for use by naive GIS people -Get stats of number of images captured by satellite, show in chart -Get stats of number of cloud free images by satellite -Run search with a geojson of 7 small to moderate sized fields, try out a few filters -Run search and turn results into STAC output, to: -Understand results with stac-terminal -Add to an existing stac catalog with stactools - -planet orders list - -` --pretty or | jq` - -``` - -Order and download a scene id found from explorer - -```console -20220605_124027_64_242b -``` - -```console -$ -``` - -```console -$ -``` - -```console -$ -``` - -As surface reflectance -Order and download the results of 3-4 of the first set of searches above -Order and download the above as STAC to: -Add to an existing STAC catalog with stactools -Browse resulting catalog with STAC Browser -Visualize results in Unfolded Studio -Do this with both surface reflectance data as well as clipped, composite, harmonized data. -Run search of geojson of 7 moderate-sized fields, download clipped, cloud-free imagery of each. -Do clip & composite of them -Clip, composite & harmonize -Clip, composite and NDVI -COG’s for each of the outputs - -Run a search for planetscope data on a bounding box in Iowa -Run search for planetscope with cloud cover less than 20 -Run search with permission filter to just be data that can be downloaded -Run search with just good sun angles -Run search for sentinel 2 images -Run search for landsat8 images -Run search for sentinel 1 images -Run search for all data from Planet & public satellites for the last 5 grow seasons in Iowa (April 15 - October 15th) -Get it into a shapefile for use by naive GIS people -Get stats of number of images captured by satellite, show in chart -Get stats of number of cloud free images by satellite -Run search with a geojson of 7 small to moderate sized fields, try out a few filters -Run search and turn results into STAC output, to: -Understand results with stac-terminal -Add to an existing stac catalog with stactools +### Future workflows to support +* Get the last 5 strips for an AOI, clipped & composited to the AOI (need either 'composite by strip' or some client-side code + to sort which results are in the same strip) +* STAC output +* Cloud delivery (could do this today with editing JSON, but should wait till we have it a bit better) +* Get most recent acquired (using sort) -Download 3 months of imagery over a small field stored in a shapefile, but only download the imagery that is 90% cloud free over the field. -https://hello.planet.com/code/benjamin/cloud_free_order can help us -Set up a subscription of planetscope data over a bound box in iowa -With cloud cover less than 25% -Set up a subscription clipped to the 7 small to moderate-sized fields -And then do it with harmonization & composites -Only use those that are 90% cloud-free over the actual AOI -Backfill a subscription for the previous 6 months From 8aaea274b742a357c8f0e73df51e9cc8a5bc1870 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Mon, 27 Jun 2022 22:22:42 -0700 Subject: [PATCH 09/30] initial commit of more advanced tutorial stuff --- docs/cli-plus-tutorial.md | 119 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 docs/cli-plus-tutorial.md diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md new file mode 100644 index 000000000..1c584e31d --- /dev/null +++ b/docs/cli-plus-tutorial.md @@ -0,0 +1,119 @@ +## About + +This document shows off a range of more advanced command-line workflows, making use of a wider range +of tools in the geospatial ecosystem. Some of them can be a pain to install,like GDAL/OGR, and +several pop in and out of web tools, so these are kept out of the main tutorial section. + +## Tools used + +* **[GDAL/OGR](https://gdal.org)** - We'll mostly use OGR, the vector tooling. +Great for things like format conversion and basic simplification. +* **[Keplergl_cli](https://github.com/kylebarron/keplergl_cli#usage)** - Nice tool to call the +awesome kepler.gl library from the commandline. Useful for visualization of large amounts of +geojson. +* **[GeoJSON.io](https://geojson.io/)** - Simple tool to do editing of geojson, useful for creating +AOI's. Integrates with github, but doesn't seem to work so well. +* **[Placemark.io](https://placemark.io)** - More advanced tool from the creator of GeoJSON, very +nice for creating AOI's and piping them in. +* **[MapShaper](https://github.com/mbloch/mapshaper)** - Tool to do interactive simplification of +GeoJSON, has a nice CLI. +* **[STACTools](https://github.com/stac-utils/stactools)** - CLI for working with STAC data. There +is also a [planet plugin](https://github.com/stactools-packages/planet) that may be useful in the +future. + +## Workflows + + +Draw in placemark and run a search in area you made, visualize output in Kepler. + - filter by acquistion, show animation of time. + +```console +$ curl -s https://api.placemark.io/api/v1/map/Dse5hjAzfrLyXDU9WiFn7/feature/11b80520-f672-11ec-98c0-5b51b58d2b56 | planet data filter --geom - | planet data search-quick SkySatCollect - | kepler +``` + +Show the latest 2500 collects for the state, across assets. + - filter by provider, and instrument, gsd + - show customizing the pop-up of properties + +```console +$ curl -s https://raw.githubusercontent.com/ropensci/geojsonio/master/inst/examples/california.geojson | planet data filter --geom - | planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect,Sentinel1 --limit 2500 - | kepler +``` + +Download lots of scenes (current version of CLI may crap out before it gets there, will take like 10 minutes) + +```console +$ planet data filter | planet data search-quick SkySatCollect --limit 200000 > skysat-large.json +``` + +Put into a feature collection - often this will clean things up if other programs can't open it (but not always): + +```console +$ planet data collect skysat-large.json > skysat-large-clean.json +``` + +Turn into geopackage (or shapefile) for a spatial index, and simplify (don't need so many points to visualize) + - simplify .001 matches fidelity very close, mostly just removes points from lines. .008 drops a bit of info but generally good. .1 + messes with things a lot, but still good for visualization. + +```console +$ ogr2ogr skysat-large.gpkg skysat-large-clean.json -simplify .008 +``` + +Can open `skysat-large.gpkg` with kepler, or other tools. + +Other simplification options for large datasets: + +* Use QGIS, run 'convex hull' (Vector -> Geoprocessing -> Convex Hull). Good idea to convert to gpkg or shapefile before you open in qgis if large. + +Draw on geojson.io, copy the geojson as input to search +(pbpaste on mac. other options for windows and linux TODO: figure these out) + +```console +$ pbpaste | planet data filter --geom - | planet data search-quick SkySatCollect - +``` + +Copy output to clipboard. + - paste to geojson.io + - paste to placemark (not quite working at the moment) + +```console +$ planet data filter --string-in strip_id 5743640 | planet data search-quick PSScene - | planet collect - | pbcopy +``` + +Show the latest skysat image on github as a gist. + +```console +$ planet data filter | planet data search-quick SkySatCollect --limit 1 - | planet collect - | gh gist create -f mysearch.geojson -w +``` + +Show all ps-scenes in a strip on github gist. +(may need to reload the page, for some reason it wasn't showing up immediately after open) + +```console +$ planet data filter --string-in strip_id 5743640 | planet data search-quick PSScene - | gh gist create -f ps-search.geojson -w +``` + +TODO: get command that gets the latest strip id and uses that as input in one line. May need to update filter commands to take stdin? + +```console +$ strip-id=`planet data filter | planet data search-quick PSScene - --limit 1 | jq -r '.properties.strip_id' | sed 's/\\[tn]//g'` +$ planet data filter --string-in strip_id $stripid | planet data search-quick PSScene - +``` + + + +```console +$ +``` + +```console +$ +``` +```console +$ +``` + +```console +$ +``` + From 46503549d7dcc24a0756229db4e22e37434b84ed Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Wed, 29 Jun 2022 16:19:58 -0700 Subject: [PATCH 10/30] minor changes --- docs/cli-plus-tutorial.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md index 1c584e31d..69061a2be 100644 --- a/docs/cli-plus-tutorial.md +++ b/docs/cli-plus-tutorial.md @@ -20,6 +20,7 @@ GeoJSON, has a nice CLI. * **[STACTools](https://github.com/stac-utils/stactools)** - CLI for working with STAC data. There is also a [planet plugin](https://github.com/stactools-packages/planet) that may be useful in the future. +* **[Github commandline](https://cli.github.com/)** - Nice CLI for quickly posting things to gists. ## Workflows @@ -77,7 +78,7 @@ Copy output to clipboard. - paste to placemark (not quite working at the moment) ```console -$ planet data filter --string-in strip_id 5743640 | planet data search-quick PSScene - | planet collect - | pbcopy +$ planet data filter --string-in strip_id 5743669 | planet data search-quick PSScene - | planet collect - | pbcopy ``` Show the latest skysat image on github as a gist. From 828f19f28147dfb0b059103e3138caaa2baaf1a6 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Fri, 1 Jul 2022 08:55:36 -0700 Subject: [PATCH 11/30] added a nice way to not copy/paste order ids --- docs/cli-tutorial.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md index ba7a6a03f..064f5731d 100644 --- a/docs/cli-tutorial.md +++ b/docs/cli-tutorial.md @@ -130,6 +130,14 @@ $ planet orders wait 65df4eb0-e416-4243-a4d2-38afcf382c30 \ && planet orders download 65df4eb0-e416-4243-a4d2-38afcf382c30 ``` +You can also use a unix variable to store the order id of your most recently placed order, +and then use that for the wait and download commands: + +```console +$ orderid=`planet orders list --limit 1 | jq -r .id` +$ planet orders wait $orderid && planet orders download $orderid +``` + Create an order from a request, wait for it and download when ready ```console From fe2b23f3e6d7614ee6cb063b19cac1f95989388c Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Fri, 15 Jul 2022 11:05:38 -0700 Subject: [PATCH 12/30] more updates --- docs/cli-concepts.md | 43 +++++++++- docs/cli-plus-tutorial.md | 31 +++---- docs/cli-tutorial.md | 172 ++++++++++++++++++++++++-------------- 3 files changed, 167 insertions(+), 79 deletions(-) diff --git a/docs/cli-concepts.md b/docs/cli-concepts.md index 80b49b9dd..856587dc8 100644 --- a/docs/cli-concepts.md +++ b/docs/cli-concepts.md @@ -2,16 +2,53 @@ Planet's command-line interface is built of composable pieces that combine in powerful ways. To take full advantage of all that Planet's CLI offers there are a few concepts and tools -that are worth explaining. +that are worth explaining. ## Core Unix Concepts -These are core functions from Unix +If you are completely new to the command-line we recommend starting with a real introductory +guide like (TODO, link to a good guide). But we wanted to go over a few key concepts that +are a bit more 'advanced', as they allow you to get the most out of Planet's CLI. These are +all built into any unix command-line, including Linux and the Mac terminal. If you're on +Windows you can use [Windows Subsytem for Linux](https://docs.microsoft.com/en-us/windows/wsl/about) +or [Cygwin](https://www.cygwin.com/). + +### Piping & Redirection + +Several commands in the Planet CLI are used to create files that are used as input to +other commands. The default of these commands is to just print the output on your screen. +Seeing it on the screen can be useful for making sure it's right, but you'll most likely +want to make use of it. This is where the concept of 'redirection' in. If you use the +`>` character and then specify a file name the command-line will save its output to that file. +So if you say: + +``` + $ planet data filter --range cloud_percent lt 10 > filter.json +``` + +Then the output will be saved. This output is referred to as STDOUT, or 'standard output'. +There is much more in this vein that you can do, like use `>>` to append to an existing +file, or `<` to send what's in the file as input for a command. + +One of the most powerful concepts that we use extensively in the Planet CLI is 'piping'. +The `|` is the pipe symbol, and it's a special command that lets you pass the output from +one command to be the input for the next one. So instead of having to save to a file and +then referring to it you can just do it all in one call: + +``` +planet data filter --range cloud_percent lt 10 | planet data search-quick PSScene - +``` + +The pipe says to take + +There's a good tutorial on https://ryanstutorials.net/linuxtutorial/piping.php + -### Piping incuding reading from a pipe with - +https://www.baeldung.com/linux/dash-in-command-line-parameters + stringing several together ### Output to file diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md index 69061a2be..3e3729782 100644 --- a/docs/cli-plus-tutorial.md +++ b/docs/cli-plus-tutorial.md @@ -29,7 +29,7 @@ Draw in placemark and run a search in area you made, visualize output in Kepler. - filter by acquistion, show animation of time. ```console -$ curl -s https://api.placemark.io/api/v1/map/Dse5hjAzfrLyXDU9WiFn7/feature/11b80520-f672-11ec-98c0-5b51b58d2b56 | planet data filter --geom - | planet data search-quick SkySatCollect - | kepler +curl -s https://api.placemark.io/api/v1/map/Dse5hjAzfrLyXDU9WiFn7/feature/11b80520-f672-11ec-98c0-5b51b58d2b56 | planet data filter --geom - | planet data search-quick SkySatCollect - | kepler ``` Show the latest 2500 collects for the state, across assets. @@ -37,19 +37,19 @@ Show the latest 2500 collects for the state, across assets. - show customizing the pop-up of properties ```console -$ curl -s https://raw.githubusercontent.com/ropensci/geojsonio/master/inst/examples/california.geojson | planet data filter --geom - | planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect,Sentinel1 --limit 2500 - | kepler +curl -s https://raw.githubusercontent.com/ropensci/geojsonio/master/inst/examples/california.geojson | planet data filter --geom - | planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect,Sentinel1 --limit 2500 - | kepler ``` Download lots of scenes (current version of CLI may crap out before it gets there, will take like 10 minutes) ```console -$ planet data filter | planet data search-quick SkySatCollect --limit 200000 > skysat-large.json +planet data filter | planet data search-quick SkySatCollect --limit 200000 > skysat-large.json ``` Put into a feature collection - often this will clean things up if other programs can't open it (but not always): ```console -$ planet data collect skysat-large.json > skysat-large-clean.json +planet data collect skysat-large.json > skysat-large-clean.json ``` Turn into geopackage (or shapefile) for a spatial index, and simplify (don't need so many points to visualize) @@ -57,7 +57,7 @@ Turn into geopackage (or shapefile) for a spatial index, and simplify (don't nee messes with things a lot, but still good for visualization. ```console -$ ogr2ogr skysat-large.gpkg skysat-large-clean.json -simplify .008 +ogr2ogr skysat-large.gpkg skysat-large-clean.json -simplify .008 ``` Can open `skysat-large.gpkg` with kepler, or other tools. @@ -70,7 +70,7 @@ Draw on geojson.io, copy the geojson as input to search (pbpaste on mac. other options for windows and linux TODO: figure these out) ```console -$ pbpaste | planet data filter --geom - | planet data search-quick SkySatCollect - +pbpaste | planet data filter --geom - | planet data search-quick SkySatCollect - ``` Copy output to clipboard. @@ -78,43 +78,44 @@ Copy output to clipboard. - paste to placemark (not quite working at the moment) ```console -$ planet data filter --string-in strip_id 5743669 | planet data search-quick PSScene - | planet collect - | pbcopy +planet data filter --string-in strip_id 5743669 | planet data search-quick PSScene - | planet collect - | pbcopy ``` Show the latest skysat image on github as a gist. ```console -$ planet data filter | planet data search-quick SkySatCollect --limit 1 - | planet collect - | gh gist create -f mysearch.geojson -w +planet data filter | planet data search-quick SkySatCollect - --sort 'acquired desc' --limit 1 \ +| planet collect - | jq | gh gist create -f latest-skysat.geojson -w ``` Show all ps-scenes in a strip on github gist. (may need to reload the page, for some reason it wasn't showing up immediately after open) ```console -$ planet data filter --string-in strip_id 5743640 | planet data search-quick PSScene - | gh gist create -f ps-search.geojson -w +planet data filter --string-in strip_id 5743640 | planet data search-quick PSScene - | gh gist create -f ps-search.geojson -w ``` TODO: get command that gets the latest strip id and uses that as input in one line. May need to update filter commands to take stdin? ```console -$ strip-id=`planet data filter | planet data search-quick PSScene - --limit 1 | jq -r '.properties.strip_id' | sed 's/\\[tn]//g'` -$ planet data filter --string-in strip_id $stripid | planet data search-quick PSScene - +strip-id=`planet data filter | planet data search-quick PSScene - --limit 1 | jq -r '.properties.strip_id' | sed 's/\\[tn]//g'` +planet data filter --string-in strip_id $stripid | planet data search-quick PSScene - ``` ```console -$ + ``` ```console -$ + ``` ```console -$ + ``` ```console -$ + ``` diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md index 064f5731d..4dfbc80fe 100644 --- a/docs/cli-tutorial.md +++ b/docs/cli-tutorial.md @@ -1,105 +1,146 @@ ## About -This tutorial is designed to show off the core Planet command-line interface capabilities, without -requiring +This tutorial is designed to show off the core Planet command-line interface (CLI) capabilities. +The Planet depends on several more advanced command-line concepts, but this tutorial should let +you get a sense of what you can do, with enough examples that you should be able to adapt the +commands for what you want to do. If you're interested in deeper understanding what is going on +then check out our [CLI Concepts](cli-concepts.md) guide. ## Commands ### Auth -Initialize with user name and password +We start with the `auth` package of tools, to ensure you can get started. + +#### Initialize + +The main way to initialize the Planet CLI is to use `init`, which will prompt +you for your Planet username and password, and then store your API key. ```console -$ planet auth init +planet auth init ``` -See API key that is being used for requests +#### View API Key + +You can easily see your API key that is being used for requests: ```console -$ planet auth value +planet auth value +``` + +This is a very convenient way to quickly grab your API key. + +#### Set environment variable + +You can also set an 'environment variable' in your terminal: + +``` +export PL_API_KEY= ``` -*NOTE: Right now you must init before you get help, until https://github.com/planetlabs/planet-client-python/issues/463 is resolved* +This currently will override your key set with 'init'. Be warned that if you have this +set `planet auth value` won't report. + ### Get Help -Most commands are pretty self-documenting, can use `--help` after any command to get usage info. And many will give you help -if you don't form a complete command. +Most commands are pretty self-documenting, and you can use `--help` after any command to get usage info. +Many will show the help text if you don't form a complete command. ```console -$ planet --help -$ planet -$ planet orders -$ planet orders list --help +planet --help +planet +planet orders +planet orders list --help ``` ### Orders -See recent orders you've made +The orders command enables interaction with the [Orders API](https://developers.planet.com/apis/orders/), +that lets you activate and download Planet products in bulk, and apply various 'tools' to your processes. + +#### See Recent Orders + +You can use the `list` command to show your recent orders: ```console -$ planet orders list +planet orders list ``` -Print them more nicely -(both are options, jq is a bit more advanced, but prints a bit nicer and is more powerful) +Note you can also get a nice list online as well, at https://www.planet.com/account/#/orders + +#### Recent Orders with Formatting + +You can also print the list of orders more nicely: ```console -$ planet orders list | jq -$ planet orders list --pretty +planet orders list | jq +planet orders list --pretty ``` -Get the number of your recent orders. (-s collects objects into a single array) +These are two different ways to do it. The `--pretty` tag is built into planet commands, while +the `jq` command will format any JSON more nicely, and can be used for more advanced processing. + +#### Number of recent orders + +You can use jq to process the output for more insight, like +get a count of how many recent orders you've done. ```console -$ planet orders list | jq -s length +planet orders list | jq -s length ``` +This uses `-s` to collect the output into a single array, and `length` then tells the +length of the array. + + + Get more info on a single order ```console -$ planet orders get 782b414e-4e34-4f31-86f4-5b757bd062d7 +planet orders get 782b414e-4e34-4f31-86f4-5b757bd062d7 ``` Download the order ```console -$ planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 +planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 ``` Download it to a different directory (relative to where the command is run) ```console -$ mkdir psscene -$ planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --directory psscene +mkdir psscene +planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --directory psscene ``` Download to a directory specified absolutely (in this case to my desktop) ```console -$ planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --directory /Users/cholmes/Desktop/ +planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --directory /Users/cholmes/Desktop/ ``` Verify the checksum to make sure the file I got wasn't corrupted along the way (during download, etc) ```console -$ planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --checksum MD5 +planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --checksum MD5 ``` Create a new order request from a scene ID found in Explorer (Note that \ just tells the console to treat the next line as the same one - used here to be a bit easier to read) ```console -$ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ +planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ --item-type PSScene ``` Save to a file in the directory where the command was run ```console -$ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ +planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ --item-type PSScene > request-1.json ``` @@ -107,26 +148,26 @@ Make a new order with that request (should be the same as [this one](https://gist.githubusercontent.com/cholmes/892f851d5c55f7cf93e210595750ecfe/raw/4c255dcdf0973e0f72d43d82c776d8251fb7545e/request-1.json)) ```console -$ planet orders create request-1.json +planet orders create request-1.json ``` Create the request and make a new order with it all in one call ```console -$ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ +planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ --item-type PSScene | planet orders create ``` Download the order. Note this only works when the order is ready: ```console -$ planet orders download 65df4eb0-e416-4243-a4d2-38afcf382c30 +planet orders download 65df4eb0-e416-4243-a4d2-38afcf382c30 ``` Wait until the order is ready for download, then order it: ```console -$ planet orders wait 65df4eb0-e416-4243-a4d2-38afcf382c30 \ +planet orders wait 65df4eb0-e416-4243-a4d2-38afcf382c30 \ && planet orders download 65df4eb0-e416-4243-a4d2-38afcf382c30 ``` @@ -134,14 +175,14 @@ You can also use a unix variable to store the order id of your most recently pla and then use that for the wait and download commands: ```console -$ orderid=`planet orders list --limit 1 | jq -r .id` -$ planet orders wait $orderid && planet orders download $orderid +orderid=`planet orders list --limit 1 | jq -r .id` +planet orders wait $orderid && planet orders download $orderid ``` Create an order from a request, wait for it and download when ready ```console -$ id=`planet orders create request-1.json | jq -r '.id'` && planet orders wait $id && planet orders download $id +id=`planet orders create request-1.json | jq -r '.id'` && planet orders wait $id && planet orders download $id ``` Create an order clipped to a geometry @@ -150,14 +191,14 @@ Create an order clipped to a geometry from github, or [preview](https://gist.github.com/cholmes/c7736ac5241d77605524d01ed2dc57a1)) ```console -$ planet orders request --item-type PSScene --clip geometry.geojson --name clipped-geom \ +planet orders request --item-type PSScene --clip geometry.geojson --name clipped-geom \ --bundle analytic_sr_udm2 --id 20220605_124027_64_242b | planet orders create ``` Order Two Scenes in one order ```console -$ planet orders request --item-type PSScene --name "Two Scenes" \ +planet orders request --item-type PSScene --name "Two Scenes" \ --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b | planet orders create ``` @@ -177,7 +218,7 @@ Use this, save as tools-composite.json Then create command and order ```console -$ planet orders request --item-type PSScene --name "Two Scenes Composited" \ +planet orders request --item-type PSScene --name "Two Scenes Composited" \ --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b --tools tools-composite.json | planet orders create ``` @@ -201,7 +242,7 @@ Get the output as a cloud-optimized GeoTIFF: The following command just shows the output, you can pipe into orders create if you'd like: ```console -$ planet orders request --item-type PSScene --name "Two Scenes Composited" \ +planet orders request --item-type PSScene --name "Two Scenes Composited" \ --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b --tools tools-cog.json ``` @@ -209,33 +250,43 @@ To clip and composite you need to specify the clip in the tools (instead of `--c Can get it from [this gist](https://gist.github.com/cholmes/378d050a263ae433ddbbb91c3439994b) ```console -$ planet orders request --item-type PSScene --name "Two Scenes Clipped and Composited" \ +planet orders request --item-type PSScene --name "Two Scenes Clipped and Composited" \ --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b --tools tools-clip-composite.json ``` Or can even stream that in: ```console -$ url -s https://gist.githubusercontent.com/cholmes/378d050a263ae433ddbbb91c3439994b/raw/ebcaa54cacdc6f696a0506705785f8ff8dae9af1/ \ +curl -s https://gist.githubusercontent.com/cholmes/378d050a263ae433ddbbb91c3439994b/raw/ebcaa54cacdc6f696a0506705785f8ff8dae9af1/ \ tools-clip-composite.json | planet orders request --item-type PSScene --name "Streaming Clip & Composite" \ --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b --tools - | planet orders create ``` TODO: STAC Output +One useful thing to note is that the order JSON that reports status and location is a valid Orders API request. +It reports all the parameters that were used to make the previous order, but you can also use it directly as a +request. So the following call is a quick way to exactly redo a previous order request: + +```console +planet orders get | planet orders create - +``` + +Realistically you'd more likely want to + ### Data Run a search for all the latest planetscope scenes that you have download access to: ```console -$ planet data filter > filter.json -$ planet data search-quick PSScene filter.json +planet data filter > filter.json +planet data search-quick PSScene filter.json ``` Or using piping (recommended) ```console -$ planet data filter | planet data search-quick PSScene - +planet data filter | planet data search-quick PSScene - ``` Run a search for all the latest PS Scenes in a bounding box in Iowa: @@ -282,69 +333,68 @@ Save this as geometry.json ``` ```console -$ planet data filter --geom geometry.json | planet data search-quick PSScene - +planet data filter --geom geometry.json | planet data search-quick PSScene - ``` And turn into GeoJSON (many gis programs do better with 'real' geojson, the default output is newline delimited geojson) ```console -$ planet data filter --geom geometry.json | planet data search-quick PSScene - | planet collect +planet data filter --geom geometry.json | planet data search-quick PSScene - | planet collect - ``` Search just for 8-band assets ```console -$ planet data filter --asset ortho_analytic_8b_sr | planet data search-quick PSScene - +planet data filter --asset ortho_analytic_8b_sr | planet data search-quick PSScene - ``` Search for 8-band assets that also have a UDM ```console -$ planet data filter --asset ortho_analytic_8b_sr --asset udm2 | planet data search-quick PSScene - +planet data filter --asset ortho_analytic_8b_sr --asset udm2 | planet data search-quick PSScene - ``` Search for any 8 band data, not just ones that you have download access to (default is only those you have access to) ```console -$ planet data filter --permission false --asset ortho_analytic_8b_sr | planet data search-quick PSScene - +planet data filter --permission false --asset ortho_analytic_8b_sr | planet data search-quick PSScene - ``` Make a filter for data acquired in July 2021 ```console -$ planet data filter --date-range acquired gte 2021-07-01 --date-range acquired lt 2021-08-01 +planet data filter --date-range acquired gte 2021-07-01 --date-range acquired lt 2021-08-01 ``` Make a filter for data with clear pixels greater than 90% ```console -$ planet data filter --range clear_percent gt 90 +planet data filter --range clear_percent gt 90 ``` Make a filter for all data in a single strip: ```console -$ planet data filter --string-in strip_id 5743640 +planet data filter --string-in strip_id 5743640 ``` Search for all landsat 8, sentinel 2 and planetscope images in an area of interest ```console -$ planet data filter --geom geometry.json | planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G +planet data filter --geom geometry.json | planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G - ``` Get the id of the most recent skysat image taken (that you have download access to) ```console -$ planet data filter | planet data search-quick SkySatCollect --limit 1 - | jq -r .id +planet data filter | planet data search-quick SkySatCollect --limit 1 - | jq -r .id ``` -Order the most recent skysat image published -(TODO: we need 'sort' to be able to show most recent acquired) +Order the most recent skysat image published. ```console -$ planet orders request --name "SkySat Latest" --item-type SkySatCollect --bundle analytic \ ---id `planet data filter | planet data search-quick SkySatCollect --limit 1 - | jq -r .id` \ +planet orders request --name "SkySat Latest" --item-type SkySatCollect --bundle analytic \ +--id `planet data filter | planet data search-quick SkySatCollect --sort 'acquired desc' --limit 1 - | jq -r .id` \ | planet orders create ``` @@ -355,9 +405,9 @@ Get the 5 latest cloud free images in an area and create an order that clips to ```console -$ ids=`planet data filter --geom geometry.geojson --range clear_percent gt 90 | planet data \ +ids=`planet data filter --geom geometry.geojson --range clear_percent gt 90 | planet data \ search-quick PSScene --limit 5 - | jq -r .id | tr '\n' , | sed 's/.$//'` -$ planet orders request --name "Clipped Scenes" --item-type PSScene --bundle analytic_sr_udm2 \ +planet orders request --name "Clipped Scenes" --item-type PSScene --bundle analytic_sr_udm2 \ --id $ids --clip geometry.geojson | planet orders create - ``` From 18e49ca3c177e9e12b70a9569b813f609b04f38a Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Fri, 15 Jul 2022 22:37:10 -0700 Subject: [PATCH 13/30] improvements --- docs/cli-tutorial.md | 198 +++++++++++++++++++++++++++++++++++-------- 1 file changed, 161 insertions(+), 37 deletions(-) diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md index 4dfbc80fe..1357b4e2b 100644 --- a/docs/cli-tutorial.md +++ b/docs/cli-tutorial.md @@ -95,82 +95,149 @@ planet orders list | jq -s length This uses `-s` to collect the output into a single array, and `length` then tells the length of the array. +#### Info on an Order - -Get more info on a single order +For a bit more information about an order, including the location of any downloads, use +the `get` command, using the order id. ```console planet orders get 782b414e-4e34-4f31-86f4-5b757bd062d7 ``` -Download the order - -```console -planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 -``` - -Download it to a different directory (relative to where the command is run) +#### Create an Order Request +To create an order you need a name, a [bundle](https://developers.planet.com/apis/orders/product-bundles-reference/), + one or more id's, and an [item type](https://developers.planet.com/docs/apis/data/items-assets/#item-types): ```console -mkdir psscene -planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --directory psscene +planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ + --item-type PSScene ``` -Download to a directory specified absolutely (in this case to my desktop) +This should output the JSON needed to create an order: -```console -planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --directory /Users/cholmes/Desktop/ +```json +{"name": "My First Order", "products": [{"item_ids": ["20220605_124027_64_242b"], "item_type": "PSScene", "product_bundle": "analytic_sr_udm2"}]} ``` -Verify the checksum to make sure the file I got wasn't corrupted along the way (during download, etc) +Note that `\` just tells the command-line to treat the next line as the same one. It's used here so it's +easier to read, but you can still copy and paste the full line into your command-line and it should work. -```console -planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --checksum MD5 -``` - -Create a new order request from a scene ID found in Explorer -(Note that \ just tells the console to treat the next line as the same one - used here to be a bit easier to read) +You can also use `jq` here to make it a bit more readable: ```console planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ - --item-type PSScene + --item-type PSScene | jq ``` -Save to a file in the directory where the command was run +```json +{ + "name": "My First Order", + "products": [ + { + "item_ids": [ + "20220605_124027_64_242b" + ], + "item_type": "PSScene", + "product_bundle": "analytic_sr_udm2" + } + ] +} +``` + +#### Save an Order Request + +The above command just prints out the necessary JSON to create an order. To actually use it you can +save the output into a file: ```console -planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ +planet orders request --name "My first Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ --item-type PSScene > request-1.json ``` -Make a new order with that request (should be the same as -[this one](https://gist.githubusercontent.com/cholmes/892f851d5c55f7cf93e210595750ecfe/raw/4c255dcdf0973e0f72d43d82c776d8251fb7545e/request-1.json)) +This saves the above JSON in a file called `request-1.json` + +#### Create an Order + +From there you can create the order with the request you just saved: ```console planet orders create request-1.json ``` -Create the request and make a new order with it all in one call +The output of that command is the JSON returned from the server, that reports the status: + +```json +{ + "_links": { + "_self": "https://api.planet.com/compute/ops/orders/v2/3b1f250e-72a0-41bb-94ea-8109b6b34e44" + }, + "created_on": "2022-07-16T05:04:36.998Z", + "error_hints": [], + "id": "3b1f250e-72a0-41bb-94ea-8109b6b34e44", + "last_message": "Preparing order", + "last_modified": "2022-07-16T05:04:36.998Z", + "name": "My Second Order", + "products": [ + { + "item_ids": [ + "20220605_124027_64_242b", + "20220605_124025_34_242b" + ], + "item_type": "PSScene", + "product_bundle": "analytic_sr_udm2" + } + ], + "state": "queued" +} +``` + +Note the default output will be a bit more 'flat' - if you'd like the above formatting in your +command-line just use `jq` as above: `planet orders create request-1.json` (but remember +if you run that command again it will create a second order). + +#### Create Request and Order in One Call. + +Using a unix command called a 'pipe', which looks like `|`, you can skip the step of saving to disk, +passing the output of the `orders request` command directly to be the input of the `orders create` +command: ```console -planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b \ +planet orders request --name "My Second Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b \ --item-type PSScene | planet orders create ``` -Download the order. Note this only works when the order is ready: +The Planet CLI is designed to work well with piping, as it aims at small commands that can be +combined in powerful ways, so you'll see it used in a number of the examples. + +#### Download an order + +To download all files in an order you use the `download` command: ```console planet orders download 65df4eb0-e416-4243-a4d2-38afcf382c30 ``` +Note this only works when the order is ready for download. To do that you can +keep running `orders get` until the `state` is `success`. Or for a better +way see the next example. -Wait until the order is ready for download, then order it: +#### Wait then download an order + +The `wait` command is a small, dedicated command that polls the server to +see if an order is ready for downloading, showing the status. It's not +so useful by itself, but can be combined with the `download` command to +only start the download once the order is ready: ```console planet orders wait 65df4eb0-e416-4243-a4d2-38afcf382c30 \ && planet orders download 65df4eb0-e416-4243-a4d2-38afcf382c30 ``` +This uses the logical AND operator (`&&`) to say "don't run the second command +until the first is done". + +#### Save order ID + You can also use a unix variable to store the order id of your most recently placed order, and then use that for the wait and download commands: @@ -179,16 +246,69 @@ orderid=`planet orders list --limit 1 | jq -r .id` planet orders wait $orderid && planet orders download $orderid ``` -Create an order from a request, wait for it and download when ready +This can be nicer than copying and pasting it in. + +You could also save the id right when you place the order: + +```console +orderid=`planet orders create request-1.json | jq -r .id` +``` + +To check the current value of `orderid` just run `echo $orderid`. + +#### Create an order and download when ready + +You can then combine these all into one call, to create the order and +download it when it's available: + +```console +id=`planet orders create request-1.json | jq -r '.id'` && \ +planet orders wait $id && planet orders download $id +``` + +#### Download to a different directory + +You can use the `--directory` flag to save the output to a specific directory. This +call saves it to a directory called `psscene`, at whatever location you are at +currently: + +```console +mkdir psscene +planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --directory psscene +``` + +You can also specify absolute directories (int his case to my desktop): + +```console +planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --directory /Users/cholmes/Desktop/ +``` + +#### Verify checksum + +The `--checksum` command will do an extra step to make sure the file you got +wasn't corrupted along the way (during download, etc). It checks that the bytes +downloaded are the same as the ones on the server. By default it doesn't show +anything if the checksums match. ```console -id=`planet orders create request-1.json | jq -r '.id'` && planet orders wait $id && planet orders download $id +planet orders download 782b414e-4e34-4f31-86f4-5b757bd062d7 --checksum MD5 ``` -Create an order clipped to a geometry +This command isn't often necessary in single download commands, but is quite +useful if you are downloading thousands of files with a script, as the likelihood +of at least one being corrupted in creases + +#### Create an order clipped to a geometry + +Now we'll dive into the variety of ways to customize your order. These can all be +combined with all the commands listed above. + +We'll work with a geojson saved online. You should download the +[raw json](https://gist.githubusercontent.com/cholmes/c7736ac5241d77605524d01ed2dc57a1/raw/7d24e02ba894e64c4c737c253a0cce4cac54167c/geometry.geojson) +from github. If you want to see what it looks like check the +[preview](https://gist.github.com/cholmes/c7736ac5241d77605524d01ed2dc57a1)). -(download [raw json](https://gist.githubusercontent.com/cholmes/c7736ac5241d77605524d01ed2dc57a1/raw/7d24e02ba894e64c4c737c253a0cce4cac54167c/geometry.geojson) -from github, or [preview](https://gist.github.com/cholmes/c7736ac5241d77605524d01ed2dc57a1)) +Use that saved geometry to clip the scene. ```console planet orders request --item-type PSScene --clip geometry.geojson --name clipped-geom \ @@ -272,7 +392,9 @@ request. So the following call is a quick way to exactly redo a previous order r planet orders get | planet orders create - ``` -Realistically you'd more likely want to +Realistically you'd more likely want to get a previous order and then change it in some way (new id's, different +tools, etc.). You can remove the 'extra' JSON fields that report on status if you'd like, but the Orders +API will just ignore them if they are included in a request. ### Data @@ -419,5 +541,7 @@ planet orders request --name "Clipped Scenes" --item-type PSScene --bundle analy to sort which results are in the same strip) * STAC output * Cloud delivery (could do this today with editing JSON, but should wait till we have it a bit better) -* Get most recent acquired (using sort) +* Use jq to show a list of order id's, names and created time, to more easily grab the id. +* get the id of the most recently created order using jq +* use jq to get the id of the an order by it's name From eb63965c4445385ccd1f964ab695be2b3d6f170d Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Sat, 16 Jul 2022 19:56:11 -0700 Subject: [PATCH 14/30] finishing off the first pass --- docs/cli-plus-tutorial.md | 47 +++++-- docs/cli-tutorial.md | 281 ++++++++++++++++++++++++++++++++------ 2 files changed, 274 insertions(+), 54 deletions(-) diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md index 3e3729782..2b8c49939 100644 --- a/docs/cli-plus-tutorial.md +++ b/docs/cli-plus-tutorial.md @@ -24,12 +24,35 @@ future. ## Workflows +### Geometry Inputs -Draw in placemark and run a search in area you made, visualize output in Kepler. - - filter by acquistion, show animation of time. +While the command-line can often be quicker than using a UI, one place that can be slower is +getting the geometry input for searching or clipping. Hand-editing GeoJSON is a huge pain, so most +people will open up a desktop tool like QGIS or ArcGIS Pro and save the file. But there are a few +tools that can get you back into the CLI workflow more quickly. + +Draw on geojson.io, copy the geojson as input to search +(pbpaste on mac. other options for windows and linux TODO: figure these out) ```console -curl -s https://api.placemark.io/api/v1/map/Dse5hjAzfrLyXDU9WiFn7/feature/11b80520-f672-11ec-98c0-5b51b58d2b56 | planet data filter --geom - | planet data search-quick SkySatCollect - | kepler +pbpaste | planet data filter --geom - | planet data search-quick SkySatCollect - +``` + +placemark + +```console +curl -s https://api.placemark.io/api/v1/map/Dse5hjAzfrLyXDU9WiFn7/feature/11b80520-f672-11ec-98c0-5b51b58d2b56 | planet data filter --geom - | planet data search-quick SkySatCollect - +``` + +### Geometry Visualization + + +Copy output to clipboard. + - paste to geojson.io + - paste to placemark (not quite working at the moment) + +```console +planet data filter --string-in strip_id 5743669 | planet data search-quick PSScene - | planet collect - | pbcopy ``` Show the latest 2500 collects for the state, across assets. @@ -40,6 +63,17 @@ Show the latest 2500 collects for the state, across assets. curl -s https://raw.githubusercontent.com/ropensci/geojsonio/master/inst/examples/california.geojson | planet data filter --geom - | planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect,Sentinel1 --limit 2500 - | kepler ``` +Draw in placemark and run a search in area you made, visualize output in Kepler. + - filter by acquistion, show animation of time. + +```console +curl -s https://api.placemark.io/api/v1/map/Dse5hjAzfrLyXDU9WiFn7/feature/11b80520-f672-11ec-98c0-5b51b58d2b56 | planet data filter --geom - | planet data search-quick SkySatCollect - | kepler +``` + + + +#### Large Dataset Visualization + Download lots of scenes (current version of CLI may crap out before it gets there, will take like 10 minutes) ```console @@ -73,13 +107,6 @@ Draw on geojson.io, copy the geojson as input to search pbpaste | planet data filter --geom - | planet data search-quick SkySatCollect - ``` -Copy output to clipboard. - - paste to geojson.io - - paste to placemark (not quite working at the moment) - -```console -planet data filter --string-in strip_id 5743669 | planet data search-quick PSScene - | planet collect - | pbcopy -``` Show the latest skysat image on github as a gist. diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md index 1357b4e2b..ab642378e 100644 --- a/docs/cli-tutorial.md +++ b/docs/cli-tutorial.md @@ -69,6 +69,24 @@ You can use the `list` command to show your recent orders: planet orders list ``` +If you've not placed any orders with Explorer, the CLI or the API directly, then the results +of this call will be blank, so you may want to try out some of the create order commands below. + +Sometimes that list gets too long, so you can put a limit on how many are returned: + +```console +planet orders list --limit 5 +``` + +You can also filter orders by their `state`, which can be useful to just see orders in +progress: + +```console +planet orders --state running +``` + +The other options are queued, failed, success, partial and cancelled. + Note you can also get a nice list online as well, at https://www.planet.com/account/#/orders #### Recent Orders with Formatting @@ -83,6 +101,16 @@ planet orders list --pretty These are two different ways to do it. The `--pretty` tag is built into planet commands, while the `jq` command will format any JSON more nicely, and can be used for more advanced processing. +This gives you a nicely formatted JSON. You can use `jq` to just show select information, which +can be useful for getting a quick sense of the state of the state of things and pull out +id's for other operations: + +```console +planet orders list | jq -rs '.[] | "\(.id) \(.created_on) \(.state) \(.name)"' +``` + +You can customize which fields you want to show. + #### Number of recent orders You can use jq to process the output for more insight, like @@ -114,7 +142,10 @@ planet orders request --name "My First Order" --bundle analytic_sr_udm2 --id 202 --item-type PSScene ``` -This should output the JSON needed to create an order: +You should replace the `id` with a scene you have access to, using [Explorer](https://planet.com/explorer) +is probably the easiest way to do so. + +Running the above command should output the JSON needed to create an order: ```json {"name": "My First Order", "products": [{"item_ids": ["20220605_124027_64_242b"], "item_type": "PSScene", "product_bundle": "analytic_sr_udm2"}]} @@ -196,15 +227,15 @@ Note the default output will be a bit more 'flat' - if you'd like the above form command-line just use `jq` as above: `planet orders create request-1.json` (but remember if you run that command again it will create a second order). -#### Create Request and Order in One Call. +#### Create Request and Order in One Call Using a unix command called a 'pipe', which looks like `|`, you can skip the step of saving to disk, passing the output of the `orders request` command directly to be the input of the `orders create` command: ```console -planet orders request --name "My Second Order" --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b \ - --item-type PSScene | planet orders create +planet orders request --name "My Second Order" --bundle analytic_sr_udm2 \ +--id 20220605_124027_64_242b,20220605_124025_34_242b --item-type PSScene | planet orders create ``` The Planet CLI is designed to work well with piping, as it aims at small commands that can be @@ -308,23 +339,25 @@ We'll work with a geojson saved online. You should download the from github. If you want to see what it looks like check the [preview](https://gist.github.com/cholmes/c7736ac5241d77605524d01ed2dc57a1)). -Use that saved geometry to clip the scene. +You can move that geometry to your current directory and use the following command, or +tweak the geometry.geojson to refer to where you downloaded it. ```console planet orders request --item-type PSScene --clip geometry.geojson --name clipped-geom \ --bundle analytic_sr_udm2 --id 20220605_124027_64_242b | planet orders create ``` -Order Two Scenes in one order +#### Two Scenes, composited + +Ordering two scenes is easy, just add another id: ```console planet orders request --item-type PSScene --name "Two Scenes" \ --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b | planet orders create ``` -Composite them together - -Use this, save as tools-composite.json +And then you can composite them together, using the 'tools' json. You can +use this, just save it into a file called tools-composite.json. ```json [ @@ -335,15 +368,22 @@ Use this, save as tools-composite.json ] ``` -Then create command and order +Once you've got it saved you call the `--tools` flag to refer to the JSON file, and you +can pipe that to `orders create`. ```console -planet orders request --item-type PSScene --name "Two Scenes Composited" \ - --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b --tools tools-composite.json | planet orders create +planet orders request --item-type PSScene --name "Two Scenes Composited" --bundle analytic_sr_udm2 \ +--id 20220605_124027_64_242b,20220605_124025_34_242b --tools tools-composite.json | planet orders create ``` -Get the output as a cloud-optimized GeoTIFF: -(save as tools-cog.json) +(We are working on adding `tools` convenience methods, so composite and other tools work like +`--clip`. You can follow issue [#601](https://github.com/planetlabs/planet-client-python/issues/601): +comment there if you'd like to see it prioritized) + +#### Output as COG + +If you'd like the above order as a COG you Get the output as a cloud-optimized GeoTIFF: + ```json [ @@ -366,15 +406,18 @@ planet orders request --item-type PSScene --name "Two Scenes Composited" \ --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b --tools tools-cog.json ``` -To clip and composite you need to specify the clip in the tools (instead of `--clip`). -Can get it from [this gist](https://gist.github.com/cholmes/378d050a263ae433ddbbb91c3439994b) +#### Clip & Composite + +To clip and composite you need to specify the clip in the tools (instead of `--clip`), as you can +not use `--clip` and `--tools` in the same call. You can see the full JSON for the tools in +[this gist](https://gist.github.com/cholmes/378d050a263ae433ddbbb91c3439994b). ```console planet orders request --item-type PSScene --name "Two Scenes Clipped and Composited" \ --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b --tools tools-clip-composite.json ``` -Or can even stream that in: +One cool little trick is that you can even stream in the JSON directly with `curl`, piping it into the request: ```console curl -s https://gist.githubusercontent.com/cholmes/378d050a263ae433ddbbb91c3439994b/raw/ebcaa54cacdc6f696a0506705785f8ff8dae9af1/ \ @@ -382,7 +425,7 @@ tools-clip-composite.json | planet orders request --item-type PSScene --name "St --bundle analytic_sr_udm2 --id 20220605_124027_64_242b,20220605_124025_34_242b --tools - | planet orders create ``` -TODO: STAC Output +#### Using Orders output as input One useful thing to note is that the order JSON that reports status and location is a valid Orders API request. It reports all the parameters that were used to make the previous order, but you can also use it directly as a @@ -396,24 +439,69 @@ Realistically you'd more likely want to get a previous order and then change it tools, etc.). You can remove the 'extra' JSON fields that report on status if you'd like, but the Orders API will just ignore them if they are included in a request. +There is planned functionality to make it easy to 'update' an existing order, so you can easily grab a past order +and use the CLI to customize it. + ### Data -Run a search for all the latest planetscope scenes that you have download access to: +Next up is the [Data API](https://developers.planet.com/docs/apis/data/), which lets you search Planet's catalog +(including select public datasets like Sentinel 1 & 2, Landsat & MODIS). + +#### Search latest Scenes + +One of the commands you'll use most frequently is `planet data filter`, which works similar to +`planet orders request` in that it is a convenience method to create the JSON you need to run +other commands. You can run it with no arguments to see how it works: + +```console +planet data filter +``` + +The output says that by default it will only show you imagery that you have permission to actually download, +and that is of `standard` quality category. This provides some reasonable defaults, but they can be changed +with flags, for example if you wanted to search all of Planet's catalog, or to look for `test` data that +is published by doesn't meet the standard quality level. + +There are a lot of options for what you can do with the command, and we recommend running +`planet data filter --help` often to get a reference of how the commands work. And we'll try to +give lots of examples below as well. + +So you can run the filter command and save it, and then use that file with the `search-quick` +command: ```console planet data filter > filter.json planet data search-quick PSScene filter.json ``` -Or using piping (recommended) +Or the recommended route is to use a pipe (`|`), as mentioned above [above](#create-request-and-order-in-one-call): ```console planet data filter | planet data search-quick PSScene - ``` -Run a search for all the latest PS Scenes in a bounding box in Iowa: +Note the dash (`-`), which explicitly tells the CLI to use the output from the call that is piped into it. + +#### Search on Item Type + +These first searches were done on the [PSScene](https://developers.planet.com/docs/data/psscene/) 'item type', but you +can use any [Item Type](https://developers.planet.com/docs/apis/data/items-assets/#item-types) that Planet offers in +its catalog. The item type is the first argument of the `search-quick` command, followed by the 'filter'. Note that +you can specify any number of item types here: + +```console +planet data filter | planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect, - +``` + +This will search for all the most recent images captured by PlanetScope, SkySat, Sentinel 2 and Landsat 8 satellites. +Note that you'll likely mostly see PlanetScope results, as they generate far more individual images than the others. +The filter you specify will apply to all item types, but not all filters work against all satellites, so you may +inadvertently filter some out if you do some specific properties. + +#### Run a search on a bounding box -Save this as geometry.json +Most searches you'll likely want to run on a geometry. To try this out you can use the following bounding box +of Iowa. You can copy it and save as a file called `geometry.json` ```json { @@ -454,65 +542,170 @@ Save this as geometry.json } ``` +And then run it with this command: + ```console planet data filter --geom geometry.json | planet data search-quick PSScene - ``` -And turn into GeoJSON (many gis programs do better with 'real' geojson, the default output is newline delimited geojson) +Note that by default all searches with the command-line return 100 results, but you can easily increase that with +the `--limit` flag: + +```console +planet data filter --geom geometry.json | planet data search-quick --limit 500 PSScene - +``` + +Creating geometries for search can be annoying in a command-line workflow, but there are some ideas in the +[Advanced CLI Tutorial](cli-plus-tutorial.md#geometry-inputs). + +#### Output as valid GeoJSON + +By default the output of Planet's Data API is [newline-delimited GeoJSON](https://stevage.github.io/ndgeojson/), which +is much better for streaming. While more and more programs will understand the format, the CLI also provides +the `planet collect` method to transform the output from the Data API to valid GeoJSON. You just pipe the end +output to it: ```console planet data filter --geom geometry.json | planet data search-quick PSScene - | planet collect - ``` -Search just for 8-band assets +If you want to visualize this you can save it as a file: ```console -planet data filter --asset ortho_analytic_8b_sr | planet data search-quick PSScene - +planet data filter --geom geometry.json | planet data search-quick PSScene - | planet collect - > planet-search.geojson ``` -Search for 8-band assets that also have a UDM +This you can then open with your favorite GIS program, or see this +[geometry visualization](cli-plus-tutorial.md#geometry-inputs) section for some ideas that flow a bit better with +the command-line. + +#### Date Filter + +Some of the most common filtering is by date. You could get all imagery acquired before August 2021: ```console -planet data filter --asset ortho_analytic_8b_sr --asset udm2 | planet data search-quick PSScene - +planet data filter --date-range acquired lt 2021-08-01 | planet data search-quick PSScene - ``` -Search for any 8 band data, not just ones that you have download access to (default is only those you have access to) +The 'operator' in this case is 'less than' (`lt`). The options are: + * `gt` - greater than + * `gte` - greater than or equal to + * `lt` - less than + * `lte` - less than or equal to + +You must specify which date field you want to use, either `acquired` or `published`. + +You can use the flags multiple times, and they are logically 'AND'-ed together, so you can +do a search for all images in July of 2021: ```console -planet data filter --permission false --asset ortho_analytic_8b_sr | planet data search-quick PSScene - +planet data filter --date-range acquired gte 2021-07-01 --date-range acquired lt 2021-08-01 | \ +planet data search-quick PSScene - ``` -Make a filter for data acquired in July 2021 +The date input understands [RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) and +[ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) formats. You can specify just the date, +or include the time, to search for all Planetscope images acquired within 5 seconds at a time +on July 1st 2021: ```console -planet data filter --date-range acquired gte 2021-07-01 --date-range acquired lt 2021-08-01 +planet data filter --date-range acquired gte 2021-07-01:06:20:10 --date-range acquired lt 2021-07-01:06:20:15 | \ +planet data search-quick PSScene - ``` -Make a filter for data with clear pixels greater than 90% +#### Range Filter + +The range filter uses the same operators as the date filter, but works against any numerical property. The most useful +of these tend to be ones about cloudy pixels. For example you can search for data with clear pixels greater than 90%: ```console planet data filter --range clear_percent gt 90 ``` -Make a filter for all data in a single strip: +#### String-In Filter + +For properties that are strings you can use the `string-in` filter. For example search for all planetscope imagery +with PS2 instrument: ```console -planet data filter --string-in strip_id 5743640 +planet data filter --string-in instrument PS2 | planet data search-quick PSScene - ``` -Search for all landsat 8, sentinel 2 and planetscope images in an area of interest +You can specify multiple strings to match, with a comma: ```console -planet data filter --geom geometry.json | planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G - +planet data filter --string-in instrument PS2,PSB.SD | planet data search-quick PSScene - ``` -Get the id of the most recent skysat image taken (that you have download access to) +Another example is to select all data in a single strip: ```console -planet data filter | planet data search-quick SkySatCollect --limit 1 - | jq -r .id +planet data filter --string-in strip_id 5743640 | planet data search-quick PSScene - ``` -Order the most recent skysat image published. +Note that in all these commands we are piping the results into the search. If you don't include the pipe then you'll +get the filter output, which can be interested to inspect to see exactly what is sent to the server. + +#### Filter by asset + +You can limit your search to only data with a particular asset, for example search just for 8-band analytic assets: + +```console +planet data filter --asset ortho_analytic_8b_sr | planet data search-quick PSScene - +``` + +Or 8-band assets that also have a UDM. + +```console +planet data filter --asset ortho_analytic_8b_sr --asset udm2 | planet data search-quick PSScene - +``` + +You can find the list of available assets in each Item Type Page, like +[available assets](https://developers.planet.com/docs/data/psscene/#available-asset-types) for PSScene. You can see +[a table of all Item Types](https://developers.planet.com/docs/data/psscene/#available-asset-types), which links to +the page for each with their list of asset types. + +Note that the asset filter doesn't perform any validation, so if your searches aren't returning anything check to make +sure you got the asset right, and it's valid for the item-types you're searching. + +#### Permission Filter + +The 'permission filter' is set to true by default, since most people want to search only for data they have access to +and are able to download. But if you'd like to just get search Planet's catalog and get a sense of what is out there +you can set the permission filter to false: + +```console +planet data filter --permission false --asset ortho_analytic_8b_sr | planet data search-quick PSScene - +``` + +#### Sort + +You can also specify the sorting with your searches. The default sort is ordered by the most recent published +images. But you can also sort by `acquired`, which is often more useful. You can sort in ascending or +descending order. The options are are: + + * 'acquired asc' + * 'acquired desc' + * 'published asc' + * 'published desc' + +The lets you do things like get the id of the most recent skysat image taken (that you have download access to): + +```console +planet data filter | planet data search-quick SkySatCollect --sort 'acquired desc' --limit 1 - +``` + +And you can also just get the ID, using `jq` + +```console +planet data filter | planet data search-quick SkySatCollect --sort 'acquired desc' --limit 1 - | jq -r .id +``` + +#### Bringing it all together + +The cool thing is you can combine the data and search commands, to make calls like ordering the most recent skysat +image that was published: + ```console planet orders request --name "SkySat Latest" --item-type SkySatCollect --bundle analytic \ @@ -520,11 +713,8 @@ planet orders request --name "SkySat Latest" --item-type SkySatCollect --bundle | planet orders create ``` -Get the 5 latest cloud free images in an area and create an order that clips to that area, using +Or get the 5 latest cloud free images in an area and create an order that clips to that area, using [geometry.geojson](https://gist.github.com/cholmes/378d050a263ae433ddbbb91c3439994b) from above: -(this one uses variables in unix) -(note that you need the tr and sd to format the output of jq into the comma delimited list we need as input. There is likely some better way to do this...) - ```console ids=`planet data filter --geom geometry.geojson --range clear_percent gt 90 | planet data \ @@ -533,6 +723,9 @@ planet orders request --name "Clipped Scenes" --item-type PSScene --bundle analy --id $ids --clip geometry.geojson | planet orders create - ``` +This one uses some advanced unix capabilities like `sed` and `tr`, along with unix variables, so more +properly belongs in the [Advanced CLI Tutorial](cli-plus-tutorial.md), but we'll leave it here to give a taste +of what's possible. ### Future workflows to support @@ -541,7 +734,7 @@ planet orders request --name "Clipped Scenes" --item-type PSScene --bundle analy to sort which results are in the same strip) * STAC output * Cloud delivery (could do this today with editing JSON, but should wait till we have it a bit better) -* Use jq to show a list of order id's, names and created time, to more easily grab the id. +* Use jq to show just orders that have a given item type, like just skysat. * get the id of the most recently created order using jq * use jq to get the id of the an order by it's name From 46967000ed39ed68a059852b661ee5dd082796ef Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Sun, 17 Jul 2022 10:04:55 -0700 Subject: [PATCH 15/30] more improvements --- docs/cli-plus-tutorial.md | 70 +++++++++++++++++++++++++++++++++------ docs/cli-tutorial.md | 4 +-- 2 files changed, 61 insertions(+), 13 deletions(-) diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md index 2b8c49939..203ac150a 100644 --- a/docs/cli-plus-tutorial.md +++ b/docs/cli-plus-tutorial.md @@ -1,8 +1,9 @@ ## About This document shows off a range of more advanced command-line workflows, making use of a wider range -of tools in the geospatial ecosystem. Some of them can be a pain to install,like GDAL/OGR, and -several pop in and out of web tools, so these are kept out of the main tutorial section. +of tools in the command-line & geospatial ecosystem. Some of them can be a pain to install, like +GDAL/OGR, and several pop in and out of web tools, so these are kept out of the main tutorial +section. ## Tools used @@ -31,30 +32,66 @@ getting the geometry input for searching or clipping. Hand-editing GeoJSON is a people will open up a desktop tool like QGIS or ArcGIS Pro and save the file. But there are a few tools that can get you back into the CLI workflow more quickly. -Draw on geojson.io, copy the geojson as input to search -(pbpaste on mac. other options for windows and linux TODO: figure these out) +#### Draw with GeoJSON.io + +One great tool for quickly drawing on a map and getting GeoJSON output is +[GeoJSON.io](https://geojson.io). You can draw and save the file, but an even faster workflow +is to use your operating system's clipboard to command-line tools. + +Just draw with the tools, then copy (with right click and 'copy' from the menu or 'ctrl c') the +JSON from the text box: + +![Select on GeoJSON.io](https://user-images.githubusercontent.com/407017/179411184-fe3061aa-32ad-4bf7-bf6a-18c59bfdfff7.png) + +On a mac you can use `pbpaste` to grab whatever is currently in your clipboard: ```console pbpaste | planet data filter --geom - | planet data search-quick SkySatCollect - ``` -placemark +(TODO: Find alternatives for windows and linux) + +#### Draw with Placemark + +A really fantastic tool for working with GeoJSON is [Placemark](https://placemark.io). It is a +commercial tool that you'll have to pay for, but it's got a really nice feature that makes it very +compatible with command-line workflows. You can easily grab the URL of any individual GeoJSON +feature and stream it in as your geometry using `curl`: + +![Stream from Placemark](https://user-images.githubusercontent.com/407017/179412209-2365d79a-9260-47e5-9b08-9bc5b84b6ddc.gif) ```console -curl -s https://api.placemark.io/api/v1/map/Dse5hjAzfrLyXDU9WiFn7/feature/11b80520-f672-11ec-98c0-5b51b58d2b56 | planet data filter --geom - | planet data search-quick SkySatCollect - +curl -s https://api.placemark.io/api/v1/map/a0BWUEErqU9A1EDHZWHez/feature/278cd610-05ee-11ed-8fdd-15633e4f8f01 | \ +planet data filter --geom - | jq ``` ### Geometry Visualization +Looking at a big list of GeoJSON is a lot less useful then actually being able to see the footprints. This is often +done by saving the output files and opening with a desktop GIS program, but there are some nice alternatives that +let you pipe (`|`) the output more directly. + +#### Copy GeoJSON to clipboard + +One of the quicker routes to visualizing search output is to copy the output to your clipboard and paste into a +tool that will take GeoJSON and visualize it. -Copy output to clipboard. - - paste to geojson.io - - paste to placemark (not quite working at the moment) +You can do this on GeoJSON.io + +(TODO: record example) + +Or also on Placemark, which tends to perform a bit better (especially when you get above 1000 features) + +For both it's recommended to pass the output through `planet collect` to get properly formatted GeoJSON: ```console planet data filter --string-in strip_id 5743669 | planet data search-quick PSScene - | planet collect - | pbcopy ``` +(TODO: Get pbcopy equivalents for windows and linux) + +#### Kepler + Show the latest 2500 collects for the state, across assets. - filter by provider, and instrument, gsd - show customizing the pop-up of properties @@ -129,11 +166,24 @@ strip-id=`planet data filter | planet data search-quick PSScene - --limit 1 | jq planet data filter --string-in strip_id $stripid | planet data search-quick PSScene - ``` +### Advanced jq +- get id by array number ```console - +planet orders list | jq -rs '.[3] | "\(.id) \(.created_on) \(.name) \(.state)"' ``` +(limit can get the most recent, but not a second or third) + +* Use jq to show just orders that have a given item type, like just skysat. + +planet orders list | jq -rs '.[] | "\(.id) \(.created_on) \(.state) \(.products[0].item_type)"' will show the item type + +https://gist.github.com/ipbastola/2c955d8bf2e96f9b1077b15f995bdae3 has ideas for contains, but haven't got it right yet + +* use jq to get the id of the an order by it's name + +* get total number of items, add up each year of `stats` ```console diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md index ab642378e..45f283e06 100644 --- a/docs/cli-tutorial.md +++ b/docs/cli-tutorial.md @@ -734,7 +734,5 @@ of what's possible. to sort which results are in the same strip) * STAC output * Cloud delivery (could do this today with editing JSON, but should wait till we have it a bit better) -* Use jq to show just orders that have a given item type, like just skysat. -* get the id of the most recently created order using jq -* use jq to get the id of the an order by it's name + From 5e5a52415ca70e16f4efc5e8857e852ef80a9dc3 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Sun, 17 Jul 2022 21:38:58 -0700 Subject: [PATCH 16/30] more kepler info --- docs/cli-plus-tutorial.md | 40 +++++++++++++++++++++++++++++++-------- 1 file changed, 32 insertions(+), 8 deletions(-) diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md index 203ac150a..4f375759b 100644 --- a/docs/cli-plus-tutorial.md +++ b/docs/cli-plus-tutorial.md @@ -90,24 +90,48 @@ planet data filter --string-in strip_id 5743669 | planet data search-quick PSSce (TODO: Get pbcopy equivalents for windows and linux) -#### Kepler +#### Kepler.gl -Show the latest 2500 collects for the state, across assets. - - filter by provider, and instrument, gsd - - show customizing the pop-up of properties +One of the best tools to visualize large amounts of imagery footprints is a tool called [kepler.gl](https://kepler.gl/), +which has a really awesome command-line version which is perfect for working with Planet's CLI. To get the CLI go to +[keplergl_cli](https://github.com/kylebarron/keplergl_cli) and follow the +[installation instructions](https://github.com/kylebarron/keplergl_cli#install). Be sure to get a Mapbox API key (from +the [access tokens](https://account.mapbox.com/access-tokens/) page) - just sign up for a free account if you don't have +one already. The kepler CLI won't work at all without getting one and setting it as the `MAPBOX_API_KEY` environment +variable. + +Once it's set up you can just pipe any search command directly to `kepler` (it usually does fine even without +`planet collect` to go from ndgeojson to geojson). For example: ```console -curl -s https://raw.githubusercontent.com/ropensci/geojsonio/master/inst/examples/california.geojson | planet data filter --geom - | planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect,Sentinel1 --limit 2500 - | kepler +curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json | planet data filter --permission false --geom - \ +| planet data search-quick PSScene - | kepler ``` -Draw in placemark and run a search in area you made, visualize output in Kepler. - - filter by acquistion, show animation of time. +Note `--permission false` is added to these examples so that anyone with Planet access can try them, even if you don't +have download permission for the area shown. + +(TODO: Add animated gif, showing some options) + +Kepler really excels at larger amounts of data, so try it out with larger limits: ```console -curl -s https://api.placemark.io/api/v1/map/Dse5hjAzfrLyXDU9WiFn7/feature/11b80520-f672-11ec-98c0-5b51b58d2b56 | planet data filter --geom - | planet data search-quick SkySatCollect - | kepler +curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json | planet data filter --permission false --geom - \ +| planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect,Sentinel1 \ +--sort 'acquired desc' --limit 1500 - | kepler ``` +(show animated gif with 600 - lower amount so it takes less time to load). + +And you can bring it all together using Placemark for input and Kepler for output: +![Placemark & Kepler with Planet CLI](https://storage.googleapis.com/open-geodata/ch/planet-cli-pm-kepler.gif) + +```console +curl -s https://api.placemark.io/api/v1/map/a0BWUEErqU9A1EDHZWHez/feature/91a07390-0652-11ed-8fdd-15633e4f8f01 \ +| planet data filter --permission false --geom - \ +| planet data search-quick PSScene,Landsat8L1G,SkySatCollect,Sentinel1 - | kepler +``` #### Large Dataset Visualization From b998300311f39a26b007771b56e48a2af7683a50 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Sun, 17 Jul 2022 21:45:17 -0700 Subject: [PATCH 17/30] small fixes --- docs/cli-plus-tutorial.md | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md index 4f375759b..9014cbea6 100644 --- a/docs/cli-plus-tutorial.md +++ b/docs/cli-plus-tutorial.md @@ -104,8 +104,10 @@ Once it's set up you can just pipe any search command directly to `kepler` (it u `planet collect` to go from ndgeojson to geojson). For example: ```console -curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json | planet data filter --permission false --geom - \ -| planet data search-quick PSScene - | kepler +curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json \ +| planet data filter --permission false --geom - \ +| planet data search-quick PSScene - \ +| kepler ``` Note `--permission false` is added to these examples so that anyone with Planet access can try them, even if you don't @@ -116,16 +118,18 @@ have download permission for the area shown. Kepler really excels at larger amounts of data, so try it out with larger limits: ```console -curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json | planet data filter --permission false --geom - \ +curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json \ +| planet data filter --permission false --geom - \ | planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect,Sentinel1 \ ---sort 'acquired desc' --limit 1500 - | kepler +--sort 'acquired desc' --limit 1500 - \ +| kepler ``` (show animated gif with 600 - lower amount so it takes less time to load). And you can bring it all together using Placemark for input and Kepler for output: -![Placemark & Kepler with Planet CLI](https://storage.googleapis.com/open-geodata/ch/planet-cli-pm-kepler.gif) +![Placemark and Kepler with Planet CLI](https://storage.googleapis.com/open-geodata/ch/planet-cli-pm-kepler.gif) ```console curl -s https://api.placemark.io/api/v1/map/a0BWUEErqU9A1EDHZWHez/feature/91a07390-0652-11ed-8fdd-15633e4f8f01 \ @@ -168,6 +172,7 @@ Draw on geojson.io, copy the geojson as input to search pbpaste | planet data filter --geom - | planet data search-quick SkySatCollect - ``` +### Post to Github as gist Show the latest skysat image on github as a gist. From 518130e083085572bffa53d3305edb5cf4f98e2b Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Mon, 18 Jul 2022 19:57:58 -0700 Subject: [PATCH 18/30] more improvements --- docs/cli-plus-tutorial.md | 51 +++++++++++++++++++++++++++++++++------ 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md index 9014cbea6..6f8f1b410 100644 --- a/docs/cli-plus-tutorial.md +++ b/docs/cli-plus-tutorial.md @@ -129,6 +129,8 @@ curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json \ And you can bring it all together using Placemark for input and Kepler for output: +(TODO: Figure out why this link isn't working... Maybe needs to be smaller?) + ![Placemark and Kepler with Planet CLI](https://storage.googleapis.com/open-geodata/ch/planet-cli-pm-kepler.gif) ```console @@ -139,24 +141,35 @@ curl -s https://api.placemark.io/api/v1/map/a0BWUEErqU9A1EDHZWHez/feature/91a073 #### Large Dataset Visualization -Download lots of scenes (current version of CLI may crap out before it gets there, will take like 10 minutes) +Oftentimes it can be useful to visualize a large amount of data, to really get a sense of the +coverage and then do some filtering of the output. For this we recommend downloading the output +to disk. Getting 200,000 skysat collects will take around 10 minutes, and will be over 800 megabytes +in GeoJSON. ```console -planet data filter | planet data search-quick SkySatCollect --limit 200000 > skysat-large.json +planet data filter | planet data search-quick SkySatCollect --limit 200000 > skysat-large.geojson ``` -Put into a feature collection - often this will clean things up if other programs can't open it (but not always): +Many GIS programs will open that geojson file by default. But if you have any trouble it can be useful to run it +through `planet collect`: ```console planet data collect skysat-large.json > skysat-large-clean.json ``` +This turns it into a real GeoJSON, instead of a newline-delimited one, which more programs understand. + +GeoJSON is an amazing format for communicating online, but is less good with really large amounts of data, so +we recommend converting it to a format that has a spatial index. We recommend a geopackage, but a shapefile can work +as well. The `ogr2ogr` of [GDAL/OGR](https://gdal.org/) is a great tool for this. We recommend using the +[binaries](https://gdal.org/download.html#binaries), or if you're on a Mac then use [homebrew] + Turn into geopackage (or shapefile) for a spatial index, and simplify (don't need so many points to visualize) - simplify .001 matches fidelity very close, mostly just removes points from lines. .008 drops a bit of info but generally good. .1 messes with things a lot, but still good for visualization. ```console -ogr2ogr skysat-large.gpkg skysat-large-clean.json -simplify .008 +ogr2ogr skysat-large.gpkg skysat-large.json -simplify .008 ``` Can open `skysat-large.gpkg` with kepler, or other tools. @@ -185,13 +198,16 @@ Show all ps-scenes in a strip on github gist. (may need to reload the page, for some reason it wasn't showing up immediately after open) ```console -planet data filter --string-in strip_id 5743640 | planet data search-quick PSScene - | gh gist create -f ps-search.geojson -w +planet data filter --string-in strip_id 5743640 | planet data search-quick PSScene - \ +| gh gist create -f ps-search.geojson -w ``` -TODO: get command that gets the latest strip id and uses that as input in one line. May need to update filter commands to take stdin? +TODO: get a command that gets the latest strip id and uses that as input in one line. May need to update filter commands to take stdin? +This current command doesn't quite work. ```console -strip-id=`planet data filter | planet data search-quick PSScene - --limit 1 | jq -r '.properties.strip_id' | sed 's/\\[tn]//g'` +strip-id=`planet data filter | planet data search-quick PSScene - --limit 1 \ +| jq -r '.properties.strip_id' | sed 's/\\[tn]//g'` planet data filter --string-in strip_id $stripid | planet data search-quick PSScene - ``` @@ -214,6 +230,27 @@ https://gist.github.com/ipbastola/2c955d8bf2e96f9b1077b15f995bdae3 has ideas for * get total number of items, add up each year of `stats` +### Simplify Geometries to 500 vertices + +One of the limits of Planet's API's is that they demand geometries have less than 500 vertices. This section shows some +tools that can help + +#### Mapshaper + +- simplify CLI +- simplify UI, download and use +- count number of resulting vertices +`ogrinfo -al simplify_test.geojson | grep POLYGON | sed 's/$/,/' | tr -d -c "," | wc` +But there should be a JQ way to do the same... + +#### Placemark + +- see count of vertices +- union together +- buffer and union +- simplify + + ```console ``` From 087770b1872a08a77e6a72bfe26b19f2f6edeeb2 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Tue, 19 Jul 2022 14:37:35 -0700 Subject: [PATCH 19/30] changed quick-search to search, updates --- docs/cli-plus-tutorial.md | 160 +++++++++++++++++++++++++------------- docs/cli-tutorial.md | 44 +++++------ 2 files changed, 130 insertions(+), 74 deletions(-) diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md index 6f8f1b410..34e6e957a 100644 --- a/docs/cli-plus-tutorial.md +++ b/docs/cli-plus-tutorial.md @@ -5,6 +5,9 @@ of tools in the command-line & geospatial ecosystem. Some of them can be a pain GDAL/OGR, and several pop in and out of web tools, so these are kept out of the main tutorial section. +**WORK IN PROGRESS**: This document is still under construction, with a number of TODO's remaining, +but we are publishing as there's a lot of good information here. + ## Tools used * **[GDAL/OGR](https://gdal.org)** - We'll mostly use OGR, the vector tooling. @@ -46,7 +49,7 @@ JSON from the text box: On a mac you can use `pbpaste` to grab whatever is currently in your clipboard: ```console -pbpaste | planet data filter --geom - | planet data search-quick SkySatCollect - +pbpaste | planet data filter --geom - | planet data search SkySatCollect - ``` (TODO: Find alternatives for windows and linux) @@ -76,20 +79,54 @@ let you pipe (`|`) the output more directly. One of the quicker routes to visualizing search output is to copy the output to your clipboard and paste into a tool that will take GeoJSON and visualize it. -You can do this on GeoJSON.io +You can do this on GeoJSON.io: (TODO: record example) -Or also on Placemark, which tends to perform a bit better (especially when you get above 1000 features) +Or also on Placemark, which tends to perform a bit better (especially when you get above 1000 features). + +(TODO: record example) For both it's recommended to pass the output through `planet collect` to get properly formatted GeoJSON: ```console -planet data filter --string-in strip_id 5743669 | planet data search-quick PSScene - | planet collect - | pbcopy +planet data filter --string-in strip_id 5743669 | planet data search PSScene - | planet collect - | pbcopy ``` (TODO: Get pbcopy equivalents for windows and linux) +#### Post to Github as gist + +Another easy option that is a bit more persistent is to post to Github using the +[`gh` cli tool](https://github.com/cli/cli). Specifically using the `gist create` command. + +The following command will get the latest SkySat image captured, upload to github, and open +your browser to see it: + +```console +planet data filter | planet data search SkySatCollect - --sort 'acquired desc' --limit 1 \ +| planet collect - | jq | gh gist create -f latest-skysat.geojson -w +``` + +Or you can show all ps-scenes in a strip on github gist. +(You may need to reload the page, for some reason it doesn't always showing up immediately after open) + +```console +planet data filter --string-in strip_id 5743640 | planet data search PSScene - \ +| planet collect - | gh gist create -f ps-search.geojson -w +``` + + +TODO: get a command that gets the latest strip id and uses that as input in one line. May need to update filter commands to take stdin? +This current command doesn't quite work. + +```console +strip-id=`planet data filter | planet data search PSScene - --limit 1 \ +| jq -r '.properties.strip_id' | sed 's/\\[tn]//g'` +planet data filter --string-in strip_id $stripid | planet data search PSScene - +``` + + #### Kepler.gl One of the best tools to visualize large amounts of imagery footprints is a tool called [kepler.gl](https://kepler.gl/), @@ -106,7 +143,7 @@ Once it's set up you can just pipe any search command directly to `kepler` (it u ```console curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json \ | planet data filter --permission false --geom - \ -| planet data search-quick PSScene - \ +| planet data search PSScene - \ | kepler ``` @@ -120,7 +157,7 @@ Kepler really excels at larger amounts of data, so try it out with larger limits ```console curl -s https://storage.googleapis.com/open-geodata/ch/vermont.json \ | planet data filter --permission false --geom - \ -| planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect,Sentinel1 \ +| planet data search PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect,Sentinel1 \ --sort 'acquired desc' --limit 1500 - \ | kepler ``` @@ -136,81 +173,110 @@ And you can bring it all together using Placemark for input and Kepler for outpu ```console curl -s https://api.placemark.io/api/v1/map/a0BWUEErqU9A1EDHZWHez/feature/91a07390-0652-11ed-8fdd-15633e4f8f01 \ | planet data filter --permission false --geom - \ -| planet data search-quick PSScene,Landsat8L1G,SkySatCollect,Sentinel1 - | kepler +| planet data search PSScene,Landsat8L1G,SkySatCollect,Sentinel1 - | kepler ``` #### Large Dataset Visualization Oftentimes it can be useful to visualize a large amount of data, to really get a sense of the coverage and then do some filtering of the output. For this we recommend downloading the output -to disk. Getting 200,000 skysat collects will take around 10 minutes, and will be over 800 megabytes -in GeoJSON. +to disk. Getting 20,000 skysat collects will take at least a couple of minutes, and will be over +100 megabytes of GeoJSON on disk. ```console -planet data filter | planet data search-quick SkySatCollect --limit 200000 > skysat-large.geojson +planet data filter | planet data search SkySatCollect --limit 20000 > skysat-large.geojson +``` + +Kepler can fairly easily handle 20,000 skysat footprints, try: + +```console +kepler skysat-large.geojson ``` Many GIS programs will open that geojson file by default. But if you have any trouble it can be useful to run it through `planet collect`: ```console -planet data collect skysat-large.json > skysat-large-clean.json +planet data collect skysat-large.geojson > skysat-large-clean.geojson ``` - This turns it into a real GeoJSON, instead of a newline-delimited one, which more programs understand. +If you want to visualize even larger sets of footprints there's a few things we recommend: + +##### Convert to GeoPackage + GeoJSON is an amazing format for communicating online, but is less good with really large amounts of data, so -we recommend converting it to a format that has a spatial index. We recommend a geopackage, but a shapefile can work -as well. The `ogr2ogr` of [GDAL/OGR](https://gdal.org/) is a great tool for this. We recommend using the -[binaries](https://gdal.org/download.html#binaries), or if you're on a Mac then use [homebrew] +we recommend converting it to a format that has a spatial index and isn't so large on disk. We recommend +[geopackage](https://www.geopackage.org/), but shapefile can work as well. The `ogr2ogr` of +[GDAL/OGR](https://gdal.org/) is a great tool for this. We recommend using the +[binaries](https://gdal.org/download.html#binaries), or if you're on a Mac then use +[homebrew](https://brew.sh/) (run `brew install gdal` after you get it set up). If you're having +trouble getting GDAL working well a good backup can be to use docker and the +[osgeo/gdal](https://hub.docker.com/r/osgeo/gdal) package. -Turn into geopackage (or shapefile) for a spatial index, and simplify (don't need so many points to visualize) - - simplify .001 matches fidelity very close, mostly just removes points from lines. .008 drops a bit of info but generally good. .1 - messes with things a lot, but still good for visualization. +To convert to a geopackage run: ```console -ogr2ogr skysat-large.gpkg skysat-large.json -simplify .008 +ogr2ogr skysat-large.gpkg skysat-large.geojson ``` -Can open `skysat-large.gpkg` with kepler, or other tools. +##### Simplification with OGR -Other simplification options for large datasets: - -* Use QGIS, run 'convex hull' (Vector -> Geoprocessing -> Convex Hull). Good idea to convert to gpkg or shapefile before you open in qgis if large. +The other thing you'll likely want to do to visualize large amounts of data is to simplify it +some. Many simplification tools call for a 'tolerance', often set in degrees. For SkySat some useful values are: + +| tolerance | result | +|-----------|-----------------------------------------------------------------------------------------------------------------| +| 0.001 | Mostly removes unnecessary points, visually looks pretty much the same, but much easier for programs to render. | +| 0.01 | Messes with the shape a bit, but the footprint generally looks the same, with a couple vertices off. | +| 0.1 | Mashes the shape, often into a triangle, but still useful for understanding broad coverage. | -Draw on geojson.io, copy the geojson as input to search -(pbpaste on mac. other options for windows and linux TODO: figure these out) +It's worth experimenting with options between these as well. The more simplification the easier it is for programs to +render the results. `ogr2ogr` includes the ability to simplify any output: ```console -pbpaste | planet data filter --geom - | planet data search-quick SkySatCollect - +ogr2ogr skysat-large.gpkg skysat-large.json -simplify .008 ``` -### Post to Github as gist +#### Simplification with Mapshaper -Show the latest skysat image on github as a gist. +Another great tool is [Mapshaper](https://github.com/mbloch/mapshaper), which excels at simplification. It offers a +web-based user interface to see the results of simplification, and also a command-line tool you can use if you +find a simplification percentage you're happy with. After you get it +[installed](https://github.com/mbloch/mapshaper#installation) you can fire up the UI with: ```console -planet data filter | planet data search-quick SkySatCollect - --sort 'acquired desc' --limit 1 \ -| planet collect - | jq | gh gist create -f latest-skysat.geojson -w +mapshaper-gui skysat-large.geojson ``` -Show all ps-scenes in a strip on github gist. -(may need to reload the page, for some reason it wasn't showing up immediately after open) +(TODO: Show animated gif of recording) + +It's easy to get a sense of how much simplification affects the shape. You can download the output from the web +interface, or you can also run the command-line program: ```console -planet data filter --string-in strip_id 5743640 | planet data search-quick PSScene - \ -| gh gist create -f ps-search.geojson -w +mapshaper -i footprints.geojson -simplify 15% -o simplified.geojson ``` -TODO: get a command that gets the latest strip id and uses that as input in one line. May need to update filter commands to take stdin? -This current command doesn't quite work. +Once you find a simplification amount you're happy with you can use it as a piped output. ```console -strip-id=`planet data filter | planet data search-quick PSScene - --limit 1 \ -| jq -r '.properties.strip_id' | sed 's/\\[tn]//g'` -planet data filter --string-in strip_id $stripid | planet data search-quick PSScene - +planet data filter | planet data search --limit 20 SkySatCollect - | planet collect - | mapshaper -i - -simplify 15% -o skysat-ms2.geojson ``` +Mapshaper also has more simplification algorithms to try out, so we recommend diving into the +[CLI options](https://github.com/mbloch/mapshaper/wiki/Command-Reference). + +#### Simplification with QGIS + +Another good tool for simplification is QGIS. + +TODO: Flesh out this section, add in command-line qgis_processing option. + +Other simplification options for large datasets: + +* Use QGIS, run 'convex hull' (Vector -> Geoprocessing -> Convex Hull). Good idea to convert to gpkg or shapefile before you open in qgis if large. + ### Advanced jq - get id by array number @@ -233,7 +299,9 @@ https://gist.github.com/ipbastola/2c955d8bf2e96f9b1077b15f995bdae3 has ideas for ### Simplify Geometries to 500 vertices One of the limits of Planet's API's is that they demand geometries have less than 500 vertices. This section shows some -tools that can help +tools that can help you to do that simplification. + +TODO: flesh these out. #### Mapshaper @@ -250,15 +318,3 @@ But there should be a JQ way to do the same... - buffer and union - simplify - -```console - -``` -```console - -``` - -```console - -``` - diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md index 45f283e06..3e09cc314 100644 --- a/docs/cli-tutorial.md +++ b/docs/cli-tutorial.md @@ -466,18 +466,18 @@ There are a lot of options for what you can do with the command, and we recommen `planet data filter --help` often to get a reference of how the commands work. And we'll try to give lots of examples below as well. -So you can run the filter command and save it, and then use that file with the `search-quick` +So you can run the filter command and save it, and then use that file with the `search` command: ```console planet data filter > filter.json -planet data search-quick PSScene filter.json +planet data search PSScene filter.json ``` Or the recommended route is to use a pipe (`|`), as mentioned above [above](#create-request-and-order-in-one-call): ```console -planet data filter | planet data search-quick PSScene - +planet data filter | planet data search PSScene - ``` Note the dash (`-`), which explicitly tells the CLI to use the output from the call that is piped into it. @@ -486,11 +486,11 @@ Note the dash (`-`), which explicitly tells the CLI to use the output from the c These first searches were done on the [PSScene](https://developers.planet.com/docs/data/psscene/) 'item type', but you can use any [Item Type](https://developers.planet.com/docs/apis/data/items-assets/#item-types) that Planet offers in -its catalog. The item type is the first argument of the `search-quick` command, followed by the 'filter'. Note that +its catalog. The item type is the first argument of the `search` command, followed by the 'filter'. Note that you can specify any number of item types here: ```console -planet data filter | planet data search-quick PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect, - +planet data filter | planet data search PSScene,Sentinel2L1C,Landsat8L1G,SkySatCollect, - ``` This will search for all the most recent images captured by PlanetScope, SkySat, Sentinel 2 and Landsat 8 satellites. @@ -545,14 +545,14 @@ of Iowa. You can copy it and save as a file called `geometry.json` And then run it with this command: ```console -planet data filter --geom geometry.json | planet data search-quick PSScene - +planet data filter --geom geometry.json | planet data search PSScene - ``` Note that by default all searches with the command-line return 100 results, but you can easily increase that with the `--limit` flag: ```console -planet data filter --geom geometry.json | planet data search-quick --limit 500 PSScene - +planet data filter --geom geometry.json | planet data search --limit 500 PSScene - ``` Creating geometries for search can be annoying in a command-line workflow, but there are some ideas in the @@ -566,13 +566,13 @@ the `planet collect` method to transform the output from the Data API to valid G output to it: ```console -planet data filter --geom geometry.json | planet data search-quick PSScene - | planet collect - +planet data filter --geom geometry.json | planet data search PSScene - | planet collect - ``` If you want to visualize this you can save it as a file: ```console -planet data filter --geom geometry.json | planet data search-quick PSScene - | planet collect - > planet-search.geojson +planet data filter --geom geometry.json | planet data search PSScene - | planet collect - > planet-search.geojson ``` This you can then open with your favorite GIS program, or see this @@ -584,7 +584,7 @@ the command-line. Some of the most common filtering is by date. You could get all imagery acquired before August 2021: ```console -planet data filter --date-range acquired lt 2021-08-01 | planet data search-quick PSScene - +planet data filter --date-range acquired lt 2021-08-01 | planet data search PSScene - ``` The 'operator' in this case is 'less than' (`lt`). The options are: @@ -600,7 +600,7 @@ do a search for all images in July of 2021: ```console planet data filter --date-range acquired gte 2021-07-01 --date-range acquired lt 2021-08-01 | \ -planet data search-quick PSScene - +planet data search PSScene - ``` The date input understands [RFC 3339](https://datatracker.ietf.org/doc/html/rfc3339) and @@ -610,7 +610,7 @@ on July 1st 2021: ```console planet data filter --date-range acquired gte 2021-07-01:06:20:10 --date-range acquired lt 2021-07-01:06:20:15 | \ -planet data search-quick PSScene - +planet data search PSScene - ``` #### Range Filter @@ -628,19 +628,19 @@ For properties that are strings you can use the `string-in` filter. For example with PS2 instrument: ```console -planet data filter --string-in instrument PS2 | planet data search-quick PSScene - +planet data filter --string-in instrument PS2 | planet data search PSScene - ``` You can specify multiple strings to match, with a comma: ```console -planet data filter --string-in instrument PS2,PSB.SD | planet data search-quick PSScene - +planet data filter --string-in instrument PS2,PSB.SD | planet data search PSScene - ``` Another example is to select all data in a single strip: ```console -planet data filter --string-in strip_id 5743640 | planet data search-quick PSScene - +planet data filter --string-in strip_id 5743640 | planet data search PSScene - ``` Note that in all these commands we are piping the results into the search. If you don't include the pipe then you'll @@ -651,13 +651,13 @@ get the filter output, which can be interested to inspect to see exactly what is You can limit your search to only data with a particular asset, for example search just for 8-band analytic assets: ```console -planet data filter --asset ortho_analytic_8b_sr | planet data search-quick PSScene - +planet data filter --asset ortho_analytic_8b_sr | planet data search PSScene - ``` Or 8-band assets that also have a UDM. ```console -planet data filter --asset ortho_analytic_8b_sr --asset udm2 | planet data search-quick PSScene - +planet data filter --asset ortho_analytic_8b_sr --asset udm2 | planet data search PSScene - ``` You can find the list of available assets in each Item Type Page, like @@ -675,7 +675,7 @@ and are able to download. But if you'd like to just get search Planet's catalog you can set the permission filter to false: ```console -planet data filter --permission false --asset ortho_analytic_8b_sr | planet data search-quick PSScene - +planet data filter --permission false --asset ortho_analytic_8b_sr | planet data search PSScene - ``` #### Sort @@ -692,13 +692,13 @@ descending order. The options are are: The lets you do things like get the id of the most recent skysat image taken (that you have download access to): ```console -planet data filter | planet data search-quick SkySatCollect --sort 'acquired desc' --limit 1 - +planet data filter | planet data search SkySatCollect --sort 'acquired desc' --limit 1 - ``` And you can also just get the ID, using `jq` ```console -planet data filter | planet data search-quick SkySatCollect --sort 'acquired desc' --limit 1 - | jq -r .id +planet data filter | planet data search SkySatCollect --sort 'acquired desc' --limit 1 - | jq -r .id ``` #### Bringing it all together @@ -709,7 +709,7 @@ image that was published: ```console planet orders request --name "SkySat Latest" --item-type SkySatCollect --bundle analytic \ ---id `planet data filter | planet data search-quick SkySatCollect --sort 'acquired desc' --limit 1 - | jq -r .id` \ +--id `planet data filter | planet data search SkySatCollect --sort 'acquired desc' --limit 1 - | jq -r .id` \ | planet orders create ``` @@ -718,7 +718,7 @@ Or get the 5 latest cloud free images in an area and create an order that clips ```console ids=`planet data filter --geom geometry.geojson --range clear_percent gt 90 | planet data \ -search-quick PSScene --limit 5 - | jq -r .id | tr '\n' , | sed 's/.$//'` +search PSScene --limit 5 - | jq -r .id | tr '\n' , | sed 's/.$//'` planet orders request --name "Clipped Scenes" --item-type PSScene --bundle analytic_sr_udm2 \ --id $ids --clip geometry.geojson | planet orders create - ``` From 302ab93689628a4c15c829700fa3b39513be7541 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Tue, 19 Jul 2022 22:20:09 -0700 Subject: [PATCH 20/30] added warning messages, more info --- docs/cli-concepts.md | 31 ++++++++++++++++++------------- docs/cli-plus-tutorial.md | 6 ++++++ 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/docs/cli-concepts.md b/docs/cli-concepts.md index 856587dc8..64b7e8e40 100644 --- a/docs/cli-concepts.md +++ b/docs/cli-concepts.md @@ -4,6 +4,9 @@ Planet's command-line interface is built of composable pieces that combine in po To take full advantage of all that Planet's CLI offers there are a few concepts and tools that are worth explaining. +**NOTE:** This document is still a work in progress, but we'll work to flesh it out. We'll +leave it up since there's still a bit of good information. + ## Core Unix Concepts If you are completely new to the command-line we recommend starting with a real introductory @@ -39,24 +42,26 @@ then referring to it you can just do it all in one call: planet data filter --range cloud_percent lt 10 | planet data search-quick PSScene - ``` -The pipe says to take - -There's a good tutorial on https://ryanstutorials.net/linuxtutorial/piping.php - - +The pipe says to take the output of the first command and pass it to the input of +the second. You'll notice that the planet command has a dash (`-`), this is a convention +that is often used by different CLI programs to explicitly say 'read from +standard out'. Most Planet CLI commands require it, but one or two will implicitly +read from standard out if it's not explicitly included. Using the dash to mean +'read from standard out' is a general convention used by many programs, but it's +not universal, so check the docs of the program you're using as to how it reads +from piped input. For example GDAL/OGR uses a specific `/vsistdin/` convention to +read from a pipe. -incuding reading from a pipe with - +If you'd like to learn more about these topics then check out +[this tutorial](https://ryanstutorials.net/linuxtutorial/piping.php). And if you'd +like to learn more about the dash (`-`) see +[this tutorial](https://www.baeldung.com/linux/dash-in-command-line-parameters). -https://www.baeldung.com/linux/dash-in-command-line-parameters - -stringing several together - -### Output to file - -> ### head & tail +Also less/more? + ### Running a command within another ### variables diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md index 34e6e957a..6ba0fcd34 100644 --- a/docs/cli-plus-tutorial.md +++ b/docs/cli-plus-tutorial.md @@ -238,6 +238,12 @@ render the results. `ogr2ogr` includes the ability to simplify any output: ogr2ogr skysat-large.gpkg skysat-large.json -simplify .008 ``` +Alternative - use convex hull. TODO: test this, write it up + +```console +ogr2ogr skysat-convex.gpkg skysat.geojson ogr2ogr -sql "select st_convexhull(geometry) from skysat" -dialect sqlite +``` + #### Simplification with Mapshaper Another great tool is [Mapshaper](https://github.com/mbloch/mapshaper), which excels at simplification. It offers a From 0e02559a67072298709fc703caa5b89fc57fb7bd Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Tue, 19 Jul 2022 22:29:35 -0700 Subject: [PATCH 21/30] added note to add centroid with gdal --- docs/cli-plus-tutorial.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md index 6ba0fcd34..e3528b3bc 100644 --- a/docs/cli-plus-tutorial.md +++ b/docs/cli-plus-tutorial.md @@ -244,6 +244,9 @@ Alternative - use convex hull. TODO: test this, write it up ogr2ogr skysat-convex.gpkg skysat.geojson ogr2ogr -sql "select st_convexhull(geometry) from skysat" -dialect sqlite ``` +Other alternative for really big ones, centroid. GDAL should be able to do this, need to figure out the similar +sql. + #### Simplification with Mapshaper Another great tool is [Mapshaper](https://github.com/mbloch/mapshaper), which excels at simplification. It offers a From 50916aa32de20fd197034dc1be6422ce5ffc314f Mon Sep 17 00:00:00 2001 From: Chris Holmes Date: Wed, 20 Jul 2022 07:11:45 -0700 Subject: [PATCH 22/30] Update docs/cli-concepts.md Co-authored-by: Jennifer Reiber Kyle --- docs/cli-concepts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cli-concepts.md b/docs/cli-concepts.md index 64b7e8e40..14fc8de1b 100644 --- a/docs/cli-concepts.md +++ b/docs/cli-concepts.md @@ -21,7 +21,7 @@ or [Cygwin](https://www.cygwin.com/). Several commands in the Planet CLI are used to create files that are used as input to other commands. The default of these commands is to just print the output on your screen. Seeing it on the screen can be useful for making sure it's right, but you'll most likely -want to make use of it. This is where the concept of 'redirection' in. If you use the +want to make use of it. This is where the concept of 'redirection' comes in. If you use the `>` character and then specify a file name the command-line will save its output to that file. So if you say: From db90a0322ddb1590e6b579639701ae2bac4bd8bf Mon Sep 17 00:00:00 2001 From: Chris Holmes Date: Wed, 20 Jul 2022 07:19:07 -0700 Subject: [PATCH 23/30] Update docs/cli-tutorial.md Co-authored-by: Jennifer Reiber Kyle --- docs/cli-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md index 3e09cc314..6ce125e24 100644 --- a/docs/cli-tutorial.md +++ b/docs/cli-tutorial.md @@ -703,7 +703,7 @@ planet data filter | planet data search SkySatCollect --sort 'acquired desc' --l #### Bringing it all together -The cool thing is you can combine the data and search commands, to make calls like ordering the most recent skysat +The cool thing is you can combine the data and order commands, to make calls like ordering the most recent skysat image that was published: From 7839a5b16bc81a527248c038f6f504c0d2343240 Mon Sep 17 00:00:00 2001 From: Chris Holmes Date: Wed, 20 Jul 2022 11:48:10 -0700 Subject: [PATCH 24/30] Update docs/cli-concepts.md Co-authored-by: Jennifer Reiber Kyle --- docs/cli-concepts.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cli-concepts.md b/docs/cli-concepts.md index 14fc8de1b..fa91579e5 100644 --- a/docs/cli-concepts.md +++ b/docs/cli-concepts.md @@ -26,7 +26,7 @@ want to make use of it. This is where the concept of 'redirection' comes in. If So if you say: ``` - $ planet data filter --range cloud_percent lt 10 > filter.json + planet data filter --range cloud_percent lt 10 > filter.json ``` Then the output will be saved. This output is referred to as STDOUT, or 'standard output'. From a2d97aee1a0a77f301d6e4c8ebb1d8bc21cdf3b3 Mon Sep 17 00:00:00 2001 From: Chris Holmes Date: Wed, 20 Jul 2022 11:50:32 -0700 Subject: [PATCH 25/30] Update docs/cli-plus-tutorial.md --- docs/cli-plus-tutorial.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md index e3528b3bc..4a8cb6c34 100644 --- a/docs/cli-plus-tutorial.md +++ b/docs/cli-plus-tutorial.md @@ -129,7 +129,7 @@ planet data filter --string-in strip_id $stripid | planet data search PSScene - #### Kepler.gl -One of the best tools to visualize large amounts of imagery footprints is a tool called [kepler.gl](https://kepler.gl/), +One of the best tools to visualize large numbers of imagery footprints is a tool called [kepler.gl](https://kepler.gl/), which has a really awesome command-line version which is perfect for working with Planet's CLI. To get the CLI go to [keplergl_cli](https://github.com/kylebarron/keplergl_cli) and follow the [installation instructions](https://github.com/kylebarron/keplergl_cli#install). Be sure to get a Mapbox API key (from From fef22547ddd535ffa741d5c1036d9915c0d993ab Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Thu, 21 Jul 2022 14:28:21 -0700 Subject: [PATCH 26/30] added to ToC, improvements --- docs/cli-plus-tutorial.md | 6 +++--- docs/cli-tutorial.md | 3 +-- mkdocs.yml | 5 +++++ 3 files changed, 9 insertions(+), 5 deletions(-) diff --git a/docs/cli-plus-tutorial.md b/docs/cli-plus-tutorial.md index 4a8cb6c34..62d9a3831 100644 --- a/docs/cli-plus-tutorial.md +++ b/docs/cli-plus-tutorial.md @@ -16,9 +16,9 @@ Great for things like format conversion and basic simplification. awesome kepler.gl library from the commandline. Useful for visualization of large amounts of geojson. * **[GeoJSON.io](https://geojson.io/)** - Simple tool to do editing of geojson, useful for creating -AOI's. Integrates with github, but doesn't seem to work so well. -* **[Placemark.io](https://placemark.io)** - More advanced tool from the creator of GeoJSON, very -nice for creating AOI's and piping them in. +AOI's. It integrates with github, but the ability to save a GeoJSON to github doesn't seem to work so well. +* **[Placemark.io](https://placemark.io)** - More advanced tool from the creator of GeoJSON.io, very +nice for creating AOI's and piping them in, with lots of rich geometry editing features. * **[MapShaper](https://github.com/mbloch/mapshaper)** - Tool to do interactive simplification of GeoJSON, has a nice CLI. * **[STACTools](https://github.com/stac-utils/stactools)** - CLI for working with STAC data. There diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md index 6ce125e24..4cec0ca33 100644 --- a/docs/cli-tutorial.md +++ b/docs/cli-tutorial.md @@ -335,8 +335,7 @@ Now we'll dive into the variety of ways to customize your order. These can all b combined with all the commands listed above. We'll work with a geojson saved online. You should download the -[raw json](https://gist.githubusercontent.com/cholmes/c7736ac5241d77605524d01ed2dc57a1/raw/7d24e02ba894e64c4c737c253a0cce4cac54167c/geometry.geojson) -from github. If you want to see what it looks like check the +[geometry](data/geometry.geojson). If you want to see what it looks like check the [preview](https://gist.github.com/cholmes/c7736ac5241d77605524d01ed2dc57a1)). You can move that geometry to your current directory and use the following command, or diff --git a/mkdocs.yml b/mkdocs.yml index 797041c6e..18ee60e4b 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -31,9 +31,14 @@ nav: - User Guide: 'guide.md' - Upgrading: 'upgrading.md' - Early Access: 'v2_earlyaccess.md' + - venv Tutorial: 'venv-tutorial.md' - API Reference: 'api.md' + - CLI Tutorial: 'cli-tutorial.md' + - CLI Concepts: 'cli-concepts.md' + - CLI Workflows: 'cli-plus-tutorial.md' - CLI Reference: 'cli.md' + markdown_extensions: - pymdownx.highlight - pymdownx.superfences From ba11ed2394a6e96921eeb355083d695008b1198e Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Thu, 21 Jul 2022 14:30:45 -0700 Subject: [PATCH 27/30] added request geojson to be in this repo --- docs/data/geometry.geojson | 51 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 51 insertions(+) create mode 100644 docs/data/geometry.geojson diff --git a/docs/data/geometry.geojson b/docs/data/geometry.geojson new file mode 100644 index 000000000..f4ea8601a --- /dev/null +++ b/docs/data/geometry.geojson @@ -0,0 +1,51 @@ +{ + "type": "Feature", + "properties": {}, + "geometry": { + "type": "Polygon", + "coordinates": [ + [ + [ + -48.49249707097496, + -1.4915864583752096 + ], + [ + -48.44331521120179, + -1.4866355971611682 + ], + [ + -48.42341130238833, + -1.4688569574592236 + ], + [ + -48.39783207794321, + -1.475195014769298 + ], + [ + -48.4146752248578, + -1.4989837652960034 + ], + [ + -48.46320120708097, + -1.5236040581230128 + ], + [ + -48.47883295535294, + -1.5173754152805117 + ], + [ + -48.49369951314944, + -1.5031696716383607 + ], + [ + -48.49369951314944, + -1.5031696716383607 + ], + [ + -48.49249707097496, + -1.4915864583752096 + ] + ] + ] + } +} \ No newline at end of file From 444e4e2aeab73394eb904246cf79cf744b54e2ac Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Thu, 21 Jul 2022 14:34:10 -0700 Subject: [PATCH 28/30] added clip-composite request --- docs/data/tools-clip-composite.json | 110 ++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) create mode 100644 docs/data/tools-clip-composite.json diff --git a/docs/data/tools-clip-composite.json b/docs/data/tools-clip-composite.json new file mode 100644 index 000000000..4c1293aca --- /dev/null +++ b/docs/data/tools-clip-composite.json @@ -0,0 +1,110 @@ +[ + { + "composite": + {} + }, + { + "file_format": + { + "format": "COG" + } + }, + { + "clip": + { + "aoi": + { + "type": "Polygon", + "coordinates": + [ + [ + [ + -48.446206167875374, + -1.4701167307243566 + ], + [ + -48.440921981977795, + -1.457614905615003 + ], + [ + -48.44743914458502, + -1.435604481829941 + ], + [ + -48.43863216808879, + -1.4324349632741047 + ], + [ + -48.44137477622803, + -1.421467932813556 + ], + [ + -48.43339627982283, + -1.4132426257606312 + ], + [ + -48.39973699811188, + -1.4020262510461947 + ], + [ + -48.30741661469506, + -1.3918205489938913 + ], + [ + -48.34960127062325, + -1.3168465606586466 + ], + [ + -48.42995299620014, + -1.3111564767828838 + ], + [ + -48.45204972073415, + -1.3041275317935828 + ], + [ + -48.46276328414464, + -1.2800281434470122 + ], + [ + -48.49256038237908, + -1.2930820071886302 + ], + [ + -48.487538399530735, + -1.3724078648216107 + ], + [ + -48.50059555493678, + -1.423616840601639 + ], + [ + -48.508630727494506, + -1.47013904276578 + ], + [ + -48.494569175518876, + -1.4801796796806457 + ], + [ + -48.45439331272971, + -1.4798449925136623 + ], + [ + -48.45405851387321, + -1.4798449925136623 + ], + [ + -48.45405851387321, + -1.4798449925136623 + ], + [ + -48.446206167875374, + -1.4701167307243566 + ] + ] + ] + } + } + } +] \ No newline at end of file From 14459a42615f19c2fab13ac4dd38d2973022dacf Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Thu, 21 Jul 2022 14:35:54 -0700 Subject: [PATCH 29/30] update paths --- docs/cli-tutorial.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/cli-tutorial.md b/docs/cli-tutorial.md index 4cec0ca33..d296fbbf2 100644 --- a/docs/cli-tutorial.md +++ b/docs/cli-tutorial.md @@ -335,8 +335,7 @@ Now we'll dive into the variety of ways to customize your order. These can all b combined with all the commands listed above. We'll work with a geojson saved online. You should download the -[geometry](data/geometry.geojson). If you want to see what it looks like check the -[preview](https://gist.github.com/cholmes/c7736ac5241d77605524d01ed2dc57a1)). +[geometry](data/geometry.geojson). You can move that geometry to your current directory and use the following command, or tweak the geometry.geojson to refer to where you downloaded it. @@ -409,7 +408,7 @@ planet orders request --item-type PSScene --name "Two Scenes Composited" \ To clip and composite you need to specify the clip in the tools (instead of `--clip`), as you can not use `--clip` and `--tools` in the same call. You can see the full JSON for the tools in -[this gist](https://gist.github.com/cholmes/378d050a263ae433ddbbb91c3439994b). +[here](data/tools-clip-composite.json). ```console planet orders request --item-type PSScene --name "Two Scenes Clipped and Composited" \ @@ -713,7 +712,7 @@ planet orders request --name "SkySat Latest" --item-type SkySatCollect --bundle ``` Or get the 5 latest cloud free images in an area and create an order that clips to that area, using -[geometry.geojson](https://gist.github.com/cholmes/378d050a263ae433ddbbb91c3439994b) from above: +[geometry.geojson](data/geometry.json) from above: ```console ids=`planet data filter --geom geometry.geojson --range clear_percent gt 90 | planet data \ From 3c0bbbbcce6ec003612b3050c7f5ef110fea8956 Mon Sep 17 00:00:00 2001 From: Christopher Holmes Date: Thu, 21 Jul 2022 14:37:34 -0700 Subject: [PATCH 30/30] moved to data directory --- docs/{ => data}/order-request.json | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename docs/{ => data}/order-request.json (100%) diff --git a/docs/order-request.json b/docs/data/order-request.json similarity index 100% rename from docs/order-request.json rename to docs/data/order-request.json