update documentation, comments

Author: scottbale

README.md

@@ -34,28 +34,11 @@ outputs. aws-api uses the same data descriptions to expose a
 data-oriented interface, using service descriptions, documentation,
 and specs which are generated from the source descriptions.
 
-Most AWS SDKs have their own copies of these data descriptions in their
-github repos. We use [aws-sdk-js](https://github.com/aws/aws-sdk-js/) as
-the source for these, and release separate artifacts for each api.
-The [api descriptors](https://github.com/aws/aws-sdk-js/tree/master/apis)
-include the AWS `api-version` in their filenames (and in their data). For
-example you'll see both of the following files listed:
-
-    dynamodb-2011-12-05.normal.json
-    dynamodb-2012-08-10.normal.json
-
-Whenever we release com.cognitect.aws/dynamodb, we look for the
-descriptor with the most recent API version. If aws-sdk-js-v2.351.0
-contains an update to dynamodb-2012-08-10.normal.json, or a new
-dynamodb descriptor with a more recent api-version, we'll make a
-release whose version number includes the 2.351.0 from the version
-of aws-sdk-js.
-
-We also include the revision of our generator in the version. For example,
-`com.cognitect.aws/dynamo-db-653.2.351.0` indicates revision `653` of the
-generator, and tag `v2.351.0` of aws-sdk-js.
-
-* See [Versioning](/doc/versioning.md) for more about how we version releases.
+Most AWS SDKs have their own copies of these data descriptions in their github
+repos. We use [aws-sdk-java-2](https://github.com/aws/aws-sdk-java-v2) as the
+source for these, and release separate artifacts for each api.
+
+* See [Versioning](/doc/versioning.md) for more info about how we version releases.
 * See [latest releases](latest-releases.edn) for a list of the latest releases of
 `api`, `endpoints`, and all supported services.
 
@@ -471,7 +454,7 @@ operation requires a custom endpoint, e.g.
 
 This indicates that the data in the `com.cognitect.aws/endpoints` lib
 (which is derived from
-[endpoints.json](https://github.com/aws/aws-sdk-java/blob/master/aws-java-sdk-core/src/main/resources/com/amazonaws/partitions/endpoints.json))
+[endpoints.json](https://github.com/aws/aws-sdk-java-v2/blob/master/core/regions/src/main/resources/software/amazon/awssdk/regions/internal/region/endpoints.json))
 does not support the `:api`/`:region` combination you are trying to
 access.
 

doc/versioning.md

@@ -2,19 +2,48 @@
 
 To use aws-api in your application, you depend on
 `com.cognitect.aws/api`, `com.cognitect.aws/endpoints` and
-`com.cognitect.aws/<service>`, e.g. `com.cognitect.aws/s3`. `api`, `endpoints`,
-and `<service>`
-all have different release schedules and version semantics.
+`com.cognitect.aws/<service>`, e.g. `com.cognitect.aws/s3`.
 
 ## Version semantics
 
+The artifacts fall into one of three categories:
+
+* `com.cognitect.aws/api`
+* `com.cognitect.aws/endpoints`
+* each `<service>` artifact (of which there are hundreds)
+
+Each of these three categories of artifacts are capable of being released and versioned independently.
+
 ## com.cognitect.aws/api
 
     0.8.[rev]
 
 `rev` is the revision in the git repository.
 
-## com.cognitect.aws/endpoints
+## endpoints and services
+
+    [major-rev].[source.repo.tag]
+
+Starting with release `871.2.29.35`, `endpoints` and service artifact versions are being prefixed
+with the same major version prefix, currently `871`. The `2.29.35` is a tag in the AWS SDK source
+repository, currently [AWS SDK Java v2](https://github.com/aws/aws-sdk-java-v2), from which both
+`endpoints` and service artifacts are generated.
+
+These artifacts are released independently and sporadically, only as needed - a given artifact will
+only see a new release if the current tag of the AWS SDK source repository contains changes for that
+artifact, or if a new service descriptor is discovered which has not yet been released.
+
+Prior to release `871.2.29.35`, `endpoints` and services had different major version prefixes. This
+is because they were generated from different, older AWS SDK repositories: [aws-sdk-java
+v1](https://github.com/aws/aws-sdk-java) and [aws-sdk-js](https://github.com/aws/aws-sdk-js),
+respectively. With the deprecation of both of those repositories, it was necessary to migrate to the
+current Java v2 repository and to release new versions of all artifacts, which was release
+`871.2.29.35`.
+
+The following two sections provides more detail and historical context about `endpoints` and service
+artifacts, respectively.
+
+### com.cognitect.aws/endpoints
 
     [source-rev].[source.repo.tag]
 
@@ -37,20 +66,23 @@ we'll add a row to this table.
 
     [generator-rev].[source.repo.tag]
 
-`generator-rev` indicates the revision of the code we use to
-generate all the services and `source.repo.tag` is the tag in the aws
+`source-rev` indicates which aws-sdk (or other resource) we are
+generating from and `source.repo.tag` is the tag in the aws
 repository from which we are generating service libraries. For example,
 `s3-631.2.347.0` is sourced from [aws-sdk-js](https://github.com/aws/aws-sdk-js),
 per the table below, at the `2.347.0` tag.
 
-| generator-rev | source repository | example source tag | example service version |
-|---------------|-------------------|--------------------|-------------------------|
-| >= 871        | aws-sdk-java-v2   | 2.29.35            | 871.2.29.35             |
-| >= 631        | aws-sdk-js        | v2.347.0           | 631.2.347.0             |
+| source-rev | source repository | example source tag | example service version |
+|------------|-------------------|--------------------|-------------------------|
+| >= 871     | aws-sdk-java-v2   | 2.29.35            | 871.2.29.35             |
+| >= 631     | aws-sdk-js        | v2.347.0           | 631.2.347.0             |
 
 If we start to source services from a different repository,
 we'll add a row to this table.
 
+Prior to source revision `871`, the source revision was linked to the version of the code used to
+generate the service artifacts. Beginning with revision `871` that is no longer true.
+
 ## Release schedules
 
 ### com.cognitect.aws/api
@@ -64,24 +96,42 @@ their release schedules.
 ### com.cognitect.aws/endpoints
 
 This is generated from source data in an aws github repository, so we
-periodically check to see if there are any new release tags in the
-source repo. When there are, we diff the endpoint resource at the
-first new tag with the last released tag. If there is a diff, we
-cut a release at that new tag, and then do the same thing, comparing
-the next new tag to the last released tag, until we run out of new
-tags.
-
-For example, if the last version we released is `1.1.11.444`, we look
-to see if there are any tags newer than `1.11.444` in the source
-repo. If we find e.g. `1.11.445`, `1.11.446`, and `1.11.447`, we'll
-diff `1.11.445` against `1.11.444`. If the endpoints resource changed,
-we'll cut a `1.1.11.445` release and then continue, comparing
-`1.1.446` to `1.1.445`, and so on.
-
-If there is no difference, we move to the next tag (`1.1.446`) and
-compare it to our last-released basis (`1.1.444`), and so on.
+periodically check to see if there are any changes to the endpoints resource at
+the repository's latest tag, since the last released tag. If there is a diff, we
+cut a release at that new tag.
+
+For example, if the last version we released is `871.1.11.444`, and the
+repository's most recent tag is `1.11.678`, we'll diff `1.11.445` against
+`1.11.678`. If the endpoints resource changed, we'll cut a `871.1.11.678`
+release.
 
 ### com.cognitect.aws/&lt;service>
 
 This works just like `endpoints`, above, except that we do the same
 thing for every aws service available in the source repository.
+
+
+## Service API Versions
+
+The api descriptors include the AWS `api-version` in their `:metadata`. If, for
+any given service, the AWS SDK repository contains descriptors for more than one
+api version, only the most recent api version will have an artifact generated
+and released.
+
+As of this writing, the [AWS SDK Java
+v2](https://github.com/aws/aws-sdk-java-v2) repository has no such examples of
+this. But if and when it does, this policy will continue to hold true.
+
+For example: historically, services were generated from the
+[aws-sdk-js](https://github.com/aws/aws-sdk-js) repository, which contains at
+least one instance of this situation. In that repository you'll find both of
+the following files:
+
+    dynamodb-2011-12-05.normal.json
+    dynamodb-2012-08-10.normal.json
+
+In that case, whenever we released `com.cognitect.aws/dynamodb`, we look for the
+descriptor with the most recent API version. If `aws-sdk-js-v2.351.0` contains
+an update to `dynamodb-2012-08-10.normal.json`, or a new dynamodb descriptor
+with a more recent api-version, we'd make a release whose version number
+includes the `2.351.0` from the version of aws-sdk-js.

src/cognitect/aws/signers.clj

@@ -81,7 +81,7 @@
       (.isEmpty encoded-path)
       "/"
 
-      ;; https://github.com/aws/aws-sdk-java/blob/fd409de/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AbstractAWSSigner.java#L392-L397
+      ;; https://github.com/aws/aws-sdk-java-v2/blob/61d16e0/core/auth/src/main/java/software/amazon/awssdk/auth/signer/internal/AbstractAws4Signer.java#L546-L555
       ;; Normalization can leave a trailing slash at the end of the resource path,
       ;; even if the input path doesn't end with one. Example input: /foo/bar/.
       ;; Remove the trailing slash if the input path doesn't end with one.

src/cognitect/aws/util.clj

@@ -207,7 +207,7 @@
   [^String s]
   (-> s
       (URLEncoder/encode "UTF-8")
-      ;; https://github.com/aws/aws-sdk-java/blob/fd409de/aws-java-sdk-core/src/main/java/com/amazonaws/util/SdkHttpUtils.java#L77-L91
+      ;; https://github.com/aws/aws-sdk-java-v2/blob/61d16e0/utils/src/main/java/software/amazon/awssdk/utils/http/SdkHttpUtils.java#L170-L176
       (.replace "+" "%20")
       (.replace "*" "%2A")))
 

test/src/cognitect/aws/client/shared_test.clj

@@ -9,11 +9,10 @@
   (testing "Creating a client with custom attributes does not deref the default delayed shared resources."
     ;; As a precondition of this test, the delays must be unrealized.
     (require 'cognitect.aws.client.shared :reload)
-    (do
-      (aws/client {:api :s3
-                   :http-client (reify http/HttpClient)
-                   :region-provider :test-provider
-                   :credentials-provider :test-provider})
-      (is (not (realized? @#'shared/shared-http-client)))
-      (is (not (realized? @#'shared/shared-region-provider)))
-      (is (not (realized? @#'shared/shared-credentials-provider))))))
+    (aws/client {:api :s3
+                 :http-client (reify http/HttpClient)
+                 :region-provider :test-provider
+                 :credentials-provider :test-provider})
+    (is (not (realized? @#'shared/shared-http-client)))
+    (is (not (realized? @#'shared/shared-region-provider)))
+    (is (not (realized? @#'shared/shared-credentials-provider)))))

test/src/cognitect/aws/generators.clj

@@ -25,10 +25,10 @@
             query-ks (gen/vector (gen/such-that seq gen/string-alphanumeric))
             query-vs (gen/vector (gen/such-that seq gen/string-alphanumeric) (count query-ks))
             method (gen/elements [:get :post])
-            ;; https://github.com/aws/aws-sdk-java/blob/d35b018/aws-java-sdk-core/src/main/java/com/amazonaws/auth/internal/SignerKey.java#L30-L34
-            ;; date must be >1 day past epoch and <= today
-            ;; 1668999574880 == 2022-11-20
-            epoch (gen/large-integer* {:min 86400000 :max 1668999574880})
+            ;; https://github.com/aws/aws-sdk-java-v2/blob/61d16e0/core/auth/src/main/java/software/amazon/awssdk/auth/signer/internal/SignerKey.java#L44
+            ;; date will be >1 day past epoch and <= today
+            ;; 1735838904634 == 2025-01-02
+            epoch (gen/large-integer* {:min 86400000 :max 1735838904634})
             body gen/string]
     {:request-method method
      :body           (.getBytes ^String body "UTF-8")