Feat: add docs for external models (#873)

* Feat: add docs for external models

* feedback

Author: tobymao

docs/concepts/models/external_models.md

@@ -0,0 +1,58 @@
+# External models
+Some SQLMesh models query "external" tables that were created by code unavailable to SQLMesh. External models are a special model kind used to store metadata about those external tables.
+
+External models allow SQLMesh to provide column-level lineage and data type information for external tables queried with SELECT *.
+
+## Generating external models schema
+External models consist of an external table's schema information stored in the `schema.yaml` file in the SQLMesh project's root directory.
+
+You can add schema information to the file by either (i) writing the YAML by hand or (ii) allowing SQLMesh to query the external table and add the information itself with the create_external_models CLI command.
+
+Consider this example FULL model that queries an external table external_db.external_table:
+
+```sql
+MODEL (
+  name my_db.my_table,
+  kind FULL
+);
+
+SELECT
+  *
+FROM
+  external_db.external_table;
+```
+
+The following sections demonstrate how to create an external model containing metadata about external_db.external_table, which contains columns column_a and column_b.
+
+## Writing YAML by hand
+This example demonstrates how the schema.yaml file should be formatted.
+
+```yaml
+- name: external_db.external_table
+  description: An external table
+  columns:
+    column_a: int
+    column_b: text
+```
+
+All the external models in a SQLMesh project are stored in one schema.yaml file. The file might look like this with an additional external model:
+
+```yaml
+- name: external_db.external_table
+  description: An external table
+  columns:
+    column_a: int
+    column_b: text
+- name: external_db.external_table_2
+  description: Another external table
+  columns:
+    column_c: bool
+    column_d: float
+```
+
+### Using the create_external_models CLI command
+Instead of writing the external model YAML by hand, SQLMesh can create it for you with the [create_external_models](../../../reference/cli#create_external_models) CLI command.
+
+The command locates all external tables queried in your SQLMesh project, executes the queries, and infers the tables' column names and types from the results.
+
+It then writes that information to the schema.yaml file.

docs/concepts/models/model_kinds.md

@@ -8,7 +8,7 @@ Models of the `INCREMENTAL_BY_TIME_RANGE` kind are computed incrementally based
 
 Only missing time intervals are processed during each execution for `INCREMENTAL_BY_TIME_RANGE` models. This is in contrast to the [FULL](#full) model kind, where the entire dataset is recomputed every time the model is executed.
 
-An `INCREMENTAL_BY_TIME_RANGE` model query must contain an expression in its SQL `WHERE` clause that filters the upstream records by time range. SQLMesh provides special macros that represent the start and end of the time range being processed: `@start_date` / `@end_date` and `@start_ds` / `@end_ds`. 
+An `INCREMENTAL_BY_TIME_RANGE` model query must contain an expression in its SQL `WHERE` clause that filters the upstream records by time range. SQLMesh provides special macros that represent the start and end of the time range being processed: `@start_date` / `@end_date` and `@start_ds` / `@end_ds`.
 
 Refer to [Macros](../macros.md#predefined-variables) for more information.
 
@@ -30,7 +30,7 @@ WHERE
 ```
 
 ### Time column
-SQLMesh needs to know which column in the model's output represents the timestamp or date associated with each record. 
+SQLMesh needs to know which column in the model's output represents the timestamp or date associated with each record.
 
 The `time_column` is used to determine which records will be overridden during data [restatement](../plans.md#restatement-plans) and provides a partition key for engines that support partitioning (such as Apache Spark):
 
@@ -85,7 +85,7 @@ WHERE
 ```
 
 ### Idempotency
-It is recommended that queries of models of this kind are [idempotent](../glossary.md#idempotency) to prevent unexpected results during data [restatement](../plans.md#restatement-plans). 
+It is recommended that queries of models of this kind are [idempotent](../glossary.md#idempotency) to prevent unexpected results during data [restatement](../plans.md#restatement-plans).
 
 Note, however, that upstream models and tables can impact a model's idempotency. For example, referencing an upstream model of kind [FULL](#full) in the model query automatically causes the model to be non-idempotent.
 
@@ -167,7 +167,7 @@ Depending on the target engine, models of the `INCREMENTAL_BY_UNIQUE_KEY` kind a
 | DuckDB     | not supported       |
 
 ## FULL
-Models of the `FULL` kind cause the dataset associated with a model to be fully refreshed (rewritten) upon each model evaluation. 
+Models of the `FULL` kind cause the dataset associated with a model to be fully refreshed (rewritten) upon each model evaluation.
 
 The `FULL` model kind is somewhat easier to use than incremental kinds due to the lack of special settings or additional query considerations. This makes it suitable for smaller datasets, where recomputing data from scratch is relatively cheap and doesn't require preservation of processing history. However, using this kind with datasets containing a large volume of records will result in significant runtime and compute costs.
 
@@ -201,7 +201,7 @@ Depending on the target engine, models of the `FULL` kind are materialized using
 | DuckDB     | CREATE OR REPLACE TABLE          |
 
 ## VIEW
-The model kinds described so far cause the output of a model query to be materialized and stored in a physical table. 
+The model kinds described so far cause the output of a model query to be materialized and stored in a physical table.
 
 The `VIEW` kind is different, because no data is actually written during model execution. Instead, a non-materialized view (or "virtual table") is created or replaced based on the model's query.
 
@@ -222,7 +222,7 @@ FROM db.employees;
 ```
 
 ## EMBEDDED
-Embedded models are a way to share common logic between different models of other kinds. 
+Embedded models are a way to share common logic between different models of other kinds.
 
 There are no data assets (tables or views) associated with `EMBEDDED` models in the data warehouse. Instead, an `EMBEDDED` model's query is injected directly into the query of each downstream model that references it.
 
@@ -240,3 +240,6 @@ FROM db.employees;
 
 ## SEED
 The `SEED` model kind is used to specify [seed models](./seed_models.md) for using static CSV datasets in your SQLMesh project.
+
+## EXTERNAL
+The EXTERNAL model kind is used to specify [external models](./external_models.md) that store metadata about external tables. External models are special; they are not specified in .sql files like the other model kinds. They are optional but useful for propagating column and type information for external tables queried in your SQLMesh project.

docs/reference/cli.md

@@ -180,3 +180,10 @@ Usage: sqlmesh migrate
 
   Please contact your SQLMesh administrator before doing this.
 ```
+
+## create_external_models
+```
+Usage: sqlmesh create_external_models [OPTIONS]
+
+  Create a schema file containing external model schemas.
+```