diff --git a/docs/source/guide/admin_settings.md b/docs/source/guide/admin_settings.md index 26d3764440b0..ce360b75bdd4 100644 --- a/docs/source/guide/admin_settings.md +++ b/docs/source/guide/admin_settings.md @@ -16,6 +16,6 @@ There are several places where you can configure org-wide settings: | ------------- | ------------ | ------------ | | [**Usage & License**](admin_usage) | Owner
Admin | This page has a mix of settings that can be set by Administrators (enabling email notifications) and Owners (enabling AI, enabling storage proxy, enabling early adopter features). | | [**Access Token**](access_tokens) | Owner
Admin | Control which types of access tokens are available in the organization. | -| [**Model Providers**](model_providers) | Owner
Admin | Set up model providers that can be used with [Prompts](prompts_overview) and [Chat](/tags/chat). | +| [**Model Providers**](model_providers) | Owner
Admin | Set up model providers that can be used with [Prompts](prompts_overview) and [Chat](/tags/chat.html). | | [**Permissions**](admin_permissions) | Owner | Customize certain permissions for roles. | | [**Support reports**](support_reports) | Owner
Admin | Generate anonymized operational reports that help HumanSignal support understand your deployment, diagnose issues, and recommend workflow and performance improvements. | \ No newline at end of file diff --git a/docs/source/guide/model_providers.md b/docs/source/guide/model_providers.md index 14d3fb66dc3e..7c42f5f19e77 100644 --- a/docs/source/guide/model_providers.md +++ b/docs/source/guide/model_providers.md @@ -15,7 +15,7 @@ date: 2025-02-18 12:03:59 To use certain AI features across your organization, you must first set up a model provider. You can set up model providers from **Organization > Settings**. -For example, if you want to interact with an LLM when using the [`` tag](/tags/chat), you will first need to configure access to the model. +For example, if you want to interact with an LLM when using the [`` tag](/tags/chat.html), you will first need to configure access to the model. !!! note diff --git a/docs/source/guide/project_settings_lse.md b/docs/source/guide/project_settings_lse.md index 541f38378fd7..61241f5429cd 100644 --- a/docs/source/guide/project_settings_lse.md +++ b/docs/source/guide/project_settings_lse.md @@ -559,7 +559,7 @@ By default, each task only needs to be annotated by one annotator. If you want m The number of distinct annotations you want to allow per task. -Note that in certain situations, this may be exceeded. For example, if there are long-standing drafts within a project or you have a very low [task reservation](#task-lock) time. +Note that in certain situations, this may be exceeded. For example, if there are long-standing drafts within a project or you have a very low [task reservation](#lock-tasks) time. Also note that only annotations created by distinct users count towards the overlap. For example, if the overlap is `2` and a user creates and submits two annotations on a single task (which can be done in Quick View), the overlap threshold will not be reached until another user submits an annotation. @@ -642,7 +642,12 @@ For more information about pausing annotators, including how to manually pause s
-Evaluate annotators against [ground truths](ground_truths) within a project. A “ground truth” annotation is a verified, high-quality annotation that serves as the correct answer for a specific task. +!!! note + Annotator Evaluation settings are only available when the project is configured to [automatically assign tasks](#distribute-tasks). If you are using Manual distribution, this section will not appear in your project settings. + + If you switch a project from Automatic to Manual distribution, annotator evaluation is automatically disabled. + +Evaluate annotators against [ground truths](ground_truths) within a project. A "ground truth" annotation is a verified, high-quality annotation that serves as the correct answer for a specific task. When enabled, this setting looks at the agreement score for the annotator when compared solely against ground truth annotations. You can decide to automatically pause an annotator within the project if their ground truth agreement score falls below a certain threshold. @@ -681,7 +686,7 @@ When annotators enter the labeling stream, they are first presented with tasks t Use the counter to determine how many ground truth tasks should be presented first before the annotator progresses through the remaining project tasks. -**Note:** This option is only active when the project is configured to [automatically assign tasks](#distribute-tasks). If you are using Manual distribution, annotators will see tasks ordered by ID number. If you would like them to see ground truth tasks first, you should add ground truth annotations in the same order. +Set this counter to zero if you want to skip onboarding and only use continuous evaluation. diff --git a/docs/source/guide/release_notes/onprem/2.29.0.md b/docs/source/guide/release_notes/onprem/2.29.0.md index 6ffb1bcebe78..241efcd200f3 100644 --- a/docs/source/guide/release_notes/onprem/2.29.0.md +++ b/docs/source/guide/release_notes/onprem/2.29.0.md @@ -18,7 +18,7 @@ Chat conversations are now a native data type in Label Studio, so you can annota For more information, see: -[**Chat tag page**](/tags/chat) +[**Chat tag page**](/tags/chat.html) [**Chat labeling templates**](/templates/gallery_chat) diff --git a/docs/source/guide/troubleshooting.md b/docs/source/guide/troubleshooting.md index d54635c7d4f9..6a4d4bd9e508 100644 --- a/docs/source/guide/troubleshooting.md +++ b/docs/source/guide/troubleshooting.md @@ -38,7 +38,7 @@ To resolve this issue, update the host specified as an environment variable or w * If you want to upload a large volume of data (thousands of items), consider doing that at a time when people are not labeling or use a different database backend such as PostgreSQL or Redis. You can run Docker Compose from the root directory of Label Studio to use PostgreSQL: `docker-compose up -d`, or see [Sync data from cloud or database storage](storage). -* If you are using a labeling schema that has many thousands of labels, consider using an [external taxonomy](/tags/taxonomy) instead. +* If you are using a labeling schema that has many thousands of labels, consider using an [external taxonomy](/tags/taxonomy.html) instead. ### Image/audio/resource loading error while labeling diff --git a/docs/source/templates/chat_eval.md b/docs/source/templates/chat_eval.md index 3b6c8da657a3..853dbb484d50 100644 --- a/docs/source/templates/chat_eval.md +++ b/docs/source/templates/chat_eval.md @@ -97,7 +97,7 @@ The `Chat` tag provides an interface where the annotator can type and send messa You can customize this parameter with `user`, `assistant`, `system`, `tool`, and `developer`. -For more information and additional parameters, see the [Chat tag](/tags/chat). +For more information and additional parameters, see the [Chat tag](/tags/chat.html). ## Input data @@ -146,11 +146,11 @@ You can also import demo chat messages as follows: !!! attention The chat messages that you import are not selectable. This means that you cannot edit them or apply annotations (ratings, choices, etc) to them. - You can only select and annotate messages that are added to the chat by an annotator or that are imported as [predictions](/tags/chat#Prediction-format). + You can only select and annotate messages that are added to the chat by an annotator or that are imported as [predictions](/tags/chat.html#Prediction-format). ## Related tags -* [Chat](/tags/chat) +* [Chat](/tags/chat.html) * [Style](/tags/style) * [View](/tags/view) * [Text](/tags/text) \ No newline at end of file diff --git a/docs/source/templates/chat_llm_eval.md b/docs/source/templates/chat_llm_eval.md index 6f98371d1737..ad5aec6a7414 100644 --- a/docs/source/templates/chat_llm_eval.md +++ b/docs/source/templates/chat_llm_eval.md @@ -16,7 +16,7 @@ While this template focuses on conversation-level evaluation, you can modify it !!! error Enterprise This template requires Label Studio Enterprise. - Starter Cloud users can use the `Chat` tag, but have limited access to LLM integration. Instead, you can conduct a manual chat or import messages as predictions. See the [Chat tag documentation](/tags/chat#Prediction-format). + Starter Cloud users can use the `Chat` tag, but have limited access to LLM integration. Instead, you can conduct a manual chat or import messages as predictions. See the [Chat tag documentation](/tags/chat.html#Prediction-format). For Community users, see our [Conversation AI templates](gallery_conversational_ai) or the [Multi-Turn Chat Evaluation template](multi_turn_chat). @@ -163,7 +163,7 @@ The `Chat` tag provides an interface where the annotator can type and send messa * `value`: This is required, and should use a variable referencing your [input data](#Input-data). In this example, we use `$chat` because the input JSON uses `"chat"`. -* `llm`: Messages from the annotator will be sent to an LLM and the response returned within the chat area of the labeling configuration. For more information, see [Chat tag - Use with an LLM](/tags/chat#Use-with-an-LLM). +* `llm`: Messages from the annotator will be sent to an LLM and the response returned within the chat area of the labeling configuration. For more information, see [Chat tag - Use with an LLM](/tags/chat.html#Use-with-an-LLM). * `minMessages`: The minimum number of messages users must submit to complete the task. You can also set a maximum. @@ -171,7 +171,7 @@ The `Chat` tag provides an interface where the annotator can type and send messa * `editable`: Messages from the annotator and from the LLM are editable. To modify this so that only messages from certain roles are editable, you can specify them (for example, `editable="user,assistant"`). -For more information and additional parameters, see the [Chat tag](/tags/chat). +For more information and additional parameters, see the [Chat tag](/tags/chat.html). ### Choices and TextArea @@ -224,11 +224,11 @@ You can also import demo chat messages as follows: !!! attention The chat messages that you import are not selectable. This means that you cannot edit them or apply annotations (ratings, choices, etc) to them. - You can only select and annotate messages that are added to the chat by an annotator or that are imported as [predictions](/tags/chat#Prediction-format). + You can only select and annotate messages that are added to the chat by an annotator or that are imported as [predictions](/tags/chat.html#Prediction-format). ## Related tags -* [Chat](/tags/chat) +* [Chat](/tags/chat.html) * [Style](/tags/style) * [Text](/tags/text) * [View](/tags/view) diff --git a/docs/source/templates/chat_red_team.md b/docs/source/templates/chat_red_team.md index 8da09079540f..80093603ab10 100644 --- a/docs/source/templates/chat_red_team.md +++ b/docs/source/templates/chat_red_team.md @@ -14,7 +14,7 @@ Stress‑test your GenAI agent with structured red‑teaming. Use this template !!! error Enterprise This template requires Label Studio Enterprise. - Starter Cloud users can use the `Chat` tag, but have limited access to LLM integration. Instead, you can conduct a manual chat or import messages as predictions. See the [Chat tag documentation](/tags/chat#Prediction-format). + Starter Cloud users can use the `Chat` tag, but have limited access to LLM integration. Instead, you can conduct a manual chat or import messages as predictions. See the [Chat tag documentation](/tags/chat.html#Prediction-format). For Community users, see our [Conversation AI templates](gallery_conversational_ai) or the [Multi-Turn Chat Evaluation template](multi_turn_chat). @@ -175,7 +175,7 @@ The `Chat` tag provides an interface where the annotator can type and send messa * `value`: This is required, and should use a variable referencing your [input data](#Input-data). In this example, we use `$chat` because the input JSON uses `"chat"`. -* `llm`: Messages from the annotator will be sent to an LLM and the response returned within the chat area of the labeling configuration. For more information, see [Chat tag - Use with an LLM](/tags/chat#Use-with-an-LLM). +* `llm`: Messages from the annotator will be sent to an LLM and the response returned within the chat area of the labeling configuration. For more information, see [Chat tag - Use with an LLM](/tags/chat.html#Use-with-an-LLM). * `minMessages`: The minimum number of messages users must submit to complete the task. You can also set a maximum. @@ -183,7 +183,7 @@ The `Chat` tag provides an interface where the annotator can type and send messa * `editable`: In this example, you are not allowing the annotator to edit messages. You can set this to `true` or modify it so that only messages from certain roles are editable (for example, `editable="user,assistant"`). -For more information and additional parameters, see the [Chat tag](/tags/chat). +For more information and additional parameters, see the [Chat tag](/tags/chat.html). ### Per-message evaluation @@ -263,11 +263,11 @@ You can also import demo chat messages as follows: !!! attention The chat messages that you import are not selectable. This means that you cannot edit them or apply annotations (ratings, choices, etc) to them. - You can only select and annotate messages that are added to the chat by an annotator or that are imported as [predictions](/tags/chat#Prediction-format). + You can only select and annotate messages that are added to the chat by an annotator or that are imported as [predictions](/tags/chat.html#Prediction-format). ## Related tags -* [Chat](/tags/chat) +* [Chat](/tags/chat.html) * [Style](/tags/style) * [View](/tags/view) * [Text](/tags/text) diff --git a/docs/source/templates/chat_rlhf.md b/docs/source/templates/chat_rlhf.md index 1d0f3ae51b49..4cebf868e758 100644 --- a/docs/source/templates/chat_rlhf.md +++ b/docs/source/templates/chat_rlhf.md @@ -195,10 +195,10 @@ The `Chat` tag provides an interface where the annotator can type and send messa * `editable`: In this example, you are not allowing the annotator to edit messages. -For more information and additional parameters, see the [Chat tag](/tags/chat). +For more information and additional parameters, see the [Chat tag](/tags/chat.html). !!! note - This template is designed to be used to evaluate an imported conversation, so you will likely want to import messages from an external source as [predictions](/tags/chat#Prediction-format). + This template is designed to be used to evaluate an imported conversation, so you will likely want to import messages from an external source as [predictions](/tags/chat.html#Prediction-format). ### Conversation evaluation @@ -296,7 +296,7 @@ You can also import demo chat messages as follows: ### Predictions -If you want to be able to select messages and evaluate them, then you can use [predictions](/tags/chat#Prediction-format). For example: +If you want to be able to select messages and evaluate them, then you can use [predictions](/tags/chat.html#Prediction-format). For example: ```json [ @@ -346,7 +346,7 @@ If you want to be able to select messages and evaluate them, then you can use [p ## Related tags -* [Chat](/tags/chat) +* [Chat](/tags/chat.html) * [Style](/tags/style) * [View](/tags/view) * [Text](/tags/text) diff --git a/docs/source/templates/chatbot.md b/docs/source/templates/chatbot.md index 9996d0bf4b0f..9c08d1734199 100644 --- a/docs/source/templates/chatbot.md +++ b/docs/source/templates/chatbot.md @@ -195,10 +195,10 @@ The `Chat` tag provides an interface where the annotator can type and send messa * `editable`: In this example, you are not allowing the annotator to edit messages. -For more information and additional parameters, see the [Chat tag](/tags/chat). +For more information and additional parameters, see the [Chat tag](/tags/chat.html). !!! note - This template is designed to be used to evaluate an imported conversation, so you will likely want to import messages from an external source as [predictions](/tags/chat#Prediction-format). + This template is designed to be used to evaluate an imported conversation, so you will likely want to import messages from an external source as [predictions](/tags/chat.html#Prediction-format). ### Per-message evaluation @@ -275,7 +275,7 @@ You can also import demo chat messages as follows: ### Predictions -If you want to be able to select messages and evaluate them, then you can use [predictions](/tags/chat#Prediction-format). For example: +If you want to be able to select messages and evaluate them, then you can use [predictions](/tags/chat.html#Prediction-format). For example: ```json @@ -326,7 +326,7 @@ If you want to be able to select messages and evaluate them, then you can use [p ## Related tags -* [Chat](/tags/chat) +* [Chat](/tags/chat.html) * [Style](/tags/style) * [Text](/tags/text) * [View](/tags/view)