api-entities/concepts/filter-concepts.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "[![GitHub Repository](https://img.shields.io/badge/GitHub-Repository-181717?style=for-the-badge&logo=GitHub&link=https://github.com/Mearman/openalex-docs/tree/main)](https://github.com/Mearman/openalex-docs/tree/main)[![Open in GitHub](https://img.shields.io/badge/Open%20in-GitHub-181717?style=for-the-badge&logo=github&link=https://github.com/Mearman/openalex-docs/blob/main/api-entities/concepts/filter-concepts.ipynb)](https://github.com/Mearman/openalex-docs/blob/main/api-entities/concepts/filter-concepts.ipynb)[![Open in Colab](https://img.shields.io/badge/Open%20in-Colab-F9AB00?style=for-the-badge&logo=Google%20Colab&link=https://colab.research.google.com/github/Mearman/openalex-docs/blob/main/api-entities/concepts/filter-concepts.ipynb)](https://colab.research.google.com/github/Mearman/openalex-docs/blob/main/api-entities/concepts/filter-concepts.ipynb)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "%pip install --upgrade \"git+https://github.com/Mearman/openalex-python-pydantic-v1.git\"\n",
        "%pip install pandasai"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "import json\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from openalex_api import Configuration, ApiClient, AutocompleteApi, AuthorsApi, ConceptsApi, FundersApi, InstitutionsApi, PublishersApi, SourcesApi, WorksApi\n",
        "\n",
        "configuration = Configuration(host=\"https://api.openalex.org\")\n",
        "autocomplete_api = AutocompleteApi(ApiClient(configuration))\n",
        "authors_api = AuthorsApi(ApiClient(configuration))\n",
        "concepts_api = ConceptsApi(ApiClient(configuration))\n",
        "funders_api = FundersApi(ApiClient(configuration))\n",
        "institutions_api = InstitutionsApi(ApiClient(configuration))\n",
        "publishers_api = PublishersApi(ApiClient(configuration))\n",
        "sources_api = SourcesApi(ApiClient(configuration))\n",
        "works_api = WorksApi(ApiClient(configuration))\n",
        "\n",
        "from pandasai import SmartDataframe\n",
        "from pandasai.llm import OpenAI"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title  { run: \"auto\", display-mode: \"form\" }\n",
        "openapi_token = \"\" # @param {type:\"string\"}"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Filter concepts\n",
        "\n",
        "You can filter concepts with the `filter` parameter:\n",
        "\n",
        "* Get concepts that are at level 0 (top level)\n",
        "  [`https://api.openalex.org/concepts?filter=level:0`](https://api.openalex.org/concepts?filter=level:0)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/concepts?filter=level:0\n",
        "filter=\"level:0\" # @param {type: \"string\"}\n",
        "\n",
        "response = concepts_api.get_concepts(\n",
        "\tfilter=filter\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "{% hint style=\"info\" %}\n",
        "It's best to [read about filters](./../../how-to-use-the-api/get-lists-of-entities/filter-entity-lists.ipynb) before trying these out. It will show you how to combine filters and build an AND, OR, or negation query\n",
        "{% endhint %}\n",
        "\n",
        "### `/concepts` attribute filters\n",
        "\n",
        "You can filter using these attributes of the [`Concept`](./concept-object.ipynb) object (click each one to view their documentation on the `Concept` object page):\n",
        "\n",
        "* [`ancestors.id`](./concept-object.md#ancestors)\n",
        "* [`cited_by_count`](./concept-object.md#cited_by_count)\n",
        "* [`ids.openalex`](./concept-object.md#ids) (alias: `openalex`)\n",
        "* [`level`](./concept-object.md#level)\n",
        "* [`summary_stats.2yr_mean_citedness`](./concept-object.md#summary_stats) (accepts float, null, !null, can use range queries such as < >)\n",
        "* [`summary_stats.h_index`](./concept-object.md#summary_stats) (accepts integer, null, !null, can use range queries)\n",
        "* [`summary_stats.i10_index`](./concept-object.md#summary_stats) (accepts integer, null, !null, can use range queries)\n",
        "* [`works_count`](./concept-object.md#works_count)\n",
        "\n",
        "### `/concepts` convenience filters\n",
        "\n",
        "These filters aren't attributes of the [`Concept`](./concept-object.ipynb) object, but they're included to address some common use cases:\n",
        "\n",
        "#### `default.search`\n",
        "\n",
        "Value: a search string\n",
        "\n",
        "This works the same as using the [`search` parameter](./search-concepts.md#search-concepts) for Concepts.\n",
        "\n",
        "#### `display_name.search`\n",
        "\n",
        "Value: a search string\n",
        "\n",
        "Returns: concepts with a [`display_name`](./concept-object.md#display_name) containing the given string; see the [search page](./search-concepts.md#search-a-specific-field) for details.\n",
        "\n",
        "* Get concepts with `display_name` containing \"electrodynamics\":\n",
        "  [`https://api.openalex.org/concepts?filter=display_name.search:electrodynamics`](https://api.openalex.org/concepts?filter=display_name.search:electrodynamics)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/concepts?filter=display_name.search:electrodynamics\n",
        "filter=\"display_name.search:electrodynamics\" # @param {type: \"string\"}\n",
        "\n",
        "response = concepts_api.get_concepts(\n",
        "\tfilter=filter\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "{% hint style=\"info\" %}\n",
        "In most cases, you should use the [`search` parameter](./search-concepts.md#concepts-full-search) instead of this filter because it uses a better search algorithm.\n",
        "{% endhint %}\n",
        "\n",
        "#### `has_wikidata`\n",
        "\n",
        "Value: a Boolean (`true` or `false`)\n",
        "\n",
        "Returns: concepts that have or lack a [Wikidata ID](./concept-object.md#wikidata), depending on the given value. For now, all concepts in OpenAlex _do_ have Wikidata IDs.\n",
        "\n",
        "* Get concepts without Wikidata IDs:\n",
        "  [`https://api.openalex.org/concepts?filter=has_wikidata:false`](https://api.openalex.org/concepts?filter=has_wikidata:false)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "# @title { run: \"auto\", vertical-output: false }\n",
        "# https://api.openalex.org/concepts?filter=has_wikidata:false\n",
        "filter=\"has_wikidata:false\" # @param {type: \"string\"}\n",
        "\n",
        "response = concepts_api.get_concepts(\n",
        "\tfilter=filter\n",
        ")\n",
        "\n",
        "df = pd.DataFrame(response.results)\n",
        "display(df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "numeric_df = df[['id', 'display_name'] +\n",
        "\t[col for col in df.columns if df[col].dtype in ['int64', 'float64'] and col != 'relevance_score']]\n",
        "display(numeric_df)"
      ],
      "outputs": [],
      "execution_count": null
    },
    {
      "cell_type": "code",
      "metadata": {},
      "source": [
        "try:\n",
        "\tprompt = \"Visualize this data\" # @param {type:\"string\"}\n",
        "\tSmartDataframe(\n",
        "\t\tnumeric_df,\n",
        "\t\tconfig={\"llm\": (OpenAI(api_token=openapi_token))},\n",
        "\t).chat(prompt)\n",
        "except:\n",
        "\tif not openapi_token:\n",
        "\t\tprint(\"Error: openapi_token not set\")\n",
        "\telse:\n",
        "\t\tprint(\"Error when creating SmartDataframe\")"
      ],
      "outputs": [],
      "execution_count": null
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 4
}