Skip to content

Build Process

Jump to bottom
Jos Denys edited this page Oct 28, 2022 · 24 revisions

This page provides additional detail on the tasks and artefacts involved with building the iKnow engine.

The C++ Source Code

See here for an introduction on the important sections of the iKnow source code. 2 third-party modules are used to support the engine :

  • ICU (header files and libraries) for utf8-encoding and regular expressions.
  • JSON (one header file) for JSON parsing and serialization.

Building on Windows

Step 1: Setting up project dependencies

  1. Download the Win64 binaries for a recent release of the ICU library (e.g. version 69.1) and unzip to <repo_root>/thirdparty/icu (or a local folder of your choice).

  2. Download a recent release of the JSON for Modern C++ module (e.g. https://github.com/nlohmann/json/releases/download/v3.10.4/include.zip) and extract it into the <repo_root>/thirdparty/json directory.

  3. If you chose a different folder for your ICU libraries, update <repo_root>\modules\Dependencies.props to represent your local configuration. This is how it looks after download, which should be OK if you used the suggested directory paths:

  <PropertyGroup Label="UserMacros">
    <ICUDIR>$(SolutionDir)..\thirdparty\icu\</ICUDIR>
    <ICU_INCLUDE>$(ICUDIR)\include</ICU_INCLUDE>
    <ICU_LIB>$(ICUDIR)\lib64</ICU_LIB>
    <JSON_INCLUDE>$(SolutionDir)..\thirdparty\json\single_include\</JSON_INCLUDE>
  </PropertyGroup>

Step 2: Building the modules

  1. Open the Solution file <repo_root>\modules\iKnowEngine.sln in Visual Studio. We used Visual Studio Community 2019.

  2. In the Solution Explorer, choose "iKnowEngineTest" as "Set up as startup project"

  3. In Solution Configurations, choose either "Debug|x64", or "Release|x64", depending on the kind of executable you prefer.

  4. Build the solution, it will build all 31 projects.

Step 3: Testing the indexer

Once building has succeeded, you can run the test program, depending on which build config you chose:

  • <repo_root>\kit\x64\Debug\bin\iKnowEngineTest.exe
  • <repo_root>\kit\x64\Release\bin\iKnowEngineTest.exe

⚠️ Note that you'll have to add the $(ICUDIR)/bin64 directory to your PATH or copy its .dll files to this test folder in order to run the test executable.

Alternatively, you can also start a debugging session in Visual Studio and walk through the code to inspect it.

The iKnow indexing demo program will index one sentence for each of the 11 languages, and write out the sentence boundaries. That's of course not very spectacular by itself, but future iterations of this demo program will expose more of the entity and context information iKnow detects.

On Linux / Unix

Step 1: Setting up project dependencies

  1. Download the proper binaries for a recent release of the ICU library (e.g. version 69.1) and untar to <repo_root>/thirdparty/icu (or a local folder of your choice).

  2. Save the path you untarred the archive to a ICUDIR environment variable. Note that your ICU download may have a relative path inside the tar archive, so you may need to use --strip-components=4 or manually reorganise to make sure the ${ICUDIR}/include leads where you'd expect it to lead.

  3. Download a recent release of the JSON for Modern C++ module (e.g. https://github.com/nlohmann/json/releases/download/v3.10.4/include.zip) and extract it into the <repo_root>/thirdparty/json directory.

  4. Set the IKNOWPLAT environment variable to the target platform of your choice: e.g. "lnxubuntux64", "lnxrhx64" or "macx64", if you want to build for the M1 platform (MacOS ARM64), choose "macosuniversal".

  5. In the <repo_root> folder, run

     export ICUDIR=$HOME/iknow/thirdparty/icu
 export JSON_INCLUDE=$HOME/iknow/thirdparty/json/single_include
 export IKNOWPLAT=macosuniversal
 export DYLD_LIBRARY_PATH=$HOME/iknow/kit/macos/release/bin:$ICUDIR/lib:$DYLD_LIBRARY_PATH

 make all

The Python Interface

We use GitHub Actions to build iknowpy automatically and deploy it to PyPI (the Python Package Index). To trigger a new build and deployment, simply increment the version in <repo_root>/modules/iknowpy/iknowpy/version.py in the master branch and push the associated commit.

Alternatively, you can go to https://github.com/intersystems/iknow/actions/workflows/CI.yml, click the Run workflow button to the right, check the Force deployment box, and click Run workflow. This deployment option is provided as a backup in case deployment failed when you incremented the version number, and you do not want to increment the version number a second time to trigger a second deployment run.

To test the deployment tools and processes without affecting the real PyPI, you can instead edit the number in version.py so that it ends in .devN, where N is a development build number. Examples are 0.0.12.dev0 and 1.2.3.dev456. When you push a change to version.py that contains a development build number, iknowpy is automatically built and deployed to TestPyPI, an index separate from PyPI. Once the deployment of the development build is complete, you can install it with pip.

pip install --index-url https://test.pypi.org/simple/ -U iknowpy

Note: If you are pushing multiple commits at once, the edit to version.py must occur in the final commit to trigger automatic deployment. To avoid accidentally changing version.py without triggering deployment, it is recommended that you have a dedicated push that changes only version.py in the master branch.

If you want to build the Python interface locally, read on. The following directions refer to the commands pip and python. On some platforms, these commands use Python 2 by default, in which case you should execute pip3 and python3 instead to ensure that you are using Python 3.

Step 1: Build the iKnow engine

Build the iKnow engine following the above directions. If you are on Windows, choose the "Release|x64" configuration.

Step 2: Setting up Python dependencies

  1. Install Python ≥3.6. Ensure that the installation includes Python header files.

  2. Install Cython, setuptools, and wheel. You can do this by having a Python distribution that already includes these modules or by running

    pip install -U cython setuptools wheel

    On Windows, you also need pefile and machomachomangler.

    pip install -U pefile machomachomangler

  3. If you are on Mac OS, ensure that the otool and install_name_path command-line tools are present. They should be available if you have XCode installed with command-line developer tools. If you are on Linux, ensure that the patchelf tool is present and has version ≥0.9. You can install it using the package manager on your machine, or you can build it from source at https://github.com/NixOS/patchelf/releases.

Step 3: Building and installing the iknowpy module

Open a command shell in the directory <repo_root>/modules/iknowpy and execute the setup script. This builds iknowpy, creates a package containing iknowpy and its dependencies, and installs the package.

python setup.py install

Step 4: Testing the iknowpy module

The scripts at https://github.com/intersystems/iknow/tree/master/modules/iknowpy/tests provide example of how to use iknowpy. Run the scripts to call a few iKnow functions from Python and print their results. See Getting Started for more on the various entry points.

⚠️ If you are testing iknowpy via the Python interactive console, do not do so in the <repo_root>/modules/iknowpy working directory. Because of how Python resolves module names, importing iknowpy will cause Python to try importing the source package <repo_root>/modules/iknowpy/iknowpy instead of the installed package, resulting in an import error.

Step 5: (Optional) Building a Wheel

A wheel is a pre-built package that can can be distributed to others and can be installed using pip. Typically, a single wheel is specific to the build platform and the minor version of Python (e.g. 3.7 or 3.8) used to build the wheel. Thus, a wheel must be built for every platform and minor Python version for which a simple installation using pip is desired.

On Windows

  1. Open a command shell in the directory <repo_root>/modules/iknowpy.

  2. Build the wheel.

    python setup.py bdist_wheel

On Mac OS

Decide the minimum version of Mac OS that the wheel will support. Ensure that the iKnow engine, ICU, and Python were built with support for this version. Python distributions from https://www.python.org/downloads/mac-osx are the best for this situation, as they tend to be the distributions that are maximally compatible with different Mac OS versions. The following directions assume a minimum target version of Mac OS X 10.9, but you can adapt them to suit your preferences.

  1. Open a command shell in the directory <repo_root>/modules/iknowpy.

  2. Build the wheel. If you do not specify MACOSX_DEPLOYMENT_TARGET or --plat-name, then the minimum supported Mac OS version defaults to that of the Python distribution used to build the wheel.

    export MACOSX_DEPLOYMENT_TARGET=10.9
python setup.py bdist_wheel --plat-name=macosx-10.9-x86_64

On Linux

In general, binaries built on one Linux distribution are not directly runnable on another Linux distribution. If all you want is a wheel that is compatible with the build platform and its binary compatible platforms, simply execute the following in the directory <repo_root>/modules/iknowpy.

python setup.py bdist_wheel

To build a wheel that is compatible with the vast majority of modern Linux distributions, you can use the manylinux project, which is the standard way to distribute pre-built Python extension modules on Linux. If you want to build a manylinux wheel locally, see <repo_root>/.github/workflows/build_manylinux.sh to see how our continuous deployment pipeline does it. The script starts up a manylinux Docker container for the desired CPU architecture. Inside the container, the script builds ICU, the iKnow engine, and the Python interface.

Clone this wiki locally