tutorials/train_test.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob, os, subprocess\n",
    "from IPython.display import IFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Precursors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train a model, you first need to convert your sequences and targets into the input HDF5 format. Check out my tutorials for how to do that; they're linked from the [main page](../README.md).\n",
    "\n",
    "For this tutorial, grab a small example HDF5 that I constructed here with 10% of the training sequences and only GM12878 targets for various DNase-seq, ChIP-seq, and CAGE experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 71.2M  100 71.2M    0     0  16.1M      0  0:00:04  0:00:04 --:--:-- 17.3M\n",
      "x heart_l131k/\n",
      "x heart_l131k/contigs.bed\n",
      "x heart_l131k/statistics.json\n",
      "x heart_l131k/targets.txt\n",
      "x heart_l131k/sequences.bed\n",
      "x heart_l131k/tfrecords/\n",
      "x heart_l131k/tfrecords/train-0.tfr\n",
      "x heart_l131k/tfrecords/train-1.tfr\n",
      "x heart_l131k/tfrecords/train-3.tfr\n",
      "x heart_l131k/tfrecords/train-2.tfr\n",
      "x heart_l131k/tfrecords/train-5.tfr\n",
      "x heart_l131k/tfrecords/test-0.tfr\n",
      "x heart_l131k/tfrecords/train-4.tfr\n",
      "x heart_l131k/tfrecords/valid-0.tfr\n"
     ]
    }
   ],
   "source": [
    "if len(glob.glob('data/heart_l131k/tfrecords/*.tfr')) == 0:\n",
    "    subprocess.call('curl -o data/heart_l131k.tgz https://storage.googleapis.com/basenji_tutorial_data/heart_l131k.tgz', shell=True)\n",
    "    subprocess.call('tar -xzvf data/heart_l131k.tgz', shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, you need to decide what sort of architecture to use. This grammar probably needs work; my goal was to enable hyperparameter searches to write the parameters to file so that I could run parallel training jobs to explore the hyperparameter space. I included an example set of parameters that will work well with this data in models/params_small.txt."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, run [basenji_train.py](https://github.com/calico/basenji/blob/master/bin/basenji_train.py) to train a model. The program will offer training feedback via stdout and write the model output files to the prefix given by the *-s* parameter.\n",
    "\n",
    "The most relevant options here are:\n",
    "\n",
    "| Option/Argument | Value | Note |\n",
    "|:---|:---|:---|\n",
    "| -o | models/heart | Directory to save training logs and model checkpoints. |\n",
    "| params_file | models/params_small.json | JSON specified parameters to setup the model architecture and optimization. |\n",
    "| data_dir | data/heart_l131k | Data directory containing the test input and output datasets as generated by [basenji_data.py](https://github.com/calico/basenji/blob/master/bin/basenji_data.py) |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to train, uncomment the following line and run it. Depending on your hardware, it may require several hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                Output Shape                 Param #   Connected to                  \n",
      "==================================================================================================\n",
      " sequence (InputLayer)       [(None, 131072, 4)]          0         []                            \n",
      "                                                                                                  \n",
      " stochastic_reverse_complem  ((None, 131072, 4),          0         ['sequence[0][0]']            \n",
      " ent (StochasticReverseComp   ())                                                                 \n",
      " lement)                                                                                          \n",
      "                                                                                                  \n",
      " stochastic_shift (Stochast  (None, 131072, 4)            0         ['stochastic_reverse_complemen\n",
      " icShift)                                                           t[0][0]']                     \n",
      "                                                                                                  \n",
      " tf.nn.gelu (TFOpLambda)     (None, 131072, 4)            0         ['stochastic_shift[0][0]']    \n",
      "                                                                                                  \n",
      " conv1d (Conv1D)             (None, 131072, 64)           3840      ['tf.nn.gelu[0][0]']          \n",
      "                                                                                                  \n",
      " batch_normalization (Batch  (None, 131072, 64)           256       ['conv1d[0][0]']              \n",
      " Normalization)                                                                                   \n",
      "                                                                                                  \n",
      " max_pooling1d (MaxPooling1  (None, 16384, 64)            0         ['batch_normalization[0][0]'] \n",
      " D)                                                                                               \n",
      "                                                                                                  \n",
      " tf.nn.gelu_1 (TFOpLambda)   (None, 16384, 64)            0         ['max_pooling1d[0][0]']       \n",
      "                                                                                                  \n",
      " conv1d_1 (Conv1D)           (None, 16384, 64)            20480     ['tf.nn.gelu_1[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_1 (Bat  (None, 16384, 64)            256       ['conv1d_1[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " max_pooling1d_1 (MaxPoolin  (None, 4096, 64)             0         ['batch_normalization_1[0][0]'\n",
      " g1D)                                                               ]                             \n",
      "                                                                                                  \n",
      " tf.nn.gelu_2 (TFOpLambda)   (None, 4096, 64)             0         ['max_pooling1d_1[0][0]']     \n",
      "                                                                                                  \n",
      " conv1d_2 (Conv1D)           (None, 4096, 72)             23040     ['tf.nn.gelu_2[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_2 (Bat  (None, 4096, 72)             288       ['conv1d_2[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " max_pooling1d_2 (MaxPoolin  (None, 1024, 72)             0         ['batch_normalization_2[0][0]'\n",
      " g1D)                                                               ]                             \n",
      "                                                                                                  \n",
      " tf.nn.gelu_3 (TFOpLambda)   (None, 1024, 72)             0         ['max_pooling1d_2[0][0]']     \n",
      "                                                                                                  \n",
      " conv1d_3 (Conv1D)           (None, 1024, 32)             6912      ['tf.nn.gelu_3[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_3 (Bat  (None, 1024, 32)             128       ['conv1d_3[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " tf.nn.gelu_4 (TFOpLambda)   (None, 1024, 32)             0         ['batch_normalization_3[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " conv1d_4 (Conv1D)           (None, 1024, 72)             2304      ['tf.nn.gelu_4[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_4 (Bat  (None, 1024, 72)             288       ['conv1d_4[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " dropout (Dropout)           (None, 1024, 72)             0         ['batch_normalization_4[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " add (Add)                   (None, 1024, 72)             0         ['max_pooling1d_2[0][0]',     \n",
      "                                                                     'dropout[0][0]']             \n",
      "                                                                                                  \n",
      " tf.nn.gelu_5 (TFOpLambda)   (None, 1024, 72)             0         ['add[0][0]']                 \n",
      "                                                                                                  \n",
      " conv1d_5 (Conv1D)           (None, 1024, 32)             6912      ['tf.nn.gelu_5[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_5 (Bat  (None, 1024, 32)             128       ['conv1d_5[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " tf.nn.gelu_6 (TFOpLambda)   (None, 1024, 32)             0         ['batch_normalization_5[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " conv1d_6 (Conv1D)           (None, 1024, 72)             2304      ['tf.nn.gelu_6[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_6 (Bat  (None, 1024, 72)             288       ['conv1d_6[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " dropout_1 (Dropout)         (None, 1024, 72)             0         ['batch_normalization_6[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " add_1 (Add)                 (None, 1024, 72)             0         ['add[0][0]',                 \n",
      "                                                                     'dropout_1[0][0]']           \n",
      "                                                                                                  \n",
      " tf.nn.gelu_7 (TFOpLambda)   (None, 1024, 72)             0         ['add_1[0][0]']               \n",
      "                                                                                                  \n",
      " conv1d_7 (Conv1D)           (None, 1024, 32)             6912      ['tf.nn.gelu_7[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_7 (Bat  (None, 1024, 32)             128       ['conv1d_7[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " tf.nn.gelu_8 (TFOpLambda)   (None, 1024, 32)             0         ['batch_normalization_7[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " conv1d_8 (Conv1D)           (None, 1024, 72)             2304      ['tf.nn.gelu_8[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_8 (Bat  (None, 1024, 72)             288       ['conv1d_8[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " dropout_2 (Dropout)         (None, 1024, 72)             0         ['batch_normalization_8[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " add_2 (Add)                 (None, 1024, 72)             0         ['add_1[0][0]',               \n",
      "                                                                     'dropout_2[0][0]']           \n",
      "                                                                                                  \n",
      " tf.nn.gelu_9 (TFOpLambda)   (None, 1024, 72)             0         ['add_2[0][0]']               \n",
      "                                                                                                  \n",
      " conv1d_9 (Conv1D)           (None, 1024, 32)             6912      ['tf.nn.gelu_9[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_9 (Bat  (None, 1024, 32)             128       ['conv1d_9[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " tf.nn.gelu_10 (TFOpLambda)  (None, 1024, 32)             0         ['batch_normalization_9[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " conv1d_10 (Conv1D)          (None, 1024, 72)             2304      ['tf.nn.gelu_10[0][0]']       \n",
      "                                                                                                  \n",
      " batch_normalization_10 (Ba  (None, 1024, 72)             288       ['conv1d_10[0][0]']           \n",
      " tchNormalization)                                                                                \n",
      "                                                                                                  \n",
      " dropout_3 (Dropout)         (None, 1024, 72)             0         ['batch_normalization_10[0][0]\n",
      "                                                                    ']                            \n",
      "                                                                                                  \n",
      " add_3 (Add)                 (None, 1024, 72)             0         ['add_2[0][0]',               \n",
      "                                                                     'dropout_3[0][0]']           \n",
      "                                                                                                  \n",
      " tf.nn.gelu_11 (TFOpLambda)  (None, 1024, 72)             0         ['add_3[0][0]']               \n",
      "                                                                                                  \n",
      " conv1d_11 (Conv1D)          (None, 1024, 32)             6912      ['tf.nn.gelu_11[0][0]']       \n",
      "                                                                                                  \n",
      " batch_normalization_11 (Ba  (None, 1024, 32)             128       ['conv1d_11[0][0]']           \n",
      " tchNormalization)                                                                                \n",
      "                                                                                                  \n",
      " tf.nn.gelu_12 (TFOpLambda)  (None, 1024, 32)             0         ['batch_normalization_11[0][0]\n",
      "                                                                    ']                            \n",
      "                                                                                                  \n",
      " conv1d_12 (Conv1D)          (None, 1024, 72)             2304      ['tf.nn.gelu_12[0][0]']       \n",
      "                                                                                                  \n",
      " batch_normalization_12 (Ba  (None, 1024, 72)             288       ['conv1d_12[0][0]']           \n",
      " tchNormalization)                                                                                \n",
      "                                                                                                  \n",
      " dropout_4 (Dropout)         (None, 1024, 72)             0         ['batch_normalization_12[0][0]\n",
      "                                                                    ']                            \n",
      "                                                                                                  \n",
      " add_4 (Add)                 (None, 1024, 72)             0         ['add_3[0][0]',               \n",
      "                                                                     'dropout_4[0][0]']           \n",
      "                                                                                                  \n",
      " tf.nn.gelu_13 (TFOpLambda)  (None, 1024, 72)             0         ['add_4[0][0]']               \n",
      "                                                                                                  \n",
      " conv1d_13 (Conv1D)          (None, 1024, 32)             6912      ['tf.nn.gelu_13[0][0]']       \n",
      "                                                                                                  \n",
      " batch_normalization_13 (Ba  (None, 1024, 32)             128       ['conv1d_13[0][0]']           \n",
      " tchNormalization)                                                                                \n",
      "                                                                                                  \n",
      " tf.nn.gelu_14 (TFOpLambda)  (None, 1024, 32)             0         ['batch_normalization_13[0][0]\n",
      "                                                                    ']                            \n",
      "                                                                                                  \n",
      " conv1d_14 (Conv1D)          (None, 1024, 72)             2304      ['tf.nn.gelu_14[0][0]']       \n",
      "                                                                                                  \n",
      " batch_normalization_14 (Ba  (None, 1024, 72)             288       ['conv1d_14[0][0]']           \n",
      " tchNormalization)                                                                                \n",
      "                                                                                                  \n",
      " dropout_5 (Dropout)         (None, 1024, 72)             0         ['batch_normalization_14[0][0]\n",
      "                                                                    ']                            \n",
      "                                                                                                  \n",
      " add_5 (Add)                 (None, 1024, 72)             0         ['add_4[0][0]',               \n",
      "                                                                     'dropout_5[0][0]']           \n",
      "                                                                                                  \n",
      " tf.nn.gelu_15 (TFOpLambda)  (None, 1024, 72)             0         ['add_5[0][0]']               \n",
      "                                                                                                  \n",
      " conv1d_15 (Conv1D)          (None, 1024, 64)             4608      ['tf.nn.gelu_15[0][0]']       \n",
      "                                                                                                  \n",
      " batch_normalization_15 (Ba  (None, 1024, 64)             256       ['conv1d_15[0][0]']           \n",
      " tchNormalization)                                                                                \n",
      "                                                                                                  \n",
      " dropout_6 (Dropout)         (None, 1024, 64)             0         ['batch_normalization_15[0][0]\n",
      "                                                                    ']                            \n",
      "                                                                                                  \n",
      " tf.nn.gelu_16 (TFOpLambda)  (None, 1024, 64)             0         ['dropout_6[0][0]']           \n",
      "                                                                                                  \n",
      " dense (Dense)               (None, 1024, 3)              195       ['tf.nn.gelu_16[0][0]']       \n",
      "                                                                                                  \n",
      " switch_reverse (SwitchReve  (None, 1024, 3)              0         ['dense[0][0]',               \n",
      " rse)                                                                'stochastic_reverse_complemen\n",
      "                                                                    t[0][1]']                     \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 111011 (433.64 KB)\n",
      "Trainable params: 109235 (426.70 KB)\n",
      "Non-trainable params: 1776 (6.94 KB)\n",
      "__________________________________________________________________________________________________\n",
      "None\n",
      "model_strides [128]\n",
      "target_lengths [1024]\n",
      "target_crops [0]\n",
      "WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.SGD` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.SGD`.\n",
      "No checkpoints found.\n",
      "Successful first step!\n",
      "/opt/anaconda3/lib/python3.11/site-packages/keras/src/engine/training.py:3103: UserWarning: You are saving your model as an HDF5 file via `model.save()`. This file format is considered legacy. We recommend using instead the native Keras format, e.g. `model.save('my_model.keras')`.\n",
      "  saving_api.save_model(\n",
      "Epoch 0 - 93s - train_loss: 0.4236 - train_r: 0.1768 - train_r2: 0.0193 - valid_loss: 0.3745 - valid_r: 0.2643 - valid_r2: 0.0619 - best!\n",
      "Epoch 1 - 97s - train_loss: 0.3761 - train_r: 0.2383 - train_r2: 0.0510 - valid_loss: 0.3653 - valid_r: 0.2948 - valid_r2: 0.0719 - best!\n",
      "Epoch 2 - 245s - train_loss: 0.3560 - train_r: 0.2925 - train_r2: 0.0854 - valid_loss: 0.3525 - valid_r: 0.3360 - valid_r2: 0.1037 - best!\n",
      "Epoch 3 - 89s - train_loss: 0.3477 - train_r: 0.3201 - train_r2: 0.1024 - valid_loss: 0.3622 - valid_r: 0.3347 - valid_r2: 0.0668\n",
      "Epoch 4 - 90s - train_loss: 0.3435 - train_r: 0.3386 - train_r2: 0.1145 - valid_loss: 0.3528 - valid_r: 0.3400 - valid_r2: 0.0749\n",
      "Epoch 5 - 93s - train_loss: 0.3421 - train_r: 0.3467 - train_r2: 0.1200 - valid_loss: 0.3446 - valid_r: 0.3573 - valid_r2: 0.1214 - best!\n",
      "Epoch 6 - 92s - train_loss: 0.3358 - train_r: 0.3801 - train_r2: 0.1428 - valid_loss: 0.3499 - valid_r: 0.3649 - valid_r2: 0.0725\n",
      "Epoch 7 - 88s - train_loss: 0.3296 - train_r: 0.3990 - train_r2: 0.1559 - valid_loss: 0.3390 - valid_r: 0.4092 - valid_r2: 0.1668 - best!\n",
      "Epoch 8 - 92s - train_loss: 0.3276 - train_r: 0.4398 - train_r2: 0.1846 - valid_loss: 0.3358 - valid_r: 0.4388 - valid_r2: 0.1923 - best!\n",
      "Epoch 9 - 89s - train_loss: 0.3229 - train_r: 0.4638 - train_r2: 0.2073 - valid_loss: 0.3406 - valid_r: 0.4424 - valid_r2: 0.1895 - best!\n",
      "Epoch 10 - 89s - train_loss: 0.3204 - train_r: 0.4826 - train_r2: 0.2227 - valid_loss: 0.3509 - valid_r: 0.4081 - valid_r2: 0.1252\n",
      "Epoch 11 - 90s - train_loss: 0.3208 - train_r: 0.4894 - train_r2: 0.2341 - valid_loss: 0.3365 - valid_r: 0.4655 - valid_r2: 0.1899 - best!\n",
      "Epoch 12 - 91s - train_loss: 0.3152 - train_r: 0.5324 - train_r2: 0.2671 - valid_loss: 0.3367 - valid_r: 0.4656 - valid_r2: 0.1898 - best!\n",
      "Epoch 13 - 90s - train_loss: 0.3181 - train_r: 0.5467 - train_r2: 0.2908 - valid_loss: 0.3400 - valid_r: 0.5083 - valid_r2: 0.2511 - best!\n",
      "Epoch 14 - 93s - train_loss: 0.3145 - train_r: 0.5785 - train_r2: 0.3248 - valid_loss: 0.3423 - valid_r: 0.4957 - valid_r2: 0.2448\n",
      "Epoch 15 - 88s - train_loss: 0.3079 - train_r: 0.6088 - train_r2: 0.3522 - valid_loss: 0.3356 - valid_r: 0.4771 - valid_r2: 0.1974\n",
      "Epoch 16 - 88s - train_loss: 0.3078 - train_r: 0.6246 - train_r2: 0.3795 - valid_loss: 0.3381 - valid_r: 0.5160 - valid_r2: 0.2627 - best!\n",
      "Epoch 17 - 91s - train_loss: 0.3061 - train_r: 0.6050 - train_r2: 0.3643 - valid_loss: 0.3370 - valid_r: 0.5075 - valid_r2: 0.2534\n",
      "Epoch 18 - 89s - train_loss: 0.3041 - train_r: 0.6730 - train_r2: 0.4436 - valid_loss: 0.3292 - valid_r: 0.5082 - valid_r2: 0.2383\n",
      "Epoch 19 - 89s - train_loss: 0.2982 - train_r: 0.6570 - train_r2: 0.4246 - valid_loss: 0.3260 - valid_r: 0.4968 - valid_r2: 0.2416\n",
      "Epoch 20 - 88s - train_loss: 0.2988 - train_r: 0.6513 - train_r2: 0.4162 - valid_loss: 0.3271 - valid_r: 0.5020 - valid_r2: 0.2473\n",
      "Epoch 21 - 88s - train_loss: 0.2980 - train_r: 0.6572 - train_r2: 0.4272 - valid_loss: 0.3276 - valid_r: 0.5170 - valid_r2: 0.2647 - best!\n",
      "Epoch 22 - 89s - train_loss: 0.2932 - train_r: 0.7078 - train_r2: 0.4911 - valid_loss: 0.3331 - valid_r: 0.5183 - valid_r2: 0.2679 - best!\n",
      "Epoch 23 - 89s - train_loss: 0.2955 - train_r: 0.6877 - train_r2: 0.4679 - valid_loss: 0.3289 - valid_r: 0.5167 - valid_r2: 0.2412\n",
      "Epoch 24 - 90s - train_loss: 0.2888 - train_r: 0.7157 - train_r2: 0.5100 - valid_loss: 0.3268 - valid_r: 0.5076 - valid_r2: 0.0832\n",
      "Epoch 25 - 89s - train_loss: 0.2869 - train_r: 0.7433 - train_r2: 0.5520 - valid_loss: 0.3334 - valid_r: 0.5064 - valid_r2: 0.1172\n",
      "Epoch 26 - 94s - train_loss: 0.2938 - train_r: 0.7209 - train_r2: 0.5187 - valid_loss: 0.3332 - valid_r: 0.5032 - valid_r2: 0.2225\n",
      "Epoch 27 - 87s - train_loss: 0.2868 - train_r: 0.7680 - train_r2: 0.5853 - valid_loss: 0.3241 - valid_r: 0.5065 - valid_r2: 0.1751\n",
      "Epoch 28 - 87s - train_loss: 0.2841 - train_r: 0.7221 - train_r2: 0.5212 - valid_loss: 0.3246 - valid_r: 0.5083 - valid_r2: 0.1120\n",
      "Epoch 29 - 89s - train_loss: 0.2835 - train_r: 0.7421 - train_r2: 0.5503 - valid_loss: 0.3266 - valid_r: 0.5109 - valid_r2: 0.2549\n",
      "Epoch 30 - 88s - train_loss: 0.2798 - train_r: 0.7449 - train_r2: 0.5537 - valid_loss: 0.3416 - valid_r: 0.5022 - valid_r2: 0.2223\n",
      "Epoch 31 - 91s - train_loss: 0.2835 - train_r: 0.7386 - train_r2: 0.5399 - valid_loss: 0.3264 - valid_r: 0.5022 - valid_r2: 0.0452\n"
     ]
    }
   ],
   "source": [
    "! basenji_train.py -o models/heart models/params_small.json data/heart_l131k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you can just download a trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 1157k  100 1157k    0     0  3381k      0 --:--:-- --:--:-- --:--:-- 3374k\n"
     ]
    }
   ],
   "source": [
    "if not os.path.isdir('models/heart'):\n",
    "    os.mkdir('models/heart')\n",
    "if not os.path.isfile('models/heart/model_best.h5'):\n",
    "    subprocess.call('curl -o models/heart/model_best.h5 https://storage.googleapis.com/basenji_tutorial_data/model_best.h5', shell=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "models/heart/model_best.tf will now specify the name of your saved model to be provided to other programs.\n",
    "\n",
    "To further benchmark the accuracy (e.g. computing significant \"peak\" accuracy), use [basenji_test.py](https://github.com/calico/basenji/blob/master/bin/basenji_test.py).\n",
    "\n",
    "The most relevant options here are:\n",
    "\n",
    "| Option/Argument | Value | Note |\n",
    "|:---|:---|:---|\n",
    "| --ai | 0,1,2 | Make accuracy scatter plots for targets 0, 1, and 2. |\n",
    "| -o | output/heart_test | Output directory. |\n",
    "| --rc | | Average the forward and reverse complement to form an ensemble predictor. |\n",
    "| --shifts | | Average sequence shifts to form an ensemble predictor. |\n",
    "| params_file | models/params_small.json | JSON specified parameters to setup the model architecture and optimization. |\n",
    "| model_file | models/heart/model_best.h5 | Trained saved model parameters. |\n",
    "| data_dir | data/heart_l131k | Data directory containing the test input and output datasets as generated by [basenji_data.py](https://github.com/calico/basenji/blob/master/bin/basenji_data.py) |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                Output Shape                 Param #   Connected to                  \n",
      "==================================================================================================\n",
      " sequence (InputLayer)       [(None, 131072, 4)]          0         []                            \n",
      "                                                                                                  \n",
      " stochastic_reverse_complem  ((None, 131072, 4),          0         ['sequence[0][0]']            \n",
      " ent (StochasticReverseComp   ())                                                                 \n",
      " lement)                                                                                          \n",
      "                                                                                                  \n",
      " stochastic_shift (Stochast  (None, 131072, 4)            0         ['stochastic_reverse_complemen\n",
      " icShift)                                                           t[0][0]']                     \n",
      "                                                                                                  \n",
      " tf.nn.gelu (TFOpLambda)     (None, 131072, 4)            0         ['stochastic_shift[0][0]']    \n",
      "                                                                                                  \n",
      " conv1d (Conv1D)             (None, 131072, 64)           3840      ['tf.nn.gelu[0][0]']          \n",
      "                                                                                                  \n",
      " batch_normalization (Batch  (None, 131072, 64)           256       ['conv1d[0][0]']              \n",
      " Normalization)                                                                                   \n",
      "                                                                                                  \n",
      " max_pooling1d (MaxPooling1  (None, 16384, 64)            0         ['batch_normalization[0][0]'] \n",
      " D)                                                                                               \n",
      "                                                                                                  \n",
      " tf.nn.gelu_1 (TFOpLambda)   (None, 16384, 64)            0         ['max_pooling1d[0][0]']       \n",
      "                                                                                                  \n",
      " conv1d_1 (Conv1D)           (None, 16384, 64)            20480     ['tf.nn.gelu_1[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_1 (Bat  (None, 16384, 64)            256       ['conv1d_1[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " max_pooling1d_1 (MaxPoolin  (None, 4096, 64)             0         ['batch_normalization_1[0][0]'\n",
      " g1D)                                                               ]                             \n",
      "                                                                                                  \n",
      " tf.nn.gelu_2 (TFOpLambda)   (None, 4096, 64)             0         ['max_pooling1d_1[0][0]']     \n",
      "                                                                                                  \n",
      " conv1d_2 (Conv1D)           (None, 4096, 72)             23040     ['tf.nn.gelu_2[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_2 (Bat  (None, 4096, 72)             288       ['conv1d_2[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " max_pooling1d_2 (MaxPoolin  (None, 1024, 72)             0         ['batch_normalization_2[0][0]'\n",
      " g1D)                                                               ]                             \n",
      "                                                                                                  \n",
      " tf.nn.gelu_3 (TFOpLambda)   (None, 1024, 72)             0         ['max_pooling1d_2[0][0]']     \n",
      "                                                                                                  \n",
      " conv1d_3 (Conv1D)           (None, 1024, 32)             6912      ['tf.nn.gelu_3[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_3 (Bat  (None, 1024, 32)             128       ['conv1d_3[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " tf.nn.gelu_4 (TFOpLambda)   (None, 1024, 32)             0         ['batch_normalization_3[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " conv1d_4 (Conv1D)           (None, 1024, 72)             2304      ['tf.nn.gelu_4[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_4 (Bat  (None, 1024, 72)             288       ['conv1d_4[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " dropout (Dropout)           (None, 1024, 72)             0         ['batch_normalization_4[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " add (Add)                   (None, 1024, 72)             0         ['max_pooling1d_2[0][0]',     \n",
      "                                                                     'dropout[0][0]']             \n",
      "                                                                                                  \n",
      " tf.nn.gelu_5 (TFOpLambda)   (None, 1024, 72)             0         ['add[0][0]']                 \n",
      "                                                                                                  \n",
      " conv1d_5 (Conv1D)           (None, 1024, 32)             6912      ['tf.nn.gelu_5[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_5 (Bat  (None, 1024, 32)             128       ['conv1d_5[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " tf.nn.gelu_6 (TFOpLambda)   (None, 1024, 32)             0         ['batch_normalization_5[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " conv1d_6 (Conv1D)           (None, 1024, 72)             2304      ['tf.nn.gelu_6[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_6 (Bat  (None, 1024, 72)             288       ['conv1d_6[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " dropout_1 (Dropout)         (None, 1024, 72)             0         ['batch_normalization_6[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " add_1 (Add)                 (None, 1024, 72)             0         ['add[0][0]',                 \n",
      "                                                                     'dropout_1[0][0]']           \n",
      "                                                                                                  \n",
      " tf.nn.gelu_7 (TFOpLambda)   (None, 1024, 72)             0         ['add_1[0][0]']               \n",
      "                                                                                                  \n",
      " conv1d_7 (Conv1D)           (None, 1024, 32)             6912      ['tf.nn.gelu_7[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_7 (Bat  (None, 1024, 32)             128       ['conv1d_7[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " tf.nn.gelu_8 (TFOpLambda)   (None, 1024, 32)             0         ['batch_normalization_7[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " conv1d_8 (Conv1D)           (None, 1024, 72)             2304      ['tf.nn.gelu_8[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_8 (Bat  (None, 1024, 72)             288       ['conv1d_8[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " dropout_2 (Dropout)         (None, 1024, 72)             0         ['batch_normalization_8[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " add_2 (Add)                 (None, 1024, 72)             0         ['add_1[0][0]',               \n",
      "                                                                     'dropout_2[0][0]']           \n",
      "                                                                                                  \n",
      " tf.nn.gelu_9 (TFOpLambda)   (None, 1024, 72)             0         ['add_2[0][0]']               \n",
      "                                                                                                  \n",
      " conv1d_9 (Conv1D)           (None, 1024, 32)             6912      ['tf.nn.gelu_9[0][0]']        \n",
      "                                                                                                  \n",
      " batch_normalization_9 (Bat  (None, 1024, 32)             128       ['conv1d_9[0][0]']            \n",
      " chNormalization)                                                                                 \n",
      "                                                                                                  \n",
      " tf.nn.gelu_10 (TFOpLambda)  (None, 1024, 32)             0         ['batch_normalization_9[0][0]'\n",
      "                                                                    ]                             \n",
      "                                                                                                  \n",
      " conv1d_10 (Conv1D)          (None, 1024, 72)             2304      ['tf.nn.gelu_10[0][0]']       \n",
      "                                                                                                  \n",
      " batch_normalization_10 (Ba  (None, 1024, 72)             288       ['conv1d_10[0][0]']           \n",
      " tchNormalization)                                                                                \n",
      "                                                                                                  \n",
      " dropout_3 (Dropout)         (None, 1024, 72)             0         ['batch_normalization_10[0][0]\n",
      "                                                                    ']                            \n",
      "                                                                                                  \n",
      " add_3 (Add)                 (None, 1024, 72)             0         ['add_2[0][0]',               \n",
      "                                                                     'dropout_3[0][0]']           \n",
      "                                                                                                  \n",
      " tf.nn.gelu_11 (TFOpLambda)  (None, 1024, 72)             0         ['add_3[0][0]']               \n",
      "                                                                                                  \n",
      " conv1d_11 (Conv1D)          (None, 1024, 32)             6912      ['tf.nn.gelu_11[0][0]']       \n",
      "                                                                                                  \n",
      " batch_normalization_11 (Ba  (None, 1024, 32)             128       ['conv1d_11[0][0]']           \n",
      " tchNormalization)                                                                                \n",
      "                                                                                                  \n",
      " tf.nn.gelu_12 (TFOpLambda)  (None, 1024, 32)             0         ['batch_normalization_11[0][0]\n",
      "                                                                    ']                            \n",
      "                                                                                                  \n",
      " conv1d_12 (Conv1D)          (None, 1024, 72)             2304      ['tf.nn.gelu_12[0][0]']       \n",
      "                                                                                                  \n",
      " batch_normalization_12 (Ba  (None, 1024, 72)             288       ['conv1d_12[0][0]']           \n",
      " tchNormalization)                                                                                \n",
      "                                                                                                  \n",
      " dropout_4 (Dropout)         (None, 1024, 72)             0         ['batch_normalization_12[0][0]\n",
      "                                                                    ']                            \n",
      "                                                                                                  \n",
      " add_4 (Add)                 (None, 1024, 72)             0         ['add_3[0][0]',               \n",
      "                                                                     'dropout_4[0][0]']           \n",
      "                                                                                                  \n",
      " tf.nn.gelu_13 (TFOpLambda)  (None, 1024, 72)             0         ['add_4[0][0]']               \n",
      "                                                                                                  \n",
      " conv1d_13 (Conv1D)          (None, 1024, 32)             6912      ['tf.nn.gelu_13[0][0]']       \n",
      "                                                                                                  \n",
      " batch_normalization_13 (Ba  (None, 1024, 32)             128       ['conv1d_13[0][0]']           \n",
      " tchNormalization)                                                                                \n",
      "                                                                                                  \n",
      " tf.nn.gelu_14 (TFOpLambda)  (None, 1024, 32)             0         ['batch_normalization_13[0][0]\n",
      "                                                                    ']                            \n",
      "                                                                                                  \n",
      " conv1d_14 (Conv1D)          (None, 1024, 72)             2304      ['tf.nn.gelu_14[0][0]']       \n",
      "                                                                                                  \n",
      " batch_normalization_14 (Ba  (None, 1024, 72)             288       ['conv1d_14[0][0]']           \n",
      " tchNormalization)                                                                                \n",
      "                                                                                                  \n",
      " dropout_5 (Dropout)         (None, 1024, 72)             0         ['batch_normalization_14[0][0]\n",
      "                                                                    ']                            \n",
      "                                                                                                  \n",
      " add_5 (Add)                 (None, 1024, 72)             0         ['add_4[0][0]',               \n",
      "                                                                     'dropout_5[0][0]']           \n",
      "                                                                                                  \n",
      " tf.nn.gelu_15 (TFOpLambda)  (None, 1024, 72)             0         ['add_5[0][0]']               \n",
      "                                                                                                  \n",
      " conv1d_15 (Conv1D)          (None, 1024, 64)             4608      ['tf.nn.gelu_15[0][0]']       \n",
      "                                                                                                  \n",
      " batch_normalization_15 (Ba  (None, 1024, 64)             256       ['conv1d_15[0][0]']           \n",
      " tchNormalization)                                                                                \n",
      "                                                                                                  \n",
      " dropout_6 (Dropout)         (None, 1024, 64)             0         ['batch_normalization_15[0][0]\n",
      "                                                                    ']                            \n",
      "                                                                                                  \n",
      " tf.nn.gelu_16 (TFOpLambda)  (None, 1024, 64)             0         ['dropout_6[0][0]']           \n",
      "                                                                                                  \n",
      " dense (Dense)               (None, 1024, 3)              195       ['tf.nn.gelu_16[0][0]']       \n",
      "                                                                                                  \n",
      " switch_reverse (SwitchReve  (None, 1024, 3)              0         ['dense[0][0]',               \n",
      " rse)                                                                'stochastic_reverse_complemen\n",
      "                                                                    t[0][1]']                     \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 111011 (433.64 KB)\n",
      "Trainable params: 109235 (426.70 KB)\n",
      "Non-trainable params: 1776 (6.94 KB)\n",
      "__________________________________________________________________________________________________\n",
      "None\n",
      "model_strides [128]\n",
      "target_lengths [1024]\n",
      "target_crops [0]\n",
      "WARNING:absl:At this time, the v2.11+ optimizer `tf.keras.optimizers.SGD` runs slowly on M1/M2 Macs, please use the legacy Keras optimizer instead, located at `tf.keras.optimizers.legacy.SGD`.\n",
      "45/45 [==============================] - 10s 184ms/step - loss: 0.3362 - pearsonr: 0.5476 - r2: 0.2953\n",
      "\n",
      "Test Loss:         0.33624\n",
      "Test PearsonR:     0.54756\n",
      "Test R2:           0.29531\n",
      "1/1 [==============================] - 1s 1s/step\n",
      "1/1 [==============================] - 0s 182ms/step\n",
      "1/1 [==============================] - 0s 177ms/step\n",
      "1/1 [==============================] - 0s 195ms/step\n",
      "1/1 [==============================] - 0s 185ms/step\n",
      "1/1 [==============================] - 0s 169ms/step\n",
      "1/1 [==============================] - 0s 185ms/step\n",
      "1/1 [==============================] - 0s 224ms/step\n",
      "1/1 [==============================] - 0s 224ms/step\n",
      "1/1 [==============================] - 0s 226ms/step\n",
      "1/1 [==============================] - 0s 184ms/step\n",
      "1/1 [==============================] - 0s 183ms/step\n",
      "1/1 [==============================] - 0s 179ms/step\n",
      "1/1 [==============================] - 0s 178ms/step\n",
      "1/1 [==============================] - 0s 235ms/step\n",
      "1/1 [==============================] - 0s 182ms/step\n",
      "1/1 [==============================] - 0s 182ms/step\n",
      "1/1 [==============================] - 0s 171ms/step\n",
      "1/1 [==============================] - 0s 186ms/step\n",
      "1/1 [==============================] - 0s 176ms/step\n",
      "1/1 [==============================] - 0s 183ms/step\n",
      "1/1 [==============================] - 0s 171ms/step\n",
      "1/1 [==============================] - 0s 181ms/step\n",
      "1/1 [==============================] - 0s 173ms/step\n",
      "1/1 [==============================] - 0s 184ms/step\n",
      "1/1 [==============================] - 0s 173ms/step\n",
      "1/1 [==============================] - 0s 182ms/step\n",
      "1/1 [==============================] - 0s 184ms/step\n",
      "1/1 [==============================] - 0s 198ms/step\n",
      "1/1 [==============================] - 0s 179ms/step\n",
      "1/1 [==============================] - 0s 184ms/step\n",
      "1/1 [==============================] - 0s 173ms/step\n",
      "1/1 [==============================] - 0s 171ms/step\n",
      "1/1 [==============================] - 0s 188ms/step\n",
      "1/1 [==============================] - 0s 172ms/step\n",
      "1/1 [==============================] - 0s 180ms/step\n",
      "1/1 [==============================] - 0s 191ms/step\n",
      "1/1 [==============================] - 0s 178ms/step\n",
      "1/1 [==============================] - 0s 173ms/step\n",
      "1/1 [==============================] - 0s 218ms/step\n",
      "1/1 [==============================] - 0s 169ms/step\n",
      "1/1 [==============================] - 0s 172ms/step\n",
      "1/1 [==============================] - 0s 173ms/step\n",
      "1/1 [==============================] - 0s 172ms/step\n",
      "1/1 [==============================] - 0s 168ms/step\n"
     ]
    }
   ],
   "source": [
    "! basenji_test.py --ai 0,1,2 -o output/heart_test --rc --shifts \"1,0,-1\" models/params_small.json models/heart/model_best.h5 data/heart_l131k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*data/heart_test/acc.txt* is a table specifiying the Pearson correlation and R2 for each dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "index\tpearsonr\tr2\tidentifier\tdescription\n",
      "0\t0.50047\t0.23747\tCNhs11760\taorta\n",
      "1\t0.64530\t0.40714\tCNhs12843\tartery\n",
      "2\t0.49690\t0.24133\tCNhs12856\tpulmonic_valve\n"
     ]
    }
   ],
   "source": [
    "! cat output/heart_test/acc.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The directories *pr*, *roc*, *violin*, and *scatter* in *data/heart_test* contain plots for the targets indexed by 0, 1, and 2 as specified by the --ai option above.\n",
    "\n",
    "E.g."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "        <iframe\n",
       "            width=\"600\"\n",
       "            height=\"500\"\n",
       "            src=\"output/heart_test/pr/t0.pdf\"\n",
       "            frameborder=\"0\"\n",
       "            allowfullscreen\n",
       "            \n",
       "        ></iframe>\n",
       "        "
      ],
      "text/plain": [
       "<IPython.lib.display.IFrame at 0x332024150>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "IFrame('output/heart_test/pr/t0.pdf', width=600, height=500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}