Skip to content

Commit a9c2d75

Browse files
d-w-moored-w-moore
committed
[_502] experimental test harness using containers
setupssl chgs -q [_502] fixes/enhancements to test harness (inclusive misc__ commits) misc__ pre: make irods install simpler, independent of other repos run the test script using exec (The docker run command initiates an iRODS startup process via CMD). now able to define the irods version to be installed. fix relative paths; sample BATS test setupssl changes pam and ssl setup funcs build target version of Python (will parameterize version later) refactor of test containers experiment.sh -- call setupssl.py in container tighter checking in driver/test-runner corrections enable ageing out pam pw (postgresql only) misc__: add/refine options - workdir etc options include dummy_script called from experiment.sh move to test-script-parameters more utility funcs directory name change misc__ post: update json files for test restore correction to update json funcs chmod +x login_auth_test.py failing/script-to-script call demo implement wrapping virtual env provide path to original script; allow args (testnames) delete script not used include overlooked ssl-and-pam Dockerfile wrap/args demo get rid of troublesome recursive symlink abspath for orig script; -t better for early docker versions username is now param for setting up pam with iinit setup conditions properly for tests adjustments to keys. arglist is now affected by symlink basename of argument if any. changes to login_auth test changes for login_auth_test nr 2 allow different irods versions include symlink change to test script params only separate container run for env-modifying test test001 working include the other symlink In testing don't depend on strong primes for DH param generation. 362-test-harness setupssl-q-nr2
1 parent c238d9b commit a9c2d75

28 files changed

+768
-109
lines changed

‎irods/test/demo.sh

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
#!/bin/bash
2+
3+
echo "$0 running"
4+
echo args:
5+
for arg in $*; do
6+
echo $((++x)): "[$arg]"
7+
done
8+
9+
exit 118

‎irods/test/demo_A.sh

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
demo.sh

‎irods/test/demo_B.sh

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
demo.sh

‎irods/test/demo_hook.sh

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/bin/bash
2+
echo "-- HOOK RUNNING --"
3+
command "/prc/$ORIGINAL_SCRIPT_RELATIVE_TO_ROOT" $*

‎irods/test/harness/000_install-irods.Dockerfile

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
FROM ubuntu:18.04
2+
COPY install.sh /
3+
ARG irods_package_version
4+
ENV IRODS_PACKAGE_VERSION "$irods_package_version"
5+
RUN for phase in initialize install-essential-packages add-package-repo; do \
6+
bash /install.sh --w=$phase 0; \
7+
done
8+
RUN /install.sh 4
9+
COPY start_postgresql_and_irods.sh /
10+
RUN useradd -ms/bin/bash user
11+
RUN apt install -y faketime
12+
CMD bash /start_postgresql_and_irods.sh

‎irods/test/harness/001_bats-python3.Dockerfile

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
from install-irods
2+
run apt update; apt install -y python3-pip bats
3+
run python3 -m pip install --upgrade pip
4+
run python3 -m pip install virtualenv
5+
run python3 -m virtualenv /py3

‎irods/test/harness/002_ssl-and-pam.Dockerfile

+4
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
FROM bats-python3
2+
RUN apt install -y sudo
3+
RUN useradd -ms/bin/bash testuser
4+
RUN echo 'testuser ALL=(ALL) NOPASSWD: ALL' >>/etc/sudoers

‎irods/test/harness/003_prc-with-py3_12_2.Dockerfile

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
from install-irods
2+
run apt update
3+
run apt install -y wget build-essential libssl-dev zlib1g-dev
4+
run apt install wget build-essential
5+
run wget https://www.python.org/ftp/python/3.12.2/Python-3.12.2.tar.xz
6+
run tar xf Python-3.12.2.tar.xz
7+
workdir /Python-3.12.2
8+
run ./configure --prefix /root/python --with-ensurepip=install
9+
run make -j
10+
run mkdir /root/python
11+
run make install
12+
workdir /
13+
run /root/python/bin/python3 -m pip install python-irodsclient
14+
run chmod a+rx /root

‎irods/test/harness/README.txt

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
SAMPLE RUNS
2+
3+
To build required images
4+
------------------------
5+
Examples
6+
7+
1) ./build-docker.sh
8+
DEFAULT: build single-node system based on latest iRODS release
9+
10+
2) IRODS_PACKAGE_VERSION="4.2.12-1~bionic" NO_CACHE='1' ./build-docker.sh
11+
Build (ignoring docker cache) single-node system based on specified package version string.
12+
13+
simple examples
14+
---------------
15+
./docker_container_driver.sh tests/test_1.sh
16+
./docker_container_driver.sh tests/test_2.sh
17+
18+
Any script in a subdirectory of the repo (mounted at /prc within the container) can be
19+
executed and will be able to find other scripts and source include files within the tree.
20+
[See "experiment.sh" example below.]
21+
22+
Examples of options in driver script
23+
------------------------------------
24+
25+
1. To start container and run test script:
26+
C=$( ./docker_container_driver.sh -c -L -u testuser ../scripts/experiment.sh )
27+
28+
2. To manually examine results afterward:
29+
docker exec -it $C bash
30+
31+
32+
Demo / Demo hook / args
33+
------------------------
34+
35+
$ ~/python-irodsclient/irods/test/harness$ ./docker_container_driver.sh ../demo.sh
36+
ORIGINAL_SCRIPT_RELATIVE_TO_ROOT=[irods/test/demo.sh]
37+
image=[ssl-and-pam]
38+
.......-- HOOK RUNNING --
39+
/prc/irods/test/demo.sh running
40+
args:
41+
1: [arg1]
42+
2: [arg2]
43+
Killed: 1358fbff6eadac24f0915ffb414f0367deedc84b0c3e4de69a23bd3a8726298f
44+
daniel@prec3431:~/python-irodsclient/irods/test/harness$ echo $?
45+
118
46+

‎irods/test/harness/build-docker.sh

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
#!/usr/bin/env bash
2+
3+
# IRODS_PACKAGE_VERSION if defined is like "4.3.1-0~bionic"
4+
# (but contains no '~' suffix for irods versions <= 4.2.10)
5+
6+
BASE=$(basename "$0")
7+
DIR=$(realpath "$(dirname "$0")")
8+
cd "$DIR"
9+
DOCKER=docker
10+
for dockerfile in [0-9]*.Dockerfile; do
11+
image_name=${dockerfile#[0-9]*_}
12+
image_name=${image_name%.Dockerfile}
13+
if [ "$image_name" = "install-irods" ];then
14+
package_version_option=${IRODS_PACKAGE_VERSION:+"--build-arg=irods_package_version=$IRODS_PACKAGE_VERSION"}
15+
else
16+
package_version_option=""
17+
fi
18+
$DOCKER build -f $dockerfile -t $image_name . $package_version_option \
19+
${NO_CACHE+"--no-cache"} ||
20+
{ STATUS=$?; echo "*** Failure while building [$image_name]"; exit $STATUS; }
21+
done

‎irods/test/harness/docker_container_driver.sh

+109
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
#!/usr/bin/env bash
2+
3+
KILL_TEST_CONTAINER=1
4+
RUN_AS_USER=""
5+
ECHO_CONTAINER=""
6+
7+
EXPLICIT_WORKDIR=""
8+
while [[ $1 = -* ]]; do
9+
if [ "$1" = -c ]; then
10+
ECHO_CONTAINER=1
11+
shift
12+
fi
13+
if [ "$1" = -L ]; then
14+
KILL_TEST_CONTAINER=0
15+
shift
16+
fi
17+
if [ "$1" = -u ]; then
18+
RUN_AS_USER="$2"
19+
shift 2
20+
fi
21+
if [ "$1" = -w ]; then
22+
EXPLICIT_WORKDIR="$2"
23+
shift 2
24+
fi
25+
done
26+
27+
if [ "$1" = "" ]; then
28+
echo >&2 "Usage: $0 [options] /path/to/script"
29+
echo >&2 "With options: [-L] to leak, [-u username] to run as non-root user"
30+
exit 1
31+
fi
32+
33+
DIR=$(dirname $0)
34+
. "$DIR"/test_script_parameters
35+
36+
testscript=${1}
37+
38+
testscript_basename=$(basename "$testscript")
39+
arglist=${wrapper_arglist[$testscript_basename]} # arglist dominated by symbolic link name if any
40+
41+
if [ -L "$testscript" ]; then
42+
testscript=$(realpath "$testscript")
43+
testscript_basename=$(basename "$testscript")
44+
fi
45+
46+
original_testscript_abspath=$(realpath "$testscript")
47+
48+
wrapped=${wrappers["$testscript_basename"]}
49+
50+
if [ -n "$wrapped" ]; then
51+
# wrapped is assumed to contain a leading path element relative to the referencing script's containing directory
52+
testscript="$(dirname "$testscript")/$wrapped"
53+
testscript_basename=$(basename "$testscript")
54+
fi
55+
56+
testscript_abspath=$(realpath "$testscript")
57+
58+
cd "$DIR"
59+
60+
image=${images[$testscript_basename]}
61+
62+
if [ -z "$RUN_AS_USER" ]; then
63+
RUN_AS_USER=${user[$testscript_basename]}
64+
fi
65+
66+
# Tests are run as testuser by default
67+
: ${RUN_AS_USER:='testuser'}
68+
69+
WORKDIR=""
70+
if [ -n "$EXPLICIT_WORKDIR" ]; then
71+
WORKDIR="$EXPLICIT_WORKDIR"
72+
else
73+
WORKDIR=${workdirs[$RUN_AS_USER]}
74+
fi
75+
76+
reporoot=$(./print_repo_root_location)
77+
ORIGINAL_SCRIPT_RELATIVE_TO_ROOT=$(realpath --relative-to $reporoot "$original_testscript_abspath")
78+
79+
echo "ORIGINAL_SCRIPT_RELATIVE_TO_ROOT=[$ORIGINAL_SCRIPT_RELATIVE_TO_ROOT]"
80+
INNER_MOUNT=/prc
81+
82+
# Start the container.
83+
echo image="[$image]"
84+
CONTAINER=$(docker run -d -v $reporoot:$INNER_MOUNT:ro --rm $image)
85+
86+
# Wait for iRODS and database to start up.
87+
TIME0=$(date +%s)
88+
while :; do
89+
[ `date +%s` -gt $((TIME0 + 30)) ] && { echo >&2 "Waited too long for DB and iRODS to start"; exit 124; }
90+
sleep 1
91+
docker exec $CONTAINER grep '(0)' /tmp/irods_status 2>/dev/null >/dev/null
92+
[ $? -ne 0 ] && { echo -n . >&2; continue; }
93+
break
94+
done
95+
96+
docker exec ${RUN_AS_USER:+"-u$RUN_AS_USER"} \
97+
${WORKDIR:+"-w$WORKDIR"} \
98+
-e "ORIGINAL_SCRIPT_RELATIVE_TO_ROOT=$ORIGINAL_SCRIPT_RELATIVE_TO_ROOT" \
99+
$CONTAINER \
100+
$INNER_MOUNT/$(realpath --relative-to $reporoot "$testscript_abspath") \
101+
$arglist
102+
STATUS=$?
103+
104+
if [ $((0+KILL_TEST_CONTAINER)) -ne 0 ]; then
105+
echo >&2 'Killed:' $(docker stop --time=0 $CONTAINER)
106+
fi
107+
108+
[ -n "$ECHO_CONTAINER" ] && echo $CONTAINER
109+
exit $STATUS

‎irods/test/harness/install.sh

+139
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
#!/bin/bash
2+
3+
IRODS_HOME=/var/lib/irods
4+
DEV_HOME="$HOME"
5+
: ${DEV_REPOS:="$DEV_HOME/github"}
6+
7+
add_package_repo()
8+
{
9+
local R="/etc/apt/sources.list.d/renci-irods.list"
10+
echo >&2 "... installing package repo"
11+
sudo apt update
12+
sudo apt install -y lsb-release apt-transport-https
13+
wget -qO - https://packages.irods.org/irods-signing-key.asc | sudo apt-key add - && \
14+
echo "deb [arch=amd64] https://packages.irods.org/apt/ $(lsb_release -sc) main" |\
15+
sudo tee "$R"
16+
sudo apt update
17+
}
18+
19+
DIST_NAME=$(lsb_release -sc)
20+
21+
: ${IRODS_VSN:=4.3.1-0~$DIST_NAME}
22+
23+
while [[ "$1" = -* ]]; do
24+
ARG="$1"
25+
shift
26+
case $ARG in
27+
--i=* | --irods=* |\
28+
--irods-version=*) IRODS_PACKAGE_VERSION=${ARG#*=};;
29+
--w=* | --with=* | --with-options=* ) withopts=${ARG#*=} ;;
30+
-v) VERBOSE=1;;
31+
esac
32+
done
33+
34+
35+
run_phase() {
36+
37+
local PHASE=$1
38+
local with_opts=" $2 "
39+
40+
case "$PHASE" in
41+
42+
0)
43+
44+
if [[ $with_opts = *\ initialize\ * ]]; then
45+
apt-get -y update
46+
apt-get install -y apt-transport-https wget lsb-release sudo jq
47+
fi
48+
49+
if [[ $with_opts = *\ sudo-without-pw\ * ]]; then
50+
if [ `id -u` = 0 -a "${USER:-root}" = root ] ; then
51+
echo >&2 "root authorization for 'sudo' is automatic - no /etc/sudoers modification needed"
52+
else
53+
if [ -f "/etc/sudoers" ]; then
54+
if [ -n "$USER" ] ; then
55+
# add a line with our USER name to /etc/sudoers if not already there
56+
sudo su -c "sed -n '/^\s*[^#]/p' /etc/sudoers | grep '^$USER\s*ALL=(ALL)\s*NOPASSWD:\s*ALL\s*$' >/dev/null" || \
57+
sudo su -c "echo '$USER ALL=(ALL) NOPASSWD: ALL' >>/etc/sudoers"
58+
else
59+
echo >&2 "user login is '$USER' - can this be right?"
60+
fi
61+
else
62+
echo >&2 "WARNING - Could not modify sudoers files"
63+
echo -n >&2 " (hit 'Enter' to continue)"
64+
read key
65+
fi
66+
fi # not root
67+
fi # with-opts
68+
69+
#------ (needed for both package install and build from source)
70+
71+
if [[ $with_opts = *\ install-essential-packages\ * ]]; then
72+
73+
if ! dpkg -l tzdata >/dev/null 2>&1 ; then
74+
sudo su - root -c \
75+
"env DEBIAN_FRONTEND=noninteractive bash -c 'apt-get install -y tzdata'"
76+
fi
77+
sudo apt-get update
78+
sudo apt-get install -y software-properties-common postgresql
79+
sudo apt-get update && \
80+
sudo apt-get install -y libfuse2 unixodbc rsyslog ################### less python-pip
81+
fi
82+
83+
84+
if [[ $with_opts = *\ add-package-repo\ * ]]; then
85+
add_package_repo -f
86+
fi
87+
88+
89+
if [[ $with_opts = *\ create-db\ * ]]; then
90+
sudo su - postgres -c "
91+
{ dropdb --if-exists ICAT
92+
dropuser --if-exists irods ; } >/dev/null 2>&1"
93+
sudo su - postgres -c "psql <<\\
94+
________
95+
CREATE DATABASE \"ICAT\";
96+
CREATE USER irods WITH PASSWORD 'testpassword';
97+
GRANT ALL PRIVILEGES ON DATABASE \"ICAT\" to irods;
98+
________"
99+
echo >&2 "-- status of create-db = $? -- "
100+
fi
101+
;;
102+
103+
4)
104+
sudo apt install -y irods-{dev,runtime}${IRODS_PACKAGE_VERSION:+"=$IRODS_PACKAGE_VERSION"}
105+
if [[ $with_opts != *\ basic\ * ]]; then
106+
sudo apt install -y irods-{icommands,server,database-plugin-postgres}${IRODS_PACKAGE_VERSION:+"=$IRODS_PACKAGE_VERSION"}
107+
fi
108+
;;
109+
110+
5)
111+
if [ ! "$IRODS_VSN" '<' "4.3" ]; then
112+
PYTHON=python3
113+
else
114+
PYTHON=python2
115+
fi
116+
sudo $PYTHON /var/lib/irods/scripts/setup_irods.py < /var/lib/irods/packaging/localhost_setup_postgres.input
117+
;;
118+
119+
*) echo >&2 "unrecognized phase: '$PHASE'." ; QUIT=1 ;;
120+
esac
121+
return $?
122+
}
123+
124+
#-------------------------- main
125+
126+
QUIT=0
127+
while [ $# -gt 0 ] ; do
128+
ARG=$1 ; shift
129+
NOP="" ; run_phase $ARG " $withopts "; sts=$?
130+
[ $QUIT != 0 ] && break
131+
[ -n "$NOP" ] && continue
132+
echo -n "== $ARG == "
133+
if [ $sts -eq 0 ]; then
134+
echo Y >&2
135+
else
136+
[ $quit_on_phase_err ] && { echo >&2 "N - quitting"; exit 1; }
137+
echo N >&2
138+
fi
139+
done

‎irods/test/harness/print_repo_root_location

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
#!/bin/bash
2+
# The following line needs be kept updated to reflect true position relative to repository root,
3+
# in the event this script or any of its chain of containing directories (up to but not including the repo root) are moved.
4+
REPO_ROOT_RELATIVE_TO_THIS_SCRIPT=../../..
5+
realpath "$(dirname "$0")/$REPO_ROOT_RELATIVE_TO_THIS_SCRIPT"

0 commit comments

Comments
 (0)