How to use Test-Suite

Test-Suite is a micro-benchmark suite designed for validating various static analysis algorithms (particularly pointer analyses and static bug checkers) developed in SVF. It includes around 400 hand-written programs and code snippets. Test-Suite provides flexible and extendable interfaces for users to add their own tests for validating the correctness of different static analyses.

Download SVF and execute the following commandline under SVF's root directory. build.sh will build SVF and run SVF's analyses against the test cases if Test-suite folder is present.

git clone https://github.com/SVF-tools/SVF.git
cd SVF
git clone https://github.com/SVF-tools/Test-Suite
source ./build.sh
cd Release-build
ctest

The following gives the Test-Suite's folders and the corresponding SVF's options for validation.

Folder	SVF option	Description
basic_c_tests	wpa -ander -stat=false	basic test cases for C programs (flow-insensitive and field-sensitive analysis)
basic_cpp_tests	wpa -ander -stat=false	basic test cases for C++ programs (flow-insensitive and field-sensitive analysis)
fs_tests	wpa -fspta -stat=false	flow-sensitive tests
cs_tests	dvf -cxt -print-pts=false -stat=false	context-sensitive tests
path_tests		path-sensitive tests
complex_tests	wpa -ander -stat=false	complex test cases simplified from real programs
mta		multithreaded test cases
mem_leak	saber -leak -valid-tests -mempar=inter-disjoint -stat=false	memory leak test cases
double_free	saber -dfree -valid-tests -stat=false	double free test cases

Scripts

Travis CI is used to build all the test cases, It uses ./generate_bc.sh to do so and then pushes the compiled test cases into master branch.

All tests are specified in CMakeLists.txt file. The CMakeLists.txt file uses foreach loops to specify tests in bulk. To add new tests append a command and specified folder into the respective commands list and folders list in CMakeLists.txt An example of a test:

add_test(
  NAME basic_c_tests/CI-funptr.c
  COMMAND wpa -ander -stat=false ${CMAKE_CURRENT_SOURCE_DIR}/test_cases_bc/basic_c_tests/CI-funptr.c.bc
  WORKING_DIRECTORY ${PROJECT_SOURCE_DIR}/${CMAKE_BUILD_TYPE}-build/bin
)

For differential testing of two options a C++ script diff_tests.cpp is used. This script will run each option, compare the results and output. If the output is 0 then the test passed.