Build fails on Linux (both x64 and x32)

[max-kamps]

main.cpp
mkdir -p ../../intermediate/release_x64/gmake/x64/release/nativejson
g++   -MMD -MP -D__STDC_FORMAT_MACROS=1 -DNDEBUG -DHAS_BOOST=1 -I../../thirdparty -I../../thirdparty/fastjson/include -I../../thirdparty/jsonbox/include -I../../thirdparty/jsoncpp/include -I../../thirdparty/rapidjson/include -I../../thirdparty/udp-json-parser -I../../thirdparty/include -I../../thirdparty/json-voorhees/include -I../../thirdparty/json-voorhees/src -I../../thirdparty/jsoncons/src -I../../thirdparty/ArduinoJson/include -I../../thirdparty/include/jeayeson/include/dummy -I../../thirdparty/jvar/include -I../../thirdparty/pjson/inc -I../../thirdparty/ULib/include -I../../thirdparty/facil.io/lib/facil/core/types -I../../thirdparty/facil.io/lib/facil/core/types/fiobj -m64 -O3 -Wall -Wextra -march=native -Wall -Wextra -std=c++14  -o "../../intermediate/release_x64/gmake/x64/release/nativejson/main.o" -MF "../../intermediate/release_x64/gmake/x64/release/nativejson/main.d" -c "../../src/main.cpp"
In file included from ../../src/test.h:3:0,
                 from ../../src/main.cpp:12:
../../src/memorystat.h:141:16: error: ‘std::MemoryStatCalloc’ has not been declared
 #define calloc MemoryStatCalloc
                ^
../../src/memorystat.h:143:14: error: ‘void std::MemoryStatFree(void*)’ conflicts with a previous declaration
 #define free MemoryStatFree
              ^
../../src/memorystat.h:134:13: note: previous declaration ‘void MemoryStatFree(void*)’
 extern void MemoryStatFree(void* ptr);
             ^~~~~~~~~~~~~~
../../src/memorystat.h:140:16: error: ‘void* std::MemoryStatMalloc(std::size_t)’ conflicts with a previous declaration
 #define malloc MemoryStatMalloc
                ^
../../src/memorystat.h:131:14: note: previous declaration ‘void* MemoryStatMalloc(size_t)’
 extern void* MemoryStatMalloc(size_t size);
              ^~~~~~~~~~~~~~~~
../../src/memorystat.h:142:17: error: ‘void* std::MemoryStatRealloc(void*, std::size_t)’ conflicts with a previous declaration
 #define realloc MemoryStatRealloc
                 ^
../../src/memorystat.h:133:14: note: previous declaration ‘void* MemoryStatRealloc(void*, size_t)’
 extern void* MemoryStatRealloc(void* ptr, size_t size);
              ^~~~~~~~~~~~~~~~~
make[2]: *** [nativejson.make:175: ../../intermediate/release_x64/gmake/x64/release/nativejson/main.o] Error 1
make[1]: *** [benchmark.make:35: nativejson] Error 2
make[1]: Leaving directory '/home/max/Documents/nativejson-benchmark/build/gmake'
make: *** [Makefile:38: bin/nativejson_release_x64_gmake.a] Error 2```

[tomolt]

I can confirm this issue. Disabling memorystat by changing "#define USE_MEMORYSTAT 1" to "#define USE_MEMORYSTAT 0" in src/config.h circumvents this compiler error, but it is by far not enough to get the full build to work. I will report further findings in a couple of minutes ...

[tomolt]

Okay, I got it working now. Here are all the problems I encountered:

1. said MemoryStat build issue

Again, disabling MemoryStat fixes this.

2. Multiple tests fail to build

Specifically:

• src/tests/facil.io.cpp
• src/tests/configurutest.cpp
• src/tests/jeayesontest.cpp

I had to disable all of them. It should be noted however that it seems like this is caused by some compiler incompatibility. I'm running GNU C++ 7.3.1.
Clang might not have this problem.

3. premake doesn't correctly identify boost

I actually did not have the boost headers installed going into this, but premake assumed I had anyway and set HAS_BOOST=1. Installing the boost headers solves this problem.

4. ULib can't build correctly

It seems some part of it's build system setup didn't behave properly on my system.
To disable ULib, I had to disable src/tests/ULibtest.cpp.bak, and comment out two lines in build/premake5.lua: line 138 and line 239 (both begin with "linkoptions").

5. Some tests crash in the verifying stage (segfault)

Specifically:

• src/tests/jvartest.cpp (munmap_chunk(): invalid pointer)
• src/tests/ujsontest.cpp (free(): invalid pointer)

Disable them too and it should work.

Oh, and you'll also need php installed, which isn't mentioned in the readme AFAIK.

I hope this helps.

[mloskot]

4. ULib can't build correctly

This may be related #91

I disabled the ULib as explained in the #102 (comment), then run

make clean
cd build
./premake.sh
cd gmake
make -f benchmark.make config=release_x64 && make -f nativejson.make config=release_x64

and it failed with

ULibtest.cpp
In file included from ../../thirdparty/ULib/include/ulib/base/hash.h:20,
                 from ../../thirdparty/ULib/include/ulib/string.h:17,
                 from ../../thirdparty/ULib/include/ulib/container/construct.h:17,
                 from ../../thirdparty/ULib/include/ulib/container/vector.h:17,
                 from ../../thirdparty/ULib/include/ulib/container/hash_map.h:17,
                 from ../../thirdparty/ULib/include/ulib/json/value.h:17,
                 from ../../src/tests/ULibtest.cpp:16:
../../thirdparty/ULib/include/ulib/base/base.h:20:12: fatal error: ulib/internal/config.h: No such file or directory
   20 | #  include <ulib/internal/config.h>
      |            ^~~~~~~~~~~~~~~~~~~~~~~~

Removing thirdparty/ULib submodule completely should be a usable workaround, then rebuild

cd build && ./premake.sh && cd gmake && make -f benchmark.make config=release_x64 && make -f nativejson.make config=release_x64 && cd ../..

5. Some tests crash in the verifying stage (segfault)

The Configure crashes too, so

rm -rf src/tests/configurutest.cpp
rm -rf thirdparty/configuru/

and rebuild.

---

Adding more details and issues to the list from the #102 (comment):

1. Linked version of configuru is failing to compile

In file included from ../../src/tests/configurutest.cpp:8:
../../thirdparty/configuru/configuru.hpp:574:34: error: ‘function’ in namespace ‘std’ does not name a template type
  574 |   void visit_dangling(const std::function<void(const std::string& key, const Config& value)>& visitor) const;

Workaround: add missing #include <functional> in configuru.hpp.

2. Linked version of Facil.io is failing to compile

facil.io.cpp
In file included from ../../thirdparty/facil.io/lib/facil/core/types/fiobj/fiobj_ary.h:12:0,
                 from ../../thirdparty/facil.io/lib/facil/core/types/fiobj.h:8,
                 from ../../src/tests/facil.io.cpp:2:
../../thirdparty/facil.io/lib/facil/core/types/fiobj/fiobject.h: In function ‘fio_cstr_s fiobj_obj2cstr(FIOBJ)’:
../../thirdparty/facil.io/lib/facil/core/types/fiobj/fiobject.h:499:59: sorry, unimplemented: non-trivial designated initializers not supported
     return (fio_cstr_s){.buffer = (void *)"null", .len = 4};
                                                           ^
../../thirdparty/facil.io/lib/facil/core/types/fiobj/fiobject.h:499:59: sorry, unimplemented: non-trivial designated initializers not supported
../../thirdparty/facil.io/lib/facil/core/types/fiobj/fiobject.h:505:61: sorry, unimplemented: non-trivial designated initializers not supported
       return (fio_cstr_s){.buffer = (void *)"null", .len = 4};
                                                             ^
../../thirdparty/facil.io/lib/facil/core/types/fiobj/fiobject.h:505:61: sorry, unimplemented: non-trivial designated initializers not supported
../../thirdparty/facil.io/lib/facil/core/types/fiobj/fiobject.h:507:62: sorry, unimplemented: non-trivial designated initializers not supported
       return (fio_cstr_s){.buffer = (void *)"false", .len = 5};
                                                              ^
../../thirdparty/facil.io/lib/facil/core/types/fiobj/fiobject.h:507:62: sorry, unimplemented: non-trivial designated initializers not supported
../../thirdparty/facil.io/lib/facil/core/types/fiobj/fiobject.h:509:61: sorry, unimplemented: non-trivial designated initializers not supported
       return (fio_cstr_s){.buffer = (void *)"true", .len = 4};
                                                             ^
../../thirdparty/facil.io/lib/facil/core/types/fiobj/fiobject.h:509:61: sorry, unimplemented: non-trivial designated initializers not supported
nativejson.make:206: recipe for target '../../intermediate/release_x64/gmake/x64/release/nativejson/facil.io.o' failed

Workaround: Remove Facil.io completely

rm src/cjsonlibs/facil.io_all.c
rm src/tests/facil.io.cpp
rm -rf thirdparty/facil.io/

Rebuild

cd build && ./premake.sh && cd gmake && make -f benchmark.make config=release_x64 && make -f nativejson.make config=release_x64 && cd ../..

---

Having all the workarounds applied, the benchmark builds and runs without crash.

Complete set of fixes is available from my branch https://github.com/mloskot/nativejson-benchmark/tree/ml/issue-102-add-workarounds-to-build

[13steinj]

So uh, I thought I'd make a 2021 update to this issue:

Only some of the workarounds are needed (others appear unnecessary).

In my case, I disabled 1. MemoryStat and 4. ULib as mentioned by @tomolt. I also fixed/disabled configuru and facil.io as mentioned by @mloskot . However, it appears the complexity in the make-setup is unneeded and provided for one via the main Makefile.

After all of this, I found new issues.

1. The results makefile needs minor edits

In the clean directive of results/makefile, add a - preceding rm. This will allow for failure in case no html files exist. Now, the main Makefile can be used rather than a number of commands, simply via make clean all. clean directive is necessary because you (may) need to re-run premake (done via the setup directive internally) and such.

2. simdjson conformance test is broken

The simdjson conformance test is missing a variable initialization (oddly). In the conformance test, function ParseString, line 127, a parser is referred to but never declared / initialized. Adding simdjson::dom::parser parser; appears to fix this issue.

3. ujson header is missing an STL include

thirdparty/ujson/ujson/ujson.hpp refers to std::out_of_range, but only includes <exception>. You need to add an include for <stdexcept> which contains std::out_of_range (among other exception classes).

4. The v8test fails

Fails to compile because of a non-existent header "libplatform/libplatform.h". I was unable to determine which header this "actually" referred to, so I just removed the test by renaming it-- mv src/tests/v8test.cpp src/tests/v8test.cpp.bak

5. Related to 4 seemingly, v8_base and v8_nosnapshot can't be linked

I changed the build/premake5.lua line 53 to remove these linked libraries (but not the others, on the off chance that such is needed). The (condensed) patch for premake5.lua:

diff --git a/build/premake5.lua b/build/premake5.lua
index 20916ff..749f2f2 100755
--- a/build/premake5.lua
+++ b/build/premake5.lua
@@ -53 +53 @@ function gmake_common()
-        links { "v8_base", "v8_libbase", "v8_libplatform", "v8_nosnapshot" }
+        links { "v8_libbase", "v8_libplatform" }
@@ -139 +139 @@ solution "benchmark"
-      linkoptions { "../../thirdparty/ULib/src/ulib/.libs/libulib.a" }
+--      linkoptions { "../../thirdparty/ULib/src/ulib/.libs/libulib.a" }
@@ -243 +243 @@ solution "jsonstat"
-      linkoptions { "../../thirdparty/ULib/src/ulib/.libs/libulib.a" }
+--      linkoptions { "../../thirdparty/ULib/src/ulib/.libs/libulib.a" }

6. Invalid pointers, segfaults galore

After all this, you should get some tests to run, but tests will fail. You should be able to just run make all at this point and not have to deal with cleaning and rebuilding everything entirely. In my case, on munmap_chunk() (invalid pointer) for the configuru, JVar tests, free() (invalid pointer) for the ujson test. SimpleJSON segfaults on stringify. I was uninterested in these tests, so I just removed them by renaming their .cpp files to .cpp.bak.

The simdjson test "gracefully" fails in verification, but seems to work in general, and I was actually interested in this test (I wanted to in fact, see if this benchmark suite matched simdjson's, for the obvious reason that simdjson claims to have beaten the author of this suite's json libary (RapidJSON).

7. php is required, apparently

Self-explanatory. Install php for your distribution.

8. Start a basic webserver in the repo's root

The html files are located in result, but relatively source some things from resource. So simply being in the repo's root and python3 -m http.server should do the trick. Many tests still...don't work / don't have metrics. But better than nothing?

---

I also don't know if I should trust metrics given the fact that there's an incredibly large amount of warnings generated upon compilation.

In case anyone is curious, a very-abridged summary of my findings:

CPU / OS / compiler detected as [email protected]_linux64_gcc10.2 (note I have dual CPUs, don't know if both were used or not).

Parse Time Graph:

Overall Conformance Graph:

Parse Conformance Graph:

The obvious: you'll probably want to run your own benchmark. I was unable to figure out how to order the pre-made Conformance Graphs, unfortunately.