This library allows to use the NDB API of RonDB in Common Lisp programs and thus write client applications that talk to RonDB. RonDb is a key-value store providing low latency, high throughput, and high availability (actually, Class 6 Availability or "six nines").
Parallel to the key-value stores capabilities RonDB also provides SQL capabilities. As RonDB is an open source distribution of NDB Cluster, which is the distributed database system underlying MySQL Cluster, these SQL capabilities are actually provided by running MySQL Servers on top of RonDB.
As a consequence, you can use SQL to setup new databases, tables, indexes and so forth, and also to load or export data, while you can use the NDB API to efficiently retrieve data, for example, by making use of specialized indexes. Look at the instruction on how to test this library for an example for this.
NOTE: This is an early release and the library still evolving as of 2022-10-25. MGR
The library bases on a FFI binding code that was generated using SWIG 3.0.12. It consists of a complete low-level and a nicer high-level interface.
The high-level interface makes just those parts of the NDB API available
that I need to implement a new database backend based on RonDB for our
graph store but in a much nicer fashion, with scope-based macros such
as ndbapi:with-ndb, ndbapi:with-ndb-transaction et cetera, but you
can also explicitly free the objects with ndbapi:ndb-free-object or
just let the the GC take care of them, as the constructed C++ objects
are hooked up into SBCL's garbage collector with
thin Lisp wrapping classes so that they get properly freed by the respective
destructors if they are not referenced anywhere anymore.
The low-level interface makes the full NDB API available. But is is
hard to use as it includes no class hierarchy, and thus exhibits long
names, and implements no overloading. That is, the overloaded method
myNdb.startTransaction(), for example, is available as the ten
separate Lisp functions from _wrap_Ndb_startTransaction__SWIG_0 to
_wrap_Ndb_startTransaction__SWIG_9 with varying arguments.
Care has been taken that NDB is not deinitialized by NDB_END if there is still a cluster connection, or a cluster connection is not freed as long there are NDB objects that were created using it. Also double freeing is avoided, so you can combine the explicit and automatic approaches and still be sure that eventually the GC will free the resources when you have missed one.
The exported interface is defined as the package :ndbapi
in file src/interface.lisp. The implementation
is in the subdirectory src/, the examples in example/,
and the ASDF system definitions in the
top level directory in the files ndbapi.asd and
ndbapi-examples.asd.
An example application is available in file
examples/ndbapi-simple-scan.lisp.
For comparison a C++ version of the same example is implemented in file
examples/ndbapi_simple_scan_in_cpp/ndbapi_simple_scan.cpp.
It is based on other examples of ndb cluster but does not use a MySQL node.
The example uses a secondary index as a scan index to iterate over columns
in an example database observing bounds. See both files for details.
A second example application is in file
examples/ndbapi-scan-count.lisp.
It gives an estimated count of the rows matching a bound of lower
and/or upper limits. This example uses a number of more advanced features
compared to the first example to make the count efficient, for example,
interpreted programs to limit the row access to just one per partition (or fragment),
accessing special values (or pseudo columns), and masking of columns.
Please read the long comment at the beginning of the file
examples/ndbapi-scan-count.lisp
for a more detailed explanation.
The installation instructions are in the separate file INSTALL.md.
This library bases on a FFI binding code that was generated using SWIG. As CFFI support was disabled in SWIG 4.0.0 and finally completely removed in SWIG 4.1.0, SWIG 3.0.12 was used, which is part of Ubuntu 20.04.2 LTS.
NDB API is a rather substantial C++ library and uses many things not
possible in C while the CFFI module of SWIG has quite a few
limitations and cannot handle some C++ constructs. Still, SWIG
successfully generates a complete C wrapping library (using extern "C")
named src/ndbapi_wrap.cxx and a lisp library
making all C definitions available to Lisp via CFFI constructs in
src/ndbapi.lisp. Also a higher level CLOS interface
gets generated as src/ndbapi-clos.lisp, which
contains generated classes and generic methods.
The C wrapping library exhibits a rather low-level interface as all
hierarchical class names are rolled out with prefixes and all
overloaded methods are exported as separate functions. That is, the
overloaded method myNdb.startTransaction(), for examples, becomes
available as the ten separate Lisp functions from
_wrap_Ndb_startTransaction__SWIG_0 to
_wrap_Ndb_startTransaction__SWIG_9 with varying arguments. Most of the time
the methods only vary in their arity, which would allow for a
simple dispatch by arity. But there are also some methods that have
the same number of arguments but of different types, as, for example,
NdbDictionary::getValuePtr(), the constructor of the Column class,
or Table::getColumn().
SWIG 3.0.1 just created CFFI bindings with the same lisp name for a family of overloaded methods, which conflicted, of course, while the CLOS interface just tries to call these function, ignoring the fact that there is overloading supposed to be involved.
In the end, after exploring other versions of SWIG (see section
"Exploration of other SWIG versions ..." for details), the CFFI
bindings generated by SWIG 3.0.1 were used, completely ignoring the
CLOS interface in ndbapi-clos.lisp. This simple CFFI interface could
be generated for the full NDB API interface and seemed complete.
Afterwards, many translation problems or errors were manually fixed.
I have change the conflicting Lisp names in an automated fashion to
also contain the SWIG-* postfix, to have, for example, the C function
_wrap_new_Ndb__SWIG_1 available as new_Ndb/SWIG-1 in Lisp. Simple
dispatch arity is implemented in file src/overloading.lisp
by the macro ndbapi.ffi.overloading::overload-function-by-arity and
used for some methods.
I just focused on having the parts available that I need to implement a new database backend based on RonDB for our graph store rather then exporting everything. And to have this core available in a safe fashion so that the C/C++ NDB objects are properly constructed and always destructed.
An attempt was made to use the enhanced variant 'swig-cpp-to-cffi-for-common-lisp-enhancements` of SWIG which was authored by Jason E. Aten in 2011, who spent quite some time to make the CFFI module for SWIG work for "a large codebase of C and C+". Sadly, this version choked on some parts of the NDB API, as, for example, structs with constructors, and it just exited with a SEGFAULT. It was possible generate a version for a smaller subset of the NDB API, after taking out problematic parts of it that were determined by having a version of SWIG with debug logging activated and running it from within GDB.
This version of SWIG actually contained something that Aten describes as
"a synthesized dispatcher that allows natural calling of overloaded
constructors by keyword specification is provided". Meaning that for
each overloaded method first a function is generated which implemented
dispatch by arity and then for each variant generic methods are generated
that dispatch on the types. But the methods that actually would use it, as
the above mentioned NdbDictionary::getValuePtr(), seem to be just missing
from the CLOS interface.
In the end, this attempt was aborted, as the generated interface seemed to be incomplete even when only parts of the NDB API were translated.
It is noted that Nikolai Matiushev attempted to ("Revive CFFI Common Lisp support") in 2021](swig/swig#1966) but this seems to "lack an C++ support". Still, if that work is continued it might lead to a nicer automatically generated high-level interface.
Copyright (c) 2022 Max-Gerd Retzlaff [email protected], Datagraph GmbH.
This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License, version 2.0, as published by the Free Software Foundation.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License, version 2.0, for more details.
You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
A copy of the license is also reproduced in the file LICENSE in the top level directory of this repository.