v1.16.0

New Functionality

• Improvements to generated C++ code

  We continued our work to improve the C++ code generated from Spicy grammars by removing artifacts from automated parser construction, and observed throughput improvements of 20-60% in microbenchmarks. These improvements are powered by the existing machinery and new passes, mainly

  • GH-2249: Add new optimizer pass removing unused struct fields.

    Add optimizer pass that removes struct fields with no productive accesses. When such fields are printed via self they will appear as (optimized out). This optimization can be disabled on a per-type level by declaring them with linkage export, or globally by passing --strict-public-api to the compiler invocation.

  • GH-2031 which reduced the state generated parsers pass around, and reduced the number of temporary variables creating during parsing

  • GH-1781 which allowed putting more parser onto the stack eliminating heap allocations and streamlined memory access

  • GH-2247 to extend constant propagation, and also added copy propagation as an extension which took advantage of improved handling of non-mutating data access in CFG (GH-2238, GH-2254)

• Support for using Spicy on Windows

  We are grateful to Maor Hamami for his contributions to this release to make Spicy work under Windows. This included numerous changes for compiling with MSVC, fixes to codegen and to allow Windows processes to load HLTO files (which are always used even if not saved). He also added CI support for Windows which will allow us to catch regressions early. At this point Windows is not an officially supported platform, but we are happy to see support improve.

• GH-2131 added some compile-time validation of regular expressions. Previously incorrect regular expression would not fail until runtime.

• GH-2249: Allow specifying which optimizer passes should not run via environment variable HILTI_DISABLE_OPTIMIZER_PASSES.

• GH-2305: Optimize byte iterator increment operation

Changed Functionality

• GH-2279: String values in the C++ API are now passed as hilti::rt::String instead of std::string

• GH-2304: Produce better C++ code for switch statements

  We now require that cases in a switch statement are unique.

Bug fixes

• GH-2251: Fix coercion of enum values to bool
• GH-2259: Ensure that all enum types exposed by the runtime library have an Undef variant
• GH-2297: Doğukan Çağatay contributed a fix so DWARF debug info can be found with debugsource packaging
• GH-2302: Fix potential memory leak when using sinks
• GH-2300: Fix construction of tuples from map keys
• GH-2304: Fix operator resolving involving global constants
• GH-2321: Fix missing application of &bitorder for bitfields coming with a constraint.

Documentation

• GH-2186: library function names now include HTML anchors to simplify linking to them

v1.14.1

Bug fixes

• GH-2302: Properly clean up sink state
• GH-2187: Remove doctest minimum macOS version hack

v1.15.0

New Functionality

• Control-flow-based optimizations enabled and expanded

  We built out the control-flow-based optimization first introduced as an experimental feature in spicy-1.14, and with this release these optimizer passes are enabled by default. For that we made the framework more efficient and cleaned up issues in the implementation.

  We also introduced a new pass performing constant propagation (GH-2137, GH-2150) and reducing nested scopes to simplify data flow analysis (GH-1425).

• GH-2197: Support coercion from empty list to struct

  Spicy struct fields can already be declared with a &default value to reduce data duplication. We now allow constructing struct values from empty lists [] which effectively allows default-construction.

Changed Functionality

• GH-2183: Use dedicated C++ types for representing optional and tuple in the runtime library

• GH-2194: Reimplement hilti::rt::currentExecutable without external dependency

• GH-2201: HILTI_OPTIMIZER_PASSES has been removed

• GH-2204: Minimum required GCC version bumped to gcc-12

• GH-2222: Properly resolve and validate capture groups

  Previously, incorrect uses of regular expression capture groups (e.g., $1, $2) were emitted to C++ without any analysis, causing compilation failures or unexpected behavior. Now, we validate capture groups and reject invalid Spicy code that was previously accepted.

• GH-2245: Extend our notion of reserved C++ identifiers

  When generating C++ code from Spicy sources, we transform Spicy identifiers which are not valid in C++. We extended the set of identifiers we transform, which might be visible when directly working with the generated C++ code, like e.g., in custom host applications.

Bug fixes

• GH-2144: Function parameter and local variable name clash leads to C++ error
• GH-2154: Coercion from Null gets unhandled internal error
• GH-2162: Fix ASAN false positive on ARM
• GH-2175: Fix tuple assignment when coercing individual elements
• GH-2177: Fix C++ code generation for struct parameters passed as references
• GH-2184: FunctionParamVisitor fails to remove some redundant parameters
• GH-2205: Uglify internal identifiers
• Multiple fixes for issues reported by static analysis tools (clang-tidy, Coverity)
• zeek/zeek#5008: Do not normalize ID names inside type information

Documentation

• GH-2218: Add FAQ for code optimizers can remove

v1.14.0

New Functionality

• GH-2028: New interprocedural optimizations.

  We added infrastructure for performing interprocedural optimizations, and as a first user added a pass which removes unused function parameters in GH-2030. While this works on any code it is mainly intended to simply generated parser code for better runtime performance.

• GH-1697: Remove some dead statements based on control and data flow.

  We now collect control and data flow information. We use this to detect and remove "dead statements", i.e., statements which are not seen by any other needed computations. Currently we handle two classes of dead statements:

  • assignments which are override before being used
  • unreachable code, e.g., due to preceding return, break or throw

  The implementation for this is still not able to cover all possible Spicy language constructs, so it is behind a feature flag and not enabled by default. To enable it one needs to set the environment variable HILTI_OPTIMIZER_ENABLE_CFG=1 when compiling Spicy code with e.g., spicyc.

  We encourage users to test this compilation mode and if possible use the compiled parsers in production. If parsers compiled this way show the intended runtime behavior in tests they should also be fine to use in production.

Changed Functionality

• GH-2050: Prefer stdout over stderr for --help messages.

  Spicy tools now emit --help output to stdout instead of stderr.

• GH-2068: Allow disabling building of tests.

  We added a new CMake option SPICY_ENABLE_TESTS which if toggled on forces building of test and benchmark binaries; it is ON by default. This flag can be used by projects building Spicy to disable building of tests if they are not interested in them. We also provide a configure flag --disable-tests which has has the effect of turning it off.

• GH-1663: Speed up checking of iterator compatibility.

  We were previously using a control block which held a weak_ptr to the protected data. This was pretty inefficient for a number of reasons:

  • access to the controlled data always required a weak_ptr::lock which created a temporary shared_ptr copy and immediately destroyed it after access
  • to check whether the control block was expired we used lock instead of expired which introduced the same overhead
  • to check compatibility of iterators we compared shared_ptrs to the control data which again required full locks instead of using owner_before

  This manifested in e.g., loops often being less performant than possible. We now changed how we hold data to make iterating collections cheaper.

• GH-2086: Fix scope resolution of local variables.

  If usage of a local comes before its declaration, we now no longer resolve that usage to this local. It'll either be resolved to an upper layer ID (if there is one of the same name), or rejected if it's otherwise unknown.

• GH-2066: When C++ compilation fails, ask user for help.

  We do expect C++ code generated by Spicy to be valid, so C++ compiler errors in generated code are likely bugs. We now record the output of the C++ compiler in a dedicated file hilti-jit-error.log and ask users to file a ticket in case C++ compilation failed.

• GH-1660: When printing anonymous bitfields inside a struct, lift up the fields.

  This now prints, e.g., [$fin=1, $rsv=0, $opcode=2, $remaining=255] instead of [$<anon>=(1, 0, 2, 255)].

  In addition, we also prettify non-anonymous bitfields. They now print as, e.g., [$y=(a: 4, b: 8)] instead of [$y=(4, 8)].

• GH-1085: Allow registering a module twice.

  So far, if one compiled the same HILTI module twice, each into its own HLTO, then when loading the two HLTOs, the runtime system would skip the second instance. However, that's not really what we want: a module could intentionally be part of multiple HLTOs, in which case each should get its own copy of that module state (i.e., its globals).

  This change allows the same module to be registered multiple times, with the HLTO linker scope distinguishing between the instances at runtime, as usual. To make that work, we move computation of the scope from compile time to runtime, using the library's absolute path as the scope.

• GH-1905: Fix operator precedence in Spicy grammar.

  We fixed the precedence of a number of operators to be closer to what users would expect from other language like C++ or Python.

  • we reduced the precedence of the in operator
  • pre- and postfix operators ++ and -- now have same precedence and are right associative
  • unary negate was change to match the precedence of other unary operators.

• Switch compilation to C++20.

  Like Zeek Spicy now requires a C++ compiler. As part of this change we cleaned up the implementation to take advantage of C++ functionality in a number of places. We also moved from the external libraries linb::any to std::any, and ghc::filesystem to std::filesystem.

• Update supported platforms.

  We dropped support for the following platforms:

  • debian-11
  • fedora-40

  We added support for

  • debian-13
  • fedora-42

• GH-1660: Render all bitfield instances with included field names.

• GH-2099: Fully implement iterator interface for set::Iterator.

• GH-2052: Move calling convention from function to function type.

Bug fixes

• GH-2057: Fix bytes iterator dereference operation.
• GH-2065: Error for redefined locals from statement inits.
• GH-2061: Fix cyclic usage of units types inside other types.
• GH-2074: Fix fiber abortion.
• GH-2063: Fix C++ compilation issue with weak->strong refs.
• GH-2064: Ensure generated typeinfos are declared before used.
• GH-2044: Catch if methods are implemented multiple times.
• GH-2078: Fix C++ output for constants of constant type.
• GH-1988: Enforce that block-local declarations must be variables.
• GH-1996: Catch exceptions in processInput gracefully.
• GH-2091: Fix strong->value reference coercion in calls.
• GH-2100: Add missing deref operations for struct try-member/has-member operators.
• GH-2119: Fix missing inline functions in enum prototypes.
• GH-2142, GH-2134: Complete information exposed for reflection in typeinfo.
• GH-2135: Add &cxx-any-as-ptr attribute.

Documentation

• GH-1905: Document operator precedence.

v1.11.6

Bug fixes

• GH-2074: Fix fiber abortion.

  When aborting a fiber, we need to activate it once more, to then leave it for good by raising an AbortException. Problem was that that exception ended up being caught by user code because it was derived from std::exception`. This change removes the base class so that the exception is guaranteed to go back to the managing fiber code, where we just ignore it.

• GH-2073: Prevent throwing naked exception when yielding from aborted fiber.

v1.13.2

Bug fixes

• GH-2119: Fix missing inline functions in enum prototypes.

  Our prototype generation could miss function bodies for inline functions.

• GH-2074: Fix fiber abortion.

  When aborting a fiber, we need to activate it once more, to then leave it for good by raising an AbortException. Problem was that that exception ended up being caught by user code because it was derived from std::exception`. This change removes the base class so that the exception is guaranteed to go back to the managing fiber code, where we just ignore it.

v1.13.1

Bug fixes

• GH-2057: Fix bytes iterator dereference operation.

v1.11.5

Bug fixes

• GH-2057: Fix bytes iterator dereference operation.

v1.13.0

New Functionality

• GH-1788: We now support decoding and encoding to UTF16, in particular
  the new UTF16LE and UTF16BE charsets for little and big endian
  encoding, respectively.

• GH-1961: We now support creating type values in Spicy code. The
  primary use case for this is to pass type information to host
  applications, and debugging.

  A type value is typically created from either typeinfo(TYPE) or
  typinfo(value), or coercion from an existing ID of a custom type
  like global T: type = MyStruct;. The resulting value can be printed,
  or stored in a variable of type type, e.g.,
  global bool_t: type = typeinfo(bool);.

• GH-1971: Extend unit switch based on look-ahead to support blocks of
  items.

  In 1.12.0 we added support grouping related unit fields in blocks;
  there the primary use case were if blocks to group fields with
  identical dependencies. We now also support such blocks inside unit
  switch constructs with lookahead so one can write the following
  code:

  # Parses either `a` followed by another `a`, or `b`.
  type X = unit {
      switch {
          -> {
              : b"a";
              : b"a";
          }
          -> : b"b";
      };
  };

• GH-1538: Implement compound statements ({...}). This allows
  introducing local scopes, e.g., to group related code.

• GH-1946: string's encode method gained an optional errors
  argument to influence error handling. The parameter defaults to
  DecodeErrorStrategy::REPLACE reproducing the previous implicit
  behavior.

• GH-2010: bytes and string gained ends_with methods

• GH-1965: Add support for case-insensitive matching to regular
  expressions.

  By adding an i flag to a regular expression pattern, it will now be
  matched case-insensitively (e.g. /foobar/i).

• GH-1962: Add spicy-dump option to enable profiling.

Changed Functionality

• GH-1981, GH-1982, GH-1991: We now catch more user errors in defining
  function overloads. Previously these would likely (hopefully) have
  failed in C++ compilation down the line, but are now cleanly rejected.

• GH-1977: We now reject function overloads which only differ in their
  return type.

• GH-1991: We now reject function prototypes without &cxxname.

  Since in Spicy global declarations can be in any order there is no
  need to introduce a function with a prototype if it is declared later.
  The only valid use case for function prototypes was if the function
  was implemented in C++ and bound to the Spicy name with &cxxname.

• We have cleaned up our implementation for runtime type information,
  primarily intended for custom host applications.

  • type_info::Value instances obtained through runtime type
    introspection can now be rendered to a user-facing representation
    with a new to_string method.
  • The runtime representation was changed to correctly encode that
    tuple elements can remain unset. A Spicy-side tuple
    tuple<T1, T2, T3> now gets turned into
    std::tuple<std::optional<T1>, std::optional<T2>, std::optional<T3>>
    which captures the full semantics.
  • We added type information for types previously not exposed, namely
    Null, Nothing and List. We also fixed the exposed type
    information for result<void>.

• GH-2011: We have optimized allocations for unit fields extracting
  vectors which should speed up extracting especially small and
  medium-size vectors.

• GH-2035: We have dropped support for Ubuntu 20.04 (Focal Fossa) since
  it has reached end of standard support upstream.

• GH-2026: Speed up matching of character classes in regexps

Bug fixes

• GH-1580: Catch when functions aren't called.
• GH-1961: Fix generated C++ prototype header.
• GH-1966: Reject anonymous units in variables and fields.
• GH-1967: Fix inactive stack size check during module initialization.
• GH-1968: Fix coercion of function call arguments.
• GH-1976: Fix unit &max-size not returning to proper loc.
• GH-2007: Fix using &try with &max-size, and potentially other
  cases.
• GH-2016: Fix &size expressions evaluating multiple times.
• GH-2038: Prevent escape of non-HILTI exception in lower-level driver
  functions.
• GH-2047: Make sure bytes::to[U]Int returns runtime integers.
• GH-2049: Add #include <cstdint> for fixed-width integers

Documentation

• GH-1155: Document iteration over maps/set/vectors.
• GH-1963: Document assert-exception.
• GH-1964: Document use of $$ inside &{while,until,until-including}.
• GH-1973: Remove documentation of unsupported &nosub.
• GH-1974: Add documentation on how to interpret stack traces involving
  fibers.
• GH-1975: Fix possibly-incorrect custom host compile command
• GH-2039: Touchup docs style section.
• GH-1970, GH-2003: Fix minor typos in documentation.

v1.11.4

Bug fixes

• GH-2047: Make sure bytes::to[U]Int returns runtime integers.
• GH-2049: Fix building with GCC15.
• GH-1999, GH-2004: Adjust build setup for cmake-4.
• GH-2038: Prevent escape of non-HILTI exception in lower-level driver functions.
• GH-1918: Fix potential segfault with stream iterators.
• GH-1871: Fix &max-size on unit containing a switch.