Liquibase fails to initialize databasechangeloglock table in Azure Databricks

[BillCM]

• liquibase installed on OSX via Homebrew: Version: 4.25.1
• DatabricksJDBC42-2.6.36.1062
• liquibase-databricks-1.1.2.jar
• Azure Databricks using Unity Catalog
• openjdk64-17.0.9

1. run liquibase --changeLogFile=changelog.databricks.sql generateChangeLog: success
2. add create view to the changelog
3. run liquibase --changeLogFile=changelog.databricks.sql update

• liquibase creates databasechangelog
• liquibase creates databasechangeloglock
  • but it fails to initialize the table, leading to 10 retries and eventual failure

[2024-01-05 16:36:32] FINE [liquibase.executor] Create Database Lock Table
[2024-01-05 16:36:32] FINE [liquibase.executor] CREATE TABLE development_sandbox.test.DATABASECHANGELOGLOCK (ID INT NOT NULL, LOCKED BOOLEAN NOT NULL, LOCKGRANTED TIMESTAMP, LOCKEDBY VARCHAR(255), CONSTRAINT PK_DATABASECHANGELOGLOCK PRIMARY KEY (ID))
[2024-01-05 16:36:35] FINE [liquibase.executor] -1 row(s) affected
[2024-01-05 16:36:35] FINE [liquibase.executor] Changelog query completed.
[2024-01-05 16:36:35] FINE [liquibase.lockservice] Created database lock table with name: development_sandbox.test.DATABASECHANGELOGLOCK
[2024-01-05 16:36:36] FINE [liquibase.lockservice] Failed to create or initialize the lock table, trying again, iteration 1 of 10
liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: Error executing SQL SELECT COUNT(*) FROM development_sandbox.test.DATABASECHANGELOGLOCK: [Databricks][DatabricksJDBCDriver](500618) Error occured while deserializing arrow data: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
	at liquibase.lockservice.StandardLockService.isDatabaseChangeLogLockTableInitialized(StandardLockService.java:219)
	at liquibase.lockservice.StandardLockService.init(StandardLockService.java:135)
	at liquibase.command.core.helpers.DatabaseChangelogCommandStep.checkLiquibaseTables(DatabaseChangelogCommandStep.java:145)
	at liquibase.command.core.helpers.DatabaseChangelogCommandStep.run(DatabaseChangelogCommandStep.java:91)
	at liquibase.command.CommandScope.execute(CommandScope.java:217)
	at liquibase.integration.commandline.CommandRunner.call(CommandRunner.java:55)
	at liquibase.integration.commandline.CommandRunner.call(CommandRunner.java:24)
	at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
	at picocli.CommandLine.access$1500(CommandLine.java:148)
	at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
	at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
	at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
	at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
	at picocli.CommandLine.execute(CommandLine.java:2170)
	at liquibase.integration.commandline.LiquibaseCommandLine.lambda$execute$2(LiquibaseCommandLine.java:391)
	at liquibase.Scope.child(Scope.java:195)
	at liquibase.Scope.child(Scope.java:171)
	at liquibase.integration.commandline.LiquibaseCommandLine.lambda$execute$3(LiquibaseCommandLine.java:366)
	at liquibase.Scope.child(Scope.java:195)
	at liquibase.Scope.child(Scope.java:171)
	at liquibase.integration.commandline.LiquibaseCommandLine.execute(LiquibaseCommandLine.java:363)
	at liquibase.integration.commandline.LiquibaseCommandLine.main(LiquibaseCommandLine.java:103)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at liquibase.integration.commandline.LiquibaseLauncher.main(LiquibaseLauncher.java:116)
Caused by: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: Error executing SQL SELECT COUNT(*) FROM development_sandbox.test.DATABASECHANGELOGLOCK: [Databricks][DatabricksJDBCDriver](500618) Error occured while deserializing arrow data: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
	at liquibase.executor.jvm.ChangelogJdbcMdcListener.query(ChangelogJdbcMdcListener.java:60)
	at liquibase.lockservice.StandardLockService.isDatabaseChangeLogLockTableInitialized(StandardLockService.java:216)
	... 25 more
Caused by: liquibase.exception.DatabaseException: Error executing SQL SELECT COUNT(*) FROM development_sandbox.test.DATABASECHANGELOGLOCK: [Databricks][DatabricksJDBCDriver](500618) Error occured while deserializing arrow data: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
	at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:98)
	at liquibase.executor.jvm.JdbcExecutor.query(JdbcExecutor.java:224)
	at liquibase.executor.jvm.JdbcExecutor.query(JdbcExecutor.java:232)
	at liquibase.executor.jvm.JdbcExecutor.queryForObject(JdbcExecutor.java:240)
	at liquibase.executor.jvm.JdbcExecutor.queryForObject(JdbcExecutor.java:255)
	at liquibase.executor.jvm.JdbcExecutor.queryForInt(JdbcExecutor.java:276)
	at liquibase.executor.jvm.JdbcExecutor.queryForInt(JdbcExecutor.java:271)
	at liquibase.lockservice.StandardLockService.lambda$isDatabaseChangeLogLockTableInitialized$2(StandardLockService.java:216)
	at liquibase.executor.jvm.ChangelogJdbcMdcListener.lambda$query$1(ChangelogJdbcMdcListener.java:54)
	at liquibase.Scope.child(Scope.java:195)
	at liquibase.Scope.child(Scope.java:171)
	at liquibase.executor.jvm.ChangelogJdbcMdcListener.query(ChangelogJdbcMdcListener.java:54)
	... 26 more
Caused by: java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500618) Error occured while deserializing arrow data: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
	at com.databricks.client.spark.arrow.ArrowBuffer.deserializeBatch(Unknown Source)
	at com.databricks.client.spark.arrow.ArrowBuffer.handleInitializeBuffer(Unknown Source)
	at com.databricks.client.hivecommon.api.HiveServer2BaseBuffer.initializeBuffer(Unknown Source)
	at com.databricks.client.hivecommon.api.RowsetBuffer.initializeBuffer(Unknown Source)
	at com.databricks.client.hivecommon.api.HS2Client.getRowSetInformation(Unknown Source)
	at com.databricks.client.hivecommon.api.HS2Client.fetchFromServer(Unknown Source)
	at com.databricks.client.spark.jdbc.DowloadableFetchClient.fetchNRows(Unknown Source)
	at com.databricks.client.hivecommon.api.HS2Client.fetchRows(Unknown Source)
	at com.databricks.client.hivecommon.dataengine.BackgroundFetcher.run(Unknown Source)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Caused by: com.databricks.client.support.exceptions.GeneralException: [Databricks][DatabricksJDBCDriver](500618) Error occured while deserializing arrow data: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
	... 13 more
Caused by: java.lang.UnsupportedOperationException: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available
	at com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil.directBuffer(MemoryUtil.java:167)
	at com.databricks.client.jdbc42.internal.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:228)
	at com.databricks.client.jdbc42.internal.apache.arrow.memory.ArrowBuf.nioBuffer(ArrowBuf.java:223)
	at com.databricks.client.jdbc42.internal.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:87)
	at com.databricks.client.jdbc42.internal.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:727)
	at com.databricks.client.jdbc42.internal.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:363)
	at com.databricks.client.spark.arrow.ArrowBuffer.deserializeBatch(Unknown Source)
	at com.databricks.client.spark.arrow.ArrowBuffer.handleInitializeBuffer(Unknown Source)
	at com.databricks.client.hivecommon.api.HiveServer2BaseBuffer.initializeBuffer(Unknown Source)
	at com.databricks.client.hivecommon.api.RowsetBuffer.initializeBuffer(Unknown Source)
	at com.databricks.client.hivecommon.api.HS2Client.getRowSetInformation(Unknown Source)
	at com.databricks.client.hivecommon.api.HS2Client.fetchFromServer(Unknown Source)
	at com.databricks.client.spark.jdbc.DowloadableFetchClient.fetchNRows(Unknown Source)
	at com.databricks.client.hivecommon.api.HS2Client.fetchRows(Unknown Source)
	at com.databricks.client.hivecommon.dataengine.BackgroundFetcher.run(Unknown Source)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)

[BillCM]

Ah, looks like I need Java 11 from this closed ticket. I'll test that
#64