Category of Bug / Issue

Converter bug

Current Behavior

Using transpile can sometimes hang rather than complete normally. This typically happens when using version 0.1.20 or later of the BB plugin. This can happen when the underlying dbconv process completes and the BB plugin is sending the conversion results back to the CLI. The logs from the BB plugin, written to stderr, are never visible in the CLI.

Expected Behavior

The CLI should not hang, and the logs from the BB plugin should be processed normally irrespective of their size.

Steps To Reproduce

Install the following versions:

• Lakebridge 0.11.0
• BB 0.20.0

During the configuration questionnaire choose a SQL dialect such as mssql and ensure that the BB backend is configured.

Generate a SQL file that is larger than 64KiB in size (e.g. 100KiB), then try to convert it via databricks labs lakebridge transpile. The SQL file will convert, but the CLI will hang because the stderr processing fails if more than 64KiB is present on a single line.