Shader translation via HLSLcc

[0xf4b1]

This contains an attempt to use the HLSLcc library to translate DXBC to OpenGL. The translation works for the test DXBC object, however it crashes on dumped DXBC objects from unity for yet unknown reasons.

closes #11

[Masterjun3]

however it crashes on dumped DXBC objects from unity for yet unknown reasons

Yeah, it seems the dumped shaders from Unity are shuffled.
The DXBC that should appear at the start is somewhere in the middle of the file, and sometimes it's not even there in the first place! But when it's not there, I always found a cut-off XBC without the inital D, with the 01 00 00 00 Version number below it, showing that this is really part of the header.

This indicates to me that there's some shuffling going on.

EDIT: Oops sorry, this is a lot more obvious when you look at the data of the shader that can be parsed by asset tools. The part I mentioned with the DXBC is literally inside a "compressedBlob" section.

[Masterjun3]

Okay I researched this a bit more and after a lot of back and forth, it turns out the compressedBlob really is just LZ4 compression.
The section has no header and no footer, they're just immediate datablocks. Conventional LZ4 tools (like unix lz4) work if we manually add a good header and footer.
First adding the FOURCC 04 22 4D 18, and then adding some good flags 60 70 73, and then a small endian size of the compressedBlob. Finally at the end we append four 00 to signify the end.

prepend with:
	04 22 4D 18  60 70 73    FF 05 00 00
	| FOURCC  |  |FLAGS | |SIZE e.g. 0x5FF|

append with (does not increase the SIZE in header):
	00 00 00 00

After decompressing this, the file has some DXBC shaders that can be used with HLSLcc! Though, mine had 8 of those shaders! And there was some more stuff inbetween...

[0xf4b1]

This sounds interesting, what I did was to inspect the shader objects in AssetRipper, took the compressedBlob data, base64 decoded it and then cut off all the bytes before the DXBC bytes and then used it as input for hlslcc but this just crashed.

Does the output you are getting look reasonable? Would be a nice step forward if we can compile it and in the best case produce the same output like unity would do. Easiest would be to take a unity project and export with directx and opengl, then try to translate the shaders from directx to opengl and compare the result with the shaders in the opengl version.

Should this be working, the next issue would be to figure out how to write the whole shader objects as I described here because only replacing the compressedBlob section will most likely not work.