Goa: add support for Jam and h/o/g ?

[ttcoder]

Picking up the discussion with @nfeske on the mailing-list, about taking advantage of Be apps on SculptOS : this might be done with new repos/ports/*.port files or (more likely!) with Goa projects.

If there's ever a wish to try this out, I've just pushed a few commits with simple apps:

• DeskCalc scientific calculator
• Sudoku game
• the previous demo "classics" are still there too, including Mandelbrot

By working on such 'easy' ports we would detect early issues and porting challenges in a relatively simple environment, before tackling more complex ports.

In the abstract, I guess Goa should perform these tasks:

• download/expand the tarball or zip at https://chiselapp.com/user/ttcoder/repository/genode-haiku/zip/_an_hex_hash_might_be_used_here_ (EDIT: let's use the latest commit : 25e4905634
• configure the build system's environment vars (e.g. $GenodeRepos, possibly the toolchain's rev too) [edit]
• call jam haiku.lib.so
• call jam DeskCalc
• create a package for the lib
• create a package for the app
• test: deploy the library, deploy the app, does the calculator show up in a Motif window as it should ?

In practice, there will be things to work out, to make haiku.lib.so work with standard SculptOS, like configuring it to not use xattr:

• GE_XATTR_DISABLED=1 (otherwise linking will fail with "no such Genode function: path_for_genode_fd()")
• GE_DECORATIONS_AND_WM=1 (otherwise haiku.lib.so creates client-side decorations etc which would conflict with Genode's wm and decorator)

So that would mean building the lib and buliding the app by invoking these:
GE_XATTR_DISABLED=1 GE_DECORATIONS_AND_WM=1 jam -q -j2 haiku.lib.so
and
GE_XATTR_DISABLED=1 GE_DECORATIONS_AND_WM=1 jam -q -j2 DeskCalc

And in order to invoke that, one would need to build the "reference" Jam (rather than use ft-jam or lib-boost jam, which are not quite compatible). (we don't want to go again through troublesome debugging that occured when I suggested Norman use non-compatible Jam variants on Linux, I am still feeling guilty about that!)

Other possible issues:

• my jamfiles are a bit messy currently, in that they don't (all) build to genode/build_dir, some build inside the fossil tree (!) Should probably fix that.
• they also hardcode building for x86_64, but the whole Be code is probably not completely in shape for compiling on Arm etc anyway, so it may be too soon to make the jamfiles architecture-agnostic yet.

If you ever tackle this, hit me with any question you have, I'll do my best to answer, and tweak my jamfiles as needed.

[cnuke]

I am going to take a look at your repository and will try to reproduce building and running the simple examples. I will report back any obstacles I encountered.

Adding general support for jam like was done for e.g. cmake to goa and providing HoG in way that it can be picked up easily sounds reasonable to me.

[ttcoder]

In case it helps I sync'ed the Setup page so that it mentions the newly added "GE_HAVE..." defines mentionned above, and also did a "from scratch" build (fossil side only) in an empty directory like this:

mkdir hog && cd hog
fossil clone http://chiselapp.com/user/ttcoder/repository/genode-haiku/  genode-haiku.fossil
fossil open genode-haiku.fossil
cd genode-haiku
export GenodeRepos=/GenCharlie/dev_ge/ge23_11/genode/repos
GE_XATTR_DISABLED=1  GE_DECORATIONS_AND_WM=1  jam -q -j4 DeskCalc

It completed in a couple minutes without error, as pre-requirements were met : 1) the "jam" build tool and 2) genode/repos with headers and build-dir with expected contents

It ended up building haiku.lib.so automatically by just mentionning "DeskCalc", since haiku.lib.so is a dependency of the latter, so I guess there's no need to use two build lines, one suffices in fact.

[cnuke]

Following your nice documentation I was able to build and run DeskCalc (via a custom run-script) and also hog-demos.iso6 without any immediate problems - a pleasent endeavor. For jam I checked out the Haiku buildtools repository and built and installed the binaries. Since I only have access to the current tool-chain version 23.05 I had to adjust the check in the BeBuild.h header and set the tool-chain related variables (I have not tested much but on the surface things appear to work using the newer version).

I started to look into your Jam* files and used jam -da -dc -dx … to get a peek under the hood just to wrap my head around a possible jam and for that matter haiku.lib.so integration. As most things center around depot archives these days
providing haiku (or hog) api, src and bin archives that are used by apps like DeskCalc is natural. Seeing how your build-system is currently tied directly to Genode's build directory as you already mentioned - which is not available in goa - makes me wonder if assembling such archives manually for the time being and focusing more on the application side of things is a reasonable interim step.

(Commit 0fb4af8 adds your xattr-related function discussed in genodelabs/genode#4346 in an adapted form, including the caveats, we could upstream from my point of view.)

[ttcoder]

Agreed, apps could get precedence. I took a look at the api and src tarballs of SDL (since that library might more or less mirror our haiku library situation):
https://depot.genode.org/genodelabs/api/sdl/
https://depot.genode.org/genodelabs/src/sdl/
And it looks involved indeed. After extracting the archives I see that the api and used_api files are surprisingly short, just a few lines, but there is a big-time long list in the lib/symbols/sdl text file, and I expect the haiku.lib.so symbols list to be much longer than the SDL one.

I see Goa has this script, presumably used by the "goa extract-abi-symbols" command : https://github.com/genodelabs/goa/blob/9919605309011e71c5b306b46f35f250078d8333/share/goa/abi/abi_symbols#L4
Wonder if that's the one used for managing Qt and SDL, and if it could also help with creating the symbols list for haiku.lib.so (C symbols and C++ classes symbols), or if it's completely unrelated.
Alternatively if necessary I could try to replicate that script in the Jamfiles/Jamrules. Though even after running the script, the produced symbol list will need manual curation I guess.

Anyway,
it makes sense to craft a temporary deployement for haiku.lib.so, and focus on apps for now. Talking of which, there is also FtpPositive which we could test (it has few external dependancies), and next week I'll look into adding a couple more apps I have in mind.

Today's news: I've started pushing new commit for haiku.lib.so:
I'm enabling so-called 'layers' 7 and 8, to expand the library with TrackerKit and MediaKit classes. When I'm done (probably tonight) it should be possible to link more demanding apps like Sudoku against haiku.lib.so without getting "missing symbols : Tracker FilePanel etc" errors.
One of the commits also fixes the toolchain reference (from 21.05 to 23.05 etc) and the BeBuild.h header.
Later on I'll also change my code to refer to __get_fdpath() instead of path_for_genode_fd().

[trimpim]

I see Goa has this script, presumably used by the "goa extract-abi-symbols" command : https://github.com/genodelabs/goa/blob/9919605309011e71c5b306b46f35f250078d8333/share/goa/abi/abi_symbols#L4
Wonder if that's the one used for managing Qt and SDL, and if it could also help with creating the symbols list for haiku.lib.so (C symbols and C++ classes symbols), or if it's completely unrelated.

This is indeed used to create the list of ABI symbols via goa extract-abi-symbols. This uses the following as input:

• the artifacts file
• the results of goa build

With this it extracts the symbols for all .lib.so files it finds in the list of artifacts. For each it creates a file in the symbols directory.

[cnuke]

Agreed, apps could get precedence. I took a look at the api and src tarballs of SDL (since that library might more or less mirror our haiku library situation): https://depot.genode.org/genodelabs/api/sdl/ https://depot.genode.org/genodelabs/src/sdl/ And it looks involved indeed. After extracting the archives I see that the api and used_api files are surprisingly short, just a few lines, but there is a big-time long list in the lib/symbols/sdl text file, and I expect the haiku.lib.so symbols list to be much longer than the SDL one.

I'll spend a day this week and give this approach a try (assembling by hand) with DeskCalc…

(The symbols files should be curated anyway as they initially contain all found symbols and this is sometimes undesirable.)

Anyway, it makes sense to craft a temporary deployement for haiku.lib.so, and focus on apps for now. Talking of which, there is also FtpPositive which we could test (it has few external dependancies), and next week I'll look into adding a couple more apps I have in mind.

… and if that works we could focus on other applications as well (icon-o-matic would be nice to have).

[cnuke]

Unfortately I did not manage to spend as much time as I originally planned last week. That being said, I produced a bash script that assembles the required depot archives and more closely looked into the Jamfiles and how the actual build-system (or rather the rules used by the these files) is provided by the Jamrules. I have not yet looked at the vanilla Jamfiles used by Haiku but I assume the rules are somewhat similar.

So my basic idea for now is to provide the necessary hooks for jam in Goa to pull in the provided Jamrules in the HoG API archive but extended them either via the command-line (or by a generated rules files if that's not feasible) that picks up all archives from the used_apis file and features the required include paths and libraries to be linked against.

[ttcoder]

If you didn't do so already, maybe use "jam t1" as an interim milestone, if you'd rather get a minimal use-case working in full before embarking on a whole-scale one. Running jam -n t1 will outline a minimal working scenario with a haiku.lib.so that is reduced to just a small selection of the smaller classes (mainly BString and BMallocIO). Once you get that working in Goa, it's "just" (cough, cough) a matter of scaling up to the rest of the classes.

I just tried "jam t1" now and realize it was broken a couple commits ago though, it no longer links. To restore it one needs to edit haiku-on-genode/kits/app/Jamfile and comment out all refs to Roster(..).cpp

To recap the big picture on the way Jam works, there is 1) a Jambase that is embedded inside the jam.exe binary, with a few built-in rules (Library to build a lib, Main to build a simple exe, SubDir/SubInclude to organize deep hierarchies, etc) ; then 2) there is Jamrules where I override those built-in rules and provide new ones -- some of the Jamrules contents is actually in separate files in the jam/ folder and included into Jamrules, as it was getting too big ; then 3) there are the various Jamfiles that rely on those defined rules indeed.

Re-using the Jamfiles involved in building haiku.lib.so, but without re-using Jamrules (if I understand you correctly) sounds like it would be the way to an elegant/clean build for SculptOS indeed, and allow targetting the api instead of the (forbidden/not-provided) Genode objects dir. I need to wrap my head around that, hmmm.

Looking at the "Application Kit" jamfile again, I think a reduced Jamrules might need to provide a (re)implementation of:

• SubDir : sets some variables (like TOP and SUBDIR). BTW when working on that stuff I tend to refer to the original https://git.haiku-os.org/buildtools/plain/jam/Jambase file to look at the definition of SubDir etc.
• SubDirHdrs : paths for headers
• HaikuKit : this is the one that actually builds (part of) haiku.lib.so and is half-nifty-trick half-hack. The trick is that it uses the together rule modifiers (just like 'piecemeal' etc) via the LinkSolib rule. That means in each case, the .cpp file is compiled into an .o object (taking advantage of the locally defined header paths) as usual, BUT the linking is deferred : for dealing with .o objects, all calls to that rule in all Jamfiles are read and consolidated together first, and then the linker is called, linking it all in one step, instead of doing interim linking stages in each kit as interims of the final haiku.lib.so

Upstream-wise,
I didn't re-use the actual Haiku jam stuff because it's too complex (does more than I want) and wouldn't fit well with Genode. Actually it all evolved as a replication of the Genode build system, and then evolved to be (mostly) compatible with Haiku Jamfiles so that I can import the apps and compile them with minimal build tweaks ^^.

Latest:
Started working on Vision IRC this morning. This one I will not import into my fossil repo, but will instead configure its build as an "out of tree" port, to demonstrate how that can be done and make it easier to make ports in the future. Once Vision works I'll get started on Icon-o-matic (that one will be in-tree since it's part of the Haiku distribution). From my initial foray into that latter one, it's going to be a challenge, but if I can pull it off there'll be benefits for h/o/g as well, due to importing the AGG graphics library (hog lacks graphic rendering routines).

[cnuke]

Thanks for your explanation and thoughts.

If you didn't do so already, maybe use "jam t1" as an interim milestone, if you'd rather get a minimal use-case working in full before embarking on a whole-scale one. Running jam -n t1 will outline a minimal working scenario with a haiku.lib.so that is reduced to just a small selection of the smaller classes (mainly BString and BMallocIO). Once you get that working in Goa, it's "just" (cough, cough) a matter of scaling up to the rest of the classes.

[…]

That sounds reasonable. I used the 6ish target because that is what I already had at hand.

To recap the big picture on the way Jam works, there is 1) a Jambase that is embedded inside the jam.exe binary, with a few built-in rules (Library to build a lib, Main to build a simple exe, SubDir/SubInclude to organize deep hierarchies, etc) ; then 2) there is Jamrules where I override those built-in rules and provide new ones -- some of the Jamrules contents is actually in separate files in the jam/ folder and included into Jamrules, as it was getting too big ; then 3) there are the various Jamfiles that rely on those defined rules indeed.

Re-using the Jamfiles involved in building haiku.lib.so, but without re-using Jamrules (if I understand you correctly) sounds like it would be the way to an elegant/clean build for SculptOS indeed, and allow targetting the api instead of the (forbidden/not-provided) Genode objects dir. I need to wrap my head around that, hmmm.

For the time being I deliberately did not address building haiku.lib.so but focused on what DeskCalc needs. So it is actually the other way around. The API archive should contain a special-tailored Jamrules file that sets all the necessary variables (includes paths and so on) and provides the rules required by the Jamfile of DeskCalc (as a first target). Of course certain settings like the tool-chain and additional dependencies pulled in via the used_apis file (like libc and stdcxx) need to somehow extend that information.

Looking at the "Application Kit" jamfile again, I think a reduced Jamrules might need to provide a (re)implementation of:

* SubDir : sets some variables (like TOP and SUBDIR). BTW when working on that stuff I tend to refer to the original https://git.haiku-os.org/buildtools/plain/jam/Jambase file to look at the definition of `SubDir` etc.

* SubDirHdrs : paths for headers

* HaikuKit : this is the one that actually builds (part of) haiku.lib.so and is half-nifty-trick half-hack. The trick is that it uses the `together` rule modifiers (just like 'piecemeal' etc) via the LinkSolib rule. That means in each case, the .cpp file is compiled into an .o object (taking advantage of the locally defined header paths) as usual, BUT the linking is deferred : for dealing with .o objects, all calls to that rule in all Jamfiles are read and consolidated together first, and _then_ the linker is called, linking it all in one step, instead of doing interim linking stages  in each kit as interims of the final haiku.lib.so

Yeah, that is one of the reasons I started with the example application first - for the moment the HoG depot archives are a given and if the example application is built successfully I think it is easier to address HoG itself. At least for someone like me that hasn't used jam in years ☺.

Upstream-wise, I didn't re-use the actual Haiku jam stuff because it's too complex (does more than I want) and wouldn't fit well with Genode. Actually it all evolved as a replication of the Genode build system, and then evolved to be (mostly) compatible with Haiku Jamfiles so that I can import the apps and compile them with minimal build tweaks ^^.

[ttcoder]

In the latest commit : https://chiselapp.com/user/ttcoder/repository/genode-haiku/info/b566ab69ea97fc4c
I've added an alternate way to build an app which I believe might help : one may now use a Jamfile that is 'detached' from the hierarchy (it does not call SubDir at all), and stands alone (with just one companion 'support' jam file to back it up).

The "infrastructure" jam file, it might not be impossible(?) to use with little or no changes. The commit also includes an "example port" recipe Jamfile which could serve as inspiration (as I did below with Mandelbrot) for compiling in-tree apps. The example-port Jamfile in that commit is for Vision and is not completely trivial, as it involves downloading and unzipping an archive -- but here's a simplified variant for building Mandelbrot (it could also be DeskCalc) :

# Jamfile for building Mandelbrot as a out-of-tree port (!)

# General/mandatory config
app_name = Mandel_port ;
SEARCH_SOURCE += ./mandelbrot ;
 
list_src =
 	Mandelbrot.cpp
 	FractalEngine.cpp
 	;
 
# App-specific : Mandelbrot needs special header paths
list_private_headers = interface shared system ;
 
# Situation-specific : we are not in the usual place for doing ports, the path to top is shorter
HoG_TOP = .. ;
 
# We may now call the 'generic port engine', parametrized by the above setup
include ../hai-ports/_contrib_remote/Jam_generic.inc ;

You may write that contents in "the_above_jamfile" inside misc-apps. Then to invoke it, cd to the directory and invoke jam, instructing it to use that special 'detached' jam file instead of the Jamfile which already exists in that folder:

cd misc-apps
JAMFILE=the_above_jamfile  jam -q

With that kind of simplified infrastructure, your work might be easier : the dependencies are laid bare, easier to spot since the files are smaller. And it is easier to customize them for Goa, since one can spot how they are directed with configuration variables :

• LayerLevel: defaults to 7, easy to override on the command-line (-sLayerLevel=6 for instance)
• LOCATE_ToBuildFolder : that one might need a little more push as it only allows configuring the target folder name, not its location
• SetupHeadersHoG : it's now clear where that rule is called
• etc

Next I'll take a look at Icon-o-matic, so you might not hear from me for a while 😁

[ttcoder]

Icon-O-Matic wasn't all that hard, once I really got into it, it was fairly straightforward :

Commits pushed today to chiselapp. Seems to work, but I'd need to gain a better understanding of the interface, and run it side by side with the original to compare behavior. Something to explore for the 24.05 dev cycle I guess.

[cnuke]

That's awesome ☺.

(I haven't had the time in the last weeks to further work on this topic myself but I'll pick it up again probably after the release.)

[nfeske]

@ttcoder your screenshot made my day! It took me a white to identify what's visible in the upper-right corner of the Genode window. ;-)