log09.txt

    It's pretty wild what I *don't* yet have in this
    language:

        * Basic math operations
        * Conditionals
        * Loops
        * User-defined variables

    The easiest one to rectify, I think, will be some basic
    math operations. With that, I can at least have a decent
    RPN calculator.

    Before I do that, I'd like to now start separating the
    language primitives I used in the creation of the
    interpreter versus everyhing else.

    Putting everything else in another file and including it
    turns out to be super easy with NASM:

        %include 'stdlib.asm'

    So far, stdlib.asm contains just these:

        ps (print stack)
        inspect
        inspect_all
        all (all names)

    So now I'll...YUCK, nevermind! NASM doesn't report the
    line number of an error in an included file - it really
    pretends the included content is actually in the file
    that did the including. It was probably premature to
    split this up anyway.

    So everything will stay in meow5.asm as before.

    Okay, so the math operations are easy because I'm just
    popping the arguments, calling the CPU instructions, and
    pushing the answer:

$ mr
20 8 - ps
12
4 * ps
48
3 / ps
0 16
"4 divided by 3 is $ remainder $\n" print
4 divided by 3 is 16 remainder 0
bin
1011 0100 + ps
1111
1111 hex ps
f f
0beef ps
f f beef
+ ps
f befe
+ ps
bf0d
Goodbye.
Exit status: 0

    Increment and decrement! Bam! Easy!

45 inc ps
46
dec dec dec ps
43

    I love easy stuff!

    I'll need to figure out some more substantial stuff
    next. Variables, maybe. Or conditionals.

    Next week: I chose to implement variables first. Then I
    took some time off to rest a bit.

    Now I've got a first stab at 'var'.

    (I originally planned to have variables put their VALUES
    on the stack when called, but after I started
    implementing them, I've decided to follow yet another
    Forth convention: variables will put their ADDRESSES on
    the stack. It's just so much simpler and more flexible
    that way.)

    So far, 'var' borrows from three existing words:

        1. From 'colon', it takes the idea of getting the
           next token from the input stream as a name.

        2. From 'quote', it takes the compiling of the
           machine code to push the immediate value
           containing the address of the variable's memory.

        3. From 'semicolon' - actually, it doesn't borrow,
           it flat out *includes* semicolon.

    Does it work?

var foo
hex foo "Address of foo: $\n" print
Address of foo: 804c16c
var bar
hex bar "Address of bar: $\n" print
Address of bar: 804c189

    Seems plausible! So I guess I need new words to get
    values to/from that address and the stack.

    I think I'll call them "get" and "set" rather than the !
    ("store") and @ ("fetch") terminology from Forth.

    They're easy to write in assembly. Just a handful of
    instructions.

    Here goes:

var foo
42 foo set
: foo? foo "Foo contains $\n" print ;
foo?
./build.sh: line 34:  1465 Segmentation fault      ./$F
Exit status: 139

    Oops, I didn't even write that correctly. I forgot the
    'get' in the definition of 'foo?'. But it should still
    have printed the address of foo. So something has gone
    awry. Now I know what I'm doing tomorrow night.

    Next night: Okay, let's see where this crashes using
    GDB. First, Can I set and get the variable?

Starting program: /home/dave/meow5/meow5
var foo
55 foo set
foo get
ps
134525204 55

    Yes! Okay, I've got some extra garbage on the stack
    (looks like an address that I'm not cleaing up at some
    point). I'll deal with that in good time.

    But 55 on the stack totally means 'foo get' worked.
    Neat!

    And can I print it from a string in immediate mode?

"foo is $\n" print
foo is 55

    No problemo.

    Now in a compiled word where it crashed last night.
    (This time I remembered to do a 'get' as well):

: foo? foo get "foo=$\n" ;
foo?

Program received signal SIGSEGV, Segmentation fault.
0x0804b114 in token_buffer ()

    Yeah, so there we are. Apparently it crashes
    while...what? Trying to execute code in token_buffer?

    Well, I think the real problem is probably something
    fundamental. Like, I haven't really thought through how
    something works in a compiled word versus immediate
    mode.

    Next night: Okay, LOL, so first of all, I never
    implemented numeric literals for compile mode. So I
    can't even test strings like this:

$ mr
: x 5 "five is $" print ;
TODO: a new compile number word?Exit status: 0

    And second of all, I don't think I ever tested number
    interpolation (via '$' placeholders) in compiled words
    either. Because that crashes:

$ mr
: x "stack has $" print ;
5 x
./build.sh: line 34:  1390 Segmentation fault      ./$F
Exit status: 139

    So I should probably get those working. I'll start with
    the second problem and work my way back to variable
    printing:

        [ ] Get $ placeholders working in compiled words
        [ ] Implement compiled number literals
        [ ] Implement compiled variable get/set

    I'm not sure yet if that third item is even needed.

    Okay, so I'm reviewing my 'quote' definition and the
    thing that immediately stands out is that since I call
    'quote' from the interpreter in immediate mode, it
    should be "baking" in the placeholder value when the
    string is put in memory.

    So, for example:

        12

        : foo "I have $ coins." ;

        6

        foo

    Should be printing "I have 12 coins." instead of "I have 6
    coins."

    Instead, it's crashing.

    But before I fix that, I think I should re-think how
    this works. Because I'm pretty sure as a programmer, I
    would expect to have 'foo' output "I have 6 coins."

    I mean, both ways have advantages and disadvantages. But
    doing "late binding" on the placeholder seems much more
    sensible and useful. I want the string to reflect the
    value on the stack when I *run* 'foo', not when I define
    it.

    So I think what I need is for 'print' to contain the
    placeholder stuff, not 'quote'!

    Yuck, that's a pretty significant chnage. But I don't
    really see a way around it if I want the language to
    make any sense.

    So, the new TODO is gonna have to be:

        [ ] Make 'quote' just store the string with '$'
        [ ] Make 'print' evaluate '$' placeholders in
            strings (at "run time")

    I guess this will also require a new output buffer so I
    can build up strings to output. Or maybe I store them in
    my general "free" memory as temporary storage?

    Next night: Okay, let's get to this.

    Step one is to not have 'quote' replace placeholder '$'
    with numbers anymore. I'll just comment them out for
    now.

    Here's the simple example that crashed above:

: x "stack has $" print ;
5 x
stack has $

    Easy.

    Next, 'print' needs to print a number from the stack
    when it sees a '$' in the string.

    There are two ways I could do this:

        1. Make yet *another* copy of this string for
           temporary printing purposes that has the number
           string incorporated and print that.

        2. Print the string up to the place holder, then
           print the number, then print the next bit of
           string after the placeholder, etc.

    Hmmm... I'm not *loving* either of those options, The
    disadvantage of #1 is that I need to find another place
    to store the string - ideally temporary. The
    disadvantage of #2 is that it's more complicated.

    I feel like #1 can't be made "nicer". But maybe with
    some thoughtful design, #2 could be broken into
    palatable chunks of functionality.

    Next night: Okay, after some painful stack and register
    management, I've got it!

var foo
: foo? foo get "foo=$\n" print$ ;
55 foo set
foo?
foo=55

    Okay, so I stuck with the plan to put the '$'
    placeholder functionality in the printing mechanism
    rather than the string building mechanism ('quote').

    Rather than turn 'print' into an abomination, I simply
    added a 'print$' word that does number interpolation.

    Here you can see 'print' vs 'print$' for comparison:

42

"--$--" print
--$--

"--$--" print$
--42--

    And 'print$' really wasn't too bad. I mean, it's still
    quite compact. It was no fun to write.

    I like how it cleaned up 'quote' a bit as well. That
    word was way too long.

    Okay, I need to add a *bunch* of stuff to my test script
    next to double-check that what I have so far is
    rock-solid.

    I already know that something (semicolon?) is leaving an
    address on the stack that it shouldn't. So I'll be
    fixing that as well.

        [ ] Fix whatever is leaving an addr on stack

    Wait, I almost forgot the fun part: I was going to add a
    new printing convenience word to wrap up 'print$' plus a
    newline: 'say'. I love convenience!

1 2 + "I have $ coins." say
I have 3 coins.

    Ah! I love it so much. Yes! What a difference one little
    handy word makes.

    I'm going update my hello world examples with 'say' now.

    Done.

    And now a confession: I've not been happy with my
    Expect-based tests. Expect waits for the input you asked
    for (until a timeout is reached) and it is perfectly okay with getting things you
    *didn't* ask for. Which makes plenty of sense for
    interacting with some applications.

    But that does fit what I want to see, which is *exact*
    output. And I want *immediate* failure when something
    other than that is provided.

    No doubt I can force Expect to do that, but I'm gonna
    give shell scripts a shot.

    First of all, I can't remember if I've tried piping
    input into my interpreter yet, so here goes:

$ echo '"Hello" say' | ./meow5
Hello
Goodbye.

    And we can make sure that I get my exact output with a
    grep search:

$ echo '"Hello" say' | ./meow5 | grep '^Hello$'
Hello
$ echo $?
0

    grep's 0 exit status means match was found.

$ echo '"Hello" say' | ./meow5 | grep '^foobar$'
$ echo $?
1

    And 1 exit status is not found.

    So scripting this is no problem. Here's my shell
    function:

        function try {
            # -q tells grep to not echo, just return match status
            if echo "$1" | ./meow5 | grep -q "^$2\$"
            then
                printf '.'
            else
                echo
                echo "Error!"
                echo "Wanted:"
                echo "-------------------------------------------"
                echo $2
                echo "-------------------------------------------"
                echo
                echo "But got:"
                echo "-------------------------------------------"
                echo "$1" | ./meow5
                echo "-------------------------------------------"
                echo
            fi
        }

    And some simple tests:

        try '"Hello" say' 'Hello'
        try '"Hello" say' 'beans'

$ ./test.sh
.
Error!
Wanted:
-------------------------------------------
beans
-------------------------------------------

But got:
-------------------------------------------
Hello
Goodbye.
-------------------------------------------

    Cool, then I can re-write the Expect script with this
    and verify the functionality so far.

    Oh, and implement compiled numeric literals. That's
    still an open TODO.

    Next night: Cool as a consequence of the new test
    script, I've fixed the address being left on the stack
    (I had guessed 'semicolon', but it was actually 'colon'.
    I was pushing the source address to copy the token
    string for the new word/function name, but 'gettoken'
    was already doing that, so it was redundant.)

    I also implemented numeric literals in compile mode. I'm
    getting a LOT of mileage out of that x86 opcode to push
    an immediate 32 bit value onto the stack!

    Now my tests look like:

#   Input                  Expected Result
#   -------------------    ------------------------
try 'ps'                   ''
try '5 ps'                 '5 ' # note space
try '5 5 5 + ps'           '5 10 '
try '9 2 * ps'             '18 '
try '18 5 / ps'            '3 3 '
try '5 2 - ps'             '3 '
try '"Hello$\\\$\n" print' 'Hello$\\$'
try '"Hello" say'          'Hello'
try ': five 5 ; five ps'   '5 '
try ': m "M." print ; m '' say' 'M.'
try ': m "M." print ; : m5 m m m m m ; m5 '' say' 'M.M.M.M.M.'

    And they all pass:

$ ./test.sh
...........
Passed!

    However, it looks like 'var' is pushing one to many
    addresses because there's still one on my precious stack
    that fouls up this test:

$ ./test.sh
...........
Error!
Wanted:
-------------------------------------------
var x 4 x set x get x ps
4 
-------------------------------------------

But got:
-------------------------------------------
var x 4 x set x get x ps
134525172 4 134529332 
Goodbye.
-------------------------------------------

    But I'm getting there! Feeling good about having these
    tests to check against regressions as i add features.

    I'll debug the stack issue with a bunch of DEBUG
    statements to narrow it down like I did for 'colon'.

    Next night: Yup, in fact it was the exact same bug I had
    in 'colon'. Shame.

    By the way, I found both of these by just peppering the
    offending area my DEBUG macros and printing the value of
    the esp register to see where something was being left
    on the stack that shouldn't have been. It looked like
    this once I fixed it:

var x
var start: ffafb9c0
var copys: ffafb9c0
var push:  ffafb9bc
var end:   ffafb9c0
4 x set
x get ps
4
x ps
4 134529376
get ps
4 4

    Now I've fixed my test:

        try 'var x 4 x set x get ps' '4 '

    And all is well:

$ mt
............
Passed!

    And to finish it off, I need to test using (not
    creating...yet) variables within compiled words.

    The test:

        var x 
        4 x set
        : x? x get "x=$" say ;
        x?

    Expected result:

        x=4

    Crossing fingers:

$ mt
.............
Passed!

    Yay!

    So it looks like that checks everything off for this
    log:

        [x] Get $ placeholders working in compiled words
        [x] Implement compiled number literals
        [x] Implement compiled variable get/set
        [x] Make 'quote' just store the string with '$'
        [x] Make 'print' evaluate '$' placeholders in
            strings (at "run time")
        [x] Fix whatever is leaving an addr on stack (twice)

    See you in log10.txt!