…at least this one thing.
In the old 8-bit Microsoft BASICs, the tokenizer routine scans the input line and converts keywords into one or two byte tokens. This crunches the line down so it takes up less memory, and speeds up execution since the detokenizer just has to grab a token then run the related function. If BASIC were stored as ASCII text, it would have to scan the line text and parse multiple bytes to figure out it is a “PRINT” versus just looking at a one byte token.
In Color BASIC on the Radio Shack Color Computer, the tokenizer needs a space between variable names and keywords so it can tell where the variable stops and the keyword begins. For example:
IFSC=100THEN500
That line works fine, since “SC” (a score variable in this case) is followed by an equal sign. BASIC can find it easily. But if you were comparing against another variable:
IFSC>HSTHEN500
…you would get an ?SN ERROR on that line. I believe this is because Color BASIC “supports” longer variable names, but only honors the first two characters. Thus, these are all the same variable:
LO=1
LONG=1
LONGER=1
LONGEST=1
They are recognized as LO In the “IFSC>HSTHEN500” example, how is BASIC to know that we are using “HS” versus “HST” or even “HSTHEN” as a variable? It must be scanning until it hits something that is clearly not part of a variable name.
To make things even more confusing — variable names begin with a letter and can end with a number (or numbers, that are ignored), you can make variable names like this:
HSTHEN500=10
OK
PRINT HSTHEN500
10
And with that understanding, of course a space is required after variables before the following token. You have to add one between the variable and the next keyword:
IFSC>HS THEN500
BASIC can easily figure out where SC starts since a variable cannot contain a “>”, but it needs the space to know where HS ends. Thus, a FOR/NEXT loop like this requires spaces.
I fired up the “work in progress” Tandy CoCo engine in the cool Clock Signal emulator to test it out.
And then VIC-20 enters the arena…
But my VIC-20 did not need the space. Its parser can figure this out. Once again, I used the Clock Signal emulator which also has VIC-20.
When I was relearning VIC-20 a few years ago, I ran into this difference and wondered why. As I began revisiting it recently, I thought maybe the VIC-20 does not allow longer variable names, and has a better way to tell where a variable ends? I tried to use “START=1” but got an error. “Ah, it must not!” But then I found “ST=1” also did not work, and assumed “ST” must be some kind of reserved keyword.
Oddly, “AAAAA=1” works, and shows the same value as “AABBB=5”. For that test, it works like Color BASIC.
But “LO=1” works while “LON=1” does not work. Is LON some keyword too??? I clearly never learned all the keywords when I had my VIC.
SUPERMAN=1
SUPERGIRL=2
PRINT SU
2
I guess the VIC-20 BASIC does work like Color BASIC, but with some different keywords making the VIC-20 list of forbidden variables different than the CoCo’s list of forbidden variables. (See that link — I’ve now written about this at least three times, before this post.)
When I had my VIC-20, I did not know Microsoft had created the BASIC it uses. It just says “CBM BASIC” so I’d always thought (back then) that Commodore Business Machines wrote their own BASIC. It wasn’t until the modern Internet that I learned Microsoft wrote the 6502 BASIC used by Commodore. (If you have never dug into this, check it out sometime. There is some interesting history between Microsoft and Commodore. Apparently, that’s what led Microsoft into placing hidden “MICROSOFT” easter eggs in their other BASICs to be able to prove it was their code. But I digress…)
I find this interesting. I do not know 6502 assembly, but I am tempted to try to find the VIC-20 equivalent of “Color BASIC Unravelled” and see if I can learn how the Microsoft 6502 BASIC parser works compared to their 6809 parser.
But hopefully one of you knows, and can tell us all in the comments.
Until then…
It simply searches for reserved words first, and analyzes everything else afterward.
Another thing is that the commands have the space already stored in the token, so they don’t waste space on the line. For example,
printis abbreviated with theprinttoken.But it let me make a variable name with a keyword in it?
LON (and LONG, etc) doesn’t work because “ON” is a keyword. Its tokenizer literally just looks for tokens at every single character position in the line.
Interesting! I will experiment.
Tokens first, then the rest? That sounds more complex, but it saves valuable bytes.
As you are already on to, ST is a reserved keyword. In particular one that Commodore added, ST = status for I/O operations.
I wonder which other Microsoft Basic interpreters does this “the VIC 20 way” v.s. “The CoCo way”?
I need to research, but I have seen references to the “6502 source” versus the “6809 source”. I am wondering if there is a Family Tree (genealogy?) of Microsoft BASIC that would show where things split off.
Then there is the 6800 version. I would guess that it’s related both to the 6502 and the 6809 version. Would be interesting to know if the 6800 version were the first and then the 6502 and 6809 versions forked off from the 6800 basic in their own directions, or if the 6502 basic influenced the 6809 basic?
Seems like a disassembly is out there for the 6800 basic (and the source is available for both the 6502 and 6809 versions)
Another difference: The VIC-20 does not need “CLEAR” to reserve string space. It just works.
When I was re-learning VIC-20 a few years ago, I was understanding things I never understood as a kid. Where the memory locations little endien? On CoCo, PEEK(26)*256+PEEK(27) but I think I learned a different version of that on CBM BASIC.
IIRC we’ve discussed needing or not needing to set aside a specified size of memory for strings, and the reason for that is that the 6809 has a 16-bit stack pointer while the 6502 only has an 8-bit stack pointer. Thus the stack is always at $100-$1FF on a 6502, while on the 6809 it sits between string space and the rest of memory.
In theory you could create a BASIC where strings are stored above regular variables, all below the 6809 stack, but then you’d need to move all strings each time you allocate a variable. A compromise of sorts would perhaps be to move the strings a bunch of bytes in one go each time they need to be moved, so it doesn’t happen that often. Or perhaps let them start halfway through free memory and then move them whenever you run out of space for regular variables, strings or in theory stack. Would require more code and would have slowdowns that would be hard to determine when they happen.
It kind of surprises me that Tandy+Microsoft didn’t just set aside a fixed memory space for the stack though. Like there must be a finite amount of stack space that is needed when running BASIC and whatever disk/whatnot things also run, kind of sort of. When using BASIC to load something that’s not basic then you can just avoid strings, or manually set aside a larger stack size before using strings.