The road to Basic(s)

[ggn]

So, something inbetween a blog post: How the heck did the Jaguar get Basic? And why two of them?


Let's rewind back to December 1st 2014. Raptor library was out and we were really doubtful anyone would pick up and use it (Der Luchs proved us wrong there, cheers!). For quite some time now (months, prehaps years) we showed the library to people who promised they'd do C or BASIC backends for it (you know who you are, you naughty people!). So I decided to take a look at that myself.

While evaluating what's out there CJ asked me if we could do Pascal as he kind of likes it. First hit for "free pascal compiler" was of course Free Pascal . It looked really nice and hackable, there was a 68000 port for it and a good looking IDE there as well (Lazarus). So all this was looking good. After 1 hour or so I even had the cross compiler for 68k ready to go! Alas, the 68k port of the libraries was horribly outdated and since I don't know Pascal that much (well, not at all actually!), compiling the standard library proved a bit too much for me so I abandoned this. (Of course if anyone wants to bring the 68k port to working shape I can definitely consider picking it up again). That was all done in 1 day.

Next day I started evaluating Basics. Compiled, interpreted, it didn't matter too much. A couple of suitable candidates were found, all written in raw 68k (which was good for me, no faffing about again ). In the end I picked EhBasic; it looked it was being updated (although we learned that its author unfortunately isn't with us any more ), it was a monolithic source file (just the way I prefer projects) and it looked easy to get working.

Withing 1 hour or two it booted up on ST (my preferred target as the debugging tools are better), and I quickly hacked in I/O and forcing it to load any ASCII file. Initially it was targetting the EASy68K environment which seems to have some sort of operating system for it so it could bring up a file selector etc. To honor the openness of the original project I decided to share the code changes to the public so I set up Github and Bitbucket repositories for it. So, success from the first day!

The 2nd day was spent tossing the code over between me and CJ, getting it to run on the Jaguar which wasn't too bad once we sorted out the assembler syntax differences. Next day EhBasic started up on VJ and printed something - yay! Then we simply neeeded to figure out how to add our own commands and functions, which it turns out that is quite easy - double yay!

Then it was over to CJ to interface EhBasic with raptor and after a few days it was ready! Hardware accelerated Basic interpreter for your Jaguar!

Of course, while benchmarking it, we did saw that the interpreter's speed isn't much to write home about. And line numbers might get on people's nerves. And we'd be probably be forced to write a compiler at some point. And we couldn't add too many extra commands because the parsing would slow down considerably (that and there doesn't seem to be room in the tokens for a lot of extra commands). All of these things were itching me. So, fast forward to December 10, and this IRC log snippet:

10:57 < ggn~> bcx raptor next?
10:58 < CJ^AU > if you want
10:58 < ggn~> maybe I'll look at this today

How hard could it be, right? After all a Basic's a Basic's a Basic! Well, here's an intro to bcx Basic: Like its acronym says, it's a Basic to C translator. Which means you get to type Basic or Basic-like code and it'll convert the source code into C code that you can compile afterwards. Which of course opened a few can o'worms for me in the process!

C. I really don't like that language. I mean I use it at home, I can understand why people like it and on which projects it's beneficial - but still meh! With such a positive attitude ( ) and some experience in my back I considered which one to use. Ideally it would have to be a cross compiler, without many dependencies, able to produce a Jaugar binary or compatible object files for linking, and of course free so I can distribute it on a zip file)

There's the one by Atari which is based on gcc v1.5 or something (meh), some good optimising compilers on the ST like Pure C (meh, would tie the developemnt down to having to use a 16/32 machine and/or emulators), vbcc (not bad, supported and can output Jaguar binaries from what I gather, but I really don't like its interface and I really have no clue how mature its optimiser is. Of course a lot of Amiga people say it's good. But still, would avoid it if I can) and modern cross-compiled gcc led by Vincent Rivière (good solid optimising compiler, needs Cygwin which is bad for my use case). While the last choice wasn't the most ideal and possibly vbcc would be more hassle-free, I chose gcc in the end.

So, starting to fiddle with bcx next. I found out that the compiler seems to produce C sources suitable for Windows only - Linux or anything else isn't supported although I dunno why, I think it wouldn't be that hard to make it cross-platform. Commenting out some header files in the produced source seems to have fixed this, so I was left with something that gcc could eat up and spit an .o file.

At this point I should maybe describe my work set-up. Obviously Cygwin would have to be used to run gcc. The bcx compiler itself is (again) a big monolithic file that is self-compiled, i.e. if you want to build a new version you have to run the old bcx compiler on itself, then compile the produced C code, then use this compiler to convert your test programs to C code and then run gcc to convert it to a m68000 .o file, which then you can link with a Jaguar linker like rln. Sounds complex? It was . Now, I chose mingw as my C compiler for bcx because Cygwin for some reason couldn't produce a Windows executable of bcx! And since I copied the dev environment and build script from Raptor Basic, I had to use the Windows command line too! Luckily I used Conemu which can show all of the above in the same screen, in tabs, and with horizontal/vertical splits - so I had all the madness at once on my screen .

Shortly after the initial environment was sorted, a PLOT/COLOUR pair of commands were coded to see if I could put anything on screen. Of course I spent some time on this because I had to see how C passes parameters to routines (and with the added bonus of compiler optimisations on ). After some confusion and a few hours of looking at compiler generated source code (gas syntax yay), I managed to get that working, kind of!

Of course, the real problems were just starting: gcc's standard libs object files are packed in archives with the extension .a. rln didn't know how to eat up those, so I raised a support ticket to Shamus (bless him!) and in the meantime I dumped all .o files from .a libraries and started resolving dependencies by hand while linking even the simplest of sources (pro tip: don't ever do that!). Check out this commit log for example to see what fun times I had . While Shamus was slaving away on rln (again, bless him!), I started to backport all EhBasic commands CJ added to bcx, remaining syntax compatible when possible.

Fast forward to December 20, where Shamus finally adds .a support to rln so linking problems become a thing of the past . Spirits went up and we targeted for a Decemeber 25 release - what could go wrong? Next few days were spent polishing up things and adding more commands...

 

[EDIT]Forgot to mention here that I built gcc on mingw to get rid of Cygwin's dll dependency hell if I wanted to do a tidy release!)[/EDIT]

And then we're one day before Christmas. Panic mode on! Due to compiler zealously optimising out stuff and inlining some of the commands I put (generally it made a total mess out of things), I decided to move all extra Raptor/U235SE functions to an external .o file. Guess what - remember when I said that Shamus fixed rln linking? Well, an awesome bug manifested when trying to link the above and rln would crash! Urgh! Also, trying to get C's sprintf to work so we can use rich output formatting would result in literally tons of unresolved externals which I had no idea how to solve! Not to mention that Raptor's print fucntion accepted -1 terminated strings, which are a "bit" hard to do in basic-translated-c-code! Things were looking bleak indeed...

So I began with the hard stuff first: Shamus was really puzzled as rln was working for him on linux - so I brought the source over to Visual Studio, solved a ton of warnings-Visual-Studio-likes-to-call-errors-for-some-reason-maybe-there's-a-switch-to-pass-so-it-won't-no-time-to-research-release-was-a-few-hours-away-zounds, and miraculously fixed the bug in 20 mins! Yay me! After this my spirits went up again and fixed the rest of the problems: sprintf was out which means the user will be responsible for formatting the string before sending it off to be printed. But null-terminated strings were a problem until I thought: hey, don't I have access to Raptor source code? DUH! 5 minutes later and I had a custom raptor.o which accepted null terminated strings! Removed optimisations from compilation (too scared to leave that on for a public release, I think it should be alright though), fixed the build environment, tested that all examples work, CJ woke up in .au, he tested some more things, uploaded to server and made the release posts! Pheeeeeew!

So this is more or less the history of how these 2 Basics came to existence. There are still stuff to be added and probably a helper program for asset insertion in Basic projects, but in due time - we made the first step, now we're waiting for people to pick it up and tell us what they like and dislike, and what they'd want to see next.


Thanks for reading, and since it'll be a new year in a few hours, I'd also like to wish you all a happy new year - please use Raptor Basic and Basic+ to bring some life to the Jaguar - we'd be ever so glad!

Edited December 31, 2014 by ggn

[Christos]

I'll stick to RBASIC now, since I hardly ever boot to windows. The more interesting part, as I understand it, is that you basically link files created with gcc, so any language that has 2C compiler can be modified for RAPTOR and linked. That would include my personal favourite X11BASIC and others. These are exciting times for jaguar development. I guess Atari should have hired you guys back in 1993 to work on these.

[+Gemintronic]

I think both methods are awesome! I guess it depends on which RBASIC people are more hungry for feedback and bug reports on.

[ggn]

I'll stick to RBASIC now, since I hardly ever boot to windows. The more interesting part, as I understand it, is that you basically link files created with gcc, so any language that has 2C compiler can be modified for RAPTOR and linked. That would include my personal favourite X11BASIC and others. These are exciting times for jaguar development. I guess Atari should have hired you guys back in 1993 to work on these.

 

Yeah, basically any compiler/interpreter that can produce a MC68000 object file and modified so it can talk to Raptor can be used here. I actually gave x11basic a shot while evaluating Basics (forgot to mention that!) but trying to compile it for windows really kicked my arse. Also the devs themselves admit that the Windows port needs some updating and they're doing some really arcane things to compile it under Linux.

 

Also, forgot to mention that Raptor Basic+ should in theory be usable on Linux and Mac. If you check out Vincent's site, cross m68k gcc has been ported to those systems. Which leaves us to the sticky point of running bcx, but I'm positive that wine should help here. Can anyone try that? In theory you could take the build.bat script we have bundled and change the line that says "bc PROJECTS\%1\%1.bas -q" to something like "wine bc PROJECTS\%1\%1.bas -q". I suppose some path fiddling might be needed but in theory it should produce a C file.

[Chilly Willy]

This is my makefile for building gcc for the m68k in *nix. It build gcc 4.8.3, binutils 2.24, and newlib 2.1.0. It builds c, c++, objc, obj-c++, libc, and libstdc++ for the 68000. It's basically the same way I build gcc for the Sega Genesis.

[+CyranoJ]

I do understand there is the bare minimum interfacing with RAPTOR at the moment. If any of the other commands are required I'll be happy to implement them (for both Basics). Although, TXG, this doesn't mean you can submit a post with all the commands in it - You'll have to convince me you're actually needing any of them first

 

Whilst on the subject of 'how these came to be' - it has to be said, without the API, making the BASICS would have been orders of magnitudes harder. Imagine the daunting task of having to write the BASIC command plus all the actual back end hardware management, RAPTOR API put us in a great position where new commands had to simply be wrappers to pre-existing functions. And, of course, updates to RAPTOR make the API more functional whilst providing additional offloaded functionality to the BASICS. Everyone wins, from the assembler junkies to the BASIC junkies.

 

The RAPTOR API Object file is also, due to debugging and memory constraints, slightly behind our internal RAPTOR versions - ie, things like QR code support are not included (yet!). It also lacks our calls to generate webcodes. We'd be happy to discuss allowing access to the webcode module with anoyone who has a complete project they'd like to integrate with online support, but I don't think it'll go out in a public revision just yet as it makes the score tables on our website vulnerable to mischief. Although, I'm sure the Jaguar community is above such things

 

To get the most out of RBasic you will need to exploit RAPTOR as the execution speed is not that great. For example, use RAPTOR auto update objects and don't try to control them directly from basic. What I mean by this is if you try to manually manipulate a lot of objects you will quickly run out of frame time to update them. However, if you set up a lot of objects on pre-defined paths, or give them auto-update co-ordinates you can have many objects moving on screen.

 

A typical example of this would be something like our game, Expressway (Freeway for the 2600 boffins!) - you can initialize and start moving all the cars with just one command, From then on, you don't have to manually control all of their movement, you just have to check if the player hit any of them. By doing it this way you can have, probably, around 60+ cars moving on the screen - from INTERPRETED Basic - running at 60fps.

 

Something where you need to actually control each objects path, like, say, Kobayashi Maru (Time Pilot) - is more suited to RBasic+ as each object will need manual updates.

 

However you chose to use them, and whichever one you chose to use, BASIC, BASIC+, or the Assembler API - let's bring a bunch of new stuff to Jaguar in 2015.

 

Happy new year, everyone.

Edited December 31, 2014 by CyranoJ

[ggn]

This is my makefile for building gcc for the m68k in *nix. It build gcc 4.8.3, binutils 2.24, and newlib 2.1.0. It builds c, c++, objc, obj-c++, libc, and libstdc++ for the 68000. It's basically the same way I build gcc for the Sega Genesis.

 

Thanks, but well, if you read my WoT (Wall of Text © sh3) above you'll noticed me saying I'm not exactly fond of C? Well, guess what I think about makefiles . What I did to build a cross-gcc was to simply take Vincent's build script, modify some paths, put the required files to the required places, ran it and presto! A cross gcc without all the useless cygwin environment gobbledygook! I was actually amazed it worked without my PC exploding into deep space! I won't push my luck further though, 0 makefiles created by me = enough to last a lifetime .

[Chilly Willy]

Thanks, but well, if you read my WoT (Wall of Text © sh3) above you'll noticed me saying I'm not exactly fond of C? Well, guess what I think about makefiles . What I did to build a cross-gcc was to simply take Vincent's build script, modify some paths, put the required files to the required places, ran it and presto! A cross gcc without all the useless cygwin environment gobbledygook! I was actually amazed it worked without my PC exploding into deep space! I won't push my luck further though, 0 makefiles created by me = enough to last a lifetime .

Yeah, that's good for Windows folks. My makefile makes things easier on *nix folks. My makefiles are pretty straightforward. Makefiles can get REALLY hairy on complex projects. I've seen some that would take a PHD in CS to understand.

 

Basic can be a fun language - it was probably the first language many of us from the 8 and 16 bit eras learned. It's pretty easy to learn, but still quite capable... and no makefiles!

[ggn]

Yeah, that's good for Windows folks.

 

Nope, Vincent's page also has packages for Linux flavours too. And RHEL done by other people too, not just Vincent. So I think that script should function in most linux distros and Windows.

[lp060]

 

Yeah, basically any compiler/interpreter that can produce a MC68000 object file and modified so it can talk to Raptor can be used here. I actually gave x11basic a shot while evaluating Basics (forgot to mention that!) but trying to compile it for windows really kicked my arse. Also the devs themselves admit that the Windows port needs some updating and they're doing some really arcane things to compile it under Linux.

 

Also, forgot to mention that Raptor Basic+ should in theory be usable on Linux and Mac. If you check out Vincent's site, cross m68k gcc has been ported to those systems. Which leaves us to the sticky point of running bcx, but I'm positive that wine should help here. Can anyone try that? In theory you could take the build.bat script we have bundled and change the line that says "bc PROJECTS\%1\%1.bas -q" to something like "wine bc PROJECTS\%1\%1.bas -q". I suppose some path fiddling might be needed but in theory it should produce a C file.

 

Also forgot to mention GFA-Basic for the ST was breifly considered. I did get an e-mail about that. However, reverse engineering the entire package is a lot of work to say the least and although it is all rebuilt from the ground up, a large percentage of it is still not fully understood and documented. GFA for the ST was quickly ruled out as a candidate.

 

That bcx-basic looks pretty good, no more line numbers. Cool!

Edited January 1, 2015 by lp060

[Chilly Willy]

Nope, Vincent's page also has packages for Linux flavours too. And RHEL done by other people too, not just Vincent. So I think that script should function in most linux distros and Windows.

It has PACKAGES for (like) Ubuntu. That's different from building it yourself. That makes is very easy to USE, not to build.

 

It also has mint specific packages, not generic packages. My makefile makes a generic m68k-elf build, not a mint build.

[ggn]

It has PACKAGES for (like) Ubuntu. That's different from building it yourself. That makes is very easy to USE, not to build.

 

It also has mint specific packages, not generic packages. My makefile makes a generic m68k-elf build, not a mint build.

 

I'm glad I went and released the basic first, then discussed about it, otherwise I'd be talking SEMANTICS for months

[108 Stars]

I really find it awesome what you have done here.

I'll probably try it out for fun myself when I have time. DerLuchs success in coding after having no experience at all 3 years ago, and now having released stuff on Jaguar, Lynx, Vectrex and whatnot is really an inspiration that n00bs CAN get something interesting done without years of learning by doing dull stuff.

This Raptor BASIC and LX.NET's in-depth tutorials on Lynx development are great stuff for beginners.

 

What is missing is the other way around... a Lynx BASIC, and tutorials for beginners on Jaguar BASIC.

 

 

Maybe you could tell us total n00bs where to start? There's so many BASIC variants out there, how should you go about beginning when your goal is to have something run on Jaguar eventually (preferably sooner than later)? Some specific tutorials, or some PC based BASIC to learn the basic workings quickly? Maybe it sounds silly to ask, or maybe you have answered this somewhere, but when you come from nothing, it may be difficult to think of a good starting point to learning.

[Christos]

It's pretty easy to get started actually. First of all there's the manual for each basic (le's say eh). You can learn the syntax of each command. Secondly there's the raptor part, you can read the raptor manual and learn about that. But that will only tell you the words of a language.

If I write some sentences in greek and you read them with a dictionary, it's not likely you'll understand much. For that the best way is to play with the code already available. Change stuff there and see what happens. That's the best way to learn IMO.

[ST-Oldie]

Hi,

Also forgot to mention GFA-Basic for the ST was breifly considered. I did get an e-mail about that. However, reverse engineering the entire package is a lot of work to say the least and although it is all rebuilt from the ground up, a large percentage of it is still not fully understood and documented. GFA for the ST was quickly ruled out as a candidate.

I remember the disassemble project for Atari ST GFB Basic. Some years ago i got the source code for the planned GFA Basic 4.00, written in C with the use of ACS Pro. A few years later i had contacrt with someone interested in GFA Basic. He took a look in the source code for GFA 4.00 and told me, there was a disassemble project. The goal was to understand whya GFA basic was so fast and (maybe?) build a successor which is as fast then the original interpreter.

 

As far as ai know, the token code (.GFA files) ist fully understand. So im am pretty sure there is no need to disassemble the old GFA interpreter. I would say, with the knowledge of the token code it is possible to write a interpreter whihc is (nearly'?) as fast as the original.

 

Best regards

Michael

[lp060]

I can't speak for ggn, but my impression was they wanted to modify a basic that was already written to save time. Perhaps it was a friend of mine that contacted you. I was, well still am, part of a small group that helped dis-assembled the gfa editor. It was never fully commented as most of the members went idle.

Edited January 10, 2015 by lp060