meta A new language for the Atari

[Kaj de Vos]

I touched on that several times in this thread. The language and compiler are designed to fit like a glove around REBOL, C and WebAssembly, and eventually also LLVM. Years ago I was already preparing this project, and I was working towards an LLVM binding. Before I made the final decision to go ahead ten months ago, I revisited all my research. By then I wasn't as attached to Red and its model anymore as I once was. I always wanted to support Atari, but never thought it would be economical, and certainly not feasible without a native backend. Generating C always seemed like a distant option, a kludge. I was actually considering more to write my own C compiler on the same stack, just to get rid of the current humongous C toolchains.

 

Revisiting all that, more puzzle pieces clicked together in my head, I found new pieces on the Internet and I found that several pieces had developed considerably since my last rounds. Always when such a step happens, one's horizon broadens. LLVM suddenly seemed limiting instead of all-encompassing. I had always been looking for alternatives, but they were always problematic. On the other hand, CC65 looked pretty reasonable, active and much more developed than its origins. Following this path further, I found Z88DK. In general, it seems more evolved than CC65, so a picture developed of CC65 as the perfect baseline for a C generator. That's how I'm using it now: if I get something to work on CC65, with few exceptions, I know it will work on almost any C toolchain. It used to be popular, but these days few languages generate C, indeed because LLVM has taken over this baton. Generating C is indeed a kludge compared to LLVM, but it means I can be more than competitive in supporting target platforms. Even other modern languages generating C are much more complex and fail to target retro platforms.

 

There is no problem in competing with C, because my language has almost the same performance as C. Where there's a good C compiler, it's also competitive with Pascal. And I allowed myself the indulgence of implementing the integrated assembler, so I already support the Atari and 6502 natively and close to optimal through that. It was also a good design study, leading to some changes in the language design, and preparation for other future assemblers.

 

Still, it's frustrating all the time to know that a native backend could be much more optimal. EMPTY.XEX should be 7 bytes instead of 517. On the other hand, the C backend also still has a lot of room for improvement. Everywhere I look, things are unfinished. Everywhere I look, things can be optimised. I have design studies and proofs of concept of several backends, but even finishing just one is an enormous amount of work. Without an 8-bit target, the drive to optimise the C backend would be much lower, and I might end up like other languages: a lot of missed opportunities if you know what is possible.

 

Soon, I will need to start to collect funding, or I may not be able to continue the project. I can't expect that to all come from the Atari 8-bit audience, so I need to branch out, collect users from everywhere to share the development cost. To achieve that, I need to strongly focus my planning, on the cross-platform features. As I said before, I can't afford to do the full Atari backend now, as it would endanger the project. However, if it would happen that early funding from Atari donations would be highest, I will work on it first.

[Kaj de Vos]

I see we have reached page 6 here. ?

[ivop]

3 hours ago, Kaj de Vos said:

I was actually considering more to write my own C compiler on the same stack

I have considered doing that myself, too. Many times. There a a few pretty good C parsers, like tinycc, sparse, 8cc/chibicc, that are begging for a 6502 backend

 

But then comes the question of how to implement that backend. Software stack ABI, we already have. So either like an embedded system, and use the hardware stack, or implement a VM that uses all extended memory as a continuous memory space, up to 1MB with Ultimate, and 4MB with an Antonia upgrade, and target that.

 

Edited April 4, 2021 by ivop

[Kaj de Vos]

I have a C backend now, so I could compile C to C. Bwoehahahaa!

 

Actually, no, because my intermediate language is safe: it doesn't support pointers and such. This is also one of the reasons why adding a native backend would be a lot of extra work: to do it right requires an extra, lower-level intermediate format.

 

Yeah, there are many options for mapping memory. I think 8-bit CPU's shouldn't have had 16-bit address spaces, they should have been 24-bit.

[+MrFish]

19 hours ago, Kaj de Vos said:

I see we have reached page 6 here. ?

Can't be: there are only 120 some odd posts so far. Call again when you've reached 1536+.

 

Page 6: Dumping ground for 95% of all BASIC machine language subroutines ever written.

 

So, what if you want to use more than one machine language subroutine in your program? If you're lucky, one of them is relocatable, and can be stuffed into a string (preferable safe memory), or CASBUF (if it's small enough).

 

BTW, don't try and retrieve more than 128 bytes using the INPUT statement, if you've got something of value in Page 6: you'll find your data hacked up after the command uses this treasured space for its buffer overrun. "Safe memory"? Usually...

 

Apparently OSS decided users didn't need all of Page 6, when they decided to take a chunk out for BASIC A+. Could have factored into the success of this DOS, or lack thereof...

 

Edited April 5, 2021 by MrFish

[Kaj de Vos]

Hmm, I went through all those phases... ? Last week I read that even Mac/65 uses page 6, so you can't test your programs. There's so much I must have known back then, but that is almost gone from my brain. I'm really not sure if I knew about INPUT, though.

 

Anyway, I heard nobody wants a safe language on 8-bit, so it's all fine, isn't it?

[danwinslow]

18 minutes ago, Kaj de Vos said:

Anyway, I heard nobody wants a safe language on 8-bit, so it's all fine, isn't it?

Mad Pascal is probably the closest, although if you mean 'safe' to include no pointers, then not it either. Safety usually costs some amount of either complexity, speed or memory. The 8 bit is so constrained it's usually not worth it. Java can tell you that 'no pointers' by itself does not equal safe. I come from an Ada background, a famously 'safe' language, and programmers generally hated it because of what it wouldn't let you do. It happens to still be my favorite language, but nowadays everybody uses Javascript, which I don't event consider a full language and is so bizarrely unsafe it reminds me of C, although at least C has a coherent syntax. Ah, progress.

[+MrFish]

21 minutes ago, Kaj de Vos said:

Last week I read that even Mac/65 uses page 6, so you can't test your programs.

Yeah, but there's a fairly elegant workaround built into the language to compensate -- assembling with an offset. Let's you test from the land of make believe.

  

21 minutes ago, Kaj de Vos said:

Anyway, I heard nobody wants a safe language on 8-bit, so it's all fine, isn't it?

I just want a child-proof keyboard -- because I'm pretty rough with the things.

 

[ivop]

9 minutes ago, danwinslow said:

I come from an Ada background

Oh, I remember that. Had to do an assignment in Ada once. Wasn't that what NASA used BITD?

 

Also did Smalltalk, Miranda, and some other long forgotten languages. The horror, the horror.

[Kaj de Vos]

Hmm, I'm going to list Miranda as one of the influences on my language. ? Twentel, too.

[ivop]

Haha, I was about to edit my post, and add that Miranda was not that bad Another language I liked BITD was TCL (and TCL/TK, interactive GUI building )

[+CharlieChaplin]

1 hour ago, MrFish said:

Can't be: there are only 120 some odd posts so far. Call again when you've reached 1536+.

 

Page 6: Dumping ground for 95% of all BASIC machine language subroutines ever written.

 

 

Hmmm, I thought page 6 is where they publish the nude girls?

 

[flashjazzcat]

28 minutes ago, CharlieChaplin said:

 

Hmmm, I thought page 6 is where they publish the nude girls?

 

That's page three, although if Page 6 had done so, they may have lasted a bit longer.

[Gury]

I am so excited. It would be really good to go through Atari binaries with software monitor, I would maybe find something.

 

[+MrFish]

2 hours ago, CharlieChaplin said:

Hmmm, I thought page 6 is where they publish the nude girls?

 

Yeah, they included inflatable girlfriends with each issue for a while...

 

[Kaj de Vos]

On 4/5/2021 at 5:37 PM, danwinslow said:

Safety usually costs some amount of either complexity, speed or memory. The 8 bit is so constrained it's usually not worth it.

Thinking about this, the cost in my language is in complexity of the type system. I'm introducing many more types than REBOL, but I'm managing so far to leave this complexity in the compiler. It is designed to compile away the complexity to be able to target tiny systems.

 

Compared to something such as FastBasic, which is a simple language to still be hosted on the 8-bit, my language is much too big to easily host the compiler on 8-bit, but it is designed to run the programs there.

 

In itself, a type hierarchy is not that complex, it's mostly just large. Basically like how languages like C and C++ use many named record and object types. While programming, one seldomly notices it, because the REBOL design abstracts away types in most language expressions. In my language, most types are inferred, and enable the compiler to generate optimised code.

 

So far, I support a static style of program code. One of REBOL's strengths is dynamic code (a far descendant of self-modifying code). When I start adding that, programs will need to include more runtime and their size will explode. My aim is to get as far as possible with still fitting that in small machines.

[Kaj de Vos]

Rainbows galore!

 

I posted an improved Rainbow here:

 

https://language.meta.frl/examples/platforms/Atari/8-bit/rainbow.list
https://language.meta.frl/examples/platforms/Atari/8-bit/rainbow.labels
https://language.meta.frl/examples/platforms/Atari/8-bit/rainbow.map

 

 

Here is the code:

unsafe!!! constant reference volatile byte! [
    ; OS

    RTCLOK3= ~14
    SDMCTL=  ~022F

    ; ANTIC

    COLBK=   ~D01A
    WSYNC=   ~D40A
    VCOUNT   ; ~D40B
]

SDMCTL: 0  ; Turn off display DMA

forever [COLBK: WSYNC: overflow VCOUNT << 1 + RTCLOK3]

 

[Kaj de Vos]

I've been doing more structural work on the language implementation. I started out by inlining all generated code, but this produces too much code for 8-bit. I am now creating functions for multiple use and collecting them in a runtime library. This makes the language implementation quite a bit more complex, because C linkers want every function in separate source and object files to be able to select and deselect them for inclusion. Now all those files need to be managed and you suddenly need extra build tools for managing files and dependencies. Compiling all those little files is much slower, so it also causes the need for incremental compilation of dependencies. I also think it blocks compilers from being able to do certain optimisations. All in all, a rather outdated architecture.

 

Anyway, once the runtime library is compiled, compiling a user program on top of it can be much faster. It will keep compilation time down when the runtime library grows. Compiling a program takes less than a second on a small laptop.

 

On systems with a runtime loader/linker, the runtime library can be compiled as a dynamic library, sharing the memory of the library between running instances of programs written in the language.

 

With some code wrapped into runtime functions, I have started implementing error checking, currently simply using the facilities of the C library. This explodes the size of program executables. Hello World is now 4 KB. However, it still compares favourably with something such as Atari BASIC, where you get error checking at the cost of a runtime of 8 KB. Eventually, I will implement options for different levels of error reporting, to control size.

 

I have republished Hello World. The download links are the same as before.

 

If you clicked on my links from an Atari Age digest mail, you got nothing, because there seems to be a bug in the Atari Age mailer that constructs invalid links. I have programmed a detection and compensation for this in my web server. In my own language, of course.

[ivop]

2 hours ago, Kaj de Vos said:

Compiling all those little files is much slower, so it also causes the need for incremental compilation of dependencies. I also think it blocks compilers from being able to do certain optimisations. All in all, a rather outdated architecture.

That was fixed with pre-compiled headers, and link time optimization. Both gcc and clang/llvm do that. Cc65 does not   But really, how long does a cc65 compile last anyway, even with shell and make overhead? That won't get significantly shorter with forementioned optimizations, as it is already below a few seconds.

[Kaj de Vos]

Pre-compiled headers are just the headers, right? They do get included insanely more often with all those little source files, so it helps, but it's not what I'm talking about.

 

Both the CC65 and CLang linkers include an object file fully: they don't optimise away functions you don't use. There may be some internal technical reason for that, but it forces a rigid structure on your source: one function per source file. This is bad, it explodes management of all those files and makes you loose overview. I haven't found a way to avoid this, if you know one, I'm all ears.

 

Compilation of the library is much slower this way. Sure, at this early stage it's still under a second for the language. But runtimes for REBOL-style languages grow relatively big, and then come all the external libraries you want to bind. The biggest problem is the sudden cascade of extra complexity and needed build tools. It feels like something simple suddenly exploding in your face.

 

It's clear that LLVM was designed to fix some of this. Its linker is supposed to do optimisations that a compiler normally does, but cannot do after the object files and library are generated. However, I don't see LLVM doing this yet, and I wonder if it can with these traditional formats. Perhaps I need to force it to generate LLVM intermediate code in the object files and library, but it doesn't seem to operate like that by default.

[dmsc]

Hi!

4 hours ago, Kaj de Vos said:

Pre-compiled headers are just the headers, right? They do get included insanely more often with all those little source files, so it helps, but it's not what I'm talking about.

 

Both the CC65 and CLang linkers include an object file fully: they don't optimise away functions you don't use. There may be some internal technical reason for that, but it forces a rigid structure on your source: one function per source file. This is bad, it explodes management of all those files and makes you loose overview. I haven't found a way to avoid this, if you know one, I'm all ears.

Modern ELF linkers (like LD from binutils/GCC or LLD from CLANG) allows using wildcard in the section specifications, allowing to write one bit object file with one section per function or data object - this allows to filter out individual functions from one object file, reducing the number of source files needed.

 

Currently LD65 does not support this, but it could be added so you could write a linker config like this (simple sample from FastBasic linker config):

FEATURES {
    STARTADDRESS: default = $2000;
}
SYMBOLS {
    __STARTADDRESS__: type = export, value = %S;
}
MEMORY {
    ZP:      file = "", define = yes, start = $0094, size = $0040;
    MAIN:    file = %O, define = yes, start = %S,    size = $BC20 - %S;
}
FILES {
    %O: format = atari;
}
FORMATS {
    atari: runad = start;
}
SEGMENTS {
    ZEROPAGE: load = ZP,      type = zp,  optional = yes;
    CODE.*:   load = MAIN,    type = rw,                  define = yes;
    DATA.*:   load = MAIN,    type = rw   optional = yes, define = yes;
    BSS:      load = MAIN,    type = bss, optional = yes, define = yes;
    HEAP:     load = MAIN,    type = bss, optional = yes, define = yes, align = $100;
}

Now, you could write assembly like this:

	.secgment "CODE.00001"
.proc print_eol
	.import print_char
    .export print_eol
    lda #155
	jmp  print_char
,endproc

	.secgment "CODE.00002"
.proc print_char
    .export print_char
    tay
    lda     ICAX1,X
    sta     ICAX1Z
    lda     ICPTH, x
    pha
    lda     ICPTL, x
    pha
    tya
    rts
,endproc

	.secgment "CODE.00003"
.proc print_str
    .import print_char
    .export print_str
    ; .......... some code that uses print_char
    rts
,endproc

 

Now, you need to also implement the option for garbage-collect unused linker sections - this is the most difficult part as you need to only include sections that contains symbols that are referenced, and discard sections that aren't. Hacking LD65 is not that difficult, you could try!.

 

4 hours ago, Kaj de Vos said:

Compilation of the library is much slower this way. Sure, at this early stage it's still under a second for the language. But runtimes for REBOL-style languages grow relatively big, and then come all the external libraries you want to bind. The biggest problem is the sudden cascade of extra complexity and needed build tools. It feels like something simple suddenly exploding in your face.

Also, if you are writing a cross-compiler, current PCs are really fast, so this is not really a problem. When compling FastBasic, compiling the C++ parser is ten times slower than compiling the full library ? 

 

4 hours ago, Kaj de Vos said:

It's clear that LLVM was designed to fix some of this. Its linker is supposed to do optimisations that a compiler normally does, but cannot do after the object files and library are generated. However, I don't see LLVM doing this yet, and I wonder if it can with these traditional formats. Perhaps I need to force it to generate LLVM intermediate code in the object files and library, but it doesn't seem to operate like that by default.

 

Many compilers (both GCC and CLANG, for example, with the "-flto" option) support link-time-optimizations, and the technology is always the same: instead of compiling to machine language, compile to some intermediate representation. On link time, the liner calls a plugin that is passed all the needed object code, the plugin calls the compiler again passing all the intermediate code as one big chunk.

 

Have Fun!

[Kaj de Vos]

Thanks again for the pro tips! It seemed strange to me that modern linkers would not support this, but it would probably have taken me a long time to find this.

 

I will experiment with it. However, if there are older C compilers/linkers that don't support this, I can't use it, because the runtime needs to be shared between a number of toolchains.

 

Compilation speed is currently very good, but the runtime will grow a lot over time. Not just for Atari, also for bigger systems. I am traumatised by compiling the Syllable operating system for many years, which took most of the day and doubled every year just by updating other open source projects it contained, and by the Red language, which takes a number of minutes to compile its language runtime. I need to offer my compiler as a network service, so to scale that, I need all the speed I can get.

[Kaj de Vos]

-flto with CLang gives me the optimisation I expected. I need to use it on both the compilation of the library and the compilation of the program, or it confuses library file types.

 

This is what I was looking for for modern toolchains. I'm surprised that this isn't the default, and that traditional toolchains are so primitive.

[ivop]

You might also want to look into ThinLTO, which can be enabled by -flto=thin with clang.

[Alfred]

I'm curious as to what sort of programming projects you think that this language would perform better or be more easily implemented than the existing array of native or cross-compiled languages ? Usually a programmer has an idea for some project which drives the toolchain development, although there's nothing wrong with doing it for it's own sake. I've looked over some of the documentation, and while I'm not in the target audience, I'm curious as to who you think is that audience, and for what purposes. I expect as time goes by and features are added, the Atari will be left behind due to its memory limitations.