Removers Library - a.out Binary Emulation Dropped From Linux Kernel

[+BitJag]

Reloaded a most recent flavor of Ubuntu and noticed that a.out binary emulation is now gone from the Linux kernel.  As much as I am able to understand this means that MADMAC and ALN aren't options anymore in Linux, making it impossible to build with the Removers Library as it currently sits.  I understand that there was allot of effort put into some replacements a few years back with RMAC and RLN, but RMAC doesn't seem to be a good replacement for the Removers Library at the moment, according to Seb.  This was 5 years ago, and I might be missing a forum thread somewhere on the web that might shed some light on moving over to RMAC & RLN/Seb's Linker.

 

Besides the obvious other options, like moving to Windows or learning to code my own macro assembler over the next 20 years, are there any other options I am missing for getting the Removers Library/MADMAC/ALN working in Linux?  I can't complain that I have been able to use Linux and the Removers Library for over a decade with what ever system I am using for my day job, but I don't fancy the idea of being isolated to a static/non-updatable dev system for the foreseeable future.

[zzip]

4 minutes ago, lachoneus said:

Reloaded a most recent flavor of Ubuntu and noticed that a.out binary emulation is now gone from the Linux kernel.  As much as I am able to understand this means that MADMAC and ALN aren't options anymore in Linux, making it impossible to build with the Removers Library as it currently sits.  I understand that there was allot of effort put into some replacements a few years back with RMAC and RLN, but RMAC doesn't seem to be a good replacement for the Removers Library at the moment, according to Seb.  This was 5 years ago, and I might be missing a forum thread somewhere on the web that might shed some light on moving over to RMAC & RLN/Seb's Linker.

 

Besides the obvious other options, like moving to Windows or learning to code my own macro assembler over the next 20 years, are there any other options I am missing for getting the Removers Library/MADMAC/ALN working in Linux?  I can't complain that I have been able to use Linux and the Removers Library for over a decade with what ever system I am using for my day job, but I don't fancy the idea of being isolated to a static/non-updatable dev system for the foreseeable future.

you could run a version of linux that still supports it in a virtual machine like KVM or virtualbox

[+BitJag]

14 minutes ago, zzip said:

you could run a version of linux that still supports it in a virtual machine like KVM or virtualbox

Thanks for the suggestion.  I have used virtual machines before, but run into the inconvenience of not being able to to get virtualjaguar emulating well inside a virutal machine.  It has been a while since I have tried this though,  I should give it another shot. 

 

A work around for slow Jaguar emulation in a virtual machine is to create a shared drive between the virtual machine and my host machine and then run virtualjaguar outside of the virtual machine.  This does slow down my workflow though. At that point my personal preference would be to have a dedicated, non-updatable linux pc, considering how many times I am booting up virtualjaguar to test code.  

 

As a side note, I am having a hard time finding the source code for RMAC and RLN, everything I have found so far here and here are windows/mac builds.  It seems the original repo isn't available anymore. Does anyone know where the source is available for these? Is it supposed to be available?

[zzip]

10 minutes ago, lachoneus said:

At that point my personal preference would be to have a dedicated, non-updatable linux pc, considering how many times I am booting up virtualjaguar to test code.  

That's a good option too.   From your post, I thought you were just trying to get development tools running, which would be a fine use for a VM,  but running VirtualJaguar in it I agree would be a problem.

[swapd0]

These are the latest version that I have. 

rln.zip rmac.zip

[+BitJag]

12 minutes ago, zzip said:

That's a good option too.   From your post, I thought you were just trying to get development tools running, which would be a fine use for a VM,  but running VirtualJaguar in it I agree would be a problem.

Getting the tools back to a usable state is the best situation for my current workflow.  But I am beginning to think the reality is, without having to invest time into learning how to push RMAC and RLN myself, this might be the end of the line for Removers Library in Linux.

4 minutes ago, swapd0 said:

These are the latest version that I have. 

rln.zip 224.09 kB · 1 download rmac.zip 1.28 MB · 1 download

Thank you!  I didn't want to give up on these yet and I wanted to try building these in Linux ti see if I run into the same problems that Seb did or not when trying to build with RMAC.

[ggn]

FWIW I'm taking a look on why Seb's code won't assemble under rmac, so far it seems that a macro is making the assembler mis-behave. (I guess worst case the macro can be rewritten to comply but I'll take a crack at fixing the issue first. Since it's working with madmac and all...)

[+BitJag]

38 minutes ago, ggn said:

FWIW I'm taking a look on why Seb's code won't assemble under rmac, so far it seems that a macro is making the assembler mis-behave. (I guess worst case the macro can be rewritten to comply but I'll take a crack at fixing the issue first. Since it's working with madmac and all...)

gnn thanks!  Let me know if there is anything that I could possibly help out with.  I don't consider myself anywhere near the same caliber of programmer as anyone that has been involved with the removers or RMAC/RLN, but having this so I can continue to dev in my current and future Linux environments is a huge incentive for me to be as useful as I can.

[ggn]

Ok, so first things first: rmac and rln do have an official distro here: http://virtualjaguar.kicks-ass.net/builds/ or if you need to build from source: "git clone http://shamusworld.gotdns.org/git/rmac" and "git clone http://shamusworld.gotdns.org/git/rln". The first link holds the windows and mac builds, so there's no need to upload versions anywhere. Also I noticed that the version @swapd0 uploaded is an older one, so it's also another reason :). For obvious reasons we don't host any linux binaries so people are encouraged to build from source in that o/s.

 

With that out of the way, I had a go at building the removers' libraries, which meant I had to fiddle with makefiles (yuck!). I'll get back to the topic in a bit, I'll just skip forward a bit to mention the rmac assembly issues:

a) There is a small issue with rmac not handling * as program count on expressions. I didn't fix that inside rmac itself but I bypassed it by altering the offending code. If you open display\display.s you'll find a line that says "padding_nop    (G_RAM+$10-*)" and another "padding_nop    (G_RAM+$40-*)" which both give out some cryptic errors. These can be overcome by equating * to a symbol in the previous lines. I.e. go to the first padding_nop line and add a line directly above that says "pc1=*", then modify the offending line to ""padding_nop    (G_RAM+$10-pc1)". Likewise, go to the second line and add a "pc2=*" immediately above and modify line to "padding_nop    (G_RAM+$40-pc2)". That should take care of that.

b) But there's another issue inside rmac that can't be fixed that easily unfortunately. For this you need to modify the code and recompile rmac for now. Namely, open file expr.c and go to line 421. The line there should read "*a_attr = cursect | DEFINED;". Change that to "*a_attr = DEFINED;" and recompile rmac.

 

If you do these two steps correctly and you've set up your paths to point to the correct tools you should be able to compile the code all the way through. At least on my setup with gcc 9.2.0-m68k it seems to create the library no problems. I just skipped the doxygen doc generation as I don't want to install that.

 

Getting back to the main topic now. The whole a.out requirement is a bit weird to me. a.out is for creating binaries that are relocatable so the operating system can load them, relocate them where they're loaded and then execute them. Jaguar binaries are by default absolute, i.e. you have to tell your linker to hardcode them into specific addresses (for example $802000 for ROMs or $4000 for a typical bjl binary). This can be done without generating a.out binaries at all, just tell the linker to grab the libraries in whatever format they are and the object files, jam them together and hardcode all addresses, then write the binary to disk.

 

It's true that a.out is also a format for object files and libraries as well. I encountered this problem a few years ago when trying to build a cross gcc compiler for Atari ST binaries. I mistakenly thought that the a.out format was dropped back then and only elf was available (that was true to some extent). So I managed to use the elf format for gcc and covert the elf binaries to tos executables. (sorry for babbling, we're getting to the point!) The side effect is that we had to add elf support to rmac, so it's there by default now.

 

What this all means is that you can now use the elf format for gcc and rmac, and produce object files that are compatible with each other. I'm pretty sure that the gcc linker (ld) then can generate absolute binaries.

 

Anyway, that's as far as I got to this evening. If you'd like to try my fixes above and see if at least this fixes the rmac issues we could figure out the rest. Ideally a small test project that uses the library and builds a small binary would be a great help at this point, since I'm not familiar with the library at all!

 

Hope this helps a bit!

 

(also I hope Seb will read this and comment!)

Edited October 24, 2019 by ggn

[Alex Czarnowski]

Well, a.out is really a thing of past but you can always do one of 2 things: compile your own kernel with a.out support enabled or install dosbox and run original Atari dev tools inside it. It is available as binary package for most Linux distros I suppose. 

 

I highly doubt that any other available assemblers do follow all of idiosyncrasies of MADMAC and ALN. I recall that one of MADMAC clones (was it RMAC?) has been rewritten using original Atari source code but the source package has been missing Jaguar support. On the other hand it has been ages since I've used any of Jag dev tools so I may be very wrong here. 

[+BitJag]

7 hours ago, ggn said:

Ok, so first things first: rmac and rln do have an official distro here: http://virtualjaguar.kicks-ass.net/builds/ or if you need to build from source: "git clone http://shamusworld.gotdns.org/git/rmac" and "git clone http://shamusworld.gotdns.org/git/rln". The first link holds the windows and mac builds, so there's no need to upload versions anywhere. Also I noticed that the version @swapd0 uploaded is an older one, so it's also another reason :). For obvious reasons we don't host any linux binaries so people are encouraged to build from source in that o/s.

A clear indicator to how uninformed I am, I tried just pasting the links for the git repository, without "git clone" to see if I could just see the file structure.  Turns out you get a 404 when you do that.  Learned that you have to use git to download these.  Got em' now.

7 hours ago, ggn said:

With that out of the way, I had a go at building the removers' libraries, which meant I had to fiddle with makefiles (yuck!). I'll get back to the topic in a bit, I'll just skip forward a bit to mention the rmac assembly issues:

a) There is a small issue with rmac not handling * as program count on expressions. I didn't fix that inside rmac itself but I bypassed it by altering the offending code. If you open display\display.s you'll find a line that says "padding_nop    (G_RAM+$10-*)" and another "padding_nop    (G_RAM+$40-*)" which both give out some cryptic errors. These can be overcome by equating * to a symbol in the previous lines. I.e. go to the first padding_nop line and add a line directly above that says "pc1=*", then modify the offending line to ""padding_nop    (G_RAM+$10-pc1)". Likewise, go to the second line and add a "pc2=*" immediately above and modify line to "padding_nop    (G_RAM+$40-pc2)". That should take care of that.

b) But there's another issue inside rmac that can't be fixed that easily unfortunately. For this you need to modify the code and recompile rmac for now. Namely, open file expr.c and go to line 421. The line there should read "*a_attr = cursect | DEFINED;". Change that to "*a_attr = DEFINED;" and recompile rmac.

 

If you do these two steps correctly and you've set up your paths to point to the correct tools you should be able to compile the code all the way through. At least on my setup with gcc 9.2.0-m68k it seems to create the library no problems. I just skipped the doxygen doc generation as I don't want to install that.

Thanks for this.  I am going to give this a go tonight and tomorrow morning to see if I can get the libraries built.  I am hoping that I don't have to rebuild my cross compiler, I will cross that bridge if I need to.

7 hours ago, ggn said:

mistakenly thought that the a.out format was dropped back then and only elf was available (that was true to some extent).

A few years back they just disabled a.out binary emulation in the kernel .config file (marked to just not build with the kernel).  To remedy this, so I could execute MAC and ALN for compiling, I just needed to re-enable this in the kernel .config file, and rebuild my kernel.  I just tried doing this with the latest kernel, and the option is completely gone.  I don't know if Ubuntu is just a branch of the kernel, and it only effects that branch, but I did pull the latest kernel from kernel.org (5.3.7 if I remember right), and the flag to enable a.out binary emulation is completely gone.  I found that bug report I linked in an early post, and I assumed it was pulled completely from the kernel at this point.  I haven't dug any further than that though. 

 

I would still like to get away from the original MAC and ALN anyways for long term support.  Because even if a.out emulation isn't gone yet, based on that bug report, the people working on the kernel are assuming no one uses it anymore and it causes more problems than its worth.

 

7 hours ago, ggn said:

Ideally a small test project that uses the library and builds a small binary would be a great help at this point, since I'm not familiar with the library at all!

Thanks again gnn, I believe this will help out allot!  I will do my best to get to this point over the next couple days.

 

4 hours ago, Alex Czarnowski said:

compile your own kernel with a.out support enabled

Alex, I don't know if you have the time, but could you check to see if the option to enable a.out emulation is still available on the latest stable version of the Linux kernel?  I have tried it a couple times, and don't see it through opening and manually editing the .config file, or by using the make menuconfig command.  Its just gone as far as I can tell.  The only mention that I was able to find that supports its disappearance is the one bug report where they say that keeping this option in the kernel is actually causing problems.

[Zerosquare]

Support for a.out was marked deprecated last March and planned to be removed a few releases later:

https://www.phoronix.com/scan.php?page=news_item&px=Linux-Dropping-A.Out

 

So it's likely it has now been removed completely.

[+BitJag]

1 minute ago, Zerosquare said:

Support for a.out was marked deprecated last March and planned to be removed a few releases later:

https://www.phoronix.com/scan.php?page=news_item&px=Linux-Dropping-A.Out

 

So it's likely it has now been removed completely.

Well, there's some confirmation.  Thanks Zerosquare!

[+BitJag]

Before I hit the sack.  This is where I was able to get to.

 

I was able to get jlibc (from git repo) built using rmac from git, and the ppa for m68k-atari-minit cross-tools, and making the appropriate adjustments to makefiles and environment paths. I ran into an error with sound.s in rmvlib (from git repo), after implementing the two suggested fixes to display.s and to rmac's source file.

 

rmac -fb  sound.s
adding 8 padding nop
adding 0 padding nop
sound.s 192: Error: illegal relative address
sound.s 192: Error: mis-nested .endr
adding 8 padding nop
adding 8 padding nop
adding 8 padding nop
adding 8 padding nop
sound.s 141: Warning: NULL offset in LOAD ignored
sound.s 192: Warning: NULL offset in LOAD ignored
sound.s 266: Warning: NULL offset in STORE ignored
Sound driver code size (DSP): 27C
Available DSP Ram after D_RAM+22C
make[1]: *** [../Makefile.template:6: sound.o] Error 2
make[1]: Leaving directory '/home/lachoneus/tmp/Jaguar/src/rmvlib/sound'
make: *** [Makefile:39: sound] Error 2

 

Thinking it was the version of gcc provided by the ppa (version 4 something), I decided to try and build the cross compiler from scratch using source files from here http://tho-otto.de/crossmint.php . Dropped an additional pile of hours on top of the many hours over the last decade trying to figured this one out, no luck.

 

So, resorting to the pre-compiled binaries provided by the same site, I was able to build jlibc again, with a newer version of gcc (9.1.1), but building rmvlib gives me the same error that I was receiving at the beginning of this post.  I am not sure if it is a bug/incompatibility with rmac, a mistake in my Makefile  (I don't think this is the problem because it is compiling other assembler files just fine up until it gets to sound.s), or something to do with my cross compiler tool set.  When it comes to troubleshooting my setup of cross compiler tools, I will need some help stepping through this, because there is something my head just isn't wrapping around when it comes to this.

 

I am planning on spending more time on this tomorrow.

[+Stephen Moss]

16 hours ago, lachoneus said:

Besides the obvious other options, like moving to Windows or learning to code my own macro assembler over the next 20 years, are there any other options I am missing for getting the Removers Library/MADMAC/ALN working in Linux?

I don't know if belboz' Slax.iso uses the files you want but IIRC that is a bootable Linux (or Linux style) interface so if you want to keep working in a Linux style environment perhaps it could be easily adapted to use the files you need.

[JagChris]

Belboz has the removers library in there I think.

 

 

[ggn]

2 hours ago, lachoneus said:

Before I hit the sack.  This is where I was able to get to.

 

I was able to get jlibc (from git repo) built using rmac from git, and the ppa for m68k-atari-minit cross-tools, and making the appropriate adjustments to makefiles and environment paths. I ran into an error with sound.s in rmvlib (from git repo), after implementing the two suggested fixes to display.s and to rmac's source file.

 

rmac -fb  sound.s
adding 8 padding nop
adding 0 padding nop
sound.s 192: Error: illegal relative address
sound.s 192: Error: mis-nested .endr
adding 8 padding nop
adding 8 padding nop
adding 8 padding nop
adding 8 padding nop
sound.s 141: Warning: NULL offset in LOAD ignored
sound.s 192: Warning: NULL offset in LOAD ignored
sound.s 266: Warning: NULL offset in STORE ignored
Sound driver code size (DSP): 27C
Available DSP Ram after D_RAM+22C
make[1]: *** [../Makefile.template:6: sound.o] Error 2
make[1]: Leaving directory '/home/lachoneus/tmp/Jaguar/src/rmvlib/sound'
make: *** [Makefile:39: sound] Error 2

 

Thinking it was the version of gcc provided by the ppa (version 4 something), I decided to try and build the cross compiler from scratch using source files from here http://tho-otto.de/crossmint.php . Dropped an additional pile of hours on top of the many hours over the last decade trying to figured this one out, no luck.

 

So, resorting to the pre-compiled binaries provided by the same site, I was able to build jlibc again, with a newer version of gcc (9.1.1), but building rmvlib gives me the same error that I was receiving at the beginning of this post.  I am not sure if it is a bug/incompatibility with rmac, a mistake in my Makefile  (I don't think this is the problem because it is compiling other assembler files just fine up until it gets to sound.s), or something to do with my cross compiler tool set.  When it comes to troubleshooting my setup of cross compiler tools, I will need some help stepping through this, because there is something my head just isn't wrapping around when it comes to this.

 

I am planning on spending more time on this tomorrow.

Sorry you couldn't get this to work. From your log I see that display.s gets compiled correctly now but sound.s isn't. I just went and double checked, no issues here. I can only assume that for some reason it's using the unmodded rmac version. Can you double check that your Makefile.config points to the freshly modded/compiled rmac? (MADMAC line at the top of the file).

 

Also, this is not a C compiler program as the errors seem to lie on rmac, so for now you should focus there.

[+BitJag]

10 hours ago, Stephen Moss said:

I don't know if belboz' Slax.iso uses the files you want but IIRC that is a bootable Linux (or Linux style) interface so if you want to keep working in a Linux style environment perhaps it could be easily adapted to use the files you need.

Thanks for the suggestions for the slax image.  This is actually where I originally got the rebuilt cross compiler from that I was still using up to this point, so my current dev environment is a grand child of this image that Michael Hill put together 10 years ago.  My main goal with this thread is to see if we can bring the removers library out from under the now unsupported a.out binary emulation that Michael Hills tools set relies on.

9 hours ago, ggn said:

Sorry you couldn't get this to work. From your log I see that display.s gets compiled correctly now but sound.s isn't. I just went and double checked, no issues here. I can only assume that for some reason it's using the unmodded rmac version. Can you double check that your Makefile.config points to the freshly modded/compiled rmac? (MADMAC line at the top of the file).

 

Also, this is not a C compiler program as the errors seem to lie on rmac, so for now you should focus there.

So I have attempted this a few times from scratch to see if there was some kind of small mistake I made along the way, but I am still coming out with the same error.  I did a screen recording and I have also attached my build logs for rmac, jlibc, and rmvlib to see if there is something you might see that I am missing due to inexperience.

 

A written break down of what I am doing.

1. Create a folder to work from in my home directory for JAGPATH variable in make files
2. Set that environment variable
3. create a lib and src folder. lib folder is for jlibc and rmvlib to install header files and other essential files after they successfully build. src folder is for building everything
4. git clone jlibc, rmvlib and rmac into src folder
5. Install cross tools and gcc from m68k-mint-atari paa
6. Make suggested change to source in rmac, and then build rmac
7. Change Makefiles in jlibc to point to newly built rmac, and remove the need to build doxygen stuff
8. make and install jlibc
9. Change Makefiles in rmvlib to point to newly built rmac, and remove the need to build doxygen stuff
10. make and get error.

 

I am assuming it is something to do with my environment at this point, so I am going to spend more time in the makefiles to see if there is something I still need to change there.  Also, I may spin up a clean version of linux to work from to remove other potential issues. Thanks again for your time and help on this!

jlibc_build_log.txt rmac_build_log.txt rmvlib_build_log.txt

[+BitJag]

Got it!  Thank you so much ggn.

 

I had to alter paula.s and sound.s in the same way that display.s was modified.  Just replacing that * token with an equate as explained by ggn above.

 

I then downloaded and modified Seb's example 1 makefiles to point to the appropriate executables and include/lib folders.  I had to patch and rebuild RLN so RLN doesn't complain about the -y flag that is used with ALN, using Seb's patch on jagware .

 

And as a last step, I had to manually lin libgcc.a to the JAGPATH/lib/lib folder so RLN can find it.

 

ggn, let me know if there is any information you need from me.  I am going to try build Crescent Memories with this as a final test, and then I want to build a quick bash file with the patches as a way of recording this process for anyone else who might be interested in getting these libraries up and running on Linux in the future.

[+BitJag]

So Crescent Memories builds without much trouble.  I had to comment out any calls to open_custom_console(), a function from the removers library.  I am thinking this is a bug in the removers library, as calling the same function in Seb's example gives the same result. I am not 100% sure though.  I will attach a log and the c file from the example so other know what I mean. 

 

Not having this function is a small sacrifice for knowing that I can continue to dev in a modern OS for the foreseeable future.  Huge thank you ggn! Really, this made my year.

 

Now off to making that bash file.

rmvlib_open_custom_console_bug.log.txt example1.c

[+BitJag]

For ggn and others to test, here is an archive with bash file that will download, modify, patch, build, and install gcc mint cross compiler tools from Ubuntu PPA, RMAC, RLN, JLIBC and RMVLIB.  It also has a modified version of Seb's example1 program to test the installation of the tool set. Also included is a uninstall script that will reverse everything the install script does.

 

The script installs everything, but the gcc cross compiler tools, to your home directory in a folder called "Jaguar".  Also, the environment variable JAGPATH is only set for the session, and it will disappear if you logout/shutdown.  If you want this to be persistent, add the following line to /etc/environment or to /home/<your user name>/.bash_profile  .  You will need to logout/reboot in order for the change to reflect on your set environment variables.

 

export JAGPATH=/home/<your user name>/Jaguar

 

A few notes:

 

This has only been tested in one distribution, Kubuntu 19.10.  Should work on any flavor of ubuntu, and any other flavor of linux if you can properly install the m68k mint atari cross compiler tools.

 

This script is dependent on specific versions of these tools, and the script may break in the future due to continued development.  GCC 4.6.7, RMAC 2.0.0, RLN 1.6.1, JLIBC 0.5.11, RMVLIB 1.3.6.

 

You will need to download and build virtualjaguar, or JCP to dump to skunk, in order to test the compiled example1.cof file yourself.  I will eventually extend this script to do these two things along with a few other tools for creating graphics.  If anyone is interested in a more comprehensive script, let me know and I will make sure to post it here when the script gets to that point.

RMAC_RLN_JLIBC_RMVLIB_Setup_Scripts.zip

[ggn]

Okay, many thanks for sticking with this - there's nothing more annoying than setting up the toolchain and build system and THEN having to fight with buggy tools :).

 

Think of the current rmac workarounds as temporary. Me and Shamus will try to get to the bottom of this and fix these issues in the official source tree so nobody has to undergo the ordeal you did! I'll give your script a go soon though, thanks for sharing.

 

The rmac authors hope that people use it in their toolchain to create useful software and move away from moldy old software that require VMs to use :).

 

P.S. any other issue you encounter with rmac, let us know!

[Welshworrier]

Nothing to see

 

Edited October 27, 2019 by Welshworrier
Because I reread above

[+BitJag]

23 hours ago, ggn said:

Okay, many thanks for sticking with this - there's nothing more annoying than setting up the toolchain and build system and THEN having to fight with buggy tools :).

 

Think of the current rmac workarounds as temporary. Me and Shamus will try to get to the bottom of this and fix these issues in the official source tree so nobody has to undergo the ordeal you did! I'll give your script a go soon though, thanks for sharing.

 

The rmac authors hope that people use it in their toolchain to create useful software and move away from moldy old software that require VMs to use :).

 

P.S. any other issue you encounter with rmac, let us know!

No problem.  I really can't emphasize enough how liberating this is to get out from under executable formats that are slowly disappearing.  Huge thanks to you and Shamus, and any others involved, for your efforts to modernize these tools. 

 

I will look forward to an update of rmac that doesn't require the minor alteration needed to get this to work, and I will definitely keep you informed if we find anything unusual with rmac.  Thanks again!

[Zerosquare]

By the way, have you tried pinging SebRmv on his e-mail address?