XEX Filter tool

vitoco · January 12, 2020

In the 80's I wrote many tools to manage binary files, in order analyze them (see the XEX file structure), join them (insert a splash screen), remove portions from them (remove a slash screen), split them, reallocate them (load in another address), squeeze them (extracts runs of zeros to reduce file size), refill (removes headings and puts heart bytes, usefull for Mac65 assembies), change the loading sequence of the chunks, remove headers and get just a block of data (or picture to edit), etc. Those mini tools were all written in Atari BASIC and a set of USR routines to perform "fast I/O" (the same as BGET and BPUT from TBXL).

During the recent years, I've been writing some simple scripts to perform some of those things directly from my PC, just to manage XEX file quickly in the same way I did before, but without having to setup an ATR and loading my tools in an emulator. Some days ago, I decided to merge all those tools in a single one, and XEX Filter toolkit was born.

"xex-filter.pl" is a small script written in perl language, and runs on any machine with the perl interpreter installed (no special modules are required). It might be a linux box (perl is native there), Windows (with Strawberry Perl or ActivePerl) or Mac. It don't have a GUI, it just works in the command line, and there are many options to combine and apply one or more actions to a file or set of them and get a new one.

Running it without parameters shows the following help:

xex-filter version 1.5 (2020-01-11)
Copyright (c) 2020 by Victor Parada <https://www.vitoco.cl/atari/>

Usage: xex-filter.pl [-option ...] [--] FILE1.XEX [FILE2.XEX ...]

Options:
  -c c1,c2-c3,...  Chunks list to select in process
  -s a1,a2-a3,...  List of addresses where to split chunks
  -e a1,a2-a3,...  List of memory addresses for data extraction
  -a a1,a2-a3,...  New address list to assign to output chunks
  -z max           Max number of bytes to fill between chunks
  -r               Removes zeros if there are more than 4 in a row
  -d               Writes data without address pointers or header
  -b               Reads input as a single chunk of data at address $0000
  -f               Fix corrupt files by filling or discarding data
  -m               Makes a memory map with the selected chunks
  -o NEW.XEX       Output file (mandatory for -c -s -e -a -z -r -d and -m)

Addresses between 0 and 65535 or in $hhhh format. Use "-" for a range.

The most simple usage is just giving it a path to an XEX file, and it will display the file structure.

For example, running without parameters over Alley Cat (downloaded from Atarimania) it gives the following report:

C:\Atari\XEX-Filter>xex-filter.pl "Alley Cat.xex"
Analyzing "Alley Cat.xex"...
-: 65535 [$FFFF] (binhead)
1: 708-712 [$02C4-$02C8] (5) PROG/DATA
-: 65535 [$FFFF] (binhead)
2: 48724-48920 [$BE54-$BF18] (197) PROG/DATA
3: 48925-48930 [$BF1D-$BF22] (6) PROG/DATA
4: 48938-48967 [$BF2A-$BF47] (30) PROG/DATA
5: 48974-49151 [$BF4E-$BFFF] (178) PROG/DATA
6: 560-561 [$0230-$0231] (2) -> 48724 [$BE54]
7: 708-712 [$02C4-$02C8] (5) PROG/DATA
8: 1024-1097 [$0400-$0449] (74) PROG/DATA
9: 736-737 [$02E0-$02E1] (2) -> 1024 [$0400]
10: 7936-9048 [$1F00-$2358] (1113) PROG/DATA
11: 9053-9056 [$235D-$2360] (4) PROG/DATA
12: 9061-9064 [$2365-$2368] (4) PROG/DATA
13: 9069-9159 [$236D-$23C7] (91) PROG/DATA
... (deleted)
46: 22122-22534 [$566A-$5806] (413) PROG/DATA
47: 22539-24320 [$580B-$5F00] (1782) PROG/DATA
48: 24320-24408 [$5F00-$5F58] (89) PROG/DATA
49: 24416-24471 [$5F60-$5F97] (56) PROG/DATA
... (deleted)
151: 40319-40320 [$9D7F-$9D80] (2) -> 1 [$0001]
152: 40327-40328 [$9D87-$9D88] (2) -> 60 [$003C]
153: 40337-40349 [$9D91-$9D9D] (13) PROG/DATA
154: 40512-40513 [$9E40-$9E41] (2) -> 255 [$00FF]
155: 40526-40530 [$9E4E-$9E52] (5) PROG/DATA

Clearly it was squeezed (zeros were removed), so we can restore them adding up to 128 zero bytes between chunks:

C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT.XEX -z 128 "Alley Cat.xex"
Analyzing "Alley Cat.xex"...
... (same info than before removed)
Writing "ALLEYCAT.XEX"...
-: 65535 [$FFFF] (binhead)
1: 708-712 [$02C4-$02C8] (5)
2: 48724-49151 [$BE54-$BFFF] (428)
3: 560-561 [$0230-$0231] (2)
4: 708-712 [$02C4-$02C8] (5)
5: 1024-1097 [$0400-$0449] (74)
6: 736-737 [$02E0-$02E1] (2)
7: 7936-24320 [$1F00-$5F00] (16385)
8: 24320-40349 [$5F00-$9D9D] (16030)
9: 40512-40530 [$9E40-$9E52] (19)
32988 bytes written

Adding more than 128 bytes would also fill OS data between the splash display list pointers at address 560 and the color registers at 708-712, but some more zeros were required to fill the gap between address 40349 ($9D9D) and 40512 ($9E40), that can be also added with this tool if the splash screen is removed before and added again after the fill, with thi commands (output info omitted):

C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-splash.XEX -c 1-6 ALLEYCAT.XEX
...
C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-game.XEX -c 7-9 ALLEYCAT.XEX
...
C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-game-filled.XEX -z 1000 ALLEYCAT-game.XEX
...
C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-new.XEX ALLEYCAT-splash.XEX ALLEYCAT-game-filled.XEX
...

What it is strange is that there are 2 chunks that overlaps at memory address 24320 ($5F00), and it could also be analyzed with this tool by splitting the file at that byte and running it to display the resulting XEX structure:

C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-split.XEX -s $5F00-$5F00 ALLEYCAT-new.XEX
...
C:\Atari\XEX-Filter>xex-filter.pl ALLEYCAT-split.XEX
Analyzing "ALLEYCAT-split.XEX"...
-: 65535 [$FFFF] (binhead)
1: 708-712 [$02C4-$02C8] (5) PROG/DATA
2: 48724-49151 [$BE54-$BFFF] (428) PROG/DATA
3: 560-561 [$0230-$0231] (2) -> 48724 [$BE54]
4: 708-712 [$02C4-$02C8] (5) PROG/DATA
5: 1024-1097 [$0400-$0449] (74) PROG/DATA
6: 736-737 [$02E0-$02E1] (2) -> 1024 [$0400]
7: 7936-24319 [$1F00-$5EFF] (16384) PROG/DATA
8: 24320-24320 [$5F00-$5F00] (1) $00 "00000000"
9: 24320-24320 [$5F00-$5F00] (1) $10 "00010000"
10: 24321-40530 [$5F01-$9E52] (16210) PROG/DATA

Hum!!! Looking at chunks number 8 and 9, we can see two different values for the same memory location. A bug from the packager? As it normally loads sequentially, we can remove the first one at chunk 8 and merge chuncks 7, 9 and 10, and also move the startup routine and RUNAD pointer to the end of the file, all this steps in one command:

C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-final.XEX -z 0 -c 1-4,7,9-10,5-6 ALLEYCAT-split.XEX
Analyzing "ALLEYCAT-split.XEX"...
-: 65535 [$FFFF] (binhead)
1: 708-712 [$02C4-$02C8] (5) PROG/DATA
2: 48724-49151 [$BE54-$BFFF] (428) PROG/DATA
3: 560-561 [$0230-$0231] (2) -> 48724 [$BE54]
4: 708-712 [$02C4-$02C8] (5) PROG/DATA
5: 1024-1097 [$0400-$0449] (74) PROG/DATA
6: 736-737 [$02E0-$02E1] (2) -> 1024 [$0400]
7: 7936-24319 [$1F00-$5EFF] (16384) PROG/DATA
8: 24320-24320 [$5F00-$5F00] (1) $00 "00000000"
9: 24320-24320 [$5F00-$5F00] (1) $10 "00010000"
10: 24321-40530 [$5F01-$9E52] (16210) PROG/DATA
Writing "ALLEYCAT-final.XEX"...
-: 65535 [$FFFF] (binhead)
1: 708-712 [$02C4-$02C8] (5)
2: 48724-49151 [$BE54-$BFFF] (428)
3: 560-561 [$0230-$0231] (2)
4: 708-712 [$02C4-$02C8] (5)
5: 7936-40530 [$1F00-$9E52] (32595)
6: 1024-1097 [$0400-$0449] (74)
7: 736-737 [$02E0-$02E1] (2)
33141 bytes written

There are many more tricks that could be done with this tool, and I've attached it here to share it with the community.

Depending on the machine where this script will run, it might be needed to change the first line to any of the following lines:

#!/usr/bin/perl
#!/bin/perl
#!/usr/local/bin/perl
#!/usr/sbin/perl

Bug reports, feature suggestions and any kind of comments are welcome!

Have fun!

++vitoco

xex-filter.pl

ascrnet · January 12, 2020

Thank you for sharing your tool, as long as I have time I will try it ?

regards

baktra · January 12, 2020

This is a good tool. It seems that all of us who need to process the binary load files create such a tool sooner or later.

I like the way you can select multiple chunks from the command line.

I hope the tool will earn its rightful place at your website or a public repository, so it doesn't get lost in this forum.

vitoco · January 12, 2020

3 hours ago, baktra said:

This is a good tool. It seems that all of us who need to process the binary load files create such a tool sooner or later.

Yep, I decided to merge all of those specialized tools into a general and flexible one, to avoid having to modify existing ones or to create new ones each time.

With this one, you can convert a font file to a XEX chunk and include it in a game, or extract a charset from a XEX to be edited and then put it back to customize fonts... Or even to do the same with an image! ?

3 hours ago, baktra said:

I like the way you can select multiple chunks from the command line.

You can also specify ranges in reverse order, and get a squeezed file that loads backwards... ? Just try:

C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-reverse.XEX -c 1,5-2,6-9,155-49,47,48,46-10 "Alley Cat.xex"

The game range cannot be directly 155-10 because of the double byte at $5F00, and the latest must be preserved, or the game will crash.

3 hours ago, baktra said:

I hope the tool will earn its rightful place at your website or a public repository, so it doesn't get lost in this forum.

You are right. I'll add a page for it in my website in order to publish the most recent version there. I already found a missing feature that I will add in the following days (or hours).?

dmsc · January 13, 2020

Hi!

On 1/11/2020 at 10:35 PM, vitoco said:

"xex-filter.pl" is a small script written in perl language, and runs on any machine with the perl interpreter installed (no special modules are required). It might be a linux box (perl is native there), Windows (with Strawberry Perl or ActivePerl) or Mac. It don't have a GUI, it just works in the command line, and there are many options to combine and apply one or more actions to a file or set of them and get a new one.

Great!

Another very similar tool is "ataricom" by @HiassofT, https://github.com/HiassofT/AtariSIO/blob/master/README-tools . I use it all the time to analyze XEX files, this is the short help:

ataricom 0.30-190211
(c) 2008-2019 Matthias Reichl <hias@horus.com>
usage: ataricom [options]... file [outfile]
  -c address      create COM file from raw data file
  -r address      add RUN block with specified address at end of file
  -i address      add INIT block with specified address at end of file
  -b start[-end]  only process specified blocks
  -x start[-end]  exclude specified blocks
  -m start-end    merge specified blocks
  -s block,adr... split block at given addresses
  -n              write raw data blocks (no COM headers)
  -X              show block length and file offset in hex

In analysis mode it also shows the INIT and RUN addresses after any chunk that writes that ranges, you could add that to your tool.

Have Fun!

vitoco · January 13, 2020

2 hours ago, dmsc said:
Hi!

Great!

Another very similar tool is "ataricom" by @HiassofT, https://github.com/HiassofT/AtariSIO/blob/master/README-tools . I use it all the time to analyze XEX files, this is the short help:
ataricom 0.30-190211
(c) 2008-2019 Matthias Reichl <hias@horus.com>
usage: ataricom [options]... file [outfile]
  -c address      create COM file from raw data file
  -r address      add RUN block with specified address at end of file
  -i address      add INIT block with specified address at end of file
  -b start[-end]  only process specified blocks
  -x start[-end]  exclude specified blocks
  -m start-end    merge specified blocks
  -s block,adr... split block at given addresses
  -n              write raw data blocks (no COM headers)
  -X              show block length and file offset in hex
In analysis mode it also shows the INIT and RUN addresses after any chunk that writes that ranges, you could add that to your tool.

Have to look at it...

About INIAD and RUNAD, that was what I thought it was missing, even when they could be added using some tricks with my tool. But I extended that function and it is possible to add any word (-w) or byte (-v) at any location. This is how it looks like now the help screen:

xex-filter version 1.6 (2020-01-12)
Copyright (c) 2020 by Victor Parada <https://www.vitoco.cl/atari/>

Usage: xex-filter.pl [-option ...] [--] FILE1.XEX [FILE2.XEX ...]

Options:
  -c c1,c2-c3,...  Chunks list to select in process
  -s a1,a2-a3,...  List of addresses where to split chunks
  -e a1,a2-a3,...  List of memory addresses for data extraction
  -a a1,a2-a3,...  New address list to assign to output chunks
  -v a1=v1,v2,...  Append chunks with byte value at $0000 or address
  -w a1=v1,v2,...  Append chunks with word value at RUNAD or address
  -z max           Max number of bytes to fill between chunks
  -r               Removes zeros if there are more than 4 in a row
  -d               Writes data without address pointers or header
  -b               Reads input as chunks up to 64K of data at $0000
  -f               Fix corrupt files by filling or discarding data
  -m               Makes a memory map with the selected chunks
  -o NEW.XEX       Output file (mandatory for -c -s -e -a -z -r -d and -m)

Addresses between 0 and 65535 or in $hhhh format. Use "-" for a range.

I'll share it later, when I publish it in my webpage.

vitoco · January 13, 2020

I forgot to mention that this tool has always been displaying the RUNAD and INITAD addresses during the XEX analysis, as well as any other vector that it is loaded as a chunk. That can be seen in my first example in this thread, where RUNAD ($02E0) and SDLSTL ($0230) vectors are displayed in chunks 9 and 6 respectively (without those labels anyway). It did (and does) the same with chunks of a single byte, displaying it in hex and it's bits.

I haven't tested ataricom yet because I don't have the required environment to compile its sources, but I could see from the help screen @dmsc shared that it refers to "blocks" instead of "chunks". As my tool was programmed in Spanish and all the messages were also in Spanish, in order to release the tool here, I just tried to find a proper English name for "trozo" and I liked "chunk". I also changed the assigned letter to some options to match its new name and I had to get used to that! Now, open question to the community: should I change that to "block"?

Edited January 13, 2020 by vitoco
More info added

baktra · January 13, 2020

3 hours ago, vitoco said:

I haven't tested ataricom yet because I don't have the required environment to compile its sources, but I could see from the help screen @dmsc shared that it refers to "blocks" instead of "chunks". As my tool was programmed in Spanish and all the messages were also in Spanish, in order to release the tool here, I just tried to find a proper English name for "trozo" and I liked "chunk". I also changed the assigned letter to some options to match its new name and I had to get used to that! Now, open question to the community: should I change that to "block"?

This is a very good question.

I believe block or segment, would do. It might be good idea to check scans of literature from Atari to see how they named those.

For example Atari DOS 2 reference doesn't specify any name. I would recommend to avoid the word chunk, as it is associated with elements of tape images (.cas).

vitoco · January 19, 2020

After some suggestions and improvements, I've published version 1.6 in my page: https://www.vitoco.cl/atari/xex-filter/


xex-filter version 1.6 (2020-01-18)
Copyright (c) 2020 by Victor Parada
<https://www.vitoco.cl/atari/xex-filter/>

Usage: xex-filter.pl [-option]... [--] FILE1.XEX [FILE2.XEX]...

Options:
  -i n1,n2-n3,...  List of indexes for blocks selection
  -s a1,a2-a3,...  List of addresses where to split blocks
  -e a1,a2-a3,...  List of memory addresses for data extraction
  -a a1,a2-a3,...  New address list to assign to output blocks
  -v a1=v1,v2,...  Append blocks with byte value at $0000 or address
  -w a1=v1,v2,...  Append blocks with word value at RUNAD or address
  -z max           Max number of bytes to fill between blocks
  -r               Removes zeros if there are more than 4 in a row
  -d               Writes data without address pointers or header
  -b               Reads input as blocks up to 64K of data at $0000
  -f               Fix corrupt files by filling or discarding data
  -m               Makes a memory map with the selected blocks
  -o NEW.XEX       Output file (mandatory for -c -s -e -a -z -r -d and -m)

Addresses between 0 and 65535 or in $hhhh format. Use "-" for a range.

As it could be seen, I've changed "chunks" to "blocks", which also made me to change an option letter. Now, it also supports some labels for common addresses like INITAD, RUNAD and PORTB, for both command options and for analysis output.

The download link is in my page... Comments are welcome here as ever.

Have fun!!!

+DjayBee · February 1, 2020

Hi vitoco,

downloading the file gives only an error 500.

candle · February 1, 2020

this is great!

having big chunks of data to be loaded off modern media (cf, sd, hard drives) is more efficient than small ones thus greatly reducing loading speeds

vitoco · February 1, 2020

Thanks for the info. I'll check for that.

In the meanwhile, latest version 1.6 (2020-01-18) is attached.

xex-filter.pl

FULS · May 15, 2020

Víctor,

I also Love your program "ATASCII printer V.0.4". ATASCIIprinter V.0.4.htm

Is there a way to save the Basic file to Notepad instead of a BMP file?

Thanks,

Doug

vitoco · May 15, 2020

9 hours ago, FULS said:

Víctor,

I also Love your program "ATASCII printer V.0.4". ATASCIIprinter V.0.4.htm

Is there a way to save the Basic file to Notepad instead of a BMP file?

What do you need?

The ATASCII Printer service takes a source listing in ATASCII code and "draws" it in a series of BMP images (one image per page column), but if you provide a (Turbo) BASIC tokenized file, the printer untokenizes it on the fly.

If you just want to get the untokenized file, you might use some of the already available detokenizers or procedures (read, read, read, ...).

I guess that I could isolate the on-the-fly detokenizer from my ATASCII Printer service to create source TXT files, but it won't work very well because they will be in ATASCII coding instead of ASCII for the PC, unless some extra code is added to filter newlines and other control characters, including inverse video. I'll think about this... ;-)

lbaeza · May 15, 2020

13 hours ago, FULS said:

Víctor,

I also Love your program "ATASCII printer V.0.4". ATASCIIprinter V.0.4.htm

Is there a way to save the Basic file to Notepad instead of a BMP file?

Thanks,

Doug

+1

+DjayBee · May 15, 2020

14 hours ago, FULS said:

Is there a way to save the Basic file to Notepad instead of a BMP file?

If you are running Windows then get Memo Pad.

It can read text files and tokenized BASIC or MAC/65 files and save to ATASCII or ASCII.

FULS · May 15, 2020

Hi DjayBee,

I could not get this tokenized file loaded with Memo Pad without "Error importing Basic file. var name len error"

SP.bas (Strip Poker)

The var name table was Destroyed/Protected by the author.

ATASCII printer V.0.4 displays it, but only in .BMP picture. Maybe ATASCII printer V.0.4 could make a Hex file output to load into a Hex Editor.

Or fix Memo Pad to load it with a replaced var name table.

I know there are other Atari progams out there that will do this, but these two are Windows based and thus more convenient.

Thanks,

Doug

+DjayBee · May 15, 2020

19 minutes ago, FULS said:

The var name table was Destroyed/Protected by the author.

Loaded to Altirra and

.basic_rebuildvnt
.basic_save trythis.bas

It gives an error when loading it to Memo Pad. Remove the last line 32310.

Disclaimer: I did NOT try if it really works.

trythis.bas

FULS · May 15, 2020

Yes, I forgot to mention Altirra. Very, very great Emulator, full of features.

Also ATRTools helps also with Basic files. But, Altirra and ATRTools are not the simplest programs to use, because they are so powerful.

'ATASCII printer V.0.4' and 'Memo Pad' would be so simple if they added some new features.

Thanks,

Doug

vitoco · May 16, 2020

4 hours ago, FULS said:

'ATASCII printer V.0.4' and 'Memo Pad' would be so simple if they added some new features.

I downloaded your protected file and filter it through the detokenizer of the Printer service, and got the attached file. Note that the line 32310 (the last one) is corrupted, so it has to be deleted after ENTERed again into BASIC (it should display a syntax error) and before SAVED again into a disk drive. I haven't tried it anyway.

Anyway, I won't add such a "feature" to the printer, because it does what I wanted and I designed it for "pretty printing" purposes only, in order to display ATASCII characters like type-in games published in old magazines. I think I might create a new tool for this... let me ask to my pillow.

SP.lst

FULS · May 16, 2020

31 minutes ago, vitoco said:

I think I might create a new tool for this... let me ask to my pillow.

SP.lst 19.44 kB · 2 downloads

Your .lst file is exactly what I was after!!

I have other protected Basic programs, so if you find the time to create a new tool, I know I will enjoy it!

Thank You,

Doug

_The Doctor__ · May 16, 2020

heh heh heh, poke 82,0

+Stephen · May 16, 2020

55 minutes ago, _The Doctor__ said:

heh heh heh, poke 82,0

Isn't it funny how many memory locations and error codes we can still remember on these old machines?

ivop · May 16, 2020

When I got back in the scene ten years ago, I still remembered $D40A

And poke 580,1 for coldtart.

Edited May 16, 2020 by ivop

rossum · May 16, 2020

Sirs,

You might find this tool useful. It does BAS and XEX disassembly in a browser and can produce nice printable Basic listings in the classic font.

lmk if it is useful or if you have any feature requests.

cheers

atr_image_explorer.htm

XEX Filter tool

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Join the conversation

Recently Browsing 0 members