Jump to content

Recommended Posts

In the 80's I wrote many tools to manage binary files, in order analyze them (see the XEX file structure), join them (insert a splash screen), remove portions from them (remove a slash screen), split them, reallocate them (load in another address), squeeze them (extracts runs of zeros to reduce file size), refill (removes headings and puts heart bytes, usefull for Mac65 assembies), change the loading sequence of the chunks, remove headers and get just a block of data (or picture to edit), etc. Those mini tools were all written in Atari BASIC and a set of USR routines to perform "fast I/O" (the same as BGET and BPUT from TBXL).

 

During the recent years, I've been writing some simple scripts to perform some of those things directly from my PC, just to manage XEX file quickly in the same way I did before, but without having to setup an ATR and loading my tools in an emulator. Some days ago, I decided to merge all those tools in a single one, and XEX Filter toolkit was born.

 

"xex-filter.pl" is a small script written in perl language, and runs on any machine with the perl interpreter installed (no special modules are required). It might be a linux box (perl is native there), Windows (with Strawberry Perl or ActivePerl) or Mac. It don't have a GUI, it just works in the command line, and there are many options to combine and apply one or more actions to a file or set of them and get a new one.


Running it without parameters shows the following help:

xex-filter version 1.5 (2020-01-11)
Copyright (c) 2020 by Victor Parada <https://www.vitoco.cl/atari/>

Usage: xex-filter.pl [-option ...] [--] FILE1.XEX [FILE2.XEX ...]

Options:
  -c c1,c2-c3,...  Chunks list to select in process
  -s a1,a2-a3,...  List of addresses where to split chunks
  -e a1,a2-a3,...  List of memory addresses for data extraction
  -a a1,a2-a3,...  New address list to assign to output chunks
  -z max           Max number of bytes to fill between chunks
  -r               Removes zeros if there are more than 4 in a row
  -d               Writes data without address pointers or header
  -b               Reads input as a single chunk of data at address $0000
  -f               Fix corrupt files by filling or discarding data
  -m               Makes a memory map with the selected chunks
  -o NEW.XEX       Output file (mandatory for -c -s -e -a -z -r -d and -m)

Addresses between 0 and 65535 or in $hhhh format. Use "-" for a range.

The most simple usage is just giving it a path to an XEX file, and it will display the file structure.

 

For example, running without parameters over Alley Cat (downloaded from Atarimania) it gives the following report:

C:\Atari\XEX-Filter>xex-filter.pl "Alley Cat.xex"
Analyzing "Alley Cat.xex"...
-: 65535 [$FFFF] (binhead)
1: 708-712 [$02C4-$02C8] (5) PROG/DATA
-: 65535 [$FFFF] (binhead)
2: 48724-48920 [$BE54-$BF18] (197) PROG/DATA
3: 48925-48930 [$BF1D-$BF22] (6) PROG/DATA
4: 48938-48967 [$BF2A-$BF47] (30) PROG/DATA
5: 48974-49151 [$BF4E-$BFFF] (178) PROG/DATA
6: 560-561 [$0230-$0231] (2) -> 48724 [$BE54]
7: 708-712 [$02C4-$02C8] (5) PROG/DATA
8: 1024-1097 [$0400-$0449] (74) PROG/DATA
9: 736-737 [$02E0-$02E1] (2) -> 1024 [$0400]
10: 7936-9048 [$1F00-$2358] (1113) PROG/DATA
11: 9053-9056 [$235D-$2360] (4) PROG/DATA
12: 9061-9064 [$2365-$2368] (4) PROG/DATA
13: 9069-9159 [$236D-$23C7] (91) PROG/DATA
... (deleted)
46: 22122-22534 [$566A-$5806] (413) PROG/DATA
47: 22539-24320 [$580B-$5F00] (1782) PROG/DATA
48: 24320-24408 [$5F00-$5F58] (89) PROG/DATA
49: 24416-24471 [$5F60-$5F97] (56) PROG/DATA
... (deleted)
151: 40319-40320 [$9D7F-$9D80] (2) -> 1 [$0001]
152: 40327-40328 [$9D87-$9D88] (2) -> 60 [$003C]
153: 40337-40349 [$9D91-$9D9D] (13) PROG/DATA
154: 40512-40513 [$9E40-$9E41] (2) -> 255 [$00FF]
155: 40526-40530 [$9E4E-$9E52] (5) PROG/DATA

Clearly it was squeezed (zeros were removed), so we can restore them adding up to 128 zero bytes between chunks:

C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT.XEX -z 128 "Alley Cat.xex"
Analyzing "Alley Cat.xex"...
... (same info than before removed)
Writing "ALLEYCAT.XEX"...
-: 65535 [$FFFF] (binhead)
1: 708-712 [$02C4-$02C8] (5)
2: 48724-49151 [$BE54-$BFFF] (428)
3: 560-561 [$0230-$0231] (2)
4: 708-712 [$02C4-$02C8] (5)
5: 1024-1097 [$0400-$0449] (74)
6: 736-737 [$02E0-$02E1] (2)
7: 7936-24320 [$1F00-$5F00] (16385)
8: 24320-40349 [$5F00-$9D9D] (16030)
9: 40512-40530 [$9E40-$9E52] (19)
32988 bytes written

Adding more than 128 bytes would also fill OS data between the splash display list pointers at address 560 and the color registers at 708-712, but some more zeros were required to fill the gap between address 40349 ($9D9D) and 40512 ($9E40), that can be also added with this tool if the splash screen is removed before and added again after the fill, with thi commands (output info omitted):

C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-splash.XEX -c 1-6 ALLEYCAT.XEX
...
C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-game.XEX -c 7-9 ALLEYCAT.XEX
...
C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-game-filled.XEX -z 1000 ALLEYCAT-game.XEX
...
C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-new.XEX ALLEYCAT-splash.XEX ALLEYCAT-game-filled.XEX
...

 What it is strange is that there are 2 chunks that overlaps at memory address 24320 ($5F00), and it could also be analyzed with this tool by splitting the file at that byte and running it to display the resulting XEX structure:

C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-split.XEX -s $5F00-$5F00 ALLEYCAT-new.XEX
...
C:\Atari\XEX-Filter>xex-filter.pl ALLEYCAT-split.XEX
Analyzing "ALLEYCAT-split.XEX"...
-: 65535 [$FFFF] (binhead)
1: 708-712 [$02C4-$02C8] (5) PROG/DATA
2: 48724-49151 [$BE54-$BFFF] (428) PROG/DATA
3: 560-561 [$0230-$0231] (2) -> 48724 [$BE54]
4: 708-712 [$02C4-$02C8] (5) PROG/DATA
5: 1024-1097 [$0400-$0449] (74) PROG/DATA
6: 736-737 [$02E0-$02E1] (2) -> 1024 [$0400]
7: 7936-24319 [$1F00-$5EFF] (16384) PROG/DATA
8: 24320-24320 [$5F00-$5F00] (1) $00 "00000000"
9: 24320-24320 [$5F00-$5F00] (1) $10 "00010000"
10: 24321-40530 [$5F01-$9E52] (16210) PROG/DATA

Hum!!! Looking at chunks number 8 and 9, we can see two different values for the same memory location. A bug from the packager? As it normally loads sequentially, we can remove the first one at chunk 8 and merge chuncks 7, 9 and 10, and also move the startup routine and RUNAD pointer to the end of the file, all this steps in one command:

C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-final.XEX -z 0 -c 1-4,7,9-10,5-6 ALLEYCAT-split.XEX
Analyzing "ALLEYCAT-split.XEX"...
-: 65535 [$FFFF] (binhead)
1: 708-712 [$02C4-$02C8] (5) PROG/DATA
2: 48724-49151 [$BE54-$BFFF] (428) PROG/DATA
3: 560-561 [$0230-$0231] (2) -> 48724 [$BE54]
4: 708-712 [$02C4-$02C8] (5) PROG/DATA
5: 1024-1097 [$0400-$0449] (74) PROG/DATA
6: 736-737 [$02E0-$02E1] (2) -> 1024 [$0400]
7: 7936-24319 [$1F00-$5EFF] (16384) PROG/DATA
8: 24320-24320 [$5F00-$5F00] (1) $00 "00000000"
9: 24320-24320 [$5F00-$5F00] (1) $10 "00010000"
10: 24321-40530 [$5F01-$9E52] (16210) PROG/DATA
Writing "ALLEYCAT-final.XEX"...
-: 65535 [$FFFF] (binhead)
1: 708-712 [$02C4-$02C8] (5)
2: 48724-49151 [$BE54-$BFFF] (428)
3: 560-561 [$0230-$0231] (2)
4: 708-712 [$02C4-$02C8] (5)
5: 7936-40530 [$1F00-$9E52] (32595)
6: 1024-1097 [$0400-$0449] (74)
7: 736-737 [$02E0-$02E1] (2)
33141 bytes written

There are many more tricks that could be done with this tool, and I've attached it here to share it with the community.

 

Depending on the machine where this script will run, it might be needed to change the first line to any of the following lines:

#!/usr/bin/perl
#!/bin/perl
#!/usr/local/bin/perl
#!/usr/sbin/perl

Bug reports, feature suggestions and any kind of comments are welcome!

 

Have fun!

 

++vitoco

 

xex-filter.pl

  • Like 11
  • Thanks 2

Share this post


Link to post
Share on other sites

Thank you for sharing your tool, as long as I have time I will try it  😎

 

regards

Share this post


Link to post
Share on other sites

This is a good tool. It seems that all of us who need to process the binary load files create such a tool sooner or later.

I like the way you can select multiple chunks from the command line.

 

I hope the tool will earn its rightful place at your website or a public repository, so it doesn't get lost in this forum.

 

 

Share this post


Link to post
Share on other sites
3 hours ago, baktra said:

This is a good tool. It seems that all of us who need to process the binary load files create such a tool sooner or later.

Yep, I decided to merge all of those specialized tools into a general and flexible one, to avoid having to modify existing ones or to create new ones each time.

 

With this one, you can convert a font file to a XEX chunk and include it in a game, or extract a charset from a XEX to be edited and then put it back to customize fonts... Or even to do the same with an image! 😎

 

3 hours ago, baktra said:

I like the way you can select multiple chunks from the command line.

You can also specify ranges in reverse order, and get a squeezed file that loads backwards... 😉 Just try:

C:\Atari\XEX-Filter>xex-filter.pl -o ALLEYCAT-reverse.XEX -c 1,5-2,6-9,155-49,47,48,46-10 "Alley Cat.xex"

The game range cannot be directly 155-10 because of the double byte at $5F00, and the latest must be preserved, or the game will crash.

 

3 hours ago, baktra said:

I hope the tool will earn its rightful place at your website or a public repository, so it doesn't get lost in this forum.

You are right. I'll add a page for it in my website in order to publish the most recent version there. I already found a missing feature that I will add in the following days (or hours).😬

 

  • Like 2

Share this post


Link to post
Share on other sites

Hi!

On 1/11/2020 at 10:35 PM, vitoco said:

 

 

"xex-filter.pl" is a small script written in perl language, and runs on any machine with the perl interpreter installed (no special modules are required). It might be a linux box (perl is native there), Windows (with Strawberry Perl or ActivePerl) or Mac. It don't have a GUI, it just works in the command line, and there are many options to combine and apply one or more actions to a file or set of them and get a new one.

 

Great!

 

Another very similar tool is "ataricom" by @HiassofThttps://github.com/HiassofT/AtariSIO/blob/master/README-tools . I use it all the time to analyze XEX files, this is the short help:

ataricom 0.30-190211
(c) 2008-2019 Matthias Reichl <[email protected]>
usage: ataricom [options]... file [outfile]
  -c address      create COM file from raw data file
  -r address      add RUN block with specified address at end of file
  -i address      add INIT block with specified address at end of file
  -b start[-end]  only process specified blocks
  -x start[-end]  exclude specified blocks
  -m start-end    merge specified blocks
  -s block,adr... split block at given addresses
  -n              write raw data blocks (no COM headers)
  -X              show block length and file offset in hex

In analysis mode it also shows the INIT and RUN addresses after any chunk that writes that ranges, you could add that to your tool.

 

Have Fun!

 

Share this post


Link to post
Share on other sites
2 hours ago, dmsc said:

Hi!

Great!

 

Another very similar tool is "ataricom" by @HiassofThttps://github.com/HiassofT/AtariSIO/blob/master/README-tools . I use it all the time to analyze XEX files, this is the short help:

ataricom 0.30-190211
(c) 2008-2019 Matthias Reichl <[email protected]>
usage: ataricom [options]... file [outfile]
  -c address      create COM file from raw data file
  -r address      add RUN block with specified address at end of file
  -i address      add INIT block with specified address at end of file
  -b start[-end]  only process specified blocks
  -x start[-end]  exclude specified blocks
  -m start-end    merge specified blocks
  -s block,adr... split block at given addresses
  -n              write raw data blocks (no COM headers)
  -X              show block length and file offset in hex

In analysis mode it also shows the INIT and RUN addresses after any chunk that writes that ranges, you could add that to your tool.

Have to look at it...

 

About INIAD and RUNAD, that was what I thought it was missing, even when they could be added using some tricks with my tool. But I extended that function and it is possible to add any word (-w) or byte (-v) at any location. This is how it looks like now the help screen:

xex-filter version 1.6 (2020-01-12)
Copyright (c) 2020 by Victor Parada <https://www.vitoco.cl/atari/>

Usage: xex-filter.pl [-option ...] [--] FILE1.XEX [FILE2.XEX ...]

Options:
  -c c1,c2-c3,...  Chunks list to select in process
  -s a1,a2-a3,...  List of addresses where to split chunks
  -e a1,a2-a3,...  List of memory addresses for data extraction
  -a a1,a2-a3,...  New address list to assign to output chunks
  -v a1=v1,v2,...  Append chunks with byte value at $0000 or address
  -w a1=v1,v2,...  Append chunks with word value at RUNAD or address
  -z max           Max number of bytes to fill between chunks
  -r               Removes zeros if there are more than 4 in a row
  -d               Writes data without address pointers or header
  -b               Reads input as chunks up to 64K of data at $0000
  -f               Fix corrupt files by filling or discarding data
  -m               Makes a memory map with the selected chunks
  -o NEW.XEX       Output file (mandatory for -c -s -e -a -z -r -d and -m)

Addresses between 0 and 65535 or in $hhhh format. Use "-" for a range.

I'll share it later, when I publish it in my webpage.

 

 

Share this post


Link to post
Share on other sites

I forgot to mention that this tool has always been displaying the RUNAD and INITAD addresses during the XEX analysis, as well as any other vector that it is loaded as a chunk. That can be seen in my first example in this thread, where RUNAD ($02E0) and SDLSTL ($0230) vectors are displayed in chunks 9 and 6 respectively (without those labels anyway). It did (and does) the same with chunks of a single byte, displaying it in hex and it's bits.

 

I haven't tested ataricom yet because I don't have the required environment to compile its sources, but I could see from the help screen @dmsc shared that it refers to "blocks" instead of "chunks". As my tool was programmed in Spanish and all the messages were also in Spanish, in order to release the tool here, I just tried to find a proper English name for "trozo" and I liked "chunk". I also changed the assigned letter to some options to match its new name and I had to get used to that! Now, open question to the community: should I change that to "block"?

Edited by vitoco
More info added

Share this post


Link to post
Share on other sites
3 hours ago, vitoco said:

I haven't tested ataricom yet because I don't have the required environment to compile its sources, but I could see from the help screen @dmsc shared that it refers to "blocks" instead of "chunks". As my tool was programmed in Spanish and all the messages were also in Spanish, in order to release the tool here, I just tried to find a proper English name for "trozo" and I liked "chunk". I also changed the assigned letter to some options to match its new name and I had to get used to that! Now, open question to the community: should I change that to "block"?

This is a very good question.

 

I believe block or segment, would do. It might be good idea to check scans of literature from Atari to see how they named those.

For example Atari DOS 2 reference doesn't specify any name. I would recommend to avoid the word chunk, as it is associated with elements of tape images (.cas).

 

Share this post


Link to post
Share on other sites

After some suggestions and improvements, I've published version 1.6 in my page: https://www.vitoco.cl/atari/xex-filter/

 

xex-filter version 1.6 (2020-01-18)
Copyright (c) 2020 by Victor Parada
<https://www.vitoco.cl/atari/xex-filter/>

Usage: xex-filter.pl [-option]... [--] FILE1.XEX [FILE2.XEX]...

Options:
  -i n1,n2-n3,...  List of indexes for blocks selection
  -s a1,a2-a3,...  List of addresses where to split blocks
  -e a1,a2-a3,...  List of memory addresses for data extraction
  -a a1,a2-a3,...  New address list to assign to output blocks
  -v a1=v1,v2,...  Append blocks with byte value at $0000 or address
  -w a1=v1,v2,...  Append blocks with word value at RUNAD or address
  -z max           Max number of bytes to fill between blocks
  -r               Removes zeros if there are more than 4 in a row
  -d               Writes data without address pointers or header
  -b               Reads input as blocks up to 64K of data at $0000
  -f               Fix corrupt files by filling or discarding data
  -m               Makes a memory map with the selected blocks
  -o NEW.XEX       Output file (mandatory for -c -s -e -a -z -r -d and -m)

Addresses between 0 and 65535 or in $hhhh format. Use "-" for a range.

 

As it could be seen, I've changed "chunks" to "blocks", which also made me to change an option letter. Now, it also supports some labels for common addresses like INITAD, RUNAD and PORTB, for both command options and for analysis output.

 

The download link is in my page... Comments are welcome here as ever.

 

Have fun!!!

 

  • Thanks 1

Share this post


Link to post
Share on other sites

this is great!

having big chunks of data to be loaded off modern media (cf, sd, hard drives) is more efficient than small ones thus greatly reducing loading speeds

 

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...