VGM Compression Tool

[Tursi]

So do you mean? what would be the advantage of setting a flag inside the nmi? Wouldn't all other processes be waiting for the nmi to end anyways before continuing.

Yes, more like this:

 

nmi()
{
  updatesprites(0,16);
  flag=1;
}

... elsewhere, in the non-interrupt code...

// wait for frame sync before continuing
while (flag == 0) { }
vdp_status = VDPST; // clear VDP status
flag = 0;
stplay();
// continue with another frame

For this to work 'flag' needs to be defined as volatile.

 

That's just how I do it, coming from a background where the VDP interrupt was NOT an NMI, I'm used to saying when I'm ready to do something about it.

Edited February 19, 2016 by Tursi

[Tursi]

I should note that my example code there is really over-simplified. You need to be really careful if you do VDP access /both/ in the NMI and outside of the NMI, you need to be able to guarantee that they can't stomp on top of each other. The easiest way to guarantee it is to put all your VDP access code in the NMI - it can't be interrupted. That's not always easy to do. Or you can put it all outside the NMI, and then even if it's interrupted nothing happens. Or you can never clear the VDP interrupt, then you'll never get another NMI. However, then you can't read the status register at all, so it's awfully hard to get frame sync.

 

The approach I have in Super Space Acer (and so by extension, libti99Coleco) gives me the ability to simulate turning interrupts on and off without actually doing so (meaning that it is guaranteed atomic) by tracking the state in the NMI itself. It took a number of passes to get it going but I think it's right now, but it allows me to have 'critical sections' during which the interrupt can be turned off, and yet not lost so that when I turn it back on, any missed interrupt is processed. (Unless more than one frame was missed, of course). That way I can do VDP work without fear of NMI interrupting it and yet also have the sprite list copy inside the NMI.

 

My library is available here, it's just a port of my libti99 The interrupt interaction is in vdp.h (the enable/disable macros) and vdp_ints.c

 

https://github.com/tursilion/libti99coleco

[Pixelboy]

I should note that my example code there is really over-simplified. You need to be really careful if you do VDP access /both/ in the NMI and outside of the NMI, you need to be able to guarantee that they can't stomp on top of each other. The easiest way to guarantee it is to put all your VDP access code in the NMI - it can't be interrupted.

I haven't read the entire thread, but the NMI can interrupt itself, for example if I write too many bytes to VRAM and the VDP sends another VBLANK interrupt signal before I'm done.

 

The approach I have in Super Space Acer (and so by extension, libti99Coleco) gives me the ability to simulate turning interrupts on and off without actually doing so (meaning that it is guaranteed atomic) by tracking the state in the NMI itself. It took a number of passes to get it going but I think it's right now, but it allows me to have 'critical sections' during which the interrupt can be turned off, and yet not lost so that when I turn it back on, any missed interrupt is processed. (Unless more than one frame was missed, of course). That way I can do VDP work without fear of NMI interrupting it and yet also have the sprite list copy inside the NMI.

I'm considering doing another rewrite of Jewel Panic later this year (it will be the third one since I picked up the source code from Daniel Bienvenu) because the current version of the software uses a clunky "write-a-bunch-of-bytes-to-vram-and-keep-fingers-crossed-that-it's-done-before-next-vblank" method. This leads to graphic glitches, and I really think I can do better. But I'll have to restructure the code to do all VRAM writes within the NMI function, and partially disable NMI processing until I'm done writing to VRAM. I say "partially" because I don't want my "massive" VRAM writes to interfere with PSG sound output.

[youki]

But even if you "disable" the NMI , VRAM corruption can occurs, because the NMI is in fact never Disabled. In Daniel 's Kit , when you disable the NMI the only thing it does , it is that it won't call your Nmi routine when the NMI occurs. But the NMI still occurs. And if it occurs while you are accessing the VDP in general you have a VRAM corruption.

 

the NMI is NOT MASKABLE INTERRUPT . I still don't understand why Coleco wire the Vsync on a NMI , you don't have that issue on SG1000 and MSX. Here you can really disable the VSynch interruption. Not on coleco.

[Pixelboy]

But even if you "disable" the NMI , VRAM corruption can occurs, because the NMI is in fact never Disabled. In Daniel 's Kit , when you disable the NMI the only thing it does , it is that it won't call your Nmi routine when the NMI occurs. But the NMI still occurs. And if it occurs while you are accessing the VDP in general you have a VRAM corruption.

 

the NMI is NOT MASKABLE INTERRUPT . I still don't understand why Coleco wire the Vsync on a NMI , you don't have that issue on SG1000 and MSX. Here you can really disable the VSynch interruption. Not on coleco.

Right. By "partially disabling NMI processing", I was talking about setting up and checking a global entry flag in the NMI function, which I will use to skip the NMI altogether if the flag is set (and I will unset the flag once I'm done with my VRAM writes). But I'll update the sound output before checking the flag, so even if I skip an NMI, the sound output won't be affected.

 

EDIT: Essentially, unless I'm mistaken, this will mean I'll stop using "disable_nmi" and "enable_nmi" in my code. I'll just manage everything related to the NMI myself. This will likely lead to a significant source code rewrite, because right now I'm calling those two functions very often throughout the code.

[Kiwi]

But even if you "disable" the NMI , VRAM corruption can occurs, because the NMI is in fact never Disabled. In Daniel 's Kit , when you disable the NMI the only thing it does , it is that it won't call your Nmi routine when the NMI occurs. But the NMI still occurs. And if it occurs while you are accessing the VDP in general you have a VRAM corruption.

It'll run the nmi{} for sure. Spunky's controls are in that bracket so it'll work for all 6 game loops. If the controls are in the gameloop and not in NMI, then the controls will lag when the screen get too busy with lots of tilebased graphics. I don't how exactly disable_nmi works but it appears to skip the os7 routines, uploading sound data to the soundchip, retrieving controller data. Some routines like pause() will run enable the nmi and will go back to disabled if the nmi is disable before pause(). The plus side, it allows large vram writes. I do large tile/color/sprite write before level start in RockCutter. When it times to change to the next screen, I disabled the nmi to run the metamapping routines, which turns screen off get vram level data from vram to ram, if new sprites needed(like bosses) is written from ROM to VRAM, run write_screen, when screen's finished, write additional put frame object like boss gate, switches on to that screen, turn on screen, enable_nmi. One draw back, it sustain the sound effect until the screen is finish. Of course disable/enable sound will help. Or I could insert delay(1) into my screen writing routines to leave nmi_enabled.

 

If you want to have nmi enabled at all time, then delay(1) function will help. Limit your data upload to vpd to about 128-200 bytes a frame. Do this after delay(1), updatesprite(0,64); You can use the 2nd screen to build your screen while the game is active. You could swap the tilegraphics to sprites, but it may be a problem if you clear 6 tiles horizonally, unless you use a sprite priority swapping routines to rotate the sprites.

[Tursi]

I haven't read the entire thread, but the NMI can interrupt itself, for example if I write too many bytes to VRAM and the VDP sends another VBLANK interrupt signal before I'm done.

That can only happen if you clear the interrupt by reading the VDP status register before you are finished. When you read the status register, that's the end of the interrupt. You might still be in the interrupt handler, but you have told the hardware that you are done.

 

But even if you "disable" the NMI , VRAM corruption can occurs, because the NMI is in fact never Disabled. In Daniel 's Kit , when you disable the NMI the only thing it does , it is that it won't call your Nmi routine when the NMI occurs. But the NMI still occurs. And if it occurs while you are accessing the VDP in general you have a VRAM corruption.

You are confusing what /causes/ the corruption. The VDP corruption is not caused by the NMI, it's caused by accessing the VDP while another VDP access is underway. When my code has the interrupt "disabled", it DOES NOT TOUCH the VDP when the NMI occurs. It just sets a flag in CPU memory and returns. The interrupt is not cleared, the status byte is not read, and no VDP processing occurs. Yes, the interrupt occurred, but it didn't do anything that can cause corruption. When I later re-allow the interrupt processing, my code checks the flag and if it was set (by a missed NMI), then it jumps off and processes the VDP side of things immediately. The trick with that code was making it handle the edge conditions, since such a design is inherently racy.

 

Most NMI handlers in Coleco libs I've seen read the VDP status right away. Reading the VDP status byte changes the VDP's internal address pointer, and if that happens while the main code is doing a read or write, that is what causes the corruption.

Edited February 20, 2016 by Tursi

[Tursi]

EDIT: Essentially, unless I'm mistaken, this will mean I'll stop using "disable_nmi" and "enable_nmi" in my code. I'll just manage everything related to the NMI myself. This will likely lead to a significant source code rewrite, because right now I'm calling those two functions very often throughout the code.

Give my code a chance. You can pull my interrupt handling code out of libti99coleco and use it - it's not dependent on the rest of the lib.

It gives you the disable and enable wrappers with the above described functionality. You'll need three parts, here in the spoiler.

 

 

 

 

The crt0.s NMI handler (this replaces the NMI code in your existing crt0.s, and manages the flags. It calls "my_nmi" when enabled and simply sets the flag and exits when it's not).

 

	.area _BSS
_vdpLimi:		; 0x80 - interrupt set, other bits used by library
	.ds 1

    .area _CODE
nmi:
; all we do is set the MSB of _vdpLimi, and then check
; if the LSB is set. If so, we call user code now. if
; not, the library will deal with it when enabled.
	push af					; save flags (none affected, but play safe!)
	push hl
	
	ld hl,#_vdpLimi
	bit 0,(hl)				; check LSb (enable)
	jp z,notokay
	
; okay, full on call, save off the (other) regs
	push bc
	push de
	;push ix ; saved by callee
	push iy
	call _my_nmi			; call the lib version
	pop iy
	;pop ix
	pop de
	pop bc	
	jp clrup				

notokay:
	set 7,(hl)				; set MSb (flag)

clrup:
	pop hl					
	pop af
	retn

The header code (macros):

 

extern volatile unsigned char vdpLimi;
void my_nmi();

// we enable interrupts via a mask byte, as Coleco ints are NMI
// Note that the enable therefore needs to check a flag!
#define VDP_INT_ENABLE			{ __asm__("\tpush hl\n\tld hl,#_vdpLimi\n\tset 0,(hl)\n\tpop hl"); if (vdpLimi&0x80) my_nmi(); }
#define VDP_INT_DISABLE			{ __asm__("\tpush hl\n\tld hl,#_vdpLimi\n\tres 0,(hl)\n\tpop hl"); }
#define VDP_INT_POLL	\
	VDP_INT_ENABLE;		\
	VDP_INT_DISABLE;

And the VDP interrupt handling code:

 

// storage for VDP status byte
volatile unsigned char VDP_STATUS_MIRROR = 0;

// lock variable to prevent NMI from doing anything
// Z80 int is edge triggered, so it won't break us
// 0x80 = interrupt pending, 0x01 - interrupts enabled
// (This must be defined in the crt0.s)
//volatile unsigned char vdpLimi = 0;		// NO ints by default!

// address of user interrupt function
static void (*userint)() = 0;

// interrupt counter
volatile unsigned char VDP_INT_COUNTER = 0;

// May be called from true NMI or from VDP_INTERRUPT_ENABLE, depending on
// the flag setting when the true NMI fires. Do not call directly.
void my_nmi() {
	// I think we're okay from races. There are only two conditions this is called:
	//
	// VDP_INTERRUPT_ENABLE - detects that vdpLimi&0x80 was set by the interrupt code.
	//						  Calls this code. But the interrupt line is still active,
	//						  so we can't retrigger until it's cleared by the VDPST read
	//						  below. At that time the vdpLimi is zeroed, and so we can't loop.
	//
	// nmi -				  detects that vdpLimi&0x01 is valid, and calls directly.
	//						  Again, the interrupt line is still active.
	//
	// I think the edge cases are covered. Except if the user is manually reading VDPST,
	// then the state of vdpLimi could be out of sync with the real interrupt line, and cause
	// a double call. User apps should only read the mirror variable. Should I enforce that?

	vdpLimi = 0;			// clear the interrupt flags - do this before clearing the VDP
	VDP_STATUS_MIRROR = VDPST;	// release the VDP - we could instantly trigger again, but the vdpLimi is zeroed, so no loop
	VDP_INT_COUNTER++;		// count up the frames

	// the TI is running with ints off, so it won't retrigger in the
	// user code, even if it's slow. Our current process won't either because
	// the vdpLimi is set to 0.
	if (0 != userint) userint();

	// the TI interrupt would normally exit with the ints disabled
	// if it fired, so we will do the same here and not reset it.
}

// NOT atomic! Do NOT call with interrupts enabled!
void setUserIntHook(void (*hookfn)()) {
	userint = hookfn;
}

// NOT atomic! Do NOT call with interrupts enabled!
void clearUserIntHook() {
	userint = 0;
}

Finally, somewhere very early in your initialization code (I put it in the init function called from the crt0), you need to clear vdpLimi and set VDP_STATUS_MIRROR. For the sake of F18A systems which need a little longer to startup, I also have a delay in there before the first VDP init - the time wasn't calculated to be perfect but it's short enough not to notice and long enough to work.

 

// called automatically by crt0.S (not in TI version)
void vdpinit() {
	volatile unsigned int x;
	
	// shut off the sound generator - if the cart skips the BIOS screen, this is needed.
	SOUND = 0x9f;
	SOUND = 0xbf;
	SOUND = 0xdf;
	SOUND = 0xff;

	// interrupts off
	vdpLimi = 0;

	// before touching VDP, a brief delay. This gives time for the F18A to finish
	// initializing before we touch the VDP itself. This is needed on the Coleco if
	// you don't use the BIOS startup delay. This is roughly 200ms.
	x=60000;
	while (++x != 0) { }		// counts till we loop at 65536

	VDP_STATUS_MIRROR = VDPST;	// init and clear any pending interrupt
}

Usage is simple enough -- once you have the init code all working, the game will start with interrupts disabled. Any time you are safe to process interrupts, just use VDP_INT_ENABLE and VDP_INT_DISABLE. If you have your own interrupt code that needs to run, set the function with setUserIntHook or remove it with clearUserIntHook. You can do that as often as you like (just be sure interrupts are disabled when you call the function, otherwise you may crash if interrupted halfway through the change). VDP_INT_POLL is provided because on the TI it's common to only enable interrupts briefly at the top of your loop, and that's all it takes. If you need to read the VDP status register, and you're not doing raster effects, just read VDP_STATUS_MIRROR -- reading VDPST directly will race the NMI code. VDP_INT_COUNTER will increment every processed interrupt and can be used for timing. And finally, if you have interrupts disabled but want to know if an interrupt is pending, just check (vdpLimi&0x80) - if it's set, there is a pending interrupt. It will be called when you call VDP_INT_ENABLE.

 

"LIMI" is the interrupt enable/disable instruction on the TI, and that's where that comes from in the variable name.

 

 

[Pixelboy]

Thanks Tursi. Just before resuming work on Jewel Panic later this year, I'll have a closer look at your code above. Now is not really a good time for this, I'm knee-deep into Team Pixelboy activities.

[digress]

if this can fix the occasional corruption i'm experiencing i'm going to try it. It's getting frustrating as i'm spending more time problem solving corruption than actually adding to the program. Thanks for the help.

[Tursi]

Good luck!

 

As I learned in my Super Space Acer thread -- sometimes corruption really is your own fault, though, so don't discount that! Divide and conquer can help -- comment out parts of the code and see if you can isolate where it comes from... with a narrower scope it can be easier to determine if it's timing or code.

[digress]

"main.c:455: error 20: Undefined identifier 'VDPST'"

this doesn't seem to be defined.

 

I never doubt that something could be my fault. It's so occasional. I can play a complete round of my game for 20-25 minutes completing every stage and sometimes nothing happens. and then sometimes after that long the screen in the middle of play has the tiles suddenly and completely corrupted. It's not in the middle of the screen rewrite and pletter or anything. Just regular tile updates to the scoreboard or something minor.

 

So if it happen right away every time I start it would be easy to find but I literally am playing & playing & playing waiting for it to happen and there seems to be no sense to why it happens. It's almost like memory is slowly getting eaten up til it has no more open memory so it starts eating the video memory.

 

Yeah so that has been my method to comment out suspect parts of the code and everytime I think I got the suspect part it happens again seemingly disproving my theory.

 

Which of course means it could be more than 1 part.

Good luck!

As I learned in my Super Space Acer thread -- sometimes corruption really is your own fault, though, so don't discount that! Divide and conquer can help -- comment out parts of the code and see if you can isolate where it comes from... with a narrower scope it can be easier to determine if it's timing or code.

[alekmaul]

Hi Tursi,

It iseems that the version on your github (https://github.com/tursilion/libti99coleco) is different from the one in the zip file on your website.

What is the correct version to use ? I tried the one on github but it does not work for me (and , of course), the version in zip file works fine (from here:http://www.harmlesslion.com/cgi-bin/onesoft.cgi?88).

[Tursi]

Hmm... the version that's on Github should be the version that I use in SSA (I assume you mean the player, not the compression tool itself.) I'll have to check my archives to see which is the current version, sorry that they fell out of sync! I will have to wait till Monday to check.

 

In a quick download and diff, though, I believe the github version got a number of optimizations, and I didn't go back and update the archive in the compressor. There is no reason not to use the one in the archive, the format did not change. I might have forgotten to push a change (I use the libti99Coleco version in Super Space Acer). Once I check what went wrong I'll let you know, but for now you can use the one in the zip.

 

Also.... if you are using it without the rest of my lib, that might be a factor. I've never tested it standalone since the earliest days.

[alekmaul]

Also.... if you are using it without the rest of my lib, that might be a factor. I've never tested it standalone since the earliest days.

I'm only using the player, not all the lib.

And yes, it is not the same, I saw the optimizations you did but I have no sound with this version, will check again if it is my code that is not good with this one.

[Tursi]

There were some initialization changes in my local repo that I had not pushed, so I've updated that now. Go ahead and see if it works for you! Thanks for the update.

[alekmaul]

Well, sorry, but it is not working as previous player is working with my VGM files :/ The sound seems to freeze after a certain time , I will investigate why I have such behavior ...

[Tursi]

I'll run up some more tests, but that's the code that's in Super Space Acer... when the sound stops, are you saying the whole system locks up or you just stop hearing audio? Does the player think the music has ended (check the playing flags)?

[alekmaul]

Well, yes, the sound locks up and I only hear a noise after a few seconds, that's really strange. And the player I used from the zip file works perfectly with my VGM files :/, that is really odd ...

[digress]

Are you doing a lot of tile scrolling? I find the player works fine. I have current problem when I try to play music in the planes section which has 32x18 area of screen scrolling that it won't play at all unless I crash the plane or fly straight up so the scrolling stops.

 

I'm investigating why this is happening today. I expect it's too much data transfer per cycle.

 

 

>>> FOLLOWUP

 

I tried a bunch of different things including extra delay. skipping the scrolling altogether and ... I got it working.

 

For me the problem was I was getting and putting too much graphic data per step. I was moving 155 bytes sometimes

 

I reduced this to smaller 1 line at a time get/puts of 31 bytes . 1 byte was being redrawn anyways. and that seems to have fixed it. Music is no longer cutting out.

 

 

Well, yes, the sound locks up and I only hear a noise after a few seconds, that's really strange. And the player I used from the zip file works perfectly with my VGM files :/, that is really odd ...

[Tursi]

Be careful about drawing conclusions before investigating... you can spend a lot of time running down the wrong theories. The music player has no connection to drawing whatsoever, all it cares is when you call "stplay". So work backwards from there -- you can safely call it anywhere, but for proper playback it needs to be called once per frame. So you'd want to start at your stplay call and work backwards till you see why it's not happening in that segment - perhaps your scrolling code just returns early or such.

 

Scrolling can cost more than a full frame, so it may still slow down in such a case. In that case it gets a bit trickier to keep the proper rate while still protecting VDP.

 

Alek, I haven't had a chance to test properly yet, but I will by this weekend... keep using the older version for now.

[digress]

Well it could have also been the freed up memory. byte scr[155] versus byte scr[31]

 

Be careful about drawing conclusions before investigating... you can spend a lot of time running down the wrong theories. The music player has no connection to drawing whatsoever, all it cares is when you call "stplay". So work backwards from there -- you can safely call it anywhere, but for proper playback it needs to be called once per frame. So you'd want to start at your stplay call and work backwards till you see why it's not happening in that segment - perhaps your scrolling code just returns early or such.

 

Scrolling can cost more than a full frame, so it may still slow down in such a case. In that case it gets a bit trickier to keep the proper rate while still protecting VDP.

 

Alek, I haven't had a chance to test properly yet, but I will by this weekend... keep using the older version for now.

[alekmaul]

Are you doing a lot of tile scrolling? I find the player works fine. I have current problem when I try to play music in the planes section which has 32x18 area of screen scrolling that it won't play at all unless I crash the plane or fly straight up so the scrolling stops.

No, it is a screen with no scrolling, and the player is the only thing running when the sound locks

[Tursi]

Alekmaul, sorry you are having problems with it. I've spent the last couple of hours running tests and double checking what is up on github, and it seems to be working for me here. The one thing I can suggest is to check your map file and make sure you aren't overlapping RAM locations used by the library (or maybe running out of stack). You could try moving the storage for the player data earlier in the link process to see if the problem moves to support such a theory.

 

Besides the restructuring of the code, the newer version of the player uses a little more RAM to gain back the speed, so if you are near the edge, that might explain why one library works and not the other. Another thing to try might be to make a simple test app - just the player and your sound bank, no game code. If that still screws up, please, send it my way and I'll help you troubleshoot why. If it works, then the problem may well be memory corruption.

 

Although I use the player with my own libti99, it has no dependencies on the rest of the library. The major differences with cvLib are that I don't use the BIOS functions and so don't have fixed RAM addresses reserved for those BIOS calls (there's also the different NMI handling that we discussed elsewhere, but that doesn't directly affect the player). I can't think of any reason it wouldn't work standalone.

Edited September 4, 2016 by Tursi

[alekmaul]

ok, thanks for your tests. Will check more deeper in my code to find the issue :/
regarding ram, it seems ok :

 

Area Addr Size Decimal Bytes (Attributes)
-------------------------------- ---- ---- ------- ----- ------------
_DATA 00007000 0000037C = 892. bytes (REL,CON)