Yes, just like PuzZLeR I rewrote the code before going to bed. Eventually I will try to make a clean example of what happened, but it looked roughly like this, where x0 was unsigned, x1 and tx were signed:
tx = tx + SGN(x1-x0)
Changing x0 to signed made no difference. I assigned a temporary variable to the SGN result, and the cross-procedure jumping bug went away. Yes, it sounds entirely bizarre and that is why I want to make a clean example before really submitting a bug report.
There's something fishy going on in the SGN() function. I see in the resulting Assembly that it tries to do some fancy jumping around but at least one of the jumps seems to land inadvertently on the wrong place.
Here's some test IntyBASIC code:
tx = 5
x1 = 1
x0 = 1
tx = tx + SGN(x1-x0)
FOO = 10
And here's the Assembled code from the Listing file:
5255 0280 0110 MVI V3,R0
5257 0300 010F SUB V4,R0
5259 0204 0005 BEQ $+7
525B 02B8 0001 MVII #1,R0
525D 0203 0001 BPL $+3
525F 0300 010E SUB V2,R0
5261 0020 NEGR R0
5262 0240 010E MVO R0,V2
; FOO = 10
5264 02B8 000A MVII #10,R0
5266 0240 010D MVO R0,V5
The jump in line $5259, actually lands on $5260, which is the second encoded word of the SUB instruction, which is interpreted as
$5260 010E SUBR R1,R6
That's gotta freak out the stack.
It can be better seen in a dissassembly of the running code in the emulator:
$5255: 0280 0110 MVI V3 ($0110),R0 | tx = tx + SGN(x1-x0)
$5257: 0300 010F SUB V4 ($010F),R0 | tx = tx + SGN(x1-x0)
$5259: 0204 0005 BEQ $5260 | tx = tx + SGN(x1-x0)
$525B: 02B8 0001 MVII #$0001,R0 | tx = tx + SGN(x1-x0)
$525D: 0203 0001 BPL $5260 | tx = tx + SGN(x1-x0)
$525F: 0300 010E SUB V2 ($010E),R0 | tx = tx + SGN(x1-x0)
$5261: 0020 NEGR R0 | tx = tx + SGN(x1-x0)
$5262: 0240 010E MVO R0,V2 ($010E) | tx = tx + SGN(x1-x0)
$5264: 02B8 000A MVII #$000A,R0 | FOO = 10
$5266: 0240 010D MVO R0,V5 ($010D) | FOO = 10
You can see that in line $5259, the target address of the "BEQ $+7" instruction has been computed as $5260 instead of $5262 (skip the subtraction and negation, and store the zero).
As you discovered, assigning the SGN() function value directly to a variable avoids the mess:
FOO = SGN(x1-x0)
tx = tx + FOO
$5255: 0280 0110 MVI V3 ($0110),R0 | FOO = SGN(x1-x0)
$5257: 0300 010F SUB V4 ($010F),R0 | FOO = SGN(x1-x0)
$5259: 0204 0005 BEQ $5260 | FOO = SGN(x1-x0)
$525B: 02B8 0001 MVII #$0001,R0 | FOO = SGN(x1-x0)
$525D: 0203 0001 BPL $5260 | FOO = SGN(x1-x0)
$525F: 0020 NEGR R0 | FOO = SGN(x1-x0)
$5260: 0240 010D MVO R0,V5 ($010D) | FOO = SGN(x1-x0)
Now the jump at $5259 lands correctly at $5260, just as the computer gods intended. It's precisely the clever code that tries to continue with the expression by adding/subtracting the result to the next operand, which causes the problem. It doesn't account for the length of that instruction (2 words) in the jump.
This is a bug. I recommend to everyone to always assign the return value of SGN() directly to a variable until it is corrected in the compiler.
Oscar, a quick and easy solution to these things is to inject labels at judicious places during code generation, and branch to those instead. That's what I do in P-Machinery, rather than assuming the length of the instructions generated, since that is always a bit dangerous.