My TI 99/4A doing medical research!

[+Vorticon]

Nothing beats XB for a rapid check on an algorithm, at least for me, so when I started a research project on pediatric croup, my TI was super useful.

 

Quick background: 75% of admissions for pediatric croup are unnecessary in the US, a huge waste of healthcare dollars and an unnecessary burden on the child and family. I started a research project about a year ago trying to come up with a risk calculator that could predict with reasonable accuracy whether any particular croup patients required admission. Data collection was completed a couple of months ago but extensive statistical analysis has failed to produce strong associations between patient characteristics and need for admission, possibly hindered by the fact that only 117 patients in my hospital pool met research inclusion criteria. So I decided to test out a neural network to see if I could tease out hidden associations beyond the realm of statistics.

 

I started out with programs on the TI in XB initially with a simple 1 neuron perceptron which failed spectacularly (80+ % error rate), then moving on to a much more involved neural net with 2 medial synaptic layers and backpropagation which was much more promising (30% error rate). Since neural networks require a large number of trials to converge on a solution, I just had the program process the same dataset repeatedly to simulate that. Yes I know that this is still a high error rate, but it's still better than the 75% currently in the medical community! Essentially my TI experimentations provided the needed reassurance that my algorithms were working, although implementation was severely limited by available memory (I could only do 25 neurons per synaptic layer) and numeric precision (the Fermi function used to normalize the data and it's reverse function used in backpropagation could produce very small and very large numbers and I kept bumping into overflow errors). Therefore I fully expect much better error rates on a modern computer, and I'm currently working on creating a javascript program for that purpose.

 

Who said the TI was obsolete?

 

For the curious, here's the neural net program in its current state:

 

 

 

// Croup Neural Network
// By Walid Maalouli
// May 2018

// Initialization
RANDOMIZE
OPTION BASE 1
CALL CLEAR
N=25 ! Number of medial neurons
F=12 ! Number of input factors + 1

DIM SYN1(12,25),SYN2(25,25),SYN3(25),MEDIN(25),MEDIN2(25),MEDOUT(25),MEDOUT2(25)
DIM INP(12),SIGMA(25),SIGMOID(25)

RATE=0.01
DELTA=0
COUNT=0
MINERR=100
INP(F)=1
OPEN #1:"DSK1.ERRLST",OUTPUT,INTERNAL,FIXED 15 :: CLOSE #1

// Randomize synapses initial conditions
PRINT "INITIALIZING INPUT SYNAPSES"
FOR I=1 TO N
	FOR J=1 TO F 
		SYN1(J,I)=0.1*RND
	NEXT J
NEXT I

PRINT "INITIALIZING MIDDLE LAYER"
FOR I=1 TO N
	FOR J=1 TO N
		SYN2(J,I)=0.1*RND
	NEXT J
NEXT I

PRINT "INITIALIZING OUTPUT SYNAPSES"
PRINT
FOR I=1 TO N
	SYN3(I)=0.1*RND
NEXT I

GOSUB SaveSynapses

// Forward propagation
// Read input
ReadData:
FOR I=1 TO F-1
	READ INP(I)
NEXT I
READ TARG1
READ TARG2
IF TARG1=1 OR TARG2>0 THEN 
	TARGET=1 ELSE TARGET=0

IF INP(1)=0 THEN DataDone
COUNT=COUNT+1
PRINT "PROCESSING RECORD# ";COUNT

// Modify input synapses
FOR I=1 TO N
	TEMP=MEDIN(I)
	MEDIN(I)=0
	FOR J=1 TO F
		MEDIN(I)=MEDIN(I)+SYN1(J,I)*INP(J) ! Summate all inputs for each medial neuron
	NEXT J
	IF MEDIN(I)<10E-12 THEN MEDIN(I)=TEMP
	MEDOUT(I)=1/(1-EXP(-MEDIN(I))) ! Normalize medial neurons output
NEXT I

// Process middle layer
FOR I=1 TO N
	TEMP1=MEDIN2(I)
	MEDIN2(I)=0
	FOR J=1 TO N
		MEDIN2(I)=MEDIN2(I)+SYN2(J,I)*MEDOUT(I)
	NEXT J
	IF MEDIN2(I)<10E-12 THEN MEDIN2(I)=TEMP1
	MEDOUT2(I)=1/(1-EXP(-MEDIN2(I)))
NEXT I

// Process output
OUT=0
FOR I=1 TO N
	OUT=OUT+SYN3(I)*MEDOUT2(I) ! Summate all inputs to each output neuron
NEXT I

TERROR=TARGET-OUT ! Calculate output error
PRINT "OUT:";OUT;" TARGET:";TARGET;" ERROR:";TERROR :: PRINT
TOTERR=TOTERR+ABS(TERROR)
IF INT(COUNT/10)=COUNT/10 THEN PRINT "AVERAGE ERROR:";TOTERR/10 ELSE SkipSave
OPEN #1:"DSK1.ERRLST",INTERNAL,APPEND,FIXED 15
PRINT #1:TOTERR/10 :: CLOSE #1
IF TOTERR/10<MINERR THEN MINERR=TOTERR/10::GOSUB SaveSynapses
TOTERR=0

SkipSave:
PRINT "MIN ERROR: ";MINERR
PRINT
	
// Back propagation
// Apply error to output synapses
FOR I=1 TO N
	SYN3(I)=SYN3(I)+RATE*MEDOUT2(I)*TERROR
NEXT I

// Invert the sigmoid signal for middle layer
FOR I=1 TO N
	SIGMA(I)=0
	SIGMA(I)=SIGMA(I)+TERROR*SYN3(I)
	IF MEDIN2(I)>10E48 THEN MEDIN2(I)=TEMP1
	SIGMOID(I)=MEDIN2(I)*(1-MEDIN2(I))
NEXT I

// Adjust the middle synaptic layer
FOR I=1 TO N
	FOR J=1 TO N
		DELTA=RATE*SIGMOID(J)*SIGMA(J)*MEDOUT(I)
		SYN2(I,J)=SYN2(I,J)+DELTA
	NEXT J
NEXT I

// Adjust the input synaptic layer
FOR I=1 TO N
	SIGMA(I)=0
	FOR J=1 TO N
		SIGMA(I)=SIGMA(I)+TERROR*SYN2(I,J)
	NEXT J
	IF MEDIN(I)>10E48 THEN MEDIN(I)=TEMP
	SIGMOID(I)=MEDIN(I)*(1-MEDIN(I))
NEXT I

FOR I=1 TO F-1
	FOR J=1 TO N
		DELTA=RATE*SIGMOID(J)*SIGMA(J)*INP(I)
		SYN1(I,J)=SYN1(I,J)+DELTA
	NEXT J
NEXT I

GOTO ReadData

// Training done. Verify net predictions
DataDone:
RESTORE RawData
GOTO ReadData

//Saving state of synapses routine
SaveSynapses:
OPEN #1:"DSK1.SYNSAV",OUTPUT,INTERNAL,FIXED 15
FOR I=1 TO F
	FOR J=1 TO N
		PRINT #1:SYN1(I,J)
	NEXT J
NEXT I
FOR I=1 TO N
	FOR J=1 TO N
		PRINT #1:SYN2(I,J)
	NEXT J
NEXT I
FOR I=1 TO N
	PRINT #1:SYN3(I)
NEXT I
CLOSE #1
RETURN

// Patient data
RawData:
DATA 14,2,1,3.8,4,23,2,5,0,0,1,0,1
DATA 14,1,1,3.1,1,53,2,1,0,1,0,0,0
DATA 5,1,1,3.1,5,24,1,2,0,0,0,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 6,1,1,2.6,4,40,2,3,5,1,0,0,0
DATA 79,1,2,5.9,3,6,1,0,0,1,0,0,0
DATA 23,1,1,4.2,3,37,2,2,0,0,0,0,0
DATA 18,1,2,2.2,6,18,2,0,0,0,1,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 40,2,1,3.4,1,99,2,0,0,0,0,0,0
DATA 35,1,1,2.4,4,56,3,3,0,0,1,0,0
DATA 22,1,1,2.6,4,87,3,0,0,1,1,0,3
DATA 7,1,1,2.9,4,24,2,2,1,1,0,0,0
DATA 19,2,1,3.3,4,65,1,4,0,0,0,0,3
DATA 4,1,1,2.9,2,25,2,0,0,0,1,0,0
DATA 15,1,1,1.5,2,127,1,2,0,1,0,0,1
DATA 61,1,1,5.4,2,116,2,0,0,0,0,0,0
DATA 15,1,1,4.0,2,112,1,0,0,0,0,0,0
DATA 8,2,1,3.9,2,74,1,1,0,0,1,0,0
DATA 4,1,1,3.2,4,36,1,3,1,0,0,0,0
DATA 49,1,1,3.1,2,43,2,0,6,0,0,0,0
DATA 16,1,1,3.8,3,25,2,1,0,1,0,0,1
DATA 6,2,3,2.2,3,43,1,1,5,0,1,0,0
DATA 50,1,1,3.4,5,35,2,0,6,0,1,0,0
DATA 8,1,1,3.0,5,87,1,1,1,0,1,0,0
DATA 11,1,1,2.5,4,45,2,1,0,1,1,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 11,1,1,5.8,3,32,2,2,0,0,1,0,0
DATA 14,1,1,2.6,2,33,2,0,1,0,0,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 9,2,1,4.2,2,80,1,0,0,0,0,0,0
DATA 16,1,1,4.1,4,44,2,1,0,0,1,0,0
DATA 24,1,1,2.8,5,82,1,0,0,0,1,0,0
DATA 37,1,1,4.5,3,62,2,2,0,0,0,0,0
DATA 25,2,1,2.2,6,28,3,4,0,1,1,0,3
DATA 23,1,1,2.5,2,28,3,0,0,0,1,0,0
DATA 62,1,1,4.4,5,23,1,0,1,0,1,0,2
DATA 14,2,1,2.8,4,33,2,0,0,0,0,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 101,1,1,4.5,2,58,2,0,0,0,0,0,0
DATA 11,1,1,5.1,3,20,1,1,0,0,1,0,0
DATA 103,2,5,5.1,3,42,2,0,0,0,1,0,0
DATA 51,1,6,2.0,4,40,2,2,0,0,1,0,0
DATA 24,2,1,5.6,1,21,2,0,0,0,0,0,0
DATA 84,1,1,4.3,1,22,2,0,0,0,0,0,0
DATA 66,1,1,2.7,2,26,2,0,0,0,0,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 10,1,1,2.3,3,28,2,0,0,0,0,0,0
DATA 12,2,1,4.7,3,40,1,1,3,0,1,0,1
DATA 42,1,1,2.5,2,42,2,3,0,1,0,0,0
DATA 17,2,1,5.4,3,40,3,2,0,0,1,0,0
DATA 14,1,1,3.2,4,36,1,0,0,0,1,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 19,2,1,2.1,5,46,2,1,0,1,0,0,0
DATA 19,1,1,3.0,0,64,1,0,0,0,0,0,0
DATA 10,1,1,2.7,2,38,2,2,0,0,0,0,0
DATA 124,2,1,3.3,2,54,2,0,0,1,1,0,0
DATA 138,1,1,2.6,3,22,1,0,0,0,0,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 93,1,1,4.7,1,32,2,0,0,0,1,0,0
DATA 25,2,1,3.8,2,19,2,2,0,0,1,0,0
DATA 17,1,1,3.0,3,27,2,2,0,0,0,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 23,1,1,5.3,3,81,2,1,0,0,1,0,0
DATA 20,1,1,2.7,4,23,2,0,0,0,1,0,1
DATA 27,1,1,2.9,3,39,2,0,0,1,0,0,0
DATA 40,1,1,3.5,2,64,2,0,0,0,1,0,0
DATA 14,1,0,3.0,4,17,3,5,0,0,0,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 29,2,2,2.8,2,30,2,2,0,0,1,0,0
DATA 29,2,1,3.4,3,41,2,2,0,0,1,0,0
DATA 21,1,1,3.5,2,44,2,4,0,0,0,0,0
DATA 21,1,1,2.7,4,17,3,3,1,0,0,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 17,2,2,6.6,2,63,2,2,6,0,0,0,0
DATA 82,1,1,2.0,5,19,2,2,1,1,0,0,2
DATA 73,1,1,2.5,2,69,2,3,0,0,0,0,0
DATA 8,1,1,4.6,2,41,2,0,0,0,0,0,0
DATA 61,1,1,4.2,2,19,2,0,0,0,0,0,0
DATA 24,1,2,4.1,2,41,2,0,0,0,1,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 13,1,1,3.5,2,181,2,0,0,1,0,0,0
DATA 18,1,1,3.8,3,28,2,3,0,0,1,0,0
DATA 2,1,1,2.7,2,71,2,0,0,0,0,0,0
DATA 5,1,1,1.7,2,16,1,0,0,0,1,0,1
DATA 42,1,1,2.2,2,38,2,3,0,1,1,0,0
DATA 17,1,1,4.9,3,41,2,1,7,0,0,1,0
DATA 20,2,1,4.4,2,43,2,0,6,0,0,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 52,1,1,5.4,3,33,3,0,0,1,0,0,0
DATA 19,1,1,3.1,3,51,1,2,0,0,0,0,0
DATA 21,2,7,3.1,3,37,2,0,5,0,0,0,3
DATA 15,1,1,5.5,3,80,3,1,0,0,1,0,0
DATA 16,2,1,2.8,3,22,2,3,0,0,0,0,0
DATA 18,2,1,5.4,2,29,2,0,0,0,1,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 4,1,8,3.1,4,129,2,3,0,0,1,0,0
DATA 12,1,1,2.4,3,24,2,0,0,0,0,0,0
DATA 22,1,1,2.3,3,43,1,1,0,1,0,0,0
DATA 11,1,1,4.1,2,26,2,1,0,1,0,0,0
DATA 31,2,2,3.1,3,33,2,2,0,0,1,0,0
DATA 69,2,0,2.8,1,45,0,1,0,0,1,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 17,1,1,3.2,3,40,1,0,0,0,0,0,0
DATA 82,1,1,2.6,2,16,3,1,0,0,0,0,0
DATA 26,1,1,3.0,2,24,2,0,0,1,0,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 4,1,1,3.6,2,25,2,0,0,0,1,0,0
DATA 18,1,1,3.5,4,18,2,2,0,1,1,0,0
DATA 33,2,1,3.4,2,61,1,0,0,0,0,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 11,2,1,3.1,3,34,2,0,0,0,0,0,0
DATA 23,2,3,2.5,3,9,2,3,0,0,0,0,1
DATA 16,1,1,2.4,2,96,2,2,0,0,0,0,0
DATA 59,1,1,2.2,3,13,2,2,6,0,1,0,1
DATA 44,1,1,4.2,3,43,2,1,0,0,0,0,0
DATA 15,1,1,3.1,2,21,2,1,0,0,0,0,0
DATA 15,2,1,2.9,2,58,2,4,0,1,1,0,2
DATA 62,2,1,3.5,3,20,2,0,6,0,0,0,0
DATA 110,1,1,4.6,2,15,1,0,0,0,0,0,0
DATA 17,2,1,3.7,3,9,2,2,0,1,0,0,0
DATA 7,1,1,3.9,1,60,2,3,0,0,0,0,1
DATA 5,1,1,4.3,3,128,1,2,0,0,0,0,0
DATA 81,1,1,3.8,2,67,2,4,0,0,0,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 14,2,1,3.4,3,49,1,2,0,0,1,0,0
DATA 43,1,6,5.1,1,74,1,1,6,0,0,0,0
DATA 16,1,1,2.5,3,34,2,2,0,0,1,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 9,1,1,3.8,3,105,2,3,0,0,1,0,0
DATA 26,2,1,6.7,3,39,2,4,0,1,1,0,0
DATA 19,2,1,5.6,3,83,3,4,0,0,0,0,1
DATA 16,1,1,3.1,1,22,2,3,0,0,1,0,0
DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1
DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3
DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2
DATA 0,0,0,0,0,0,0,0,0,0,0,0,0 

 

 

[+TheBF]

If you were getting overflow errors on the TI then you could still get them on a modern computer depending on the floating point data size.

The TI-99 can handle a mantissa to 127 so it's pretty big.

 

This is really cool work. Are you an MD?

[+TheBF]

Also, would be able to save some size by reading your DATA from an input file to make more room for the neuron layers?

[+Lee Stewart]

I have not carefully analyzed what you are doing mathematically and you may already be doing this, but, if you are dealing with summations of very large and very small numbers in the same summation, it is best , if possible, to order the addends from smallest to largest to minimize truncation errors. It may even be necessary to compartmentalize calculations for very small and very large terms until the end of the process. I had to do something like this in fitting ethanol-water composition/density data to a polynomial because the critical parts of the coefficients often required more digits of precision than were available in the C program package I was using (MS Visual C++ circa 2006 or so). This was due to the fitting equations depending on very small differences in very large numbers. What I did was to split the coefficients into high and low parts, thus doubling the number of terms, and summing the smallest terms first.

 

...lee

[+Vorticon]

Yes I'm an MD. Javascript arithmetic is double-precision floating point on a modern PC, i.e. numbers between 10E-308 and 10+E308 can be represented, much larger than the TI. I'm hoping this will minimize the overflow errors issue.

As for the data format, I could indeed have it on disk but it will slow things down significantly. Even as things stand it is painfully slow and requires Classic99 in overdrive mode to run marginally well. Besides, the savings in memory achieved will allow for only a handful of extra neurons per layer at best, which will have negligible impact on the results. I'm planning on having at least 150-200 neurons per medial synaptic layer with javascript, maybe more depending on my ultimate results.

[+Vorticon]

I have not carefully analyzed what you are doing mathematically and you may already be doing this, but, if you are dealing with summations of very large and very small numbers in the same summation, it is best , if possible, to order the addends from smallest to largest to minimize truncation errors. It may even be necessary to compartmentalize calculations for very small and very large terms until the end of the process. I had to do something like this in fitting ethanol-water composition/density data to a polynomial because the critical parts of the coefficients often required more digits of precision than were available in the C program package I was using (MS Visual C++ circa 2006 or so). This was due to the fitting equations depending on very small differences in very large numbers. What I did was to split the coefficients into high and low parts, thus doubling the number of terms, and summing the smallest terms first.

 

...lee

Ordering the addition terms will certainly be feasible but with a larger number of neurons, say 200 per medial synaptic layer, we are looking at 40000 terms for each of the medial layers, and I'm not sure what kind of performance hit this will generate. I might consider it if the error rate remains high. The same goes for compatmentalization. Great trick though!

Thinking about this a bit more though, my experimentation on the TI showed that very small intermediate numbers are generated during forward propagation, and conversely very large ones during back-propagation, so ordering might not be very helpful in this situation...