+Vorticon Posted June 2, 2018 Share Posted June 2, 2018 Nothing beats XB for a rapid check on an algorithm, at least for me, so when I started a research project on pediatric croup, my TI was super useful. Quick background: 75% of admissions for pediatric croup are unnecessary in the US, a huge waste of healthcare dollars and an unnecessary burden on the child and family. I started a research project about a year ago trying to come up with a risk calculator that could predict with reasonable accuracy whether any particular croup patients required admission. Data collection was completed a couple of months ago but extensive statistical analysis has failed to produce strong associations between patient characteristics and need for admission, possibly hindered by the fact that only 117 patients in my hospital pool met research inclusion criteria. So I decided to test out a neural network to see if I could tease out hidden associations beyond the realm of statistics. I started out with programs on the TI in XB initially with a simple 1 neuron perceptron which failed spectacularly (80+ % error rate), then moving on to a much more involved neural net with 2 medial synaptic layers and backpropagation which was much more promising (30% error rate). Since neural networks require a large number of trials to converge on a solution, I just had the program process the same dataset repeatedly to simulate that. Yes I know that this is still a high error rate, but it's still better than the 75% currently in the medical community! Essentially my TI experimentations provided the needed reassurance that my algorithms were working, although implementation was severely limited by available memory (I could only do 25 neurons per synaptic layer) and numeric precision (the Fermi function used to normalize the data and it's reverse function used in backpropagation could produce very small and very large numbers and I kept bumping into overflow errors). Therefore I fully expect much better error rates on a modern computer, and I'm currently working on creating a javascript program for that purpose. Who said the TI was obsolete? For the curious, here's the neural net program in its current state: // Croup Neural Network // By Walid Maalouli // May 2018 // Initialization RANDOMIZE OPTION BASE 1 CALL CLEAR N=25 ! Number of medial neurons F=12 ! Number of input factors + 1 DIM SYN1(12,25),SYN2(25,25),SYN3(25),MEDIN(25),MEDIN2(25),MEDOUT(25),MEDOUT2(25) DIM INP(12),SIGMA(25),SIGMOID(25) RATE=0.01 DELTA=0 COUNT=0 MINERR=100 INP(F)=1 OPEN #1:"DSK1.ERRLST",OUTPUT,INTERNAL,FIXED 15 :: CLOSE #1 // Randomize synapses initial conditions PRINT "INITIALIZING INPUT SYNAPSES" FOR I=1 TO N FOR J=1 TO F SYN1(J,I)=0.1*RND NEXT J NEXT I PRINT "INITIALIZING MIDDLE LAYER" FOR I=1 TO N FOR J=1 TO N SYN2(J,I)=0.1*RND NEXT J NEXT I PRINT "INITIALIZING OUTPUT SYNAPSES" PRINT FOR I=1 TO N SYN3(I)=0.1*RND NEXT I GOSUB SaveSynapses // Forward propagation // Read input ReadData: FOR I=1 TO F-1 READ INP(I) NEXT I READ TARG1 READ TARG2 IF TARG1=1 OR TARG2>0 THEN TARGET=1 ELSE TARGET=0 IF INP(1)=0 THEN DataDone COUNT=COUNT+1 PRINT "PROCESSING RECORD# ";COUNT // Modify input synapses FOR I=1 TO N TEMP=MEDIN(I) MEDIN(I)=0 FOR J=1 TO F MEDIN(I)=MEDIN(I)+SYN1(J,I)*INP(J) ! Summate all inputs for each medial neuron NEXT J IF MEDIN(I)<10E-12 THEN MEDIN(I)=TEMP MEDOUT(I)=1/(1-EXP(-MEDIN(I))) ! Normalize medial neurons output NEXT I // Process middle layer FOR I=1 TO N TEMP1=MEDIN2(I) MEDIN2(I)=0 FOR J=1 TO N MEDIN2(I)=MEDIN2(I)+SYN2(J,I)*MEDOUT(I) NEXT J IF MEDIN2(I)<10E-12 THEN MEDIN2(I)=TEMP1 MEDOUT2(I)=1/(1-EXP(-MEDIN2(I))) NEXT I // Process output OUT=0 FOR I=1 TO N OUT=OUT+SYN3(I)*MEDOUT2(I) ! Summate all inputs to each output neuron NEXT I TERROR=TARGET-OUT ! Calculate output error PRINT "OUT:";OUT;" TARGET:";TARGET;" ERROR:";TERROR :: PRINT TOTERR=TOTERR+ABS(TERROR) IF INT(COUNT/10)=COUNT/10 THEN PRINT "AVERAGE ERROR:";TOTERR/10 ELSE SkipSave OPEN #1:"DSK1.ERRLST",INTERNAL,APPEND,FIXED 15 PRINT #1:TOTERR/10 :: CLOSE #1 IF TOTERR/10<MINERR THEN MINERR=TOTERR/10::GOSUB SaveSynapses TOTERR=0 SkipSave: PRINT "MIN ERROR: ";MINERR PRINT // Back propagation // Apply error to output synapses FOR I=1 TO N SYN3(I)=SYN3(I)+RATE*MEDOUT2(I)*TERROR NEXT I // Invert the sigmoid signal for middle layer FOR I=1 TO N SIGMA(I)=0 SIGMA(I)=SIGMA(I)+TERROR*SYN3(I) IF MEDIN2(I)>10E48 THEN MEDIN2(I)=TEMP1 SIGMOID(I)=MEDIN2(I)*(1-MEDIN2(I)) NEXT I // Adjust the middle synaptic layer FOR I=1 TO N FOR J=1 TO N DELTA=RATE*SIGMOID(J)*SIGMA(J)*MEDOUT(I) SYN2(I,J)=SYN2(I,J)+DELTA NEXT J NEXT I // Adjust the input synaptic layer FOR I=1 TO N SIGMA(I)=0 FOR J=1 TO N SIGMA(I)=SIGMA(I)+TERROR*SYN2(I,J) NEXT J IF MEDIN(I)>10E48 THEN MEDIN(I)=TEMP SIGMOID(I)=MEDIN(I)*(1-MEDIN(I)) NEXT I FOR I=1 TO F-1 FOR J=1 TO N DELTA=RATE*SIGMOID(J)*SIGMA(J)*INP(I) SYN1(I,J)=SYN1(I,J)+DELTA NEXT J NEXT I GOTO ReadData // Training done. Verify net predictions DataDone: RESTORE RawData GOTO ReadData //Saving state of synapses routine SaveSynapses: OPEN #1:"DSK1.SYNSAV",OUTPUT,INTERNAL,FIXED 15 FOR I=1 TO F FOR J=1 TO N PRINT #1:SYN1(I,J) NEXT J NEXT I FOR I=1 TO N FOR J=1 TO N PRINT #1:SYN2(I,J) NEXT J NEXT I FOR I=1 TO N PRINT #1:SYN3(I) NEXT I CLOSE #1 RETURN // Patient data RawData: DATA 14,2,1,3.8,4,23,2,5,0,0,1,0,1 DATA 14,1,1,3.1,1,53,2,1,0,1,0,0,0 DATA 5,1,1,3.1,5,24,1,2,0,0,0,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 6,1,1,2.6,4,40,2,3,5,1,0,0,0 DATA 79,1,2,5.9,3,6,1,0,0,1,0,0,0 DATA 23,1,1,4.2,3,37,2,2,0,0,0,0,0 DATA 18,1,2,2.2,6,18,2,0,0,0,1,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 40,2,1,3.4,1,99,2,0,0,0,0,0,0 DATA 35,1,1,2.4,4,56,3,3,0,0,1,0,0 DATA 22,1,1,2.6,4,87,3,0,0,1,1,0,3 DATA 7,1,1,2.9,4,24,2,2,1,1,0,0,0 DATA 19,2,1,3.3,4,65,1,4,0,0,0,0,3 DATA 4,1,1,2.9,2,25,2,0,0,0,1,0,0 DATA 15,1,1,1.5,2,127,1,2,0,1,0,0,1 DATA 61,1,1,5.4,2,116,2,0,0,0,0,0,0 DATA 15,1,1,4.0,2,112,1,0,0,0,0,0,0 DATA 8,2,1,3.9,2,74,1,1,0,0,1,0,0 DATA 4,1,1,3.2,4,36,1,3,1,0,0,0,0 DATA 49,1,1,3.1,2,43,2,0,6,0,0,0,0 DATA 16,1,1,3.8,3,25,2,1,0,1,0,0,1 DATA 6,2,3,2.2,3,43,1,1,5,0,1,0,0 DATA 50,1,1,3.4,5,35,2,0,6,0,1,0,0 DATA 8,1,1,3.0,5,87,1,1,1,0,1,0,0 DATA 11,1,1,2.5,4,45,2,1,0,1,1,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 11,1,1,5.8,3,32,2,2,0,0,1,0,0 DATA 14,1,1,2.6,2,33,2,0,1,0,0,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 9,2,1,4.2,2,80,1,0,0,0,0,0,0 DATA 16,1,1,4.1,4,44,2,1,0,0,1,0,0 DATA 24,1,1,2.8,5,82,1,0,0,0,1,0,0 DATA 37,1,1,4.5,3,62,2,2,0,0,0,0,0 DATA 25,2,1,2.2,6,28,3,4,0,1,1,0,3 DATA 23,1,1,2.5,2,28,3,0,0,0,1,0,0 DATA 62,1,1,4.4,5,23,1,0,1,0,1,0,2 DATA 14,2,1,2.8,4,33,2,0,0,0,0,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 101,1,1,4.5,2,58,2,0,0,0,0,0,0 DATA 11,1,1,5.1,3,20,1,1,0,0,1,0,0 DATA 103,2,5,5.1,3,42,2,0,0,0,1,0,0 DATA 51,1,6,2.0,4,40,2,2,0,0,1,0,0 DATA 24,2,1,5.6,1,21,2,0,0,0,0,0,0 DATA 84,1,1,4.3,1,22,2,0,0,0,0,0,0 DATA 66,1,1,2.7,2,26,2,0,0,0,0,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 10,1,1,2.3,3,28,2,0,0,0,0,0,0 DATA 12,2,1,4.7,3,40,1,1,3,0,1,0,1 DATA 42,1,1,2.5,2,42,2,3,0,1,0,0,0 DATA 17,2,1,5.4,3,40,3,2,0,0,1,0,0 DATA 14,1,1,3.2,4,36,1,0,0,0,1,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 19,2,1,2.1,5,46,2,1,0,1,0,0,0 DATA 19,1,1,3.0,0,64,1,0,0,0,0,0,0 DATA 10,1,1,2.7,2,38,2,2,0,0,0,0,0 DATA 124,2,1,3.3,2,54,2,0,0,1,1,0,0 DATA 138,1,1,2.6,3,22,1,0,0,0,0,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 93,1,1,4.7,1,32,2,0,0,0,1,0,0 DATA 25,2,1,3.8,2,19,2,2,0,0,1,0,0 DATA 17,1,1,3.0,3,27,2,2,0,0,0,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 23,1,1,5.3,3,81,2,1,0,0,1,0,0 DATA 20,1,1,2.7,4,23,2,0,0,0,1,0,1 DATA 27,1,1,2.9,3,39,2,0,0,1,0,0,0 DATA 40,1,1,3.5,2,64,2,0,0,0,1,0,0 DATA 14,1,0,3.0,4,17,3,5,0,0,0,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 29,2,2,2.8,2,30,2,2,0,0,1,0,0 DATA 29,2,1,3.4,3,41,2,2,0,0,1,0,0 DATA 21,1,1,3.5,2,44,2,4,0,0,0,0,0 DATA 21,1,1,2.7,4,17,3,3,1,0,0,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 17,2,2,6.6,2,63,2,2,6,0,0,0,0 DATA 82,1,1,2.0,5,19,2,2,1,1,0,0,2 DATA 73,1,1,2.5,2,69,2,3,0,0,0,0,0 DATA 8,1,1,4.6,2,41,2,0,0,0,0,0,0 DATA 61,1,1,4.2,2,19,2,0,0,0,0,0,0 DATA 24,1,2,4.1,2,41,2,0,0,0,1,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 13,1,1,3.5,2,181,2,0,0,1,0,0,0 DATA 18,1,1,3.8,3,28,2,3,0,0,1,0,0 DATA 2,1,1,2.7,2,71,2,0,0,0,0,0,0 DATA 5,1,1,1.7,2,16,1,0,0,0,1,0,1 DATA 42,1,1,2.2,2,38,2,3,0,1,1,0,0 DATA 17,1,1,4.9,3,41,2,1,7,0,0,1,0 DATA 20,2,1,4.4,2,43,2,0,6,0,0,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 52,1,1,5.4,3,33,3,0,0,1,0,0,0 DATA 19,1,1,3.1,3,51,1,2,0,0,0,0,0 DATA 21,2,7,3.1,3,37,2,0,5,0,0,0,3 DATA 15,1,1,5.5,3,80,3,1,0,0,1,0,0 DATA 16,2,1,2.8,3,22,2,3,0,0,0,0,0 DATA 18,2,1,5.4,2,29,2,0,0,0,1,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 4,1,8,3.1,4,129,2,3,0,0,1,0,0 DATA 12,1,1,2.4,3,24,2,0,0,0,0,0,0 DATA 22,1,1,2.3,3,43,1,1,0,1,0,0,0 DATA 11,1,1,4.1,2,26,2,1,0,1,0,0,0 DATA 31,2,2,3.1,3,33,2,2,0,0,1,0,0 DATA 69,2,0,2.8,1,45,0,1,0,0,1,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 17,1,1,3.2,3,40,1,0,0,0,0,0,0 DATA 82,1,1,2.6,2,16,3,1,0,0,0,0,0 DATA 26,1,1,3.0,2,24,2,0,0,1,0,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 4,1,1,3.6,2,25,2,0,0,0,1,0,0 DATA 18,1,1,3.5,4,18,2,2,0,1,1,0,0 DATA 33,2,1,3.4,2,61,1,0,0,0,0,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 11,2,1,3.1,3,34,2,0,0,0,0,0,0 DATA 23,2,3,2.5,3,9,2,3,0,0,0,0,1 DATA 16,1,1,2.4,2,96,2,2,0,0,0,0,0 DATA 59,1,1,2.2,3,13,2,2,6,0,1,0,1 DATA 44,1,1,4.2,3,43,2,1,0,0,0,0,0 DATA 15,1,1,3.1,2,21,2,1,0,0,0,0,0 DATA 15,2,1,2.9,2,58,2,4,0,1,1,0,2 DATA 62,2,1,3.5,3,20,2,0,6,0,0,0,0 DATA 110,1,1,4.6,2,15,1,0,0,0,0,0,0 DATA 17,2,1,3.7,3,9,2,2,0,1,0,0,0 DATA 7,1,1,3.9,1,60,2,3,0,0,0,0,1 DATA 5,1,1,4.3,3,128,1,2,0,0,0,0,0 DATA 81,1,1,3.8,2,67,2,4,0,0,0,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 14,2,1,3.4,3,49,1,2,0,0,1,0,0 DATA 43,1,6,5.1,1,74,1,1,6,0,0,0,0 DATA 16,1,1,2.5,3,34,2,2,0,0,1,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 9,1,1,3.8,3,105,2,3,0,0,1,0,0 DATA 26,2,1,6.7,3,39,2,4,0,1,1,0,0 DATA 19,2,1,5.6,3,83,3,4,0,0,0,0,1 DATA 16,1,1,3.1,1,22,2,3,0,0,1,0,0 DATA 7,2,3,4.0,1,30,1,0,0,0,0,0,1 DATA 13,2,1,3.3,5,83,2,0,0,0,0,1,3 DATA 71,2,1,3.4,7,35,1,0,0,0,0,0,2 DATA 0,0,0,0,0,0,0,0,0,0,0,0,0 5 Quote Link to comment Share on other sites More sharing options...
+TheBF Posted June 2, 2018 Share Posted June 2, 2018 If you were getting overflow errors on the TI then you could still get them on a modern computer depending on the floating point data size. The TI-99 can handle a mantissa to 127 so it's pretty big. This is really cool work. Are you an MD? Quote Link to comment Share on other sites More sharing options...
+TheBF Posted June 2, 2018 Share Posted June 2, 2018 Also, would be able to save some size by reading your DATA from an input file to make more room for the neuron layers? Quote Link to comment Share on other sites More sharing options...
+Lee Stewart Posted June 3, 2018 Share Posted June 3, 2018 I have not carefully analyzed what you are doing mathematically and you may already be doing this, but, if you are dealing with summations of very large and very small numbers in the same summation, it is best , if possible, to order the addends from smallest to largest to minimize truncation errors. It may even be necessary to compartmentalize calculations for very small and very large terms until the end of the process. I had to do something like this in fitting ethanol-water composition/density data to a polynomial because the critical parts of the coefficients often required more digits of precision than were available in the C program package I was using (MS Visual C++ circa 2006 or so). This was due to the fitting equations depending on very small differences in very large numbers. What I did was to split the coefficients into high and low parts, thus doubling the number of terms, and summing the smallest terms first. ...lee Quote Link to comment Share on other sites More sharing options...
+Vorticon Posted June 3, 2018 Author Share Posted June 3, 2018 Yes I'm an MD. Javascript arithmetic is double-precision floating point on a modern PC, i.e. numbers between 10E-308 and 10+E308 can be represented, much larger than the TI. I'm hoping this will minimize the overflow errors issue. As for the data format, I could indeed have it on disk but it will slow things down significantly. Even as things stand it is painfully slow and requires Classic99 in overdrive mode to run marginally well. Besides, the savings in memory achieved will allow for only a handful of extra neurons per layer at best, which will have negligible impact on the results. I'm planning on having at least 150-200 neurons per medial synaptic layer with javascript, maybe more depending on my ultimate results. Quote Link to comment Share on other sites More sharing options...
+Vorticon Posted June 3, 2018 Author Share Posted June 3, 2018 I have not carefully analyzed what you are doing mathematically and you may already be doing this, but, if you are dealing with summations of very large and very small numbers in the same summation, it is best , if possible, to order the addends from smallest to largest to minimize truncation errors. It may even be necessary to compartmentalize calculations for very small and very large terms until the end of the process. I had to do something like this in fitting ethanol-water composition/density data to a polynomial because the critical parts of the coefficients often required more digits of precision than were available in the C program package I was using (MS Visual C++ circa 2006 or so). This was due to the fitting equations depending on very small differences in very large numbers. What I did was to split the coefficients into high and low parts, thus doubling the number of terms, and summing the smallest terms first. ...lee Ordering the addition terms will certainly be feasible but with a larger number of neurons, say 200 per medial synaptic layer, we are looking at 40000 terms for each of the medial layers, and I'm not sure what kind of performance hit this will generate. I might consider it if the error rate remains high. The same goes for compatmentalization. Great trick though! Thinking about this a bit more though, my experimentation on the TI showed that very small intermediate numbers are generated during forward propagation, and conversely very large ones during back-propagation, so ordering might not be very helpful in this situation... Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.