Ahl's Benchmark?

[JamesD]

Panasonic JR-200 (0.89MHz MN1800A, 6802 Clone)

Time: 58 secs (Hand Timed)

Accuracy: 2.1481514E-4

[thorfdbg]

I wonder what that would do for BASIC XL. 220 secs or less I would guess. (I have a BASIC XL cart).

 

It does not make much of a difference since the benchmark is limited by math, not by BASIC. I get here a time of 210.46 secs (instead of the 230 secs on the stock Atari BASIC). Accuracy is the same since the math pack is the same. All numbers for the Os++ mathpack (same memory footprint, same call-in's, same ROM size), not for the original mathpack which is slower and less accurate.

[thorfdbg]

The difference in accuracy makes me nervous. They *should* be the same.

 

The difference between A=A*A and A=A^2 is easily explained. The A*A is a straight multiply. However, Atari BASIC is not specifically clever in computing powers. Instead of testing whether the exponent is an integer and performing a chained multiply, it rather implements A^B for *all* A and B by the identity A^B = EXP(LOG(A)*B), plus some very stupid rounding (round up to the next integer if A and B are integer, which can lead to *incorrect* results!). Thus, for A^2, Atari BASIC has to go through two transcendental functions, EXP and LOG. The speedup you see with Os++ math is that both, EXP and LOG, are just more carefully implemented, and use a shorter but more accurate polynominal approximation (the minimax-polynomial). Multiplication is also a bit faster, but the speedup is mainly due to EXP and LOG.

[JamesD]

It does not make much of a difference since the benchmark is limited by math, not by BASIC. I get here a time of 210.46 secs (instead of the 230 secs on the stock Atari BASIC). Accuracy is the same since the math pack is the same. All numbers for the Os++ mathpack (same memory footprint, same call-in's, same ROM size), not for the original mathpack which is slower and less accurate.

I was actually hoping it might get under 200 secs but BASIC XL was obviously as much or more about new features as it was about speed optimizations.

Still, 20 (ish) seconds is a decent improvement over Atari BASIC.

[JamesD]

Panasonic JR-200 (0.89MHz MN1800A, 6802 Clone)

Time: 58 secs (Hand Timed)

Accuracy: 2.1481514E-4

it's pretty obvious from this number and from the Acorn machine benchmarks that Microsoft BASIC wasn't very fast.

 

The 6802 is basically a 6800 with some hardware interface changes and maybe some improved clock cycle times.

It still only has two 8 bit accumulators so any 16 bit math (calculating large numbers or addresses) is still done similar to how you'd do it on a 6502 with multiple steps.

The 6803 added the 16 bit D accumulator (A and B combined), a multiply instruction, a few other instructions and more optimized execution. A 16 bit add or subtraction is one instruction and the multiply can make a huge difference on some code.

 

To be blunt, a 6802 of a similar clock speed shouldn't beat a 6803 let alone a 6809 and yet the JR-200 BASIC benchmarks at twice the speed of an MC-10 and an MC-10 is slightly faster than a standard CoCo... at least until you put the CoCo in double speed mode.

 

That should also tell you just how bad Atari BASIC was from a speed standpoint.

 

You have to wonder what would have happened if someone had stepped in with a faster BASIC in the Altair days.

Every machine currently with a Microsoft BASIC could have run BASIC at least twice as fast.

[+nanochess]

Every machine currently with a Microsoft BASIC could have run BASIC at least twice as fast.

At least MSX-Basic used BCD representation for floating point numbers, that was painfully slow. http://edmond.orignac.pagesperso-orange.fr/detokenizer.html

 

Besides MS was never good for making fast programs, I remember installing MS-Multiplan in my CP/M Emulator just for testing and being amazed of how slow was it compared with other similar programs.

[thorfdbg]

That should also tell you just how bad Atari BASIC was from a speed standpoint.

 

Yes, but... Ahi's benchmark is actually, as a benchmark, pretty bad either. This doesn't make Atari Basic any better, but it doesn't expose its "slowness" in all aspects. Atari BASIC is slow for more than one reason, and what this benchmark measures is only the math pack speed. The random test is a pure joke - if you want to see decent tests for random-ness (for whatever this word means), please check volume 2 of Knuth's "The Art of Computing". That is a good source.

 

The other reasons why Atari BASIC was pretty slow was that it managed loops in a lousy way, probably to avoid a race condition when a loop was interrupted, a line was inserted in the middle and the program was continued. As Atari BASIC only stores line numbers and offsets, this operation is safe. It is not for TurboBasic, and I wonder what it does to avoid trouble in such a case.

 

Atari BASIC was pretty good in other aspects, for example for elementary graphics and sound support (you couldn't do anything interesting on a C64 without a pile of peeks and pokes), and that it detected errors right away when you typed in the line. But math was lousy in particular, indeed.

 

For a machine that really ruined the BASIC speed, just see how complicated TI did it. An interpreter running on top of an interpreter. *That* is bad design of the finest.

[JamesD]

Ahl's benchmark is pretty limited but it's big enough to bring out some flaws in machines such as speed, accuracy in the math library, and a poor random number generator.

Given the number of machines Ahl got to test at Creative Computing I'm sure he saw some machines that did poorly or he wouldn't have built the test like that.

 

Frankly, Atari BASIC does poorly in every BASIC benchmark you can find. An improved math library will help in some cases but I'm not sure how much it would help in others.

An example:

http://www.retroprogramming.com/2010/01/8-bit-home-computer-benchmarks.html

The Atari seems more competitive because the only math is simple addition and the poor math library flaw isn't as exposed. However, the Sinclair Spectrum's complex parsing and other issues don't benefit from the math simplicity so the Atari beats it this time. YAY Atari! But the Atari still gets beat by every other machine tested. A CoCo 3 may have a 6809 running roughly the same clock rate of the Atari but benchmarking twice as fast? And we already know that other BASICs could be twice as fast as Microsoft BASIC.

 

Yeah the Atari has more graphics and sound support than the C64. But it's easier to move sprites around on the C64 than the Atari.

To be honest, an Apple II has more graphics support in BASIC than a C64 and Atari BASIC pales in comparison to Extended Color BASIC on the CoCo.

Other BASICs also have better string manipulation, text formatting and file I/O.

Picking and choosing favorable comparisons to show a machine in a good light isn't really representative of the machine overall.

I think a combination of string manipulation, math, looping, etc... should all be in a good benchmark for a BASIC interpreter. Not only that, I'd like to see the benchmark have hundreds of lines of code so that GOTO and GOSUB line searching flaws come to light. A program with 10 lines of code won't really do that very well.

 

TI BASIC was designed to run on a new CPU that engineering wanted. If they had known it would never be built I'm sure they wouldn't have designed it as running an interpreter on an interpreter and with that custom CPU it would run very fast. Just remember, in spite of that flaw, TI BASIC beat regular Atari BASIC in Ahl's benchmark and even with the improved math lib TI BASIC was close behind.

 

The point of my comment was that many of these old machines could have run a lot faster in BASIC.

Seriously, what if Microsoft had been beaten by a faster competitor on the Altair?

Then that company might have dominated writing BASIC for multiple CPUs. Apple, Commodore, Tandy, etc... would have had that BASIC.

Now, what if Atari had approached that company and was offered better terms than what Microsoft did?

Then Atari might have had a BASIC that could run the benchmark in under a minute.

I know I would have loved it if BASIC had run twice as fast on my CoCo, imagine about 4 times as fast on the Atari.

[JamesD]

At least MSX-Basic used BCD representation for floating point numbers, that was painfully slow. http://edmond.origna...etokenizer.html

Yeah, the MSX Turbo R results look fairly impressive vs other machines but I don't think most people grasp just how fast that machine really is because of the slow floating point math.

The Amstrad ran it in about 39 seconds and the Turbo R did it in about 30 seconds. At a glance you might think the Turbo R is 25% faster but in reality it's way more than that.

[jstimson]

Here is a quick-and-dirty scan of the original article from Creative Computing (March 1994).

[JamesD]

LOL, the Coleco ADAM beat most of the 8 bits we've benchmarked. Who knew?
They obviously didn't use the high speed POKE on the CoCo.

[carlsson]

I just run Ahl's benchmark on Creativision BASIC 1.0. It actually has the A**2 function. These are the results:

 

Time: 3 min 13 sec (approx)

Accuracy: 0.37207 (is this the worst so far?)

Random: 4.55102 (not sure if this is good or bad)

 

It can be compared with the Laser 2001 BASIC that I benchmarked six years ago, clocking in at a mere 45 seconds. It means that the VRAM based BASIC in the Creativision in theory takes four times the amount of time as the CPU RAM based BASIC in the Laser 2001 model, while otherwise being just about identical in hardware.

 

Some of those days I might re-benchmark the BBC Master Compact and make sure it uses an exponential function. ISTR it somehow pre-compiles parts of the program on the fly, but I may have misunderstood its inner workings.

Edited December 16, 2013 by carlsson

[+Larry]

Thanks for the post. It got me thinking again about the original article. (Your post is a follow-up article with many more results.)

 

It has been scanned and posted here:

 

http://archive.org/stream/creativecomputing-1983-11/Creative_Computing_v09_n11_1983_November#page/n269/mode/2up

 

Reading the original benchmarks, I was chagrined to see that the Atari 2600 Graduate ran substantially faster! That is a "low blow!"

 

-Larry

[+Larry]

Here is a quick-and-dirty scan of the original article from Creative Computing (March 1994).

 

Are you sure about the date of the article? Pretty sure that CC ceased publication in the 1980's. If you have that article could you please check.

 

Thanks,

Larry

 

EDIT: I found it -- March, 1984

 

Here is a link to the article and entire mag scan:

 

https://archive.org/stream/creativecomputing-1984-03/Creative_Computing_v10_n03_1984_Mar#page/n7/mode/2up

[jstimson]

Sorry, typo in message, it should read 1984, not 1994.

[carlsson]

Atari Microsoft Basic Time=101.4 Accuracy .150879 Random=2.06506, 2.06506 (5.60 when RANDOMIZEd)

Is that accuracy correct or did you put the decimal comma in the wrong place? The reason for asking is that most other Microsoft BASIC derivates, e.g. the Commodore but also Oric yield close to 1/100 of that error. While not every version of BASIC will have the same precision, it surprises me if Microsoft's implementation on the Atari would be one of the five (*) least precise BASIC implementations ever, all systems counted.

 

(*) I think the bottom list currently looks like CreatiVision 0.37, COMX-35 0.32, Mattel Aquarius 0.18, NEC PC-8201 0.18, Atari Microsoft 0.15, while the two TI-99 BASIC's and the MSX Turbo-R would be on top of the accuracy list.

Edited December 17, 2013 by carlsson

[+Larry]

Hi-

 

"Is that accuracy correct or did you put the decimal comma in the wrong place?"

 

"Atari Microsoft Basic Time=101.4 Accuracy .150879 Random=2.06506, 2.06506 (5.60 when RANDOMIZEd)"

 

BTW, this wasn't double precision.

 

I re-ran it this AM and got essentially the same results as shown on the screen shot below.

 

-Larry

[JamesD]

I just run Ahl's benchmark on Creativision BASIC 1.0. It actually has the A**2 function. These are the results:

 

Time: 3 min 13 sec (approx)

Accuracy: 0.37207 (is this the worst so far?)

Random: 4.55102 (not sure if this is good or bad)

 

It can be compared with the Laser 2001 BASIC that I benchmarked six years ago, clocking in at a mere 45 seconds. It means that the VRAM based BASIC in the Creativision in theory takes four times the amount of time as the CPU RAM based BASIC in the Laser 2001 model, while otherwise being just about identical in hardware.

 

Some of those days I might re-benchmark the BBC Master Compact and make sure it uses an exponential function. ISTR it somehow pre-compiles parts of the program on the fly, but I may have misunderstood its inner workings.

The Creatavision and Laser 2001 seem to have different BASICs (???? vs Microsoft) even thought the Laser version is backwards compatible.

It may be more than the VRAM based BASIC code and variables responsible for that time.

But then it's still faster than the TI-99/4A and Atari BASIC.

 

It's interesting that the Laser 2001 and ADAM benchmarks are so close. They have similar hardware except the CPU. The ADAM doesn't use a Microsoft BASIC so that may impact the timing but if it doesn't, it pretty much demonstrates a 2:1 clock ratio for Z80 vs 6502 performance.

 

Random doesn't really mean much unless the number always comes up the same from one run to another. Then it's not random.

[JamesD]

Is that accuracy correct or did you put the decimal comma in the wrong place? The reason for asking is that most other Microsoft BASIC derivates, e.g. the Commodore but also Oric yield close to 1/100 of that error. While not every version of BASIC will have the same precision, it surprises me if Microsoft's implementation on the Atari would be one of the five (*) least precise BASIC implementations ever, all systems counted.

 

(*) I think the bottom list currently looks like CreatiVision 0.37, COMX-35 0.32, Mattel Aquarius 0.18, NEC PC-8201 0.18, Atari Microsoft 0.15, while the two TI-99 BASIC's and the MSX Turbo-R would be on top of the accuracy list.

That number is published in the magazine article. FWIW, it sounds odd to me as well since all other 6502 Microsoft BASICs seem to use the same math code.

[carlsson]

Ok, thanks for the Atari Microsoft re-run. I don't know about which precision the Commodore, Oric and whichever else Microsoft BASIC for 6502 systems (perhaps the Laser 2001 counts towards those after all, can't recall how Applesoft BASIC relates to Microsoft) use. It'd be interesting to run the benchmark on a PET 2001 or similar with Commodore BASIC V1, can be arranged through emulation with fairly high accuracy on timing. Perhaps also Microtan 65 (precursor to the Oric series) which was benchmarked in a previous (2003) run of the Byte benchmarks. If the Microsoft BASIC available for Atari is a rather early one with poor math library, there should be some other (6502) based versions with similar specs.

 

As for comparing systems with similar hardware, this one doesn't depend a lot of graphics or sound chips, and as long as the BASIC listing and workspace memory is in CPU accessible RAM, I think just about all of them can be compared to eachother regarding type of CPU, clock frequency and BASIC vendor. For that matter, there are Z80 implementations of BBC BASIC too, it could also be interesting to locate and try on a supported system, e.g. ZX Spectrum IIRC.

 

Edit: Using WinVICE, I get the estimates of a PET 2001 with BASIC 1: 157 seconds (TI$ function), accuracy 1.04E-03 which equals the other Commodore and Oric results. If it wasn't for the Oric, I'd say that Commodore rewrote and improved the SQR and ^ functions, but I'm not sure if they contributed such code to Microsoft for them to sell to future customers. Also, the PET in 1977 would still predate the Atari 400/800 if there were plans to equip those with Microsoft BASIC (but which didn't fit in the ROM space with added commands for graphics and sound, IIRC).

Edited December 17, 2013 by carlsson

[The Usotsuki]

Isn't Atari Microsoft Basic 32-bit FP, and Commodore and Applesoft are 40-bit FP?

[JamesD]

Ok, thanks for the Atari Microsoft re-run. I don't know about which precision the Commodore, Oric and whichever else Microsoft BASIC for 6502 systems (perhaps the Laser 2001 counts towards those after all, can't recall how Applesoft BASIC relates to Microsoft) use. It'd be interesting to run the benchmark on a PET 2001 or similar with Commodore BASIC V1, can be arranged through emulation with fairly high accuracy on timing. Perhaps also Microtan 65 (precursor to the Oric series) which was benchmarked in a previous (2003) run of the Byte benchmarks. If the Microsoft BASIC available for Atari is a rather early one with poor math library, there should be some other (6502) based versions with similar specs.

 

As for comparing systems with similar hardware, this one doesn't depend a lot of graphics or sound chips, and as long as the BASIC listing and workspace memory is in CPU accessible RAM, I think just about all of them can be compared to eachother regarding type of CPU, clock frequency and BASIC vendor. For that matter, there are Z80 implementations of BBC BASIC too, it could also be interesting to locate and try on a supported system, e.g. ZX Spectrum IIRC.

It you want to see how 6502 Microsoft BASICs relate, this is a pretty good place to start:

http://www.pagetable.com/?p=46

 

The difference in math might be traced to here:

http://www.pagetable.com/?p=45

 

Applesoft II *is* Microsoft BASIC but includes some things for backwards compatibility with Integer BASIC and the Apple monitor.

It has the same accuracy as Commodore.

 

Commodore bought their copy for a one time fee and then started modifying it themselves. It stayed mostly the same from the PET until their business machines, the Plus/4 and C128 where they added new features.

 

As for comparing the hardware speed... ultimately it depends on how efficient the BASIC interpreter is on a given machine. A small change in how an interpreter works on a given CPU to take advantage of it's features could mean a big change in performance. If the authors didn't do any optimization it's not real representative of what a CPU can do. And you have to be careful with math libraries obviously. Different precision, accuracy or implementation (BCD for example) pretty much biases the results... obviously.

Edited December 17, 2013 by JamesD

[carlsson]

The Creativision [..] is still faster than the TI-99/4A and Atari BASIC.

Actually Ahl's benchmark gives a very skewed picture in this case. Tonight I ran the Byte 1977 benchmark suite (7 programs + another one from Creative Computing in 1983) and it showed with painful effectiveness that Creativision BASIC sucks bowling balls through straws when it comes to loops, assignments, jumps and comparisons. However, it appears to have rather speedy arithmetic, logarithmic and trigonometric functions, which means in a program spending most of its time doing mathematical calculations, the trickier the better, it appears to be average or even just above average. I don't know what that boils down to, perhaps the VDP RAM usage after all? Then again TI BASIC also stores its programs in VDP RAM and despite generally being called slow, runs in circles around the Creativision. Either the 3 MHz TMS9900 is that much faster than a 2 MHz 6502, or there are additional factors coming into play. Check the CreatiVEmu forum if you want raw numbers.

[JamesD]

The complex math routines on the TI-99 are written in GPL and the math "Value Stack" is in VRAM. I guess it's the equivalent of Microsoft BASIC's floating point accumulators which are usually on page 0 on the 6502, 6803 and 6809. If the Creativision uses page zero for this that would explain the difference in speed for the math.

http://atariage.com/forums/topic/219836-basic-floating-point-routines-in-gpl-or-native-9900-code/

Edited December 18, 2013 by JamesD

[+Larry]

In the above posted *followup* to the original article, Mr. Ahl mentioned an improved benchmark. Is anyone aware if this was ever published?

 

IIRC, it was Bill Wilkinson that was quite critical of the original benchmark which put Atari Basic in such a poor light compared to its peers.

 

-Larry

 

 

EDIT: In July of 1984, another update of the benchmarks indicated that the improved benchmark would be published in early Fall, (1984). I sure can't find it! I went issue-by-issue through the middle of 1985 and found no new benchmark published.