User Tools

Site Tools


benchmarks:savage

Calc bench: Savage

The idea
One of the original discussion on (MoHPC)
My discussion is limited to calculators that do finite precision BCD floating point arithmetic.

In the V8N1 issue of Datafile, Wlodek Mier-Jedrzejowicz had an article about the “Savage” benchmark. The benchmark is called “Savage” not because it is vicious, but because Bill Savage presented it in Byte magazine.

It is, in HP71 Basic:

5 RADIANS //the calculation is performed with the angle mode in radians.
10 A=1
20 FOR I=1 TO 2499
30 A=TAN(ATN(EXP(LOG(SQR(A*A)))))+1
40 NEXT I
50 PRINT A

Upon first examination, one thinks that the calculator should get a result of 2500.00000000 if it did everything as it should. But this is not so. The HP71 actually gets 2499.99948647. Should we be disappointed by this result? (All the other Saturn based machines, such as the HP48 get this result also.) What result should we expect, if not 2500.00000000?

Similar benchmarks
A lot of similar benchmarks have been made about the accuracy, but if they test different operations, can't be stricly compared. It seems that the Savage benchmark is quite popular, so we focus on this. (If you know similar benchmarks, report them in other wiki pages!)

About the accuracy
Interesting remark on MoHPC and another one about accuracy, and again. For those remarks maybe it's better to count only timings of results with a not so high error (less than 1 for example), since the accuracy seems to be not stable. Moreover, even for only timings, the savage benchmark is good because it test a lot of non trivial operators.

More on accuracy: (from a discussion on MoHPC's forum)

I really really dislike that version of the Savage Benchmark:

2. It first squares the number, then takes the square root. That's a little cheesy, because if every other result were perfect the “square, then square root” sequence would be operating only on whole numbers. Better to take square root first, then do square.
I disagree. Because if you do 2*2 and the square root, you have a chance to get 2. If you do square root of 2 first you have no chances to get 2, due to the fact that sqrt(2) is irrational and the calculators has a limited memory. But even you can't get 2 back if you do computations numerically, and not symbolically, on a paper. To get the perfect sqrt(2)*sqrt(2)=2 handling sqrt(2) as 1.414…. you need infinite time just to compute sqrt(2). So, imo, it's easy to get a false value if you do square root first. Then, accordingly to what i said, is better the version with square first for me.

What do we check?
Both times of execution and accuracy (the result should be: 2500) in terms of absolute error |result-n| (done at maximum precision, i'm computing it with powertoycalc for windows with 512 bit of precision). Note: maybe the calculator is more accurate that what it shows! Check in any way the entire register of the result!

How to report the results

A result is composed by the following list
- the device used plus the language used, eventual overclock, eventual custom firmware and so on.
- the number of digits used by the device [optional]
- time elapsed for a given n in seconds (see below). NOTE: with same accuracy the faster speed is preferred in the accuracy section and a result with the time is preferred to one without it.
- the result printed [optional]
- the code used.

Speed section within an error of 1 (2499)

Physical calculators

  1. Casio FX-9860GSD SDK C
    • ? digits
    • Time: around 0.42 secs @29mhz
    • Result: ?
    • Absolute error: assuming less than 1
    • Code: ?
  2. HP 50g, 2.15, HPGCC
  3. HP Prime
  4. HP 39gII
    • ? digits
    • Time: 3.6 s
    • Result: 2499.99946106
    • Absolute error: 5.3894 e-4
    • Code:
      EXPORT SAVAGE()
      BEGIN
      A:=0
        FOR I FROM 1 TO 2500 DO
         A:=TAN(ATAN(EXP(LN(sqrt(A+1)²)))); 
        END:
      END; 
  5. HP 30b
  6. Casio fx-CG 10 PRIZM, OS version 01.04.3200, Casio-BASIC
  7. HP 15c LE
    • ? digits
    • Time: around 48 secs
    • Result: 2499.969898
    • Absolute error: 3.0102 e-2
    • Code: 5)
    • Source: Message #8 Posted by Tom Grydeland on 17 Sept 2013 on http://www.hpmuseum.org
    • ? digits
    • Time: around 50 secs
    • Result: 2499.970322
    • Absolute error: 2.9678 e-2
    • Code: 6)
  8. HP 50g, 2.15 with HPGCC patch, SATURN ASM
    • 12 digits - that the number is stored in BCD with 3 digits for the exponent (one of them also stores the exponent sign - positive for 0…4, negative for 5…9), 12 digits for the mantissa and 1 separate digit for the mantissa sign (either 0 for positive or 9 for negative)
    • Time: 52.5085 s
    • Result: 2499.99948647
    • Abs err: 5.1353 e-4
    • Notes: The loop has a 2499 and 0 as parameter (ZERO_DO provides the 0 for me) because the end is always exclusive, not inclusive like with FOR loops in Basic-like languages. So if I want to run it 2499 times, I must write 2499 0 DO (or collapse the 0 DO into ZERO_DO, which is a single command) instead of 2498 0 DO.
    • Code: 7)
    • 15 digits - extended reals have two additional exponent digits and three additional mantissa digits, just like on the HP48.
    • Time: 72.7184 s
    • Result: 2499.99999106989 @extended reals
    • Abs err: 8.93011 e-6
    • Notes: Extended reals with the [double %]ATAN solution posted here
    • Code: 8)
  9. HP 48gx, SATURN ASM ( screen and keyboard scanning turned off )
    • Source: Usenet message on comp.sys.hp48 by Jonathan Busby Savage benchmarks
    • 15 digits
    • Time: around 62 secs
    • Result: 2499.99998450891
    • Absolute error: -1.549108e-5
    • Code: 9)
  10. HP 48gx, SATURN ASM ( screen and keyboard scanning turned on )
    • Source: Usenet message on comp.sys.hp48 by Jonathan Busby Savage benchmarks
    • 15 digits
    • Time: around 70 secs
    • Result: 2499.99998450891
    • Absolute error: -1.549108e-5
    • Code: 10)
  11. HP 50g, 2.15, userRPL
  12. Ti 92+, TI-ASM (BCD)
  13. HP 86b
    • ? digits
    • Time: around 82 secs
    • Result: 2499.99942403
    • Absolute error: 5.7597 e-4
    • Code:
      5 RAD
      7 t=TIME 
      10 A=1
      20 FOR I=1 TO 2499
      30 A=TAN (ATN (EXP (LOG (SQR (A*A)))))+1
      40 NEXT I
      50 PRINT A
      55 PRINT HMS$ (TIME -t)
      60 END
  14. HP 49g sysRPL
  15. Ti 89T HW4 AMS 3.10 patched with tiosmod+amspatch, GCC4TI
    • 16 digits BCD floats
    • Time: around 97 secs
    • Result: 2500.000002527092
    • Absolute error: 2.527092 e-9
    • Code: 14)
  16. HP 49g userRPL
    • ? digits
    • Time: around 111 secs
    • Result: 2499.99948647
    • Absolute error: 5.1353 e-4
    • Code: See 50g
  17. Hp 48g
  18. HP 48gx
    • ? digits
    • Time: around 118 secs
    • Result: 2499.99948647
    • Absolute error: 5.1353 e-4
    • Code: See 50g
  19. HP 48Sx
    • ? digits
    • Time: around 193 secs
    • Result: 2499.99948647
    • Absolute error: 5.1353 e-4
    • Code: See 50g
  20. Ti 92+, ti basic
  21. HP 200 LX
    • ? digits
    • Time: around 210 secs
    • Result: 2499.999999998308
    • Absolute error: 1.692 e-9
    • Code:
      0*L(A,1)+SIGMA(I,1,N,1,
      0*L(A,1+TAN(ATAN(EXP(LN(SQRT(g(A)*g(A))))))))
      +L(B,g(A))-B
      
      2499 N and solving for B
  22. HP 28s
  23. HP 27
    • ? digits
    • Time: around 280 s
    • Result: ?
    • Absolute error: ? Assuming that it's less than 1
    • Code: ?
  24. Ti 81, ti basic
  25. HP 20s
  26. HP 19BII
    • ? digits
    • Time: around 360 secs
    • Result: 2500.00005173
    • Absolute error: 5.173 e-5
    • Code:
      0*L(A,1)+SIGMA(I,1,N,1,
      0*L(A,1+TAN(ATAN(EXP(LN(SQRT(g(A)*g(A))))))))
      +L(B,g(A))-B
      
      2499 N and solving for B
  27. HP71 Basic
    • 12 digits
    • Result: 2499.99948647
    • Absolute error: 5.1353 e-4
    • Note: All the other Saturn based machines, such as the HP48, should get this result also (but maybe they have more digits!)
    • >RADIANS
      >1 T=TIME @ A=0 @ FOR I=1 TO 2500 @ B=SQR(A+1) @ A=TAN(ATAN(EXP(LN(B*B)))) @ NEXT I
      >2 DISP A;TIME-T
  28. HP 32SII
    • ? digits
    • Time: around 390
    • Result: 2499.99946106
    • Absolute error: 5.3894 e-4
    • Code: ?
  29. HP 28s
    • ? digits
    • Time: around 398 secs
    • Result: 2499.99948647
    • Absolute error: 5.1353 e-4
    • Code: see 28s
  30. HP 32S
  31. Ti 86, ti basic
  32. HP 33s
  33. HP 42s
  34. HP 35s
    • Source: HP 32SII
    • ? digits
    • Time: around 630 secs
    • Result: 2499.99939863
    • Absolute error: 6.0137 e-4
    • Code: ?
  35. fx-115ES Plus
  36. fx-115ES Plus
  37. HP 41cx
  38. HP 15c
    • ? digits
    • Time: around 5'840 secs
    • Result: 2499.969898
    • Absolute error: 3.0102 e-2
    • Code: see 15c le
  39. HP 67
  40. HP 25
  41. HP 19c
  42. HP 34c
  43. Electronika Mk-61

Emulators on mobile/handheld devices smaller than 7'' (7'' included)

Accuracy

1)
Just out of curiosity, I ran the same benchmark with HPGCC double:
int main()
{
  double f;
 
  f=1.0;
  while(f<=2499.0)
  {
    f=sqrt(f);
    f=f*f;
    f=log(f);
    f=exp(f);
    f=atan(f);
    f=tan(f);
    f+=1.0;
  }
 
  sat_push_real(f);
  return 0;
}
2)
EXPORT SAVAGE()
  BEGIN
  A:=1;
  FOR I FROM 1 TO 2499 DO
    A:=TAN(ATAN(e^(LN(\|(A*A)))))+1
  END;
END;
3)
00 - SH*   @ assign to shift-multiply button
01 - 2
02 - 4
03 - 9
04 - 9
05 - STO 0
06 - 1
07 - LBL 00
08 - X^2
09 - [sqrt]
10 - Ln
11 - e^X
12 - Math
13 - Input
14 - Down
15 - Down
16 - Down
17 - Input   @ ATAN
18 - Tan
19 - 1
20 - +
21 - DSE 0
22 - Gto 00
4)
Rad
Fix 9
1->A
For 1->I To 2499
  (tan tan^-1 e^ln sqrt(A*A))+1->A
Next
A
5)
001 LBL B
002 STO 0
003 RAD
004 FIX 9
005 0
006 LBL 0
007 1
008 +
009 SQRT
010 X^2
011 LN
012 e^X
013 ARCTAN
014 TAN
015 DSE 0
016 GTO 0
017 DEG
018 RTN
6)
43,22,11 @ LBL A
2
4
9
9
44 1     @ STO 1
1
43,22, 0 @ LBL 0
43 11    @ x^2
11       @ SQRT
43 12    @ LN
12       @ e^x
43 25    @ ATAN
25       @ TAN
1
40       @ +
42, 5, 1 @ DSE 1
22 0     @ GTO 0
43 32    @ RTN
7)
::
  %1
  2500 ZERO_DO
    DUP %* %SQRT
    %LN %EXP
    %ATAN %TAN
    %1+
  LOOP
;
8)
::
  %%1
  2499 ZERO_DO
        DUP %%* %%SQRT
        %%LN %%EXP
        %%1 SWAPDUP %%* %%1 %%+ %%SQRT %%/ %%ACOSRAD
        %%TANRAD
        %%1+_
  LOOP
;
9)
And here is the code for your reference ( Jazz syntax ):

CODE

speedup        EQU        1

SQRTF        EQU        #2B9F3
LNF        EQU        #2B698
EXPF        EQU        #2B6AA
ATANF        EQU        #2B6FB
TANF        EQU        #2B6F2
GETANGMODE EQU        #2AEF6
PUSH%%LOOP EQU        #2A235

        GOSBVL        =SAVPTR

        IF        speedup
        GOSBVL        =DispOff
        INTOFF
        ENDIF

        A=0        W
        B=0        W
        P=        14
        B=B+1        P
        P=        0
        LC(5)        #2498
        R4=C.F  A
        
        GOSBVL         GETANGMODE

        SETDEC

-        C=B        W
        D=C        W
        C=A        W
        GOSBVL        =MULTF
        GOSBVL        SQRTF
        GOSBVL        LNF
        GOSBVL        EXPF
        GOSBVL        ATANF
         GOSBVL        TANF
        
        C=0        W
        D=0        W
        P=        14
        D=D+1        P

        GOSBVL        =RADDF

        C=R4.F        A
        C=C-1        A
        R4=C.F        A
        GONC        -        

        IF        speedup
        GOSBVL        =DispOn
        INTON
        ENDIF

        GOVLNG        PUSH%%LOOP
ENDCODE
10)
And here is the code for your reference ( Jazz syntax ):

CODE

speedup        EQU        0

SQRTF        EQU        #2B9F3
LNF        EQU        #2B698
EXPF        EQU        #2B6AA
ATANF        EQU        #2B6FB
TANF        EQU        #2B6F2
GETANGMODE EQU        #2AEF6
PUSH%%LOOP EQU        #2A235

        GOSBVL        =SAVPTR

        IF        speedup
        GOSBVL        =DispOff
        INTOFF
        ENDIF

        A=0        W
        B=0        W
        P=        14
        B=B+1        P
        P=        0
        LC(5)        #2498
        R4=C.F  A
        
        GOSBVL         GETANGMODE

        SETDEC

-        C=B        W
        D=C        W
        C=A        W
        GOSBVL        =MULTF
        GOSBVL        SQRTF
        GOSBVL        LNF
        GOSBVL        EXPF
        GOSBVL        ATANF
         GOSBVL        TANF
        
        C=0        W
        D=0        W
        P=        14
        D=D+1        P

        GOSBVL        =RADDF

        C=R4.F        A
        C=C-1        A
        R4=C.F        A
        GONC        -        

        IF        speedup
        GOSBVL        =DispOn
        INTON
        ENDIF

        GOVLNG        PUSH%%LOOP
ENDCODE
11)
@ "Savage benchmark" for 48 and 49 series.
@ 48 series checksum: # ECAh
@ 48 series size:        159
@ 49 series checksum: # B0C9h
@ 49 series size:        159.
\<<             @
  STD           @ Force standard display mode.
  RCLF          @ Get original flags.
  -55. SF       @ Force last arguments disabled.
  64. STWS      @ Force wordsize.
  RAD           @ Force radians mode.
  MEM DROP      @ Force a GC.
  TICKS         @ Initial system time.
  1.            @ Initial value.
  1. 2499.      @ Loop start/stop values.
  START         @
    DUP *       @ Square.
    \v/         @ Square root command.
    LN          @
    EXP         @
    ATAN        @
    TAN         @
    1. +        @
  NEXT          @
  TICKS         @ Ending system time.
  ROT           @ Move initial time to level 1.
  -             @ Elapsed time.
  B\->R         @ Convert binary to real.
  "Ticks"       @
  \->TAG        @
  DUP           @
  8192. /       @ Convert ticks to seconds.
  3. RND        @ Round to 3 decimal places.
  "Seconds"     @
  \->TAG        @
  4. ROLL STOF  @ Restore original flags.
\>>
12)
<<
RAD TICKS
0.
1. 2500. START
1. + sqrt SQ LN EXP ATAN TAN
NEXT
SWAP TICKS SWAP - B->R 8192. /
DEG
800. 2. BEEP
>> 
13)
%%1
SWAP
DUP
%%*
%%1
%%+
%%SQRT
%%/
%%ACOSRAD
14)
// savage.c: Savage benchmark
 
#define MIN_AMS 101
#define USE_TI89
#define USE_TI92P
#define USE_V200
#define USE_TI89T
#define NO_CALC_DETECT
#define OPTIMIZE_ROM_CALLS
#define RETURN_VALUE
 
#include <stdint.h>
#include <system.h>
#include <args.h>
#include <estack.h>
#include <intr.h>
#include <timath.h>
 
#define TIMER_START_VAL (100000UL)
 
/*
5 RADIANS
10 A=1
20 FOR I=1 TO 2499
30 A=TAN(ATN(EXP(LOG(SQR(A*A)))))+1
40 NEXT I
50 PRINT A
*/
 
void _main(void) {
    uint16_t i;
    short orig_rate = PRG_getRate();
    unsigned short orig_start = PRG_getStart();
    unsigned long val = 0;
    double a = 1;
 
    // Make the system timer an order of magnitude more precise;
    // NOTE: this code assumes a HW2+ TI-68k, i.e. anything since 1999.
    PRG_setRate(1); // Increment counter at a rate of 2^19/2^9 Hz
    PRG_setStart(0xCE); // Trigger the interrupt every 257 - 0xCE = 51 increments ~ 20.07 Hz.
 
    // The PRG_getStart() above effectively waited for the interrupt to trigger, so we don't need another wait.
    /*OSRegisterTimer(USER_TIMER, 1);
    while (!OSTimerExpired(USER_TIMER));
    OSFreeTimer(USER_TIMER);*/
    OSRegisterTimer(USER_TIMER, TIMER_START_VAL);
 
    // Main loop :)
    for (i = 1; i < 2500; i++) {
        a = tan(atan(exp(log(sqrt(a * a))))) + 1;
    }
 
    // Retrieve timer value.
    val = TIMER_START_VAL - OSTimerCurVal(USER_TIMER);
    OSFreeTimer(USER_TIMER);
 
    // Push arguments onto the RPN stack: clean arguments up, then create a list.
    while (GetArgType (top_estack) != END_TAG) {
        top_estack = next_expression_index (top_estack);
    }
    top_estack--;
    push_END_TAG();
    push_Float(a); // Note: rounds to 14 digits.
    push_longint(val);
    push_LIST_TAG();
 
    // Restore old system state.
    PRG_setRate(orig_rate);
    PRG_setStart(orig_start);
}
 
Compiler options:
tigcc -v -O3 -Wall -W -mpcrel --optimize-code --cut-ranges --reorder-sections --remove-unused --merge-constants -fmerge-all-constants -Wa,--all-relocs -Wa,-l -fverbose-asm -save-temps -o savage savage.c
15)
@ "Savage benchmark" for 28 series.
@ A 28C must be in HEX mode when this program is entered to ensure
@ that the correct address is supplied to SYSEVAL!
\<<             @
  STD           @
  RCLF          @ Get original flags.
  31 CF         @ Force last arguments disabled.
  64 STWS       @ Force wordsize.
  RAD           @ Force radians mode.
  MEM DROP      @ Force a GC.

@ Uncomment the binary integer in one of the following 3 lines.
  @ #123E @     @ For 28C ROM version 1BB.
  @ #1266 @     @ For 28C ROM version 1CC.
  @ #11CAh @    @ For 28S ROM version 2BB.


  SYSEVAL       @ Initial system time.
  1             @ Initial value.
  1 2499        @ Loop start/stop values.
  START         @
    DUP *       @ Square.
    \v/         @ Square root command.
    LN          @
    EXP         @
    ATAN        @
    TAN         @
    1  +        @
  NEXT          @


@ Uncomment the binary integer in one of the following 3 lines.
  @ #123E @     @ For 28C ROM version 1BB.
  @ #1266 @     @ For 28C ROM version 1CC.
  @ #11CAh @    @ For 28S ROM version 2BB.


  SYSEVAL       @ Ending system time.
  ROT           @ Move initial time to level 1.
  -             @ Elapsed time.
  B\->R         @ Convert binary to real.
  DUP           @
  \->STR        @ Convert real to character string.
  "Ticks="      @
  SWAP +        @
  SWAP          @
  8192 /        @ Convert ticks to seconds.
  3 FIX RND     @ Round to 3 decimal places.
  \->STR        @ Convert real to character string.
  "Seconds="    @
  SWAP +        @
  4 ROLL STOF   @ Restore original flags.
\>>
16)
Example of Algebraic version on HP 20S: (Put 2500 in display and execute B)

LBL B
RAD
STO 0
0
LBL 1
+
1
=
sqrt
X^2
LN
E^X
ATAN
TAN
STO 1
1
STO-0
RCL 0
X=0?
GTO 2
RCL 1
GTO 1
LBL 2
STO 1
DEG
RTN
17)
LBL S  @ 015.5 bytes; CHKSUM=C573
RAD
2499
STO U
1
LBL U  @ 016.5 bytes; CHKSUM=EC87
x^2
SQRT
LN
e^x
ATAN
TAN
1
+
DSE U
GTO U
18)
LBL A
1
STO A
STO I
LBL B
RCL A
x^2
\|x (Square root of x)
LN
e^x
ATAN
TAN
1
+
STO A
1
STO+ I
RCL I
2,499
x>=y?
GTO B
RCL A
RTN
19)
01 LBL "SB"
02 RAD
03 0
04 LBL 01
05 1
06 +
07 SQRT
08 X^2
09 LN
10 E^X
11 ATAN
12 TAN
13 DSE ST Y
14 GTO 01
15 BEEP
16 DEG
17 END
20)
{32-Byte Prgm}
LBL "SAVAGE"
RAD
2499
1
LBL 00
X^2
SQRT
LN
EXP
ATAN
TAN
1
+
DSE ST Y
GTO 00
END
21)
LBL 'SAVAGE
TIME
RAD
2499
1
LBL 00
X^2
SQRT
LN
E^X
ATAN
TAN
1
+
DSE Y
GTO 00
TIME
R^
HMS-
BEEP
22) , 25)
01 LBL B
02 RAD
03 STO I
04 0
05 LBL 1
06 1
07 +
08 SQRT
09 X^2
10 LN
11 e^X
12 ATAN
13 TAN
14 DSE I
15 GTO 1
16 DEG
17 RTN
23)
01 LBL B
02 RAD
03 ST I
04 0
05 LBL 1
06 1
07 +
08 SQRT
09 X^2
10 LN
11 e^X
12 ATAN
13 TAN
14 DSZ I
15 GTO 1
16 DEG
17 RTN
24)
01 RAD
02 STO 0
03 0
04 STO 1
05 RCL 1
06 1
07 +
08 SQRT
09 X^2
10 LN
11 e^X
12 ATAN
13 TAN
14 STO 1
15 RCL 0
16 1
17 -
18 STO 0
19 X=0?
20 GTO 22
21 GTO 05
22 RCL 1
23 DEG
24 GTO 00
26)
00	1
01	STO 0
02	STO 1
03	2499
07	STO 2
08	RCL 2
09	RCL 1
10	-
11	x!=0 29
13	RCL 0
14	x^2
15	SQRT
16	LN
17	e^x
18	ATAN
19	TAN
20	1
21	+
22	STO 0
23	RCL 1
24	1
25	+
26	STO 1
27	GTO 08
29	R/S
benchmarks/savage.txt · Last modified: 2019/01/07 14:14 by jdb2