The testbed was, as last time, an array of 2048 single-precision floating point numbers, this time interpreted as a packed list of 682 three-dimensional vectors. This number was chosen so that both it and the output array were sure to fit in the L1 cache; however, because three floats add up to twelve bytes, this means that three out of four vectors were not aligned to a 16-byte boundary, which is significant for the SIMD test case as I had to use the movups unaligned load op. Each timing case consisted of looping over this array of vectors 2048 times, normalizing each and writing the result to memory.

Each normalize function computed the length of the vector 1/√(x² + y² + z²), multiplied each component by the reciprocal, and then wrote it back through an output pointer. The main difference was in how the reciprocal square root was computed:

• via the x87 FPU, by simply compiling 1.0f/sqrt( x*x + y*y + z*z )
• via the SSE scalar unit, by compiling 1.0f/sqrt( x*x + y*y + z*z ) with the /arch:SSE2 option set; this causes the compiler to issue a sqrtss followed by an fdiv — ie, it computes the square root and then divides one by it
• via the SSE scalar unit, by using the estimated reciprocal square root intrinsic and then performing one step of Newton-Raphson iteration
• via the SSE SIMD unit, working on the whole vector at once

In all cases the results were accurate to 22 bits of precision. The results for 1,396,736 vector normalizations were:

Two things jump out here. First, even when the square root op is surrounded by lots of other math — multiplies, adds, loads, stores — optimizations such as this can make a huge difference. It’s not just the cost of the sqrt itself, but also that it’s unpipelined, which means it ties up an execution unit and prevents any other work from being done until it’s entirely completed.

Second, in this case, SIMD is only a very modest benefit. That’s because the input vectors are unaligned, and the two key steps of this operation, the dot product and the square root, are scalar in nature. (This is what’s meant by “horizontal” SIMD computation — operations between the components of one vector, rather than between the corresponding words of two vectors. Given a vector V ∋ <x,y,z>, the sum x + y + z is horizontal, but with two vectors V₁ and V₂, V₃ = <x₁+x₂, y₁+y₂, z₁+z₂> is vertical.) So it really doesn’t play to SIMD’s strengths at all.

On the other hand, if I were to normalize four vectors at a time, so that four dot products and four rsqrts could be performed in parallel in the four words of a vector register, then the speed advantage of SIMD would be much greater. But, again, my goal wasn’t to test performance in tight loops over packed data — it was to figure out the best way to do something like an angle check in the middle of a character’s AI, where you usually deal with one vector at a time.

Source code for my testing functions below the jump. Note that each function writes the normalized vector through an out pointer, but also returns the original vector’s length. The hand-written intrinsic versions probably aren’t totally optimal, but they ought to be good enough to make the point.

Naive vector normalize, x87 FPU or SSE scalar

// Normalizes an assumed 3-element vector starting
// at pointer V, and returns the length of the original
// vector.
inline float NaiveTestNormalize( float * RESTRICT vOut, const float * RESTRICT vIn )
{
        const float l = vIn[0]*vIn[0] + vIn[1]*vIn[1] + vIn[2]*vIn[2];
        const float rsqt = 1.0f / sqrt(l);
        vOut[0] = vIn[0] * rsqt;
        vOut[1] = vIn[1] * rsqt;
        vOut[2] = vIn[2] * rsqt;
        return rsqt * l;
}

_TEXT   SEGMENT
_vOut$ = 8                                              ; size = 4
_vIn$ = 12                                              ; size = 4
?TestNormalize@@YAMPIAMPIBM@Z PROC                      ; TestNormalize, COMDAT
 
; 396  :        const float l = vIn[0]*vIn[0] + vIn[1]*vIn[1] + vIn[2]*vIn[2];
 
        mov     eax, DWORD PTR _vIn$[esp-4]
        fld     DWORD PTR [eax+8]
 
; 397  :        const float rsqt = 1.0f / sqrt(l);
; 398  :        vOut[0] = vIn[0] * rsqt;
 
        mov     ecx, DWORD PTR _vOut$[esp-4]
        fld     DWORD PTR [eax+4]
        fld     DWORD PTR [eax]
        fmul    ST(0), ST(0)
        fld     ST(1)
        fmulp   ST(2), ST(0)
        faddp   ST(1), ST(0)
        fld     ST(1)
        fmulp   ST(2), ST(0)
        faddp   ST(1), ST(0)
        fld     ST(0)
        fsqrt
        fld1
        fdivrp  ST(1), ST(0)
        fld     DWORD PTR [eax]
        fmul    ST(0), ST(1)
        fstp    DWORD PTR [ecx]
 
; 399  :        vOut[1] = vIn[1] * rsqt;
 
        fld     ST(0)
        fmul    DWORD PTR [eax+4]
        fstp    DWORD PTR [ecx+4]
 
; 400  :        vOut[2] = vIn[2] * rsqt;
 
        fld     ST(0)
        fmul    DWORD PTR [eax+8]
        fstp    DWORD PTR [ecx+8]
 
; 401  :        return rsqt * l;
 
        fmulp   ST(1), ST(0)
 
; 402  : }
 
        ret     0
?TestNormalize@@YAMPIAMPIBM@Z ENDP                      ; TestNormalize
_TEXT   ENDS

_TEXT   SEGMENT
_l$ = -4                                                ; size = 4
_vOut$ = 8                                              ; size = 4
_rsqt$ = 12                                             ; size = 4
_vIn$ = 12                                              ; size = 4
?TestNormalize@@YAMPIAMPIBM@Z PROC                      ; TestNormalize, COMDAT
 
; 392  : {
 
        push    ecx
 
; 393  :        const float l = vIn[0]*vIn[0] + vIn[1]*vIn[1] + vIn[2]*vIn[2];
 
        mov     eax, DWORD PTR _vIn$[esp]
        movss   xmm1, DWORD PTR [eax+4]
        movss   xmm2, DWORD PTR [eax]
        movss   xmm0, DWORD PTR [eax+8]
 
; 394  :        const float rsqt = 1.0f / sqrt(l);
; 395  :        vOut[0] = vIn[0] * rsqt;
 
        mov     eax, DWORD PTR _vOut$[esp]
        movaps  xmm3, xmm2
        mulss   xmm3, xmm2
        movaps  xmm4, xmm1
        mulss   xmm4, xmm1
        addss   xmm3, xmm4
        movaps  xmm4, xmm0
        mulss   xmm4, xmm0
        addss   xmm3, xmm4
        movss   DWORD PTR _l$[esp+4], xmm3
        sqrtss  xmm4, xmm3   ;; slow full-precision square root gets stored in xmm4
        movss   xmm3, DWORD PTR __real@3f800000  ;; store 1.0 in xmm3
        divss   xmm3, xmm4  ;; divide 1.0 / xmm4 to get the reciprocal square root !?!
        movss   DWORD PTR _rsqt$[esp], xmm3
 
; 396  :        vOut[1] = vIn[1] * rsqt;
; 397  :        vOut[2] = vIn[2] * rsqt;
; 398  :        return rsqt * l;
 
        fld     DWORD PTR _rsqt$[esp]
        mulss   xmm2, xmm3
        fmul    DWORD PTR _l$[esp+4]
        mulss   xmm1, xmm3
        mulss   xmm0, xmm3
        movss   DWORD PTR [eax], xmm2
        movss   DWORD PTR [eax+4], xmm1
        movss   DWORD PTR [eax+8], xmm0
 
; 399  : }
 
        pop     ecx
        ret     0
?TestNormalize@@YAMPIAMPIBM@Z ENDP                      ; TestNormalize
_TEXT   ENDS

Vector normalize, hand-written SSE scalar by intrinsics

// SSE scalar reciprocal sqrt using rsqrt op, plus one Newton-Rhaphson iteration
inline __m128 SSERSqrtNR( const __m128 x )
{
	__m128 recip = _mm_rsqrt_ss( x );  // "estimate" opcode
	const static __m128 three = { 3, 3, 3, 3 }; // aligned consts for fast load
	const static __m128 half = { 0.5,0.5,0.5,0.5 };
	__m128 halfrecip = _mm_mul_ss( half, recip );
	__m128 threeminus_xrr = _mm_sub_ss( three, _mm_mul_ss( x, _mm_mul_ss ( recip, recip ) ) );
	return _mm_mul_ss( halfrecip, threeminus_xrr );
}
 
 
inline __m128 SSE_ScalarTestNormalizeFast(  float * RESTRICT vOut, float * RESTRICT vIn )
{
        __m128 x = _mm_load_ss(&vIn[0]);
        __m128 y = _mm_load_ss(&vIn[1]);
        __m128 z = _mm_load_ss(&vIn[2]);
 
        const __m128 l =  // compute x*x + y*y + z*z
                _mm_add_ss(
                 _mm_add_ss( _mm_mul_ss(x,x),
                             _mm_mul_ss(y,y)
                            ),
                 _mm_mul_ss( z, z )
                );
 
 
        const __m128 rsqt = SSERSqrtNR( l );
        _mm_store_ss( &vOut[0] , _mm_mul_ss( rsqt, x ) );
        _mm_store_ss( &vOut[1] , _mm_mul_ss( rsqt, y ) );
        _mm_store_ss( &vOut[2] , _mm_mul_ss( rsqt, z ) );
 
        return _mm_mul_ss( l , rsqt );
}

_TEXT   SEGMENT
_vOut$ = 8                                              ; size = 4
_vIn$ = 12                                              ; size = 4
?SSE_ScalarTestNormalizeFast@@YA?AT__m128@@PIAM0@Z PROC ; SSE_ScalarTestNormalizeFast, COMDAT
 
    push    ebp
    mov     ebp, esp
    and     esp, -16                                ; fffffff0H
 
    mov     eax, DWORD PTR _vIn$[ebp]
    movss   xmm0, DWORD PTR [eax]
 
    movss   xmm3, DWORD PTR [eax+4]
 
    movaps  xmm7, XMMWORD PTR ?three@?1??SSERSqrtNR@@YA?AT__m128@@T2@@Z@4T2@B
    movaps  xmm2, xmm0
    movss   xmm0, DWORD PTR [eax+8]
 
    mov     eax, DWORD PTR _vOut$[ebp]
    movaps  xmm4, xmm0
    movaps  xmm0, xmm2
    mulss   xmm0, xmm2
    movaps  xmm1, xmm3
    mulss   xmm1, xmm3
    addss   xmm0, xmm1
    movaps  xmm1, xmm4
    mulss   xmm1, xmm4
    addss   xmm0, xmm1
    movaps  xmm1, xmm0
    rsqrtss xmm1, xmm1
    movaps  xmm5, xmm1
    mulss   xmm1, xmm5
    movaps  xmm6, xmm0
    mulss   xmm6, xmm1
    movaps  xmm1, XMMWORD PTR ?half@?1??SSERSqrtNR@@YA?AT__m128@@T2@@Z@4T2@B
    mulss   xmm1, xmm5
    subss   xmm7, xmm6
    mulss   xmm1, xmm7
    movaps  xmm5, xmm1
    mulss   xmm5, xmm2
    movss   XMMWORD PTR [eax], xmm5
    movaps  xmm2, xmm1
    mulss   xmm2, xmm3
 
    movss   XMMWORD PTR [eax+4], xmm2
    movaps  xmm2, xmm1
    mulss   xmm2, xmm4
 
    movss   XMMWORD PTR [eax+8], xmm2
 
    mulss   xmm0, xmm1
 
    mov     esp, ebp
    pop     ebp
    ret     0
?SSE_ScalarTestNormalizeFast@@YA?AT__m128@@PIAM0@Z ENDP ; SSE_ScalarTestNormalizeFast
_TEXT   ENDS

Vector normalize, hand-written SSE SIMD by intrinsics

inline __m128 SSE_SIMDTestNormalizeFast( float * RESTRICT vOut, float * RESTRICT vIn  )
{
        // load as a SIMD vector
        const __m128 vec = _mm_loadu_ps(vIn);
        // compute a dot product by computing the square, and
        // then rotating the vector and adding, so that the
        // dot ends up in the low term (used by the scalar ops)
        __m128 dot = _mm_mul_ps( vec, vec );
        // rotate x under y and add together   
        __m128 rotated = _mm_shuffle_ps( dot, dot, _MM_SHUFFLE( 0,3,2,1 ) ); // YZWX ( shuffle macro is high to low word )
        dot = _mm_add_ss( dot, rotated ); // x^2 + y^2 in the low word
        rotated = _mm_shuffle_ps( rotated, rotated, _MM_SHUFFLE( 0,3,2,1 ) ); // ZWXY
        dot = _mm_add_ss( dot, rotated ); // x^2 + y^2 + z^2 in the low word
 
        __m128 recipsqrt = SSERSqrtNR( dot ); // contains reciprocal square root in low term
        recipsqrt = _mm_shuffle_ps( recipsqrt, recipsqrt, _MM_SHUFFLE( 0, 0, 0, 0 ) ); // broadcast low term to all words
 
        // multiply 1/sqrt(dotproduct) against all vector components, and write back
        const __m128 normalized = _mm_mul_ps( vec, recipsqrt );
        _mm_storeu_ps(vOut, normalized);
        return _mm_mul_ss( dot , recipsqrt );
}

_TEXT   SEGMENT
_vOut$ = 8                                              ; size = 4
_vIn$ = 12                                              ; size = 4
?SSE_SIMDTestNormalizeFast@@YA?AT__m128@@PIAM0@Z PROC   ; SSE_SIMDTestNormalizeFast, COMDAT
 
 
    push    ebp
    mov     ebp, esp
    and     esp, -16                                ; fffffff0H
 
    mov     eax, DWORD PTR _vIn$[ebp]
    movups  xmm2, XMMWORD PTR [eax] ;; load the input vector
    movaps  xmm5, XMMWORD PTR ?three@?1??SSERSqrtNR@@YA?AT__m128@@T2@@Z@4T2@B ;; load the constant "3"
    mov     ecx, DWORD PTR _vOut$[ebp]
    movaps  xmm0, xmm2
    mulps   xmm0, xmm2
    movaps  xmm1, xmm0
    shufps  xmm1, xmm0, 57	; shuffle to YZWX
    addss   xmm0, xmm1      ; add Y to low word of xmm0
    shufps  xmm1, xmm1, 57	; shuffle to ZWXY
    addss   xmm0, xmm1      ; add Z to low word of xmm0
 
    movaps  xmm1, xmm0        
    rsqrtss xmm1, xmm1      ; reciprocal square root estimate
    movaps  xmm3, xmm1
    mulss   xmm1, xmm3
    movaps  xmm4, xmm0
    mulss   xmm4, xmm1
    movaps  xmm1, XMMWORD PTR ?half@?1??SSERSqrtNR@@YA?AT__m128@@T2@@Z@4T2@B
    mulss   xmm1, xmm3
    subss   xmm5, xmm4
    mulss   xmm1, xmm5      ; Newton-Raphson finishes here; 1/sqrt(dot) is in xmm1's low word
 
    shufps  xmm1, xmm1, 0   ; broadcast so that xmm1 has 1/sqrt(dot) in all words
    movaps  xmm3, xmm1
    mulps   xmm3, xmm2      ; multiply all words of original vector by 1/sqrt(dot)
    movups  XMMWORD PTR [ecx], xmm3   ; unaligned save to memory
 
	; return dot * 1 / sqrt(dot) == sqrt(dot) == length of vector
    mulss   xmm0, xmm1
 
    mov     esp, ebp
    pop     ebp
    ret     0
?SSE_SIMDTestNormalizeFast@@YA?AT__m128@@PIAM0@Z ENDP   ; SSE_SIMDTestNormalizeFast
_TEXT   ENDS

What is the fastest way to compute a square root? It would seem that if the CPU has a native square-root opcode, there’s no beating the hardware, but is it really true?

Such questions vex me, so I went and measured all the different means of computing the square root of a scalar single-precision floating point number that I could think of. I ran trials on my Intel Core 2 and on the Xenon, comparing each technique for both speed and accuracy, and some of the results were surprising!

In this article I’ll describe my results for the Intel hardware; next week I’ll turn to the Xenon PPC.

Experimental setup

I’ll post the whole source code for my tests elsewhere, but basically each of these trials consists of iterating N times over an array of floating point numbers, calling square root upon each of them and writing it to a second output array.

(see pseudocode)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

inline float TestedFunction( float x ) 
{
   return sqrt(x); // one of many implementations..
}
void TimeSquareRoot()
{
   float numbersIn[ ARRAYSIZE ];    // ARRAYSIZE chosen so that both arrays 
   float numbersOut[ ARRAYSIZE ];  // fit in L1 cache
   // assume that numbersIn is filled with random positive numbers, and both arrays are 
   // prefetched to cache...
   StartClockCycleCounter();
   for ( int i = 0 ; i < NUMITERATIONS ; ++i ) 
      for ( int j = 0 ; j < ARRAYSIZE ; ++j ) // in some cases I unroll this loop
      {
         numbersOut[j] = TestedFunction( numbersIn[j] ); 
      }
   StopClockCycleCounter();
   printf( "%.3f millisec for %d floats\n",  
             ClockCycleCounterInMilliseconds(), ARRAYSIZE * NUMITERATIONS ); 
 
   // now measure accuracy
   float error = 0;
   for ( int i = 0 ; i < ARRAYSIZE ; ++i )
   {
       double knownAccurate = PerfectSquareRoot( numbersIn[i] );
       error += fabs( numbersOut[i] - knownAccurate  ) / knownAccurate ;
   }
   error /= ARRAYSIZE ;
   printf( "Average error: %.5f%%\n", error * 100.0f );
}

In each case I verified that the compiler was not eliding any computations (it really was performing ARRAYSIZE × NUMITERATIONS many square roots), that it was properly inlining the tested function, and that all the arrays fit into L1 cache so that memory latency wasn’t affecting the results. I also only tested scalar square root functions — SIMD would clearly be the fastest way of working on large contiguous arrays, but I wanted to measure the different techniques of computing one square root at a time, as is usually necessary in gameplay code.

Because some of the speedup techniques involve trading off accuracy, I compared the resulting numbers against the perfectly-accurate double-precision square root library routine to get an average error for each test run.

And I performed each run multiple times with different data, averaging the final results together.

x86 results

I ran my tests on a 2.66Ghz Intel Core 2 workstation. An x86 chip actually has two different means of performing scalar floating-point math. By default, the compiler uses the old x87 FPU, which dates back to 1980 with a stack-based instruction set like one of those old RPN calculators. In 1999, Intel introduced SSE, which added a variety of new instructions to the processor. SSE is mostly thought of as a SIMD instruction set — for operating on four 32-bit floats in a single op — but it also includes an entire set of scalar floating point instructions that operate on only one float at a time. It’s faster than the x87 operations and was meant to deprecate the old x87 pathway. However, both the MSVC and GCC compilers default to exclusively using the x87 for scalar math, so unless you edit the “code generation” project properties panel (MSVC) or provide a cryptic obscure command line option (GCC), you’ll be stuck with code that uses the old slow way.

I timed the following techniques for square root:

1. The compiler’s built in sqrt() function (which compiled to the x87 FSQRT opcode)
2. The SSE “scalar single square root” opcode sqrtss, which MSVC emits if you use the _mm_sqrt_ss intrinsic or if you set /arch:SSE2
3. The “magic number” approximation technique invented by Greg Walsh at Ardent Computer and made famous by John Carmack in the Quake III source code.
4. Taking the estimated reciprocal square root of a via the SSE opcode rsqrtss, and multiplying it against a to get the square root via the identity x / √x = √x.
5. Method (4), with one additional step of Newton-Raphson iteration to improve accuracy.
6. Method (5), with the loop at line 13 of the pseudocode above unrolled to process four floats per iteration.

I also tested three ways of getting the reciprocal square root: Carmack’s technique, the rsqrtss SSE op via compiler intrinsic, and rsqrtss with one Newton-Raphson step.

The results, for 4096 loops over 4096 single-precision floats, were:

Discussion

Looking at these results, it’s clear that there’s a dramatic difference in performance between different approaches to performing square root; which one you choose really can have a significant impact on framerate and accuracy. My conclusions are:

Don’t trust the compiler to do the right thing. The received wisdom on performance in math functions is usually “don’t reinvent the wheel; the library and compiler are smart and optimal.” We see here that this is completely wrong, and in fact calling the library sqrt(x) causes the compiler to do exactly the worst possible thing. The compiler’s output for y = sqrt(x); is worse by orders of magnitude compared to any other approach tested here.

The x87 FPU is really very slow. Intel has been trying to deprecate the old x87 FPU instructions for a decade now, but no compiler in the business defaults to using the new, faster SSE scalar opcodes in place of emulating a thirty-year-old 8087. In the case of y = sqrt(x) , by default MSVC and GCC emit something like

fld DWORD PTR [ecx]
fsqrt  ;; slow x87 flop
fstp DWORD PTR [eax]

But if I set the /arch:SSE2 option flag, telling the compiler “assume this code will run on a machine with SSE2″, it will instead emit the following, which is 2x faster.

sqrtss xmm0, DWORD PTR [ecx]  ;; faster SSE scalar flop
movss DWORD PTR [eax], xmm0

There was a time when not every PC on the market had SSE2, meaning that there was some sense in using the older, more backwards-compatible operations, but that time has long since passed. SSE2 was introduced in 2001 with the Pentium 4. No one is ever going to try to play your game on a machine that doesn’t support it. If your customer’s PC has DirectX 9, it has SSE2.

You can beat the hardware. The most surprising thing about these results for me was that it is faster to take a reciprocal square root and multiply it, than it is to use the native sqrt opcode, by an order of magnitude. Even Carmack’s trick, which I had assumed was obsolete in an age of deep pipelines and load-hit-stores, proved faster than the native SSE scalar op. Part of this is that the reciprocal sqrt opcode rsqrtss is an estimate, accurate to twelve bits; but it only takes one step of Newton’s Method to converge that estimate to an accuracy of 24 bits while still being four times faster than the hardware square root opcode.

The question that then bothered me was, why is SSE’s built-in-to-hardware square root opcode slower than synthesizing it out of two other math operations? The first hint came when I tried unrolling the loop so that it performed four ops inside the inner for():

   for ( int i = 0 ; i < NUMITERATIONS ; ++i ) 
      for ( int j = 0 ; j < ARRAYSIZE ; j += 4  ) // in some cases I unroll this loop
      {
         numbersOut[j + 0] = TestedSqrt( numbersIn[j + 0] );
         numbersOut[j + 1] = TestedSqrt( numbersIn[j + 1] ); 
         numbersOut[j + 2] = TestedSqrt( numbersIn[j + 2] ); 
         numbersOut[j + 3] = TestedSqrt( numbersIn[j + 3] );  
      }
 
// two implementations of

As you can see from the results above, when TestedSqrt was the rsqrtss followed by a multiply and one step of Newton iteration, unrolling the loop this way provided a modest 8.8% improvement in speed. But when I tried the same thing with the “precise square root” op sqrtss, the difference was negligible:

SSE sqrt: 200.395 msec
average error 0.0000%

SSE sqrt, unrolled four: 196.741 msec
average error 0.0000%

What this suggests is that unrolling the loop this way allowed the four rsqrt paths to be pipelined, so that while an individual rsqrtss might take 6 cycles to execute before its result was ready, other work could proceed during that time so that the four square root operations in the loop overlapped. On the other hand, the non-estimated sqrtss op apparently cannot be overlapped; one sqrt must finish before the next can begin. A look at the Intel® 64 and IA-32 Architectures Optimization Reference Manual confirms: sqrtss is an unpipelined instruction.

Pipelined operations make a big difference. When the CPU hits an unpipelined instruction, every other instruction in the pipeline has to stop and wait for it to retire before proceeding, so it’s like putting the handbrake on your processor. You can identify nonpipelined operations in appendix C of the Optimization Reference Manual as the ones that have a throughput equal to latency and greater than 4 cycles.

In the case of ssqrt, the processor is probably doing the same thing internally that I’m doing in my “fast” function — taking an estimated reciprocal square root, improving it with Newton’s method, and then multiplying it by the input parameter. Taken all together, this is far too much work to fit into a single execution unit, so the processor stalls until it’s all done. But if you break up the work so that each of those steps is its own instruction, then the CPU can pipeline them all, and get a much higher throughput even if the latency is the same.

Pipeline latency and microcoded instructions are a much bigger deal on the 360 and PS3, whose CPUs don’t reorder operations to hide bubbles; there the benefit from unrolling is much greater, as you’ll see next week.

Conclusion

Not all square root functions are created equal, and writing your own can have very real performance benefits over trusting the compiler to optimize your code for you (at which it fails miserably). In many cases you can trade off some accuracy for a massive increase in speed, but even in those places where you need full accuracy, writing your own function to leverage the rsqrtss op followed by Newton’s method can still give you 32 bits of precision at a 4x-8x improvement over what you will get with the built-in sqrtf() function.

And if you have lots of numbers you need to square root, of course SIMD (rsqrtps) will be faster still.

If you want to run this test on your own hardware, I’ve put my code below the jump. You’ll have to build your own CFastTimer class, but it should be pretty clear what it does — it simply reads out of the CPU clock-cycle counter and computes a difference.

file 1: class definitions header

class TestVector4_Virtual
{
public:
	virtual float GetX() const;
	virtual float SetX( float in );
	virtual float GetY() const;
	virtual float SetY( float in );
	virtual float GetZ() const;
	virtual float SetZ( float in );
	virtual float GetW() const;
	virtual float SetW( float in );
private:
	float x,y,z,w;
};
 
class TestVector4_Direct
{
public:
	__declspec(noinline) float GetX() const;
	__declspec(noinline) float SetX( float in );
	__declspec(noinline) float GetY() const;
	__declspec(noinline) float SetY( float in );
	__declspec(noinline) float GetZ() const;
	__declspec(noinline) float SetZ( float in );
	__declspec(noinline) float GetW() const;
	__declspec(noinline) float SetW( float in );
private:
	float x,y,z,w;
};
 
class TestVector4_Inline
{
public:
	inline float GetX() const;
	inline float SetX( float in );
	inline float GetY() const;
	inline float SetY( float in );
	inline float GetZ() const;
	inline float SetZ( float in );
	inline float GetW() const;
	inline float SetW( float in );
private:
	float x,y,z,w;
};
 
inline float TestVector4_Inline::GetX() const
{
	return x;
}
inline float TestVector4_Inline::SetX( float in )
{
	return x = in;
}
 
/* and so on for GetY, Z, W... */

file 2: class definitions cpp

These functions are defined here to prevent the compiler from inlining them when they’re used.

float TestVector4_Virtual::GetX() const
{
	return x;
}
float TestVector4_Virtual::SetX( float in )
{
	return x = in;
}
/* and so on for y,z,w... */
 
float TestVector4_Direct::GetX() const
{
	return x;
}
float TestVector4_Direct::SetX( float in )
{
	return x = in;
}
/* and so on for y,z,w... */

file 3: test loop

#define ARRAY_SIZE 1024
#define TEST_ITERATIONS 10000
 
template <class T>
void InitWithRandom( T *ptr, int num )
{
	while( num > 0 )
	{
		ptr->SetX( RandomFloat(-1024.f, 1024.0f) );
		ptr->SetY( RandomFloat(-1024.f, 1024.0f) );
		ptr->SetZ( RandomFloat(-1024.f, 1024.0f) );
		ptr->SetW( RandomFloat(-1024.f, 1024.0f) );
		++ptr;
		--num;
	}
}
 
template <class T>
void SumTest( T * RESTRICT in1, T * RESTRICT in2, T * RESTRICT out, const int num )
{
	for ( int i = 0; i < num ; ++i )
	{
		out[i].SetX( in1[i].GetX() + in2[i].GetX() );
		out[i].SetY( in1[i].GetY() + in2[i].GetY() );
		out[i].SetZ( in1[i].GetZ() + in2[i].GetZ() );
		out[i].SetW( in1[i].GetW() + in2[i].GetW() );
	}
}
 
template <class T>
float TestTimings( )
{
	// set up input and output and preheat the cache
	T A[ ARRAY_SIZE ];
	T B[ ARRAY_SIZE ];
	T C[ ARRAY_SIZE ];
 
	InitWithRandom( A , ARRAY_SIZE );
	InitWithRandom( B , ARRAY_SIZE );
	InitWithRandom( C , ARRAY_SIZE );
 
	uint64 retval = 0;
	CFastTimer t1;
	int dontOptimizeThisLoopToNothing = 0;
	for ( int i = 0 ; i < N_ITERS ; ++i )
	{
		t1.Start();
		SumTest( A, B, C, ARRAY_SIZE );
		t1.End();
		dontOptimizeThisLoopToNothing  += i;
		retval += t1.GetClockCycleDelta();
	}
	// force compiler to actually use the data so it doesn't vanish the loop above
	float ac = 0;
	for ( int i = 0 ; i < ARRAY_SIZE ; ++i )
	{
		ac += C[i].GetX();
		ac += C[i].GetY();
		ac += C[i].GetZ();
		ac += C[i].GetW();
	}
	printf( "%f %d\n", ac, dontOptimizeThisLoopToNothing  ); // just ignore these
	return CyclesToMilliseconds(retval) ;
}
 
void RunTest()
{
	// get timings for each type
	float tVirt, tDirect, tInline;
	tVirt = TestTimings< TestVector4_Virtual >();
	tDirect = TestTimings< TestVector4_Direct >();
	tInline = TestTimings< TestVector4_Inline >();
 
	printf( "\n%d iterations over %d vectors\n", TEST_ITERATIONS , ARRAY_SIZE );
	printf( "virtual: %.3f ms\n", tVirt );
	printf( "direct: %.3f ms\n", tDirect );
	printf( "inline: %.3f ms\n", tInline );
}

Assembly output

And, just in case you’re curious, here’s the assembly the compiler generates for the different versions of SumTest:

Direct Function

; Begin code for function: ??$SumTest@VTestVector4_Direct@@@@YAXPIAVTestVector4_Direct@@00H@Z
 
; 58   : {
 
	mflr         r12
	bl           __savegprlr_26
	stfd         fr31,-40h(r1)
	stwu         r1,-90h(r1)
.endprolog
$M89780:
 
; 59   : 	for ( int i = 0; i < num ; ++i )
 
	cmpwi        cr6,r6,0
	ble          cr6,$LN1@SumTest@2
	mr           r31,r4
	subf         r27,r4,r3
	subf         r26,r4,r5
	mr           r28,r6
$LL3@SumTest@2:
 
; 60   : 	{
; 61   : 		out[i].SetX( in1[i].GetX() + in2[i].GetX() );
 
	add          r30,r27,r31
	add          r29,r26,r31
	mr           r3,r30
	bl           ?GetX@TestVector4_Direct@@QBAMXZ
	mr           r3,r31
	fmr          fr31,fr1
	bl           ?GetX@TestVector4_Direct@@QBAMXZ
	mr           r3,r29
	fadds        fr1,fr31,fr1
	bl           ?SetX@TestVector4_Direct@@QAAMM@Z
 
; 62   : 		out[i].SetY( in1[i].GetY() + in2[i].GetY() );
 
	mr           r3,r30
	bl           ?GetY@TestVector4_Direct@@QBAMXZ
	mr           r3,r31
	fmr          fr31,fr1
	bl           ?GetY@TestVector4_Direct@@QBAMXZ
	mr           r3,r29
	fadds        fr1,fr31,fr1
	bl           ?SetY@TestVector4_Direct@@QAAMM@Z
 
; 63   : 		out[i].SetZ( in1[i].GetZ() + in2[i].GetZ() );
 
	mr           r3,r30
	bl           ?GetZ@TestVector4_Direct@@QBAMXZ
	mr           r3,r31
	fmr          fr31,fr1
	bl           ?GetZ@TestVector4_Direct@@QBAMXZ
	mr           r3,r29
	fadds        fr1,fr31,fr1
	bl           ?SetZ@TestVector4_Direct@@QAAMM@Z
 
; 64   : 		out[i].SetW( in1[i].GetW() + in2[i].GetW() );
 
	mr           r3,r30
	bl           ?GetW@TestVector4_Direct@@QBAMXZ
	mr           r3,r31
	fmr          fr31,fr1
	bl           ?GetW@TestVector4_Direct@@QBAMXZ
	mr           r3,r29
	fadds        fr1,fr31,fr1
	bl           ?SetW@TestVector4_Direct@@QAAMM@Z
	addic.       r28,r28,-1		; 0FFFFh
	addi         r31,r31,16		; 10h
	bne          $LL3@SumTest@2
$LN1@SumTest@2:
 
; 65   : 	}
; 66   : }
 
	addi         r1,r1,144		; 90h
	lfd          fr31,-40h(r1)
	b            __restgprlr_26
$M89781:
; End code for function: ??$SumTest@VTestVector4_Direct@@@@YAXPIAVTestVector4_Direct@@00H@Z

Virtual Function

??$SumTest@VTestVector4_Virtual@@@@YAXPIAVTestVector4_Virtual@@00H@Z PROC NEAR ; SumTest<TestVector4_Virtual>, COMDAT
 
; Begin code for function: ??$SumTest@VTestVector4_Virtual@@@@YAXPIAVTestVector4_Virtual@@00H@Z
 
; 58   : {
 
	mflr         r12
	bl           __savegprlr_25
	stfd         fr31,-48h(r1)
	stwu         r1,-0A0h(r1)
.endprolog
$M89754:
 
; 59   : 	for ( int i = 0; i < num ; ++i )
 
	cmpwi        cr6,r6,0
	ble          cr6,$LN1@SumTest
	mr           r31,r4
	subf         r30,r4,r3
	subf         r29,r4,r5
	mr           r26,r6
$LL3@SumTest:
 
; 60   : 	{
; 61   : 		out[i].SetX( in1[i].GetX() + in2[i].GetX() );
 
	lwz          r11,0(r31)
	add          r28,r29,r31
	lwzx         r25,r29,r31
	add          r27,r30,r31
	mr           r3,r31
	lwz          r10,0(r11)
	mtctr        r10
	bctrl
	lwzx         r9,r30,r31
	mr           r3,r27
	fmr          fr31,fr1
	lwz          r8,0(r9)
	mtctr        r8
	bctrl
	lwz          r7,4(r25)
	mr           r3,r28
	fadds        fr1,fr31,fr1
	mtctr        r7
	bctrl
 
; 62   : 		out[i].SetY( in1[i].GetY() + in2[i].GetY() );
 
	lwz          r6,0(r31)
	mr           r3,r31
	lwz          r5,8(r6)
	lwzx         r25,r29,r31
	mtctr        r5
	bctrl
	lwzx         r4,r30,r31
	mr           r3,r27
	fmr          fr31,fr1
	lwz          r11,8(r4)
	mtctr        r11
	bctrl
	lwz          r10,0Ch(r25)
	mr           r3,r28
	fadds        fr1,fr31,fr1
	mtctr        r10
	bctrl
 
; 63   : 		out[i].SetZ( in1[i].GetZ() + in2[i].GetZ() );
 
	lwz          r9,0(r31)
	mr           r3,r31
	lwz          r8,10h(r9)
	lwzx         r25,r29,r31
	mtctr        r8
	bctrl
	lwzx         r7,r30,r31
	mr           r3,r27
	fmr          fr31,fr1
	lwz          r6,10h(r7)
	mtctr        r6
	bctrl
	lwz          r5,14h(r25)
	mr           r3,r28
	fadds        fr1,fr31,fr1
	mtctr        r5
	bctrl
 
; 64   : 		out[i].SetW( in1[i].GetW() + in2[i].GetW() );
 
	lwz          r4,0(r31)
	mr           r3,r31
	lwz          r11,18h(r4)
	lwzx         r25,r29,r31
	mtctr        r11
	bctrl
	lwzx         r10,r30,r31
	fmr          fr31,fr1
	mr           r3,r27
	lwz          r9,18h(r10)
	mtctr        r9
	bctrl
	lwz          r8,1Ch(r25)
	fadds        fr1,fr31,fr1
	mr           r3,r28
	mtctr        r8
	bctrl
	addic.       r26,r26,-1		; 0FFFFh
	addi         r31,r31,20		; 14h
	bne          $LL3@SumTest
$LN1@SumTest:
 
; 65   : 	}
; 66   : }
 
	addi         r1,r1,160		; 0A0h
	lfd          fr31,-48h(r1)
	b            __restgprlr_25
$M89755:
; End code for function: ??$SumTest@VTestVector4_Virtual@@@@YAXPIAVTestVector4_Virtual@@00H@Z

Inlined Function

(notice the use of software pipelining to reduce hazards)

; Begin code for function: ??$SumTest@VTestVector4_Inline@@@@YAXPIAVTestVector4_Inline@@00H@Z
 
; 58   : {
 
	mflr         r12
	bl           __savegprlr_29
	stfd         fr29,-38h(r1)
	stfd         fr30,-30h(r1)
	stfd         fr31,-28h(r1)
.endprolog
$M89879:
 
; 59   : 	for ( int i = 0; i < num ; ++i )
 
	li           r7,0
	cmpwi        cr6,r6,4
	blt          cr6,$LC33@SumTest@3
	addi         r11,r6,-4		; 0FFFCh
	addi         r9,r3,16		; 10h
	srwi         r11,r11,2
	addi         r10,r5,8
	addi         r8,r11,1
	addi         r11,r4,4
 
; 64   : 		out[i].SetW( in1[i].GetW() + in2[i].GetW() );
 
	subf         r31,r4,r3
	subf         r30,r4,r5
	subf         r29,r5,r3
	slwi         r7,r8,2
$LL34@SumTest@3:
	lfs          fr0,-4(r11)
	addic.       r8,r8,-1		; 0FFFFh
	lfs          fr13,-10h(r9)
	lfsx         fr12,r31,r11
	fadds        fr11,fr0,fr13
	lfs          fr10,0(r11)
	fadds        fr8,fr12,fr10
	lfsx         fr9,r10,r29
	lfs          fr7,4(r11)
	lfs          fr6,8(r11)
	fadds        fr5,fr9,fr7
	lfs          fr4,-4(r9)
	lfs          fr3,0Ch(r11)
	fadds        fr2,fr6,fr4
	lfs          fr1,0(r9)
	lfs          fr0,10h(r11)
	fadds        fr13,fr3,fr1
	lfs          fr12,4(r9)
	lfs          fr10,14h(r11)
	fadds        fr9,fr0,fr12
	lfs          fr7,8(r9)
	lfs          fr6,18h(r11)
	fadds        fr4,fr10,fr7
	lfs          fr3,0Ch(r9)
	lfs          fr1,1Ch(r11)
	fadds        fr0,fr6,fr3
	lfs          fr12,10h(r9)
	lfs          fr10,20h(r11)
	fadds        fr7,fr1,fr12
	lfs          fr6,14h(r9)
	lfs          fr3,24h(r11)
	fadds        fr1,fr10,fr6
	lfs          fr12,18h(r9)
	fadds        fr6,fr3,fr12
	lfs          fr3,1Ch(r9)
	lfs          fr10,28h(r11)
	fadds        fr10,fr10,fr3
	lfs          fr3,20h(r9)
	lfs          fr12,2Ch(r11)
	fadds        fr12,fr12,fr3
	lfs          fr31,30h(r11)
	lfs          fr3,24h(r9)
	fadds        fr3,fr31,fr3
	lfs          fr30,34h(r11)
	lfs          fr31,28h(r9)
	fadds        fr31,fr30,fr31
	lfs          fr29,38h(r11)
	lfs          fr30,2Ch(r9)
	addi         r9,r9,64		; 40h
	fadds        fr30,fr29,fr30
	stfs         fr11,-8(r10)
	stfsx        fr8,r30,r11
	addi         r11,r11,64		; 40h
	stfs         fr5,0(r10)
	stfs         fr2,4(r10)
	stfs         fr13,8(r10)
	stfs         fr9,0Ch(r10)
	stfs         fr4,10h(r10)
	stfs         fr0,14h(r10)
	stfs         fr7,18h(r10)
	stfs         fr1,1Ch(r10)
	stfs         fr6,20h(r10)
	stfs         fr10,24h(r10)
	stfs         fr12,28h(r10)
	stfs         fr3,2Ch(r10)
	stfs         fr31,30h(r10)
	stfs         fr30,34h(r10)
	addi         r10,r10,64		; 40h
	bne          $LL34@SumTest@3
$LC33@SumTest@3:
 
; 59   : 	for ( int i = 0; i < num ; ++i )
 
	cmpw         cr6,r7,r6
	bge          cr6,$LN32@SumTest@3
	slwi         r11,r7,4
	subf         r31,r4,r3
	add          r8,r11,r4
	add          r10,r11,r5
	add          r9,r11,r3
	addi         r11,r8,4
	subf         r4,r4,r5
	addi         r10,r10,8
	subf         r5,r5,r3
	subf         r8,r7,r6
$LC3@SumTest@3:
 
; 60   : 	{
; 61   : 		out[i].SetX( in1[i].GetX() + in2[i].GetX() );
 
	lfs          fr0,-4(r11)
	addic.       r8,r8,-1		; 0FFFFh
	lfs          fr13,0(r9)
 
; 62   : 		out[i].SetY( in1[i].GetY() + in2[i].GetY() );
 
	lfsx         fr12,r31,r11
	fadds        fr11,fr0,fr13
	lfs          fr10,0(r11)
 
; 63   : 		out[i].SetZ( in1[i].GetZ() + in2[i].GetZ() );
 
	lfsx         fr9,r10,r5
	fadds        fr8,fr12,fr10
	lfs          fr7,4(r11)
 
; 64   : 		out[i].SetW( in1[i].GetW() + in2[i].GetW() );
 
	lfs          fr6,8(r11)
	fadds        fr5,fr9,fr7
	lfs          fr4,0Ch(r9)
	addi         r9,r9,16		; 10h
	fadds        fr3,fr6,fr4
	stfs         fr11,-8(r10)
	stfsx        fr8,r4,r11
	addi         r11,r11,16		; 10h
	stfs         fr5,0(r10)
	stfs         fr3,4(r10)
	addi         r10,r10,16		; 10h
	bne          $LC3@SumTest@3
$LN32@SumTest@3:
 
; 65   : 	}
; 66   : }
 
	lfd          fr29,-38h(r1)
	lfd          fr30,-30h(r1)
	lfd          fr31,-28h(r1)
	b            __restgprlr_29
$M89880:
; End code for function: ??$SumTest@VTestVector4_Inline@@@@YAXPIAVTestVector4_Inline@@00H@Z

Our beloved Xenon CPU is in-order, however, so I got curious whether that myth is truly busted for us, and as any Mythbuster can tell you, the only way to know is to build it and test!

I’ll talk about the test results first and then try to explain them in a later article. I built a simple 4-dimensional vector class with accessor functions for x,y,z, and w. Then I set up three arrays (A, B, C) each containing 1024 of these classes (so everything fits into the L1 cache) and ran a loop that simply added them together one component at a time.

class Vector4Test {
  float x,y,z,w;
public:
  float GetX() { return x; ]
  float SetX( float x_ ) { return x=x_; }
  // and so on
}
Vector4Test A[1024], B[1024], C[1024];
 
for (int n = 0 ; n = NUM_TESTS ; ++n)
for (int i=0; i < 1024 ; ++i) {
   C[i].SetX( A[i].GetX + () B[i].GetX();
   // and so on for y, z, and w
}

By specifying whether the Get and Set functions are inline, direct, or virtual, it’s easy to compare the overhead of one kind of function call versus another. Each run through the loop would make three function calls per component times four components times 1024 elements in the array for a total of 12,288 function calls. The inline function is essentially the control group since it measures just the cost of the memory accesses, loop conditionals, and floating-point math without any function call overhead at all. Here’s the results:

A couple of things are immediately obvious. First, virtual functions are slower than direct function calls. But by how much? In the upper trial, the virtual-function test took 91.894ms longer than the direct functions; divided by the 12.288×10⁶ function calls, that works out a differential overhead of about 7 nanoseconds. So, there is a definite cost there, but probably not something to worry about unless it’s a function that gets called thousands of times per frame.

Later I’ll get further into the causes of these disparities, why virtual functions are slower than direct calls, and when inlining is advantageous. In the meantime I can tell you for sure that the problem is not the cost of looking up the indirect function pointer from the vtable — that’s only a single unproblematic load operation. Rather the issues lie in branch prediction and the way that marshalling parameters for the calling convention can get in the way of good instruction scheduling.

• By default it will call a function _ftol2_sse, which tests the CPU to see if it has SSE2 functionality. If so, it uses the native SSE2 instruction cvttsd2si. If not, it calls _ftol(). This is quite slow because it has to perform that CPU test for every single conversion, and because there is that overhead of a function call.
• With /QIfist specified, the compiler simply emits a fistp opcode to convert the x87 floating point register to an integer in memory directly. It uses whatever rounding mode happens to be set in the CPU at the moment.
• With /arch:SSE2 specified, the compiler assumes that the program will only run on CPUs with SSE2, so it emits the cvttsd2si opcode directly instead of calling _ftol2_sse. Like /QIfist, this replaces a function call with a single instruction, but it’s even faster and not deprecated. As commenter cb points out, the intrinsics also let you specify truncation or rounding without having to fool around with CPU modes.

I raced the different techniques against each other and the clear winner was the function compiled with /arch:SSE2 set. Thus, if you can assume that your customer will have a CPU with SSE2 enabled, setting that simple compiler switch will provide you with superior performance for basically no work. The only caveat is that the SSE scalar operations operate at a maximum of double-precision floats, whereas the old x87 FPU instructions are internally 80-bit — but I’ve never seen a game application where that level of precision makes a difference.

According to the Steam Hardware Survey, 95% of our customers have SSE2-capable CPUs. The rest are probably not playing your most recent releases anyway.

This particular performance suck has two major sources: register shuffling and rounding. The first one has to do with hardware and affects all modern CPUs; the second is older and more specific to x86 compilers.

In both cases, one simple rule holds true: whenever you find yourself typing *((int *)(&anything)), you’re in for some pain.

Register Sets: The Multiple-Personality CPU

Casting floats to ints and back is something that can happen in routine tasks, like

int a = GetSystemTimeAsInt(); 
float seconds = a;

Or occasionally you may be tempted to perform bit-twiddling operations on a floating point number; for example, this might seem like a fast way to determine whether a float is positive or negative:

bool IsFloatPositive(float f)
{
    int signBit = *((int *)(&f)) & 0x80000000;
    return signBit == 0;
}

The problem here is that casting from one register type to another like this is an almost sure way to induce a load-hit-store. In the x86 and the PPC, integers, floating-point numbers, and SIMD vectors are kept in three separate register sets. As Becky Heineman wrote in Gamasutra, you can “think of the PowerPC as three completely separate CPUs, each with its own instruction set, register set, and ways of performing operations on the data.”

Integer operations (like add, sub, and bitwise ops) work only on integer registers, floating-point operations (like fadd, fmul, fsqrt) only on the FPU registers, and SIMD ops only touch vector registers. This is true of both the PowerPC and the x86 and nearly every modern CPU (with one exception, see below).

This makes the CPU designer’s job much easier (because each of these units can have a pipeline of different depth), but it means there is no means of directly moving data from one kind of register to another. There is no “move” operation that has one integer and one float operand. Basically, there are simply no wires that run directly between the int, float, and vector registers.

So, whenever you move data from an int to a float, the CPU first stores the integer from the int register to memory, and then in the next instruction reads from that memory address into the float register. This is the very definition of a load-hit-store stall, because that first store may take as many as 40 cycles to make it all the way out to the L1 cache, and the subsequent load can’t proceed until it has finished. On an in-order processor like the 360 or the PS3’s PPC, that means everything stops dead for between 40 and 80 cycles; on an out-of-order x86, the CPU will try to skip ahead to some of the subsequent instructions, but can usually only hide a little bit of that latency.

It’s not the actual conversion of ints to floats that is slow — const int *pA; float f = *pA; can happen in two cycles if the contents of pA are already in memory — but moving data between the different kinds of registers that is slow because the data has to get to and from the memory first.

What this all boils down to is that you should simply avoid mixing ints, floats, and vectors in the same calculation. So, for example, instead of

struct {int a,b} Foo;
float func( Foo *data )
{
  float x = fsqrt((( data->a << 1 ) - data->b));
  return x;
}

you are really better off with

float func( Foo *data )
{
  float fA = data->a;
  float fB = data->b;
  float x = fsqrt((( fA * 2.0f ) - fB ));
  return x;
}

More importantly, if you have wrapped your native SIMD type in a union like

union {
  __vector V;  // native 128-bit VMX register
  struct { float x,y,z,w; }
} vec4;

then you really need to avoid accessing the individual float members after working with it as a vector. Never do this:

vec4 A,B,C;
C = VectorCrossProductInVMX(A, B);
float x = C.y * A.x * 2.0f;
return x;

There is one notable exception: the Cell processor SPU as found in the PS3. In the SPUs, all operations — integers, floats, and vectors — operate on the same set of registers and you can mix them as much as you like.

Now, to put this in context, 80 cycles isn’t the end of the world. If you’re performing a hundred float-to-int casts per frame in high level functions, it’ll amount to less than a microsecond. On the other hand it only takes 20,000 such casts to eat up 5% of your frame on the 360, so if this is the sort of thing you’re doing in, say, AccumulateSoundBuffersByAddingTogetherEverySample(), it may be something to look at.

Rounding: Say Hello To My Little fist

In times gone by one of the easiest performance speedups to be had on the PC and its x86-based architecture was usually found in the way your program rounded floats.

The most obvious and least hardware-specific cost of int a = float b is that you somehow have to get rid of the fractional part of the number to turn it into a whole integer; in other words, the CPU has to turn 3.14159 into 3 without involving the Indiana State Legislature. That’s simple enough, but what if the number is 3.5 — do you round it up, or down? How about -3.5 — up or down? And how about on x86-based chips, where floating point numbers are calculated inside the CPU as 80 bits but an int is 32 bits?

At the time the Intel x87 floating-point coprocessor was invented, the IEEE 754 floating point standard specified that rounding could happen in one of four ways:

1. Round to nearest – rounds to the nearest value; if the number falls midway it is rounded to the nearest even number
   ( 3.3 → 3 , 3.5 → 4, -2.5 → -2 )
2. Round to zero – also known as truncate, simply throws away everything after the decimal point
   ( 3.3 → 3 , 3.5 → 3, -2.5 → -2 )
3. Round up –
   ( 3.3 → 4 , 3.5 → 4, -2.5 → -2 )
4. Round down –
   ( 3.3 → 3 , 3.5 → 3, -2.5 → -3 ).

The x87 allows you to select any of these modes by setting or clearing a couple of bits in a special control register. Reading and writing that register is a very slow operation, because it means the processor has to totally throw away anything that came behind it in the pipeline and start over, so it’s best to change modes as little as possible, or not at all.

The actual rounding operation can be done in one instruction (the amusingly named fist op, which means “float-to-int store”), but there’s a snag. The ANSI C standard decrees that one and only one of these modes may ever be used for int a = float b: truncate. But, because the compiler can never be sure what rounding mode is set when you enter any particular function (you might have called into some library that set it differently), it would call a function called _ftol(), which set this mode each and every time a number was rounded. In fact, what it actually did for every cast was:

1. Call into _ftol()
2. Check the old rounding mode and save it to memory
3. Set the rounding mode to “truncate” (this causes a pipeline clear)
4. Round the number
5. Set the rounding mode back to the one it saved in step one (another pipeline clear)
6. Return from _ftol()

Because of this it wasn’t unusual to see a game spending over 6% of its time inside _ftol() alone. (In fact I can say with a straight face that I once saw a profile where a game spent fully 8% of each frame on fisting.) This is an extreme case of the compiler choosing correctness over speed.

You’re thinking the answer is “well, how about I just set the rounding mode to start with and tell the compiler not to obsess so much about the exact correctness?” and you’re right. The solution in MSVC is to supply the /QIfist compiler option, which tells the compiler to assume the current rounding mode is correct and simply issue the hardware float-to-int op directly. This saves you the function call and two pipeline clears. If your rounding mode gets changed elsewhere in the program you might get unexpected results, but… you know.. don’t do that.

Microsoft’s documentation claims that /QIfist is “deprecated” due to their floating-point code being much faster now, but if you try it out you’ll see they’re fibbing. What happens now is that they call to _ftol2_sse() which uses the modeless SSE conversion op cvttsd2si instead of old _ftol(). This has some advantages — you can pick between truncation and rounding for each operation without having to change the CPU’s rounding mode — it’s still a needless function call where an opcode would do, and it shuffles data between the FPU and SSE registers which brings us back to the LHS issue mentioned above. On my Intel Core2 PC, a simple test of calling the function below is twice as fast with compiler options /fp:fast /QIfist specified compared with only /fp:fast.

void test1(volatile int *a, volatile float *f)
{
  for (int i=0; i < 1000000 ; ++i)
    *a = (int) *f;
}

On the other hand, in an absolute sense _ftol2_sse() is pretty fast so it may be good enough.

It’s also possible to convert floats to ints by adding them to a certain magic number, but this isn’t always a benefit. In times of yore the fistp op was slow, so there was an advantage to replacing the fist with a fadd, but this doesn’t seem to be the case any more. It is faster than an implicit call to _ftol2_sse, and it has the advantage of not depending on the CPU’s current rounding mode (since you can pick your magic number to choose between rounding and truncation). On the other hand if you’ve specified /arch:sse2 and the compiler is using SSE scalar operations instead of the x87 FPU generally, then it’s faster to let it use the native cvttss2si op.

On the 360/PS3 CPU, the magic number technique is usually a performance hit, because most of the magic-number tricks involve an integer step on the floating-point number and run into register-partitioning issue mentioned above.

Further Reading

• My primary source for the register-type load-hit-store issue was the XDK documentation; consult your own for more details.
• David Goldberg’s What Every Computer Scientist Should Know About Floating-Point Arithmetic is the definitive survival guide for the IEEE754 wilds.
• You can learn more about the “magic integer” technique for converting between floats and ints here and here, and I also found an interesting article about the state of affairs circa 2000.

The Slow Tree Is Full Of Branches

In opcode parlance, any instruction that causes the CPU to skip from one instruction to another is called a branch. If the jump occurs based on some condition being met, like in an if ( x == y ) { doSomething() ; } block, it’s called a conditional branch: in this case, the program branches to either run doSomething() or not depending on the result of a “compare x to y” operation.

For various reasons, a branch instruction can be pretty slow to execute. The important one here is that modern CPUs process their instructions in many steps along a pipeline. Like parts moving along a conveyor belt in a factory, a CPU have as many as eighty program instructions in flight at once, each of them being processed a little bit more at each of the eighty steps along the assembly line until it is finished and the final results written back to memory.

When a conditional branch instruction executes, the CPU can’t know which instructions are supposed to come after the branch on the assembly line. Usually instructions are executed in sequence, so that opcode #1000 is followed by #1001 and that by #1002 and so on, but if #1002 turns out to be a branch to instruction #2000, then any work the compiler might have done on instructions #1003-#1010 at earlier stages in the pipeline has to be thrown away. Modern CPUs and compilers have gotten quite good at mitigating this effect through branch prediction but in some cases, particularly involving floating-point numbers, branch prediction may be impossible in a way that causes the entire assembly line to be cleared out and restarted.

To understand this, first consider this pseudoassembly for how we could compile x = (a >= 0 ? b : c) using a compare followed by a branch instruction:

// assume four registers named ra, rb, rc, rx
// where rx is the output
// and r0 contains zero
COMPARE ra, r0 // sets "ge" condition bit to 1 if ra >= 0
JUMPGE cbranch // executes a GOTO cbranch: if the "ge" bit is 1
MOVE rx, rb    // sets rx = rb
JUMP finish
cbranch:
MOVE rx, rc    // sets rx = rc
finish:
RETURN rx

There are a couple of ways that this can slow down the instruction pipeline. Firstly, it always executes at least one branch operation, which is expensive: on the PowerPC such a branch can stall execution for between five and twenty cycles depending on whether it is predicted correctly. Also, the conditional jump occurs immediately after the COMPARE operation, which can lead to another stall if the result of the COMPARE isn’t ready within a single cycle.

But with floating-point compares the situation can be much worse because float pipelines are often much longer than the integer/branch pipeline. This means that the result of the COMPARE may not be available to the branch unit for many cycles after it dispatches. When the CPU tries to branch on a result that isn’t available yet, it has no choice but to flush the entire pipeline, wait for the compare to finish and start over. For a modern 40-cycle long pipeline this can be quite costly indeed! For example, consider this simplified pipeline of a hypothetical machine.

The fetch stage takes six cycles, then instructions are dispatched after a three cycle delay to either an integer, branch, or float pipeline.

a float-compare enters the fetch stage, immediately followed by a branch-on-greater-or-equal

The fcmp instruction begins to execute, but its results won’t be ready until it leaves the eight-stage float pipeline. So, the branch instruction immediately following it can’t execute yet.

The fcmp begins to execute, but the branch pipeline can't evaluate until the results are ready.

It (and all the instructions behind it) gets flushed from the pipeline and has to start over from the beginning.

The fcmp continues to execute, while everything behind it in the pipeline is flushed altogether.

Once the fcmp is done, the branch is allowed to enter the pipe again,

Branch instruction starts over while fcmp executes.

and then finally executes some time later. Even though the branch immediately follows the fcmp in the program, it doesn’t actually get executed until over twenty cycles later!

Branch instruction finally executes after fcmp completes.

To solve all of these issues and others, hardware designers have invented instructions that can perform all this work in a single branchless operation. Unfortunately, compilers aren’t always smart enough to use them, so sometimes you need to do a little work yourself.

The conditional move

Because branches can be slow, many architectures implement a means of selecting between two different values based on a third without having to actually execute a comparison and jump operation. This is usually called a “conditional move” or “branchless select” and expresses the operation: if a ≥ 0 then b else c; in C++ that is

// int a, b, c;
int x = a >= 0 ? b : c ;

( if (>= a 0) b c )

There are numerous reasons why this is useful but from my point of view the best is improving performance by eliminating pipeline bubbles. Unlike an if/else implemented as a branch op, a conditional move doesn’t change the flow of the program or cause it to jump from one instruction or another; it simply assigns either value a or b to the contents of register x in a single instruction. Thus, it can never cause the pipeline to clear or invalidate any of the instructions that come after it.

The floating-point conditional move operator on the PPC is called fsel, which works like:

fsel f0, f1, f2, f3 // f0 = ( f1 >= 0 ? f2 : f3 )

The most recent version of the compiler I use is pretty good at compiling this automatically for cases like this:

and it even does the right thing with this:

// float a, b, c, d, e, f;
return ( a >= 0 ? b + 1 : c + 2 ) +
   ( d >= 0 ? e + 1 : f + 2 ) ;

; fr1 = a, fr2 = b, fr3 = c,
; fr4 = d, fr5 = e, fr6 = f
; fr0 = 1.0, fr13 = 2.0f
 fadds   fr12,fr2,fr0      ; fr12 = a + 1
 fadds   fr11,fr3,fr13     ; fr11 = c + 2
 fadds   fr10,fr5,fr0      ; fr10 = e + 1
 fadds   fr9,fr6,fr13      ; fr9  = f + 2
 fsel    fr8,fr1,fr12,fr11 ; fr8  = a >= 0 ? fr12 : fr11
 fsel    fr7,fr4,fr10,fr9  ; fr7  = d >= 0 ? fr10 : fr9
 fadds   fr1,fr8,fr7     ; return = fr8 + fr7

but it has trouble with more complicated scenarios like this:

; fr1 = a, fr2 = b, fr3 = c, fr4 = d
fcmpu        cr6,fr1,fr2  ; compare fr1 and fr2
blt          cr6,$LN3     ; if compare result was
                          ; "less than", GOTO $LN3
fmr          fr1,fr3      ; move fr3 to fr1
blr                       ; return
$LN3:
fmr          fr1,fr4      ; move fr4 to fr1
blr                       ; return

and if you use an if/else block instead of the ternary operator, all bets are off.

So, for these more complex cases, the compiler exposes as an intrinsic with a prototype something like:

float fsel( float a, float x, float y );
// equivalent to { return a >= 0 ? x : y ; }

A compiler intrinsic looks like an ordinary C++ function but compiles directly to the native hardware instruction, providing you with direct access to the opcode without having to actually write assembly. The caveat is that the hardware fcmp can only perform a ≥ 0 comparison (on the PPC anyway), so for complex conditionals like

if ( a / 2 < b + 3 ) { return c; } else { return d; }

you need to do a little algebra to transform the inequality a / 2 < b + 3 into a - 2b - 6 < 0 and then, because x < 0 ≡ ¬(x ? 0), flip the order of the operands to get return _fsel( a - 2*b - 6, d, c );. Once you’ve got everything fed into the intrinsic, the compiler can do a good job of scheduling your other instructions.

Life beyond if

The advantage of conditional moves comes when you use them to eliminate branching from a complex mathematical operation altogether. The most obvious cases are something like the good old min( float a, float b ) { return a <= b ? a : b } which becomes return _fsel( b – a, a, b ); which means you get through something wacky like x += min(a,b) * max(c,d) without any branches at all.

Because math is costly, it may seem like a good idea to use an if statement to prevent unnecessary calculations from being performed, but this can be the wrong thing to do. Sometimes using a floating-point branch to early-out of some calculations may actually hurt performance. In some cases, the cost of the branch may be much greater than the cost of the calculation itself, and so if you’re using if/else to choose between performing one of two different calculations, it may be better to do both calculations and use fsel to choose between them afterwards.

Often the compiler can interleave the calculations so the cost of both is the same as doing only one, and you avoid the pipeline clear caused by the fcmp. For example, this code:

float out[N], inA[N], inB[N], cond[N];
for (int i = 0 ; i < N ; ++i )
{
  if ( cond[i] >= 0 )
    {     out[i] = cond[i] + inA[i];   }
  else
    {     out[i] = cond[i] + inB[i];   }
}

can be turned into:

for (int i = 0 ; i < N ; ++i )
{
    out[i] = fsel( cond[i], cond[i] + inA[i], cond[i] + inB[i] );
}

In the top version, we choose to do one of two additions based on the sign of cond[i]. In the bottom version, we perform two additions and throw away one result, but even so it is much faster! When I tested 200,000,000 iterations of the above code, the if/else version took 5.243 seconds compared with 0.724 seconds for the fsel version: a 7x speedup for not avoiding an addition!

Another example is if you have a large quantity of such assignments to do one after another, like
struct CFoo { float x, y, z, w; }
// CFoo out, A, B
// float cond[4]; (some number that acts as a selector)
out.x += fsel( cond[0], A.x, B.x );
out.y += fsel( cond[1], A.y, B.y );
out.z += fsel( cond[2], A.z, B.z );
out.w += fsel( cond[3], A.w, B.w );

In this case, because all of the fsel operations are independent (the result of one line does not depend on the result of another), they can all be scheduled to occur immediately after one another for a throughput of one per cycle. Had if/else been used instead (eg if ( cond[0] >= 0 ) out.x += A.x; else out.x += B.x;), then each branch would have depended on the result of the preceding comparison, meaning that we would need to pay the full 60-cycle fcmp/branch penalty for each line before starting the next one.

By now you’re probably looking at the example above and thinking, “for four floats packed together like that, I could use a SIMD instruction to compare and assign them all at once,” and if so you’re absolutely right. Both VMX (on the PowerPC) and SSE (on the x86) also have conditional move instructions that work very similarly to the scalar float operations I’ve described above. Consult your documentation for the exact semantics, but in general they work by performing a comparison operation to create a mask which is then fed into a subsequent “select” operation.

The general principle is that it can be better to perform both arms of a calculation and then select between them afterwards than to use an if/else to perform only one arm of the calculation. This is even more true for SIMD units since the pipelines are often deeper.

Integer Conditional Move

All the discussion above has focused on floating-point conditional move operations. The x86 platform offers an analogous integer conditional move, called CMOV, but some PowerPC implementations lack this native opcode. Instead, you can manufacture one yourself by creating a mask and then using a bitwise-and to select between two values.

The key is to remember that for any signed int a < 0, the leftmost (most significant) bit of a will be 1, and that the C++ arithmetic-shift operator >> always preserves the sign of the word you are shifting. So, int mask = a >> 31; means the same as int mask = (a < 0 ? -1 : 0).

Once you have your mask, you can combine it with a bitwise-and and an add to make your conditional move. Given that x + ( y – x ) = y, you can generate your integer select function like so:

// if a >= 0, return x, else y
int isel( int a, int x, int y )
{
    int mask = a >> 31; // arithmetic shift right, splat out the sign bit
    // mask is 0xFFFFFFFF if (a < 0) and 0x00 otherwise.
    return x + ((y - x) & mask);
};

The assembly for this works out to about four instructions. The performance gain usually isn’t as great as with fsel, because the integer pipeline tends to be much shorter and so there’s no pipeline clear on an integer compared followed by a branch; however this construction does give the compiler more latitude in scheduling instructions, and lets you avoid the innate prediction penalty that always occurs with any branch.

In addition to this most PowerPCs have a “count leading zeroes” instruction that lets a smart compiler do something really cool with return a + ( (b - a) & ((Q == W) ? 0 : -1) );. Try it out in your disassembler and see!

Further Reading

Mike Acton writes extensively about branch elimination at his CellPerformance blog. In addition to fsel specifically, he describes many other techniques of branchless programming.

• Escape Pod’s short-fiction podcast has not one but two Christmas specials.
• X Minus One, The Lifeboat Mutiny. 50s-era science fiction radio play: two prospectors find themselves in possession of a lifeboat that’s more than it seems. Worth listening to for the Drone National Anthem alone!
• G.K. Chesterton’s  The Wisdom Of Father Brown is available as a public-domain audio book from LibriVox, read by Martin Clifton. I found this collection of short stories about the eponymous priest-turned-private-detective to be fine treadmill listening.
• WNYC’s science documentary special Radio Lab is always a singular pleasure.
• And Subterranean Press was kind enough to produce a free audio book of Charlie Stross’ Trunk and Disorderly, a humorous adventure in a style that defies description (the closest anyone’s come is “Heinlein meets Wodehouse in space with a woolly mammoth”).

I’ll be back next week with some thoughts on replacing floating-point conditionals with branchless math.

Game applications cannot leak memory. If an application leaks even a small amount of memory, it eventually dies.

I say preach it, brother. I can think of a couple of titles where memory leaks actually hurt a game’s sales. In fact I’ve been known to blame reckless use of new for the death of entire studios. malloc() — there’s no evil too great to lay at its feet.

Paul’s sentence comes in the context of a group of justifications for EA building its own Standard Template Library, which are so concisely spot-on that I’ll reproduce them here. Call them Paul’s Commandments:

• No matter how powerful any game computer ever gets, it will never have any free memory or CPU cycles.
• Game developers are very concerned about software performance and software development practices.
• Game software often doesn’t use conventional synchronous disk IO such as <stdio.h> or <fstream> but uses asynchronous IO.
• Game applications cannot leak memory. If an application leaks even a small amount of memory, it eventually dies.
• Every byte of allocated memory must be accounted for and trackable. This is partly to assist in leak detection but is also to enforce budgeting.
• Game software rarely uses system-provided heaps but uses custom heaps instead.
• A lot of effort is expended in reducing memory fragmentation.
• A lot of effort is expended in creating memory analysis tools and debugging heaps.
• A lot of effort is expended in improving source and data build times.
• Application code and libraries cannot be very slow in debug builds.
• Memory allocation of any type is avoided to the extent possible.
• Operator new overrides (class and global) are the rule and not the exception.
• Use of built-in global operator new is verboten, at least with shareable libraries.
• Any memory a library allocates must be controllable by the user.
• Game software must be savvy to non-default memory alignment requirements.
• Memory pools are sometimes used in order to avoid fragmentation, even though they necessarily waste some memory themselves.
• Branching (if/else/while/for/do) is avoided to the extent possible, especially mispredicted branches.
• Virtual functions are avoided to the extent possible, especially in bottleneck code.
• Exception handling is usually disabled.
  
• RTTI is usually disabled or at least unused in shipping code.

You can find the whole article here; it’s a worthwhile read. If you’re in a hurry, you can safely skip the large tables; the meat of the article is in “Motivation” and the sections following “Game Software Issues“.

Whenever I need to edit my %PATH% (which is often) I find myself copying the whole thing out of the Windows dialog, pasting it into Notepad, editing it there, and then copying-and-pasting back into the Windows dialog.

Enter the freeware Rapid Environment Editor, a sweet little tool that is infinitely better than the Windows builtin for editing %PATH% and all your other envvars without pain. Long semicolon-seperated environment variables are displayed as drop-down trees (like my PATH below), and it even highlights broken folders with red text!

It’s a quick 800kb download and the best thing to happen to me today.