HoloLens has been detailed at the Hot Chips conference in Cupertino, with Microsoft teasing that there's a "holographic processing unit", or HPU inside of HoloLens, that is capable of around 1 trillion operations per second.
TSMC build the "Tensilica" chip for HoloLens on the 28nm process, with 65 million logic gates, 8MB of ultra-fast static RAM (SRAM), and 1GB of DDR3 RAM. Microsoft managed to fit all of this into a 12mm x 12mm (half-inch square) package that's designed to process input from the inertial chip, environmental cameras, depth cameras, and other sensors on HoloLens.
HoloLens is capable of speeding up calculations 200 times over software alone, with ready-to-use positional data to the headset's main Intel Cherry Trail-based CPU, so that it's free to run apps and games. It makes sense now why the headset is $3000.