Adaptec by PMC ASR-72405 Enterprise RAID Controller Review

Introduction

30

Adaptec by PMC is hoping to ride a wave of resurgence into the enterprise RAID controller market with the recently launched Series 7. At the heart of this new series of RAID controllers is the radical new processor developed by PMC-Sierra, the PM8015, which boasts speeds of 6,600MB/s and 450,000 random read IOPS.

PMC-Sierra, a large semiconductor innovator, acquired Adaptec in 2010. Developed prior to the acquisition, the Adaptec 6 Series controllers were released shortly after the PMC merger. The Series 7 RAID controllers we are testing today are the first from Adaptec to leverage the benefits of the merger, with PMC-Sierra's newest ROC (RAID On Chip) providing a drastic increase in power.

The backbone of the datacenter is constructed around slow spinning HDDs and has not evolved nearly as quickly as the server platform. In the past, this has not required vast advances in storage controller technology and related infrastructure. For years the pace of innovation has been merely evolutionary in the storage controller market.

SSDs have burst into the datacenter, and this disruptive technology creates new challenges. As the sophistication of flash-based devices increases, and prices plunge, and suddenly there is an intense need for faster storage systems as SSDs stress existing infrastructure.

In 2011 SSDs provided 16GB of storage per cubic inch, compared to 1GB per inch for HDDs. HDDs also supply 4.2 IOPS per cubic inch of space, dwarfed by the 1,250 IOPS provided by SSDs. This calls for revolutionary new products to deliver not only the performance to handle mass deployments of exponentially faster devices, but products that also mesh well with the increased density enabled by these solutions. With smaller 2U servers featuring up to 24 x 2.5" SAS/SATA bays, the ability to pack as much connectivity into the smallest possible solution is imperative.

30

The PM8015 is a revolutionary approach to RAID controller design and integration. This new controller, engineered by PMC, sports 24 native SAS 6Gb/s ports. This is a radical departure from the current crop of storage processors, which feature only 8 ports. This tripling of native ports allows administrators to connect more devices to the RAID controller without the use of expanders.

Adaptec paired this powerful controller with the new Mini-SAS HD (High Density) cables. This new form of RAID controller interconnects is the only solution that can provide the required number of connections for taking advantage of the ROC's 24 available ports. This combination provides a revolutionary new controller with advanced cabling provides extreme performance, a slim form factor, and enhanced density that no other product currently on the market can match.

Series 7 Architecture and Specifications

Series 7 Architecture

30

One of the immediate goals for the engineers at PMC-Sierra was to create a controller to saturate the PCIe 3.0 connection. PCIe Gen3 offers a tremendous amount of bandwidth and saturating that is simply not possible with existing solutions. Typical RAID architectures consist of 8 x 6Gb/s SAS ports. This provides a limited pathway for the controller, topping out at 4.8 GB/s of useable bandwidth.

The PM8015 ROC features 24 x 6Gb/s SAS native ports combined with the x8 PCIe Gen3 interface. 16 ports alone can saturate the bus by providing 9.8 GB/s, well above the 8 GB/s supplied by PCIe Gen3. This allows Adaptec to pack as many lanes on board a small form-factor card as possible. The resulting Low Profile/MD-2 ASR-71605 provides an unmatched 16 ports of SAS/SATA connectivity in the half-height category, and the ASR-71405 that we are testing today features an unmatched 24 native SAS/SATA ports in the full-height half-length category.

Specifications

30

One great feature of the ASR-72405 that we were repeatedly thankful for during testing is the ability to switch the controller into HBA mode. This functionality allows users to switch the adapter from RAID to HBA mode. After a reboot, the attached devices are presented to the host directly. In terms of device management, this is extremely useful.

The Adaptec Series 7 controllers boast impressive specifications. With a RAID 0 configuration with 24x SSDs, Adaptec observed 534,000 IOPS with a 4k random read workload. The maximum 4k write IOPS recorded with the same configuration totaled 374,000 IOPS. The maximum 1MB sequential read speed weighed in at 6,400 MB/s and sequential write speed was clocked of 5229 MB/s. With the 512B file size sequential data transfers topped 1.02 million IOPS, and the write speed topped out at 1.07 million IOPS.

With a RAID 5 configuration, the Series 7 adapters perform at up to 6.6 GB/s with sequential read and 2.6 GB/s in sequential write. With 16 direct connected SSDs they delivered 450,000 IOPS.

Adaptec RAID Code (ARC) delivers RAID levels 0, 1, 1E, 5, 6, 10, 50 and 60 (71605E supports 0, 1, 1E, and 10, only). ARC also offers RAID Level Migration (the ability to easily migrate RAID levels), Online Capacity Expansion (expand capacity without powering down the server), and Copyback Hot Spare (when a failed drive has been replaced, data is automatically copied from the hot spare back to the restored drive).

Adaptec's Intelligent Power Management techniques can purportedly reduce storage power consumption, and thus cost, by up to 70%. This is enabled by shifting the power for the connected devices into one of three modes; full on, low power and spin down.

The controllers are rated for 700,000 hours MTBF at 40C, though it is noted that they are still undergoing testing, and this number may increase. The adapter carries a standard three year warranty.

Expanders and maxView Storage Manager

Expanders

IDC estimates that the cost of operating a piece of IT equipment over a four year period can be up to four times greater than the initial cost of acquiring the product. This is a sobering statistic for administrators evaluating the cost of deploying a solution into an enterprise environment.

Expanders add another layer of hidden costs by incurring additional OpEX to the storage solution. Internal and external SAS expanders add upfront costs to the purchase, require power to function, and generate heat. SAS Expanders also require more cabling and take up space, effectively reducing the amount of storage that can be crammed directly into the server chassis. Building expanders directly onto the PCB of the RAID controller also increases cost and power consumption.

The Series 7 controllers will also work with expanders to provide up to 256 devices, but the option of forgoing internal and external expanders in many applications is an attractive feature that reduces the overall footprint.

Even more concerning is that SAS expanders incur latency and bandwidth penalties, hampering the performance of the solution. Largely overlooked in the past, with HDDs simply not providing enough speed, the performance limitations of expanders have remained hidden. With the changing dynamics of SSD accelerated infrastructure, these restrictions are becoming more visible.

Performance challenges can be partially overcome by leveraging dual-port SAS devices through an expander. However, this limits the selection of devices, and there is still an expander adding to the footprint and complexity of the deployment.

maxView Storage Manager

30

There are a number of ways to manage the ASR-72405, but we found the maxView Storage Manager GUI to be a lightweight user-friendly application that provides administrators with all of the functionality needed to manage the adapter. The web-based GUI can be viewed in standard desktop browsers without the need for a separate software installation. The utility features a wide range of compatibility and allows for management of direct attached or networked storage. The manger also provides full monitoring of an installed supercap and the AFM status.

The maxView Storage Manager provides advanced monitoring statistics along with SMART and temperature data. The documentation for the program is thorough and easily accessible from within the application. The only feature that we feel could be improved upon is a status monitor for ongoing tasks, such as RAID initialization and rebuilds, and a means of initializing multiple drives simultaneously.

30

The Series 7 adapters allow three configuration options for attached volumes with the addition of FlexConfig. This aids ease of configurability, with the option for automatic setup for both physical and logical volumes.

Physical drive mode: This configures the drive in a pass-through configuration, much like an HBA. There will be no addition of metadata to the drive and no use of the DRAM caching.

Simple volume mode: This allows the drive to utilize the DRAM caching and configures metadata onto the drive. This value added proposition gives HBA functionality with DRAM caching and the protection of supercaps (if attached).

RAID: Provides the full set of options for manually configuring all RAID parameters.

ZMCP and MaxCache 3.0

ZMCP

ZMCP 3.0 (Zero-Maintenance Cache Protection) is a unique new take on protecting data stored in the volatile cache of the controller during a power outage. This approach allows RAID controllers to function with write back caching without endangering data. Write back caching provides the maximum performance for connected drives. Li-Ion (Lithium Ion) batteries have several disadvantages, and the AFM-700 ZMCP solution from Adaptec looks to change the utilization of cache protection systems.

30

The black container that forms the 'battery' of the solution is actually a super capacitor connected to a PCB containing 4GB of SLC NAND flash. In the event of a power failure, the super capacitor will provide enough power for the controller to flush the contents of the DRAM to the attached SLC NAND. SLC is employed due to its endurance, resistance to heat and very fast write access time.

A benefit of using non-volatile NAND to store the flushed data is that when data is committed to the NAND, it can be stored for years without power. SLC typically has data retention of 10 years without power. Upon power restoration, the data is copied back from the NAND to the array and the adapter functions as normal. Li-Ion solutions merely keep the DRAM onboard the controller alive with power, preserving the data in the volatile DRAM until the battery discharges entirely, usually within 72 hours.

Li-Ion batteries also require hours to charge when installed, and this leads to compromised performance due to adapters switching out of write back mode until the battery is charged. Another unfortunate characteristic of Li-Ion batteries is that they require discharging to determine the life expectancy of the battery. The performance of the adapter is compromised during this cycle. While discharging and recharging the adapter switches back to write thru mode.

The AFM-700 super capacitor takes a few minutes to charge during boot up, and then does not require powering down for monitoring. The super capacitor is monitored from the MaxView Storage Manager without disrupting operations or impacting performance. The AFM-700 also reports its temperature, allowing administrators to monitor the server temperature remotely.

With a longer lifespan (five years at 50C) and more durability in high heat scenarios than Li-Ion solutions, the AFM-700 modules will cut OpEX and TCO requirements significantly.

maxCache 3.0

30

maxCache 3.0 provides users with a means of caching HDD pools with faster SSDs, expanding the performance of the underlying array exponentially. This latest version of maxCache supports both read and write caching. Adaptec has also expanded maxCache to introduce protection from SSD failure via redundancy with RAID 1E and RAID 5.

30

The new ODU (Optimized Disk Usage) approach of forming a cache pool allows for more flexibility when configuring the attached flash. This approach will help to maximize the investment in flash-based devices by also allowing the pool to have usable volumes, or even boot volumes. This could also allow for interesting tiering configurations with third party applications.

ASR-72405 and SAS HD Cabling

30

The ASR-72405 comes in a half-length full-height form factor and measures 4.198" x 6.6". This adapter provides 24 native SAS/SATA ports with new SAS HD cables. Four SFF-8643 connectors face upwards near the bracket, and two more face to the rear. The adapter connects via a PCIe Gen3 x8 slot, though it is backwards compatible with PCIe Gen2. The controller houses 1024MB of DDR3-1333 DRAM for caching.

30

The adapters function from 0-55C without the ZMCP module, and 0-55C with it installed. The card requires 200LFM of airflow for optimum operation. The adapter will throttle if the heat threshold exceeds the limit. The Series 7 adapters consume 12 to 18 Watts, dependent upon the port count. These power specifications were measured with PCIe Gen3 systems and 6G drives.

30

The heart of the Series 7 RAID controllers is the PMC8015 processor. This very small chip packs quite the punch, delivering 6,600MB/s in sequential speed and 450,000 IOPS in random read speed. Developed by PMC-Sierra, this cornerstone of the Series 7 controller provides Adaptec with a 24-port SAS 2.0 native solution that easily outstrips an x8 PCIe Gen3 connection.

30

The enhanced density that is a key building block of the Adaptec Series 7 controllers is empowered by the use of the new SAS HD cables. This new type of cable will experience wide deployment from other vendors with the 12Gb/s SAS products. Adaptec made the choice to begin using these cables a year early to maximize the potential of the PM8015. These cables provide a smaller connection that allows Adaptec to pack more connections onto the PCB.

30

After years of utilizing standard SAS cables these are a refreshing change. Cable routing and management is much simpler due to the thinner profile of the cabling itself. In comparison to previous cables the SAS HD cables are like strands of hair, and flexible enough for tight spaces and quick turns.

Adaptec has a new product line of 6Gb/s SAS HD cables with reasonable price points for use with the Series 7 adaptors. They are also the first to market with unique right angle connections. This will provide the flexibility for deployment into slim form factor servers. The only notable cable type that isn't present in Adaptec's line is dual-port SAS cables.

Test System and Methodology

Test System

30

The fun of testing HBA and RAID controllers is that we can use the fastest SSDs at our disposal in Fresh Out of Box (F.O.B.) conditions. One of the more tedious aspects of standard enterprise testing is the long drawn out conditioning runs to push the SSDs into steady state. This can take up to 12 hours for some SSDs, and consists of a lot of waiting on our part.

With HBA and RAID controller testing, we secure erase the SSDs repetitively to keep them at the absolute top speed. The ability to switch to the HBA mode with the ASR-72405 was a tremendous time saver when secure erasing the SSDs. There is obvious value to this functionality in mass deployments.

30

For throughput testing we turned to our trusty Crucial RealSSD C400s. Physically identical to the consumer M4, this 8 x 256GB array of SSDs has served us well in our testing by offering solid and consistent performance. We also had to bring in an additional 8 x 256GB M4's to help shoulder the load. Providing read speeds of 500 MB/s and 260 MB/s write speeds each, we can saturate the read bandwidth of the ASR-72405.

30

The second prong of our attack consists of the SanDisk ESS Lightning LS 300S EFDs (Enterprise Flash Devices). The LS 300S are 300GB SLC enterprise class SSDs that come in both a 3.5" and 2.5" form factor. These drives are made for taking a beating, and are warrantied for five years of unlimited writing with no throttling. They are simply solid as a rock, offering consistent performance regardless of the workload.

Rated at a blistering 160,000 IOPS per device, they are definitely the heavyweights of enterprise flash storage, even though we are using the 3Gb/s versions. There aren't many devices that can lay claim to this type of performance in steady state conditions.

Methodology

Many forms of testing involve utilizing peak and average measurements over a given time period. While these average values can give a basic understanding of the performance of the storage solution, they fall short in providing the clearest view possible of the QOS (Quality of Service) of the I/O. The degree of variability is especially pertinent, as many applications can hang or lag as they wait for one I/O to complete.

Our testing illustrates the performance variability expected in these types of scenarios, including the average measurements, during the measurement window. By providing ongoing measurements from our workloads with one-second reporting intervals, we can illustrate the difference between different products in relation to the purity of the QOS.

Consistent latency is the goal of every storage device, and measurements such as Maximum Latency only illuminate the single longest I/O received during testing. This can be misleading, as a single 'outlying I/O' can skew the view of an otherwise superb solution. Standard Deviation measurements take the average distribution of the I/O into consideration, but do not always effectively illustrate the entire I/O distribution with enough granularity to provide a clear picture of system performance.

By utilizing scatter charts readers can gain a basic understanding of the latency distribution of the I/O stream without directly observing numerous graphs. We also present histograms to illuminate the latency of every single I/O during the measurement window.

RAID 0 Sequential Testing

128K Sequential Read

30

Testing the ASR-72045 was challenging because its speeds are very hard to saturate. In a RAID 0 configuration the ASR-72405 delivers tremendous sequential speeds, higher than we have achieved from any other single slot solution. We configured our 16 SSDs into a RAID 0 with a stripe size of 256K.

We tested at QD256 with a 128K file size and achieved 6,640 MB/s. This speed was very consistent, with little variability during the test period.

30

The majority of read requests were served with the 4-6ms range, with 55% (or 53,899,281 I/Os) falling into that range. 24% (23,899,281 I/Os) fell into the 2-4ms range, and 16% (15,384,532 I/Os) fell into the 6-8ms range.

128K Sequential Write

30

Write speeds are the hardest aspect to test with a controller with so much speed on tap. It is important to note that we began with the SSDs freshly secure erased. The nature of SSDs leads to a descent into steady state, and from the test below, we can see that the SSDs hit the write cliff during our testing. This lower level of performance is due to the limitations of our hardware, and not any limitations from the ASR-72405 RAID controller.

We started out at 4,000 MB/s, and then the SSDs began to sink into their steady state performance. The sequential write speed of the C400's utilized for testing is 260MB/s each, and the write speed measured during the fresh condition represents perfect scaling from these SSDs. We simply do not have the required speed from our SSDs to saturate the sequential write speed of the ASR-72405, which tops out at 5,229MB/s.

30

The sequential write commands fell into a wide berth of latency, with 30% (16,348,896 I/Os) falling into the 4-6ms range, 28% (15,620,482 I/Os) into the 2-4ms range, and 14% (8,140,931 I/Os) coming in at 6-8ms.

RAID 0 4K Random Read

4K Random Read

30

We deployed the SanDisk Lightning LS 300S EFD's to saturate the controller with 4K random read speed. We ran into a limitation again with the random write speed testing, with none of our solutions coming close to handling the incredible write speed of the ASR-72405.

The Lightning's were configured into a RAID 0 with 64K stripe size, and then we created four clients to maximize results. We topped out at 557,000 IOPS, and experienced an average of 534,763 IOPS during the test period.

30

The random read latency from the ASR-72405 is superb, with the majority of read requests, 43% (488,834,830 I/Os), fell into the 0.00-0.02 range. 15% fell in the 1-2ms range, 12% at .6-.8ms, 11% at .4-.6 and 10% at .8-1ms.

RAID 5 Testing

128K Sequential Read

30

We utilized the C400 array to test the RAID 5 sequential read speed from the ASR-72405. We configured the 16 SSDs into RAID 5 with a 256K stripe size. The read pattern fell into a bimodal pattern, with one extreme hovering around 3,100MB/s, and the higher range topping 6.6GB/s. The majority of the read requests fell into the higher extreme, and our average for the test period was 5,500 MB/s.

30

With our staggered distribution of read results in the test above, we fully expected a larger latency distribution for the measurement window.

Surprisingly, the range of latency distribution was somewhat tight, with 44% (35,201,326 I/Os) falling into the 4-6ms range, 35% (28,040,901 I/Os) falling into the 2-4ms range, and 15% (12,133,712 I/Os) falling into the 6-8ms range.

4K Random Read

30

The 4K random read results were impressive, with an average speed of 531,600 IOPS. There was a nice even distribution in the results.

30

The similarities between the read latency from our RAID 0 4k random read testing and the RAID 5 random read is surprising. They are almost similar in the percentage of I/Os that fell in each category, though the quantity of I/Os differ.

43% (13,802,310 I/Os) fell into the 0.00-.02 range, 15% fell into the 1-2ms range, 13% fell into the .6-.8 range, 11% into the .4-.6ms range, and 10% fell into the .8-1ms range.

Final Thoughts

30

Today's primary challenge in the datacenter is to do more with less, with tight budgets demanding reductions in Cost of Service (COS) per user, CapEX and OpEX expenses. Solving problems with high performance storage solutions in a dense package is necessary, and the Adaptec Series 7 RAID controllers are designed to fit this requirement.

The cornerstone of the Adaptec Series 7 RAID controllers is the PM8015 processor. This new processor provides speeds that lives up to its billing, and in our testing matched or exceeded every test that our hardware could handle. The PM8015's ability to saturate the PCIe Gen3 interface provides an assurance that every drop of storage power is fully utilized.

The sheer speed of the controller and the 24-port architecture also created a problem for the Adaptec by PMC team. Standard 6Gb/s cables and their large connectors would not allow for enough connections on a limited amount of PCB real estate. The need for a dense solution prompted Adaptec to utilize the new SAS HD cables, one generation early. These cables are easy to use and great for routing through tight bends and spaces. Their thin profile and available right angle connections also allow easy utilization in slim servers.

After the hardware end of a product is designed and successfully integrated, the onus falls onto the software developers to deliver an easy to use solution that just works. The maxView management software is responsive, intuitive and easy to use. It could use the addition of a status monitor for RAID rebuilds and other tasks.

Overall, the RAID controller is easy to install and use, and the inclusion of a new HBA Mode for the adapter is a welcome addition. This allows users to switch into HBA mode with a reboot, and then manage the connected devices directly. This is useful in a number of scenarios and simplifies management tremendously.

30

12Gb/s hardware is on the horizon, but surprisingly 3Gb/s solutions are still prevalent in the channel. The vast majority of customers are now broadly deploying 6Gb/s solutions, and being able to secure the highest performance and port count is essential. Customers with existing infrastructure can benefit from the Series 7 RAID controllers, with one card able to do the work of two with passive backplanes. There is also no need for an expander with an active backplane.

One area that Adaptec has vastly improved is the interoperability of their solution, with the new series tested with 25 leading vendors for servers, enclosures, motherboards, drives and software. The ASR-72405 performed well with our equipment, including the relatively rare wideport SanDisk Lightning SAS EFD's that we utilized for IOPS saturation.

The Series 7 adapters have many features currently not available elsewhere. It is the only PCIe Gen3 solution with 24 native ports, the only adapters to saturate the PCIe Gen3 interface, the only solution with 6,660MB/s in throughput, and the only PCIe Gen3 LP/MD2 form-factor with 16 native ports. These impressive specifications, and the competitive pricing, combine to make the Adaptec a sure winner.

It seems that the acquisition by PMC-Sierra has breathed new life into Adaptec's product line. Adaptec by PMC is offering solutions that are ambitious in scope and offer compelling features for their customers. For their class leading performance and density, we give the ASR-72405 the TweakTown Editor's Choice Award.