The Benefits of Byte-Level WAN Deduplication
By Silver Peak
Published 2007, Posted April 2008

Abstract:

WAN deduplication (also known as disk-based data reduction) is a critical component of state-of-the-art WAN acceleration. By eliminating the transfer of repetitive data across the WAN, deduplication saves bandwidth and increases application throughput (from 2x to 100x under the right circumstances.)

WAN acceleration appliances that utilize deduplication rely on storing commonly used patterns on the appliance’s disk drives, which are later referenced instead of transmitting repetitive data across the WAN. In a deduplication scenario, the first time data is transferred across the WAN (called a “first pass”) data patterns are stored in both appliances on either side of the WAN link. Subsequent transfers of the same or similar data (called “second passes”) take advantage of the fact that both appliances have already “seen” and stored the data. During these passes, the originating appliance refers to the data in the remote appliance’s data store, enabling the remote appliance to deliver the data to the end-station locally.

There are two ways of performing deduplication - “token-based” and “instruction-based”. This paper discusses how the two methods work, the differences between the two, and why each performs differently in “dynamic data” scenarios, such as AutoCAD, Microsoft Excel, and video streaming.