Cloud applications are increasingly distributing data across multiple regions and cloud providers due to privacy regulations, availability of specialized hardware, and to prevent vendor lock-in. Unfortunately, wide-area bulk data transfers are often slow, bottlenecking applications. We demonstrate that it is possible to significantly improve inter-cloud bulk transfer throughput by adapting network overlays to the cloud setting—that is, by routing data through indirect paths at the application layer. However, directly applying network overlays in this setting results in unacceptable increases in cloud egress prices. We present Overcloud, a system for bulk data transfer between cloud object stores that uses network overlays to optimally navigate the trade-off between price and performance. Overcloud's planner uses mixed-integer linear programming to determine the optimal overlay path and resource allocation for data transfer, subject to user-provided constraints on price or performance. Overcloud's data plane saturates the available bandwidth found by the planner with parallel reads/writes and integrates well with object stores, aligning with the stores’ internal layouts. Together, these techniques allow Overcloud to significantly improve object transfer throughput at very low cost overheads.
0 commit comments