-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy patharchitecture.html
127 lines (127 loc) · 8.59 KB
/
architecture.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<title>WiredTiger: WiredTiger Architecture</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="resize.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<script type="text/javascript">
$(document).ready(initResizable);
$(window).load(resizeHeight);
</script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="wiredtiger.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<td id="projectlogo"><a href="http://wiredtiger.com/"><img alt="Logo" src="LogoFinal-header.png" alt="WiredTiger" /></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
 <span id="projectnumber">Version 2.0.1</span>
</div>
</td>
</tr>
</tbody>
</table>
</div>
<div class="banner">
<a href="https://github.com/wiredtiger/wiredtiger">Fork me on GitHub</a>
<a class="last" href="http://groups.google.com/group/wiredtiger-users">Join my user group</a>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.3.1 -->
<div id="navrow1" class="tabs">
<ul class="tablist">
<li><a href="index.html"><span>Main Page</span></a></li>
<li class="current"><a href="pages.html"><span>Related Pages</span></a></li>
<li><a href="modules.html"><span>Modules</span></a></li>
<li><a href="examples.html"><span>Examples</span></a></li>
<li><a href="community.html"><span>Community</span></a></li>
<li><a href="license.html"><span>License</span></a></li>
</ul>
</div>
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
<div id="nav-tree">
<div id="nav-tree-contents">
<div id="nav-sync" class="sync"></div>
</div>
</div>
<div id="splitbar" style="-moz-user-select:none;"
class="ui-resizable-handle">
</div>
</div>
<script type="text/javascript">
$(document).ready(function(){initNavTree('architecture.html','');});
</script>
<div id="doc-content">
<div class="header">
<div class="headertitle">
<div class="title">WiredTiger Architecture </div> </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p>The WiredTiger data engine is a high performance, scalable, transactional, production quality, open source, NoSQL data engine, created to maximize the value of each computer you buy:</p>
<ul>
<li>WiredTiger offers both low latency and high throughput (in-cache reads require no latching, writes typically require a single latch),</li>
</ul>
<ul>
<li>WiredTiger handles data sets much larger than RAM without performance or resource degradation,</li>
</ul>
<ul>
<li>WiredTiger has predictable behavior under heavy access and large volumes of data,</li>
</ul>
<ul>
<li>WiredTiger offers transactional semantics without blocking,</li>
</ul>
<ul>
<li>WiredTiger stores are not corrupted by torn writes, reverting to the last checkpoint after system failure,</li>
</ul>
<ul>
<li>WiredTiger supports petabyte tables, records up to 4GB, and record numbers up to 64-bits.</li>
</ul>
<p>WiredTiger's design is focused on a few core principles:</p>
<h1><a class="anchor" id="multi_core"></a>
Multi-core scaling</h1>
<p>WiredTiger scales on modern, multi-CPU architectures. Using a variety of programming techniques such as hazard pointers, lock-free algorithms, fast latching and message passing, WiredTiger performs more work per CPU core than alternative engines.</p>
<p>WiredTiger's transactions use optimistic concurrency control algorithms that avoid the bottleneck of a centralized lock manager. Transactional operations in one thread do not block operations in other threads, but strong isolation is provided and update conflicts are detected to preserve data consistency.</p>
<h1><a class="anchor" id="cache"></a>
Hot caches</h1>
<p>WiredTiger supports both row-oriented storage (where all columns of a row are stored together), and column-oriented storage (where groups of columns are stored in separate files), resulting in more efficient memory use. When reading and writing column-stores, only the columns required for any particular query are maintained in memory. Column-store keys are derived from the value's location in the table rather than being physically stored in the table, further minimizing memory requirements. Finally, row-and column-stores can be mixed-and-matched at the table level: for example, a row-store index can be created on a column-store table.</p>
<p>WiredTiger supports <a class="el" href="lsm.html">Log-Structured Merge Trees</a>, where updates are buffered in small files that fit in cache for fast random updates, then automatically merged into larger files in the background so that read latency approaches that of traditional Btree files. LSM trees automatically create Bloom filters to avoid unnecessary reads from files that cannot containing matching keys.</p>
<p>WiredTiger supports different-sized Btree internal and leaf pages in the same file. Applications can maximize the amount of data transferred in each I/O by configuring large leaf pages, and still minimize CPU cache misses when searching the tree.</p>
<p>WiredTiger supports key prefix compression and value dictionaries, reducing the amount of memory keys and values require.</p>
<p>WiredTiger supports static encoding with a configurable Huffman engine, which typically reduces the amount of information maintained in memory by 20-50%.</p>
<h1><a class="anchor" id="io"></a>
Making I/O more valuable</h1>
<p>WiredTiger uses compact file formats to minimize on-disk overhead. WiredTiger does not store page content indexing information on disk, instead, WiredTiger instantiates content indexing information either when pages are read from disk or on demand. This simplifies the on-disk file format and in the case of small key/value pairs, typically reduces the amount of information written to disk by 20-50%.</p>
<p>WiredTiger supports variable-length pages, meaning there is less wasted space for large objects, and no need for compaction as pages grow and shrink naturally when key/value pairs are inserted or deleted.</p>
<p>WiredTiger supports block compression on table pages. Because WiredTiger supports variable-length pages, pages do not have to shrink by a fixed amount in order to benefit from block compression. Block compression is selectable on a per-table basis, allowing applications to choose the compression algorithm most appropriate for their data. Block compression typically reduces the amount of information written to disk by 30-80%.</p>
<p>WiredTiger supports leaf pages of up to 512MB in size. Disk seeks are less likely when reading large amounts of data from disk, significantly improving table scan performance.</p>
<p>Also, as noted in the <a class="el" href="architecture.html#cache">Hot caches</a> section, WiredTiger supports column-store formats, prefix compression and static encoding. While each of these features makes WiredTiger's use of memory more efficient, they also maximize the amount of useful data transferred per disk I/O.</p>
<h1><a class="anchor" id="quality"></a>
Production quality</h1>
<p>WiredTiger is production quality, supported software, engineered for the most demanding application environments. For example, as a no-overwrite data engine, torn writes can never corrupt a WiredTiger data store.</p>
<p>WiredTiger includes verification support so you can verify data sets, and salvage support as a last-ditch protection: data can be retrieved even if it somehow becomes corrupted.</p>
<h1><a class="anchor" id="nosql"></a>
NoSQL and Open Source</h1>
<p>WiredTiger is an Open Source, NoSQL data engine. See the <a class="el" href="license.html">WiredTiger license</a> for details. </p>
</div></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
<li class="navelem"><a class="el" href="index.html">Reference Guide</a></li>
<li class="footer">Copyright (c) 2008-2013 WiredTiger, Inc. All rights reserved. Contact <a href="mailto:[email protected]">[email protected]</a> for more information.</li>
</ul>
</div>
</body>
</html>