-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathtune_page_sizes.html
129 lines (129 loc) · 11.7 KB
/
tune_page_sizes.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<title>WiredTiger: Page and overflow key/value sizes</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="resize.js"></script>
<script type="text/javascript" src="navtreedata.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<script type="text/javascript">
$(document).ready(initResizable);
$(window).load(resizeHeight);
</script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="wiredtiger.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<td id="projectlogo"><a href="http://wiredtiger.com/"><img alt="Logo" src="LogoFinal-header.png" alt="WiredTiger" /></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
 <span id="projectnumber">Version 2.8.0</span>
</div>
</td>
</tr>
</tbody>
</table>
</div>
<div class="banner">
<a href="https://github.com/wiredtiger/wiredtiger">Fork me on GitHub</a>
<a class="last" href="http://groups.google.com/group/wiredtiger-users">Join my user group</a>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.9.1 -->
<div id="navrow1" class="tabs">
<ul class="tablist">
<li><a href="index.html"><span>Main Page</span></a></li>
<li class="current"><a href="pages.html"><span>Related Pages</span></a></li>
<li><a href="modules.html"><span>Modules</span></a></li>
<li><a href="examples.html"><span>Examples</span></a></li>
<li><a href="community.html"><span>Community</span></a></li>
<li><a href="license.html"><span>License</span></a></li>
</ul>
</div>
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
<div id="nav-tree">
<div id="nav-tree-contents">
<div id="nav-sync" class="sync"></div>
</div>
</div>
<div id="splitbar" style="-moz-user-select:none;"
class="ui-resizable-handle">
</div>
</div>
<script type="text/javascript">
$(document).ready(function(){initNavTree('tune_page_sizes.html','');});
</script>
<div id="doc-content">
<div class="header">
<div class="headertitle">
<div class="title">Page and overflow key/value sizes </div> </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p>There are seven page and key/value size configuration strings:</p>
<ul>
<li>allocation size (<code>allocation_size</code>),</li>
<li>page sizes (<code>internal_page_max</code> and <code>leaf_page_max</code>),</li>
<li>key and value sizes (<code>internal_key_max</code>, <code>leaf_key_max</code> and <code>leaf_value_max</code>), and the</li>
<li>page-split percentage (<code>split_pct</code>).</li>
</ul>
<p>All seven are specified to the <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file. ">WT_SESSION::create</a> method, in other words, they are configurable on a per-file basis.</p>
<p>Applications commonly configure page sizes, based on their workload's typical key and value size. Once the correct page size has been chosen, appropriate defaults for the other configuration values are derived from the page sizes, and relatively few applications will need to modify the other page and key/value size configuration options.</p>
<p>An example of configuring page and key/value sizes:</p>
<div class="fragment"><div class="line"> ret = session->create(session,</div>
<div class="line"> <span class="stringliteral">"table:mytable"</span>, <span class="stringliteral">"key_format=S,value_format=S,"</span></div>
<div class="line"> <span class="stringliteral">"internal_page_max=16KB,leaf_page_max=1MB,leaf_value_max=64KB"</span>);</div>
</div><!-- fragment --> <h1><a class="anchor" id="tune_page_sizes_sizes"></a>
Page, key and value sizes</h1>
<p>The <code>internal_page_max</code> and <code>leaf_page_max</code> configuration values specify a maximum size for Btree internal and leaf pages. That is, when an internal or leaf page grows past that size, it splits into multiple pages. Generally, internal pages should be sized to fit into on-chip caches in order to minimize cache misses when searching the tree, while leaf pages should be sized to maximize I/O performance (if reading from disk is necessary, it is usually desirable to read a large amount of data, assuming some locality of reference in the application's access pattern).</p>
<p>The default page size configurations (2KB for <code>internal_page_max</code>, 32KB for <code>leaf_page_max</code>), are appropriate for applications with relatively small keys and values.</p>
<ul>
<li>Applications doing full-table scans through out-of-memory workloads might increase both internal and leaf page sizes to transfer more data per I/O.</li>
<li>Applications focused on read/write amplification might decrease the page size to better match the underlying storage block size.</li>
</ul>
<p>When block compression has been configured, configured page sizes will not match the actual size of the page on disk. Block compression in WiredTiger happens within the I/O subsystem, and so a page might split even if subsequent compression would result in a resulting page size small enough to leave as a single page. In other words, page sizes are based on in-memory sizes, not on-disk sizes. Applications needing to write specific sized blocks may want to consider implementing a <a class="el" href="struct_w_t___c_o_m_p_r_e_s_s_o_r.html#ab19c8959b2fe1ee6abf1a51aac0c258f" title="Callback to compress a list of byte strings. ">WT_COMPRESSOR::compress_raw</a> function.</p>
<p>The page sizes also determine the default size of overflow items, that is, keys and values too large to easily store on a page. Overflow items are stored separately in the file from the page where the item logically appears, and so reading or writing an overflow item is more expensive than an on-page item, normally requiring additional I/O. Additionally, overflow values are not cached in memory. This means overflow items won't affect the caching behavior of the application, but it also means that each time an overflow value is read, it is re-read from disk.</p>
<p>For both of these reasons, applications should avoid creating large numbers of commonly referenced overflow items. This is especially important for keys, as keys on internal pages are referenced during random searches, not just during data retrieval. Generally, applications should make every attempt to avoid creating overflow keys.</p>
<ul>
<li>Applications with large keys and values, and concerned with latency, might increase the page size to avoid creating overflow items, in order to avoid the additional cost of retrieving them.</li>
<li>Applications with large keys and values, doing random searches, might decrease the page size to avoid wasting cache space on overflow items that aren't likely to be needed.</li>
<li>Applications with large keys and values, doing table scans, might increase the page size to avoid creating overflow items, as the overflow items must be read into memory in all cases, anyway.</li>
</ul>
<p>The <code>internal_key_max</code>, <code>leaf_key_max</code> and <code>leaf_value_max</code> configuration values allow applications to change the size at which a key or value will be treated as an overflow item.</p>
<p>The value of <code>internal_key_max</code> is relative to the maximum internal page size. Because the number of keys on an internal page determines the depth of the tree, the <code>internal_key_max</code> value can only be adjusted within a certain range, and the configured value will be automatically adjusted by WiredTiger, if necessary to ensure a reasonable number of keys fit on an internal page.</p>
<p>The values of <code>leaf_key_max</code> and <code>leaf_value_max</code> are not relative to the maximum leaf page size. If either is larger than the maximum page size, the page size will be ignored when the larger keys and values are being written, and a larger page will be created as necessary.</p>
<p>Most applications should not need to tune the maximum key and value sizes. Applications requiring a small page size, but also having latency concerns such that the additional work to retrieve an overflow item is an issue, may find them useful.</p>
<p>An example of configuring a large leaf overflow value:</p>
<div class="fragment"><div class="line"> ret = session->create(session,</div>
<div class="line"> <span class="stringliteral">"table:mytable"</span>, <span class="stringliteral">"key_format=S,value_format=S,"</span></div>
<div class="line"> <span class="stringliteral">"leaf_page_max=16KB,leaf_value_max=256KB"</span>);</div>
</div><!-- fragment --> <h1><a class="anchor" id="tune_page_sizes_split_percentage"></a>
Split percentage</h1>
<p>The <code>split_pct</code> configuration string configures the size of a split page. When a page grows sufficiently large that it must be written as multiple disk blocks, the newly written block size is <code>split_pct</code> percent of the maximum page size. This value should be selected to avoid creating a large number of tiny pages or repeatedly splitting whenever new entries are inserted. For example, if the maximum page size is 1MB, a <code>split_pct</code> value of 10% would potentially result in creating a large number of 100KB pages, which may not be optimal for future I/O. Or, if the maximum page size is 1MB, a <code>split_pct</code> value of 90% would potentially result in repeatedly splitting pages as the split pages grow to 1MB over and over. The default value for <code>split_pct</code> is 75%, intended to keep large pages relatively large, while still giving split pages room to grow.</p>
<p>Most applications should not need to tune the split percentage size.</p>
<h1><a class="anchor" id="tune_page_sizes_allocation_size"></a>
Allocation size</h1>
<p>The <code>allocation_size</code> configuration value is the underlying unit of allocation for the file. As the unit of file allocation, it sets the minimum page size and how much space is wasted when storing small amounts of data and overflow items. For example, if the allocation size is set to 4KB, an overflow item of 18,000 bytes requires 5 allocation units and wastes about 2KB of space. If the allocation size is 16KB, the same overflow item would waste more than 10KB.</p>
<p>The default allocation size is 4KB, chosen for compatibility with virtual memory page sizes and direct I/O requirements on common server platforms.</p>
<p>Most applications should not need to tune the allocation size; it is primarily intended for applications coping with the specific requirements some file systems make to support features like direct I/O. </p>
</div></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
<li class="navelem"><a class="el" href="index.html">Reference Guide</a></li><li class="navelem"><a class="el" href="programming_lang_java.html">Writing WiredTiger applications in Java</a></li>
<li class="footer">Copyright (c) 2008-2016 MongoDB, Inc. All rights reserved. Contact <a href="mailto:[email protected]">[email protected]</a> for more information.</li>
</ul>
</div>
</body>
</html>