-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathtuning.html
253 lines (253 loc) · 34.5 KB
/
tuning.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<title>WiredTiger: Performance Tuning</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="resize.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<script type="text/javascript">
$(document).ready(initResizable);
$(window).load(resizeHeight);
</script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="wiredtiger.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<td id="projectlogo"><a href="http://wiredtiger.com/"><img alt="Logo" src="LogoFinal-header.png" alt="WiredTiger" /></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
 <span id="projectnumber">Version 2.0.1</span>
</div>
</td>
</tr>
</tbody>
</table>
</div>
<div class="banner">
<a href="https://github.com/wiredtiger/wiredtiger">Fork me on GitHub</a>
<a class="last" href="http://groups.google.com/group/wiredtiger-users">Join my user group</a>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.3.1 -->
<div id="navrow1" class="tabs">
<ul class="tablist">
<li><a href="index.html"><span>Main Page</span></a></li>
<li class="current"><a href="pages.html"><span>Related Pages</span></a></li>
<li><a href="modules.html"><span>Modules</span></a></li>
<li><a href="examples.html"><span>Examples</span></a></li>
<li><a href="community.html"><span>Community</span></a></li>
<li><a href="license.html"><span>License</span></a></li>
</ul>
</div>
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
<div id="nav-tree">
<div id="nav-tree-contents">
<div id="nav-sync" class="sync"></div>
</div>
</div>
<div id="splitbar" style="-moz-user-select:none;"
class="ui-resizable-handle">
</div>
</div>
<script type="text/javascript">
$(document).ready(function(){initNavTree('tuning.html','');});
</script>
<div id="doc-content">
<div class="header">
<div class="headertitle">
<div class="title">Performance Tuning </div> </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><h1><a class="anchor" id="tuning_cache"></a>
WiredTiger's cache</h1>
<h2><a class="anchor" id="tuning_cache_size"></a>
Cache size</h2>
<p>The cache size for the database is configurable by setting the <code>cache_size</code> configuration string when calling the <a class="el" href="group__wt.html#ga9e6adae3fc6964ef837a62795c7840ed" title="Open a connection to a database.">wiredtiger_open</a> function.</p>
<p>The effectiveness of the cache can be measured by reviewing the page eviction statistics for the database.</p>
<p>An example of setting a cache size to 500MB:</p>
<div class="fragment"><div class="line"> <span class="keywordflow">if</span> ((ret = <a class="code" href="group__wt.html#ga9e6adae3fc6964ef837a62795c7840ed" title="Open a connection to a database.">wiredtiger_open</a>(home, NULL,</div>
<div class="line"> <span class="stringliteral">"create,cache_size=500M"</span>, &conn)) != 0)</div>
<div class="line"> fprintf(stderr, <span class="stringliteral">"Error connecting to %s: %s\n"</span>,</div>
<div class="line"> home, <a class="code" href="group__wt.html#gac95e70a24d09cf6928398512990e1474" title="Return information about an error as a string; wiredtiger_strerror is a superset of the ISO C99/POSIX...">wiredtiger_strerror</a>(ret));</div>
</div><!-- fragment --> <h2><a class="anchor" id="tuning_read_only_objects"></a>
Read-only objects</h2>
<p>Cursors opened on checkpoints (either named, or using the special "last
checkpoint" name "WiredTigerCheckpoint") are read-only objects. Unless memory mapping is configured off (using the "mmap" configuration string to <a class="el" href="group__wt.html#ga9e6adae3fc6964ef837a62795c7840ed" title="Open a connection to a database.">wiredtiger_open</a>), read-only objects are mapped into process memory instead of being read through the WiredTiger cache. Using read-only objects where possible minimizes the amount of buffer cache memory required by WiredTiger applications and the work required for buffer cache management, as well as reducing the number of memory copies from the operating system buffer cache into application memory.</p>
<p>To open a named checkpoint, use the configuration string "checkpoint" to the <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#afb5b4a69c2c5cafe411b2b04fdc1c75d" title="Open a new cursor on a data source or duplicate an existing cursor.">WT_SESSION::open_cursor</a> method: </p>
<div class="fragment"><div class="line"> ret = session-><a class="code" href="struct_w_t___s_e_s_s_i_o_n.html#afb5b4a69c2c5cafe411b2b04fdc1c75d" title="Open a new cursor on a data source or duplicate an existing cursor.">open_cursor</a>(session,</div>
<div class="line"> <span class="stringliteral">"table:mytable"</span>, NULL, <span class="stringliteral">"checkpoint=midnight"</span>, &cursor);</div>
</div><!-- fragment --><p> To open the last checkpoint taken in the object, use the configuration string "checkpoint" with the name "WiredTigerCheckpoint" to the <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#afb5b4a69c2c5cafe411b2b04fdc1c75d" title="Open a new cursor on a data source or duplicate an existing cursor.">WT_SESSION::open_cursor</a> method: </p>
<div class="fragment"><div class="line"> ret = session-><a class="code" href="struct_w_t___s_e_s_s_i_o_n.html#afb5b4a69c2c5cafe411b2b04fdc1c75d" title="Open a new cursor on a data source or duplicate an existing cursor.">open_cursor</a>(session,</div>
<div class="line"> <span class="stringliteral">"table:mytable"</span>, NULL, <span class="stringliteral">"checkpoint=WiredTigerCheckpoint"</span>, &cursor);</div>
</div><!-- fragment --> <h2><a class="anchor" id="tuning_bulk_load"></a>
Bulk load</h2>
<p>When loading a large amount of data into a new object, using a cursor with the <code>bulk</code> configuration string enabled and loading the data in sorted order will be much faster than doing out-of-order inserts. See <a class="el" href="bulk_load.html">Bulk-load</a> for more information.</p>
<h2><a class="anchor" id="tuning_cache_resident"></a>
Cache resident objects</h2>
<p>Cache resident objects (objects never considered for the purposes of cache eviction), can be configured with the <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">WT_SESSION::create</a> "cache_resident" configuration string.</p>
<p>Configuring a cache resident object has two effects: first, once the object's pages have been instantiated in memory, no further I/O cost is ever paid for object access, minimizing potential latency. Second, in-memory objects can be accessed faster than objects tracked for potential eviction, and applications able to guarantee sufficient memory that an object need never be evicted can significantly increase their performance.</p>
<p>An example of configuring a cache-resident object:</p>
<div class="fragment"><div class="line"> ret = session-><a class="code" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">create</a>(session,</div>
<div class="line"> <span class="stringliteral">"table:mytable"</span>, <span class="stringliteral">"key_format=r,value_format=S,cache_resident=true"</span>);</div>
</div><!-- fragment --> <h1><a class="anchor" id="tuning_memory_allocator"></a>
Memory allocator</h1>
<p>The performance of heavily-threaded WiredTiger applications can be dominated by memory allocation because the WiredTiger engine has to free and re-allocate memory as part of many queries. Replacing the system's malloc implementation with one that has better threaded performance (for example, Google's <a href="http://goog-perftools.sourceforge.net/doc/tcmalloc.html">tcmalloc</a>, or <a href="http://www.canonware.com/jemalloc">jemalloc</a>), can dramatically improve throughput.</p>
<h1><a class="anchor" id="transparent_huge_pages"></a>
Linux transparent huge pages</h1>
<p>The performance of WiredTiger applications can be adversely affected when transparent huge pages are enabled on Linux systems. Some Linux systems enable transparent huge pages by default. Further information about transparent huge pages is <a href="https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Performance_Tuning_Guide/s-memory-transhuge.html">here</a>. To disable transparent huge pages run the following command as the root user:</p>
<div class="fragment"><div class="line">echo never > /sys/kernel/mm/transparent_hugepage/enabled</div>
</div><!-- fragment --><h1><a class="anchor" id="numa_zone_reclamation"></a>
Linux zone reclamation memory management</h1>
<p>The performance of WiredTiger applications can be sensitive to the zone reclamation algorithms used in Linux Non Uniform Memory Management (NUMA). Details of NUMA with specific reference to Linux are in this <a href="http://queue.acm.org/detail.cfm?id=2513149">article</a> by Christoph Lameter. If a WiredTiger database cache is assigned a significant proportion of system memory, it will likely be beneficial to disable zone reclamation. Zone reclamation can be disabled by running the following command as the root user:</p>
<div class="fragment"><div class="line">echo 0 > /proc/sys/vm/zone_reclaim_mode</div>
</div><!-- fragment --><h1><a class="anchor" id="tuning_cursor_persistence"></a>
Cursor persistence</h1>
<p>Opening a new cursor is a relatively expensive operation in WiredTiger (especially in table objects and Log-Structured Merge Trees (LSM) trees, where a logical cursor may require multiple, underlying object cursors), and caching cursors can improve performance. On the other hand, cursors hold positions in objects, and therefore long-lived cursor positions can decrease performance. The best combination is to cache cursors, but use the <a class="el" href="struct_w_t___c_u_r_s_o_r.html#afc1b42c22c9c85e1ba08ce3b34437565" title="Reset the position of the cursor.">WT_CURSOR::reset</a> method to discard the cursor's position in the object when the position is no longer needed.</p>
<p>Additionally, cursors are automatically reset whenever a transaction boundary is crossed; when a transaction is started with the <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a7e26b16b26b5870498752322fad790bf" title="Start a transaction in this session.">WT_SESSION::begin_transaction</a> or ended with either <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a712226eca5ade5bd123026c624468fa2" title="Commit the current transaction.">WT_SESSION::commit_transaction</a> or <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#ab45f521464ad9e54d9b15efc2ffe20a1" title="Roll back the current transaction.">WT_SESSION::rollback_transaction</a>, all open cursors are automatically reset, there is no need to call the <a class="el" href="struct_w_t___c_u_r_s_o_r.html#afc1b42c22c9c85e1ba08ce3b34437565" title="Reset the position of the cursor.">WT_CURSOR::reset</a> method explicitly, and the cursor can be immediately reused.</p>
<h1><a class="anchor" id="tuning_page_size"></a>
Page and overflow sizes</h1>
<p>There are four page and item size configuration values: <code>internal_page_max</code>, <code>internal_item_max</code>, <code>leaf_page_max</code> and <code>leaf_item_max</code>. All four should be specified to the <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">WT_SESSION::create</a> method, that is, they are configurable on a per-file basis.</p>
<p>The <code>internal_page_max</code> and <code>leaf_page_max</code> configuration values specify the maximum size for Btree internal and leaf pages. That is, when an internal or leaf page reaches the specified size, it splits into two pages. Generally, internal pages should be sized to fit into the system's L1 or L2 caches in order to minimize cache misses when searching the tree, while leaf pages should be sized to maximize I/O performance (if reading from disk is necessary, it is usually desirable to read a large amount of data, assuming some locality of reference in the application's access pattern).</p>
<p>The <code>internal_item_max</code> and <code>leaf_item_max</code> configuration values specify the maximum size at which an object will be stored on-page. Larger items will be stored separately in the file from the page where the item logically appears. Referencing overflow items is more expensive than referencing on-page items, requiring additional I/O if the object is not already cached. For this reason, it is important to avoid creating large numbers of overflow items that are repeatedly referenced, and the maximum item size should probably be increased if many overflow items are being created. Because pages must be large enough to store any item that is not an overflow item, increasing the size of the overflow items may also require increasing the page sizes.</p>
<p>With respect to compression, page and item sizes do not necessarily reflect the actual size of the page or item on disk, if block compression has been configured. Block compression in WiredTiger happens within the disk I/O subsystem, and so a page might split even if subsequent compression would result in a resulting page size that would be small enough to leave as a single page. In other words, page and overflow sizes are based on in-memory sizes, not disk sizes.</p>
<p>There are two other, related configuration values, also settable by the <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">WT_SESSION::create</a> method. They are <code>allocation_size</code>, and <code>split_pct</code>.</p>
<p>The <code>allocation_size</code> configuration value is the underlying unit of allocation for the file. As the unit of file allocation, it has two effects: first, it limits the ultimate size of the file, and second, it determines how much space is wasted when storing overflow items.</p>
<p>By limiting the size of the file, the allocation size limits the amount of data that can be stored in a file. For example, if the allocation size is set to the minimum possible (512B), the maximum file size is 2TB, that is, attempts to allocate new file blocks will fail when the file reaches 2TB in size. If the allocation size is set to the maximum possible (512MB), the maximum file size is 2EB.</p>
<p>The unit of allocation also determines how much space is wasted when storing overflow items. For example, if the allocation size were set to the minimum value of 512B, an overflow item of 1100 bytes would require 3 allocation sized file units, or 1536 bytes, wasting almost 500 bytes. For this reason, as the allocation size increases, page sizes and overflow item sizes will likely increase as well, to ensure that significant space isn't wasted by overflow items.</p>
<p>The last configuration value is <code>split_pct</code>, which configures the size of a split page. When a page grows sufficiently large that it must be split, the newly split page's size is <code>split_pct</code> percent of the maximum page size. This value should be selected to avoid creating a large number of tiny pages or repeatedly splitting whenever new entries are inserted. For example, if the maximum page size is 1MB, a <code>split_pct</code> value of 10% would potentially result in creating a large number of 100KB pages, which may not be optimal for future I/O. Or, if the maximum page size is 1MB, a <code>split_pct</code> value of 90% would potentially result in repeatedly splitting pages as the split pages grow to 1MB over and over. The default value for <code>split_pct</code> is 75%, intended to keep large pages relatively large, while still giving split pages room to grow.</p>
<p>An example of configuring page sizes:</p>
<div class="fragment"><div class="line"> ret = session-><a class="code" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">create</a>(session, <span class="stringliteral">"file:example"</span>,</div>
<div class="line"> <span class="stringliteral">"key_format=u,"</span></div>
<div class="line"> <span class="stringliteral">"internal_page_max=32KB,internal_item_max=1KB,"</span></div>
<div class="line"> <span class="stringliteral">"leaf_page_max=1MB,leaf_item_max=32KB"</span>);</div>
</div><!-- fragment --> <h1><a class="anchor" id="tuning_system_file_block"></a>
File block allocation</h1>
<h2><a class="anchor" id="tuning_system_file_block_grow"></a>
File growth</h2>
<p>It's faster on some filesystems to grow a file in chunks rather than to extend it a block at a time as new blocks are written. By configuring the wiredtiger_open functions <code>file_extend</code> value, applications can grow files ahead of the blocks being written.</p>
<div class="fragment"><div class="line"> ret = <a class="code" href="group__wt.html#ga9e6adae3fc6964ef837a62795c7840ed" title="Open a connection to a database.">wiredtiger_open</a>(</div>
<div class="line"> home, NULL, <span class="stringliteral">"create,file_extend=(data=16MB)"</span>, &conn);</div>
</div><!-- fragment --> <h2><a class="anchor" id="tuning_system_file_block_allocation"></a>
File allocation</h2>
<p>By default, when file blocks are being reused, WiredTiger attempts to avoid file fragmentation by selecting the smallest available block rather than splitting a larger available block into two. The <code>block_allocation</code> configuration string to <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">WT_SESSION::create</a> can be set to <code>first</code> to change the algorithm to first-fit, that is, take the first available block in the file. Applications where file size is more of an issue than file fragmentation (for example, applications with fixed-size blocks) might want to configure this way.</p>
<div class="fragment"><div class="line"> ret = session-><a class="code" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">create</a>(session, <span class="stringliteral">"table:mytable"</span>,</div>
<div class="line"> <span class="stringliteral">"key_format=S,value_format=S,block_allocation=first"</span>);</div>
</div><!-- fragment --> <h1><a class="anchor" id="tuning_system_buffer_cache"></a>
System buffer cache</h1>
<h2><a class="anchor" id="tuning_system_buffer_cache_direct_io"></a>
Direct I/O</h2>
<p>WiredTiger optionally supports direct I/O. Configuring direct I/O may be useful for applications wanting to:</p>
<ul>
<li>minimize the operating system cache effects of I/O to and from WiredTiger's buffer cache,</li>
<li>avoid double-buffering of blocks in WiredTiger's cache and the operating system buffer cache, and</li>
<li>avoid stalling underlying solid-state drives by writing a large number of dirty blocks.</li>
</ul>
<p>Direct I/O is configured using the "direct_io" configuration string to the <a class="el" href="group__wt.html#ga9e6adae3fc6964ef837a62795c7840ed" title="Open a connection to a database.">wiredtiger_open</a> function. An example of configuring direct I/O for WiredTiger's data files:</p>
<div class="fragment"><div class="line"> ret = <a class="code" href="group__wt.html#ga9e6adae3fc6964ef837a62795c7840ed" title="Open a connection to a database.">wiredtiger_open</a>(home, NULL, <span class="stringliteral">"create,direct_io=[data]"</span>, &conn);</div>
</div><!-- fragment --><p> Direct I/O implies a writing thread waits for the write to complete (which is a slower operation than writing into the system buffer cache), and configuring direct I/O is likely to decrease overall application performance.</p>
<p>Many Linux systems do not support mixing <code>O_DIRECT</code> and memory mapping or normal I/O to the same file, and attempting to do so can result in data loss or corruption. For this reason:</p>
<ul>
<li>WiredTiger silently ignores the setting of the <code>mmap</code> configuration to the wiredtiger_open function in those cases, and will never memory map a file which is configured for direct I/O.</li>
</ul>
<ul>
<li>If <code>O_DIRECT</code> is configured for data files on Linux systems, any system utilities used to copy database files for the purposes of hot backup should also specify <code>O_DIRECT</code> when configuring their file access. A standard Linux system utility that supports <code>O_DIRECT</code> is the <code>dd</code> utility, when using the <code>iflag=direct</code> command-line option.</li>
</ul>
<p>Additionally, many Linux systems require specific alignment for buffers used for I/O when direct I/O is configured, and using the wrong alignment can cause data loss or corruption. When direct I/O is configured on Linux systems, WiredTiger aligns I/O buffers to 4KB; if different alignment is required by your system, the <code>buffer_alignment</code> configuration to the wiredtiger_open call should be configured to the correct value.</p>
<p>Finally, if direct I/O is configured on any system, WiredTiger will silently change the file unit allocation size and the maximum leaf and internal page sizes to be at least as large as the <code>buffer_alignment</code> value as well as a multiple of that value.</p>
<p>Direct I/O is based on the non-standard <code>O_DIRECT</code> flag to the POSIX 1003.1 open system call and may not be available on all platforms.</p>
<h2><a class="anchor" id="tuning_system_buffer_cache_os_cache_dirty_max"></a>
os_cache_dirty_max</h2>
<p>As well as direct I/O, WiredTiger supports two additional configuration options related to the system buffer cache:</p>
<p>The first is <code>os_cache_dirty_max</code>, the maximum dirty bytes an object is allowed to have in the system buffer cache. Once this many bytes from an object are written into the system buffer cache, WiredTiger will attempt to schedule writes for all of the dirty blocks the object has in the system buffer cache. This configuration option allows applications to flush dirty blocks from the object, avoiding stalling any underlying drives when the object is subsequently flushed to disk as part of a durability operation.</p>
<p>An example of configuring <code>os_cache_dirty_max:</code> </p>
<div class="fragment"><div class="line"> ret = session-><a class="code" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">create</a>(</div>
<div class="line"> session, <span class="stringliteral">"table:mytable"</span>, <span class="stringliteral">"os_cache_dirty_max=500MB"</span>);</div>
</div><!-- fragment --><p> The <code>os_cache_dirty_max</code> configuration may not be used in combination with direct I/O.</p>
<p>The <code>os_cache_dirty_max</code> configuration is based on the non-standard Linux <code>sync_file_range</code> system call and may not available on all platforms.</p>
<h2><a class="anchor" id="tuning_system_buffer_cache_os_cache_max"></a>
os_cache_max</h2>
<p>The second configuration option related to the system buffer cache is <code>os_cache_max</code>, the maximum bytes an object is allowed to have in the system buffer cache. Once this many bytes from an object are either read into or written from the system buffer cache, WiredTiger will attempt to evict all of the object's blocks from the buffer cache. This configuration option allows applications to evict blocks from the system buffer cache to limit double-buffering and system buffer cache overhead.</p>
<p>An example of configuring <code>os_cache_max:</code> </p>
<div class="fragment"><div class="line"> ret = session-><a class="code" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">create</a>(session, <span class="stringliteral">"table:mytable"</span>, <span class="stringliteral">"os_cache_max=1GB"</span>);</div>
</div><!-- fragment --><p> The <code>os_cache_max</code> configuration may not be used in combination with direct I/O.</p>
<p>The <code>os_cache_max</code> configuration is based on the POSIX 1003.1 standard <code>posix_fadvise</code> system call and may not available on all platforms.</p>
<p>Configuring direct I/O, <code>os_cache_dirty_max</code> or <code>os_cache_max</code> all have the side effect of turning off memory-mapping of objects in WiredTiger.</p>
<h1><a class="anchor" id="tuning_checksums"></a>
Checksums</h1>
<p>WiredTiger checksums file reads and writes, by default. In read-only applications, or when file compression provides any necessary checksum functionality, or when using backing storage systems where blocks require no validation, performance can be increased by turning off checksum support when calling the <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">WT_SESSION::create</a> method.</p>
<p>Checksums can be configured to be "off" or "on", as well as "uncompressed". The "uncompressed" configuration checksums blocks not otherwise protected by compression. For example, in a system where the compression algorithms provide sufficient protection against corruption, and when writing a block which is too small to be usefully compressed, setting the checksum configuration value to "uncompressed" causes WiredTiger to checksum the blocks which are not compressed:</p>
<div class="fragment"><div class="line"> ret = session-><a class="code" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">create</a>(session, <span class="stringliteral">"table:mytable"</span>,</div>
<div class="line"> <span class="stringliteral">"key_format=S,value_format=S,checksum=uncompressed"</span>);</div>
</div><!-- fragment --> <h1><a class="anchor" id="tuning_compression"></a>
Compression</h1>
<p>WiredTiger configures key prefix compression for row-store objects by default. Additional forms of compression for both row- and column-store objects, including dictionary and block compression, and Huffman encoding, are optional. Compression minimizes in-memory and on-disk resource requirements and decreases the amount of I/O, at some CPU cost when rows are read and written.</p>
<p>Configuring compression on or off may change application throughput. For example, in applications using solid-state drives (where I/O is less expensive), turning off compression may increase application performance by reducing CPU costs; in applications where I/O costs are more expensive, turning on compression may increase application performance by reducing the overall number of I/O operations.</p>
<p>For example, turning off row-store key prefix compression:</p>
<div class="fragment"><div class="line"> ret = session-><a class="code" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">create</a>(session, <span class="stringliteral">"table:mytable"</span>,</div>
<div class="line"> <span class="stringliteral">"key_format=S,value_format=S,prefix_compression=false"</span>);</div>
</div><!-- fragment --><p> For example, turning on row-store or column-store dictionary compression:</p>
<div class="fragment"><div class="line"> ret = session-><a class="code" href="struct_w_t___s_e_s_s_i_o_n.html#a358ca4141d59c345f401c58501276bbb" title="Create a table, column group, index or file.">create</a>(session, <span class="stringliteral">"table:mytable"</span>,</div>
<div class="line"> <span class="stringliteral">"key_format=S,value_format=S,dictionary=1000"</span>);</div>
</div><!-- fragment --><p> See <a class="el" href="file_formats.html#file_formats_compression">File formats and compression</a> for more information.</p>
<h1><a class="anchor" id="tuning_statistics"></a>
Performance monitoring with statistics</h1>
<p>WiredTiger optionally maintains a variety of statistics, when the <code>statistics</code> configuration string is specified to <a class="el" href="group__wt.html#ga9e6adae3fc6964ef837a62795c7840ed" title="Open a connection to a database.">wiredtiger_open</a>; see <a class="el" href="statistics.html">Statistics</a> for general information about statistics, and <a class="el" href="data_sources.html#data_statistics">Statistics Data</a> for information about accessing the statistics.</p>
<p>Note that maintaining run-time statistics involves updating shared-memory data structures and may decrease application performance.</p>
<p>The statistics gathered by WiredTiger can be combined to derive information about the system's behavior. For example, a cursor can be opened on the statistics for a table:</p>
<div class="fragment"><div class="line"> <span class="keywordflow">if</span> ((ret = session-><a class="code" href="struct_w_t___s_e_s_s_i_o_n.html#afb5b4a69c2c5cafe411b2b04fdc1c75d" title="Open a new cursor on a data source or duplicate an existing cursor.">open_cursor</a>(session,</div>
<div class="line"> <span class="stringliteral">"statistics:table:access"</span>, NULL, NULL, &cursor)) != 0)</div>
<div class="line"> <span class="keywordflow">return</span> (ret);</div>
</div><!-- fragment --><p> Then this code calculates the "fragmentation" of a table, defined here as the percentage of the table that is not part of the current checkpoint:</p>
<div class="fragment"><div class="line"> uint64_t ckpt_size, file_size;</div>
<div class="line"> ret = get_stat(cursor, <a class="code" href="group__wt.html#ga3ba4d6c12abe10285dc3b599f082a4e4" title="block manager: checkpoint size">WT_STAT_DSRC_BLOCK_CHECKPOINT_SIZE</a>, &ckpt_size);</div>
<div class="line"> ret = get_stat(cursor, <a class="code" href="group__wt.html#ga2af5c768712ce6c6b840badd17dd13b9" title="block manager: file size in bytes">WT_STAT_DSRC_BLOCK_SIZE</a>, &file_size);</div>
<div class="line"></div>
<div class="line"> printf(<span class="stringliteral">"File is %d%% fragmented\n"</span>,</div>
<div class="line"> (<span class="keywordtype">int</span>)(100 * (file_size - ckpt_size) / file_size));</div>
</div><!-- fragment --><p> The following example calculates the "write amplification", defined here as the ratio of bytes written to the filesystem versus the total bytes inserted, updated and removed by the application.</p>
<div class="fragment"><div class="line"> uint64_t app_insert, app_remove, app_update, fs_writes;</div>
<div class="line"></div>
<div class="line"> ret = get_stat(cursor, <a class="code" href="group__wt.html#gaec7145b45392656825b8404ae1fb4f46" title="cursor-insert key and value bytes inserted">WT_STAT_DSRC_CURSOR_INSERT_BYTES</a>, &app_insert);</div>
<div class="line"> ret = get_stat(cursor, <a class="code" href="group__wt.html#gaf9cc721c75f42a110e0f6af104087b4d" title="cursor-remove key bytes removed">WT_STAT_DSRC_CURSOR_REMOVE_BYTES</a>, &app_remove);</div>
<div class="line"> ret = get_stat(cursor, <a class="code" href="group__wt.html#ga21596e9d2f3d0c992bcade65b85cb5eb" title="cursor-update value bytes updated">WT_STAT_DSRC_CURSOR_UPDATE_BYTES</a>, &app_update);</div>
<div class="line"></div>
<div class="line"> ret = get_stat(cursor, <a class="code" href="group__wt.html#ga212a7e9d219e279da8b10adfaaab2afc" title="bytes written from cache">WT_STAT_DSRC_CACHE_BYTES_WRITE</a>, &fs_writes);</div>
<div class="line"></div>
<div class="line"> <span class="keywordflow">if</span> (app_insert + app_remove + app_update != 0)</div>
<div class="line"> printf(<span class="stringliteral">"Write amplification is %.2lf\n"</span>,</div>
<div class="line"> (<span class="keywordtype">double</span>)fs_writes / (app_insert + app_remove + app_update));</div>
</div><!-- fragment --><p> Both examples use this helper function to retrieve statistics values from a cursor:</p>
<div class="fragment"><div class="line"><span class="keywordtype">int</span></div>
<div class="line">get_stat(<a class="code" href="struct_w_t___c_u_r_s_o_r.html" title="A WT_CURSOR handle is the interface to a cursor.">WT_CURSOR</a> *cursor, <span class="keywordtype">int</span> stat_field, uint64_t *valuep)</div>
<div class="line">{</div>
<div class="line"> <span class="keyword">const</span> <span class="keywordtype">char</span> *desc, *pvalue;</div>
<div class="line"> <span class="keywordtype">int</span> ret;</div>
<div class="line"></div>
<div class="line"> cursor-><a class="code" href="struct_w_t___c_u_r_s_o_r.html#ad1088d719df40babc1f57d086691ebdc" title="Set the key for the next operation.">set_key</a>(cursor, stat_field);</div>
<div class="line"> <span class="keywordflow">if</span> ((ret = cursor-><a class="code" href="struct_w_t___c_u_r_s_o_r.html#a7e25b2ced2cf3ec68bd5429bf921c79f" title="Return the record matching the key.">search</a>(cursor)) != 0)</div>
<div class="line"> <span class="keywordflow">return</span> (ret);</div>
<div class="line"></div>
<div class="line"> <span class="keywordflow">return</span> (cursor-><a class="code" href="struct_w_t___c_u_r_s_o_r.html#af85364a5af50b95bbc46c82e72f75c01" title="Get the value for the current record.">get_value</a>(cursor, &desc, &pvalue, valuep));</div>
<div class="line">}</div>
</div><!-- fragment --></div></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
<li class="navelem"><a class="el" href="index.html">Reference Guide</a></li><li class="navelem"><a class="el" href="programming.html">Writing WiredTiger applications</a></li>
<li class="footer">Copyright (c) 2008-2013 WiredTiger, Inc. All rights reserved. Contact <a href="mailto:[email protected]">[email protected]</a> for more information.</li>
</ul>
</div>
</body>
</html>