-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathtune_system_buffer_cache.html
120 lines (120 loc) · 9.72 KB
/
tune_system_buffer_cache.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<title>WiredTiger: System buffer cache</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="resize.js"></script>
<script type="text/javascript" src="navtreedata.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<script type="text/javascript">
$(document).ready(initResizable);
$(window).load(resizeHeight);
</script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="wiredtiger.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<td id="projectlogo"><a href="http://wiredtiger.com/"><img alt="Logo" src="LogoFinal-header.png" alt="WiredTiger" /></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
 <span id="projectnumber">Version 2.9.2</span>
</div>
<div id="projectbrief"><!-- 2.9.2 --></div>
</td>
</tr>
</tbody>
</table>
</div>
<div class="banner">
<a href="https://github.com/wiredtiger/wiredtiger">Fork me on GitHub</a>
<a class="last" href="http://groups.google.com/group/wiredtiger-users">Join my user group</a>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.10 -->
<div id="navrow1" class="tabs">
<ul class="tablist">
<li><a href="index.html"><span>Main Page</span></a></li>
<li class="current"><a href="pages.html"><span>Related Pages</span></a></li>
<li><a href="modules.html"><span>Modules</span></a></li>
<li><a href="examples.html"><span>Examples</span></a></li>
<li><a href="community.html"><span>Community</span></a></li>
<li><a href="license.html"><span>License</span></a></li>
</ul>
</div>
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
<div id="nav-tree">
<div id="nav-tree-contents">
<div id="nav-sync" class="sync"></div>
</div>
</div>
<div id="splitbar" style="-moz-user-select:none;"
class="ui-resizable-handle">
</div>
</div>
<script type="text/javascript">
$(document).ready(function(){initNavTree('tune_system_buffer_cache.html','');});
</script>
<div id="doc-content">
<div class="header">
<div class="headertitle">
<div class="title">System buffer cache </div> </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><h1><a class="anchor" id="tuning_system_buffer_cache_direct_io"></a>
Direct I/O</h1>
<p>WiredTiger optionally supports direct I/O. Configuring direct I/O may be useful for applications wanting to:</p><ul>
<li>minimize the operating system cache effects of I/O to and from WiredTiger's buffer cache,</li>
<li>avoid double-buffering of blocks in WiredTiger's cache and the operating system buffer cache, and</li>
<li>avoid stalling underlying solid-state drives by writing a large number of dirty blocks.</li>
</ul>
<p>Direct I/O is configured using the "direct_io" configuration string to the <a class="el" href="group__wt.html#ga9e6adae3fc6964ef837a62795c7840ed" title="Open a connection to a database. ">wiredtiger_open</a> function. An example of configuring direct I/O for WiredTiger's data files:</p>
<div class="fragment"><div class="line"> ret = <a class="code" href="group__wt.html#ga9e6adae3fc6964ef837a62795c7840ed">wiredtiger_open</a>(home, NULL, <span class="stringliteral">"create,direct_io=[data]"</span>, &conn);</div>
</div><!-- fragment --><p> On Windows, the "direct_io" configuration string controls whether the operating system cache is used to buffer reads, and writes to disk, i.e., FILE_FLAG_NO_BUFFERING. When "direct_io" is off, Windows will use free RAM to cache access to files. This may had adverse effects because Windows may page out the WiredTiger buffer cache instead of its file cache. An additional configuration string "write_through" controls whether the disk is allowed to cache the writes. Enabling this flag increases write latency as the drive must ensure all writes are persisted to disk, but it ensures write durability. To get the equivalent of <code>O_DIRECT</code> on Windows, "direct_io", and "write_through" must be both set.</p>
<p>Direct I/O implies a writing thread waits for the write to complete (which is a slower operation than writing into the system buffer cache), and configuring direct I/O is likely to decrease overall application performance.</p>
<p>Many Linux systems do not support mixing <code>O_DIRECT</code> and memory mapping or normal I/O to the same file, and attempting to do so can result in data loss or corruption. For this reason:</p>
<ul>
<li>WiredTiger silently ignores the setting of the <code>mmap</code> configuration to the wiredtiger_open function in those cases, and will never memory map a file which is configured for direct I/O.</li>
<li>If <code>O_DIRECT</code> is configured for data files on Linux systems, any system utilities used to copy data files for the purposes of backup should also specify <code>O_DIRECT</code> when configuring their file access. A standard Linux system utility that supports <code>O_DIRECT</code> is the <code>dd</code> utility, when using the <code>iflag=direct</code> command-line option.</li>
</ul>
<p>Additionally, Windows, and many Linux systems require specific alignment for buffers used for I/O when direct I/O is configured, and using the wrong alignment can cause data loss or corruption. When direct I/O is configured on Windows, and Linux systems, WiredTiger aligns I/O buffers to 4KB; if different alignment is required by your system, the <code>buffer_alignment</code> configuration to the wiredtiger_open call should be configured to the correct value.</p>
<p>Finally, if direct I/O is configured on any system, WiredTiger will silently change the file unit allocation size and the maximum leaf and internal page sizes to be at least as large as the <code>buffer_alignment</code> value as well as a multiple of that value.</p>
<p>Direct I/O is based on the non-standard <code>O_DIRECT</code> flag to the POSIX 1003.1 open system call and may not be available on all platforms.</p>
<h1><a class="anchor" id="tuning_system_buffer_cache_os_cache_dirty_max"></a>
os_cache_dirty_max</h1>
<p>As well as direct I/O, WiredTiger supports two additional configuration options related to the system buffer cache:</p>
<p>The first is <code>os_cache_dirty_max</code>, the maximum dirty bytes an object is allowed to have in the system buffer cache. Once this many bytes from an object are written into the system buffer cache, WiredTiger will attempt to schedule writes for all of the dirty blocks the object has in the system buffer cache. This configuration option allows applications to flush dirty blocks from the object, avoiding stalling any underlying drives when the object is subsequently flushed to disk as part of a durability operation.</p>
<p>An example of configuring <code>os_cache_dirty_max:</code> </p>
<div class="fragment"><div class="line"> ret = session->create(</div>
<div class="line"> session, <span class="stringliteral">"table:mytable"</span>, <span class="stringliteral">"os_cache_dirty_max=500MB"</span>);</div>
</div><!-- fragment --><p> The <code>os_cache_dirty_max</code> configuration may not be used in combination with direct I/O.</p>
<p>The <code>os_cache_dirty_max</code> configuration is based on the non-standard Linux <code>sync_file_range</code> system call and will be ignored if set and that call is not available.</p>
<h1><a class="anchor" id="tuning_system_buffer_cache_os_cache_max"></a>
os_cache_max</h1>
<p>The second configuration option related to the system buffer cache is <code>os_cache_max</code>, the maximum bytes an object is allowed to have in the system buffer cache. Once this many bytes from an object are either read into or written from the system buffer cache, WiredTiger will attempt to evict all of the object's blocks from the buffer cache. This configuration option allows applications to evict blocks from the system buffer cache to limit double-buffering and system buffer cache overhead.</p>
<p>An example of configuring <code>os_cache_max:</code> </p>
<div class="fragment"><div class="line"> ret = session->create(session, <span class="stringliteral">"table:mytable"</span>, <span class="stringliteral">"os_cache_max=1GB"</span>);</div>
</div><!-- fragment --><p> The <code>os_cache_max</code> configuration may not be used in combination with direct I/O.</p>
<p>The <code>os_cache_max</code> configuration is based on the POSIX 1003.1 standard <code>posix_fadvise</code> system call and may not available on all platforms.</p>
<p>Configuring direct I/O, <code>os_cache_dirty_max</code> or <code>os_cache_max</code> all have the side effect of turning off memory-mapping of objects in WiredTiger. </p>
</div></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
<li class="navelem"><a class="el" href="index.html">Reference Guide</a></li><li class="navelem"><a class="el" href="programming_lang_java.html">Writing WiredTiger applications in Java</a></li>
<li class="footer">Copyright (c) 2008-2016 MongoDB, Inc. All rights reserved. Contact <a href="mailto:[email protected]">[email protected]</a> for more information.</li>
</ul>
</div>
</body>
</html>