-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathdurability_lang_java.html
141 lines (141 loc) · 15.8 KB
/
durability_lang_java.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<title>WiredTiger: Commit-level durability in Java</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="resize.js"></script>
<script type="text/javascript" src="navtreedata.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<script type="text/javascript">
$(document).ready(initResizable);
$(window).load(resizeHeight);
</script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="wiredtiger.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<td id="projectlogo"><a href="http://wiredtiger.com/"><img alt="Logo" src="LogoFinal-header.png" alt="WiredTiger" /></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
 <span id="projectnumber">Version 2.9.2</span>
</div>
<div id="projectbrief"><!-- 2.9.2 --></div>
</td>
</tr>
</tbody>
</table>
</div>
<div class="banner">
<a href="https://github.com/wiredtiger/wiredtiger">Fork me on GitHub</a>
<a class="last" href="http://groups.google.com/group/wiredtiger-users">Join my user group</a>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.10 -->
<div id="navrow1" class="tabs">
<ul class="tablist">
<li><a href="index.html"><span>Main Page</span></a></li>
<li class="current"><a href="pages.html"><span>Related Pages</span></a></li>
<li><a href="modules.html"><span>Modules</span></a></li>
<li><a href="examples.html"><span>Examples</span></a></li>
<li><a href="community.html"><span>Community</span></a></li>
<li><a href="license.html"><span>License</span></a></li>
</ul>
</div>
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
<div id="nav-tree">
<div id="nav-tree-contents">
<div id="nav-sync" class="sync"></div>
</div>
</div>
<div id="splitbar" style="-moz-user-select:none;"
class="ui-resizable-handle">
</div>
</div>
<script type="text/javascript">
$(document).ready(function(){initNavTree('durability_lang_java.html','');});
</script>
<div id="doc-content">
<div class="header">
<div class="headertitle">
<div class="title">Commit-level durability in Java </div> </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p>WiredTiger supports checkpoint durability by default, and optionally commit-level durability when logging is enabled. In most applications, commit-level durability impacts performance more than checkpoint durability; checkpoints offer basic operation durability across application or system failure without impacting performance (although the creation of each checkpoint is a relatively heavy-weight operation). See <a class="el" href="checkpoint_lang_java.html">Checkpoint durability in Java</a> for information on checkpoint durability.</p>
<p>Commit-level durability is implemented using a write-ahead log and is enabled using the <code>log=</code>(enabled) configuration to <code>wiredtiger.open</code>. When logging is enabled, WiredTiger writes records to the log for each transaction.</p>
<p>Transactions define which updates are made durable together; see <a class="el" href="transactions_lang_java.html">Transactions in Java</a> for details. By default, log records are buffered in memory when written by Session.commit_transaction. See <a class="el" href="durability_lang_java.html#durability_group_commit_lang_java">Group commit</a> for information about durability options.</p>
<p>When the transactional log is enabled, calling <code>wiredtiger.open</code> automatically performs a recovery step when opening the database that applies whatever changes from the log are required to bring the database up to date with the most recent transactional state. This recovery step may require extensions be available when it runs (for example, collators and compression). Therefore, applications doing recovery must configure extensions with the <code>extensions</code> keyword to <code>wiredtiger.open</code> consistently whenever re-opening the database.</p>
<p>Recovery is required after the failure of any thread of control in the application, where the failed thread might have been executing inside of the WiredTiger library or open WiredTiger handles have been lost. In most applications, if any thread of control exits unexpectedly, the application will close and re-open the database.</p>
<h1><a class="anchor" id="durability_checkpoint_lang_java"></a>
Checkpoints</h1>
<p>Checkpoints of the database should still be performed periodically when commit-level durability is configured, either explicitly from the application or periodically based on elapsed time or data size with the <code>checkpoint</code> configuration to <code>wiredtiger.open</code>.</p>
<p>Database checkpoints are necessary for two reasons: First, log files can only be archived after a checkpoint completes, and so the frequency of checkpoints determines the disk space required by log files. Second, checkpoints bound the time required for recovery to complete after application or system failure by limiting the log records that need to be processed.</p>
<h1><a class="anchor" id="durability_backup_lang_java"></a>
Backups</h1>
<p>With logging enabled, partial backups (backups where not all of the database objects are copied), may result in error messages during recovery, because data files referenced in the logs might not be found. Applications should either copy all objects and log files if commit-level durability of the copied database is required, or alternatively, copy only selected objects when backing up and not copy log files at all, then fall back to checkpoint durability when switching to the backup.</p>
<h1><a class="anchor" id="durability_bulk_lang_java"></a>
Bulk loads</h1>
<p>Bulk-loads are not commit-level durable, that is, the creation and bulk-load of an object will not appear in the database log files. For this reason, applications doing incremental backups after a full backup should repeat the full backup step after doing a bulk-load to make the bulk-load durable. In addition, incremental backups after a bulk-load can cause recovery to report errors because there are log records that apply to data files which don't appear in the backup.</p>
<h1><a class="anchor" id="durability_archiving_lang_java"></a>
Log file archival</h1>
<p>WiredTiger log files are named "WiredTigerLog.[number]" where "[number]" is a 10-digit value, for example, WiredTigerLog.0000000001". The log file with the largest number in its name is the most recent log file written. The log file size can be set using the <code>log</code> configuration to <code>wiredtiger.open</code>.</p>
<p>By default, WiredTiger automatically removes log files no longer required for recovery. Applications wanting to archive log files instead must disable log file removal using the <code>log=</code>(archive=false) configuration to <code>wiredtiger.open</code>.</p>
<p>Log files may be removed or archived after a checkpoint has completed, as long as there's not a backup in progress. Immediately after the checkpoint has completed, only the most recent log file is needed for recovery, and all other log files can be removed or archived. Note that there must always be at least one log file for the database.</p>
<p>Open log cursors prevents WiredTiger from automatically removing log files. Therefore, we recommend proactively closing log cursors when done with them. Applications manually removing log files should take care that no log cursors are opened in the log when removing files or errors may occur when trying to read a log record in a file that was removed.</p>
<h1><a class="anchor" id="durability_tuning_lang_java"></a>
Tuning commit-level durability</h1>
<h2><a class="anchor" id="durability_group_commit_lang_java"></a>
Group commit</h2>
<p>WiredTiger automatically groups the flush operations for threads that commit concurrently into single calls. This usually means multi-threaded workloads will achieve higher throughput than single-threaded workloads because the operating system can flush data more efficiently to the disk. No application-level configuration is required for this feature.</p>
<h2><a class="anchor" id="durability_flush_config_lang_java"></a>
Flush call configuration</h2>
<p>By default, log records are written to an in-memory buffer before <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a712226eca5ade5bd123026c624468fa2" title="Commit the current transaction. ">WT_SESSION::commit_transaction</a> returns, giving highest performance but not ensuring durability. The durability guarantees can be stricter but will impact performance.</p>
<p>If <code>transaction_sync=</code>(enabled=false) is configured to <a class="el" href="group__wt.html#ga9e6adae3fc6964ef837a62795c7840ed" title="Open a connection to a database. ">wiredtiger_open</a>, log records may be buffered in memory, and only flushed to disk by checkpoints, when log files switch or calls to <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a712226eca5ade5bd123026c624468fa2" title="Commit the current transaction. ">WT_SESSION::commit_transaction</a> with <code>sync=on</code>. (Note that any call to <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a712226eca5ade5bd123026c624468fa2" title="Commit the current transaction. ">WT_SESSION::commit_transaction</a> with <code>sync=on</code> will flush the log records for all committed transactions, not just the transaction where the configuration is set.) This provides the minimal guarantees, but will be significantly faster than other configurations.</p>
<p>If <code>transaction_sync=</code>(enabled=true), <code>transaction_sync=</code>(method) further configures the method used to flush log records to disk. By default, the configured value is <code>fsync</code>, which calls the operating system's <code>fsync</code> call (of <code>fdatasync</code> if available) as each commit completes.</p>
<p>If the value is set to <code>dsync</code>, the <code>O_DSYNC</code> or <code>O_SYNC</code> flag to the operating system's <code>open</code> call will be specified when the file is opened. (The durability guarantee of the <code>fsync</code> and <code>dsync</code> configurations are the same, and in our experience the <code>open</code> flags are slower, this configuration is only included for systems where that may not be the case.)</p>
<p>If the value is set to <code>none</code>, the operating system's <code>write</code> call will be called as each commit completes but does not flush to disk. This setting gives durability at the application level but not at the system level.</p>
<p>When a log file fills and the system moves to the next log file, the previous log file will always be flushed to disk prior to close. So when running in a durability mode that does not flush to disk, the risk is bounded by the most recent log file change.</p>
<p>Here is the expected performance of durability modes, in order from the fastest to the slowest (and from the fewest durability guarantees to the most durability guarantees).</p>
<table class="doxtable">
<tr>
<th>Durability Mode</th><th>Notes </th></tr>
<tr>
<td><code>log=(enabled=false)</code></td><td>checkpoint-level durability </td></tr>
<tr>
<td><code>log=(enabled),transaction_sync=(enabled=false)</code></td><td>in-memory buffered logging configured; updates durable after checkpoint or after <code>sync</code> is set in <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a7e26b16b26b5870498752322fad790bf" title="Start a transaction in this session. ">WT_SESSION::begin_transaction</a> </td></tr>
<tr>
<td><code>log=(enabled),transaction_sync=(enabled=true,method=none)</code></td><td>logging configured; updates durable after application failure, but not after system failure </td></tr>
<tr>
<td><code>log=(enabled),transaction_sync=(enabled=true,method=fsync)</code></td><td>logging configured; updates durable on application or system failure </td></tr>
<tr>
<td><code>log=(enabled),transaction_sync=(enabled=true,method=dsync)</code></td><td>logging configured; updates durable on application or system failure </td></tr>
</table>
<p>The durability setting can also be controlled directly on a per-transaction basis via the <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a712226eca5ade5bd123026c624468fa2" title="Commit the current transaction. ">WT_SESSION::commit_transaction</a> method. The <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a712226eca5ade5bd123026c624468fa2" title="Commit the current transaction. ">WT_SESSION::commit_transaction</a> supports several durability modes with the <code>sync</code> configuration that override the connection level settings.</p>
<p>If <code>sync=on</code> is configured then this commit operation will wait for its log records, and all earlier ones, to be durable to the extent specified by the <code>transaction_sync=</code>(method) setting before returning.</p>
<p>If <code>sync=off</code> is configured then this commit operation will write its records into the in-memory buffer and return immediately.</p>
<p>If <code>sync=background</code> is configured then this commit operation will write its record to an in-memory buffer, and will return. Prior to returning it will signal an internal WiredTiger worker thread to synchronize this log record. The caller may then check the status of that background synchronization with the <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a61c8c3ad80d8228172db66ca70bd90fd" title="Wait for a transaction to become synchronized. ">WT_SESSION::transaction_sync</a> method.</p>
<p>The durability of the write-ahead log can be controlled independently as well via the <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a1843292630960309129dcfe00e1a3817" title="Flush the log. ">WT_SESSION::log_flush</a> method. The <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a1843292630960309129dcfe00e1a3817" title="Flush the log. ">WT_SESSION::log_flush</a> supports several durability modes with the <code>sync</code> configuration that immediately act upon the log.</p>
<p>If <code>sync=on</code> is configured then this flush will force the current log and all earlier records to be durable on disk before returning. This method call overrides the <code>transaction_sync</code> setting and forces the data out via <code>fsync</code>.</p>
<p>If <code>sync=off</code> is configured then this flush operation will force the logging subsystem to write any outstanding in-memory buffers to the file system before returning.</p>
<p>If <code>sync=background</code> is configured then this flush operation will force the signalling of a background synchronization operation. The caller may then check the status of that background synchronization with the <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#a61c8c3ad80d8228172db66ca70bd90fd" title="Wait for a transaction to become synchronized. ">WT_SESSION::transaction_sync</a> method. </p>
</div></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
<li class="navelem"><a class="el" href="index.html">Reference Guide</a></li><li class="navelem"><a class="el" href="programming_lang_java.html">Writing WiredTiger applications in Java</a></li>
<li class="footer">Copyright (c) 2008-2016 MongoDB, Inc. All rights reserved. Contact <a href="mailto:[email protected]">[email protected]</a> for more information.</li>
</ul>
</div>
</body>
</html>