-
Notifications
You must be signed in to change notification settings - Fork 5
/
Copy pathasync.html
210 lines (210 loc) · 20.9 KB
/
async.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<title>WiredTiger: Asynchronous operations</title>
<link href="tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="jquery.js"></script>
<script type="text/javascript" src="dynsections.js"></script>
<link href="navtree.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="resize.js"></script>
<script type="text/javascript" src="navtreedata.js"></script>
<script type="text/javascript" src="navtree.js"></script>
<script type="text/javascript">
$(document).ready(initResizable);
$(window).load(resizeHeight);
</script>
<link href="doxygen.css" rel="stylesheet" type="text/css" />
<link href="wiredtiger.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<table cellspacing="0" cellpadding="0">
<tbody>
<tr style="height: 56px;">
<td id="projectlogo"><a href="http://wiredtiger.com/"><img alt="Logo" src="LogoFinal-header.png" alt="WiredTiger" /></a></td>
<td style="padding-left: 0.5em;">
<div id="projectname">
 <span id="projectnumber">Version 2.9.2</span>
</div>
<div id="projectbrief"><!-- 2.9.2 --></div>
</td>
</tr>
</tbody>
</table>
</div>
<div class="banner">
<a href="https://github.com/wiredtiger/wiredtiger">Fork me on GitHub</a>
<a class="last" href="http://groups.google.com/group/wiredtiger-users">Join my user group</a>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.10 -->
<div id="navrow1" class="tabs">
<ul class="tablist">
<li><a href="index.html"><span>Main Page</span></a></li>
<li class="current"><a href="pages.html"><span>Related Pages</span></a></li>
<li><a href="modules.html"><span>Modules</span></a></li>
<li><a href="examples.html"><span>Examples</span></a></li>
<li><a href="community.html"><span>Community</span></a></li>
<li><a href="license.html"><span>License</span></a></li>
</ul>
</div>
</div><!-- top -->
<div id="side-nav" class="ui-resizable side-nav-resizable">
<div id="nav-tree">
<div id="nav-tree-contents">
<div id="nav-sync" class="sync"></div>
</div>
</div>
<div id="splitbar" style="-moz-user-select:none;"
class="ui-resizable-handle">
</div>
</div>
<script type="text/javascript">
$(document).ready(function(){initNavTree('async.html','');});
</script>
<div id="doc-content">
<div class="header">
<div class="headertitle">
<div class="title">Asynchronous operations </div> </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p>WiredTiger supports asynchronous operations; as an example of where this can be useful, a server application handling requests from a network as fast as possible may want its worker threads to initiate a unit of work and then immediately respond to the next request, rather than waiting for the results of the first request.</p>
<p>WiredTiger supports asynchronous operations through the <a class="el" href="struct_w_t___a_s_y_n_c___o_p.html" title="A WT_ASYNC_OP handle is the interface to an asynchronous operation. ">WT_ASYNC_OP</a> handle. The work unit represented by the <a class="el" href="struct_w_t___a_s_y_n_c___o_p.html" title="A WT_ASYNC_OP handle is the interface to an asynchronous operation. ">WT_ASYNC_OP</a> handle is queued by the application and performed by an internal WiredTiger worker thread. When the work unit completes, the WiredTiger thread makes a callback to notify the application the operation is finished, along with providing any results and error values.</p>
<p>The asynchronous operation handle operates in a manner similar to a <a class="el" href="struct_w_t___c_u_r_s_o_r.html" title="A WT_CURSOR handle is the interface to a cursor. ">WT_CURSOR</a> handle. An asynchronous operation includes:</p>
<ul>
<li>getter/setters for key and value fields</li>
<li>encoding of fields to store in the data source</li>
<li>methods to modify and retrieve specific data (for example, insert and update)</li>
<li>a method to compact a table</li>
</ul>
<p>The <a class="el" href="struct_w_t___a_s_y_n_c___o_p.html" title="A WT_ASYNC_OP handle is the interface to an asynchronous operation. ">WT_ASYNC_OP</a> handle does not survive after the callback function returns into the WiredTiger library. When the application callback returns the handle is returned to the system pool. The application callback must copy out any key, value or other information that it needs before the callback function returns.</p>
<h1><a class="anchor" id="async_config"></a>
Configuring asynchronous operations</h1>
<p>To perform asynchronous operations, the application must first include the <code>async</code> configuration option when <a class="el" href="group__wt.html#ga9e6adae3fc6964ef837a62795c7840ed" title="Open a connection to a database. ">wiredtiger_open</a> is called. Additional configuration parameters include the number of WiredTiger worker threads created to handle the incoming queue of operations and the maximum number of simultaneous asynchronous operations that are expected.</p>
<p>For example, the following configures an application for asynchronous operations, with a maximum of 10 asynchronous operations and 2 supporting threads:</p>
<div class="fragment"><div class="line"> ret = <a class="code" href="group__wt.html#ga9e6adae3fc6964ef837a62795c7840ed">wiredtiger_open</a>(home, NULL,</div>
<div class="line"> <span class="stringliteral">"create,cache_size=100MB,"</span></div>
<div class="line"> <span class="stringliteral">"async=(enabled=true,ops_max=20,threads=2)"</span>, &conn);</div>
</div><!-- fragment --><p> If the number of requests exceeds the configured maximum number, a <a class="el" href="struct_w_t___a_s_y_n_c___o_p.html" title="A WT_ASYNC_OP handle is the interface to an asynchronous operation. ">WT_ASYNC_OP</a> handle won't immediately be available and an error will be returned to the application when it attempts to allocate a handle. If the number of configured worker threads are unable to keep up with the requests, requests will be forced to wait for worker threads to become available.</p>
<h1><a class="anchor" id="async_alloc"></a>
Allocating an asynchronous operations handle</h1>
<p>A <a class="el" href="struct_w_t___a_s_y_n_c___o_p.html" title="A WT_ASYNC_OP handle is the interface to an asynchronous operation. ">WT_ASYNC_OP</a> handle is allocated using the <a class="el" href="struct_w_t___c_o_n_n_e_c_t_i_o_n.html#a0cfd7bfb13c5b2ad8499f44e43d62672" title="Return an async operation handle. ">WT_CONNECTION::async_new_op</a> method. This method takes an existing object URI and a callback. For example:</p>
<div class="fragment"><div class="line"> <span class="keywordflow">while</span> ((ret = conn->async_new_op(conn,</div>
<div class="line"> <span class="stringliteral">"table:async"</span>, NULL, &ex_asynckeys.iface, &op)) != 0) {</div>
<div class="line"> <span class="comment">/*</span></div>
<div class="line"><span class="comment"> * If we used up all the handles, pause and retry to</span></div>
<div class="line"><span class="comment"> * give the workers a chance to catch up.</span></div>
<div class="line"><span class="comment"> */</span></div>
<div class="line"> fprintf(stderr,</div>
<div class="line"> <span class="stringliteral">"asynchronous operation handle not available\n"</span>);</div>
<div class="line"> <span class="keywordflow">if</span> (ret == EBUSY)</div>
<div class="line"> sleep(1);</div>
<div class="line"> <span class="keywordflow">else</span></div>
<div class="line"> <span class="keywordflow">return</span> (EXIT_FAILURE);</div>
<div class="line"> }</div>
</div><!-- fragment --><p> To aid the application in matching up an asynchronous operation with a subsequent call to the callback function, every handle contains a unique <code>uint64_t</code> identifier and <a class="el" href="group__wt.html#ga24152c6e1509c488192df653d511a82c" title="Asynchronous operation types. ">WT_ASYNC_OPTYPE</a> type. The <code>identifier</code> is assigned when the handle is allocated and the <code>type</code> is assigned when the asynchronous operation is queued.</p>
<p>To retrieve the id, use the <a class="el" href="struct_w_t___a_s_y_n_c___o_p.html#aa10a973cc683c48944f4909e2f18b3b9" title="Get the unique identifier for this operation. ">WT_ASYNC_OP::get_id</a> method:</p>
<div class="fragment"><div class="line"> <span class="comment">/* Retrieve the operation's 64-bit identifier. */</span></div>
<div class="line"> <span class="keywordtype">id</span> = op->get_id(op);</div>
</div><!-- fragment --><p> To retrieve the <a class="el" href="group__wt.html#ga24152c6e1509c488192df653d511a82c" title="Asynchronous operation types. ">WT_ASYNC_OPTYPE</a> type, use the <a class="el" href="struct_w_t___a_s_y_n_c___o_p.html#adb476ee741ca2483fad081c892aea420" title="Get the type for this operation. ">WT_ASYNC_OP::get_type</a> method:</p>
<div class="fragment"><div class="line"> <span class="comment">/* Retrieve the operation's WT_ASYNC_OPTYPE type. */</span></div>
<div class="line"> type = op->get_type(op);</div>
</div><!-- fragment --><p> WiredTiger only allows a limited number of method calls back into the library using the <a class="el" href="struct_w_t___a_s_y_n_c___o_p.html" title="A WT_ASYNC_OP handle is the interface to an asynchronous operation. ">WT_ASYNC_OP</a> handle, while in the callback function. The application is allowed to retrieve than handle's key, value, identifier and the operation type from the <a class="el" href="struct_w_t___a_s_y_n_c___o_p.html" title="A WT_ASYNC_OP handle is the interface to an asynchronous operation. ">WT_ASYNC_OP</a> handle.</p>
<p>Here is a complete example callback function implementation, from the example program <a class="el" href="ex_async_8c-example.html">ex_async.c</a>:</p>
<div class="fragment"><div class="line"><span class="keyword">typedef</span> <span class="keyword">struct </span>{</div>
<div class="line"> <a class="code" href="struct_w_t___a_s_y_n_c___c_a_l_l_b_a_c_k.html">WT_ASYNC_CALLBACK</a> iface;</div>
<div class="line"> uint32_t num_keys;</div>
<div class="line">} ASYNC_KEYS;</div>
<div class="line"></div>
<div class="line"><span class="keyword">static</span> <span class="keywordtype">int</span></div>
<div class="line">async_callback(<a class="code" href="struct_w_t___a_s_y_n_c___c_a_l_l_b_a_c_k.html">WT_ASYNC_CALLBACK</a> *cb,</div>
<div class="line"> <a class="code" href="struct_w_t___a_s_y_n_c___o_p.html">WT_ASYNC_OP</a> *op, <span class="keywordtype">int</span> wiredtiger_error, uint32_t flags)</div>
<div class="line">{</div>
<div class="line"> ASYNC_KEYS *asynckey = (ASYNC_KEYS *)cb;</div>
<div class="line"> <a class="code" href="group__wt.html#ga24152c6e1509c488192df653d511a82c">WT_ASYNC_OPTYPE</a> type;</div>
<div class="line"> <a class="code" href="group__wt.html#struct_w_t___i_t_e_m">WT_ITEM</a> k, v;</div>
<div class="line"> <span class="keyword">const</span> <span class="keywordtype">char</span> *key, *value;</div>
<div class="line"> uint64_t id;</div>
<div class="line"> <span class="keywordtype">int</span> ret;</div>
<div class="line"></div>
<div class="line"> (void)flags; <span class="comment">/* Unused */</span></div>
<div class="line"></div>
<div class="line"> ret = 0;</div>
<div class="line"></div>
<div class="line"> <span class="comment">/* Retrieve the operation's WT_ASYNC_OPTYPE type. */</span></div>
<div class="line"> type = op-><a class="code" href="struct_w_t___a_s_y_n_c___o_p.html#adb476ee741ca2483fad081c892aea420">get_type</a>(op);</div>
<div class="line"> <span class="comment">/* Retrieve the operation's 64-bit identifier. */</span></div>
<div class="line"> <span class="keywordtype">id</span> = op-><a class="code" href="struct_w_t___a_s_y_n_c___o_p.html#aa10a973cc683c48944f4909e2f18b3b9">get_id</a>(op);</div>
<div class="line"> <span class="comment">/* Check for a WiredTiger error. */</span></div>
<div class="line"> <span class="keywordflow">if</span> (wiredtiger_error != 0) {</div>
<div class="line"> fprintf(stderr,</div>
<div class="line"> <span class="stringliteral">"ID %"</span> PRIu64 <span class="stringliteral">" error %d: %s\n"</span>,</div>
<div class="line"> <span class="keywordtype">id</span>, wiredtiger_error,</div>
<div class="line"> <a class="code" href="group__wt.html#gae8bf720ddb4a7a7390b70424594c40fd">wiredtiger_strerror</a>(wiredtiger_error));</div>
<div class="line"> global_error = wiredtiger_error;</div>
<div class="line"> <span class="keywordflow">return</span> (1);</div>
<div class="line"> }</div>
<div class="line"></div>
<div class="line"> <span class="comment">/* If doing a search, retrieve the key/value pair. */</span></div>
<div class="line"> <span class="keywordflow">if</span> (type == <a class="code" href="group__wt.html#gga24152c6e1509c488192df653d511a82ca0a70cdcda8801256cc086e8ab7d78b44">WT_AOP_SEARCH</a>) {</div>
<div class="line"> ret = op-><a class="code" href="struct_w_t___a_s_y_n_c___o_p.html#a3c0ab580d7d7cc18f669ef8259254b8c">get_key</a>(op, &k);</div>
<div class="line"> key = k.<a class="code" href="group__wt.html#a57f5e62aa968275d7e398cfa70e49450">data</a>;</div>
<div class="line"> ret = op-><a class="code" href="struct_w_t___a_s_y_n_c___o_p.html#a46ec0ae7d0294b9f4f206c390dd886b3">get_value</a>(op, &v);</div>
<div class="line"> value = v.<a class="code" href="group__wt.html#a57f5e62aa968275d7e398cfa70e49450">data</a>;</div>
<div class="line"> ATOMIC_ADD(asynckey->num_keys, 1);</div>
<div class="line"> printf(<span class="stringliteral">"Id %"</span> PRIu64 <span class="stringliteral">" got record: %s : %s\n"</span>, <span class="keywordtype">id</span>, key, value);</div>
<div class="line"> }</div>
<div class="line"> <span class="keywordflow">return</span> (ret);</div>
<div class="line">}</div>
</div><!-- fragment --><h1><a class="anchor" id="async_operations"></a>
Executing asynchronous operations</h1>
<p>The <a class="el" href="struct_w_t___a_s_y_n_c___o_p.html" title="A WT_ASYNC_OP handle is the interface to an asynchronous operation. ">WT_ASYNC_OP</a> handle behaves similarly to the <a class="el" href="struct_w_t___c_u_r_s_o_r.html" title="A WT_CURSOR handle is the interface to a cursor. ">WT_CURSOR</a> handle, that is, the key and value are initialized and then an operation is performed.</p>
<p>For example, the following code does an asynchronous insert into the table:</p>
<div class="fragment"><div class="line"> <span class="comment">/*</span></div>
<div class="line"><span class="comment"> * Set the operation's string key and value, and then do</span></div>
<div class="line"><span class="comment"> * an asynchronous insert.</span></div>
<div class="line"><span class="comment"> */</span></div>
<div class="line"> (void)snprintf(k[i], <span class="keyword">sizeof</span>(k), <span class="stringliteral">"key%d"</span>, i);</div>
<div class="line"> op-><a class="code" href="struct_w_t___a_s_y_n_c___o_p.html#a9931651a57133df0d6d881e0cb9386ab">set_key</a>(op, k[i]);</div>
<div class="line"> (void)snprintf(v[i], <span class="keyword">sizeof</span>(v), <span class="stringliteral">"value%d"</span>, i);</div>
<div class="line"> op-><a class="code" href="struct_w_t___a_s_y_n_c___o_p.html#a6cb91888b737850ebffd7e01215bff53">set_value</a>(op, v[i]);</div>
<div class="line"> ret = op-><a class="code" href="struct_w_t___a_s_y_n_c___o_p.html#a60ed981eeae3a80b13ca8af708e4704b">insert</a>(op);</div>
</div><!-- fragment --><p> For example, the following code does an asynchronous search of the table:</p>
<div class="fragment"><div class="line"> <span class="comment">/*</span></div>
<div class="line"><span class="comment"> * Set the operation's string key and value, and then do</span></div>
<div class="line"><span class="comment"> * an asynchronous search.</span></div>
<div class="line"><span class="comment"> */</span></div>
<div class="line"> (void)snprintf(k[i], <span class="keyword">sizeof</span>(k), <span class="stringliteral">"key%d"</span>, i);</div>
<div class="line"> op-><a class="code" href="struct_w_t___a_s_y_n_c___o_p.html#a9931651a57133df0d6d881e0cb9386ab">set_key</a>(op, k[i]);</div>
<div class="line"> ret = op-><a class="code" href="struct_w_t___a_s_y_n_c___o_p.html#a5cd1b495d3796c5318a49422a3f7e220">search</a>(op);</div>
</div><!-- fragment --><p> When a database contains multiple tables, it may be desired to compact several tables in parallel without having to manage separate threads to each call <a class="el" href="struct_w_t___s_e_s_s_i_o_n.html#aafa7a12a4891a5bfdc98673a5b8f9c69" title="Compact a live row- or column-store btree or LSM tree. ">WT_SESSION::compact</a>. Alternatively, compacting several tables serially may take much longer. The <a class="el" href="struct_w_t___a_s_y_n_c___o_p.html#ae6421d4c43369146502e8c7b54c06ac0" title="Invoke the underlying WT_SESSION::compact method; see that method for configuration, return and error values. ">WT_ASYNC_OP::compact</a> method allows the application to compact multiple objects asynchronously.</p>
<div class="fragment"><div class="line"> <span class="comment">/*</span></div>
<div class="line"><span class="comment"> * Compact a table asynchronously, limiting the run-time to 5 minutes.</span></div>
<div class="line"><span class="comment"> */</span></div>
<div class="line"> ret = conn->async_new_op(</div>
<div class="line"> conn, <span class="stringliteral">"table:async"</span>, <span class="stringliteral">"timeout=300"</span>, &ex_asynckeys.iface, &op);</div>
<div class="line"> ret = op-><a class="code" href="struct_w_t___a_s_y_n_c___o_p.html#ae6421d4c43369146502e8c7b54c06ac0">compact</a>(op);</div>
</div><!-- fragment --> <h1><a class="anchor" id="async_flush"></a>
Waiting for outstanding operations to complete</h1>
<p>The <a class="el" href="struct_w_t___c_o_n_n_e_c_t_i_o_n.html#a43d649d7eca3c9c281a833ec4e73473b" title="Wait for all outstanding operations to complete. ">WT_CONNECTION::async_flush</a> method can be used to wait for all previous operations to complete. When that call returns, all previously queued operations are guaranteed to have been completed and their callback functions have returned.</p>
<div class="fragment"><div class="line"> <span class="comment">/* Wait for all outstanding operations to complete. */</span></div>
<div class="line"> ret = conn->async_flush(conn);</div>
</div><!-- fragment --><p> Because <a class="el" href="struct_w_t___c_o_n_n_e_c_t_i_o_n.html#af535c517df851eeac8ebf3594d40b545" title="Close a connection. ">WT_CONNECTION::close</a> implicitly does a <a class="el" href="struct_w_t___c_o_n_n_e_c_t_i_o_n.html#a43d649d7eca3c9c281a833ec4e73473b" title="Wait for all outstanding operations to complete. ">WT_CONNECTION::async_flush</a>, the call is not required in all applications.</p>
<h1><a class="anchor" id="async_transactions"></a>
Asynchronous operations and transactions</h1>
<p>Each asynchronous worker thread operates in its own session, executing a single asynchronous operation with the context of the session's transaction. Therefore, there is no way to combine multiple, related updates into a single transaction when using asynchronous operations.</p>
<p>The transaction is committed if the operation was successful and the application callback returns success, otherwise the transaction is rolled back. </p>
</div></div><!-- contents -->
</div><!-- doc-content -->
<!-- start footer part -->
<div id="nav-path" class="navpath"><!-- id is needed for treeview function! -->
<ul>
<li class="navelem"><a class="el" href="index.html">Reference Guide</a></li><li class="navelem"><a class="el" href="programming.html">Writing WiredTiger applications</a></li>
<li class="footer">Copyright (c) 2008-2016 MongoDB, Inc. All rights reserved. Contact <a href="mailto:[email protected]">[email protected]</a> for more information.</li>
</ul>
</div>
</body>
</html>