-
Notifications
You must be signed in to change notification settings - Fork 9
/
Copy pathshuf.html
169 lines (154 loc) · 17.2 KB
/
shuf.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
<!doctypehtml><html class="sidebar-visible no-js light"lang=en><head><meta charset=UTF-8><title>shuf - CLI text processing with GNU Coreutils</title><meta content="text/html; charset=utf-8"http-equiv=Content-Type><meta content="Example based guide for specialized text processing with GNU Coreutils"name=description><meta content=width=device-width,initial-scale=1 name=viewport><meta content=#ffffff name=theme-color><meta content="CLI text processing with GNU Coreutils"property=og:title><meta content=website property=og:type><meta content="Example based guide for specialized text processing with GNU Coreutils"property=og:description><meta content=https://learnbyexample.github.io/cli_text_processing_coreutils/ property=og:url><meta content=https://raw.githubusercontent.com/learnbyexample/cli_text_processing_coreutils/main/images/cli_coreutils_ls.png property=og:image><meta content=1280 property=og:image:width><meta content=720 property=og:image:height><meta content=summary_large_image property=twitter:card><meta content=@learn_byexample property=twitter:site><link href=favicon.svg rel=icon><link rel="shortcut icon"href=favicon.png><link href=css/variables.css rel=stylesheet><link href=css/general.css rel=stylesheet><link href=css/chrome.css rel=stylesheet><link href=FontAwesome/css/font-awesome.css rel=stylesheet><link href=fonts/fonts.css rel=stylesheet><link href=highlight.css rel=stylesheet><link href=tomorrow-night.css rel=stylesheet><link href=ayu-highlight.css rel=stylesheet><link href=style.css rel=stylesheet><body><script>var path_to_root = "";
var default_theme = window.matchMedia("(prefers-color-scheme: dark)").matches ? "navy" : "light";</script><script>try {
var theme = localStorage.getItem('mdbook-theme');
var sidebar = localStorage.getItem('mdbook-sidebar');
if (theme.startsWith('"') && theme.endsWith('"')) {
localStorage.setItem('mdbook-theme', theme.slice(1, theme.length - 1));
}
if (sidebar.startsWith('"') && sidebar.endsWith('"')) {
localStorage.setItem('mdbook-sidebar', sidebar.slice(1, sidebar.length - 1));
}
} catch (e) { }</script><script>var theme;
try { theme = localStorage.getItem('mdbook-theme'); } catch(e) { }
if (theme === null || theme === undefined) { theme = default_theme; }
var html = document.querySelector('html');
html.classList.remove('no-js')
html.classList.remove('light')
html.classList.add(theme);
html.classList.add('js');</script><script>var html = document.querySelector('html');
var sidebar = 'hidden';
if (document.body.clientWidth >= 1080) {
try { sidebar = localStorage.getItem('mdbook-sidebar'); } catch(e) { }
sidebar = sidebar || 'visible';
}
html.classList.remove('sidebar-visible');
html.classList.add("sidebar-" + sidebar);</script><nav aria-label="Table of contents"class=sidebar id=sidebar><div class=sidebar-scrollbox><ol class=chapter><li class="chapter-item expanded affix"><a href=cover.html>Cover</a><li class="chapter-item expanded affix"><a href=buy.html>Buy PDF/EPUB versions</a><li class="chapter-item expanded affix"><a href=preface.html>Preface</a><li class="chapter-item expanded"><a href=introduction.html><strong aria-hidden=true>1.</strong> Introduction</a><li class="chapter-item expanded"><a href=cat-tac.html><strong aria-hidden=true>2.</strong> cat and tac</a><li class="chapter-item expanded"><a href=head-tail.html><strong aria-hidden=true>3.</strong> head and tail</a><li class="chapter-item expanded"><a href=tr.html><strong aria-hidden=true>4.</strong> tr</a><li class="chapter-item expanded"><a href=cut.html><strong aria-hidden=true>5.</strong> cut</a><li class="chapter-item expanded"><a href=seq.html><strong aria-hidden=true>6.</strong> seq</a><li class="chapter-item expanded"><a class=active href=shuf.html><strong aria-hidden=true>7.</strong> shuf</a><li class="chapter-item expanded"><a href=paste.html><strong aria-hidden=true>8.</strong> paste</a><li class="chapter-item expanded"><a href=pr.html><strong aria-hidden=true>9.</strong> pr</a><li class="chapter-item expanded"><a href=fold-fmt.html><strong aria-hidden=true>10.</strong> fold and fmt</a><li class="chapter-item expanded"><a href=sort.html><strong aria-hidden=true>11.</strong> sort</a><li class="chapter-item expanded"><a href=uniq.html><strong aria-hidden=true>12.</strong> uniq</a><li class="chapter-item expanded"><a href=comm.html><strong aria-hidden=true>13.</strong> comm</a><li class="chapter-item expanded"><a href=join.html><strong aria-hidden=true>14.</strong> join</a><li class="chapter-item expanded"><a href=nl.html><strong aria-hidden=true>15.</strong> nl</a><li class="chapter-item expanded"><a href=wc.html><strong aria-hidden=true>16.</strong> wc</a><li class="chapter-item expanded"><a href=split.html><strong aria-hidden=true>17.</strong> split</a><li class="chapter-item expanded"><a href=csplit.html><strong aria-hidden=true>18.</strong> csplit</a><li class="chapter-item expanded"><a href=expand-unexpand.html><strong aria-hidden=true>19.</strong> expand and unexpand</a><li class="chapter-item expanded"><a href=basename-dirname.html><strong aria-hidden=true>20.</strong> basename and dirname</a><li class="chapter-item expanded affix"><a href=what_next.html>What next?</a><li class="chapter-item expanded affix"><a href=Exercise_solutions.html>Exercise solutions</a></li><br><hr><li class="chapter-item expanded"><i class="fa fa-github"id=git-repository-button></i><a href=https://github.com/learnbyexample/cli_text_processing_coreutils> Source code</a><li class="chapter-item expanded"><i class="fa fa-home"id=home-button></i><a href=https://learnbyexample.github.io/> My Blog</a><li class="chapter-item expanded"><i class="fa fa-book"id=book-button></i><a href=https://learnbyexample.github.io/books/> My Books</a><li class="chapter-item expanded"><i class="fa fa-envelope"id=mail-button></i><a href=https://learnbyexample.gumroad.com/l/learnbyexample-weekly> learnbyexample weekly</a><li class="chapter-item expanded"><i class="fa fa-twitter"id=twitter-button></i><a href=https://twitter.com/learn_byexample> Twitter</a></ol></div><div class=sidebar-resize-handle id=sidebar-resize-handle></div></nav><div class=page-wrapper id=page-wrapper><div class=page><div id=menu-bar-hover-placeholder></div><div class="menu-bar sticky bordered"id=menu-bar><div class=left-buttons><button aria-label="Toggle Table of Contents"title="Toggle Table of Contents"aria-controls=sidebar class=icon-button id=sidebar-toggle type=button><i class="fa fa-bars"></i></button><button aria-label="Change theme"title="Change theme"aria-controls=theme-list aria-expanded=false aria-haspopup=true class=icon-button id=theme-toggle type=button><i class="fa fa-paint-brush"></i></button><ul aria-label=Themes class=theme-popup id=theme-list role=menu><li role=none><button class=theme id=light role=menuitem>Light (default)</button><li role=none><button class=theme id=rust role=menuitem>Rust</button><li role=none><button class=theme id=coal role=menuitem>Coal</button><li role=none><button class=theme id=navy role=menuitem>Navy</button><li role=none><button class=theme id=ayu role=menuitem>Ayu</button></ul><button aria-label="Toggle Searchbar"title="Search. (Shortkey: s)"aria-controls=searchbar aria-expanded=false aria-keyshortcuts=S class=icon-button id=search-toggle type=button><i class="fa fa-search"></i></button></div><h1 class=menu-title>CLI text processing with GNU Coreutils</h1><div class=right-buttons><a aria-label=Blog href=https://learnbyexample.github.io title=Blog> <i class="fa fa-home"id=home-button></i> </a><a aria-label=Twitter href=https://twitter.com/learn_byexample title=Twitter> <i class="fa fa-twitter"id=twitter-button></i> </a><a aria-label="Git repository"title="Git repository"href=https://github.com/learnbyexample/cli_text_processing_coreutils> <i class="fa fa-github"id=git-repository-button></i> </a></div></div><div class=hidden id=search-wrapper><form class=searchbar-outer id=searchbar-outer><input placeholder="Search this book ..."aria-controls=searchresults-outer aria-describedby=searchresults-header id=searchbar name=searchbar type=search></form><div class="searchresults-outer hidden"id=searchresults-outer><div class=searchresults-header id=searchresults-header></div><ul id=searchresults></ul></div></div><script>document.getElementById('sidebar-toggle').setAttribute('aria-expanded', sidebar === 'visible');
document.getElementById('sidebar').setAttribute('aria-hidden', sidebar !== 'visible');
Array.from(document.querySelectorAll('#sidebar a')).forEach(function(link) {
link.setAttribute('tabIndex', sidebar === 'visible' ? 0 : -1);
});</script><div class=content id=content><main><div class=sidetoc><nav class=pagetoc></nav></div><h1 id=shuf><a class=header href=#shuf>shuf</a></h1><p>The <code>shuf</code> command helps you randomize the input lines. And there are features to limit the number of output lines, repeat lines and even generate random positive integers.<h2 id=randomize-input-lines><a class=header href=#randomize-input-lines>Randomize input lines</a></h2><p>By default, <code>shuf</code> will randomize the order of input lines. Here's an example:<pre><code class=language-bash>$ cat purchases.txt
coffee
tea
washing powder
coffee
toothpaste
tea
soap
tea
$ shuf purchases.txt
tea
coffee
tea
toothpaste
soap
coffee
washing powder
tea
</code></pre><blockquote><p><img alt=info src=./images/info.svg> You can use the <code>--random-source=FILE</code> option to provide your own source for randomness. With this option, the output will be the same across multiple runs. See <a href=https://www.gnu.org/software/coreutils/manual/html_node/Random-sources.html>Sources of random data</a> for more details.</blockquote><blockquote><p><img alt=warning src=./images/warning.svg> <code>shuf</code> doesn't accept multiple input files. Use <code>cat</code> for such cases.</blockquote><h2 id=limit-output-lines><a class=header href=#limit-output-lines>Limit output lines</a></h2><p>Use the <code>-n</code> option to limit the number of lines you want in the output. If the value is greater than the number of lines in the input, it would be similar to not using the <code>-n</code> option.<pre><code class=language-bash>$ printf 'apple\nbanana\ncherry\nfig\nmango' | shuf -n2
mango
cherry
</code></pre><blockquote><p><img alt=info src=./images/info.svg> As seen in the example above, <code>shuf</code> will add a newline character even if it is not present for the last input line.</blockquote><h2 id=repeated-lines><a class=header href=#repeated-lines>Repeated lines</a></h2><p>The <code>-r</code> option helps if you want to allow input lines to be repeated. This option is usually paired with <code>-n</code> to limit the number of lines in the output.<pre><code class=language-bash>$ cat fruits.txt
banana
papaya
mango
$ shuf -n3 -r fruits.txt
banana
mango
banana
$ shuf -n5 -r fruits.txt
papaya
banana
mango
papaya
papaya
</code></pre><blockquote><p><img alt=info src=./images/info.svg> If a limit using <code>-n</code> is not specified, <code>shuf -r</code> will produce output lines indefinitely.</blockquote><h2 id=specify-input-lines-as-arguments><a class=header href=#specify-input-lines-as-arguments>Specify input lines as arguments</a></h2><p>You can use the <code>-e</code> option to specify multiple input lines as arguments to the command.<pre><code class=language-bash># quote the arguments as necessary
$ shuf -e hi there 'hello world' good
hello world
good
hi
there
$ shuf -n1 -e brown green blue
blue
$ shuf -n4 -r -e brown green blue
blue
green
brown
blue
</code></pre><p>The shell will autocomplete unquoted glob patterns (provided there are files that match the given expression). You can thus easily construct a solution to get a random selection of files matching the given glob pattern.<pre><code class=language-bash>$ echo *.csv
marks.csv mixed_fields.csv report_1.csv report_2.csv scores.csv
$ shuf -n2 -e *.csv
scores.csv
marks.csv
</code></pre><h2 id=generate-random-numbers><a class=header href=#generate-random-numbers>Generate random numbers</a></h2><p>The <code>-i</code> option will help generate random positive integers. The argument has to be a range, with <code>-</code> as the separator between the two numbers.<pre><code class=language-bash>$ shuf -i 5-8
5
8
7
6
$ shuf -n3 -i 100-200
170
112
148
$ shuf -n5 -r -i 0-1
1
0
0
1
1
</code></pre><blockquote><p><img alt=info src=./images/info.svg> <code>2^64 - 1</code> is the maximum allowed integer when I tested it on my machine.<pre><code class=language-bash>$ shuf -i 18446744073709551612-18446744073709551615
18446744073709551615
18446744073709551614
18446744073709551612
18446744073709551613
$ shuf -i 18446744073709551612-18446744073709551616
shuf: invalid input range: ‘18446744073709551616’:
Value too large for defined data type
# seq can help in such cases
# but remember that shuf needs to read the entire input
$ seq 100000000000000000000000000000 100000000000000000000000000105 | shuf -n2
100000000000000000000000000039
100000000000000000000000000018
</code></pre></blockquote><p><code>seq</code> can also help when you need negative and floating-point numbers.<pre><code class=language-bash>$ seq -10 -8 | shuf
-9
-10
-8
$ seq -f'%.4f' 100 0.25 3000 | shuf -n3
1627.7500
1303.5000
2466.2500
</code></pre><blockquote><p><img alt=info src=./images/info.svg> See <a href=https://unix.stackexchange.com/q/230673/109046>unix.stackexchange: generate random strings</a> if numbers aren't enough for you.</blockquote><h2 id=specifying-output-file><a class=header href=#specifying-output-file>Specifying output file</a></h2><p>The <code>-o</code> option can be used to specify the output file to be used for saving the results. This is more useful for in-place editing, since you can simply use shell redirection to save the output in a different file.<pre><code class=language-bash>$ cat book_list.txt
Cradle
Mage Errant
Mother of Learning
Super Powereds
The Umbral Storm
The Weirkey Chronicles
$ shuf book_list.txt -o book_list.txt
$ cat book_list.txt
Super Powereds
Cradle
Mage Errant
The Weirkey Chronicles
Mother of Learning
The Umbral Storm
</code></pre><h2 id=nul-separator><a class=header href=#nul-separator>NUL separator</a></h2><p>Use the <code>-z</code> option if you want to use NUL character as the line separator. In this scenario, <code>shuf</code> will ensure to add a final NUL character even if not present in the input.<pre><code class=language-bash>$ printf 'apple\0banana\0cherry\0fig\0mango' | shuf -z -n3 | cat -v
banana^@mango^@cherry^@
</code></pre><h2 id=exercises><a class=header href=#exercises>Exercises</a></h2><blockquote><p><img alt=info src=images/info.svg> The <a href=https://github.com/learnbyexample/cli_text_processing_coreutils/tree/main/exercises>exercises</a> directory has all the files used in this section.</blockquote><p><strong>1)</strong> What's wrong with the given command?<pre><code class=language-bash>$ shuf --random-source=greeting.txt fruits.txt books.txt
shuf: extra operand ‘books.txt’
Try 'shuf --help' for more information.
# expected output
##### add your solution here
banana
Cradle:::Mage Errant::The Weirkey Chronicles
Mother of Learning::Eight:::::Dear Spellbook:Ascendant
papaya
Mark of the Fool:Super Powereds:::Ends of Magic
mango
</code></pre><p><strong>2)</strong> What do the <code>-r</code> and <code>-n</code> options do? Why are they often used together?<p><strong>3)</strong> What does the following command do?<pre><code class=language-bash>$ shuf -e apple banana cherry fig mango
</code></pre><p><strong>4)</strong> Which option would you use to generate random numbers? Given an example.<p><strong>5)</strong> How would you generate 5 random numbers between <code>0.125</code> and <code>0.789</code> with a step value of <code>0.023</code>?<pre><code class=language-bash># output shown below is a sample, might differ for you
##### add your solution here
0.378
0.631
0.447
0.746
0.723
</code></pre></main><nav aria-label="Page navigation"class=nav-wrapper><a aria-label="Previous chapter"class="mobile-nav-chapters previous"title="Previous chapter"aria-keyshortcuts=Left href=seq.html rel=prev> <i class="fa fa-angle-left"></i> </a><a aria-label="Next chapter"class="mobile-nav-chapters next"title="Next chapter"aria-keyshortcuts=Right href=paste.html rel=next> <i class="fa fa-angle-right"></i> </a><div style="clear: both"></div></nav></div></div><nav aria-label="Page navigation"class=nav-wide-wrapper><a aria-label="Previous chapter"class="nav-chapters previous"title="Previous chapter"aria-keyshortcuts=Left href=seq.html rel=prev> <i class="fa fa-angle-left"></i> </a><a aria-label="Next chapter"class="nav-chapters next"title="Next chapter"aria-keyshortcuts=Right href=paste.html rel=next> <i class="fa fa-angle-right"></i> </a></nav></div><script>window.playground_copyable = true;</script><script charset=utf-8 src=elasticlunr.min.js></script><script charset=utf-8 src=mark.min.js></script><script charset=utf-8 src=searcher.js></script><script charset=utf-8 src=clipboard.min.js></script><script charset=utf-8 src=highlight.js></script><script charset=utf-8 src=book.js></script><script src=sidebar.js></script>