Useful_Cmds

Useful Terminal Commands

Linux/Mac command lines and setup stuff that I find useful

APT

# shows which package owns /path/to/file
apt-file search /path/to/file
# allows reinstalling a package without having to remove it first (and risk removing its dependencies)
apt-get install --reinstall packagename
apt-file list packagename
apt-show-versions | grep unstable
apt-get build-dep packagename

Development

JAVA

Install Sun/Oracle's JDK

# First add "contrib" to /etc/apt-sources.list
# Then
apt-get update & apt-get install java-package
make-jpkg /path/to/jdk-*-linux-x64.tar.gz
dpkg -i oracle-java-*_amd64.deb

Administration

Email

We use ssmtp to send emails. The following uses Gmail's server.

After installing, edit the /etc/ssmtp/ssmtp.conf file

[email protected]
mailhub=smtp.gmail.com:587
rewriteDomain=
hostname=fileserver.local
UseSTARTTLS=YES
AuthUser=username
AuthPass=password
FromLineOverride=YES

Now edit /etc/ssmtp/revaliases to add accounts that we want to be able to send mails. For example,
```
root:[email protected]:smtp.gmail.com:587
auser:[email protected]:smtp.gmail.com:587
```

Finally, we can test sending mails

$ echo "test" | mail -v -s "testing ssmtp setup" [email protected]
$ echo "Do this" | mail -s "Todo" [email protected]
$ echo "A TEST" > afile
$ mail -s "Todo" [email protected] < afile

RAID

This is for RAID-5 setup, which requires a minimum of 3 drives. The usable space is (number of drives - 1) * size of smallest drive. Here we will use mdm to manage our RAID setup (apt-get install mdadm).

To create new drive partitions in the RAID array, we can use fdisk or parted (if the disk size > 2 terabytes). If use fdisk then choose the partition type to be raid (type fd). Now assume we want to use the 4 partitions sdc1, sdc1, sde1, and sdf1.

# build the RAID array
mdadm --create --verbose /dev/md0 --level=5 --raid-devices=4 /dev/sd[cdef]1

# it will take some time to build the array, we can watch its status with
watch cat /proc/mdstat

#once the array is built, we can check with
mdadm --detail /dev/md0

/dev/md0:
Version : 1.2
Creation Time : Fri Jun 24 15:04:32 2016
Raid Level : raid5
Array Size : 734954496 (700.91 GiB 752.59 GB)
Used Dev Size : 244984832 (233.64 GiB 250.86 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent

Intent Bitmap : Internal

Update Time : Sat Jun 25 00:34:10 2016
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 512K

Name : GiaoChi:0
UUID : 1d431c63:a9fe7dca:ffb4dca5:58391d2e
Events : 1097

Number   Major   Minor   RaidDevice State
0     8     33    0    active sync   /dev/sdc1
1     8     49    1    active sync   /dev/sdd1
2     8     65    2    active sync   /dev/sde1
4     8     81    3    active sync   /dev/sdf1


# create file extension and mount
mkfs.ext4 /dev/md0
mkdir /MYNEWSTORAGE
# in /etc/fstab
/dev/md0    /MYNEWSTORAGE      ext4    defaults    0    0
# finally mount the array so that we can being to access /MYNEWSTORAGE
mount -av

#edit configuration files so that mdm knows how to assemble the array when the system boots
echo "DEVICE partitions" > /etc/mdadm/mdadm.conf
echo "HOMEHOST fileserver" >> /etc/mdadm/mdadm.conf
echo "MAILADDR [email protected]" >> /etc/mdadm/mdadm.conf
mdadm --detail --scan --verbose >> /etc/mdadm/mdadm.conf

GPG

gpg --gen-key
gpg --export  -a "Name" > pub.key
gpg --export-secret-key -a "Name" > priv.key
gpg --import priv.key
gpg --list-keys
gpg --list-secret-keys
gpg --delete-key "Name"
gpg --delete-secret-key "Name"
gpg --edit-key "Name"  #here you can change passphrase of priv key with passwd

Remote Login

SSH

Generate password-less SSH public key
1. In a terminal from your machine client, type ssh-keygen -t rsa, and hit Enter on most questions, even when it asks for password, i.e., leave the password blank. This allows to log in another machine from client without requiring password.
2. The public key is stored at ~/.ssh/id_rsa.pub
3. Tips:
  - Add you SSH public key to code repo (e.g., Github, Bitbucket). This allows you to modify projects (e.g., push) without having to enter username and password.
  - You can also use this key to ssh into other machine without having to enter password (see password-less login below)
Password-less login
1. To ssh login the server host without having to enter a password:
```
$ cat ~/.ssh/id_rsa.pub | ssh username@host 'cat >> ~/.ssh/authorized_keys'
```

Disable password login on host

To disable password login, edit the /etc/ssh/sshd_config file on the server host and set the following:

PasswordAuthentication no
PubkeyAuthentication yes
PermitRootLogin no
ChallengeResponseAuthentication no
UsePAM no

Restart the ssh service

$ sudo service ssh restart # or /etc/init.d/ssh restart

Miscs:

- SSH session disconnected and got stuck: type `Enter ~ .`

Multi-hop ssh e.g., ssh to user1@host1:port1 and then ssh to user2@host2:port2 (asuser1@host1:).

# This method uses ProxyJump
# add the following entry to file ~/.ssh/config
Host AName
Hostname host1
Port port1
User user1
ProxyJump user2@host2:port2

$ ssh user2@AName
# first password asked is for user1@host1, second password asked is for user2@host2.


# This second method uses `netcat` (the `nc` command) and ProxyCommand
# add the following entry to file ~/.ssh/config
Host AName
Hostname host1
ProxyCommand ssh -q user1@host1 -p port1 nc host2 port2
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null

$ ssh user2@AName
# first password asked is for user1@host1, second password asked is for user2@host2

using scp

scp -P port_number username@host:/path/to/file /destination/dir

using rsync resume file transfer

rsync --partial --progress --rsh=ssh user@host:/path/to/file .

using sshfs to mount remote directories > source: https://linuxize.com/post/how-to-use-sshfs-to-mount-remote-directories-over-ssh/
1. Install SSHFS
  - For Debian
```
$ sudo apt install sshf
```
  - For Mac
```
$ brew cask install osxfuse
$ brew install sshfs
```
- Mount remote directories (tip: use password-less SSH to avoid entering password) shell # sshfs [user@]host:[remote_directory] mountpoint [options] # E.g., $ sshfs username@hostname:/home/dir1 /home/dir1/dir2 $ sshfs localhost:/home/tnguyen/Dropbox local2222/ -C -p 2222

Files

move

#move with overwrite (and will not ask for confirmation)
mv -f

#do *not* overwrite (and will not ask for confirmation)
mv -n

sync

#sync dirs
#todir will become exactly like fromdir
rsync -val --delete --exclude "*.ext"  --exclude "dir" fromdir todir

#syn website
#I use the this command to synchronize my webpages to the university server
rsync -azv --delete --exclude=.hg --exclude=.hgcheck -e ssh ~/www [email protected]:/home/fac/tnguyen/public_html/

recursively change directory whose permision is 777 to 755
```
find . -type d -perm 777 -print -exec chmod 755 {} \;
```
recursively delete files with extensions *.ext
```
find -name \*.ext -delete
```

change permission

# change dir to 0755
$ find /path -type d -print0 | xargs -0 chmod 0755

# change file to 0644
$ find /path -type f -print0 | xargs -0 chmod 0644

Dropbox find conflicts

find ~/Dropbox/ -path "*(*'s conflicted copy [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]*" -print

grep

grep "error" file.txt | grep -v "multiple output" | grep -v "notexist"
grep -A 10 -B 10 "hello world" file.txt:  output surrounding lines

Misc

Convert JPG's to pdf: use the convert command from imagemagick, works with various types of files and not just jpg's.
```
$ convert -auto-orient *.jpg notes.pdf
```

Benchmarks

CPU

echo '2^2^25' | time -p bc > /dev/null
time -p echo "scale=5000; a(1)*4" | bc -l
sysbench --test=cpu --cpu-max-prime=20000 run #apt-get install sysbench

Keyboard

permanent switching capslock and control keys

#add/replace to /etc/default/keyboard
XKBOPTIONS="ctrl:nocaps" # Some people prefer "ctrl:swapcaps"
sudo dpkg-reconfigure -phigh console-setup

Emacs key in XFCE

$ xfconf-query -c xsettings -p /Gtk/KeyThemeName -s Emacs
OR
Menu -> Settings -> Settings Editor -> xsettings -> Gtk -> KeyThemeName,  enter Emacs in the field

Multimedia

Burn videos, e.g., mpg, to DVD (http://www.lamolabs.org/blog/author/slmingol/)

convert video with ffmpeg

ffmpeg -i input.m4v -target ntsc-dvd output.mpg

do authoring

dvdauthor --title -o dvd -f output.mpg
dvdauthor -o dvd -T
# NOTE: --title sets the title of the DVD, -T sets the table of contents. In both
#     above commands the -o switch is referencing a directory, NOT the actual dvd.

roll the .mpg file into an ISO file

mkisofs -dvd-video -o dvdimage.iso dvd
# NOTE: mkisofs is making an actual DVD video ISO file using the directory, dvd.

burn ISO to DVD

growisofs -speed=1 -dvd-compat -Z /dev/dvd=dvdimage.iso
NOTE: -speed=1 is for use with lower quality discs, increase as necessary
NOTE: This approach can be used to convert basically any format (m4v, mp4, etc.) to a DVD. Simply change the input file accordingly.

Update `date` on Debian

Note: ntpdate is deprecated, use ntp instead shell sudo service ntp stop sudo ntpd -gq sudo service ntp start
The -gq tells the ntp daemon to correct the time regardless of the offset (g) and exit immediately (q) after setting the time.

Users

Add user

#add user
sudo adduser username
#add user to group
sudo adduser username groupname

Add user to sudo shell sudo adduser username sudo
Change username from olduser to newuser. shell # make sure olduser is not logged in or using any process, e.g., restart and log in as another user. $ sudo usermod -l newuser olduser $ sudo groupmod -n newuser olduser $ sudo usermod -d /home/newuser -m newuser
Disable users shell passwd -l username to lock , passwd -u username to unlock

Web

Website

Password Protected Website

# To protect a directory called 'SecDir' on your website,
# go to SecDir and create/edit .htaccess file as follows:
AuthName "MySecDir"
AuthType Basic
AuthUserFile /path/to/.htpasswd
require valid-user

$ htpasswd -c /path/to/.htpasswd guestname #create a passwod
$ chmod 644 /path/to/.htpasswd   #change permission

# to log in SecDir online, enter username: guestname and passwod: yourpass

Webserver

Run a local server

$ cd /path/to/dir     
$ python -m http.server #optional, can also enter the port e.g., 8080

Table of Contents in Markdown

Use gh-md-toc, e.g., $ gh-md-toc --insert cmds.md

MAC OS

Creating an ISO from a CD/DVD e.g., to back up CD

- Insert CD
- open Disk Utils
- Select the CD (not the drive)
- Create File / New / Disk Image from ... / 
- Image Format:  CD/DVD Master  ...
- rename cdr to iso

Burn ISO image to USB (e.g., to install Linux distribution from USB)

First convert ISO to dmg file

$ hdiutil convert -format UDRW -o ~/file.img ~/file.iso     
...
Speed: 427.5Mbytes/sec     Savings: 0.0%     created: ~/file.img.dmg

Then find the USB device

$ diskutil list
/dev/disk0
#:             TYPE NAME          SIZE     IDENTIFIER
0:    GUID_partition_scheme            *320.1 GB   disk0
1:            EFI             209.7 MB   disk0s1
2:          Apple_HFS Mac           319.7 GB   disk0s2
/dev/disk1
#:             TYPE NAME          SIZE     IDENTIFIER
0:   FDisk_partition_scheme            *2.0 GB   disk1
1:       Windows_FAT_32             2.0 GB   disk1s1

$ diskutil unmountDisk /dev/disk1
Unmount of all volumes on disk1 was successful

Finally, copy data to USB disk

$ sudo dd if=/Users/tnguyen/file.img.dmg of=/dev/disk1 bs=1m
687+1 records in
687+1 records out
721127424 bytes transferred in 77.176835 secs (9343833 bytes/sec)

$ diskutil eject /dev/disk1
Disk /dev/disk1 ejected

Virtual Box

SSH from Host OS to Virtualbox (VB) Guest OS using port forwarding. For example, I run Linux (Debian) on my Mac and often ssh (from my Mac) to that Linux run.
1. The following assumes the VB Guest is called myserver. Also make sure the ssh server is installed on myserver.
2. The Guest OS, by default, should have one interface already which is using NAT. First go to the Network settings Advanced/Port Forwarding, and then add a new rule:
```
Host port 2222, guest port 22, name ssh, other left blank
```
  or from command line
```
HostOS$ VBoxManage modifyvm myserver --natpf1 "ssh,tcp,,2222,,22"
```
3. To check the added rules:
```
HostOS$ VBoxManage showvminfo myserver | grep 'Rule'
NIC 1 Rule(0):   name = ssh, protocol = tcp, host ip = , host port = 2222, guest ip = ..., guest port = 22
```
4. Log in the Guest OS HostOS$ ssh -p 2222 username@localhost where username is your user name in myserver.

Others:

Setup password-less login so that we don't have to enter the password everytime.

HostOS$ cat ~/.ssh/id_rsa.pub | ssh username@localhost -p 2222 'cat >> ~/.ssh/authorized_keys'

Copy files from host to guest using scp, or vice versa

#copy from host to guest
HostOS$ scp -P 2223 /path/to/file   fse16ae@localhost:/destination/dir

#copy from guest to host
HostOS$ scp -P 2222 user@localhost:/path/to/file /destination/dir

Note that to add another Guest OS, use the same technique but choose different port number on the host OS, e.g., 2223.

Start Headless

# List virtual machines
HostOS$ VBoxManage list vms

# Start VM in headless mode
HostOS$ VBoxManage startvm Debian7 --type headless

# Power off VM
HostOS$ VBoxManage controlvm Debian7 poweroff

Emacs

Useful keys

M-x load-file
C-x k : kill buffer
C-x C-x:  start a a location l,  search for something to go to new location,  then press C-x C-x to return back to l
M-x occur:  search all line with matched regexp
C-x C-r : read only
C-x C-v:  reload file (not technically but it works well)
C-c {  in latex mode to create {}
C-j  insert paragraph break
C-c C-s:  insert section subsection paragraph and label
C-c C-e: insert environment (e.g. figure, equation, list etc)
C-c C-f C-e:  emphasize
C-c %   or  ;  :  comment out the paragraph or line
C-c C-r:  on a highlighted region to latex only that region
C-c C-c:  to run latex.
C-c `:  keep on pressing this to review  errors after running latex
C-/:  undo
C-x k: kill buffer
M-/: autocomplete
M-h : mark entire paragraph
C-x h: mark entire buffer
C-t: transpose 2 letters  , e.g.  letet C-t  => lette
M-t: transpose 2 words,  e.g.,  world hello  M-t  ->  hello word  ;   vu, nguyen M-t  => nguyen, vu  (note: cursor btw the 2 words,  Alt-b to move back a word)
C-x C-t: transpose 2 lines
M-u : uppercase letters
M-c : capitalize letters
C-s C-w:  to the end of the word (handy)
Search and replace
C-s twice to redo the last  search
C-s C-w  search the word under the cursor

;; Windows resize
C-x + : make windows same height/width
C-x - : shrink window to fit content

M-x package-refresh-contents:  to refresh melpa content

;; Tramp: open remote files via ssh
open /ssh:user@host#port:/path

;; Tab
untabify / tabify to convert from tab to space and space to tab.

TeX

M-x align-current: To align columns in LaTeX table, highlight the table region, then M-x align-current

Install Flyspell on Mac

Install cocoAspell (which provides aspell)
Install the appropriate dictionary (./configure; make; make install)

TexLive

$ tlmgr update --self
$ tlmgr update --all # update TexLive packages installation (Mac)
$ tlmgr update --list  # see what will be upgraded

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Useful_Cmds

Useful Terminal Commands

APT

Development

JAVA

Administration

Email

RAID

GPG

Remote Login

SSH

Files

Misc

Benchmarks

Keyboard

Multimedia

Update `date` on Debian

Users

Web

Website

Webserver

Table of Contents in Markdown

MAC OS

Virtual Box

Emacs

Useful keys

TeX

Install Flyspell on Mac

TexLive

Clone this wiki locally

Useful_Cmds

Useful Terminal Commands

APT

Development

JAVA

Administration

Email

RAID

GPG

Remote Login

SSH

Files

Misc

Benchmarks

Keyboard

Multimedia

Update date on Debian

Users

Web

Website

Webserver

Table of Contents in Markdown

MAC OS

Virtual Box

Emacs

Useful keys

TeX

Install Flyspell on Mac

TexLive

Clone this wiki locally

Update `date` on Debian