diff --git a/.gitignore b/.gitignore index 7e99e367..e49c446e 100644 --- a/.gitignore +++ b/.gitignore @@ -1 +1,5 @@ -*.pyc \ No newline at end of file +*.pyc +*.swp +blockchain.cache +env +joinmarket.cfg diff --git a/.travis.yml b/.travis.yml new file mode 100644 index 00000000..24171656 --- /dev/null +++ b/.travis.yml @@ -0,0 +1,38 @@ +language: python +python: + - "2.7_with_system_site_packages" +before_install: + - sudo apt-add-repository ppa:bitcoin/bitcoin -y + - sudo apt-get update -q + - sudo apt-get install --no-install-recommends --no-upgrade -qq bitcoind + - sudo apt-get install -y build-essential + - sudo apt-get install -y automake +install: + - pip install numpy + - pip install pexpect +script: + - git clone git://github.com/jedisct1/libsodium.git + - cd libsodium + - git checkout tags/1.0.3 + - ./autogen.sh + - ./configure + - make check + - sudo make install + - cd .. +#set up joinmarket.cfg + - cp test/regtest_joinmarket.cfg joinmarket.cfg +#E2E encryption (libnacl) tests + - python lib/enc_wrapper.py +#start bitcoin regtest daemon and 101 blocks + - mkdir /home/travis/.bitcoin + - cp test/bitcoin.conf /home/travis/.bitcoin/. + - chmod 600 /home/travis/.bitcoin/bitcoin.conf + - bitcoind -regtest -daemon + - sleep 5 + - bitcoin-cli -regtest setgenerate true 101 + - cd test + - python regtest.py + - python wallet-test.py +branches: + only: + - master diff --git a/README.md b/README.md new file mode 100644 index 00000000..251750f4 --- /dev/null +++ b/README.md @@ -0,0 +1,45 @@ +##What is JoinMarket ? + +The idea behind JoinMarket is to allow users to have their bitcoins mixed with other JoinMarket users in return for a fee. A form of smart contract is created, meaning the private keys will never be broadcasted outside of your computer, resulting in virtually zero risk of loss (aside from malware or bugs). + +Simply put, JoinMarket allows its users to improve the privacy of their bitcoin transactions (and therefore maintaining or even restoring fungibility) in a decentralized fashion. On the other side, there are the JoinMarket operators who provide access to their bitcoins for others to use in mixing transactions. Their incentive is in the form of a fee in return for the provision of their bitcoins, meaning JoinMarket could become a form of passive income. + +As a result of free-market forces (i.e. anyone with bitcoins can become a JoinMarket operator), the fees will eventually be next to nothing. + +##Installation + +#####REQUIRED INSTALLATION DEPENDENCIES + ++ You will need python 2.7 + ++ You will need libsodium installed + + - You can get it here: http://doc.libsodium.org/ or through apt-get as `libsodium-dev` + + - Use this line to check it was installed correctly: `python lib/enc_wrapper.py` + ++ Matplotlib for displaying the graphs in orderbook-watcher (optional) + +###DEBIAN / UBUNTU QUICK INSTALL: + +1. `sudo apt-get update -y && sudo apt-get upgrade -y && sudo apt-get install python libsodium-dev -y` +2. `sudo apt-get install python-matplotlib -y` (optional) +3. `git clone https://github.com/chris-belcher/joinmarket.git` +4. Generating your first wallet will populate the configuration file: `joinmarket.cfg`. + Check if the default settings suit your needs. + +###[WIKI PAGES FOR DETAILED ARTICLES/GUIDES](https://github.com/chris-belcher/joinmarket/wiki) + +###[FOR WINDOWS](https://github.com/chris-belcher/joinmarket/wiki/Installing-JoinMarket-on-Windows-7-(temporary)) + +--- + ++ IRC: `#joinmarket` on irc.freenode.net https://webchat.freenode.net/?channels=%23joinmarket + ++ Bitcointalk thread: https://bitcointalk.org/index.php?topic=919116.msg10096563 + ++ Subreddit: https://reddit.com/r/joinmarket + ++ Twitter: https://twitter.com/joinmarket + ++ Donation address: `1AZgQZWYRteh6UyF87hwuvyWj73NvWKpL` diff --git a/README.txt b/README.txt deleted file mode 100644 index 07c9fd19..00000000 --- a/README.txt +++ /dev/null @@ -1,101 +0,0 @@ -FIRST IMPLEMENTATION OF JOINMARKET - -you will need to know python somewhat to play around with it - also get some testnet coins - -HOWTO try -1. use bip32-tool.py to output a bunch of addresses - send testnet coins to one mixing-depth=0 receive address - do this for two wallet seeds, one for each taker and maker - -2. open taker.py and maker.py and set the wallet seed for each - one in the source code - also for taker.py set the unspent transaction output (utxo) variable - for the coin you want to spend - -3. join irc.freenode.net #joinmarket and run both taker.py and maker.py - -4. when both bots join and have announced their orders, use this - command to start a coinjoining - !%fill [counterparty] [order-id] [cj-amount] - -so for example if the maker is called 'cj-maker' and you want to mix 1.9btc - !%fill cj-maker 0 190000000 - -all values are in satoshis, the first order has order-id 0 and it counts up - -5. watch the outputs of both bots, soon enough taker.py will say it has completed - a transaction, it will not do pushtx() but instead print the tx hex - you can examine this, with a blockchain explorer or my coin-jumble app and - push it to the network yourself, or not, whatever - -theres lots that needs to be done -some other notes below.. - -#COINJOIN PROTOCOL -#when a maker joins the channel it says out all its orders -# an order contains an order id, max size, min size, fee, whether the fee is absolute or -# as a proportion of the coinjoin amount -#when a taker joins the channel, it asks for orders to be pmed to him -#taker initiates coinjoin -#tells maker(s) by pm which order it wants to fill, sends the order id and the coinjoin amount -#maker(s) pm back the utxos they will input, and exactly two addresses, the coinjoin output and the change address -#taker collects all the utxos and outputs and makes a transaction -# pms them to the maker(s) who check everything is ok -# that the miner fee is right, that the cj fee is right -# and pm back signatures -# iv checked, it easily possible to put the signatures back into a tx -#taker then signs his own and pushtx() - -#TODO -#ask people on the testnet stuff to code up a few trading algos to see if the interface/protocol that -# iv invented is general enough - -#TODO think of names -#cj-market, cjex, but this isnt really an exchange -#Indra's Net -#If we now arbitrarily select one of these jewels for inspection and look closely at it, we will discover that in its polished surface there are reflected all the other jewels in the net, infinite in number. Not only that, but each of the jewels reflected in this one jewel is also reflecting all the other jewels, so that there is an infinite reflecting process occurring. -# but it sounds a bit like 'internet' with an accent -#maybe Indra, Indra's mixer -#other allusions, hall of mirrors, mirror labyrinth -#from discussing on irc, a simple name could just be JoinMarket or CoinJoinMarket -# JoinMarket seems the best probably - -#TODO dont always pick the lowest cost order, instead have an exponentially decaying -# distribution, so most of the time you pick the lowest and sometimes you take higher ones -# this represents your uncertainty in sybil attackers, the cheapest may not always be the best -#i.e. randomly chosen makers, weighted by the price they offer - -#TODO on nickname change, change also the counterparty variable in any open orders - -#TODO use electrum json_rpc instead of the pybitcointools stuff -# problem, i dont think that supports testnet -# bitcoind json_rpc obviously supports testnet, but someone else can download -# the blockchain - -#TODO option for how many blocks deep to wait before using a utxo for more mixing -# 1 confirm is probably enough - -#TODO encrypt messages between taker and maker, to stop trivial server eavesdropping -# but that wont stop mitm -# after chats on irc, easiest is to do Trust On First Use, maker sends a pubkey over -# TOFU requires a human to verify each first time, might not be practical -# also theres some algorithm for detecting mitm - -#TODO implement something against dust -# e.g. where the change address ends up having an output of value 1000 satoshis - -#TODO completely abstract away the irc stuff, so it can be switched to something else -# e.g. twitter but more likely darkwallet obelisk and/or electrum server - -#TODO add random delays to the orderbook stuff so there isnt such a traffic spike when a new bot joins - -#TODO make sure the outputs are in random order -# i.e. so its not like the taker always gets outputs 0,1 and maker 2,3 -#from random import shuffle - -#TODO error checking so you cant crash the bot by sending malformed orders - -#TODO make an ordertype where maker publishes the utxo he will use -# this is a way to auction off the use of a desirable coin, maybe a -# very newly mined coin or one which hasnt been moved for years diff --git a/bip32-tool.py b/bip32-tool.py deleted file mode 100644 index a5f3b3d1..00000000 --- a/bip32-tool.py +++ /dev/null @@ -1,93 +0,0 @@ - -import bitcoin as btc - -#structure for cj market wallet -# m/0/ root key -# m/0/n/ nth mixing depth, where n=0 is unmixed, n=1 is coinjoined once, etc -# pay in coins to mix at n=0 addresses -# coins move up a level when they are cj'd and stay at same level if they're the change from a coinjoin -# using coins from different levels as inputs to the same tx is probably detrimental to privacy -# m/0/n/0/k kth receive address, for mixing depth n -# m/0/n/1/k kth change address, for mixing depth n - - -seed = btc.sha256('dont use brainwallets') -#seed = '256 bits of randomness' - - -master = btc.bip32_master_key(seed)#, btc.TESTNET_PRIVATE) -print 'master = ' + master - -addr_vbyte = 0x6f #testnet - -m_0 = btc.bip32_ckd(master, 0) -for n in range(3): - print 'mixing depth ' + str(n) + ' m/0/' + str(n) + '/' - m_0_n = btc.bip32_ckd(m_0, n) - for forchange in range(2): - print(' ' + ('receive' if forchange==0 else 'change') + - ' addresses m/0/%d/%d/' % (n, forchange)) - m_0_n_c = btc.bip32_ckd(m_0_n, forchange) - for k in range(4): - m_0_n_c_k = btc.bip32_ckd(m_0_n_c, k) - priv = btc.bip32_extract_key(m_0_n_c_k) - print' m/0/%d/%d/%d/ ' % (n, forchange, k) + btc.privtoaddr(priv, addr_vbyte)# + ' ' + btc.encode_privkey(priv, 'wif') - - -''' -#default key on http://bip32.org/ -m_priv =\ -'xprv9s21ZrQH143K2JF8RafpqtKiTbsbaxEeUaMnNHsm5o6wCW3z8ySyH4UxFVSfZ8n7ESu7fgir8imbZKLYVBxFPND1pniTZ81vKfd45EHKX73' - -m_pub = btc.bip32_privtopub(m_priv) -print 'm_pub = ' + m_pub - -print 'prv(hex) = ' + btc.bip32_extract_key(m_priv) -print 'prv(wif) = ' + btc.encode_privkey(btc.bip32_extract_key(m_priv), 'wif_compressed') -print 'pub(hex) = ' + btc.bip32_extract_key(m_pub) -print 'addr = ' + btc.pubtoaddr(btc.bip32_extract_key(m_pub)) - -def print_pub_priv(prefix, priv): - pub = btc.bip32_privtopub(priv) - print prefix - print 'pub = ' + pub - print 'prv = ' + priv - -#bip32 test vector -print '\nbip32 test vector\n' -m_priv =\ -'xprv9s21ZrQH143K3QTDL4LXw2F7HEK3wJUD2nW2nRk4stbPy6cq3jPPqjiChkVvvNKmPGJxWUtg6LnF5kejMRNNU3TGtRBeJgk33yuGBxrMPHi' - -#i_H = i + 2**31 -chain_m_0h = btc.bip32_ckd(m_priv, 0 + 2**31) -print_pub_priv('chain m/0H', chain_m_0h) - -chain_m_0h_1 = btc.bip32_ckd(chain_m_0h, 1) -print_pub_priv('chain m/0H/1', chain_m_0h_1) - -chain_m_0h_1_2h = btc.bip32_ckd(chain_m_0h_1, 2 + 2**31) -print_pub_priv('chain m/0H/1/2H', chain_m_0h_1_2h) - - -#bip32 test vector 2 -print '\nbip32 test vector 2\n' -m_priv =\ -'xprv9s21ZrQH143K31xYSDQpPDxsXRTUcvj2iNHm5NUtrGiGG5e2DtALGdso3pGz6ssrdK4PFmM8NSpSBHNqPqm55Qn3LqFtT2emdEXVYsCzC2U' - -print 'master(hex) = ' + btc.bip32_extract_key(m_priv) - -chain_m_0 = btc.bip32_ckd(m_priv, 0) -print_pub_priv('chain m/0', chain_m_0) - -chain_m_0_214blahH = btc.bip32_ckd(chain_m_0, 2147483647 + 2**31) -print_pub_priv('chain m/0/2147483647H', chain_m_0_214blahH) - -chain_m_0_214blahH_1 = btc.bip32_ckd(chain_m_0_214blahH, 1) -print_pub_priv('chain m/0/2147483647H/1', chain_m_0_214blahH_1) -''' - -''' -seed = '256 bits of randomness' -m_priv = btc.bip32_master_key(seed) -print 'new seed = ' + m_priv -''' diff --git a/bitcoin/__init__.py b/bitcoin/__init__.py deleted file mode 100644 index cedc2c47..00000000 --- a/bitcoin/__init__.py +++ /dev/null @@ -1,6 +0,0 @@ -from main import * -from transaction import * -from deterministic import * -from bci import * -from composite import * -from stealth import * diff --git a/bitcoin/composite.py b/bitcoin/composite.py deleted file mode 100644 index dc234b8b..00000000 --- a/bitcoin/composite.py +++ /dev/null @@ -1,40 +0,0 @@ -from main import * -from transaction import * -from bci import * -from deterministic import * - - -# Takes privkey, address, value (satoshis), fee (satoshis) -def send(frm, to, value, fee=1000): - u = unspent(privtoaddr(frm)) - u2 = select(u, value+fee) - argz = u2 + [to+':'+str(value), privtoaddr(to), fee] - tx = mksend(argz) - tx2 = signall(tx, privtoaddr(to)) - pushtx(tx2) - - -def bip32_hdm_script(*args): - if len(args) == 3: - keys, req, path = args - else: - i, keys, path = 0, [], [] - while len(args[i]) > 40: - keys.append(args[i]) - i += 1 - req = int(args[i]) - path = map(int, args[i+1:]) - pubs = sorted(map(lambda x: bip32_descend(x, path), keys)) - return mk_multisig_script(pubs, req) - - -def bip32_hdm_addr(*args): - return scriptaddr(bip32_hdm_script(*args)) - - -def setup_coinvault_tx(tx, script): - txobj = deserialize(tx) - N = deserialize_script(script)[-2] - for inp in txobj["ins"]: - inp["script"] = serialize_script([None] * (N+1) + [script]) - return serialize(txobj) diff --git a/bitcoin/deterministic.py b/bitcoin/deterministic.py deleted file mode 100644 index ac463d5d..00000000 --- a/bitcoin/deterministic.py +++ /dev/null @@ -1,162 +0,0 @@ -from main import * -import hmac, hashlib - -### Electrum wallets - -def electrum_stretch(seed): return slowsha(seed) - -# Accepts seed or stretched seed, returns master public key -def electrum_mpk(seed): - if len(seed) == 32: seed = electrum_stretch(seed) - return privkey_to_pubkey(seed)[2:] - -# Accepts (seed or stretched seed), index and secondary index -# (conventionally 0 for ordinary addresses, 1 for change) , returns privkey -def electrum_privkey(seed,n,for_change=0): - if len(seed) == 32: seed = electrum_stretch(seed) - mpk = electrum_mpk(seed) - offset = dbl_sha256(str(n)+':'+str(for_change)+':'+mpk.decode('hex')) - return add_privkeys(seed, offset) - -# Accepts (seed or stretched seed or master public key), index and secondary index -# (conventionally 0 for ordinary addresses, 1 for change) , returns pubkey -def electrum_pubkey(masterkey,n,for_change=0): - if len(masterkey) == 32: mpk = electrum_mpk(electrum_stretch(masterkey)) - elif len(masterkey) == 64: mpk = electrum_mpk(masterkey) - else: mpk = masterkey - bin_mpk = encode_pubkey(mpk,'bin_electrum') - offset = bin_dbl_sha256(str(n)+':'+str(for_change)+':'+bin_mpk) - return add_pubkeys('04'+mpk,privtopub(offset)) - -# seed/stretched seed/pubkey -> address (convenience method) -def electrum_address(masterkey,n,for_change=0,version=0): - return pubkey_to_address(electrum_pubkey(masterkey,n,for_change),version) - -# Given a master public key, a private key from that wallet and its index, -# cracks the secret exponent which can be used to generate all other private -# keys in the wallet -def crack_electrum_wallet(mpk,pk,n,for_change=0): - bin_mpk = encode_pubkey(mpk,'bin_electrum') - offset = dbl_sha256(str(n)+':'+str(for_change)+':'+bin_mpk) - return subtract_privkeys(pk, offset) - -# Below code ASSUMES binary inputs and compressed pubkeys -PRIVATE = '\x04\x88\xAD\xE4' -PUBLIC = '\x04\x88\xB2\x1E' - -# BIP32 child key derivation -def raw_bip32_ckd(rawtuple, i): - vbytes, depth, fingerprint, oldi, chaincode, key = rawtuple - i = int(i) - - if vbytes == PRIVATE: - priv = key - pub = privtopub(key) - else: - pub = key - - if i >= 2**31: - if vbytes == PUBLIC: - raise Exception("Can't do private derivation on public key!") - I = hmac.new(chaincode,'\x00'+priv[:32]+encode(i,256,4),hashlib.sha512).digest() - else: - I = hmac.new(chaincode,pub+encode(i,256,4),hashlib.sha512).digest() - - if vbytes == PRIVATE: - newkey = add_privkeys(I[:32]+'\x01',priv) - fingerprint = bin_hash160(privtopub(key))[:4] - if vbytes == PUBLIC: - newkey = add_pubkeys(compress(privtopub(I[:32])),key) - fingerprint = bin_hash160(key)[:4] - - return (vbytes, depth + 1, fingerprint, i, I[32:], newkey) - -def bip32_serialize(rawtuple): - vbytes, depth, fingerprint, i, chaincode, key = rawtuple - depth = chr(depth % 256) - i = encode(i,256,4) - chaincode = encode(hash_to_int(chaincode),256,32) - keydata = '\x00'+key[:-1] if vbytes == PRIVATE else key - bindata = vbytes + depth + fingerprint + i + chaincode + keydata - return changebase(bindata+bin_dbl_sha256(bindata)[:4],256,58) - -def bip32_deserialize(data): - dbin = changebase(data,58,256) - if bin_dbl_sha256(dbin[:-4])[:4] != dbin[-4:]: - raise Exception("Invalid checksum") - vbytes = dbin[0:4] - depth = ord(dbin[4]) - fingerprint = dbin[5:9] - i = decode(dbin[9:13],256) - chaincode = dbin[13:45] - key = dbin[46:78]+'\x01' if vbytes == PRIVATE else dbin[45:78] - return (vbytes, depth, fingerprint, i, chaincode, key) - -def raw_bip32_privtopub(rawtuple): - vbytes, depth, fingerprint, i, chaincode, key = rawtuple - return (PUBLIC, depth, fingerprint, i, chaincode, privtopub(key)) - -def bip32_privtopub(data): - return bip32_serialize(raw_bip32_privtopub(bip32_deserialize(data))) - -def bip32_ckd(data,i): - return bip32_serialize(raw_bip32_ckd(bip32_deserialize(data),i)) - -def bip32_master_key(seed): - I = hmac.new("Bitcoin seed",seed,hashlib.sha512).digest() - return bip32_serialize((PRIVATE, 0, '\x00'*4, 0, I[32:], I[:32]+'\x01')) - -def bip32_bin_extract_key(data): - return bip32_deserialize(data)[-1] - -def bip32_extract_key(data): - return bip32_deserialize(data)[-1].encode('hex') - -# Exploits the same vulnerability as above in Electrum wallets -# Takes a BIP32 pubkey and one of the child privkeys of its corresponding privkey -# and returns the BIP32 privkey associated with that pubkey -def raw_crack_bip32_privkey(parent_pub,priv): - vbytes, depth, fingerprint, i, chaincode, key = priv - pvbytes, pdepth, pfingerprint, pi, pchaincode, pkey = parent_pub - i = int(i) - - if i >= 2**31: raise Exception("Can't crack private derivation!") - - I = hmac.new(pchaincode,pkey+encode(i,256,4),hashlib.sha512).digest() - - pprivkey = subtract_privkeys(key,I[:32]+'\x01') - - return (PRIVATE, pdepth, pfingerprint, pi, pchaincode, pprivkey) - -def crack_bip32_privkey(parent_pub,priv): - dsppub = bip32_deserialize(parent_pub) - dspriv = bip32_deserialize(priv) - return bip32_serialize(raw_crack_bip32_privkey(dsppub,dspriv)) - - -def coinvault_pub_to_bip32(*args): - if len(args) == 1: - args = args[0].split(' ') - vals = map(int, args[34:]) - I1 = ''.join(map(chr, vals[:33])) - I2 = ''.join(map(chr, vals[35:67])) - return bip32_serialize((PUBLIC, 0, '\x00'*4, 0, I2, I1)) - - -def coinvault_priv_to_bip32(*args): - if len(args) == 1: - args = args[0].split(' ') - vals = map(int, args[34:]) - I2 = ''.join(map(chr, vals[35:67])) - I3 = ''.join(map(chr, vals[72:104])) - return bip32_serialize((PRIVATE, 0, '\x00'*4, 0, I2, I3+'\x01')) - - -def bip32_descend(*args): - if len(args) == 2: - key, path = args - else: - key, path = args[0], map(int, args[1:]) - for p in path: - key = bip32_ckd(key, p) - return bip32_extract_key(key) diff --git a/bitcoin/main.py b/bitcoin/main.py deleted file mode 100644 index c5358950..00000000 --- a/bitcoin/main.py +++ /dev/null @@ -1,431 +0,0 @@ -#!/usr/bin/python -import hashlib, re, sys, os, base64, time, random, hmac -import ripemd - -### Elliptic curve parameters (secp256k1) - -P = 2**256-2**32-2**9-2**8-2**7-2**6-2**4-1 -N = 115792089237316195423570985008687907852837564279074904382605163141518161494337 -A = 0 -B = 7 -Gx = 55066263022277343669578718895168534326250603453777594175500187360389116729240 -Gy = 32670510020758816978083085130507043184471273380659243275938904335757337482424 -G = (Gx,Gy) - - -def change_curve(p, n, a, b, gx, gy): - global P, N, A, B, Gx, Gy, G - P, N, A, B, Gx, Gy = p, n, a, b, gx, gy - G = (Gx, Gy) - -def getG(): - return G - -### Extended Euclidean Algorithm - -def inv(a,n): - lm, hm = 1,0 - low, high = a%n,n - while low > 1: - r = high/low - nm, new = hm-lm*r, high-low*r - lm, low, hm, high = nm, new, lm, low - return lm % n - -### Base switching - -def get_code_string(base): - if base == 2: return '01' - elif base == 10: return '0123456789' - elif base == 16: return '0123456789abcdef' - elif base == 32: return 'abcdefghijklmnopqrstuvwxyz234567' - elif base == 58: return '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz' - elif base == 256: return ''.join([chr(x) for x in range(256)]) - else: raise ValueError("Invalid base!") - -def lpad(msg,symbol,length): - if len(msg) >= length: return msg - return symbol * (length - len(msg)) + msg - -def encode(val,base,minlen=0): - base, minlen = int(base), int(minlen) - code_string = get_code_string(base) - result = "" - while val > 0: - result = code_string[val % base] + result - val /= base - return lpad(result,code_string[0],minlen) - -def decode(string,base): - base = int(base) - code_string = get_code_string(base) - result = 0 - if base == 16: string = string.lower() - while len(string) > 0: - result *= base - result += code_string.find(string[0]) - string = string[1:] - return result - -def changebase(string,frm,to,minlen=0): - if frm == to: return lpad(string,get_code_string(frm)[0],minlen) - return encode(decode(string,frm),to,minlen) - -### JSON access (for pybtctool convenience) - -def access(obj,prop): - if isinstance(obj,dict): - if prop in obj: return obj[prop] - elif '.' in prop: return obj[float(prop)] - else: return obj[int(prop)] - else: - return obj[int(prop)] - -def multiaccess(obj,prop): - return [access(o,prop) for o in obj] - -def slice(obj,start=0,end=2**200): - return obj[int(start):int(end)] - -def count(obj): - return len(obj) - -_sum = sum -def sum(obj): - return _sum(obj) - -### Elliptic Curve functions - -def isinf(p): return p[0] == 0 and p[1] == 0 - -def base10_add(a,b): - if isinf(a): return b[0],b[1] - if isinf(b): return a[0],a[1] - if a[0] == b[0]: - if a[1] == b[1]: return base10_double((a[0],a[1])) - else: return (0,0) - m = ((b[1]-a[1]) * inv(b[0]-a[0],P)) % P - x = (m*m-a[0]-b[0]) % P - y = (m*(a[0]-x)-a[1]) % P - return (x,y) - -def base10_double(a): - if isinf(a): return (0,0) - m = ((3*a[0]*a[0]+A)*inv(2*a[1],P)) % P - x = (m*m-2*a[0]) % P - y = (m*(a[0]-x)-a[1]) % P - return (x,y) - -def base10_multiply(a,n): - if isinf(a) or n == 0: return (0,0) - if n == 1: return a - if n < 0 or n >= N: return base10_multiply(a,n%N) - if (n%2) == 0: return base10_double(base10_multiply(a,n/2)) - if (n%2) == 1: return base10_add(base10_double(base10_multiply(a,n/2)),a) - -# Functions for handling pubkey and privkey formats - -def get_pubkey_format(pub): - if isinstance(pub,(tuple,list)): return 'decimal' - elif len(pub) == 65 and pub[0] == '\x04': return 'bin' - elif len(pub) == 130 and pub[0:2] == '04': return 'hex' - elif len(pub) == 33 and pub[0] in ['\x02','\x03']: return 'bin_compressed' - elif len(pub) == 66 and pub[0:2] in ['02','03']: return 'hex_compressed' - elif len(pub) == 64: return 'bin_electrum' - elif len(pub) == 128: return 'hex_electrum' - else: raise Exception("Pubkey not in recognized format") - -def encode_pubkey(pub,formt): - if not isinstance(pub,(tuple,list)): - pub = decode_pubkey(pub) - if formt == 'decimal': return pub - elif formt == 'bin': return '\x04' + encode(pub[0],256,32) + encode(pub[1],256,32) - elif formt == 'bin_compressed': return chr(2+(pub[1]%2)) + encode(pub[0],256,32) - elif formt == 'hex': return '04' + encode(pub[0],16,64) + encode(pub[1],16,64) - elif formt == 'hex_compressed': return '0'+str(2+(pub[1]%2)) + encode(pub[0],16,64) - elif formt == 'bin_electrum': return encode(pub[0],256,32) + encode(pub[1],256,32) - elif formt == 'hex_electrum': return encode(pub[0],16,64) + encode(pub[1],16,64) - else: raise Exception("Invalid format!") - -def decode_pubkey(pub,formt=None): - if not formt: formt = get_pubkey_format(pub) - if formt == 'decimal': return pub - elif formt == 'bin': return (decode(pub[1:33],256),decode(pub[33:65],256)) - elif formt == 'bin_compressed': - x = decode(pub[1:33],256) - beta = pow(x*x*x+A*x+B,(P+1)/4,P) - y = (P-beta) if ((beta + ord(pub[0])) % 2) else beta - return (x,y) - elif formt == 'hex': return (decode(pub[2:66],16),decode(pub[66:130],16)) - elif formt == 'hex_compressed': - return decode_pubkey(pub.decode('hex'),'bin_compressed') - elif formt == 'bin_electrum': - return (decode(pub[:32],256),decode(pub[32:64],256)) - elif formt == 'hex_electrum': - return (decode(pub[:64],16),decode(pub[64:128],16)) - else: raise Exception("Invalid format!") - -def get_privkey_format(priv): - if isinstance(priv,(int,long)): return 'decimal' - elif len(priv) == 32: return 'bin' - elif len(priv) == 33: return 'bin_compressed' - elif len(priv) == 64: return 'hex' - elif len(priv) == 66: return 'hex_compressed' - else: - bin_p = b58check_to_bin(priv) - if len(bin_p) == 32: return 'wif' - elif len(bin_p) == 33: return 'wif_compressed' - else: raise Exception("WIF does not represent privkey") - -def encode_privkey(priv,formt,vbyte=0): - if not isinstance(priv,(int,long)): - return encode_privkey(decode_privkey(priv),formt,vbyte) - if formt == 'decimal': return priv - elif formt == 'bin': return encode(priv,256,32) - elif formt == 'bin_compressed': return encode(priv,256,32)+'\x01' - elif formt == 'hex': return encode(priv,16,64) - elif formt == 'hex_compressed': return encode(priv,16,64)+'01' - elif formt == 'wif': - return bin_to_b58check(encode(priv,256,32),128+int(vbyte)) - elif formt == 'wif_compressed': - return bin_to_b58check(encode(priv,256,32)+'\x01',128+int(vbyte)) - else: raise Exception("Invalid format!") - -def decode_privkey(priv,formt=None): - if not formt: formt = get_privkey_format(priv) - if formt == 'decimal': return priv - elif formt == 'bin': return decode(priv,256) - elif formt == 'bin_compressed': return decode(priv[:32],256) - elif formt == 'hex': return decode(priv,16) - elif formt == 'hex_compressed': return decode(priv[:64],16) - else: - bin_p = b58check_to_bin(priv) - if len(bin_p) == 32: return decode(bin_p,256) - elif len(bin_p) == 33: return decode(bin_p[:32],256) - else: raise Exception("WIF does not represent privkey") - -def add_pubkeys(p1,p2): - f1,f2 = get_pubkey_format(p1), get_pubkey_format(p2) - return encode_pubkey(base10_add(decode_pubkey(p1,f1),decode_pubkey(p2,f2)),f1) - -def add_privkeys(p1,p2): - f1,f2 = get_privkey_format(p1), get_privkey_format(p2) - return encode_privkey((decode_privkey(p1,f1) + decode_privkey(p2,f2)) % N,f1) - -def multiply(pubkey,privkey): - f1,f2 = get_pubkey_format(pubkey), get_privkey_format(privkey) - pubkey, privkey = decode_pubkey(pubkey,f1), decode_privkey(privkey,f2) - # http://safecurves.cr.yp.to/twist.html - if not isinf(pubkey) and (pubkey[0]**3+B-pubkey[1]*pubkey[1]) % P != 0: - raise Exception("Point not on curve") - return encode_pubkey(base10_multiply(pubkey,privkey),f1) - -def divide(pubkey,privkey): - factor = inv(decode_privkey(privkey),N) - return multiply(pubkey,factor) - -def compress(pubkey): - f = get_pubkey_format(pubkey) - if 'compressed' in f: return pubkey - elif f == 'bin': return encode_pubkey(decode_pubkey(pubkey,f),'bin_compressed') - elif f == 'hex' or f == 'decimal': - return encode_pubkey(decode_pubkey(pubkey,f),'hex_compressed') - -def decompress(pubkey): - f = get_pubkey_format(pubkey) - if 'compressed' not in f: return pubkey - elif f == 'bin_compressed': return encode_pubkey(decode_pubkey(pubkey,f),'bin') - elif f == 'hex_compressed' or f == 'decimal': - return encode_pubkey(decode_pubkey(pubkey,f),'hex') - -def privkey_to_pubkey(privkey): - f = get_privkey_format(privkey) - privkey = decode_privkey(privkey,f) - if privkey == 0 or privkey >= N: - raise Exception("Invalid privkey") - if f in ['bin','bin_compressed','hex','hex_compressed','decimal']: - return encode_pubkey(base10_multiply(G,privkey),f) - else: - return encode_pubkey(base10_multiply(G,privkey),f.replace('wif','hex')) - -privtopub = privkey_to_pubkey - -def privkey_to_address(priv,magicbyte=0): - return pubkey_to_address(privkey_to_pubkey(priv),magicbyte) -privtoaddr = privkey_to_address - -def neg_pubkey(pubkey): - f = get_pubkey_format(pubkey) - pubkey = decode_pubkey(pubkey,f) - return encode_pubkey((pubkey[0],(P-pubkey[1]) % P),f) - -def neg_privkey(privkey): - f = get_privkey_format(privkey) - privkey = decode_privkey(privkey,f) - return encode_privkey((N - privkey) % N,f) - -def subtract_pubkeys(p1, p2): - f1,f2 = get_pubkey_format(p1), get_pubkey_format(p2) - k2 = decode_pubkey(p2,f2) - return encode_pubkey(base10_add(decode_pubkey(p1,f1),(k2[0],(P - k2[1]) % P)),f1) - -def subtract_privkeys(p1, p2): - f1,f2 = get_privkey_format(p1), get_privkey_format(p2) - k2 = decode_privkey(p2,f2) - return encode_privkey((decode_privkey(p1,f1) - k2) % N,f1) - -### Hashes - -def bin_hash160(string): - intermed = hashlib.sha256(string).digest() - digest = '' - try: - digest = hashlib.new('ripemd160',intermed).digest() - except: - digest = ripemd.RIPEMD160(intermed).digest() - return digest -def hash160(string): - return bin_hash160(string).encode('hex') - -def bin_sha256(string): - return hashlib.sha256(string).digest() -def sha256(string): - return bin_sha256(string).encode('hex') - -def bin_dbl_sha256(string): - return hashlib.sha256(hashlib.sha256(string).digest()).digest() -def dbl_sha256(string): - return bin_dbl_sha256(string).encode('hex') - -def bin_slowsha(string): - orig_input = string - for i in range(100000): - string = hashlib.sha256(string + orig_input).digest() - return string -def slowsha(string): - return bin_slowsha(string).encode('hex') - -def hash_to_int(x): - if len(x) in [40,64]: return decode(x,16) - else: return decode(x,256) - -def num_to_var_int(x): - x = int(x) - if x < 253: return chr(x) - elif x < 65536: return chr(253) + encode(x,256,2)[::-1] - elif x < 4294967296: return chr(254) + encode(x,256,4)[::-1] - else: return chr(255) + encode(x,256,8)[::-1] - -# WTF, Electrum? -def electrum_sig_hash(message): - padded = "\x18Bitcoin Signed Message:\n" + num_to_var_int( len(message) ) + message - return bin_dbl_sha256(padded) - -def random_key(): - # Gotta be secure after that java.SecureRandom fiasco... - entropy = os.urandom(32)+str(random.randrange(2**256))+str(int(time.time())**7) - return sha256(entropy) - -def random_electrum_seed(): - entropy = os.urandom(32)+str(random.randrange(2**256))+str(int(time.time())**7) - return sha256(entropy)[:32] - -### Encodings - -def bin_to_b58check(inp,magicbyte=0): - inp_fmtd = chr(int(magicbyte)) + inp - leadingzbytes = len(re.match('^\x00*',inp_fmtd).group(0)) - checksum = bin_dbl_sha256(inp_fmtd)[:4] - return '1' * leadingzbytes + changebase(inp_fmtd+checksum,256,58) - -def b58check_to_bin(inp): - leadingzbytes = len(re.match('^1*',inp).group(0)) - data = '\x00' * leadingzbytes + changebase(inp,58,256) - assert bin_dbl_sha256(data[:-4])[:4] == data[-4:] - return data[1:-4] - -def get_version_byte(inp): - leadingzbytes = len(re.match('^1*',inp).group(0)) - data = '\x00' * leadingzbytes + changebase(inp,58,256) - assert bin_dbl_sha256(data[:-4])[:4] == data[-4:] - return ord(data[0]) - -def hex_to_b58check(inp,magicbyte=0): - return bin_to_b58check(inp.decode('hex'),magicbyte) - -def b58check_to_hex(inp): return b58check_to_bin(inp).encode('hex') - -def pubkey_to_address(pubkey,magicbyte=0): - if isinstance(pubkey,(list,tuple)): - pubkey = encode_pubkey(pubkey,'bin') - if len(pubkey) in [66,130]: - return bin_to_b58check(bin_hash160(pubkey.decode('hex')),magicbyte) - return bin_to_b58check(bin_hash160(pubkey),magicbyte) - -pubtoaddr = pubkey_to_address - -### EDCSA - -def encode_sig(v,r,s): - vb, rb, sb = chr(v), encode(r,256), encode(s,256) - return base64.b64encode(vb+'\x00'*(32-len(rb))+rb+'\x00'*(32-len(sb))+sb) - -def decode_sig(sig): - bytez = base64.b64decode(sig) - return ord(bytez[0]), decode(bytez[1:33],256), decode(bytez[33:],256) - -# https://tools.ietf.org/html/rfc6979#section-3.2 -def deterministic_generate_k(msghash,priv): - v = '\x01' * 32 - k = '\x00' * 32 - priv = encode_privkey(priv,'bin') - msghash = encode(hash_to_int(msghash),256,32) - k = hmac.new(k, v+'\x00'+priv+msghash, hashlib.sha256).digest() - v = hmac.new(k, v, hashlib.sha256).digest() - k = hmac.new(k, v+'\x01'+priv+msghash, hashlib.sha256).digest() - v = hmac.new(k, v, hashlib.sha256).digest() - return decode(hmac.new(k, v, hashlib.sha256).digest(),256) - -def ecdsa_raw_sign(msghash,priv): - - z = hash_to_int(msghash) - k = deterministic_generate_k(msghash,priv) - - r,y = base10_multiply(G,k) - s = inv(k,N) * (z + r*decode_privkey(priv)) % N - - return 27+(y%2),r,s - -def ecdsa_sign(msg,priv): - return encode_sig(*ecdsa_raw_sign(electrum_sig_hash(msg),priv)) - -def ecdsa_raw_verify(msghash,vrs,pub): - v,r,s = vrs - - w = inv(s,N) - z = hash_to_int(msghash) - - u1, u2 = z*w % N, r*w % N - x,y = base10_add(base10_multiply(G,u1), base10_multiply(decode_pubkey(pub),u2)) - - return r == x - -def ecdsa_verify(msg,sig,pub): - return ecdsa_raw_verify(electrum_sig_hash(msg),decode_sig(sig),pub) - -def ecdsa_raw_recover(msghash,vrs): - v,r,s = vrs - - x = r - beta = pow(x*x*x+A*x+B,(P+1)/4,P) - y = beta if v%2 ^ beta%2 else (P - beta) - z = hash_to_int(msghash) - - Qr = base10_add(neg_pubkey(base10_multiply(G,z)),base10_multiply((x,y),s)) - Q = base10_multiply(Qr,inv(r,N)) - - if ecdsa_raw_verify(msghash,vrs,Q): return Q - return False - -def ecdsa_recover(msg,sig): - return encode_pubkey(ecdsa_raw_recover(electrum_sig_hash(msg),decode_sig(sig)),'hex') diff --git a/bitcoin/transaction.py b/bitcoin/transaction.py deleted file mode 100644 index 26ca1525..00000000 --- a/bitcoin/transaction.py +++ /dev/null @@ -1,380 +0,0 @@ -#!/usr/bin/python -import re, json, copy, sys -from main import * - -### Hex to bin converter and vice versa for objects - -def json_is_base(obj,base): - alpha = get_code_string(base) - if isinstance(obj,(str,unicode)): - for i in range(len(obj)): - if alpha.find(obj[i]) == -1: return False - return True - elif isinstance(obj,(int,float,long)) or obj is None: return True - elif isinstance(obj,list): - for i in range(len(obj)): - if not json_is_base(obj[i],base): return False - return True - else: - for x in obj: - if not json_is_base(obj[x],base): return False - return True - -def json_changebase(obj,changer): - if isinstance(obj,(str,unicode)): return changer(obj) - elif isinstance(obj,(int,float,long)) or obj is None: return obj - elif isinstance(obj,list): return [json_changebase(x,changer) for x in obj] - return dict((x, json_changebase(obj[x], changer)) for x in obj) - -### Transaction serialization and deserialization - -def deserialize(tx): - if re.match('^[0-9a-fA-F]*$',tx): - return json_changebase(deserialize(tx.decode('hex')),lambda x:x.encode('hex')) - # http://stackoverflow.com/questions/4851463/python-closure-write-to-variable-in-parent-scope - # Python's scoping rules are demented, requiring me to make pos an object so that it is call-by-reference - pos = [0] - - def read_as_int(bytez): - pos[0] += bytez - return decode(tx[pos[0]-bytez:pos[0]][::-1],256) - - def read_var_int(): - pos[0] += 1 - if ord(tx[pos[0]-1]) < 253: return ord(tx[pos[0]-1]) - return read_as_int(pow(2,ord(tx[pos[0]-1]) - 252)) - - def read_bytes(bytez): - pos[0] += bytez - return tx[pos[0]-bytez:pos[0]] - - def read_var_string(): - size = read_var_int() - return read_bytes(size) - - obj = { "ins" : [] , "outs" : [] } - obj["version"] = read_as_int(4) - ins = read_var_int() - for i in range(ins): - obj["ins"].append({ - "outpoint" : { - "hash" : read_bytes(32)[::-1], - "index": read_as_int(4) - }, - "script" : read_var_string(), - "sequence" : read_as_int(4) - }) - outs = read_var_int() - for i in range(outs): - obj["outs"].append({ - "value" : read_as_int(8), - "script": read_var_string() - }) - obj["locktime"] = read_as_int(4) - return obj - -def serialize(txobj): - o = [] - if json_is_base(txobj,16): - return serialize(json_changebase(txobj,lambda x: x.decode('hex'))).encode('hex') - o.append(encode(txobj["version"],256,4)[::-1]) - o.append(num_to_var_int(len(txobj["ins"]))) - for inp in txobj["ins"]: - o.append(inp["outpoint"]["hash"][::-1]) - o.append(encode(inp["outpoint"]["index"],256,4)[::-1]) - o.append(num_to_var_int(len(inp["script"]))+inp["script"]) - o.append(encode(inp["sequence"],256,4)[::-1]) - o.append(num_to_var_int(len(txobj["outs"]))) - for out in txobj["outs"]: - o.append(encode(out["value"],256,8)[::-1]) - o.append(num_to_var_int(len(out["script"]))+out["script"]) - o.append(encode(txobj["locktime"],256,4)[::-1]) - return ''.join(o) - -### Hashing transactions for signing - -SIGHASH_ALL = 1 -SIGHASH_NONE = 2 -SIGHASH_SINGLE = 3 -SIGHASH_ANYONECANPAY = 80 - -def signature_form(tx, i, script, hashcode = SIGHASH_ALL): - i, hashcode = int(i), int(hashcode) - if isinstance(tx,str): - return serialize(signature_form(deserialize(tx),i,script,hashcode)) - newtx = copy.deepcopy(tx) - for inp in newtx["ins"]: inp["script"] = "" - newtx["ins"][i]["script"] = script - if hashcode == SIGHASH_NONE: - newtx["outs"] = [] - elif hashcode == SIGHASH_SINGLE: - newtx["outs"] = newtx["outs"][:len(newtx["ins"])] - for out in range(len(newtx["ins"]) - 1): - out.value = 2**64 - 1 - out.script = "" - elif hashcode == SIGHASH_ANYONECANPAY: - newtx["ins"] = [newtx["ins"][i]] - else: - pass - return newtx - -### Making the actual signatures - -def der_encode_sig(v,r,s): - b1, b2 = encode(r,256).encode('hex'), encode(s,256).encode('hex') - if r >= 2**255: b1 = '00' + b1 - if s >= 2**255: b2 = '00' + b2 - left = '02'+encode(len(b1)/2,16,2)+b1 - right = '02'+encode(len(b2)/2,16,2)+b2 - return '30'+encode(len(left+right)/2,16,2)+left+right - -def der_decode_sig(sig): - leftlen = decode(sig[6:8],16)*2 - left = sig[8:8+leftlen] - rightlen = decode(sig[10+leftlen:12+leftlen],16)*2 - right = sig[12+leftlen:12+leftlen+rightlen] - return (None,decode(left,16),decode(right,16)) - -def txhash(tx,hashcode=None): - if re.match('^[0-9a-fA-F]*$',tx): - tx = changebase(tx,16,256) - if hashcode: return dbl_sha256(tx + encode(int(hashcode),256,4)[::-1]) - else: return bin_dbl_sha256(tx)[::-1].encode('hex') - -def bin_txhash(tx,hashcode=None): - return txhash(tx,hashcode).decode('hex') - -def ecdsa_tx_sign(tx,priv,hashcode=SIGHASH_ALL): - rawsig = ecdsa_raw_sign(bin_txhash(tx,hashcode),priv) - return der_encode_sig(*rawsig)+encode(hashcode,16,2) - -def ecdsa_tx_verify(tx,sig,pub,hashcode=SIGHASH_ALL): - return ecdsa_raw_verify(bin_txhash(tx,hashcode),der_decode_sig(sig),pub) - -def ecdsa_tx_recover(tx,sig,hashcode=SIGHASH_ALL): - z = bin_txhash(tx,hashcode) - _,r,s = der_decode_sig(sig) - left = ecdsa_raw_recover(z,(0,r,s)) - right = ecdsa_raw_recover(z,(1,r,s)) - return (encode_pubkey(left,'hex'), encode_pubkey(right,'hex')) - -### Scripts - -def mk_pubkey_script(addr): # Keep the auxiliary functions around for altcoins' sake - return '76a914' + b58check_to_hex(addr) + '88ac' - -def mk_scripthash_script(addr): - return 'a914' + b58check_to_hex(addr) + '87' - -# Address representation to output script -def address_to_script(addr): - if addr[0] == '3' or addr[0] == '2': return mk_scripthash_script(addr) - else: return mk_pubkey_script(addr) - -# Output script to address representation -def script_to_address(script,vbyte=0): - if re.match('^[0-9a-fA-F]*$',script): - script = script.decode('hex') - if script[:3] == '\x76\xa9\x14' and script[-2:] == '\x88\xac' and len(script) == 25: - return bin_to_b58check(script[3:-2],vbyte) # pubkey hash addresses - else: - if vbyte == 111: - # Testnet - scripthash_byte = 192 - else: - scripthash_byte = 5 - - return bin_to_b58check(script[2:-1],scripthash_byte) # BIP0016 scripthash addresses - - -def p2sh_scriptaddr(script, magicbyte=5): - if re.match('^[0-9a-fA-F]*$', script): - script = script.decode('hex') - return hex_to_b58check(hash160(script), magicbyte) -scriptaddr = p2sh_scriptaddr - - -def deserialize_script(script): - if re.match('^[0-9a-fA-F]*$',script): - return json_changebase(deserialize_script(script.decode('hex')),lambda x:x.encode('hex')) - out, pos = [], 0 - while pos < len(script): - code = ord(script[pos]) - if code == 0: - out.append(None) - pos += 1 - elif code <= 75: - out.append(script[pos+1:pos+1+code]) - pos += 1 + code - elif code <= 78: - szsz = pow(2,code - 76) - sz = decode(script[pos + szsz : pos : -1],256) - out.append(script[pos + 1 + szsz:pos + 1 + szsz + sz]) - pos += 1 + szsz + sz - elif code <= 96: - out.append(code - 80) - pos += 1 - else: - out.append(code) - pos += 1 - return out - -def serialize_script_unit(unit): - if isinstance(unit,int): - if unit < 16: return chr(unit + 80) - else: return chr(unit) - elif unit is None: - return '\x00' - else: - if len(unit) <= 75: return chr(len(unit))+unit - elif len(unit) < 256: return chr(76)+chr(len(unit))+unit - elif len(unit) < 65536: return chr(77)+encode(len(unit),256,2)[::-1]+unit - else: return chr(78)+encode(len(unit),256,4)[::-1]+unit - -def serialize_script(script): - if json_is_base(script,16): - return serialize_script(json_changebase(script,lambda x:x.decode('hex'))).encode('hex') - return ''.join(map(serialize_script_unit,script)) - - -def mk_multisig_script(*args): # [pubs],k or pub1,pub2...pub[n],k - if isinstance(args[0], list): - pubs, k = args[0], int(args[1]) - else: - pubs = filter(lambda x: len(str(x)) >= 32, args) - k = args[pubs] - return serialize_script([k]+pubs+[len(pubs), 174]) - -### Signing and verifying - -def verify_tx_input(tx,i,script,sig,pub): - if re.match('^[0-9a-fA-F]*$',tx): tx = tx.decode('hex') - if re.match('^[0-9a-fA-F]*$',script): script = script.decode('hex') - if not re.match('^[0-9a-fA-F]*$',sig): sig = sig.encode('hex') - hashcode = decode(sig[-2:],16) - modtx = signature_form(tx,int(i),script,hashcode) - return ecdsa_tx_verify(modtx,sig,pub,hashcode) - -def sign(tx,i,priv): - i = int(i) - if not re.match('^[0-9a-fA-F]*$',tx): - return sign(tx.encode('hex'),i,priv).decode('hex') - if len(priv) <= 33: priv = priv.encode('hex') - pub = privkey_to_pubkey(priv) - address = pubkey_to_address(pub) - signing_tx = signature_form(tx,i,mk_pubkey_script(address)) - sig = ecdsa_tx_sign(signing_tx,priv) - txobj = deserialize(tx) - txobj["ins"][i]["script"] = serialize_script([sig,pub]) - return serialize(txobj) - -def signall(tx,priv): - for i in range(len(deserialize(tx)["ins"])): - tx = sign(tx,i,priv) - return tx - -def multisign(tx,i,script,pk,hashcode = SIGHASH_ALL): - if re.match('^[0-9a-fA-F]*$',tx): tx = tx.decode('hex') - if re.match('^[0-9a-fA-F]*$',script): script = script.decode('hex') - modtx = signature_form(tx,i,script,hashcode) - return ecdsa_tx_sign(modtx,pk,hashcode) - -def apply_multisignatures(*args): # tx,i,script,sigs OR tx,i,script,sig1,sig2...,sig[n] - tx, i, script = args[0], int(args[1]), args[2] - sigs = args[3] if isinstance(args[3],list) else list(args[3:]) - - if re.match('^[0-9a-fA-F]*$',script): script = script.decode('hex') - sigs = [x.decode('hex') if x[:2] == '30' else x for x in sigs] - if re.match('^[0-9a-fA-F]*$',tx): - return apply_multisignatures(tx.decode('hex'),i,script,sigs).encode('hex') - - txobj = deserialize(tx) - txobj["ins"][i]["script"] = serialize_script([None]+sigs+[script]) - return serialize(txobj) - -def is_inp(arg): - return len(arg) > 64 or "output" in arg or "outpoint" in arg - -def mktx(*args): # [in0, in1...],[out0, out1...] or in0, in1 ... out0 out1 ... - ins, outs = [], [] - for arg in args: - if isinstance(arg,list): - for a in arg: (ins if is_inp(a) else outs).append(a) - else: - (ins if is_inp(arg) else outs).append(arg) - - txobj = { "locktime" : 0, "version" : 1,"ins" : [], "outs" : [] } - for i in ins: - if isinstance(i,dict) and "outpoint" in i: - txobj["ins"].append(i) - else: - if isinstance(i,dict) and "output" in i: i = i["output"] - txobj["ins"].append({ - "outpoint" : { "hash": i[:64], "index": int(i[65:]) }, - "script": "", - "sequence": 4294967295 - }) - for o in outs: - if isinstance(o,str): - addr = o[:o.find(':')] - val = int(o[o.find(':')+1:]) - o = {} - if re.match('^[0-9a-fA-F]*$',addr): - o["script"] = addr - else: - o["address"] = addr - o["value"] = val - - outobj = {} - if "address" in o: - outobj["script"] = address_to_script(o["address"]) - elif "script" in o: - outobj["script"] = o["script"] - else: - raise Exception("Could not find 'address' or 'script' in output.") - outobj["value"] = o["value"] - txobj["outs"].append(outobj) - - return serialize(txobj) - -def select(unspent,value): - value = int(value) - high = [u for u in unspent if u["value"] >= value] - high.sort(key=lambda u:u["value"]) - low = [u for u in unspent if u["value"] < value] - low.sort(key=lambda u:-u["value"]) - if len(high): return [high[0]] - i, tv = 0, 0 - while tv < value and i < len(low): - tv += low[i]["value"] - i += 1 - if tv < value: raise Exception("Not enough funds") - return low[:i] - -# Only takes inputs of the form { "output": blah, "value": foo } -def mksend(*args): - argz, change, fee = args[:-2], args[-2], int(args[-1]) - ins, outs = [], [] - for arg in argz: - if isinstance(arg,list): - for a in arg: (ins if is_inp(a) else outs).append(a) - else: - (ins if is_inp(arg) else outs).append(arg) - - isum = sum([i["value"] for i in ins]) - osum, outputs2 = 0, [] - for o in outs: - if isinstance(o,str): o2 = { - "address": o[:o.find(':')], - "value": int(o[o.find(':')+1:]) - } - else: o2 = o - outputs2.append(o2) - osum += o2["value"] - - if isum < osum+fee: - raise Exception("Not enough money") - elif isum > osum+fee+5430: - outputs2 += [{"address": change, "value": isum-osum-fee }] - - return mktx(ins,outputs2) diff --git a/common.py b/common.py deleted file mode 100644 index 4ec84098..00000000 --- a/common.py +++ /dev/null @@ -1,157 +0,0 @@ - -import bitcoin as btc -from decimal import Decimal -import sys -import datetime -import json - -server = 'irc.freenode.net' -port = 6667 -channel = '#joinmarket' - -command_prefix = '!' -MAX_PRIVMSG_LEN = 450 - -ordername_list = ["absorder", "relorder"] - -def debug(msg): - print datetime.datetime.now().strftime("[%Y/%m/%d %H:%M:%S] ") + msg - -def get_network(): - return 'testnet' - -#TODO change this name into get_addr_ver() or something -def get_vbyte(): - if get_network() == 'testnet': - return 0x6f - else: - return 0x00 - -MAX_MIX_DEPTH = 2 #for now -class Wallet(object): - def __init__(self, seed): - master = btc.bip32_master_key(seed) - m_0 = btc.bip32_ckd(master, 0) - mixing_depth_keys = [btc.bip32_ckd(m_0, c) for c in range(MAX_MIX_DEPTH)] - self.keys = [(btc.bip32_ckd(m, 0), btc.bip32_ckd(m, 1)) for m in mixing_depth_keys] - - #self.index = [[0, 0]]*MAX_MIX_DEPTH - self.index = [] - for i in range(MAX_MIX_DEPTH): - self.index.append([0, 0]) - - #example - #index = self.index[mixing_depth] - #key = btc.bip32_ckd(self.keys[mixing_depth][index[0]], index[1]) - - self.addr_cache = {} - self.unspent = {} - - def get_key(self, mixing_depth, forchange, i): - return btc.bip32_extract_key(btc.bip32_ckd(self.keys[mixing_depth][forchange], i)) - - def get_addr(self, mixing_depth, forchange, i): - return btc.privtoaddr(self.get_key(mixing_depth, forchange, i), get_vbyte()) - - def get_new_addr(self, mixing_depth, forchange): - index = self.index[mixing_depth] - addr = self.get_addr(mixing_depth, forchange, index[forchange]) - self.addr_cache[addr] = (mixing_depth, forchange, index[forchange]) - index[forchange] += 1 - return addr - - def get_receive_addr(self, mixing_depth): - return self.get_new_addr(mixing_depth, False) - - def get_change_addr(self, mixing_depth): - return self.get_new_addr(mixing_depth, True) - - def get_key_from_addr(self, addr): - if addr in self.addr_cache: - return self.get_key(*self.addr_cache[addr]) - else: - return None - - def download_wallet_history(self, gaplimit=6): - ''' - sets Wallet internal indexes to be at the next unused address - ''' - addr_req_count = 20 - - for mix_depth in range(MAX_MIX_DEPTH): - for forchange in [0, 1]: - unused_addr_count = 0 - while unused_addr_count < gaplimit: - addrs = [self.get_new_addr(mix_depth, forchange) for i in range(addr_req_count)] - - #TODO send a pull request to pybitcointools - # because this surely should be possible with a function from it - if get_network() == 'testnet': - blockr_url = 'http://tbtc.blockr.io/api/v1/address/txs/' - elif network == 'btc': - blockr_url = 'http://btc.blockr.io/api/v1/address/txs/' - res = btc.make_request(blockr_url+','.join(addrs)) - data = json.loads(res)['data'] - last_used_addr = '' - unused_addr_count = 0 - for dat in data: - if dat['nb_txs'] != 0: - last_used_addr = dat['address'] - else: - unused_addr_count += 1 - if unused_addr_count >= gaplimit: - break - if last_used_addr == '': - self.index[mix_depth][forchange] = 0 - else: - self.index[mix_depth][forchange] = self.addr_cache[last_used_addr][2] + 1 - - def find_unspent_addresses(self): - ''' - finds utxos in the wallet - assumes you've already called download_wallet_history() so - you know which addresses have been used - ''' - - #TODO handle the case where there are so many addresses it cant - # fit into one api call (>50 or so) - addrs = [] - for m in range(MAX_MIX_DEPTH): - for forchange in [0, 1]: - addrs += [self.get_addr(m, forchange, n) for n in range(self.index[m][forchange])] - - #TODO send a pull request to pybitcointools - # unspent() doesnt tell you which address, you get a bunch of utxos - # but dont know which privkey to sign with - if get_network() == 'testnet': - blockr_url = 'http://tbtc.blockr.io/api/v1/address/unspent/' - elif network == 'btc': - blockr_url = 'http://btc.blockr.io/api/v1/address/unspent/' - res = btc.make_request(blockr_url+','.join(addrs)) - data = json.loads(res)['data'] - if 'unspent' in data: - data = [data] - for dat in data: - for u in dat['unspent']: - self.unspent[u['tx']+':'+str(u['n'])] = {'address': - dat['address'], 'value': int(u['amount'].replace('.', ''))} - - -def calc_cj_fee(ordertype, cjfee, cj_amount): - real_cjfee = None - if ordertype == 'absorder': - real_cjfee = int(cjfee) - elif ordertype == 'relorder': - real_cjfee = int(Decimal(cjfee) * Decimal(cj_amount)) - else: - raise RuntimeError('unknown order type: ' + str(ordertype)) - return real_cjfee - -def calc_total_input_value(utxos): - input_sum = 0 - for utxo in utxos: - tx = btc.blockr_fetchtx(utxo[:64], get_network()) - input_sum += int(btc.deserialize(tx)['outs'][int(utxo[65:])]['value']) - return input_sum - - diff --git a/create-unsigned-tx.py b/create-unsigned-tx.py new file mode 100644 index 00000000..6af84705 --- /dev/null +++ b/create-unsigned-tx.py @@ -0,0 +1,225 @@ +#! /usr/bin/env python + +from optparse import OptionParser +import threading, pprint, sys, os +data_dir = os.path.dirname(os.path.realpath(__file__)) +sys.path.insert(0, os.path.join(data_dir, 'lib')) + +from common import * +import common +import taker as takermodule +from irc import IRCMessageChannel, random_nick +import bitcoin as btc +import sendpayment + +#thread which does the buy-side algorithm +# chooses which coinjoins to initiate and when +class PaymentThread(threading.Thread): + def __init__(self, taker): + threading.Thread.__init__(self) + self.daemon = True + self.taker = taker + self.ignored_makers = [] + + def create_tx(self): + crow = self.taker.db.execute('SELECT COUNT(DISTINCT counterparty) FROM orderbook;').fetchone() + counterparty_count = crow['COUNT(DISTINCT counterparty)'] + counterparty_count -= len(self.ignored_makers) + if counterparty_count < self.taker.options.makercount: + print 'not enough counterparties to fill order, ending' + self.taker.msgchan.shutdown() + return + + utxos = self.taker.utxo_data + orders = None + cjamount = 0 + change_addr = None + choose_orders_recover = None + if self.taker.cjamount == 0: + total_value = sum([va['value'] for va in utxos.values()]) + orders, cjamount = choose_sweep_orders(self.taker.db, total_value, + self.taker.options.txfee, self.taker.options.makercount, + self.taker.chooseOrdersFunc, self.ignored_makers) + if not self.taker.options.answeryes: + total_cj_fee = total_value - cjamount - self.taker.options.txfee + debug('total cj fee = ' + str(total_cj_fee)) + total_fee_pc = 1.0*total_cj_fee / cjamount + debug('total coinjoin fee = ' + str(float('%.3g' % (100.0 * total_fee_pc))) + '%') + sendpayment.check_high_fee(total_fee_pc) + if raw_input('send with these orders? (y/n):')[0] != 'y': + self.finishcallback(None) + return + else: + orders, total_cj_fee = self.sendpayment_choose_orders( + self.taker.cjamount, self.taker.options.makercount) + if not orders: + debug('ERROR not enough liquidity in the orderbook, exiting') + return + total_amount = self.taker.cjamount + total_cj_fee + self.taker.options.txfee + print 'total amount spent = ' + str(total_amount) + cjamount = self.taker.cjamount + change_addr = self.taker.changeaddr + choose_orders_recover = self.sendpayment_choose_orders + + auth_addr = self.taker.utxo_data[self.taker.auth_utxo]['address'] + self.taker.start_cj(self.taker.wallet, cjamount, orders, utxos, + self.taker.destaddr, change_addr, self.taker.options.txfee, + self.finishcallback, choose_orders_recover, auth_addr) + + def finishcallback(self, coinjointx): + if coinjointx.all_responded: + #now sign it ourselves + tx = btc.serialize(coinjointx.latest_tx) + for index, ins in enumerate(coinjointx.latest_tx['ins']): + utxo = ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index']) + if utxo != self.taker.auth_utxo: + continue + addr = coinjointx.input_utxos[utxo]['address'] + tx = btc.sign(tx, index, coinjointx.wallet.get_key_from_addr(addr)) + print 'unsigned tx = \n\n' + tx + '\n' + debug('created unsigned tx, ending') + self.taker.msgchan.shutdown() + return + self.ignored_makers += coinjointx.nonrespondants + debug('recreating the tx, ignored_makers=' + str(self.ignored_makers)) + self.create_tx() + + def sendpayment_choose_orders(self, cj_amount, makercount, nonrespondants=[], active_nicks=[]): + self.ignored_makers += nonrespondants + orders, total_cj_fee = choose_orders(self.taker.db, cj_amount, makercount, + self.taker.chooseOrdersFunc, self.ignored_makers + active_nicks) + if not orders: + return None, 0 + print 'chosen orders to fill ' + str(orders) + ' totalcjfee=' + str(total_cj_fee) + if not self.taker.options.answeryes: + if len(self.ignored_makers) > 0: + noun = 'total' + else: + noun = 'additional' + total_fee_pc = 1.0*total_cj_fee / cj_amount + debug(noun + ' coinjoin fee = ' + str(float('%.3g' % (100.0 * total_fee_pc))) + '%') + sendpayment.check_high_fee(total_fee_pc) + if raw_input('send with these orders? (y/n):')[0] != 'y': + debug('ending') + self.taker.msgchan.shutdown() + return None, -1 + return orders, total_cj_fee + + def run(self): + print 'waiting for all orders to certainly arrive' + time.sleep(self.taker.options.waittime) + self.create_tx() + +class CreateUnsignedTx(takermodule.Taker): + def __init__(self, msgchan, wallet, auth_utxo, cjamount, destaddr, changeaddr, + utxo_data, options, chooseOrdersFunc): + takermodule.Taker.__init__(self, msgchan) + self.wallet = wallet + self.auth_utxo = auth_utxo + self.cjamount = cjamount + self.destaddr = destaddr + self.changeaddr = changeaddr + self.utxo_data = utxo_data + self.options = options + self.chooseOrdersFunc = chooseOrdersFunc + + def on_welcome(self): + takermodule.Taker.on_welcome(self) + PaymentThread(self).start() + +def main(): + parser = OptionParser(usage='usage: %prog [options] [auth utxo] [cjamount] [cjaddr] [changeaddr] [utxos..]', + description='Creates an unsigned coinjoin transaction. Outputs a partially signed transaction ' + + 'hex string. The user must sign their inputs independently and broadcast them. The JoinMarket' + + ' protocol requires the taker to have a single p2pk UTXO input to use to authenticate the ' + + ' encrypted messages. For this reason you must pass auth utxo and the corresponding private key') + #for cjamount=0 do a sweep, and ignore change address + parser.add_option('-f', '--txfee', action='store', type='int', dest='txfee', + default=10000, help='miner fee contribution, in satoshis, default=10000') + parser.add_option('-w', '--wait-time', action='store', type='float', dest='waittime', + help='wait time in seconds to allow orders to arrive, default=5', default=5) + parser.add_option('-N', '--makercount', action='store', type='int', dest='makercount', + help='how many makers to coinjoin with, default=2', default=2) + parser.add_option('-C','--choose-cheapest', action='store_true', dest='choosecheapest', default=False, + help='override weightened offers picking and choose cheapest') + parser.add_option('-P','--pick-orders', action='store_true', dest='pickorders', default=False, + help='manually pick which orders to take. doesn\'t work while sweeping.') + parser.add_option('--yes', action='store_true', dest='answeryes', default=False, + help='answer yes to everything') + #TODO implement + #parser.add_option('-n', '--no-network', action='store_true', dest='nonetwork', default=False, + # help='dont query the blockchain interface, instead user must supply value of UTXOs on ' + + # ' command line in the format txid:output/value-in-satoshi') + (options, args) = parser.parse_args() + + if len(args) < 3: + parser.error('Needs a wallet, amount and destination address') + sys.exit(0) + auth_utxo = args[0] + cjamount = int(args[1]) + destaddr = args[2] + changeaddr = args[3] + cold_utxos = args[4:] + + common.load_program_config() + addr_valid1, errormsg1 = validate_address(destaddr) + #if amount = 0 dont bother checking changeaddr so user can write any junk + if cjamount != 0: + addr_valid2, errormsg2 = validate_address(changeaddr) + else: + addr_valid2 = True + if not addr_valid1 or not addr_valid2: + if not addr_valid1: + print 'ERROR: Address invalid. ' + errormsg1 + else: + print 'ERROR: Address invalid. ' + errormsg2 + return + + all_utxos = [auth_utxo] + cold_utxos + query_result = common.bc_interface.query_utxo_set(all_utxos) + if None in query_result: + print query_result + utxo_data = {} + for utxo, data in zip(all_utxos, query_result): + utxo_data[utxo] = {'address': data['address'], 'value': data['value']} + auth_privkey = raw_input('input private key for ' + utxo_data[auth_utxo]['address'] + ' :') + if utxo_data[auth_utxo]['address'] != btc.privtoaddr(auth_privkey, common.get_p2pk_vbyte()): + print 'ERROR: privkey does not match auth utxo' + return + + chooseOrdersFunc = None + if options.pickorders and amount != 0: #cant use for sweeping + chooseOrdersFunc = pick_order + elif options.choosecheapest: + chooseOrdersFunc = cheapest_order_choose + else: #choose randomly (weighted) + chooseOrdersFunc = weighted_order_choose + + common.nickname = random_nick() + debug('starting sendpayment') + + class UnsignedTXWallet(common.AbstractWallet): + def get_key_from_addr(self, addr): + debug('getting privkey of ' + addr) + if btc.privtoaddr(auth_privkey, common.get_p2pk_vbyte()) != addr: + raise RuntimeError('privkey doesnt match given address') + return auth_privkey + + wallet = UnsignedTXWallet() + irc = IRCMessageChannel(common.nickname) + taker = CreateUnsignedTx(irc, wallet, auth_utxo, cjamount, destaddr, + changeaddr, utxo_data, options, chooseOrdersFunc) + try: + debug('starting irc') + irc.run() + except: + debug('CRASHING, DUMPING EVERYTHING') + debug_dump_object(wallet, ['addr_cache', 'keys', 'wallet_name', 'seed']) + debug_dump_object(taker) + import traceback + debug(traceback.format_exc()) + +if __name__ == "__main__": + main() + print('done') + diff --git a/encryption_protocol.txt b/encryption_protocol.txt new file mode 100644 index 00000000..fb07259b --- /dev/null +++ b/encryption_protocol.txt @@ -0,0 +1,40 @@ +Encryption handshake in JoinMarket +================================== + +In the clear +============ + +TAK: !fill +MAK: !pubkey + +Both maker and taker construct a crypto Box object to allow authenticated encryption between the parties. +These Box objects are properties of the CoinJoinTx and CoinJoinOrder objects, so they are specific to +transactions and not to Maker and Taker entities. + +Encrypted +========= + +TAK: !auth +(Maker verifies the btc sig; if not valid, connection is dropped - send REJECT message) +MAK: !ioauth +(Taker verifies the btc sig; if not valid, as for previous) + +Because the !auth messages are under encryption, there is no privacy leak of bitcoin pubkeys or output addresses. + +If both verifications pass, the remainder of the messages exchanged between the two parties will continue under encryption. + +Specifically, these message types will be encrypted: +!auth +!ioauth +!tx +!sig + +Note +==== +A key part of the authorisation process is the matching between the bitcoin pubkeys used in the coinjoin +transaction and the encryption pubkeys used. This ensures that the messages we are sending are only +readable by the entity which is conducting the bitcoin transaction with us. + +To ensure this, the maker should not sign any transaction that doesn't use the previously identified +input utxo as its input, and the taker should not push/sign any transaction that doesn't use the +previously identified maker coinjoin pubkey/address as its output. \ No newline at end of file diff --git a/irclib.py b/irclib.py deleted file mode 100644 index 0cef9c86..00000000 --- a/irclib.py +++ /dev/null @@ -1,168 +0,0 @@ - -import socket -import threading -import time - -PING_INTERVAL = 40 -PING_TIMEOUT = 10 - -def get_irc_text(line): - return line[line[1:].find(':') + 2:] - -def get_irc_nick(source): - return source[1:source.find('!')] - -class PingThread(threading.Thread): - def __init__(self, irc): - threading.Thread.__init__(self) - self.daemon = True - self.irc = irc - def run(self): - while not self.irc.give_up: - time.sleep(PING_INTERVAL) - self.irc.ping_reply = False - #maybe use this to calculate the lag one day - self.irc.lockcond.acquire() - self.irc.send_raw('PING LAG' + str(int(time.time() * 1000))) - self.irc.lockcond.wait(PING_TIMEOUT) - self.irc.lockcond.release() - if not self.irc.ping_reply: - print 'irc ping timed out' - self.irc.close() - self.irc.fd.close() - self.irc.sock.close() - -#handle one channel at a time -class IRCClient(object): - def __init__(self): - self.on_privmsg = None - self.on_pubmsg = None - self.on_welcome = None - self.on_set_topic = None - self.on_leave = None - self.on_nick_change = None #TODO implement - - def close(self): - self.send_raw("QUIT") - - def shutdown(self): - self.close() - self.give_up = True - - def pubmsg(self, message): - self.send_raw("PRIVMSG " + self.channel + " :" + message) - - def privmsg(self, nick, message): - print '>> ' + nick + ' :' + message - self.send_raw("PRIVMSG " + nick + " :" + message) - - def send_raw(self, line): - #print('>> ' + line) - self.sock.sendall(line + '\r\n') - - def __handle_privmsg(self, source, target, message): - nick = get_irc_nick(source) - if message[0] == '\x01': - endindex = message[1:].find('\x01') - if endindex == -1: - return - ctcp = message[1:endindex + 1] - #self.send_raw('PRIVMSG ' + nick + ' :\x01VERSION - #TODO ctcp version here, since some servers dont let you get on without - if target == self.nick: - if self.on_privmsg != None: - self.on_privmsg(self, nick, message) - else: - if self.on_privmsg != None: - self.on_pubmsg(self, nick, message) - - def __handle_line(self, line): - line = line.rstrip() - #print('<< ' + line) - if line.startswith('PING '): - self.send_raw(line.replace('PING', 'PONG')) - return - - chunks = line.split(' ') - if chunks[1] == 'PRIVMSG': - self.__handle_privmsg(chunks[0], chunks[2], get_irc_text(line)) - if chunks[1] == 'PONG': - self.ping_reply = True - self.lockcond.acquire() - self.lockcond.notify() - self.lockcond.release() - elif chunks[1] == '376': #end of motd - self.send_raw('JOIN ' + self.channel) - elif chunks[1] == '433': #nick in use - self.nick += '_' - self.send_raw('NICK ' + self.nick) - elif chunks[1] == '366': #end of names list - self.connect_attempts = 0 - if self.on_welcome != None: - self.on_welcome(self) - elif chunks[1] == '332' or chunks[1] == 'TOPIC': #channel topic - topic = get_irc_text(line) - if self.on_set_topic != None: - self.on_set_topic(self, topic) - elif chunks[1] == 'QUIT': - nick = get_irc_nick(chunks[0]) - if nick == self.nick: - raise IOError('we quit') - else: - if self.on_leave != None: - self.on_leave(self, nick) - elif chunks[1] == 'KICK': - target = chunks[3] - if self.on_leave != None: - self.on_leave(self, nick) - elif chunks[1] == 'PART': - nick = get_irc_nick(chunks[0]) - if self.on_leave != None: - self.on_leave(self, nick) - elif chunks[1] == 'JOIN': - channel = chunks[2][1:] - nick = get_irc_nick(chunks[0]) - ''' - elif chunks[1] == '005': - self.motd_fd = open("motd.txt", "w") - elif chunks[1] == '372': - self.motd_fd.write(get_irc_text(line) + "\n") - elif chunks[1] == '251': - self.motd_fd.close() - ''' - - def run(self, server, port, nick, channel, username='username', realname='realname'): - self.nick = nick - self.channel = channel - self.connect_attempts = 0 - - self.give_up = False - self.ping_reply = True - self.lockcond = threading.Condition() - PingThread(self).start() - - while self.connect_attempts < 10 and not self.give_up: - self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) - self.sock.connect((server, port)) - self.fd = self.sock.makefile() - self.send_raw('USER %s b c :%s' % (username, realname)) - self.send_raw('NICK ' + nick) - try: - while 1: - line = self.fd.readline() - if line == None: - break - if len(line) == 0: - break - self.__handle_line(line) - except IOError as e: - print repr(e) - finally: - self.fd.close() - self.sock.close() - self.connect_attempts += 1 - time.sleep(10) - print 'reconnecting' - print 'ending irc' - self.give_up = True - diff --git a/lib/bitcoin/__init__.py b/lib/bitcoin/__init__.py new file mode 100644 index 00000000..a772073f --- /dev/null +++ b/lib/bitcoin/__init__.py @@ -0,0 +1,9 @@ +from bitcoin.py2specials import * +from bitcoin.py3specials import * +from bitcoin.main import * +from bitcoin.transaction import * +from bitcoin.deterministic import * +from bitcoin.bci import * +from bitcoin.composite import * +from bitcoin.stealth import * +from bitcoin.blocks import * diff --git a/bitcoin/bci.py b/lib/bitcoin/bci.py similarity index 58% rename from bitcoin/bci.py rename to lib/bitcoin/bci.py index 45b6719b..a9b8d15f 100644 --- a/bitcoin/bci.py +++ b/lib/bitcoin/bci.py @@ -1,31 +1,21 @@ #!/usr/bin/python -import urllib2 -import json -import re +import json, re import random import sys - -#i changed this 8/2014 to include this proxy thing -# that also required changing everything to http requests since the -# proxy just works for http in urllib2 -__proxy_handler = None - -def bci_set_proxy(handler): - global __proxy_handler - __proxy_handler = handler +try: + from urllib.request import build_opener +except: + from urllib2 import build_opener # Makes a request to a given URL (first arg) and optional params (second arg) def make_request(*args): - if __proxy_handler != None: - opener = urllib2.build_opener(__proxy_handler) - else: - opener = urllib2.build_opener() + opener = build_opener() opener.addheaders = [('User-agent', 'Mozilla/5.0'+str(random.randrange(1000000)))] try: return opener.open(*args).read().strip() - except Exception, e: + except Exception as e: try: p = e.read().strip() except: @@ -33,21 +23,31 @@ def make_request(*args): raise Exception(p) +def parse_addr_args(*args): + # Valid input formats: blockr_unspent([addr1, addr2,addr3]) + # blockr_unspent(addr1, addr2, addr3) + # blockr_unspent([addr1, addr2, addr3], network) + # blockr_unspent(addr1, addr2, addr3, network) + # Where network is 'btc' or 'testnet' + network = 'btc' + addr_args = args + if len(args) >= 1 and args[-1] in ('testnet', 'btc'): + network = args[-1] + addr_args = args[:-1] + if len(addr_args) == 1 and isinstance(addr_args, list): + addr_args = addr_args[0] + + return network, addr_args + + # Gets the unspent outputs of one or more addresses -def unspent(*args): - # Valid input formats: unspent([addr1, addr2,addr3]) - # unspent(addr1, addr2, addr3) - if len(args) == 0: - return [] - elif isinstance(args[0], list): - addrs = args[0] - else: - addrs = args +def bci_unspent(*args): + network, addrs = parse_addr_args(*args) u = [] for a in addrs: try: data = make_request('http://blockchain.info/unspent?address='+a) - except Exception, e: + except Exception as e: if str(e) == 'No free outputs to spend': continue else: @@ -71,11 +71,7 @@ def blockr_unspent(*args): # blockr_unspent([addr1, addr2, addr3], network) # blockr_unspent(addr1, addr2, addr3, network) # Where network is 'btc' or 'testnet' - network = 'btc' - addr_args = args - if len(args) >= 1 and args[-1] in ('testnet', 'btc'): - network = args[-1] - addr_args = args[:-1] + network, addr_args = parse_addr_args(*args) if network == 'testnet': blockr_url = 'http://tbtc.blockr.io/api/v1/address/unspent/' @@ -105,6 +101,41 @@ def blockr_unspent(*args): return o +def helloblock_unspent(*args): + network, addrs = parse_addr_args(*args) + if network == 'testnet': + url = 'http://testnet.helloblock.io/v1/addresses/%s/unspents?limit=500&offset=%s' + elif network == 'btc': + url = 'http://mainnet.helloblock.io/v1/addresses/%s/unspents?limit=500&offset=%s' + o = [] + for addr in addrs: + for offset in xrange(0, 10**9, 500): + res = make_request(url % (addr, offset)) + data = json.loads(res)["data"] + if not len(data["unspents"]): + break + elif offset: + sys.stderr.write("Getting more unspents: %d\n" % offset) + for dat in data["unspents"]: + o.append({ + "output": dat["txHash"]+':'+str(dat["index"]), + "value": dat["value"], + }) + return o + + +unspent_getters = { + 'bci': bci_unspent, + 'blockr': blockr_unspent, + 'helloblock': helloblock_unspent +} + + +def unspent(*args, **kwargs): + f = unspent_getters.get(kwargs.get('source', ''), bci_unspent) + return f(*args) + + # Gets the transaction output history of a given set of addresses, # including whether or not they have been spent def history(*args): @@ -154,8 +185,8 @@ def history(*args): return [outs[k] for k in outs] -# Pushes a transaction to the network using https://blockchain.info/pushtx -def pushtx(tx): +# Pushes a transaction to the network using http://blockchain.info/pushtx +def bci_pushtx(tx): if not re.match('^[0-9a-fA-F]*$', tx): tx = tx.encode('hex') return make_request('http://blockchain.info/pushtx', 'tx='+tx) @@ -194,6 +225,17 @@ def helloblock_pushtx(tx): return make_request('http://mainnet.helloblock.io/v1/transactions', 'rawTxHex='+tx) +pushtx_getters = { + 'bci': bci_pushtx, + 'blockr': blockr_pushtx, + 'helloblock': helloblock_pushtx +} + + +def pushtx(*args, **kwargs): + f = pushtx_getters.get(kwargs.get('source', ''), bci_pushtx) + return f(*args) + def last_block_height(): data = make_request('http://blockchain.info/latestblock') @@ -223,11 +265,54 @@ def blockr_fetchtx(txhash, network='btc'): return jsondata['data']['tx']['hex'] -def fetchtx(txhash): - try: - return bci_fetchtx(txhash) - except: - return blockr_fetchtx(txhash) +def helloblock_fetchtx(txhash, network='btc'): + if not re.match('^[0-9a-fA-F]*$', txhash): + txhash = txhash.encode('hex') + if network == 'testnet': + url = 'http://testnet.helloblock.io/v1/transactions/' + elif network == 'btc': + url = 'http://mainnet.helloblock.io/v1/transactions/' + else: + raise Exception( + 'Unsupported network {0} for helloblock_fetchtx'.format(network)) + data = json.loads(make_request(url + txhash))["data"]["transaction"] + o = { + "locktime": data["locktime"], + "version": data["version"], + "ins": [], + "outs": [] + } + for inp in data["inputs"]: + o["ins"].append({ + "script": inp["scriptSig"], + "outpoint": { + "index": inp["prevTxoutIndex"], + "hash": inp["prevTxHash"], + }, + "sequence": 4294967295 + }) + for outp in data["outputs"]: + o["outs"].append({ + "value": outp["value"], + "script": outp["scriptPubKey"] + }) + from bitcoin.transaction import serialize + from bitcoin.transaction import txhash as TXHASH + tx = serialize(o) + assert TXHASH(tx) == txhash + return tx + + +fetchtx_getters = { + 'bci': bci_fetchtx, + 'blockr': blockr_fetchtx, + 'helloblock': helloblock_fetchtx +} + + +def fetchtx(*args, **kwargs): + f = fetchtx_getters.get(kwargs.get('source', ''), bci_fetchtx) + return f(*args) def firstbits(address): @@ -236,3 +321,44 @@ def firstbits(address): else: return make_request( 'http://blockchain.info/q/resolvefirstbits/'+address) + + +def get_block_at_height(height): + j = json.loads(make_request("http://blockchain.info/block-height/" + + str(height)+"?format=json")) + for b in j['blocks']: + if b['main_chain'] is True: + return b + raise Exception("Block at this height not found") + + +def _get_block(inp): + if len(str(inp)) < 64: + return get_block_at_height(inp) + else: + return json.loads(make_request( + 'http://blockchain.info/rawblock/'+inp)) + + +def get_block_header_data(inp): + j = _get_block(inp) + return { + 'version': j['ver'], + 'hash': j['hash'], + 'prevhash': j['prev_block'], + 'timestamp': j['time'], + 'merkle_root': j['mrkl_root'], + 'bits': j['bits'], + 'nonce': j['nonce'], + } + + +def get_txs_in_block(inp): + j = _get_block(inp) + hashes = [t['hash'] for t in j['tx']] + return hashes + + +def get_block_height(txhash): + j = json.loads(make_request('http://blockchain.info/rawtx/'+txhash)) + return j['block_height'] diff --git a/lib/bitcoin/blocks.py b/lib/bitcoin/blocks.py new file mode 100644 index 00000000..e60dcfdd --- /dev/null +++ b/lib/bitcoin/blocks.py @@ -0,0 +1,50 @@ +from bitcoin.main import * + + +def serialize_header(inp): + o = encode(inp['version'], 256, 4)[::-1] + \ + inp['prevhash'].decode('hex')[::-1] + \ + inp['merkle_root'].decode('hex')[::-1] + \ + encode(inp['timestamp'], 256, 4)[::-1] + \ + encode(inp['bits'], 256, 4)[::-1] + \ + encode(inp['nonce'], 256, 4)[::-1] + h = bin_sha256(bin_sha256(o))[::-1].encode('hex') + assert h == inp['hash'], (sha256(o), inp['hash']) + return o.encode('hex') + + +def deserialize_header(inp): + inp = inp.decode('hex') + return { + "version": decode(inp[:4][::-1], 256), + "prevhash": inp[4:36][::-1].encode('hex'), + "merkle_root": inp[36:68][::-1].encode('hex'), + "timestamp": decode(inp[68:72][::-1], 256), + "bits": decode(inp[72:76][::-1], 256), + "nonce": decode(inp[76:80][::-1], 256), + "hash": bin_sha256(bin_sha256(inp))[::-1].encode('hex') + } + + +def mk_merkle_proof(header, hashes, index): + nodes = [h.decode('hex')[::-1] for h in hashes] + if len(nodes) % 2 and len(nodes) > 2: + nodes.append(nodes[-1]) + layers = [nodes] + while len(nodes) > 1: + newnodes = [] + for i in range(0, len(nodes) - 1, 2): + newnodes.append(bin_sha256(bin_sha256(nodes[i] + nodes[i+1]))) + if len(newnodes) % 2 and len(newnodes) > 2: + newnodes.append(newnodes[-1]) + nodes = newnodes + layers.append(nodes) + # Sanity check, make sure merkle root is valid + assert nodes[0][::-1].encode('hex') == header['merkle_root'] + merkle_siblings = \ + [layers[i][(index >> i) ^ 1] for i in range(len(layers)-1)] + return { + "hash": hashes[index], + "siblings": [x[::-1].encode('hex') for x in merkle_siblings], + "header": header + } diff --git a/lib/bitcoin/composite.py b/lib/bitcoin/composite.py new file mode 100644 index 00000000..cbceea93 --- /dev/null +++ b/lib/bitcoin/composite.py @@ -0,0 +1,128 @@ +from bitcoin.main import * +from bitcoin.transaction import * +from bitcoin.bci import * +from bitcoin.deterministic import * +from bitcoin.blocks import * + + +# Takes privkey, address, value (satoshis), fee (satoshis) +def send(frm, to, value, fee=10000): + return sendmultitx(frm, to + ":" + str(value), fee) + + +# Takes privkey, "address1:value1,address2:value2" (satoshis), fee (satoshis) +def sendmultitx(frm, tovalues, fee=10000, **kwargs): + outs = [] + outvalue = 0 + tv = tovalues.split(",") + for a in tv: + outs.append(a) + outvalue += int(a.split(":")[1]) + + u = unspent(privtoaddr(frm), **kwargs) + u2 = select(u, int(outvalue)+int(fee)) + argz = u2 + outs + [frm, fee] + tx = mksend(*argz) + tx2 = signall(tx, frm) + return pushtx(tx2, **kwargs) + + +# Takes address, address, value (satoshis), fee(satoshis) +def preparetx(frm, to, value, fee=10000, **kwargs): + tovalues = to + ":" + str(value) + return preparemultitx(frm, tovalues, fee, **kwargs) + + +# Takes address, address:value, address:value ... (satoshis), fee(satoshis) +def preparemultitx(frm, *args, **kwargs): + tv, fee = args[:-1], int(args[-1]) + outs = [] + outvalue = 0 + for a in tv: + outs.append(a) + outvalue += int(a.split(":")[1]) + + u = unspent(frm, **kwargs) + u2 = select(u, int(outvalue)+int(fee)) + argz = u2 + outs + [frm, fee] + return mksend(*argz) + + +# BIP32 hierarchical deterministic multisig script +def bip32_hdm_script(*args): + if len(args) == 3: + keys, req, path = args + else: + i, keys, path = 0, [], [] + while len(args[i]) > 40: + keys.append(args[i]) + i += 1 + req = int(args[i]) + path = map(int, args[i+1:]) + pubs = sorted(map(lambda x: bip32_descend(x, path), keys)) + return mk_multisig_script(pubs, req) + + +# BIP32 hierarchical deterministic multisig address +def bip32_hdm_addr(*args): + return scriptaddr(bip32_hdm_script(*args)) + + +# Setup a coinvault transaction +def setup_coinvault_tx(tx, script): + txobj = deserialize(tx) + N = deserialize_script(script)[-2] + for inp in txobj["ins"]: + inp["script"] = serialize_script([None] * (N+1) + [script]) + return serialize(txobj) + + +# Sign a coinvault transaction +def sign_coinvault_tx(tx, priv): + pub = privtopub(priv) + txobj = deserialize(tx) + subscript = deserialize_script(txobj['ins'][0]['script']) + oscript = deserialize_script(subscript[-1]) + k, pubs = oscript[0], oscript[1:-2] + for j in range(len(txobj['ins'])): + scr = deserialize_script(txobj['ins'][j]['script']) + for i, p in enumerate(pubs): + if p == pub: + scr[i+1] = multisign(tx, j, subscript[-1], priv) + if len(filter(lambda x: x, scr[1:-1])) >= k: + scr = [None] + filter(lambda x: x, scr[1:-1])[:k] + [scr[-1]] + txobj['ins'][j]['script'] = serialize_script(scr) + return serialize(txobj) + + +# Inspects a transaction +def inspect(tx, **kwargs): + d = deserialize(tx) + isum = 0 + ins = {} + for _in in d['ins']: + h = _in['outpoint']['hash'] + i = _in['outpoint']['index'] + prevout = deserialize(fetchtx(h, **kwargs))['outs'][i] + isum += prevout['value'] + a = script_to_address(prevout['script']) + ins[a] = ins.get(a, 0) + prevout['value'] + outs = [] + osum = 0 + for _out in d['outs']: + outs.append({'address': script_to_address(_out['script']), + 'value': _out['value']}) + osum += _out['value'] + return { + 'fee': isum - osum, + 'outs': outs, + 'ins': ins + } + + +def merkle_prove(txhash): + blocknum = str(get_block_height(txhash)) + header = get_block_header_data(blocknum) + hashes = get_txs_in_block(blocknum) + i = hashes.index(txhash) + return mk_merkle_proof(header, hashes, i) diff --git a/lib/bitcoin/deterministic.py b/lib/bitcoin/deterministic.py new file mode 100644 index 00000000..200dc4a4 --- /dev/null +++ b/lib/bitcoin/deterministic.py @@ -0,0 +1,199 @@ +from bitcoin.main import * +import hmac +import hashlib +from binascii import hexlify +# Electrum wallets + + +def electrum_stretch(seed): + return slowsha(seed) + +# Accepts seed or stretched seed, returns master public key + + +def electrum_mpk(seed): + if len(seed) == 32: + seed = electrum_stretch(seed) + return privkey_to_pubkey(seed)[2:] + +# Accepts (seed or stretched seed), index and secondary index +# (conventionally 0 for ordinary addresses, 1 for change) , returns privkey + + +def electrum_privkey(seed, n, for_change=0): + if len(seed) == 32: + seed = electrum_stretch(seed) + mpk = electrum_mpk(seed) + offset = dbl_sha256(from_int_representation_to_bytes(n)+b':'+from_int_representation_to_bytes(for_change)+b':'+binascii.unhexlify(mpk)) + return add_privkeys(seed, offset) + +# Accepts (seed or stretched seed or master pubkey), index and secondary index +# (conventionally 0 for ordinary addresses, 1 for change) , returns pubkey + + +def electrum_pubkey(masterkey, n, for_change=0): + if len(masterkey) == 32: + mpk = electrum_mpk(electrum_stretch(masterkey)) + elif len(masterkey) == 64: + mpk = electrum_mpk(masterkey) + else: + mpk = masterkey + bin_mpk = encode_pubkey(mpk, 'bin_electrum') + offset = bin_dbl_sha256(from_int_representation_to_bytes(n)+b':'+from_int_representation_to_bytes(for_change)+b':'+bin_mpk) + return add_pubkeys('04'+mpk, privtopub(offset)) + +# seed/stretched seed/pubkey -> address (convenience method) + + +def electrum_address(masterkey, n, for_change=0, version=0): + return pubkey_to_address(electrum_pubkey(masterkey, n, for_change), version) + +# Given a master public key, a private key from that wallet and its index, +# cracks the secret exponent which can be used to generate all other private +# keys in the wallet + + +def crack_electrum_wallet(mpk, pk, n, for_change=0): + bin_mpk = encode_pubkey(mpk, 'bin_electrum') + offset = dbl_sha256(str(n)+':'+str(for_change)+':'+bin_mpk) + return subtract_privkeys(pk, offset) + +# Below code ASSUMES binary inputs and compressed pubkeys +MAINNET_PRIVATE = b'\x04\x88\xAD\xE4' +MAINNET_PUBLIC = b'\x04\x88\xB2\x1E' +TESTNET_PRIVATE = b'\x04\x35\x83\x94' +TESTNET_PUBLIC = b'\x04\x35\x87\xCF' +PRIVATE = [MAINNET_PRIVATE, TESTNET_PRIVATE] +PUBLIC = [MAINNET_PUBLIC, TESTNET_PUBLIC] + +# BIP32 child key derivation + + +def raw_bip32_ckd(rawtuple, i): + vbytes, depth, fingerprint, oldi, chaincode, key = rawtuple + i = int(i) + + if vbytes in PRIVATE: + priv = key + pub = privtopub(key) + else: + pub = key + + if i >= 2**31: + if vbytes in PUBLIC: + raise Exception("Can't do private derivation on public key!") + I = hmac.new(chaincode, b'\x00'+priv[:32]+encode(i, 256, 4), hashlib.sha512).digest() + else: + I = hmac.new(chaincode, pub+encode(i, 256, 4), hashlib.sha512).digest() + + if vbytes in PRIVATE: + newkey = add_privkeys(I[:32]+B'\x01', priv) + fingerprint = bin_hash160(privtopub(key))[:4] + if vbytes in PUBLIC: + newkey = add_pubkeys(compress(privtopub(I[:32])), key) + fingerprint = bin_hash160(key)[:4] + + return (vbytes, depth + 1, fingerprint, i, I[32:], newkey) + + +def bip32_serialize(rawtuple): + vbytes, depth, fingerprint, i, chaincode, key = rawtuple + i = encode(i, 256, 4) + chaincode = encode(hash_to_int(chaincode), 256, 32) + keydata = b'\x00'+key[:-1] if vbytes in PRIVATE else key + bindata = vbytes + from_int_to_byte(depth % 256) + fingerprint + i + chaincode + keydata + return changebase(bindata+bin_dbl_sha256(bindata)[:4], 256, 58) + + +def bip32_deserialize(data): + dbin = changebase(data, 58, 256) + if bin_dbl_sha256(dbin[:-4])[:4] != dbin[-4:]: + raise Exception("Invalid checksum") + vbytes = dbin[0:4] + depth = from_byte_to_int(dbin[4]) + fingerprint = dbin[5:9] + i = decode(dbin[9:13], 256) + chaincode = dbin[13:45] + key = dbin[46:78]+b'\x01' if vbytes in PRIVATE else dbin[45:78] + return (vbytes, depth, fingerprint, i, chaincode, key) + + +def raw_bip32_privtopub(rawtuple): + vbytes, depth, fingerprint, i, chaincode, key = rawtuple + newvbytes = MAINNET_PUBLIC if vbytes == MAINNET_PRIVATE else TESTNET_PUBLIC + return (newvbytes, depth, fingerprint, i, chaincode, privtopub(key)) + + +def bip32_privtopub(data): + return bip32_serialize(raw_bip32_privtopub(bip32_deserialize(data))) + + +def bip32_ckd(data, i): + return bip32_serialize(raw_bip32_ckd(bip32_deserialize(data), i)) + + +def bip32_master_key(seed, vbytes=MAINNET_PRIVATE): + I = hmac.new(from_string_to_bytes("Bitcoin seed"), seed, hashlib.sha512).digest() + return bip32_serialize((vbytes, 0, b'\x00'*4, 0, I[32:], I[:32]+b'\x01')) + + +def bip32_bin_extract_key(data): + return bip32_deserialize(data)[-1] + + +def bip32_extract_key(data): + return safe_hexlify(bip32_deserialize(data)[-1]) + +# Exploits the same vulnerability as above in Electrum wallets +# Takes a BIP32 pubkey and one of the child privkeys of its corresponding +# privkey and returns the BIP32 privkey associated with that pubkey + + +def raw_crack_bip32_privkey(parent_pub, priv): + vbytes, depth, fingerprint, i, chaincode, key = priv + pvbytes, pdepth, pfingerprint, pi, pchaincode, pkey = parent_pub + i = int(i) + + if i >= 2**31: + raise Exception("Can't crack private derivation!") + + I = hmac.new(pchaincode, pkey+encode(i, 256, 4), hashlib.sha512).digest() + + pprivkey = subtract_privkeys(key, I[:32]+b'\x01') + + newvbytes = MAINNET_PRIVATE if vbytes == MAINNET_PUBLIC else TESTNET_PRIVATE + return (newvbytes, pdepth, pfingerprint, pi, pchaincode, pprivkey) + + +def crack_bip32_privkey(parent_pub, priv): + dsppub = bip32_deserialize(parent_pub) + dspriv = bip32_deserialize(priv) + return bip32_serialize(raw_crack_bip32_privkey(dsppub, dspriv)) + + +def coinvault_pub_to_bip32(*args): + if len(args) == 1: + args = args[0].split(' ') + vals = map(int, args[34:]) + I1 = ''.join(map(chr, vals[:33])) + I2 = ''.join(map(chr, vals[35:67])) + return bip32_serialize((MAINNET_PUBLIC, 0, b'\x00'*4, 0, I2, I1)) + + +def coinvault_priv_to_bip32(*args): + if len(args) == 1: + args = args[0].split(' ') + vals = map(int, args[34:]) + I2 = ''.join(map(chr, vals[35:67])) + I3 = ''.join(map(chr, vals[72:104])) + return bip32_serialize((MAINNET_PRIVATE, 0, b'\x00'*4, 0, I2, I3+b'\x01')) + + +def bip32_descend(*args): + if len(args) == 2: + key, path = args + else: + key, path = args[0], map(int, args[1:]) + for p in path: + key = bip32_ckd(key, p) + return bip32_extract_key(key) diff --git a/lib/bitcoin/main.py b/lib/bitcoin/main.py new file mode 100644 index 00000000..daa66bd9 --- /dev/null +++ b/lib/bitcoin/main.py @@ -0,0 +1,550 @@ +#!/usr/bin/python +from .py2specials import * +from .py3specials import * +import binascii +import hashlib +import re +import sys +import os +import base64 +import time +import random +import hmac +from bitcoin.ripemd import * + +# Elliptic curve parameters (secp256k1) + +P = 2**256 - 2**32 - 977 +N = 115792089237316195423570985008687907852837564279074904382605163141518161494337 +A = 0 +B = 7 +Gx = 55066263022277343669578718895168534326250603453777594175500187360389116729240 +Gy = 32670510020758816978083085130507043184471273380659243275938904335757337482424 +G = (Gx, Gy) + + +def change_curve(p, n, a, b, gx, gy): + global P, N, A, B, Gx, Gy, G + P, N, A, B, Gx, Gy = p, n, a, b, gx, gy + G = (Gx, Gy) + + +def getG(): + return G + +# Extended Euclidean Algorithm + + +def inv(a, n): + lm, hm = 1, 0 + low, high = a % n, n + while low > 1: + r = high//low + nm, new = hm-lm*r, high-low*r + lm, low, hm, high = nm, new, lm, low + return lm % n + + + +# JSON access (for pybtctool convenience) + + +def access(obj, prop): + if isinstance(obj, dict): + if prop in obj: + return obj[prop] + elif '.' in prop: + return obj[float(prop)] + else: + return obj[int(prop)] + else: + return obj[int(prop)] + + +def multiaccess(obj, prop): + return [access(o, prop) for o in obj] + + +def slice(obj, start=0, end=2**200): + return obj[int(start):int(end)] + + +def count(obj): + return len(obj) + +_sum = sum + + +def sum(obj): + return _sum(obj) + + +# Elliptic curve Jordan form functions +# P = (m, n, p, q) where m/n = x, p/q = y + +def isinf(p): + return p[0] == 0 and p[1] == 0 + + +def jordan_isinf(p): + return p[0][0] == 0 and p[1][0] == 0 + + +def mulcoords(c1, c2): + return (c1[0] * c2[0] % P, c1[1] * c2[1] % P) + + +def mul_by_const(c, v): + return (c[0] * v % P, c[1]) + + +def addcoords(c1, c2): + return ((c1[0] * c2[1] + c2[0] * c1[1]) % P, c1[1] * c2[1] % P) + + +def subcoords(c1, c2): + return ((c1[0] * c2[1] - c2[0] * c1[1]) % P, c1[1] * c2[1] % P) + + +def invcoords(c): + return (c[1], c[0]) + + +def jordan_add(a, b): + if jordan_isinf(a): + return b + if jordan_isinf(b): + return a + + if (a[0][0] * b[0][1] - b[0][0] * a[0][1]) % P == 0: + if (a[1][0] * b[1][1] - b[1][0] * a[1][1]) % P == 0: + return jordan_double(a) + else: + return ((0, 1), (0, 1)) + xdiff = subcoords(b[0], a[0]) + ydiff = subcoords(b[1], a[1]) + m = mulcoords(ydiff, invcoords(xdiff)) + x = subcoords(subcoords(mulcoords(m, m), a[0]), b[0]) + y = subcoords(mulcoords(m, subcoords(a[0], x)), a[1]) + return (x, y) + + +def jordan_double(a): + if jordan_isinf(a): + return ((0, 1), (0, 1)) + num = addcoords(mul_by_const(mulcoords(a[0], a[0]), 3), (A, 1)) + den = mul_by_const(a[1], 2) + m = mulcoords(num, invcoords(den)) + x = subcoords(mulcoords(m, m), mul_by_const(a[0], 2)) + y = subcoords(mulcoords(m, subcoords(a[0], x)), a[1]) + return (x, y) + + +def jordan_multiply(a, n): + if jordan_isinf(a) or n == 0: + return ((0, 0), (0, 0)) + if n == 1: + return a + if n < 0 or n >= N: + return jordan_multiply(a, n % N) + if (n % 2) == 0: + return jordan_double(jordan_multiply(a, n//2)) + if (n % 2) == 1: + return jordan_add(jordan_double(jordan_multiply(a, n//2)), a) + + +def to_jordan(p): + return ((p[0], 1), (p[1], 1)) + + +def from_jordan(p): + return (p[0][0] * inv(p[0][1], P) % P, p[1][0] * inv(p[1][1], P) % P) + return (p[0][0] * inv(p[0][1], P) % P, p[1][0] * inv(p[1][1], P) % P) + + +def fast_multiply(a, n): + return from_jordan(jordan_multiply(to_jordan(a), n)) + + +def fast_add(a, b): + return from_jordan(jordan_add(to_jordan(a), to_jordan(b))) + +# Functions for handling pubkey and privkey formats + + +def get_pubkey_format(pub): + if is_python2: + two = '\x02' + three = '\x03' + four = '\x04' + else: + two = 2 + three = 3 + four = 4 + + if isinstance(pub, (tuple, list)): return 'decimal' + elif len(pub) == 65 and pub[0] == four: return 'bin' + elif len(pub) == 130 and pub[0:2] == '04': return 'hex' + elif len(pub) == 33 and pub[0] in [two, three]: return 'bin_compressed' + elif len(pub) == 66 and pub[0:2] in ['02', '03']: return 'hex_compressed' + elif len(pub) == 64: return 'bin_electrum' + elif len(pub) == 128: return 'hex_electrum' + else: raise Exception("Pubkey not in recognized format") + + +def encode_pubkey(pub, formt): + if not isinstance(pub, (tuple, list)): + pub = decode_pubkey(pub) + if formt == 'decimal': return pub + elif formt == 'bin': return b'\x04' + encode(pub[0], 256, 32) + encode(pub[1], 256, 32) + elif formt == 'bin_compressed': + return from_int_to_byte(2+(pub[1] % 2)) + encode(pub[0], 256, 32) + elif formt == 'hex': return '04' + encode(pub[0], 16, 64) + encode(pub[1], 16, 64) + elif formt == 'hex_compressed': + return '0'+str(2+(pub[1] % 2)) + encode(pub[0], 16, 64) + elif formt == 'bin_electrum': return encode(pub[0], 256, 32) + encode(pub[1], 256, 32) + elif formt == 'hex_electrum': return encode(pub[0], 16, 64) + encode(pub[1], 16, 64) + else: raise Exception("Invalid format!") + + +def decode_pubkey(pub, formt=None): + if not formt: formt = get_pubkey_format(pub) + if formt == 'decimal': return pub + elif formt == 'bin': return (decode(pub[1:33], 256), decode(pub[33:65], 256)) + elif formt == 'bin_compressed': + x = decode(pub[1:33], 256) + beta = pow(int(x*x*x+A*x+B), int((P+1)//4), int(P)) + y = (P-beta) if ((beta + from_byte_to_int(pub[0])) % 2) else beta + return (x, y) + elif formt == 'hex': return (decode(pub[2:66], 16), decode(pub[66:130], 16)) + elif formt == 'hex_compressed': + return decode_pubkey(safe_from_hex(pub), 'bin_compressed') + elif formt == 'bin_electrum': + return (decode(pub[:32], 256), decode(pub[32:64], 256)) + elif formt == 'hex_electrum': + return (decode(pub[:64], 16), decode(pub[64:128], 16)) + else: raise Exception("Invalid format!") + +def get_privkey_format(priv): + if isinstance(priv, int_types): return 'decimal' + elif len(priv) == 32: return 'bin' + elif len(priv) == 33: return 'bin_compressed' + elif len(priv) == 64: return 'hex' + elif len(priv) == 66: return 'hex_compressed' + else: + bin_p = b58check_to_bin(priv) + if len(bin_p) == 32: return 'wif' + elif len(bin_p) == 33: return 'wif_compressed' + else: raise Exception("WIF does not represent privkey") + +def encode_privkey(priv, formt, vbyte=0): + if not isinstance(priv, int_types): + return encode_privkey(decode_privkey(priv), formt, vbyte) + if formt == 'decimal': return priv + elif formt == 'bin': return encode(priv, 256, 32) + elif formt == 'bin_compressed': return encode(priv, 256, 32)+b'\x01' + elif formt == 'hex': return encode(priv, 16, 64) + elif formt == 'hex_compressed': return encode(priv, 16, 64)+'01' + elif formt == 'wif': + return bin_to_b58check(encode(priv, 256, 32), 128+int(vbyte)) + elif formt == 'wif_compressed': + return bin_to_b58check(encode(priv, 256, 32)+b'\x01', 128+int(vbyte)) + else: raise Exception("Invalid format!") + +def decode_privkey(priv,formt=None): + if not formt: formt = get_privkey_format(priv) + if formt == 'decimal': return priv + elif formt == 'bin': return decode(priv, 256) + elif formt == 'bin_compressed': return decode(priv[:32], 256) + elif formt == 'hex': return decode(priv, 16) + elif formt == 'hex_compressed': return decode(priv[:64], 16) + elif formt == 'wif': return decode(b58check_to_bin(priv),256) + elif formt == 'wif_compressed': + return decode(b58check_to_bin(priv)[:32],256) + else: raise Exception("WIF does not represent privkey") + +def add_pubkeys(p1, p2): + f1, f2 = get_pubkey_format(p1), get_pubkey_format(p2) + return encode_pubkey(fast_add(decode_pubkey(p1, f1), decode_pubkey(p2, f2)), f1) + +def add_privkeys(p1, p2): + f1, f2 = get_privkey_format(p1), get_privkey_format(p2) + return encode_privkey((decode_privkey(p1, f1) + decode_privkey(p2, f2)) % N, f1) + + +def multiply(pubkey, privkey): + f1, f2 = get_pubkey_format(pubkey), get_privkey_format(privkey) + pubkey, privkey = decode_pubkey(pubkey, f1), decode_privkey(privkey, f2) + # http://safecurves.cr.yp.to/twist.html + if not isinf(pubkey) and (pubkey[0]**3+B-pubkey[1]*pubkey[1]) % P != 0: + raise Exception("Point not on curve") + return encode_pubkey(fast_multiply(pubkey, privkey), f1) + + +def divide(pubkey, privkey): + factor = inv(decode_privkey(privkey), N) + return multiply(pubkey, factor) + + +def compress(pubkey): + f = get_pubkey_format(pubkey) + if 'compressed' in f: return pubkey + elif f == 'bin': return encode_pubkey(decode_pubkey(pubkey, f), 'bin_compressed') + elif f == 'hex' or f == 'decimal': + return encode_pubkey(decode_pubkey(pubkey, f), 'hex_compressed') + + +def decompress(pubkey): + f = get_pubkey_format(pubkey) + if 'compressed' not in f: return pubkey + elif f == 'bin_compressed': return encode_pubkey(decode_pubkey(pubkey, f), 'bin') + elif f == 'hex_compressed' or f == 'decimal': + return encode_pubkey(decode_pubkey(pubkey, f), 'hex') + + +def privkey_to_pubkey(privkey): + f = get_privkey_format(privkey) + privkey = decode_privkey(privkey, f) + if privkey >= N: + raise Exception("Invalid privkey") + if f in ['bin', 'bin_compressed', 'hex', 'hex_compressed', 'decimal']: + return encode_pubkey(fast_multiply(G, privkey), f) + else: + return encode_pubkey(fast_multiply(G, privkey), f.replace('wif', 'hex')) + +privtopub = privkey_to_pubkey + + +def privkey_to_address(priv, magicbyte=0): + return pubkey_to_address(privkey_to_pubkey(priv), magicbyte) +privtoaddr = privkey_to_address + + +def neg_pubkey(pubkey): + f = get_pubkey_format(pubkey) + pubkey = decode_pubkey(pubkey, f) + return encode_pubkey((pubkey[0], (P-pubkey[1]) % P), f) + + +def neg_privkey(privkey): + f = get_privkey_format(privkey) + privkey = decode_privkey(privkey, f) + return encode_privkey((N - privkey) % N, f) + +def subtract_pubkeys(p1, p2): + f1, f2 = get_pubkey_format(p1), get_pubkey_format(p2) + k2 = decode_pubkey(p2, f2) + return encode_pubkey(fast_add(decode_pubkey(p1, f1), (k2[0], (P - k2[1]) % P)), f1) + + +def subtract_privkeys(p1, p2): + f1, f2 = get_privkey_format(p1), get_privkey_format(p2) + k2 = decode_privkey(p2, f2) + return encode_privkey((decode_privkey(p1, f1) - k2) % N, f1) + +# Hashes + + +def bin_hash160(string): + intermed = hashlib.sha256(string).digest() + digest = '' + try: + digest = hashlib.new('ripemd160', intermed).digest() + except: + digest = RIPEMD160(intermed).digest() + return digest + + +def hash160(string): + return safe_hexlify(bin_hash160(string)) + + +def bin_sha256(string): + binary_data = string if isinstance(string, bytes) else bytes(string, 'utf-8') + return hashlib.sha256(binary_data).digest() + +def sha256(string): + return bytes_to_hex_string(bin_sha256(string)) + + +def bin_ripemd160(string): + try: + digest = hashlib.new('ripemd160', string).digest() + except: + digest = RIPEMD160(string).digest() + return digest + + +def ripemd160(string): + return safe_hexlify(bin_ripemd160(string)) + + +def bin_dbl_sha256(s): + bytes_to_hash = from_string_to_bytes(s) + return hashlib.sha256(hashlib.sha256(bytes_to_hash).digest()).digest() + + +def dbl_sha256(string): + return safe_hexlify(bin_dbl_sha256(string)) + + +def bin_slowsha(string): + string = from_string_to_bytes(string) + orig_input = string + for i in range(100000): + string = hashlib.sha256(string + orig_input).digest() + return string + + +def slowsha(string): + return safe_hexlify(bin_slowsha(string)) + + +def hash_to_int(x): + if len(x) in [40, 64]: + return decode(x, 16) + return decode(x, 256) + + +def num_to_var_int(x): + x = int(x) + if x < 253: return from_int_to_byte(x) + elif x < 65536: return from_int_to_byte(253)+encode(x, 256, 2)[::-1] + elif x < 4294967296: return from_int_to_byte(254) + encode(x, 256, 4)[::-1] + else: return from_int_to_byte(255) + encode(x, 256, 8)[::-1] + + +# WTF, Electrum? +def electrum_sig_hash(message): + padded = b"\x18Bitcoin Signed Message:\n" + num_to_var_int(len(message)) + from_string_to_bytes(message) + return bin_dbl_sha256(padded) + + +def random_key(): + # Gotta be secure after that java.SecureRandom fiasco... + entropy = random_string(32) \ + + str(random.randrange(2**256)) \ + + str(int(time.time() * 1000000)) + return sha256(entropy) + + +def random_electrum_seed(): + entropy = os.urandom(32) \ + + str(random.randrange(2**256)) \ + + str(int(time.time() * 1000000)) + return sha256(entropy)[:32] + +# Encodings + +def b58check_to_bin(inp): + leadingzbytes = len(re.match('^1*', inp).group(0)) + data = b'\x00' * leadingzbytes + changebase(inp, 58, 256) + assert bin_dbl_sha256(data[:-4])[:4] == data[-4:] + return data[1:-4] + + +def get_version_byte(inp): + leadingzbytes = len(re.match('^1*', inp).group(0)) + data = b'\x00' * leadingzbytes + changebase(inp, 58, 256) + assert bin_dbl_sha256(data[:-4])[:4] == data[-4:] + return ord(data[0]) + + +def hex_to_b58check(inp, magicbyte=0): + return bin_to_b58check(binascii.unhexlify(inp), magicbyte) + + +def b58check_to_hex(inp): + return safe_hexlify(b58check_to_bin(inp)) + + +def pubkey_to_address(pubkey, magicbyte=0): + if isinstance(pubkey, (list, tuple)): + pubkey = encode_pubkey(pubkey, 'bin') + if len(pubkey) in [66, 130]: + return bin_to_b58check( + bin_hash160(binascii.unhexlify(pubkey)), magicbyte) + return bin_to_b58check(bin_hash160(pubkey), magicbyte) + +pubtoaddr = pubkey_to_address + +# EDCSA + + +def encode_sig(v, r, s): + vb, rb, sb = from_int_to_byte(v), encode(r, 256), encode(s, 256) + + result = base64.b64encode(vb+b'\x00'*(32-len(rb))+rb+b'\x00'*(32-len(sb))+sb) + return result if is_python2 else str(result, 'utf-8') + + +def decode_sig(sig): + bytez = base64.b64decode(sig) + return from_byte_to_int(bytez[0]), decode(bytez[1:33], 256), decode(bytez[33:], 256) + +# https://tools.ietf.org/html/rfc6979#section-3.2 + + +def deterministic_generate_k(msghash, priv): + v = b'\x01' * 32 + k = b'\x00' * 32 + priv = encode_privkey(priv, 'bin') + msghash = encode(hash_to_int(msghash), 256, 32) + k = hmac.new(k, v+b'\x00'+priv+msghash, hashlib.sha256).digest() + v = hmac.new(k, v, hashlib.sha256).digest() + k = hmac.new(k, v+b'\x01'+priv+msghash, hashlib.sha256).digest() + v = hmac.new(k, v, hashlib.sha256).digest() + return decode(hmac.new(k, v, hashlib.sha256).digest(), 256) + + +def ecdsa_raw_sign(msghash, priv): + + z = hash_to_int(msghash) + k = deterministic_generate_k(msghash, priv) + + r, y = fast_multiply(G, k) + s = inv(k, N) * (z + r*decode_privkey(priv)) % N + + return 27+(y % 2), r, s + + +def ecdsa_sign(msg, priv): + return encode_sig(*ecdsa_raw_sign(electrum_sig_hash(msg), priv)) + + +def ecdsa_raw_verify(msghash, vrs, pub): + v, r, s = vrs + + w = inv(s, N) + z = hash_to_int(msghash) + + u1, u2 = z*w % N, r*w % N + x, y = fast_add(fast_multiply(G, u1), fast_multiply(decode_pubkey(pub), u2)) + + return r == x + + +def ecdsa_verify(msg, sig, pub): + return ecdsa_raw_verify(electrum_sig_hash(msg), decode_sig(sig), pub) + + +def ecdsa_raw_recover(msghash, vrs): + v, r, s = vrs + + x = r + beta = pow(x*x*x+A*x+B, (P+1)//4, P) + y = beta if v % 2 ^ beta % 2 else (P - beta) + z = hash_to_int(msghash) + Gz = jordan_multiply(((Gx, 1), (Gy, 1)), (N - z) % N) + XY = jordan_multiply(((x, 1), (y, 1)), s) + Qr = jordan_add(Gz, XY) + Q = jordan_multiply(Qr, inv(r, N)) + Q = from_jordan(Q) + + if ecdsa_raw_verify(msghash, vrs, Q): + return Q + return False + + +def ecdsa_recover(msg, sig): + return encode_pubkey(ecdsa_raw_recover(electrum_sig_hash(msg), decode_sig(sig)), 'hex') diff --git a/lib/bitcoin/py2specials.py b/lib/bitcoin/py2specials.py new file mode 100644 index 00000000..4e2e42bb --- /dev/null +++ b/lib/bitcoin/py2specials.py @@ -0,0 +1,94 @@ +import sys, re +import binascii +import os +import hashlib + + +if sys.version_info.major == 2: + string_types = (str, unicode) + string_or_bytes_types = string_types + int_types = (int, float, long) + + # Base switching + code_strings = { + 2: '01', + 10: '0123456789', + 16: '0123456789abcdef', + 32: 'abcdefghijklmnopqrstuvwxyz234567', + 58: '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz', + 256: ''.join([chr(x) for x in range(256)]) + } + + def bin_dbl_sha256(s): + bytes_to_hash = from_string_to_bytes(s) + return hashlib.sha256(hashlib.sha256(bytes_to_hash).digest()).digest() + + def lpad(msg, symbol, length): + if len(msg) >= length: + return msg + return symbol * (length - len(msg)) + msg + + def get_code_string(base): + if base in code_strings: + return code_strings[base] + else: + raise ValueError("Invalid base!") + + def changebase(string, frm, to, minlen=0): + if frm == to: + return lpad(string, get_code_string(frm)[0], minlen) + return encode(decode(string, frm), to, minlen) + + def bin_to_b58check(inp, magicbyte=0): + inp_fmtd = chr(int(magicbyte)) + inp + leadingzbytes = len(re.match('^\x00*', inp_fmtd).group(0)) + checksum = bin_dbl_sha256(inp_fmtd)[:4] + return '1' * leadingzbytes + changebase(inp_fmtd+checksum, 256, 58) + + def bytes_to_hex_string(b): + return b.encode('hex') + + def safe_from_hex(s): + return s.decode('hex') + + def from_int_representation_to_bytes(a): + return str(a) + + def from_int_to_byte(a): + return chr(a) + + def from_byte_to_int(a): + return ord(a) + + def from_bytes_to_string(s): + return s + + def from_string_to_bytes(a): + return a + + def safe_hexlify(a): + return binascii.hexlify(a) + + def encode(val, base, minlen=0): + base, minlen = int(base), int(minlen) + code_string = get_code_string(base) + result = "" + while val > 0: + result = code_string[val % base] + result + val //= base + return code_string[0] * max(minlen - len(result), 0) + result + + def decode(string, base): + base = int(base) + code_string = get_code_string(base) + result = 0 + if base == 16: + string = string.lower() + while len(string) > 0: + result *= base + result += code_string.find(string[0]) + string = string[1:] + return result + + def random_string(x): + return os.urandom(x) diff --git a/lib/bitcoin/py3specials.py b/lib/bitcoin/py3specials.py new file mode 100644 index 00000000..be234722 --- /dev/null +++ b/lib/bitcoin/py3specials.py @@ -0,0 +1,119 @@ +import sys, os +import binascii +import hashlib + + +if sys.version_info.major == 3: + string_types = (str) + string_or_bytes_types = (str, bytes) + int_types = (int, float) + # Base switching + code_strings = { + 2: '01', + 10: '0123456789', + 16: '0123456789abcdef', + 32: 'abcdefghijklmnopqrstuvwxyz234567', + 58: '123456789ABCDEFGHJKLMNPQRSTUVWXYZabcdefghijkmnopqrstuvwxyz', + 256: ''.join([chr(x) for x in range(256)]) + } + + def bin_dbl_sha256(s): + bytes_to_hash = from_string_to_bytes(s) + return hashlib.sha256(hashlib.sha256(bytes_to_hash).digest()).digest() + + def lpad(msg, symbol, length): + if len(msg) >= length: + return msg + return symbol * (length - len(msg)) + msg + + def get_code_string(base): + if base in code_strings: + return code_strings[base] + else: + raise ValueError("Invalid base!") + + def changebase(string, frm, to, minlen=0): + if frm == to: + return lpad(string, get_code_string(frm)[0], minlen) + return encode(decode(string, frm), to, minlen) + + def bin_to_b58check(inp, magicbyte=0): + inp_fmtd = from_int_to_byte(int(magicbyte))+inp + + leadingzbytes = 0 + for x in inp_fmtd: + if x != 0: + break + leadingzbytes += 1 + + checksum = bin_dbl_sha256(inp_fmtd)[:4] + return '1' * leadingzbytes + changebase(inp_fmtd+checksum, 256, 58) + + def bytes_to_hex_string(b): + if isinstance(b, str): + return b + + return ''.join('{:02x}'.format(y) for y in b) + + def safe_from_hex(s): + return bytes.fromhex(s) + + def from_int_representation_to_bytes(a): + return bytes(str(a), 'utf-8') + + def from_int_to_byte(a): + return bytes([a]) + + def from_byte_to_int(a): + return a + + def from_string_to_bytes(a): + return a if isinstance(a, bytes) else bytes(a, 'utf-8') + + def safe_hexlify(a): + return str(binascii.hexlify(a), 'utf-8') + + def encode(val, base, minlen=0): + base, minlen = int(base), int(minlen) + code_string = get_code_string(base) + result_bytes = bytes() + while val > 0: + curcode = code_string[val % base] + result_bytes = bytes([ord(curcode)]) + result_bytes + val //= base + + pad_size = minlen - len(result_bytes) + + padding_element = b'\x00' if base == 256 else b'1' \ + if base == 58 else b'0' + if (pad_size > 0): + result_bytes = padding_element*pad_size + result_bytes + + result_string = ''.join([chr(y) for y in result_bytes]) + result = result_bytes if base == 256 else result_string + + return result + + def decode(string, base): + if base == 256 and isinstance(string, str): + string = bytes(bytearray.fromhex(string)) + base = int(base) + code_string = get_code_string(base) + result = 0 + if base == 256: + def extract(d, cs): + return d + else: + def extract(d, cs): + return cs.find(d if isinstance(d, str) else chr(d)) + + if base == 16: + string = string.lower() + while len(string) > 0: + result *= base + result += extract(string[0], code_string) + string = string[1:] + return result + + def random_string(x): + return str(os.urandom(x)) diff --git a/bitcoin/ripemd.py b/lib/bitcoin/ripemd.py similarity index 93% rename from bitcoin/ripemd.py rename to lib/bitcoin/ripemd.py index a9d652c5..4b0c6045 100644 --- a/bitcoin/ripemd.py +++ b/lib/bitcoin/ripemd.py @@ -43,10 +43,18 @@ except ImportError: pass +import sys + +is_python2 = sys.version_info.major == 2 #block_size = 1 digest_size = 20 digestsize = 20 +try: + range = xrange +except: + pass + class RIPEMD160: """Return a new RIPEMD160 object. An optional string argument may be provided; if present, this string will be automatically @@ -77,7 +85,10 @@ def hexdigest(self): dig = self.digest() hex_digest = '' for d in dig: - hex_digest += '%02x' % ord(d) + if (is_python2): + hex_digest += '%02x' % ord(d) + else: + hex_digest += '%02x' % d return hex_digest def copy(self): @@ -155,7 +166,10 @@ def R(a, b, c, d, e, Fj, Kj, sj, rj, X): def RMD160Transform(state, block): #uint32 state[5], uchar block[64] x = [0]*16 if sys.byteorder == 'little': - x = struct.unpack('<16L', ''.join([chr(x) for x in block[0:64]])) + if is_python2: + x = struct.unpack('<16L', ''.join([chr(x) for x in block[0:64]])) + else: + x = struct.unpack('<16L', bytes(block[0:64])) else: raise "Error!!" a = state[0] @@ -362,13 +376,14 @@ def RMD160Update(ctx, inp, inplen): if type(inp) == str: inp = [ord(i)&0xff for i in inp] - have = (ctx.count / 8) % 64 + have = int((ctx.count // 8) % 64) + inplen = int(inplen) need = 64 - have ctx.count += 8 * inplen off = 0 if inplen >= need: if have: - for i in xrange(need): + for i in range(need): ctx.buffer[have+i] = inp[i] RMD160Transform(ctx.state, ctx.buffer) off = need @@ -378,12 +393,12 @@ def RMD160Update(ctx, inp, inplen): off += 64 if off < inplen: # memcpy(ctx->buffer + have, input+off, len-off); - for i in xrange(inplen - off): + for i in range(inplen - off): ctx.buffer[have+i] = inp[off+i] def RMD160Final(ctx): size = struct.pack("N//2 else s # BIP62 low s + b1, b2 = encode(r, 256), encode(s, 256) + if bytearray(b1)[0] & 0x80: # add null bytes if leading byte interpreted as negative + b1 = b'\x00' + b1 + if bytearray(b2)[0] & 0x80: + b2 = b'\x00' + b2 + left = b'\x02' + encode(len(b1), 256, 1) + b1 + right = b'\x02' + encode(len(b2), 256, 1) + b2 + return safe_hexlify(b'\x30' + encode(len(left+right), 256, 1) + left + right) + +def der_decode_sig(sig): + leftlen = decode(sig[6:8], 16)*2 + left = sig[8:8+leftlen] + rightlen = decode(sig[10+leftlen:12+leftlen], 16)*2 + right = sig[12+leftlen:12+leftlen+rightlen] + return (None, decode(left, 16), decode(right, 16)) + + +def txhash(tx, hashcode=None): + if isinstance(tx, str) and re.match('^[0-9a-fA-F]*$', tx): + tx = changebase(tx, 16, 256) + if hashcode: + return dbl_sha256(from_string_to_bytes(tx) + encode(int(hashcode), 256, 4)[::-1]) + else: + return safe_hexlify(bin_dbl_sha256(tx)[::-1]) + + +def bin_txhash(tx, hashcode=None): + return binascii.unhexlify(txhash(tx, hashcode)) + + +def ecdsa_tx_sign(tx, priv, hashcode=SIGHASH_ALL): + rawsig = ecdsa_raw_sign(bin_txhash(tx, hashcode), priv) + return der_encode_sig(*rawsig)+encode(hashcode, 16, 2) + + +def ecdsa_tx_verify(tx, sig, pub, hashcode=SIGHASH_ALL): + return ecdsa_raw_verify(bin_txhash(tx, hashcode), der_decode_sig(sig), pub) + + +def ecdsa_tx_recover(tx, sig, hashcode=SIGHASH_ALL): + z = bin_txhash(tx, hashcode) + _, r, s = der_decode_sig(sig) + left = ecdsa_raw_recover(z, (0, r, s)) + right = ecdsa_raw_recover(z, (1, r, s)) + return (encode_pubkey(left, 'hex'), encode_pubkey(right, 'hex')) + +# Scripts + + +def mk_pubkey_script(addr): + # Keep the auxiliary functions around for altcoins' sake + return '76a914' + b58check_to_hex(addr) + '88ac' + + +def mk_scripthash_script(addr): + return 'a914' + b58check_to_hex(addr) + '87' + +# Address representation to output script + + +def address_to_script(addr): + if addr[0] == '3' or addr[0] == '2': + return mk_scripthash_script(addr) + else: + return mk_pubkey_script(addr) + +# Output script to address representation + + +def script_to_address(script, vbyte=0): + if re.match('^[0-9a-fA-F]*$', script): + script = binascii.unhexlify(script) + if script[:3] == b'\x76\xa9\x14' and script[-2:] == b'\x88\xac' and len(script) == 25: + return bin_to_b58check(script[3:-2], vbyte) # pubkey hash addresses + else: + if vbyte in [111, 196]: + # Testnet + scripthash_byte = 196 + else: + scripthash_byte = 5 + # BIP0016 scripthash addresses + return bin_to_b58check(script[2:-1], scripthash_byte) + + +def p2sh_scriptaddr(script, magicbyte=5): + if re.match('^[0-9a-fA-F]*$', script): + script = binascii.unhexlify(script) + return hex_to_b58check(hash160(script), magicbyte) +scriptaddr = p2sh_scriptaddr + + +def deserialize_script(script): + if isinstance(script, str) and re.match('^[0-9a-fA-F]*$', script): + return json_changebase(deserialize_script(binascii.unhexlify(script)), + lambda x: safe_hexlify(x)) + out, pos = [], 0 + while pos < len(script): + code = from_byte_to_int(script[pos]) + if code == 0: + out.append(None) + pos += 1 + elif code <= 75: + out.append(script[pos+1:pos+1+code]) + pos += 1 + code + elif code <= 78: + szsz = pow(2, code - 76) + sz = decode(script[pos+szsz: pos:-1], 256) + out.append(script[pos + 1 + szsz:pos + 1 + szsz + sz]) + pos += 1 + szsz + sz + elif code <= 96: + out.append(code - 80) + pos += 1 + else: + out.append(code) + pos += 1 + return out + + +def serialize_script_unit(unit): + if isinstance(unit, int): + if unit < 16: + return from_int_to_byte(unit + 80) + else: + return bytes([unit]) + elif unit is None: + return b'\x00' + else: + if len(unit) <= 75: + return from_int_to_byte(len(unit))+unit + elif len(unit) < 256: + return from_int_to_byte(76)+from_int_to_byte(len(unit))+unit + elif len(unit) < 65536: + return from_int_to_byte(77)+encode(len(unit), 256, 2)[::-1]+unit + else: + return from_int_to_byte(78)+encode(len(unit), 256, 4)[::-1]+unit + + +if is_python2: + def serialize_script(script): + if json_is_base(script, 16): + return binascii.hexlify(serialize_script(json_changebase(script, + lambda x: binascii.unhexlify(x)))) + return ''.join(map(serialize_script_unit, script)) +else: + def serialize_script(script): + if json_is_base(script, 16): + return safe_hexlify(serialize_script(json_changebase(script, + lambda x: binascii.unhexlify(x)))) + + result = bytes() + for b in map(serialize_script_unit, script): + result += b if isinstance(b, bytes) else bytes(b, 'utf-8') + return result + + +def mk_multisig_script(*args): # [pubs],k or pub1,pub2...pub[n],k + if isinstance(args[0], list): + pubs, k = args[0], int(args[1]) + else: + pubs = list(filter(lambda x: len(str(x)) >= 32, args)) + k = int(args[len(pubs)]) + return serialize_script([k]+pubs+[len(pubs)]) + 'ae' + +# Signing and verifying + + +def verify_tx_input(tx, i, script, sig, pub): + if re.match('^[0-9a-fA-F]*$', tx): + tx = binascii.unhexlify(tx) + if re.match('^[0-9a-fA-F]*$', script): + script = binascii.unhexlify(script) + if not re.match('^[0-9a-fA-F]*$', sig): + sig = safe_hexlify(sig) + hashcode = decode(sig[-2:], 16) + modtx = signature_form(tx, int(i), script, hashcode) + return ecdsa_tx_verify(modtx, sig, pub, hashcode) + + +def sign(tx, i, priv, hashcode=SIGHASH_ALL): + i = int(i) + if (not is_python2 and isinstance(re, bytes)) or not re.match('^[0-9a-fA-F]*$', tx): + return binascii.unhexlify(sign(safe_hexlify(tx), i, priv)) + if len(priv) <= 33: + priv = safe_hexlify(priv) + pub = privkey_to_pubkey(priv) + address = pubkey_to_address(pub) + signing_tx = signature_form(tx, i, mk_pubkey_script(address), hashcode) + sig = ecdsa_tx_sign(signing_tx, priv, hashcode) + txobj = deserialize(tx) + txobj["ins"][i]["script"] = serialize_script([sig, pub]) + return serialize(txobj) + + +def signall(tx, priv): + # if priv is a dictionary, assume format is + # { 'txinhash:txinidx' : privkey } + if isinstance(priv, dict): + for e, i in enumerate(deserialize(tx)["ins"]): + k = priv["%s:%d" % (i["outpoint"]["hash"], i["outpoint"]["index"])] + tx = sign(tx, e, k) + else: + for i in range(len(deserialize(tx)["ins"])): + tx = sign(tx, i, priv) + return tx + + +def multisign(tx, i, script, pk, hashcode=SIGHASH_ALL): + if re.match('^[0-9a-fA-F]*$', tx): + tx = binascii.unhexlify(tx) + if re.match('^[0-9a-fA-F]*$', script): + script = binascii.unhexlify(script) + modtx = signature_form(tx, i, script, hashcode) + return ecdsa_tx_sign(modtx, pk, hashcode) + + +def apply_multisignatures(*args): + # tx,i,script,sigs OR tx,i,script,sig1,sig2...,sig[n] + tx, i, script = args[0], int(args[1]), args[2] + sigs = args[3] if isinstance(args[3], list) else list(args[3:]) + + if isinstance(script, str) and re.match('^[0-9a-fA-F]*$', script): + script = binascii.unhexlify(script) + sigs = [binascii.unhexlify(x) if x[:2] == '30' else x for x in sigs] + if isinstance(tx, str) and re.match('^[0-9a-fA-F]*$', tx): + return safe_hexlify(apply_multisignatures(binascii.unhexlify(tx), i, script, sigs)) + + txobj = deserialize(tx) + txobj["ins"][i]["script"] = serialize_script([None]+sigs+[script]) + return serialize(txobj) + + +def is_inp(arg): + return len(arg) > 64 or "output" in arg or "outpoint" in arg + + +def mktx(*args): + # [in0, in1...],[out0, out1...] or in0, in1 ... out0 out1 ... + ins, outs = [], [] + for arg in args: + if isinstance(arg, list): + for a in arg: (ins if is_inp(a) else outs).append(a) + else: + (ins if is_inp(arg) else outs).append(arg) + + txobj = {"locktime": 0, "version": 1, "ins": [], "outs": []} + for i in ins: + if isinstance(i, dict) and "outpoint" in i: + txobj["ins"].append(i) + else: + if isinstance(i, dict) and "output" in i: + i = i["output"] + txobj["ins"].append({ + "outpoint": {"hash": i[:64], "index": int(i[65:])}, + "script": "", + "sequence": 4294967295 + }) + for o in outs: + if isinstance(o, string_or_bytes_types): + addr = o[:o.find(':')] + val = int(o[o.find(':')+1:]) + o = {} + if re.match('^[0-9a-fA-F]*$', addr): + o["script"] = addr + else: + o["address"] = addr + o["value"] = val + + outobj = {} + if "address" in o: + outobj["script"] = address_to_script(o["address"]) + elif "script" in o: + outobj["script"] = o["script"] + else: + raise Exception("Could not find 'address' or 'script' in output.") + outobj["value"] = o["value"] + txobj["outs"].append(outobj) + + return serialize(txobj) + + +def select(unspent, value): + value = int(value) + high = [u for u in unspent if u["value"] >= value] + high.sort(key=lambda u: u["value"]) + low = [u for u in unspent if u["value"] < value] + low.sort(key=lambda u: -u["value"]) + if len(high): + return [high[0]] + i, tv = 0, 0 + while tv < value and i < len(low): + tv += low[i]["value"] + i += 1 + if tv < value: + raise Exception("Not enough funds") + return low[:i] + +# Only takes inputs of the form { "output": blah, "value": foo } + + +def mksend(*args): + argz, change, fee = args[:-2], args[-2], int(args[-1]) + ins, outs = [], [] + for arg in argz: + if isinstance(arg, list): + for a in arg: + (ins if is_inp(a) else outs).append(a) + else: + (ins if is_inp(arg) else outs).append(arg) + + isum = sum([i["value"] for i in ins]) + osum, outputs2 = 0, [] + for o in outs: + if isinstance(o, string_types): + o2 = { + "address": o[:o.find(':')], + "value": int(o[o.find(':')+1:]) + } + else: + o2 = o + outputs2.append(o2) + osum += o2["value"] + + if isum < osum+fee: + raise Exception("Not enough money") + elif isum > osum+fee+5430: + outputs2 += [{"address": change, "value": isum-osum-fee}] + + return mktx(ins, outputs2) diff --git a/lib/blockchaininterface.py b/lib/blockchaininterface.py new file mode 100644 index 00000000..a491f18b --- /dev/null +++ b/lib/blockchaininterface.py @@ -0,0 +1,632 @@ +#from joinmarket import * +import unittest +import json, threading, abc, pprint, time, random, sys, os, re +import BaseHTTPServer, urllib +from decimal import Decimal +import bitcoin as btc + +import common +import jsonrpc + +# This can be removed once CliJsonRpc is gone. +import subprocess + +class CliJsonRpc(object): + """ + Fake JsonRpc class that uses the Bitcoin CLI executable. This is used + as temporary fall back before we switch completely (and exclusively) + to the real JSON-RPC interface. + """ + + def __init__(self, cli, testnet): + self.cli = cli + if testnet: + self.cli.append("-testnet") + + def call(self, method, params): + fullCall = [] + fullCall.extend (self.cli) + fullCall.append (method) + for p in params: + if isinstance(p, basestring): + fullCall.append(p) + else: + fullCall.append(json.dumps(p)) + + res = subprocess.check_output(fullCall) + + if res == '': + return None + + try: + return json.loads (res) + except ValueError: + return res.strip() + +def get_blockchain_interface_instance(config): + source = config.get("BLOCKCHAIN", "blockchain_source") + network = common.get_network() + testnet = network=='testnet' + if source == 'bitcoin-rpc': + rpc_host = config.get("BLOCKCHAIN", "rpc_host") + rpc_port = config.get("BLOCKCHAIN", "rpc_port") + rpc_user = config.get("BLOCKCHAIN", "rpc_user") + rpc_password = config.get("BLOCKCHAIN", "rpc_password") + rpc = jsonrpc.JsonRpc(rpc_host, rpc_port, rpc_user, rpc_password) + bc_interface = BitcoinCoreInterface(rpc, network) + elif source == 'json-rpc': + bitcoin_cli_cmd = config.get("BLOCKCHAIN", "bitcoin_cli_cmd").split(' ') + rpc = CliJsonRpc(bitcoin_cli_cmd, testnet) + bc_interface = BitcoinCoreInterface(rpc, network) + elif source == 'regtest': + rpc_host = config.get("BLOCKCHAIN", "rpc_host") + rpc_port = config.get("BLOCKCHAIN", "rpc_port") + rpc_user = config.get("BLOCKCHAIN", "rpc_user") + rpc_password = config.get("BLOCKCHAIN", "rpc_password") + rpc = jsonrpc.JsonRpc(rpc_host, rpc_port, rpc_user, rpc_password) + bc_interface = RegtestBitcoinCoreInterface(rpc) + elif source == 'blockr': + bc_interface = BlockrInterface(testnet) + else: + raise ValueError("Invalid blockchain source") + return bc_interface + +class BlockchainInterface(object): + __metaclass__ = abc.ABCMeta + def __init__(self): + pass + + def sync_wallet(self, wallet): + self.sync_addresses(wallet) + self.sync_unspent(wallet) + + @abc.abstractmethod + def sync_addresses(self, wallet): + '''Finds which addresses have been used and sets wallet.index appropriately''' + pass + + @abc.abstractmethod + def sync_unspent(self, wallet): + '''Finds the unspent transaction outputs belonging to this wallet, sets wallet.unspent''' + pass + + @abc.abstractmethod + def add_tx_notify(self, txd, unconfirmfun, confirmfun, notifyaddr): + '''Invokes unconfirmfun and confirmfun when tx is seen on the network''' + pass + + @abc.abstractmethod + def pushtx(self, txhex): + '''pushes tx to the network, returns txhash, or None if failed''' + pass + + @abc.abstractmethod + def query_utxo_set(self, txouts): + ''' + takes a utxo or a list of utxos + returns None if they are spend or unconfirmed + otherwise returns value in satoshis, address and output script + ''' + #address and output script contain the same information btw + +class BlockrInterface(BlockchainInterface): + BLOCKR_MAX_ADDR_REQ_COUNT = 20 + + def __init__(self, testnet = False): + super(BlockrInterface, self).__init__() + self.network = 'testnet' if testnet else 'btc' #see bci.py in bitcoin module + self.blockr_domain = 'tbtc' if testnet else 'btc' + self.last_sync_unspent = 0 + + def sync_addresses(self, wallet): + common.debug('downloading wallet history') + #sets Wallet internal indexes to be at the next unused address + for mix_depth in range(wallet.max_mix_depth): + for forchange in [0, 1]: + unused_addr_count = 0 + last_used_addr = '' + while unused_addr_count < wallet.gaplimit or\ + wallet.index[mix_depth][forchange] <= wallet.index_cache[mix_depth][forchange]: + addrs = [wallet.get_new_addr(mix_depth, forchange) for i in range(self.BLOCKR_MAX_ADDR_REQ_COUNT)] + + #TODO send a pull request to pybitcointools + # because this surely should be possible with a function from it + blockr_url = 'http://' + self.blockr_domain + '.blockr.io/api/v1/address/txs/' + #print 'downloading, lastusedaddr = ' + last_used_addr + ' unusedaddrcount= ' + str(unused_addr_count) + res = btc.make_request(blockr_url+','.join(addrs)) + data = json.loads(res)['data'] + for dat in data: + #if forchange == 0: + # print ' nbtxs ' + str(dat['nb_txs']) + ' addr=' + dat['address'] + ' unused=' + str(unused_addr_count) + if dat['nb_txs'] != 0: + last_used_addr = dat['address'] + unused_addr_count = 0 + else: + unused_addr_count += 1 + if last_used_addr == '': + wallet.index[mix_depth][forchange] = 0 + else: + wallet.index[mix_depth][forchange] = wallet.addr_cache[last_used_addr][2] + 1 + + def sync_unspent(self, wallet): + #finds utxos in the wallet + st = time.time() + rate_limit_time = 10*60 #dont refresh unspent dict more often than 10 minutes + if st - self.last_sync_unspent < rate_limit_time: + common.debug('blockr sync_unspent() happened too recently (%dsec), skipping' % (st - self.last_sync_unspent)) + return + wallet.unspent = {} + + addrs = wallet.addr_cache.keys() + if len(addrs) == 0: + common.debug('no tx used') + return + i = 0 + while i < len(addrs): + inc = min(len(addrs) - i, self.BLOCKR_MAX_ADDR_REQ_COUNT) + req = addrs[i:i + inc] + i += inc + + #TODO send a pull request to pybitcointools + # unspent() doesnt tell you which address, you get a bunch of utxos + # but dont know which privkey to sign with + + blockr_url = 'http://' + self.blockr_domain + '.blockr.io/api/v1/address/unspent/' + res = btc.make_request(blockr_url+','.join(req)) + data = json.loads(res)['data'] + if 'unspent' in data: + data = [data] + for dat in data: + for u in dat['unspent']: + wallet.unspent[u['tx']+':'+str(u['n'])] = {'address': + dat['address'], 'value': int(u['amount'].replace('.', ''))} + for u in wallet.spent_utxos: + wallet.unspent.pop(u, None) + + self.last_sync_unspent = time.time() + common.debug('blockr sync_unspent took ' + str((self.last_sync_unspent - st)) + 'sec') + + def add_tx_notify(self, txd, unconfirmfun, confirmfun, notifyaddr): + unconfirm_timeout = 10*60 #seconds + unconfirm_poll_period = 5 + confirm_timeout = 2*60*60 + confirm_poll_period = 5*60 + class NotifyThread(threading.Thread): + def __init__(self, blockr_domain, txd, unconfirmfun, confirmfun): + threading.Thread.__init__(self) + self.daemon = True + self.blockr_domain = blockr_domain + self.unconfirmfun = unconfirmfun + self.confirmfun = confirmfun + self.tx_output_set = set([(sv['script'], sv['value']) for sv in txd['outs']]) + self.output_addresses = [btc.script_to_address(scrval[0], + common.get_p2pk_vbyte()) for scrval in self.tx_output_set] + common.debug('txoutset=' + pprint.pformat(self.tx_output_set)) + common.debug('outaddrs=' + ','.join(self.output_addresses)) + + def run(self): + st = int(time.time()) + unconfirmed_txid = None + unconfirmed_txhex = None + while not unconfirmed_txid: + time.sleep(unconfirm_poll_period) + if int(time.time()) - st > unconfirm_timeout: + common.debug('checking for unconfirmed tx timed out') + return + blockr_url = 'http://' + self.blockr_domain + '.blockr.io/api/v1/address/unspent/' + random.shuffle(self.output_addresses) #seriously weird bug with blockr.io + data = json.loads(btc.make_request(blockr_url + ','.join(self.output_addresses) + '?unconfirmed=1'))['data'] + shared_txid = None + for unspent_list in data: + txs = set([str(txdata['tx']) for txdata in unspent_list['unspent']]) + if not shared_txid: + shared_txid = txs + else: + shared_txid = shared_txid.intersection(txs) + common.debug('sharedtxid = ' + str(shared_txid)) + if len(shared_txid) == 0: + continue + time.sleep(2) #here for some race condition bullshit with blockr.io + blockr_url = 'http://' + self.blockr_domain + '.blockr.io/api/v1/tx/raw/' + data = json.loads(btc.make_request(blockr_url + ','.join(shared_txid)))['data'] + if not isinstance(data, list): + data = [data] + for txinfo in data: + txhex = str(txinfo['tx']['hex']) + outs = set([(sv['script'], sv['value']) for sv in btc.deserialize(txhex)['outs']]) + common.debug('unconfirm query outs = ' + str(outs)) + if outs == self.tx_output_set: + unconfirmed_txid = txinfo['tx']['txid'] + unconfirmed_txhex = str(txinfo['tx']['hex']) + break + + self.unconfirmfun(btc.deserialize(unconfirmed_txhex), unconfirmed_txid) + + st = int(time.time()) + confirmed_txid = None + confirmed_txhex = None + while not confirmed_txid: + time.sleep(confirm_poll_period) + if int(time.time()) - st > confirm_timeout: + common.debug('checking for confirmed tx timed out') + return + blockr_url = 'http://' + self.blockr_domain + '.blockr.io/api/v1/address/txs/' + data = json.loads(btc.make_request(blockr_url + ','.join(self.output_addresses)))['data'] + shared_txid = None + for addrtxs in data: + txs = set([str(txdata['tx']) for txdata in addrtxs['txs']]) + if not shared_txid: + shared_txid = txs + else: + shared_txid = shared_txid.intersection(txs) + common.debug('sharedtxid = ' + str(shared_txid)) + if len(shared_txid) == 0: + continue + blockr_url = 'http://' + self.blockr_domain + '.blockr.io/api/v1/tx/raw/' + data = json.loads(btc.make_request(blockr_url + ','.join(shared_txid)))['data'] + if not isinstance(data, list): + data = [data] + for txinfo in data: + txhex = str(txinfo['tx']['hex']) + outs = set([(sv['script'], sv['value']) for sv in btc.deserialize(txhex)['outs']]) + common.debug('confirm query outs = ' + str(outs)) + if outs == self.tx_output_set: + confirmed_txid = txinfo['tx']['txid'] + confirmed_txhex = str(txinfo['tx']['hex']) + break + self.confirmfun(btc.deserialize(confirmed_txhex), confirmed_txid, 1) + + NotifyThread(self.blockr_domain, txd, unconfirmfun, confirmfun).start() + + def pushtx(self, txhex): + try: + json_str = btc.blockr_pushtx(txhex, self.network) + except Exception: + common.debug('failed blockr.io pushtx') + return None + data = json.loads(json_str) + if data['status'] != 'success': + common.debug(data) + return None + return data['data'] + + def query_utxo_set(self, txout): + if not isinstance(txout, list): + txout = [txout] + txids = [h[:64] for h in txout] + txids = list(set(txids)) #remove duplicates + #self.BLOCKR_MAX_ADDR_REQ_COUNT = 2 + if len(txids) > self.BLOCKR_MAX_ADDR_REQ_COUNT: + txids = common.chunks(txids, self.BLOCKR_MAX_ADDR_REQ_COUNT) + else: + txids = [txids] + data = [] + for ids in txids: + blockr_url = 'http://' + self.blockr_domain + '.blockr.io/api/v1/tx/info/' + blockr_data = json.loads(btc.make_request(blockr_url + ','.join(ids)))['data'] + if not isinstance(blockr_data, list): + blockr_data = [blockr_data] + data += blockr_data + result = [] + for txo in txout: + txdata = [d for d in data if d['tx'] == txo[:64]][0] + vout = [v for v in txdata['vouts'] if v['n'] == int(txo[65:])][0] + if vout['is_spent'] == 1: + result.append(None) + else: + result.append({'value': int(Decimal(vout['amount'])*Decimal('1e8')), + 'address': vout['address'], 'script': vout['extras']['script']}) + return result + + +class NotifyRequestHeader(BaseHTTPServer.BaseHTTPRequestHandler): + def __init__(self, request, client_address, base_server): + self.btcinterface = base_server.btcinterface + self.base_server = base_server + BaseHTTPServer.BaseHTTPRequestHandler.__init__(self, request, client_address, base_server) + + def do_HEAD(self): + pages = ('/walletnotify?', '/alertnotify?') + + if self.path.startswith('/walletnotify?'): + txid = self.path[len(pages[0]):] + if not re.match('^[0-9a-fA-F]*$', txid): + common.debug('not a txid') + return + tx = self.btcinterface.rpc('getrawtransaction', [txid]) + if not re.match('^[0-9a-fA-F]*$', tx): + common.debug('not a txhex') + return + txd = btc.deserialize(tx) + tx_output_set = set([(sv['script'], sv['value']) for sv in txd['outs']]) + + unconfirmfun, confirmfun = None, None + for tx_out, ucfun, cfun in self.btcinterface.txnotify_fun: + if tx_out == tx_output_set: + unconfirmfun = ucfun + confirmfun = cfun + break + if unconfirmfun == None: + common.debug('txid=' + txid + ' not being listened for') + else: + txdata = None #on rare occasions people spend their output without waiting for a confirm + for n in range(len(txd['outs'])): + txdata = self.btcinterface.rpc('gettxout', [txid, n, True]) + if txdata is not None: + break + assert txdata != None + if txdata['confirmations'] == 0: + unconfirmfun(txd, txid) + #TODO pass the total transfered amount value here somehow + #wallet_name = self.get_wallet_name() + #amount = + #bitcoin-cli move wallet_name "" amount + common.debug('ran unconfirmfun') + else: + confirmfun(txd, txid, txdata['confirmations']) + self.btcinterface.txnotify_fun.remove((tx_out, unconfirmfun, confirmfun)) + common.debug('ran confirmfun') + + elif self.path.startswith('/alertnotify?'): + common.core_alert = urllib.unquote(self.path[len(pages[1]):]) + common.debug('Got an alert!\nMessage=' + common.core_alert) + + os.system('curl -sI --connect-timeout 1 http://localhost:' + + str(self.base_server.server_address[1] + 1) + self.path) + self.send_response(200) + #self.send_header('Connection', 'close') + self.end_headers() + +class BitcoinCoreNotifyThread(threading.Thread): + def __init__(self, btcinterface): + threading.Thread.__init__(self) + self.daemon = True + self.btcinterface = btcinterface + + def run(self): + notify_host = 'localhost' + notify_port = 62602 #defaults + if 'notify_host' in common.config.options("BLOCKCHAIN"): + notify_host = common.config.get("BLOCKCHAIN", "notify_host").strip() + if 'notify_port' in common.config.options("BLOCKCHAIN"): + notify_port = int(common.config.get("BLOCKCHAIN", "notify_port")) + for inc in range(10): + hostport = (notify_host, notify_port + inc) + try: + httpd = BaseHTTPServer.HTTPServer(hostport, NotifyRequestHeader) + except Exception: + continue + httpd.btcinterface = self.btcinterface + common.debug('started bitcoin core notify listening thread, host=' + + str(notify_host) + ' port=' + str(hostport[1])) + httpd.serve_forever() + common.debug('failed to bind for bitcoin core notify listening') + +#must run bitcoind with -server +#-walletnotify="curl -sI --connect-timeout 1 http://localhost:62602/walletnotify?%s" +#and make sure curl is installed (git uses it, odds are you've already got it) + +#TODO must add the tx addresses as watchonly if case we ever broadcast a tx +# with addresses not belonging to us +class BitcoinCoreInterface(BlockchainInterface): + def __init__(self, jsonRpc, network): + super(BitcoinCoreInterface, self).__init__() + self.jsonRpc = jsonRpc + + blockchainInfo = self.jsonRpc.call("getblockchaininfo", []) + actualNet = blockchainInfo['chain'] + + netmap = {'main': 'mainnet', 'test': 'testnet', 'regtest': 'regtest'} + if netmap[actualNet] != network: + raise Exception('wrong network configured') + + self.notifythread = None + self.txnotify_fun = [] + + def get_wallet_name(self, wallet): + return 'joinmarket-wallet-' + btc.dbl_sha256(wallet.keys[0][0])[:6] + + def rpc(self, method, args): + if method != 'importaddress': + common.debug('rpc: ' + method + " " + str(args)) + res = self.jsonRpc.call(method, args) + if isinstance(res, unicode): + res = str(res) + return res + + def add_watchonly_addresses(self, addr_list, wallet_name): + common.debug('importing ' + str(len(addr_list)) + ' addresses into account ' + wallet_name) + for addr in addr_list: + self.rpc('importaddress', [addr, wallet_name, False]) + if common.config.get("BLOCKCHAIN", "blockchain_source") != 'regtest': + print 'now restart bitcoind with -rescan, unless you\'re sure the wallet is completely empty and unused' + sys.exit(0) + + def sync_addresses(self, wallet): + if isinstance(wallet, common.BitcoinCoreWallet): + return + common.debug('requesting wallet history') + wallet_name = self.get_wallet_name(wallet) + addr_req_count = 50 + wallet_addr_list = [] + for mix_depth in range(wallet.max_mix_depth): + for forchange in [0, 1]: + wallet_addr_list += [wallet.get_new_addr(mix_depth, forchange) for i in range(addr_req_count)] + wallet.index[mix_depth][forchange] = 0 + imported_addr_list = self.rpc('getaddressesbyaccount', [wallet_name]) + if not set(wallet_addr_list).issubset(set(imported_addr_list)): + self.add_watchonly_addresses(wallet_addr_list, wallet_name) + return + + buf = self.rpc('listtransactions', [wallet_name, 1000, 0, True]) + txs = buf + # If the buffer's full, check for more, until it ain't + while len(buf) == 1000: + buf = self.rpc('listtransactions', [wallet_name, 1000, + len(txs), True]) + txs += buf + used_addr_list = [tx['address'] for tx in txs if tx['category'] == 'receive'] + too_few_addr_mix_change = [] + for mix_depth in range(wallet.max_mix_depth): + for forchange in [0, 1]: + unused_addr_count = 0 + last_used_addr = '' + breakloop = False + while not breakloop: + if unused_addr_count >= wallet.gaplimit and\ + wallet.index[mix_depth][forchange] >= wallet.index_cache[mix_depth][forchange]: + break + mix_change_addrs = [wallet.get_new_addr(mix_depth, forchange) for i in range(addr_req_count)] + for mc_addr in mix_change_addrs: + if mc_addr not in imported_addr_list: + too_few_addr_mix_change.append((mix_depth, forchange)) + breakloop = True + break + if mc_addr in used_addr_list: + last_used_addr = mc_addr + unused_addr_count = 0 + else: + unused_addr_count += 1 + + if last_used_addr == '': + wallet.index[mix_depth][forchange] = 0 + else: + wallet.index[mix_depth][forchange] = wallet.addr_cache[last_used_addr][2] + 1 + + wallet_addr_list = [] + if len(too_few_addr_mix_change) > 0: + common.debug('too few addresses in ' + str(too_few_addr_mix_change)) + for mix_depth, forchange in too_few_addr_mix_change: + wallet_addr_list += [wallet.get_new_addr(mix_depth, forchange) for i in range(addr_req_count*3)] + self.add_watchonly_addresses(wallet_addr_list, wallet_name) + return + + def sync_unspent(self, wallet): + if isinstance(wallet, common.BitcoinCoreWallet): + return + st = time.time() + wallet_name = self.get_wallet_name(wallet) + wallet.unspent = {} + unspent_list = self.rpc('listunspent', []) + for u in unspent_list: + if 'account' not in u: + continue + if u['account'] != wallet_name: + continue + if u['address'] not in wallet.addr_cache: + continue + wallet.unspent[u['txid'] + ':' + str(u['vout'])] = {'address': u['address'], + 'value': int(Decimal(str(u['amount'])) * Decimal('1e8'))} + et = time.time() + common.debug('bitcoind sync_unspent took ' + str((et - st)) + 'sec') + + def add_tx_notify(self, txd, unconfirmfun, confirmfun, notifyaddr): + if not self.notifythread: + self.notifythread = BitcoinCoreNotifyThread(self) + self.notifythread.start() + one_addr_imported = False + for outs in txd['outs']: + addr = btc.script_to_address(outs['script'], common.get_p2pk_vbyte()) + if self.rpc('getaccount', [addr]) != '': + one_addr_imported = True + break + if not one_addr_imported: + self.rpc('importaddress', [notifyaddr, 'joinmarket-notify', False]) + tx_output_set = set([(sv['script'], sv['value']) for sv in txd['outs']]) + self.txnotify_fun.append((tx_output_set, unconfirmfun, confirmfun)) + + def pushtx(self, txhex): + return self.rpc('sendrawtransaction', [txhex]) + + def query_utxo_set(self, txout): + if not isinstance(txout, list): + txout = [txout] + result = [] + for txo in txout: + ret = self.rpc('gettxout', [txo[:64], int(txo[65:]), False]) + if ret is None: + result.append(None) + else: + result.append({'value': int(Decimal(str(ret['value']))*Decimal('1e8')), + 'address': ret['scriptPubKey']['addresses'][0], 'script': ret['scriptPubKey']['hex']}) + return result + + +#class for regtest chain access +#running on local daemon. Only +#to be instantiated after network is up +#with > 100 blocks. +class RegtestBitcoinCoreInterface(BitcoinCoreInterface): + def __init__(self, jsonRpc): + super(RegtestBitcoinCoreInterface, self).__init__(jsonRpc, 'regtest') + + def pushtx(self, txhex): + ret = super(RegtestBitcoinCoreInterface, self).pushtx(txhex) + class TickChainThread(threading.Thread): + def __init__(self, bcinterface): + threading.Thread.__init__(self) + self.bcinterface = bcinterface + def run(self): + time.sleep(15) + self.bcinterface.tick_forward_chain(1) + TickChainThread(self).start() + return ret + + def tick_forward_chain(self, n): + '''Special method for regtest only; + instruct to mine n blocks.''' + self.rpc('setgenerate', [True, n]) + + def grab_coins(self, receiving_addr, amt=50): + ''' + NOTE! amt is passed in Coins, not Satoshis! + Special method for regtest only: + take coins from bitcoind's own wallet + and put them in the receiving addr. + Return the txid. + ''' + if amt > 500: + raise Exception("too greedy") + ''' + if amt > self.current_balance: + #mine enough to get to the reqd amt + reqd = int(amt - self.current_balance) + reqd_blocks = int(reqd/50) +1 + if self.rpc('setgenerate', [True, reqd_blocks]): + raise Exception("Something went wrong") + ''' + #now we do a custom create transaction and push to the receiver + txid = self.rpc('sendtoaddress', [receiving_addr, amt]) + if not txid: + raise Exception("Failed to broadcast transaction") + #confirm + self.tick_forward_chain(1) + return txid + + def get_received_by_addr(self, addresses, query_params): + #NB This will NOT return coinbase coins (but wont matter in our use case). + #allow importaddress to fail in case the address is already in the wallet + res = [] + for address in addresses: + self.rpc('importaddress', [address, 'watchonly']) + res.append({'address':address,'balance':\ + int(Decimal(1e8) * Decimal(self.rpc('getreceivedbyaddress', [address])))}) + return {'data':res} + +def main(): + #TODO some useful quick testing here, so people know if they've set it up right + myBCI = RegtestBitcoinCoreInterface() + #myBCI.send_tx('stuff') + print myBCI.get_utxos_from_addr(["n4EjHhGVS4Rod8ociyviR3FH442XYMWweD"]) + print myBCI.get_balance_at_addr(["n4EjHhGVS4Rod8ociyviR3FH442XYMWweD"]) + txid = myBCI.grab_coins('mygp9fsgEJ5U7jkPpDjX9nxRj8b5nC3Hnd',23) + print txid + print myBCI.get_balance_at_addr(['mygp9fsgEJ5U7jkPpDjX9nxRj8b5nC3Hnd']) + print myBCI.get_utxos_from_addr(['mygp9fsgEJ5U7jkPpDjX9nxRj8b5nC3Hnd']) + +if __name__ == '__main__': + main() + + + + diff --git a/lib/common.py b/lib/common.py new file mode 100644 index 00000000..73dd6506 --- /dev/null +++ b/lib/common.py @@ -0,0 +1,606 @@ + +import bitcoin as btc +from decimal import Decimal, InvalidOperation +from math import factorial, exp +import sys, datetime, json, time, pprint, threading, getpass +import random +import blockchaininterface, slowaes +from ConfigParser import SafeConfigParser, NoSectionError, NoOptionError +import os, io, itertools + +JM_VERSION = 2 +nickname = '' +DUST_THRESHOLD = 546 +bc_interface = None +ordername_list = ["absorder", "relorder"] +maker_timeout_sec = 30 + +debug_file_lock = threading.Lock() +debug_file_handle = None +core_alert = None +joinmarket_alert = None +debug_silence = False + +config = SafeConfigParser() +config_location = 'joinmarket.cfg' +# FIXME: Add rpc_* options here in the future! +required_options = {'BLOCKCHAIN':['blockchain_source', 'network'], + 'MESSAGING':['host','channel','port']} + +defaultconfig =\ +""" +[BLOCKCHAIN] +blockchain_source = blockr +#options: blockr, bitcoin-rpc, json-rpc, regtest +#for instructions on bitcoin-rpc read https://github.com/chris-belcher/joinmarket/wiki/Running-JoinMarket-with-Bitcoin-Core-full-node +network = mainnet +rpc_host = localhost +rpc_port = 8332 +rpc_user = bitcoin +rpc_password = password + +[MESSAGING] +host = irc.cyberguerrilla.org +channel = joinmarket-pit +port = 6697 +usessl = true +socks5 = false +socks5_host = localhost +socks5_port = 9050 +#for tor +#host = 6dvj6v5imhny3anf.onion +#port = 6697 +#usessl = true +#socks5 = true +maker_timeout_sec = 30 + +[POLICY] +#for dust sweeping, try merge_algorithm = gradual +merge_algorithm = default +""" + +def load_program_config(): + loadedFiles = config.read([config_location]) + #Create default config file if not found + if len(loadedFiles) != 1: + config.readfp(io.BytesIO(defaultconfig)) + with open(config_location, "w") as configfile: + configfile.write(defaultconfig) + + #check for sections + for s in required_options: + if s not in config.sections(): + raise Exception("Config file does not contain the required section: "+s) + #then check for specific options + for k,v in required_options.iteritems(): + for o in v: + if o not in config.options(k): + raise Exception("Config file does not contain the required option: "+o) + + try: + global maker_timeout_sec + maker_timeout_sec = config.getint('MESSAGING', 'maker_timeout_sec') + except NoOptionError: + debug('maker_timeout_sec not found in .cfg file, using default value') + + #configure the interface to the blockchain on startup + global bc_interface + bc_interface = blockchaininterface.get_blockchain_interface_instance(config) + +def get_config_irc_channel(): + channel = '#'+ config.get("MESSAGING","channel") + if get_network() == 'testnet': + channel += '-test' + return channel + +def debug(msg): + global debug_file_handle + with debug_file_lock: + if nickname and not debug_file_handle: + debug_file_handle = open(os.path.join('logs', nickname+'.log'),'ab',1) + outmsg = datetime.datetime.now().strftime("[%Y/%m/%d %H:%M:%S] ") + msg + if not debug_silence: + if core_alert: + print 'Core Alert Message: ' + core_alert + if joinmarket_alert: + print 'JoinMarket Alert Message: ' + joinmarket_alert + print outmsg + if nickname: #debugs before creating bot nick won't be handled like this + debug_file_handle.write(outmsg + '\r\n') + + +#Random functions - replacing some NumPy features +#NOTE THESE ARE NEITHER CRYPTOGRAPHICALLY SECURE +#NOR PERFORMANT NOR HIGH PRECISION! +#Only for sampling purposes +def rand_norm_array(mu, sigma, n): + #use normalvariate instead of gauss for thread safety + return [random.normalvariate(mu, sigma) for i in range(n)] + +def rand_exp_array(lamda, n): + #'lambda' is reserved (in case you are triggered by spelling errors) + return [random.expovariate(1.0 / lamda) for i in range(n)] + +def rand_pow_array(power, n): + #rather crude in that uses a uniform sample which is a multiple of 1e-4 + #for basis of formula, see: http://mathworld.wolfram.com/RandomNumber.html + return [y**(1.0/power) for y in [x*0.0001 for x in random.sample(xrange(10000),n)]] + +def rand_weighted_choice(n, p_arr): + '''Choose a value in 0..n-1 + with the choice weighted by the probabilities + in the list p_arr. Note that there will be some + floating point rounding errors, but see the note + at the top of this section.''' + if abs(sum(p_arr)-1.0) > 1e-4: + raise ValueError("Sum of probabilities must be 1") + if len(p_arr) != n: + raise ValueError("Need: "+str(n)+" probabilities.") + cum_pr = [sum(p_arr[:i+1]) for i in xrange(len(p_arr))] + r = random.random() + return sorted(cum_pr+[r]).index(r) +#End random functions + +def chunks(d, n): + return [d[x: x+n] for x in xrange(0, len(d), n)] + +def get_network(): + '''Returns network name''' + return config.get("BLOCKCHAIN","network") + +def get_p2sh_vbyte(): + if get_network() == 'testnet': + return 0xc4 + else: + return 0x05 + +def get_p2pk_vbyte(): + if get_network() == 'testnet': + return 0x6f + else: + return 0x00 + +def validate_address(addr): + try: + ver = btc.get_version_byte(addr) + except AssertionError: + return False, 'Checksum wrong. Typo in address?' + if ver != get_p2pk_vbyte() and ver != get_p2sh_vbyte(): + return False, 'Wrong address version. Testnet/mainnet confused?' + return True, 'address validated' + +def debug_dump_object(obj, skip_fields=[]): + debug('Class debug dump, name:' + obj.__class__.__name__) + for k, v in obj.__dict__.iteritems(): + if k in skip_fields: + continue + debug('key=' + k) + if isinstance(v, str): + debug('string: len:' + str(len(v))) + debug(v) + elif isinstance(v, dict) or isinstance(v, list): + debug(pprint.pformat(v)) + else: + debug(str(v)) + +def select_gradual(unspent, value): + ''' + UTXO selection algorithm for gradual dust reduction + If possible, combines outputs, picking as few as possible of the largest + utxos less than the target value; if the target value is larger than the + sum of all smaller utxos, uses the smallest utxo larger than the value. + ''' + value, key = int(value), lambda u: u["value"] + high = sorted([u for u in unspent if key(u) >= value], key=key) + low = sorted([u for u in unspent if key(u) < value], key=key) + lowsum=reduce(lambda x,y:x+y,map(key,low),0) + if value > lowsum: + if len(high)==0: + raise Exception('Not enough funds') + else: + return [high[0]] + else: + start, end, total = 0, 0, 0 + while total < value: + total += low[end]['value'] + end += 1 + while total >= value + low[start]['value']: + total -= low[start]['value'] + start += 1 + return low[start:end] + +def select_greedy(unspent, value): + ''' + UTXO selection algorithm for rapid dust reduction + Combines the shortest run of utxos (sorted by size, from smallest) which + exceeds the target value; if the target value is larger than the sum of + all smaller utxos, uses the smallest utxo larger than the target value. + ''' + value, key = int(value), lambda u: u["value"] + high = sorted([u for u in unspent if key(u) >= value], key=key) + low = sorted([u for u in unspent if key(u) < value], key=key) + lowsum=reduce(lambda x,y:x+y,map(key,low),0) + if value > lowsum: + if len(high)==0: + raise Exception('Not enough funds') + else: + return [high[0]] + else: + end, total = 0, 0 + while total < value: + total += low[end]['value'] + end += 1 + return low[0:end] + +class AbstractWallet(object): + ''' + Abstract wallet for use with JoinMarket + Mostly written with Wallet in mind, the default JoinMarket HD wallet + ''' + def __init__(self): + self.utxo_selector = btc.select # default fallback: upstream + try: + if config.get("POLICY", "merge_algorithm") == "gradual": + self.utxo_selector = select_gradual + elif config.get("POLICY", "merge_algorithm") == "greedy": + self.utxo_selector = select_greedy + elif config.get("POLICY", "merge_algorithm") != "default": + raise Exception("Unknown merge algorithm") + except NoSectionError: + pass + + def get_key_from_addr(self, addr): + return None + def get_utxos_by_mixdepth(self): + return None + def get_change_addr(self, mixing_depth): + return None + def update_cache_index(self): + pass + def remove_old_utxos(self, tx): + pass + def add_new_utxos(self, tx, txid): + pass + + def select_utxos(self, mixdepth, amount): + utxo_list = self.get_utxos_by_mixdepth()[mixdepth] + unspent = [{'utxo': utxo, 'value': addrval['value']} + for utxo, addrval in utxo_list.iteritems()] + inputs = self.utxo_selector(unspent, amount) + debug('for mixdepth=' + str(mixdepth) + ' amount=' + str(amount) + ' selected:') + debug(pprint.pformat(inputs)) + return dict([(i['utxo'], {'value': i['value'], 'address': + utxo_list[i['utxo']]['address']}) for i in inputs]) + + def get_balance_by_mixdepth(self): + mix_balance = {} + for m in range(self.max_mix_depth): + mix_balance[m] = 0 + for mixdepth, utxos in self.get_utxos_by_mixdepth().iteritems(): + mix_balance[mixdepth] = sum([addrval['value'] for addrval in utxos.values()]) + return mix_balance + +class Wallet(AbstractWallet): + def __init__(self, seedarg, max_mix_depth, gaplimit=6, extend_mixdepth=False): + super(Wallet, self).__init__() + self.max_mix_depth = max_mix_depth + self.seed = self.get_seed(seedarg) + if extend_mixdepth and len(self.index_cache) > max_mix_depth: + self.max_mix_depth = len(self.index_cache) + self.gaplimit = gaplimit + master = btc.bip32_master_key(self.seed) + m_0 = btc.bip32_ckd(master, 0) + mixing_depth_keys = [btc.bip32_ckd(m_0, c) for c in range(self.max_mix_depth)] + self.keys = [(btc.bip32_ckd(m, 0), btc.bip32_ckd(m, 1)) for m in mixing_depth_keys] + + #self.index = [[0, 0]]*max_mix_depth + self.index = [] + for i in range(self.max_mix_depth): + self.index.append([0, 0]) + + #example + #index = self.index[mixing_depth] + #key = btc.bip32_ckd(self.keys[mixing_depth][index[0]], index[1]) + + self.addr_cache = {} + self.unspent = {} + self.spent_utxos = [] + + def get_seed(self, seedarg): + self.path = None + self.index_cache = [[0, 0]]*self.max_mix_depth + path = os.path.join('wallets', seedarg) + if not os.path.isfile(path): + if get_network() == 'testnet': + debug('seedarg interpreted as seed, only available in testnet because this probably has lower entropy') + return seedarg + else: + raise IOError('wallet file not found') + #debug('seedarg interpreted as wallet file name') + self.path = path + fd = open(path, 'r') + walletfile = fd.read() + fd.close() + walletdata = json.loads(walletfile) + if walletdata['network'] != get_network(): + print 'wallet network(%s) does not match joinmarket configured network(%s)' % ( + walletdata['network'], get_network()) + sys.exit(0) + if 'index_cache' in walletdata: + self.index_cache = walletdata['index_cache'] + decrypted = False + while not decrypted: + password = getpass.getpass('Enter wallet decryption passphrase: ') + password_key = btc.bin_dbl_sha256(password) + encrypted_seed = walletdata['encrypted_seed'] + try: + decrypted_seed = slowaes.decryptData(password_key, encrypted_seed + .decode('hex')).encode('hex') + #there is a small probability of getting a valid PKCS7 padding + #by chance from a wrong password; sanity check the seed length + if len(decrypted_seed) == 32: + decrypted = True + else: + raise ValueError + except ValueError: + print 'Incorrect password' + decrypted = False + return decrypted_seed + + def update_cache_index(self): + if not self.path: + return + if not os.path.isfile(self.path): + return + fd = open(self.path, 'r') + walletfile = fd.read() + fd.close() + walletdata = json.loads(walletfile) + walletdata['index_cache'] = self.index + walletfile = json.dumps(walletdata) + fd = open(self.path, 'w') + fd.write(walletfile) + fd.close() + + def get_key(self, mixing_depth, forchange, i): + return btc.bip32_extract_key(btc.bip32_ckd(self.keys[mixing_depth][forchange], i)) + + def get_addr(self, mixing_depth, forchange, i): + return btc.privtoaddr(self.get_key(mixing_depth, forchange, i), get_p2pk_vbyte()) + + def get_new_addr(self, mixing_depth, forchange): + index = self.index[mixing_depth] + addr = self.get_addr(mixing_depth, forchange, index[forchange]) + self.addr_cache[addr] = (mixing_depth, forchange, index[forchange]) + index[forchange] += 1 + #self.update_cache_index() + return addr + + def get_receive_addr(self, mixing_depth): + return self.get_new_addr(mixing_depth, False) + + def get_change_addr(self, mixing_depth): + return self.get_new_addr(mixing_depth, True) + + def get_key_from_addr(self, addr): + if addr in self.addr_cache: + return self.get_key(*self.addr_cache[addr]) + else: + return None + + def remove_old_utxos(self, tx): + removed_utxos = {} + for ins in tx['ins']: + utxo = ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index']) + if utxo not in self.unspent: + continue + removed_utxos[utxo] = self.unspent[utxo] + del self.unspent[utxo] + debug('removed utxos, wallet now is \n' + pprint.pformat(self.get_utxos_by_mixdepth())) + self.spent_utxos += removed_utxos.keys() + return removed_utxos + + def add_new_utxos(self, tx, txid): + added_utxos = {} + for index, outs in enumerate(tx['outs']): + addr = btc.script_to_address(outs['script'], get_p2pk_vbyte()) + if addr not in self.addr_cache: + continue + addrdict = {'address': addr, 'value': outs['value']} + utxo = txid + ':' + str(index) + added_utxos[utxo] = addrdict + self.unspent[utxo] = addrdict + debug('added utxos, wallet now is \n' + pprint.pformat(self.get_utxos_by_mixdepth())) + return added_utxos + + def get_utxos_by_mixdepth(self): + ''' + returns a list of utxos sorted by different mix levels + ''' + mix_utxo_list = {} + for m in range(self.max_mix_depth): + mix_utxo_list[m] = {} + for utxo, addrvalue in self.unspent.iteritems(): + mixdepth = self.addr_cache[addrvalue['address']][0] + if mixdepth not in mix_utxo_list: + mix_utxo_list[mixdepth] = {} + mix_utxo_list[mixdepth][utxo] = addrvalue + debug('get_utxos_by_mixdepth = \n' + pprint.pformat(mix_utxo_list)) + return mix_utxo_list + +class BitcoinCoreWallet(AbstractWallet): + def __init__(self, fromaccount): + super(BitcoinCoreWallet, self).__init__() + if not isinstance(bc_interface, blockchaininterface.BitcoinCoreInterface): + raise RuntimeError('Bitcoin Core wallet can only be used when blockchain interface is BitcoinCoreInterface') + self.fromaccount = fromaccount + self.max_mix_depth = 1 + + def get_key_from_addr(self, addr): + return bc_interface.rpc('dumpprivkey', [addr]) + + def get_utxos_by_mixdepth(self): + unspent_list = bc_interface.rpc('listunspent', []) + result = {0: {}} + for u in unspent_list: + if not u['spendable']: + continue + if self.fromaccount and (('account' not in u) or u['account'] != self.fromaccount): + continue + result[0][u['txid'] + ':' + str(u['vout'])] = {'address': u['address'], + 'value': int(Decimal(str(u['amount'])) * Decimal('1e8'))} + return result + + def get_change_addr(self, mixing_depth): + return bc_interface.rpc('getrawchangeaddress', []) + +def calc_cj_fee(ordertype, cjfee, cj_amount): + real_cjfee = None + if ordertype == 'absorder': + real_cjfee = int(cjfee) + elif ordertype == 'relorder': + real_cjfee = int((Decimal(cjfee) * Decimal(cj_amount)).quantize(Decimal(1))) + else: + raise RuntimeError('unknown order type: ' + str(ordertype)) + return real_cjfee + +def weighted_order_choose(orders, n, feekey): + ''' + Algorithm for choosing the weighting function + it is an exponential + P(f) = exp(-(f - fmin) / phi) + P(f) - probability of order being chosen + f - order fee + fmin - minimum fee in the order book + phi - scaling parameter, 63% of the distribution is within + + define number M, related to the number of counterparties in this coinjoin + phi has a value such that it contains up to the Mth order + unless M < orderbook size, then phi goes up to the last order + ''' + minfee = feekey(orders[0]) + M = int(3*n) + if len(orders) > M: + phi = feekey(orders[M]) - minfee + else: + phi = feekey(orders[-1]) - minfee + fee = [feekey(o) for o in orders] + if phi > 0: + weight = [exp(-(1.0*f - minfee) / phi) for f in fee] + else: + weight = [1.0]*len(fee) + weight = [x/sum(weight) for x in weight] + debug('phi=' + str(phi) + ' weights = ' + str(weight)) + chosen_order_index = rand_weighted_choice(len(orders), weight) + return orders[chosen_order_index] + +def cheapest_order_choose(orders, n, feekey): + ''' + Return the cheapest order from the orders. + ''' + return sorted(orders, key=feekey)[0] + +def pick_order(orders, n, feekey): + i = -1 + print("Considered orders:"); + for o in orders: + i+=1 + print(" %2d. %20s, CJ fee: %6d, tx fee: %6d" % (i, o[0], o[2], o[3])) + pickedOrderIndex = -1 + if i==0: + print("Only one possible pick, picking it.") + return orders[0] + while pickedOrderIndex == -1: + try: + pickedOrderIndex = int(raw_input('Pick an order between 0 and '+str(i)+': ')) + except ValueError: + pickedOrderIndex = -1 + continue; + + if pickedOrderIndex>=0 and pickedOrderIndex= o['minsize'] and cj_amount <= o['maxsize'] and o['counterparty'] + not in ignored_makers] + counterparties = set([o[0] for o in orders]) + if n > len(counterparties): + debug('ERROR not enough liquidity in the orderbook n=%d suitable-counterparties=%d amount=%d totalorders=%d' + % (n, len(counterparties), cj_amount, len(orders))) + return None, 0 #TODO handle not enough liquidity better, maybe an Exception + orders = sorted(orders, key=lambda k: k[2]) #sort from smallest to biggest cj fee + debug('considered orders = \n' + '\n'.join([str(o) for o in orders])) + total_cj_fee = 0 + chosen_orders = [] + for i in range(n): + chosen_order = chooseOrdersBy(orders, n, lambda k: k[2]) + orders = [o for o in orders if o[0] != chosen_order[0]] #remove all orders from that same counterparty + chosen_orders.append(chosen_order) + total_cj_fee += chosen_order[2] + debug('chosen orders = \n' + '\n'.join([str(o) for o in chosen_orders])) + chosen_orders = [o[:2] for o in chosen_orders] + return dict(chosen_orders), total_cj_fee + +def choose_sweep_orders(db, total_input_value, my_tx_fee, n, chooseOrdersBy, ignored_makers=[]): + ''' + choose an order given that we want to be left with no change + i.e. sweep an entire group of utxos + + solve for cjamount when mychange = 0 + for an order with many makers, a mixture of absorder and relorder + mychange = totalin - cjamount - mytxfee - sum(absfee) - sum(relfee*cjamount) + => 0 = totalin - mytxfee - sum(absfee) - cjamount*(1 + sum(relfee)) + => cjamount = (totalin - mytxfee - sum(absfee)) / (1 + sum(relfee)) + ''' + def calc_zero_change_cj_amount(ordercombo): + sumabsfee = 0 + sumrelfee = Decimal('0') + for order in ordercombo: + if order[0]['ordertype'] == 'absorder': + sumabsfee += int(order[0]['cjfee']) + elif order[0]['ordertype'] == 'relorder': + sumrelfee += Decimal(order[0]['cjfee']) + else: + raise RuntimeError('unknown order type: ' + str(order[0]['ordertype'])) + cjamount = (total_input_value - my_tx_fee - sumabsfee) / (1 + sumrelfee) + cjamount = int(cjamount.quantize(Decimal(1))) + return cjamount, int(sumabsfee + sumrelfee*cjamount) + + sqlorders = db.execute('SELECT * FROM orderbook WHERE minsize <= ?;', (total_input_value,)).fetchall() + orderkeys = ['counterparty', 'oid', 'ordertype', 'minsize', 'maxsize', 'txfee', 'cjfee'] + orderlist = [dict([(k, o[k]) for k in orderkeys]) for o in sqlorders + if o['counterparty'] not in ignored_makers] + #orderlist = sqlorders #uncomment this and comment previous two lines for faster runtime but less readable output + debug('orderlist = \n' + '\n'.join([str(o) for o in orderlist])) + + #choose N amount of orders + available_orders = [(o, calc_cj_fee(o['ordertype'], o['cjfee'], total_input_value)) + for o in orderlist] + available_orders = sorted(available_orders, key=lambda k: k[1]) #sort from smallest to biggest cj fee + chosen_orders = [] + while len(chosen_orders) < n: + if len(available_orders) < n - len(chosen_orders): + debug('ERROR not enough liquidity in the orderbook') + return None, 0 #TODO handle not enough liquidity better, maybe an Exception + for i in range(n - len(chosen_orders)): + chosen_order = chooseOrdersBy(available_orders, n, lambda k: k[1]) + debug('chosen = ' + str(chosen_order)) + #remove all orders from that same counterparty + available_orders = [o for o in available_orders if o[0]['counterparty'] != chosen_order[0]['counterparty']] + chosen_orders.append(chosen_order) + #calc cj_amount and check its in range + cj_amount, total_fee = calc_zero_change_cj_amount(chosen_orders) + for c in list(chosen_orders): + minsize = c[0]['minsize'] + maxsize = c[0]['maxsize'] + if cj_amount > maxsize or cj_amount < minsize: + chosen_orders.remove(c) + debug('chosen orders = \n' + '\n'.join([str(o) for o in chosen_orders])) + result = dict([(o[0]['counterparty'], o[0]['oid']) for o in chosen_orders]) + debug('cj amount = ' + str(cj_amount)) + return result, cj_amount + diff --git a/lib/enc_wrapper.py b/lib/enc_wrapper.py new file mode 100644 index 00000000..d251a25e --- /dev/null +++ b/lib/enc_wrapper.py @@ -0,0 +1,123 @@ +#A wrapper for public key +#authenticated encryption +#using Diffie Hellman key +#exchange to set up a +#symmetric encryption. + +import libnacl.public +import binascii, base64 + +def init_keypair(fname = None): + '''Create a new encryption + keypair; stored in file fname + if provided. The keypair object + is returned. + ''' + kp = libnacl.public.SecretKey() + if fname: + #Note: handles correct file permissions + kp.save(fname) + return kp + +#the next two functions are useful +#for exchaging pubkeys with counterparty +def get_pubkey(kp, as_hex=False): + '''Given a keypair object, + return its public key, + optionally in hex.''' + return kp.hex_pk() if as_hex else kp.pk + +def init_pubkey(hexpk, fname = None): + '''Create a pubkey object from a + hex formatted string. + Save to file fname if specified. + ''' + pk = libnacl.public.PublicKey(binascii.unhexlify(hexpk)) + if fname: + pk.save(fname) + return pk + +def as_init_encryption(kp, c_pk): + '''Given an initialised + keypair kp and a counterparty + pubkey c_pk, create a Box + ready for encryption/decryption. + ''' + return libnacl.public.Box(kp.sk, c_pk) +''' +After initialisation, it's possible +to use the box object returned from +as_init_encryption to directly change +from plaintext to ciphertext: + ciphertext = box.encrypt(plaintext) + plaintext = box.decrypt(ciphertext) +Notes: + 1. use binary format for ctext/ptext + 2. Nonce is handled at the implementation layer. +''' + +#TODO: Sign, verify. At the moment we are using +#bitcoin signatures so it isn't necessary. + +#encoding for passing over the wire +def encrypt_encode(msg, box): + encrypted = box.encrypt(msg) + return base64.b64encode(encrypted) + +def decode_decrypt(msg, box): + decoded = base64.b64decode(msg) + return box.decrypt(decoded) + +def test_case(case_name, alice_box, bob_box, ab_message, ba_message, num_iterations=1): + for i in range(num_iterations): + ab_message = ''.join(random.choice(string.ascii_letters) for x in range(100)) if ab_message == 'rand' else ab_message + ba_message = ''.join(random.choice(string.ascii_letters) for x in range(100)) if ba_message == 'rand' else ba_message + otw_amsg = alice_box.encrypt(ab_message) + bob_ptext = bob_box.decrypt(otw_amsg) + assert bob_ptext == ab_message, "Encryption test: FAILED. Alice sent: "\ + +ab_message+" , Bob received: " + bob_ptext + + otw_bmsg = bob_box.encrypt(ba_message) + alice_ptext = alice_box.decrypt(otw_bmsg) + assert alice_ptext == ba_message, "Encryption test: FAILED. Bob sent: "\ + +ba_message+" , Alice received: " + alice_ptext + + print "Encryption test PASSED for case: "+case_name + +def test_keypair_setup(): + alice_kp = init_keypair() + bob_kp = init_keypair() + + #this is the DH key exchange part + bob_otwpk = get_pubkey(bob_kp, True) + alice_otwpk = get_pubkey(alice_kp, True) + + bob_pk = init_pubkey(bob_otwpk) + alice_box = as_init_encryption(alice_kp, bob_pk) + alice_pk = init_pubkey(alice_otwpk) + bob_box = as_init_encryption(bob_kp, alice_pk) + + #now Alice and Bob can use their 'box' + #constructs (both of which utilise the same + #shared secret) to perform encryption/decryption + #to test the encryption functionality + return (alice_box, bob_box) + +if __name__ == "__main__": + + alice_box, bob_box = test_keypair_setup() + test_case("short ascii", alice_box, bob_box,"Attack at dawn","Not tonight Josephine!",5) + + import base64, string, random + alice_box, bob_box = test_keypair_setup() + longb641 = base64.b64encode(''.join(random.choice(string.ascii_letters) for x in range(5000))) + longb642 = base64.b64encode(''.join(random.choice(string.ascii_letters) for x in range(5000))) + test_case("long b64", alice_box, bob_box, longb641, longb642,5) + #test a large number of messages on the same connection + alice_box, bob_box = test_keypair_setup() + test_case("endless_wittering", alice_box, bob_box,'rand','rand',40000) + #edge cases + #1 character + alice_box, bob_box = test_keypair_setup() + test_case("1 char",alice_box, bob_box,'\x00','\x00',5) + print "All test cases passed - encryption and decryption should work correctly." diff --git a/lib/irc.py b/lib/irc.py new file mode 100644 index 00000000..e121b65f --- /dev/null +++ b/lib/irc.py @@ -0,0 +1,528 @@ +# +from common import * +from message_channel import MessageChannel +from message_channel import CJPeerError + +import string, random +import socket, threading, time, ssl, socks +import base64, os, re +import enc_wrapper + +MAX_PRIVMSG_LEN = 400 +COMMAND_PREFIX = '!' +PING_INTERVAL = 180 +PING_TIMEOUT = 30 +encrypted_commands = ["auth", "ioauth", "tx", "sig"] +plaintext_commands = ["fill", "error", "pubkey", "orderbook", "relorder", "absorder", "push"] + +def random_nick(nick_len=9): + vowels = "aeiou" + consonants = ''.join([chr(c) for c in range(ord('a'), ord('z')+1) if vowels.find(chr(c)) == -1]) + assert nick_len % 2 == 1 + N = (nick_len - 1) / 2 + rnd_consonants = [consonants[random.randrange(len(consonants))] for c in range(N+1)] + rnd_vowels = [vowels[random.randrange(len(vowels))] for v in range(N)] + [''] + ircnick = ''.join([i for sl in zip(rnd_consonants, rnd_vowels) for i in sl]) + ircnick = ircnick.capitalize() + print 'Generated random nickname: ' + ircnick #not using debug because it might not know the logfile name at this point + return ircnick + #Other ideas for random nickname generation: + # - weight randomness by frequency of letter appearance + # - u always follows q + # - generate different length nicks + # - append two or more of these words together + # - randomly combine phonetic sounds instead consonants, which may be two consecutive consonants + # - e.g. th, dj, g, p, gr, ch, sh, kr, + # - neutral network that generates nicks + +def get_irc_text(line): + return line[line[1:].find(':') + 2:] + +def get_irc_nick(source): + return source[1:source.find('!')] + +class PingThread(threading.Thread): + def __init__(self, irc): + threading.Thread.__init__(self) + self.daemon = True + self.irc = irc + def run(self): + debug('starting ping thread') + while not self.irc.give_up: + time.sleep(PING_INTERVAL) + try: + self.irc.ping_reply = False + #maybe use this to calculate the lag one day + self.irc.lockcond.acquire() + self.irc.send_raw('PING LAG' + str(int(time.time() * 1000))) + self.irc.lockcond.wait(PING_TIMEOUT) + self.irc.lockcond.release() + if not self.irc.ping_reply: + debug('irc ping timed out') + try: self.irc.close() + except: pass + try: self.irc.fd.close() + except: pass + try: + self.irc.sock.shutdown(socket.SHUT_RDWR) + self.irc.sock.close() + except: pass + except IOError as e: + debug('ping thread: ' + repr(e)) + debug('ended ping thread') + +#handle one channel at a time +class IRCMessageChannel(MessageChannel): + + #close implies it will attempt to reconnect + def close(self): + try: + self.send_raw("QUIT") + except IOError as e: + debug('errored while trying to quit: ' + repr(e)) + + def shutdown(self): + self.close() + self.give_up = True + + def send_error(self, nick, errormsg): + debug('error<%s> : %s' % (nick, errormsg)) + self.__privmsg(nick, 'error', errormsg) + raise CJPeerError() + + #OrderbookWatch callback + def request_orderbook(self): + self.__pubmsg(COMMAND_PREFIX + 'orderbook') + + #Taker callbacks + def fill_orders(self, nickoid_dict, cj_amount, taker_pubkey): + for c, oid in nickoid_dict.iteritems(): + msg = str(oid) + ' ' + str(cj_amount) + ' ' + taker_pubkey + self.__privmsg(c, 'fill', msg) + + def send_auth(self, nick, pubkey, sig): + message = pubkey + ' ' + sig + self.__privmsg(nick, 'auth', message) + + def send_tx(self, nick_list, txhex): + txb64 = base64.b64encode(txhex.decode('hex')) + for nick in nick_list: + self.__privmsg(nick, 'tx', txb64) + time.sleep(1) #HACK! really there should be rate limiting, see issue#31 + + def push_tx(self, nick, txhex): + txb64 = base64.b64encode(txhex.decode('hex')) + self.__privmsg(nick, 'push', txb64) + + #Maker callbacks + def announce_orders(self, orderlist, nick=None): + #nick=None means announce publicly + order_keys = ['oid', 'minsize', 'maxsize', 'txfee', 'cjfee'] + header = 'PRIVMSG ' + (nick if nick else self.channel) + ' :' + orderlines = [] + for i, order in enumerate(orderlist): + orderparams = COMMAND_PREFIX + order['ordertype'] +\ + ' ' + ' '.join([str(order[k]) for k in order_keys]) + orderlines.append(orderparams) + line = header + ''.join(orderlines) + ' ~' + if len(line) > MAX_PRIVMSG_LEN or i == len(orderlist)-1: + if i < len(orderlist)-1: + line = header + ''.join(orderlines[:-1]) + ' ~' + self.send_raw(line) + orderlines = [orderlines[-1]] + + def cancel_orders(self, oid_list): + clines = [COMMAND_PREFIX + 'cancel ' + str(oid) for oid in oid_list] + self.__pubmsg(''.join(clines)) + + def send_pubkey(self, nick, pubkey): + self.__privmsg(nick, 'pubkey', pubkey) + + def send_ioauth(self, nick, utxo_list, cj_pubkey, change_addr, sig): + authmsg = (str(','.join(utxo_list)) + ' ' + + cj_pubkey + ' ' + change_addr + ' ' + sig) + self.__privmsg(nick, 'ioauth', authmsg) + + def send_sigs(self, nick, sig_list): + #TODO make it send the sigs on one line if there's space + for s in sig_list: + self.__privmsg(nick, 'sig', s) + time.sleep(0.5) #HACK! really there should be rate limiting, see issue#31 + + def __pubmsg(self, message): + debug('>>pubmsg ' + message) + self.send_raw("PRIVMSG " + self.channel + " :" + message) + + def __privmsg(self, nick, cmd, message): + debug('>>privmsg ' + 'nick=' + nick + ' cmd=' + cmd + ' msg=' + message) + #should we encrypt? + box, encrypt = self.__get_encryption_box(cmd, nick) + #encrypt before chunking + if encrypt: + if not box: + debug('error, dont have encryption box object for ' + nick + ', dropping message') + return + message = enc_wrapper.encrypt_encode(message, box) + + header = "PRIVMSG " + nick + " :" + max_chunk_len = MAX_PRIVMSG_LEN - len(header) - len(cmd) - 4 + #1 for command prefix 1 for space 2 for trailer + if len(message) > max_chunk_len: + message_chunks = chunks(message, max_chunk_len) + else: + message_chunks = [message] + for m in message_chunks: + trailer = ' ~' if m==message_chunks[-1] else ' ;' + if m==message_chunks[0]: + m = COMMAND_PREFIX + cmd + ' ' + m + self.send_raw(header + m + trailer) + + def send_raw(self, line): + #if not line.startswith('PING LAG'): + # debug('sendraw ' + line) + self.sock.sendall(line + '\r\n') + + def check_for_orders(self, nick, chunks): + if chunks[0] in ordername_list: + try: + counterparty = nick + oid = chunks[1] + ordertype = chunks[0] + minsize = chunks[2] + maxsize = chunks[3] + txfee = chunks[4] + cjfee = chunks[5] + if self.on_order_seen: + self.on_order_seen(counterparty, oid, + ordertype, minsize, maxsize, txfee, cjfee) + except IndexError as e: + debug('index error parsing chunks') + #TODO what now? just ignore iirc + finally: + return True + return False + + def __on_privmsg(self, nick, message): + '''handles the case when a private message is received''' + if message[0] != COMMAND_PREFIX: + return + for command in message[1:].split(COMMAND_PREFIX): + chunks = command.split(" ") + #looks like a very similar pattern for all of these + # check for a command name, parse arguments, call a function + # maybe we need some eval() trickery to do it better + + try: + #orderbook watch commands + if self.check_for_orders(nick, chunks): + pass + + #taker commands + elif chunks[0] == 'pubkey': + maker_pk = chunks[1] + if self.on_pubkey: + self.on_pubkey(nick, maker_pk) + elif chunks[0] == 'ioauth': + utxo_list = chunks[1].split(',') + cj_pub = chunks[2] + change_addr = chunks[3] + btc_sig = chunks[4] + if self.on_ioauth: + self.on_ioauth(nick, utxo_list, cj_pub, change_addr, btc_sig) + elif chunks[0] == 'sig': + sig = chunks[1] + if self.on_sig: + self.on_sig(nick, sig) + + #maker commands + if chunks[0] == 'fill': + try: + oid = int(chunks[1]) + amount = int(chunks[2]) + taker_pk = chunks[3] + except (ValueError, IndexError) as e: + self.send_error(nick, str(e)) + if self.on_order_fill: + self.on_order_fill(nick, oid, amount, taker_pk) + elif chunks[0] == 'auth': + try: + i_utxo_pubkey = chunks[1] + btc_sig = chunks[2] + except (ValueError, IndexError) as e: + self.send_error(nick, str(e)) + if self.on_seen_auth: + self.on_seen_auth(nick, i_utxo_pubkey, btc_sig) + elif chunks[0] == 'tx': + b64tx = chunks[1] + try: + txhex = base64.b64decode(b64tx).encode('hex') + except TypeError as e: + self.send_error(nick, 'bad base64 tx. ' + repr(e)) + if self.on_seen_tx: + self.on_seen_tx(nick, txhex) + elif chunks[0] == 'push': + b64tx = chunks[1] + try: + txhex = base64.b64decode(b64tx).encode('hex') + except TypeError as e: + self.send_error(nick, 'bad base64 tx. ' + repr(e)) + if self.on_push_tx: + self.on_push_tx(nick, txhex) + except CJPeerError: + #TODO proper error handling + debug('cj peer error TODO handle') + continue + + def __on_pubmsg(self, nick, message): + if message[0] != COMMAND_PREFIX: + return + for command in message[1:].split(COMMAND_PREFIX): + chunks = command.split(" ") + if self.check_for_orders(nick, chunks): + pass + elif chunks[0] == 'cancel': + #!cancel [oid] + try: + oid = int(chunks[1]) + if self.on_order_cancel: + self.on_order_cancel(nick, oid) + except ValueError as e: + debug("!cancel " + repr(e)) + return + elif chunks[0] == 'orderbook': + if self.on_orderbook_requested: + self.on_orderbook_requested(nick) + else: + #TODO this is for testing/debugging, should be removed, see taker.py + if hasattr(self, 'debug_on_pubmsg_cmd'): + self.debug_on_pubmsg_cmd(nick, chunks) + + def __get_encryption_box(self, cmd, nick): + '''Establish whether the message is to be + encrypted/decrypted based on the command string. + If so, retrieve the appropriate crypto_box object + and return. Sending/receiving flag enables us + to check which command strings correspond to which + type of object (maker/taker).''' #old doc, dont trust + if cmd in plaintext_commands: + return None, False + else: + return self.cjpeer.get_crypto_box_from_nick(nick), True + + def __handle_privmsg(self, source, target, message): + nick = get_irc_nick(source) + if target == self.nick: + if message[0] == '\x01': + endindex = message[1:].find('\x01') + if endindex == -1: + return + ctcp = message[1:endindex + 1] + if ctcp.upper() == 'VERSION': + self.send_raw('PRIVMSG ' + nick + ' :\x01VERSION xchat 2.8.8 Ubuntu\x01') + return + + if nick not in self.built_privmsg: + if message[0] != COMMAND_PREFIX: + debug('message not a cmd') + return + #new message starting + cmd_string = message[1:].split(' ')[0] + if cmd_string not in plaintext_commands + encrypted_commands: + debug('cmd not in cmd_list, line="' + message + '"') + return + self.built_privmsg[nick] = [cmd_string, message[:-2]] + else: + self.built_privmsg[nick][1] += message[:-2] + box, encrypt = self.__get_encryption_box(self.built_privmsg[nick][0], nick) + if message[-1]==';': + self.waiting[nick]=True + elif message[-1]=='~': + self.waiting[nick]=False + if encrypt: + if not box: + debug('error, dont have encryption box object for ' + nick + ', dropping message') + return + #need to decrypt everything after the command string + to_decrypt = ''.join(self.built_privmsg[nick][1].split(' ')[1]) + try: + decrypted = enc_wrapper.decode_decrypt(to_decrypt, box) + except ValueError as e: + debug('valueerror when decrypting, skipping: ' + repr(e)) + return + parsed = self.built_privmsg[nick][1].split(' ')[0] + ' ' + decrypted + else: + parsed = self.built_privmsg[nick][1] + #wipe the message buffer waiting for the next one + del self.built_privmsg[nick] + debug("< MAX_PRIVMSG_LEN: + irc.privmsg(target, prefix + line) + line = '' + if len(line) > 0: + irc.privmsg(target, prefix + line) + diff --git a/lib/jsonrpc.py b/lib/jsonrpc.py new file mode 100644 index 00000000..54e69af4 --- /dev/null +++ b/lib/jsonrpc.py @@ -0,0 +1,115 @@ +# Copyright (C) 2013,2015 by Daniel Kraft +# Copyright (C) 2014 by phelix / blockchained.com +# +# Permission is hereby granted, free of charge, to any person obtaining a copy +# of this software and associated documentation files (the "Software"), to deal +# in the Software without restriction, including without limitation the rights +# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +# copies of the Software, and to permit persons to whom the Software is +# furnished to do so, subject to the following conditions: +# +# The above copyright notice and this permission notice shall be included in all +# copies or substantial portions of the Software. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +# SOFTWARE. + +import base64 +import httplib +import json + +class JsonRpcError (Exception): + """ + The called method returned an error in the JSON-RPC response. + """ + + def __init__ (self, obj): + self.code = obj["code"] + self.message = obj["message"] + +class JsonRpcConnectionError (Exception): + """ + Error thrown when the RPC connection itself failed. This means + that the server is either down or the connection settings + are wrong. + """ + + pass + +class JsonRpc (object): + """ + Simple implementation of a JSON-RPC client that is used + to connect to Bitcoin. + """ + + def __init__ (self, host, port, user, password): + self.host = host + self.port = port + self.authstr = "%s:%s" % (user, password) + + self.queryId = 1 + + def queryHTTP (self, obj): + """ + Send an appropriate HTTP query to the server. The JSON-RPC + request should be (as object) in 'obj'. If the call succeeds, + the resulting JSON object is returned. In case of an error + with the connection (not JSON-RPC itself), an exception is raised. + """ + + headers = {} + headers["User-Agent"] = "joinmarket" + headers["Content-Type"] = "application/json" + headers["Accept"] = "application/json" + headers["Authorization"] = "Basic %s" % base64.b64encode (self.authstr) + + body = json.dumps (obj) + + try: + conn = httplib.HTTPConnection (self.host, self.port) + conn.request ("POST", "", body, headers) + response = conn.getresponse () + + if response.status == 401: + conn.close () + raise JsonRpcConnectionError ("authentication for JSON-RPC failed") + + # All of the codes below are 'fine' from a JSON-RPC point of view. + if response.status not in [200, 404, 500]: + conn.close () + raise JsonRpcConnectionError ("unknown error in JSON-RPC") + + data = response.read () + conn.close () + + return json.loads (data) + + except JsonRpcConnectionError as exc: + raise exc + except Exception as exc: + raise JsonRpcConnectionError ("JSON-RPC connection failed. Err:" + repr(exc)) + + def call (self, method, params): + """ + Call a method over JSON-RPC. + """ + + currentId = self.queryId + self.queryId += 1 + + request = {"method": method, "params": params, "id": currentId} + response = self.queryHTTP (request) + + if response["id"] != currentId: + raise JsonRpcConnectionError ("invalid id returned by query") + + if response["error"] is not None: + print response["error"] + raise JsonRpcError (response["error"]) + + return response["result"] diff --git a/lib/libnacl/__init__.py b/lib/libnacl/__init__.py new file mode 100644 index 00000000..df266a2f --- /dev/null +++ b/lib/libnacl/__init__.py @@ -0,0 +1,582 @@ +# -*- coding: utf-8 -*- +''' +Wrap libsodium routines +''' +# pylint: disable=C0103 +# Import libnacl libs +from libnacl.version import __version__ +# Import python libs +import ctypes +import sys + +__SONAMES = (13, 10, 5, 4) + + +def _get_nacl(): + ''' + Locate the nacl c libs to use + ''' + # Import libsodium + if sys.platform.startswith('win'): + try: + return ctypes.cdll.LoadLibrary('libsodium') + except OSError: + pass + for soname_ver in __SONAMES: + try: + return ctypes.cdll.LoadLibrary( + 'libsodium-{0}'.format(soname_ver) + ) + except OSError: + pass + try: + return ctypes.cdll.LoadLibrary('tweetnacl') + except OSError: + msg = ('Could not locate nacl lib, searched for libsodium, ' + 'tweetnacl') + raise OSError(msg) + elif sys.platform.startswith('darwin'): + try: + return ctypes.cdll.LoadLibrary('libsodium.dylib') + except OSError: + pass + try: + return ctypes.cdll.LoadLibrary('tweetnacl.dylib') + except OSError: + msg = ('Could not locate nacl lib, searched for libsodium, ' + 'tweetnacl') + raise OSError(msg) + else: + try: + return ctypes.cdll.LoadLibrary('libsodium.so') + except OSError: + pass + try: + return ctypes.cdll.LoadLibrary('/usr/local/lib/libsodium.so') + except OSError: + pass + + for soname_ver in __SONAMES: + try: + return ctypes.cdll.LoadLibrary( + 'libsodium.so.{0}'.format(soname_ver) + ) + except OSError: + pass + try: + return ctypes.cdll.LoadLibrary('tweetnacl.so') + except OSError: + msg = 'Could not locate nacl lib, searched for libsodium.so, ' + for soname_ver in __SONAMES: + msg += 'libsodium.so.{0}, '.format(soname_ver) + msg += ' and tweetnacl.so' + raise OSError(msg) + +nacl = _get_nacl() + +# Define constants +crypto_box_SECRETKEYBYTES = nacl.crypto_box_secretkeybytes() +crypto_box_PUBLICKEYBYTES = nacl.crypto_box_publickeybytes() +crypto_box_NONCEBYTES = nacl.crypto_box_noncebytes() +crypto_box_ZEROBYTES = nacl.crypto_box_zerobytes() +crypto_box_BOXZEROBYTES = nacl.crypto_box_boxzerobytes() +crypto_box_BEFORENMBYTES = nacl.crypto_box_beforenmbytes() +crypto_scalarmult_BYTES = nacl.crypto_scalarmult_bytes() +crypto_scalarmult_SCALARBYTES = nacl.crypto_scalarmult_scalarbytes() +crypto_sign_BYTES = nacl.crypto_sign_bytes() +crypto_sign_SEEDBYTES = nacl.crypto_sign_secretkeybytes() // 2 +crypto_sign_PUBLICKEYBYTES = nacl.crypto_sign_publickeybytes() +crypto_sign_SECRETKEYBYTES = nacl.crypto_sign_secretkeybytes() +crypto_box_MACBYTES = crypto_box_ZEROBYTES - crypto_box_BOXZEROBYTES +crypto_secretbox_KEYBYTES = nacl.crypto_secretbox_keybytes() +crypto_secretbox_NONCEBYTES = nacl.crypto_secretbox_noncebytes() +crypto_secretbox_ZEROBYTES = nacl.crypto_secretbox_zerobytes() +crypto_secretbox_BOXZEROBYTES = nacl.crypto_secretbox_boxzerobytes() +crypto_secretbox_MACBYTES = crypto_secretbox_ZEROBYTES - crypto_secretbox_BOXZEROBYTES +crypto_stream_KEYBYTES = nacl.crypto_stream_keybytes() +crypto_stream_NONCEBYTES = nacl.crypto_stream_noncebytes() +crypto_auth_BYTES = nacl.crypto_auth_bytes() +crypto_auth_KEYBYTES = nacl.crypto_auth_keybytes() +crypto_onetimeauth_BYTES = nacl.crypto_onetimeauth_bytes() +crypto_onetimeauth_KEYBYTES = nacl.crypto_onetimeauth_keybytes() +crypto_generichash_BYTES = nacl.crypto_generichash_bytes() +crypto_generichash_BYTES_MIN = nacl.crypto_generichash_bytes_min() +crypto_generichash_BYTES_MAX = nacl.crypto_generichash_bytes_max() +crypto_generichash_KEYBYTES = nacl.crypto_generichash_keybytes() +crypto_generichash_KEYBYTES_MIN = nacl.crypto_generichash_keybytes_min() +crypto_generichash_KEYBYTES_MAX = nacl.crypto_generichash_keybytes_max() +crypto_scalarmult_curve25519_BYTES = nacl.crypto_scalarmult_curve25519_bytes() +crypto_hash_BYTES = nacl.crypto_hash_sha512_bytes() +crypto_hash_sha256_BYTES = nacl.crypto_hash_sha256_bytes() +crypto_hash_sha512_BYTES = nacl.crypto_hash_sha512_bytes() +# pylint: enable=C0103 + + +# Define exceptions +class CryptError(Exception): + ''' + Base Exception for cryptographic errors + ''' + +# Pubkey defs + + +def crypto_box_keypair(): + ''' + Generate and return a new keypair + + pk, sk = nacl.crypto_box_keypair() + ''' + pk = ctypes.create_string_buffer(crypto_box_PUBLICKEYBYTES) + sk = ctypes.create_string_buffer(crypto_box_SECRETKEYBYTES) + nacl.crypto_box_keypair(pk, sk) + return pk.raw, sk.raw + + +def crypto_box(msg, nonce, pk, sk): + ''' + Using a public key and a secret key encrypt the given message. A nonce + must also be passed in, never reuse the nonce + + enc_msg = nacl.crypto_box('secret message', , , ) + ''' + if len(pk) != crypto_box_PUBLICKEYBYTES: + raise ValueError('Invalid public key') + if len(sk) != crypto_box_SECRETKEYBYTES: + raise ValueError('Invalid secret key') + if len(nonce) != crypto_box_NONCEBYTES: + raise ValueError('Invalid nonce') + pad = b'\x00' * crypto_box_ZEROBYTES + msg + c = ctypes.create_string_buffer(len(pad)) + ret = nacl.crypto_box(c, pad, ctypes.c_ulonglong(len(pad)), nonce, pk, sk) + if ret: + raise CryptError('Unable to encrypt message') + return c.raw[crypto_box_BOXZEROBYTES:] + + +def crypto_box_open(ctxt, nonce, pk, sk): + ''' + Decrypts a message given the receivers private key, and senders public key + ''' + if len(pk) != crypto_box_PUBLICKEYBYTES: + raise ValueError('Invalid public key') + if len(sk) != crypto_box_SECRETKEYBYTES: + raise ValueError('Invalid secret key') + if len(nonce) != crypto_box_NONCEBYTES: + raise ValueError('Invalid nonce') + pad = b'\x00' * crypto_box_BOXZEROBYTES + ctxt + msg = ctypes.create_string_buffer(len(pad)) + ret = nacl.crypto_box_open( + msg, + pad, + ctypes.c_ulonglong(len(pad)), + nonce, + pk, + sk) + if ret: + raise CryptError('Unable to decrypt ciphertext') + return msg.raw[crypto_box_ZEROBYTES:] + + +def crypto_box_beforenm(pk, sk): + ''' + Partially performs the computation required for both encryption and decryption of data + ''' + if len(pk) != crypto_box_PUBLICKEYBYTES: + raise ValueError('Invalid public key') + if len(sk) != crypto_box_SECRETKEYBYTES: + raise ValueError('Invalid secret key') + k = ctypes.create_string_buffer(crypto_box_BEFORENMBYTES) + ret = nacl.crypto_box_beforenm(k, pk, sk) + if ret: + raise CryptError('Unable to compute shared key') + return k.raw + + +def crypto_box_afternm(msg, nonce, k): + ''' + Encrypts a given a message, using partial computed data + ''' + if len(k) != crypto_box_BEFORENMBYTES: + raise ValueError('Invalid shared key') + if len(nonce) != crypto_box_NONCEBYTES: + raise ValueError('Invalid nonce') + pad = b'\x00' * crypto_box_ZEROBYTES + msg + ctxt = ctypes.create_string_buffer(len(pad)) + ret = nacl.crypto_box_afternm(ctxt, pad, ctypes.c_ulonglong(len(pad)), nonce, k) + if ret: + raise ValueError('Unable to encrypt messsage') + return ctxt.raw[crypto_box_BOXZEROBYTES:] + + +def crypto_box_open_afternm(ctxt, nonce, k): + ''' + Decrypts a ciphertext ctxt given k + ''' + if len(k) != crypto_box_BEFORENMBYTES: + raise ValueError('Invalid shared key') + if len(nonce) != crypto_box_NONCEBYTES: + raise ValueError('Invalid nonce') + pad = b'\x00' * crypto_box_BOXZEROBYTES + ctxt + msg = ctypes.create_string_buffer(len(pad)) + ret = nacl.crypto_box_open_afternm( + msg, + pad, + ctypes.c_ulonglong(len(pad)), + nonce, + k) + if ret: + raise ValueError('unable to decrypt message') + return msg.raw[crypto_box_ZEROBYTES:] + +# Signing functions + + +def crypto_sign_keypair(): + ''' + Generates a signing/verification key pair + ''' + vk = ctypes.create_string_buffer(crypto_sign_PUBLICKEYBYTES) + sk = ctypes.create_string_buffer(crypto_sign_SECRETKEYBYTES) + ret = nacl.crypto_sign_keypair(vk, sk) + if ret: + raise ValueError('Failed to generate keypair') + return vk.raw, sk.raw + + +def crypto_sign(msg, sk): + ''' + Sign the given message witht he given signing key + ''' + sig = ctypes.create_string_buffer(len(msg) + crypto_sign_BYTES) + slen = ctypes.pointer(ctypes.c_ulonglong()) + ret = nacl.crypto_sign( + sig, + slen, + msg, + ctypes.c_ulonglong(len(msg)), + sk) + if ret: + raise ValueError('Failed to sign message') + return sig.raw + + +def crypto_sign_seed_keypair(seed): + ''' + Computes and returns the secret adn verify keys from the given seed + ''' + if len(seed) != crypto_sign_SEEDBYTES: + raise ValueError('Invalid Seed') + sk = ctypes.create_string_buffer(crypto_sign_SECRETKEYBYTES) + vk = ctypes.create_string_buffer(crypto_sign_PUBLICKEYBYTES) + + ret = nacl.crypto_sign_seed_keypair(vk, sk, seed) + if ret: + raise CryptError('Failed to generate keypair from seed') + return (vk.raw, sk.raw) + + +def crypto_sign_open(sig, vk): + ''' + Verifies the signed message sig using the signer's verification key + ''' + msg = ctypes.create_string_buffer(len(sig)) + msglen = ctypes.c_ulonglong() + msglenp = ctypes.pointer(msglen) + ret = nacl.crypto_sign_open( + msg, + msglenp, + sig, + ctypes.c_ulonglong(len(sig)), + vk) + if ret: + raise ValueError('Failed to validate message') + return msg.raw[:msglen.value] # pylint: disable=invalid-slice-index + +# Authenticated Symmetric Encryption + + +def crypto_secretbox(msg, nonce, key): + ''' + Encrypts and authenticates a message using the given secret key, and nonce + ''' + pad = b'\x00' * crypto_secretbox_ZEROBYTES + msg + ctxt = ctypes.create_string_buffer(len(pad)) + ret = nacl.crypto_secretbox(ctxt, pad, ctypes.c_ulonglong(len(pad)), nonce, key) + if ret: + raise ValueError('Failed to encrypt message') + return ctxt.raw[crypto_secretbox_BOXZEROBYTES:] + + +def crypto_secretbox_open(ctxt, nonce, key): + ''' + Decrypts a ciphertext ctxt given the receivers private key, and senders + public key + ''' + pad = b'\x00' * crypto_secretbox_BOXZEROBYTES + ctxt + msg = ctypes.create_string_buffer(len(pad)) + ret = nacl.crypto_secretbox_open( + msg, + pad, + ctypes.c_ulonglong(len(pad)), + nonce, + key) + if ret: + raise ValueError('Failed to decrypt message') + return msg.raw[crypto_secretbox_ZEROBYTES:] + +# Symmetric Encryption + + +def crypto_stream(slen, nonce, key): + ''' + Generates a stream using the given secret key and nonce + ''' + stream = ctypes.create_string_buffer(slen) + ret = nacl.crypto_stream(stream, ctypes.c_ulonglong(slen), nonce, key) + if ret: + raise ValueError('Failed to init stream') + return stream.raw + + +def crypto_stream_xor(msg, nonce, key): + ''' + Encrypts the given message using the given secret key and nonce + + The crypto_stream_xor function guarantees that the ciphertext is the + plaintext (xor) the output of crypto_stream. Consequently + crypto_stream_xor can also be used to decrypt + ''' + stream = ctypes.create_string_buffer(len(msg)) + ret = nacl.crypto_stream_xor( + stream, + msg, + ctypes.c_ulonglong(len(msg)), + nonce, + key) + if ret: + raise ValueError('Failed to init stream') + return stream.raw + + +# Authentication + + +def crypto_auth(msg, key): + ''' + Constructs a one time authentication token for the given message msg + using a given secret key + ''' + tok = ctypes.create_string_buffer(crypto_auth_BYTES) + ret = nacl.crypto_auth(tok, msg, ctypes.c_ulonglong(len(msg)), key) + if ret: + raise ValueError('Failed to auth msg') + return tok.raw[:crypto_auth_BYTES] + + +def crypto_auth_verify(msg, key): + ''' + Verifies that the given authentication token is correct for the given + message and key + ''' + tok = ctypes.create_string_buffer(crypto_auth_BYTES) + ret = nacl.crypto_auth_verify(tok, msg, ctypes.c_ulonglong(len(msg)), key) + if ret: + raise ValueError('Failed to auth msg') + return tok.raw[:crypto_auth_BYTES] + +# One time authentication + + +def crypto_onetimeauth(msg, key): + ''' + Constructs a one time authentication token for the given message msg using + a given secret key + ''' + tok = ctypes.create_string_buffer(crypto_onetimeauth_BYTES) + ret = nacl.crypto_onetimeauth(tok, msg, ctypes.c_ulonglong(len(msg)), key) + if ret: + raise ValueError('Failed to auth msg') + return tok.raw[:crypto_onetimeauth_BYTES] + + +def crypto_onetimeauth_verify(msg, key): + ''' + Verifies that the given authentication token is correct for the given + message and key + ''' + tok = ctypes.create_string_buffer(crypto_onetimeauth_BYTES) + ret = nacl.crypto_onetimeauth(tok, msg, ctypes.c_ulonglong(len(msg)), key) + if ret: + raise ValueError('Failed to auth msg') + return tok.raw[:crypto_onetimeauth_BYTES] + +# Hashing + + +def crypto_hash(msg): + ''' + Compute a hash of the given message + ''' + hbuf = ctypes.create_string_buffer(crypto_hash_BYTES) + nacl.crypto_hash(hbuf, msg, ctypes.c_ulonglong(len(msg))) + return hbuf.raw + + +def crypto_hash_sha256(msg): + ''' + Compute the sha256 hash of the given message + ''' + hbuf = ctypes.create_string_buffer(crypto_hash_sha256_BYTES) + nacl.crypto_hash_sha256(hbuf, msg, ctypes.c_ulonglong(len(msg))) + return hbuf.raw + + +def crypto_hash_sha512(msg): + ''' + Compute the sha512 hash of the given message + ''' + hbuf = ctypes.create_string_buffer(crypto_hash_sha512_BYTES) + nacl.crypto_hash_sha512(hbuf, msg, ctypes.c_ulonglong(len(msg))) + return hbuf.raw + +# Generic Hash + + +def crypto_generichash(msg, key=None): + ''' + Compute the blake2 hash of the given message with a given key + ''' + hbuf = ctypes.create_string_buffer(crypto_generichash_BYTES) + if key: + key_len = len(key) + else: + key_len = 0 + nacl.crypto_generichash( + hbuf, + ctypes.c_ulonglong(len(hbuf)), + msg, + ctypes.c_ulonglong(len(msg)), + key, + ctypes.c_ulonglong(key_len)) + return hbuf.raw + +# scalarmult + + +def crypto_scalarmult_base(n): + ''' + Computes and returns the scalar product of a standard group element and an + integer "n". + ''' + buf = ctypes.create_string_buffer(crypto_scalarmult_BYTES) + ret = nacl.crypto_scalarmult_base(buf, n) + if ret: + raise CryptError('Failed to compute scalar product') + return buf.raw + +# String cmp + + +def crypto_verify_16(string1, string2): + ''' + Compares the first crypto_verify_16_BYTES of the given strings + + The time taken by the function is independent of the contents of string1 + and string2. In contrast, the standard C comparison function + memcmp(string1,string2,16) takes time that is dependent on the longest + matching prefix of string1 and string2. This often allows for easy + timing attacks. + ''' + return not nacl.crypto_verify_16(string1, string2) + + +def crypto_verify_32(string1, string2): + ''' + Compares the first crypto_verify_32_BYTES of the given strings + + The time taken by the function is independent of the contents of string1 + and string2. In contrast, the standard C comparison function + memcmp(string1,string2,16) takes time that is dependent on the longest + matching prefix of string1 and string2. This often allows for easy + timing attacks. + ''' + return not nacl.crypto_verify_32(string1, string2) + + +# Random byte generation + +def randombytes(size): + ''' + Return a string of random bytes of the given size + ''' + buf = ctypes.create_string_buffer(size) + nacl.randombytes(buf, ctypes.c_ulonglong(size)) + return buf.raw + + +def randombytes_buf(size): + ''' + Return a string of random bytes of the given size + ''' + size = int(size) + buf = ctypes.create_string_buffer(size) + nacl.randombytes_buf(buf, size) + return buf.raw + + +def randombytes_close(): + ''' + Close the file descriptor or the handle for the cryptographic service + provider + ''' + nacl.randombytes_close() + + +def randombytes_random(): + ''' + Return a random 32-bit unsigned value + ''' + return nacl.randombytes_random() + + +def randombytes_stir(): + ''' + Generate a new key for the pseudorandom number generator + + The file descriptor for the entropy source is kept open, so that the + generator can be reseeded even in a chroot() jail. + ''' + nacl.randombytes_stir() + + +def randombytes_uniform(upper_bound): + ''' + Return a value between 0 and upper_bound using a uniform distribution + ''' + return nacl.randombytes_uniform(upper_bound) + + +# Utility functions + +def sodium_library_version_major(): + ''' + Return the major version number + ''' + return nacl.sodium_library_version_major() + + +def sodium_library_version_minor(): + ''' + Return the minor version number + ''' + return nacl.sodium_library_version_minor() + + +def sodium_version_string(): + ''' + Return the version string + ''' + func = nacl.sodium_version_string + func.restype = ctypes.c_char_p + return func() diff --git a/lib/libnacl/base.py b/lib/libnacl/base.py new file mode 100644 index 00000000..6885b548 --- /dev/null +++ b/lib/libnacl/base.py @@ -0,0 +1,64 @@ +# -*- coding: utf-8 -*- +''' +Implement the base key object for other keys to inherit convenience functions +''' +# Import libnacl libs +import libnacl.encode + +# Import python libs +import os +import stat + +class BaseKey(object): + ''' + Include methods for key management convenience + ''' + def hex_sk(self): + if hasattr(self, 'sk'): + return libnacl.encode.hex_encode(self.sk) + else: + return '' + + def hex_pk(self): + if hasattr(self, 'pk'): + return libnacl.encode.hex_encode(self.pk) + + def hex_vk(self): + if hasattr(self, 'vk'): + return libnacl.encode.hex_encode(self.vk) + + def hex_seed(self): + if hasattr(self, 'seed'): + return libnacl.encode.hex_encode(self.seed) + + def save(self, path, serial='json'): + ''' + Safely save keys with perms of 0400 + ''' + pre = {} + sk = self.hex_sk() + pk = self.hex_pk() + vk = self.hex_vk() + seed = self.hex_seed() + if sk and pk: + pre['priv'] = sk.decode('utf-8') + if pk: + pre['pub'] = pk.decode('utf-8') + if vk: + pre['verify'] = vk.decode('utf-8') + if seed: + pre['sign'] = seed.decode('utf-8') + if serial == 'msgpack': + import msgpack + packaged = msgpack.dumps(pre) + elif serial == 'json': + import json + packaged = json.dumps(pre) + + perm_other = stat.S_IWOTH | stat.S_IXOTH | stat.S_IWOTH + perm_group = stat.S_IXGRP | stat.S_IWGRP | stat.S_IRWXG + + cumask = os.umask(perm_other | perm_group) + with open(path, 'w+') as fp_: + fp_.write(packaged) + os.umask(cumask) diff --git a/lib/libnacl/blake.py b/lib/libnacl/blake.py new file mode 100644 index 00000000..8da7d697 --- /dev/null +++ b/lib/libnacl/blake.py @@ -0,0 +1,44 @@ +''' +Mimic very closely the python hashlib classes for blake2b + +NOTE: + This class does not yet implement streaming the msg into the + hash function via the update method +''' + +# Import python libs +import binascii + +# Import libnacl libs +import libnacl + + +class Blake2b(object): + ''' + Manage a Blake2b hash + ''' + def __init__(self, msg, key=None): + self.msg = msg + self.key = key + self.raw_digest = libnacl.crypto_generichash(msg, key) + self.digest_size = len(self.raw_digest) + + def digest(self): + ''' + Return the digest of the string + ''' + return self.raw_digest + + def hexdigest(self): + ''' + Return the hex digest of the string + ''' + return binascii.hexlify(self.raw_digest) + + +def blake2b(msg, key=None): + ''' + Create and return a Blake2b object to mimic the behavior of the python + hashlib functions + ''' + return Blake2b(msg, key) diff --git a/lib/libnacl/dual.py b/lib/libnacl/dual.py new file mode 100644 index 00000000..c48fb673 --- /dev/null +++ b/lib/libnacl/dual.py @@ -0,0 +1,34 @@ +''' +The dual key system allows for the creation of keypairs that contain both +cryptographic and signing keys +''' +# import libnacl libs +import libnacl +import libnacl.base +import libnacl.public +import libnacl.sign + + +class DualSecret(libnacl.base.BaseKey): + ''' + Manage crypt and sign keys in one object + ''' + def __init__(self, crypt=None, sign=None): + self.crypt = libnacl.public.SecretKey(crypt) + self.signer = libnacl.sign.Signer(sign) + self.sk = self.crypt.sk + self.seed = self.signer.seed + self.pk = self.crypt.pk + self.vk = self.signer.vk + + def sign(self, msg): + ''' + Sign the given message + ''' + return self.signer.sign(msg) + + def signature(self, msg): + ''' + Return just the signature for the message + ''' + return self.signer.signature(msg) diff --git a/lib/libnacl/encode.py b/lib/libnacl/encode.py new file mode 100644 index 00000000..efbfe998 --- /dev/null +++ b/lib/libnacl/encode.py @@ -0,0 +1,63 @@ +# -*- coding: utf-8 -*- +''' +Build in routines and classes to simplify encoding routines +''' +# Import python libs +import base64 +import binascii + + +def hex_encode(data): + ''' + Hex encode data + ''' + return binascii.hexlify(data) + + +def hex_decode(data): + ''' + Hex decode data + ''' + return binascii.unhexlify(data) + + +def base16_encode(data): + ''' + Base32 encode data + ''' + return base64.b16encode(data) + + +def base16_decode(data): + ''' + Base16 decode data + ''' + return base64.b16decode(data) + + +def base32_encode(data): + ''' + Base16 encode data + ''' + return base64.b32encode(data) + + +def base32_decode(data): + ''' + Base32 decode data + ''' + return base64.b32decode(data) + + +def base64_encode(data): + ''' + Base16 encode data + ''' + return base64.b64encode(data) + + +def base64_decode(data): + ''' + Base32 decode data + ''' + return base64.b64decode(data) diff --git a/lib/libnacl/public.py b/lib/libnacl/public.py new file mode 100644 index 00000000..81f5b342 --- /dev/null +++ b/lib/libnacl/public.py @@ -0,0 +1,81 @@ +# -*- coding: utf-8 -*- +''' +High level classes and routines around public key encryption and decryption +''' +# import libnacl libs +import libnacl +import libnacl.utils +import libnacl.encode +import libnacl.dual +import libnacl.base + + +class PublicKey(libnacl.base.BaseKey): + ''' + This class is used to manage public keys + ''' + def __init__(self, pk): + self.pk = pk + + +class SecretKey(libnacl.base.BaseKey): + ''' + This class is used to manage keypairs + ''' + def __init__(self, sk=None): + ''' + If a secret key is not passed in then it will be generated + ''' + if sk is None: + self.pk, self.sk = libnacl.crypto_box_keypair() + elif len(sk) == libnacl.crypto_box_SECRETKEYBYTES: + self.sk = sk + self.pk = libnacl.crypto_scalarmult_base(sk) + else: + raise ValueError('Passed in invalid secret key') + + +class Box(object): + ''' + TheBox class is used to create cryptographic boxes and unpack + cryptographic boxes + ''' + def __init__(self, sk, pk): + if isinstance(sk, (SecretKey, libnacl.dual.DualSecret)): + sk = sk.sk + if isinstance(pk, (SecretKey, libnacl.dual.DualSecret)): + raise ValueError('Passed in secret key as public key') + if isinstance(pk, PublicKey): + pk = pk.pk + if pk and sk: + self._k = libnacl.crypto_box_beforenm(pk, sk) + + def encrypt(self, msg, nonce=None, pack_nonce=True): + ''' + Encrypt the given message with the given nonce, if the nonce is not + provided it will be generated from the libnacl.utils.rand_nonce + function + ''' + if nonce is None: + nonce = libnacl.utils.rand_nonce() + elif len(nonce) != libnacl.crypto_box_NONCEBYTES: + raise ValueError('Invalid nonce size') + ctxt = libnacl.crypto_box_afternm(msg, nonce, self._k) + if pack_nonce: + return nonce + ctxt + else: + return nonce, ctxt + + def decrypt(self, ctxt, nonce=None): + ''' + Decrypt the given message, if a nonce is passed in attempt to decrypt + it with the given nonce, otherwise assum that the nonce is attached + to the message + ''' + if nonce is None: + nonce = ctxt[:libnacl.crypto_box_NONCEBYTES] + ctxt = ctxt[libnacl.crypto_box_NONCEBYTES:] + elif len(nonce) != libnacl.crypto_box_NONCEBYTES: + raise ValueError('Invalid nonce') + msg = libnacl.crypto_box_open_afternm(ctxt, nonce, self._k) + return msg diff --git a/lib/libnacl/secret.py b/lib/libnacl/secret.py new file mode 100644 index 00000000..12a2c235 --- /dev/null +++ b/lib/libnacl/secret.py @@ -0,0 +1,44 @@ +# -*- coding: utf-8 -*- +''' +Utilities to make secret box encryption simple +''' +# Import libnacl +import libnacl +import libnacl.utils +import libnacl.base + + +class SecretBox(libnacl.base.BaseKey): + ''' + Manage symetric encryption using the salsa20 algorithm + ''' + def __init__(self, key=None): + if key is None: + key = libnacl.utils.salsa_key() + if len(key) != libnacl.crypto_secretbox_KEYBYTES: + raise ValueError('Invalid key') + self.sk = key + + def encrypt(self, msg, nonce=None): + ''' + Encrypt the given message. If a nonce is not given it will be + generated via the rand_nonce function + ''' + if nonce is None: + nonce = libnacl.utils.rand_nonce() + if len(nonce) != libnacl.crypto_secretbox_NONCEBYTES: + raise ValueError('Invalid Nonce') + ctxt = libnacl.crypto_secretbox(msg, nonce, self.sk) + return nonce + ctxt + + def decrypt(self, ctxt, nonce=None): + ''' + Decrypt the given message, if no nonce is given the nonce will be + extracted from the message + ''' + if nonce is None: + nonce = ctxt[:libnacl.crypto_secretbox_NONCEBYTES] + ctxt = ctxt[libnacl.crypto_secretbox_NONCEBYTES:] + if len(nonce) != libnacl.crypto_secretbox_NONCEBYTES: + raise ValueError('Invalid nonce') + return libnacl.crypto_secretbox_open(ctxt, nonce, self.sk) diff --git a/lib/libnacl/sign.py b/lib/libnacl/sign.py new file mode 100644 index 00000000..9dbd83f5 --- /dev/null +++ b/lib/libnacl/sign.py @@ -0,0 +1,55 @@ +# -*- coding: utf-8 -*- +''' +High level routines to maintain signing keys and to sign and verify messages +''' +# Import libancl libs +import libnacl +import libnacl.base +import libnacl.encode + + +class Signer(libnacl.base.BaseKey): + ''' + The tools needed to sign messages + ''' + def __init__(self, seed=None): + ''' + Create a signing key, if not seed it supplied a keypair is generated + ''' + if seed: + if len(seed) != libnacl.crypto_sign_SEEDBYTES: + raise ValueError('Invalid seed bytes') + self.vk, self.sk = libnacl.crypto_sign_seed_keypair(seed) + else: + seed = libnacl.randombytes(libnacl.crypto_sign_SEEDBYTES) + self.vk, self.sk = libnacl.crypto_sign_seed_keypair(seed) + self.seed = seed + + def sign(self, msg): + ''' + Sign the given message with this key + ''' + return libnacl.crypto_sign(msg, self.sk) + + def signature(self, msg): + ''' + Return just the signature for the message + ''' + return libnacl.crypto_sign(msg, self.sk)[:libnacl.crypto_sign_BYTES] + + +class Verifier(libnacl.base.BaseKey): + ''' + Verify signed messages + ''' + def __init__(self, vk_hex): + ''' + Create a verification key from a hex encoded vkey + ''' + self.vk = libnacl.encode.hex_decode(vk_hex) + + def verify(self, msg): + ''' + Verify the message with tis key + ''' + return libnacl.crypto_sign_open(msg, self.vk) diff --git a/lib/libnacl/utils.py b/lib/libnacl/utils.py new file mode 100644 index 00000000..c8429d87 --- /dev/null +++ b/lib/libnacl/utils.py @@ -0,0 +1,68 @@ +# -*- coding: utf-8 -*- + +import struct +import time + +# Import nacl libs +import libnacl +import libnacl.encode +import libnacl.public +import libnacl.sign +import libnacl.dual + + +def load_key(path, serial='json'): + ''' + Read in a key from a file and return the applicable key object based on + the contents of the file + ''' + with open(path, 'rb') as fp_: + packaged = fp_.read() + if serial == 'msgpack': + import msgpack + key_data = msgpack.loads(packaged) + elif serial == 'json': + import json + key_data = json.loads(packaged.decode(encoding='UTF-8')) + if 'priv' in key_data and 'sign' in key_data: + return libnacl.dual.DualSecret( + libnacl.encode.hex_decode(key_data['priv']), + libnacl.encode.hex_decode(key_data['sign'])) + elif 'priv' in key_data: + return libnacl.public.SecretKey( + libnacl.encode.hex_decode(key_data['priv'])) + elif 'sign' in key_data: + return libnacl.sign.Signer( + libnacl.encode.hex_decode(key_data['sign'])) + elif 'pub' in key_data: + return libnacl.public.PublicKey( + libnacl.encode.hex_decode(key_data['pub'])) + elif 'verify' in key_data: + return libnacl.sign.Verifier(key_data['verify']) + raise ValueError('Found no key data') + + +def salsa_key(): + ''' + Generates a salsa2020 key + ''' + return libnacl.randombytes(libnacl.crypto_secretbox_KEYBYTES) + + +def rand_nonce(): + ''' + Generates and returns a random bytestring of the size defined in libsodium + as crypto_box_NONCEBYTES + ''' + return libnacl.randombytes(libnacl.crypto_box_NONCEBYTES) + + +def time_nonce(): + ''' + Generates and returns a nonce as in rand_nonce() but using a timestamp for the first 8 bytes. + + This function now exists mostly for backwards compatibility, as rand_nonce() is usually preferred. + ''' + nonce = rand_nonce() + return (struct.pack('=d', time.time()) + nonce)[:len(nonce)] + diff --git a/lib/libnacl/version.py b/lib/libnacl/version.py new file mode 100644 index 00000000..96e3ce8d --- /dev/null +++ b/lib/libnacl/version.py @@ -0,0 +1 @@ +__version__ = '1.4.0' diff --git a/lib/maker.py b/lib/maker.py new file mode 100644 index 00000000..1bdfa501 --- /dev/null +++ b/lib/maker.py @@ -0,0 +1,357 @@ +#! /usr/bin/env python + +from common import * +import common +from taker import CoinJoinerPeer +import bitcoin as btc +import base64, pprint, threading +import enc_wrapper + +class CoinJoinOrder(object): + def __init__(self, maker, nick, oid, amount, taker_pk): + self.maker = maker + self.oid = oid + self.cj_amount = amount + if self.cj_amount <= common.DUST_THRESHOLD: + self.maker.msgchan.send_error(nick, 'amount below dust threshold') + #the btc pubkey of the utxo that the taker plans to use as input + self.taker_pk = taker_pk + #create DH keypair on the fly for this Order object + self.kp = enc_wrapper.init_keypair() + #the encryption channel crypto box for this Order object + self.crypto_box = enc_wrapper.as_init_encryption(self.kp, \ + enc_wrapper.init_pubkey(taker_pk)) + + order_s = [o for o in maker.orderlist if o['oid'] == oid] + if len(order_s) == 0: + self.maker.msgchan.send_error(nick, 'oid not found') + order = order_s[0] + if amount < order['minsize'] or amount > order['maxsize']: + self.maker.msgchan.send_error(nick, 'amount out of range') + self.ordertype = order['ordertype'] + self.txfee = order['txfee'] + self.cjfee = order['cjfee'] + debug('new cjorder nick=%s oid=%d amount=%d' % (nick, oid, amount)) + self.utxos, self.cj_addr, self.change_addr = maker.oid_to_order(self, oid, amount) + self.maker.wallet.update_cache_index() + if not self.utxos: + self.maker.msgchan.send_error(nick, 'unable to fill order constrained by dust avoidance') + #TODO make up orders offers in a way that this error cant appear + #check nothing has messed up with the wallet code, remove this code after a while + import pprint + debug('maker utxos = ' + pprint.pformat(self.utxos)) + utxo_list = self.utxos.keys() + utxo_data = common.bc_interface.query_utxo_set(utxo_list) + if None in utxo_data: + debug('wrongly using an already spent utxo. utxo_data = ' + pprint.pformat(utxo_data)) + sys.exit(0) + for utxo, data in zip(utxo_list, utxo_data): + if self.utxos[utxo]['value'] != data['value']: + debug('wrongly labeled utxo, expected value ' + + str(self.utxos[utxo]['value']) + ' got ' + str(data['value'])) + sys.exit(0) + + #always a new address even if the order ends up never being + # furfilled, you dont want someone pretending to fill all your + # orders to find out which addresses you use + self.maker.msgchan.send_pubkey(nick, self.kp.hex_pk()) + + def auth_counterparty(self, nick, i_utxo_pubkey, btc_sig): + self.i_utxo_pubkey = i_utxo_pubkey + + if not btc.ecdsa_verify(self.taker_pk, btc_sig, self.i_utxo_pubkey): + print 'signature didnt match pubkey and message' + return False + #authorisation of taker passed + #(but input utxo pubkey is checked in verify_unsigned_tx). + #Send auth request to taker + #TODO the next 2 lines are a little inefficient. + btc_key = self.maker.wallet.get_key_from_addr(self.cj_addr) + btc_pub = btc.privtopub(btc_key) + btc_sig = btc.ecdsa_sign(self.kp.hex_pk(), btc_key) + self.maker.msgchan.send_ioauth(nick, self.utxos.keys(), btc_pub, self.change_addr, btc_sig) + return True + + def recv_tx(self, nick, txhex): + try: + self.tx = btc.deserialize(txhex) + except IndexError as e: + self.maker.msgchan.send_error(nick, 'malformed txhex. ' + repr(e)) + debug('obtained tx\n' + pprint.pformat(self.tx)) + goodtx, errmsg = self.verify_unsigned_tx(self.tx) + if not goodtx: + debug('not a good tx, reason=' + errmsg) + self.maker.msgchan.send_error(nick, errmsg) + #TODO: the above 3 errors should be encrypted, but it's a bit messy. + debug('goodtx') + sigs = [] + for index, ins in enumerate(self.tx['ins']): + utxo = ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index']) + if utxo not in self.utxos: + continue + addr = self.utxos[utxo]['address'] + txs = btc.sign(txhex, index, self.maker.wallet.get_key_from_addr(addr)) + sigs.append(base64.b64encode(btc.deserialize(txs)['ins'][index]['script'].decode('hex'))) + #len(sigs) > 0 guarenteed since i did verify_unsigned_tx() + + common.bc_interface.add_tx_notify(self.tx, self.unconfirm_callback, self.confirm_callback, self.cj_addr) + debug('sending sigs ' + str(sigs)) + self.maker.msgchan.send_sigs(nick, sigs) + self.maker.active_orders[nick] = None + + def unconfirm_callback(self, txd, txid): + self.maker.wallet_unspent_lock.acquire() + try: + removed_utxos = self.maker.wallet.remove_old_utxos(self.tx) + finally: + self.maker.wallet_unspent_lock.release() + debug('saw tx on network, removed_utxos=\n' + pprint.pformat(removed_utxos)) + to_cancel, to_announce = self.maker.on_tx_unconfirmed(self, txid, removed_utxos) + self.maker.modify_orders(to_cancel, to_announce) + + def confirm_callback(self, txd, txid, confirmations): + self.maker.wallet_unspent_lock.acquire() + try: + common.bc_interface.sync_unspent(self.maker.wallet) + finally: + self.maker.wallet_unspent_lock.release() + debug('tx in a block') + debug('earned = ' + str(self.real_cjfee - self.txfee)) + to_cancel, to_announce = self.maker.on_tx_confirmed(self, + confirmations, txid) + self.maker.modify_orders(to_cancel, to_announce) + + def verify_unsigned_tx(self, txd): + tx_utxo_set = set([ins['outpoint']['hash'] + ':' \ + + str(ins['outpoint']['index']) for ins in txd['ins']]) + #complete authentication: check the tx input uses the authing pubkey + input_utxo_data = common.bc_interface.query_utxo_set(list(tx_utxo_set)) + if None in input_utxo_data: + return False, 'some utxos already spent or not confirmed yet' + input_addresses = [u['address'] for u in input_utxo_data] + if btc.pubtoaddr(self.i_utxo_pubkey, get_p2pk_vbyte())\ + not in input_addresses: + return False, "authenticating bitcoin address is not contained" + my_utxo_set = set(self.utxos.keys()) + if not tx_utxo_set.issuperset(my_utxo_set): + return False, 'my utxos are not contained' + + my_total_in = sum([va['value'] for va in self.utxos.values()]) + self.real_cjfee = calc_cj_fee(self.ordertype, self.cjfee, self.cj_amount) + expected_change_value = (my_total_in - self.cj_amount + - self.txfee + self.real_cjfee) + debug('potentially earned = ' + str(self.real_cjfee - self.txfee)) + debug('mycjaddr, mychange = ' + self.cj_addr + ', ' + self.change_addr) + + times_seen_cj_addr = 0 + times_seen_change_addr = 0 + for outs in txd['outs']: + addr = btc.script_to_address(outs['script'], get_p2pk_vbyte()) + if addr == self.cj_addr: + times_seen_cj_addr += 1 + if outs['value'] != self.cj_amount: + return False, 'Wrong cj_amount. I expect ' + str(cj_amount) + if addr == self.change_addr: + times_seen_change_addr += 1 + if outs['value'] != expected_change_value: + return False, 'wrong change, i expect ' + str(expected_change_value) + if times_seen_cj_addr != 1 or times_seen_change_addr != 1: + return False, ('cj or change addr not in tx outputs once, #cjaddr=' + + str(times_seen_cj_addr) + ', #chaddr=' + str(times_seen_change_addr)) + return True, None + +class CJMakerOrderError(StandardError): + pass + +class Maker(CoinJoinerPeer): + def __init__(self, msgchan, wallet): + CoinJoinerPeer.__init__(self, msgchan) + self.msgchan.register_channel_callbacks(self.on_welcome, self.on_set_topic, + None, None, self.on_nick_leave, None) + msgchan.register_maker_callbacks(self.on_orderbook_requested, + self.on_order_fill, self.on_seen_auth, self.on_seen_tx, self.on_push_tx) + msgchan.cjpeer = self + + self.active_orders = {} + self.wallet = wallet + self.nextoid = -1 + self.orderlist = self.create_my_orders() + self.wallet_unspent_lock = threading.Lock() + + def get_crypto_box_from_nick(self, nick): + if nick not in self.active_orders: + debug('wrong ordering of protocol events, no crypto object, nick=' + nick) + return None + else: + return self.active_orders[nick].crypto_box + + def on_orderbook_requested(self, nick): + self.msgchan.announce_orders(self.orderlist, nick) + + def on_order_fill(self, nick, oid, amount, taker_pubkey): + if nick in self.active_orders and self.active_orders[nick] != None: + self.active_orders[nick] = None + debug('had a partially filled order but starting over now') + self.wallet_unspent_lock.acquire() + try: + self.active_orders[nick] = CoinJoinOrder(self, nick, oid, amount, taker_pubkey) + finally: + self.wallet_unspent_lock.release() + + def on_seen_auth(self, nick, pubkey, sig): + if nick not in self.active_orders or self.active_orders[nick] == None: + self.msgchan.send_error(nick, 'No open order from this nick') + self.active_orders[nick].auth_counterparty(nick, pubkey, sig) + #TODO if auth_counterparty returns false, remove this order from active_orders + # and send an error + + def on_seen_tx(self, nick, txhex): + if nick not in self.active_orders or self.active_orders[nick] == None: + self.msgchan.send_error(nick, 'No open order from this nick') + self.wallet_unspent_lock.acquire() + try: + self.active_orders[nick].recv_tx(nick, txhex) + finally: + self.wallet_unspent_lock.release() + + def on_push_tx(self, nick, txhex): + debug('received txhex from ' + nick + ' to push\n' + txhex) + txid = common.bc_interface.pushtx(txhex) + debug('pushed tx ' + str(txid)) + if txid == None: + self.send_error(nick, 'Unable to push tx') + + def on_welcome(self): + self.msgchan.announce_orders(self.orderlist) + self.active_orders = {} + + def on_nick_leave(self, nick): + if nick in self.active_orders: + debug('nick ' + nick + ' has left') + del self.active_orders[nick] + + def modify_orders(self, to_cancel, to_announce): + debug('modifying orders. to_cancel=' + str(to_cancel) + '\nto_announce=' + str(to_announce)) + for oid in to_cancel: + order = [o for o in self.orderlist if o['oid'] == oid] + if len(order) == 0: + debug('didnt cancel order which doesnt exist, oid=' + str(oid)) + self.orderlist.remove(order[0]) + if len(to_cancel) > 0: + self.msgchan.cancel_orders(to_cancel) + if len(to_announce) > 0: + self.msgchan.announce_orders(to_announce) + for ann in to_announce: + oldorder_s = [order for order in self.orderlist if order['oid'] == ann['oid']] + if len(oldorder_s) > 0: + self.orderlist.remove(oldorder_s[0]) + self.orderlist += to_announce + + #these functions + # create_my_orders() + # oid_to_uxto() + # on_tx_unconfirmed() + # on_tx_confirmed() + #define the sell-side pricing algorithm of this bot + #still might be a bad way of doing things, we'll see + def create_my_orders(self): + + ''' + #tells the highest value possible made by combining all utxos + #fee is 0.2% of the cj amount + total_value = 0 + for utxo, addrvalue in self.wallet.unspent.iteritems(): + total_value += addrvalue['value'] + + order = {'oid': 0, 'ordertype': 'relorder', 'minsize': 0, + 'maxsize': total_value, 'txfee': 10000, 'cjfee': '0.002'} + return [order] + ''' + + #each utxo is a single absolute-fee order + orderlist = [] + for utxo, addrvalue in self.wallet.unspent.iteritems(): + order = {'oid': self.get_next_oid(), 'ordertype': 'absorder', 'minsize': 12000, + 'maxsize': addrvalue['value'], 'txfee': 10000, 'cjfee': 100000, + 'utxo': utxo, 'mixdepth': self.wallet.addr_cache[addrvalue['address']][0]} + orderlist.append(order) + #yes you can add keys there that are never used by the rest of the Maker code + # so im adding utxo and mixdepth here + return orderlist + + + #has to return a list of utxos and mixing depth the cj address will be in + # the change address will be in mixing_depth-1 + def oid_to_order(self, cjorder, oid, amount): + ''' + unspent = [] + for utxo, addrvalue in self.wallet.unspent.iteritems(): + unspent.append({'value': addrvalue['value'], 'utxo': utxo}) + inputs = btc.select(unspent, amount) + #TODO this raises an exception if you dont have enough money, id rather it just returned None + mixing_depth = 1 + return [i['utxo'] for i in inputs], mixing_depth + ''' + + order = [o for o in self.orderlist if o['oid'] == oid][0] + cj_addr = self.wallet.get_receive_addr(order['mixdepth'] + 1) + change_addr = self.wallet.get_change_addr(order['mixdepth']) + return [order['utxo']], cj_addr, change_addr + + def get_next_oid(self): + self.nextoid += 1 + return self.nextoid + + #gets called when the tx is seen on the network + #must return which orders to cancel or recreate + def on_tx_unconfirmed(self, cjorder, txid, removed_utxos): + return ([cjorder.oid], []) + + #gets called when the tx is included in a block + #must return which orders to cancel or recreate + # and i have to think about how that will work for both + # the blockchain explorer api method and the bitcoid walletnotify + def on_tx_confirmed(self, cjorder, confirmations, txid): + to_announce = [] + for i, out in enumerate(cjorder.tx['outs']): + addr = btc.script_to_address(out['script'], get_p2pk_vbyte()) + if addr == cjorder.change_addr: + neworder = {'oid': self.get_next_oid(), 'ordertype': 'absorder', 'minsize': 12000, + 'maxsize': out['value'], 'txfee': 10000, 'cjfee': 100000, + 'utxo': txid + ':' + str(i)} + to_announce.append(neworder) + if addr == cjorder.cj_addr: + neworder = {'oid': self.get_next_oid(), 'ordertype': 'absorder', 'minsize': 12000, + 'maxsize': out['value'], 'txfee': 10000, 'cjfee': 100000, + 'utxo': txid + ':' + str(i)} + to_announce.append(neworder) + return ([], to_announce) + + +def main(): + from socket import gethostname + nickname = 'cj-maker-' + btc.sha256(gethostname())[:6] + import sys + seed = sys.argv[1] #btc.sha256('dont use brainwallets except for holding testnet coins') + + common.load_program_config() + wallet = Wallet(seed, max_mix_depth=5) + common.bc_interface.sync_wallet(wallet) + + from irc import IRCMessageChannel + irc = IRCMessageChannel(nickname) + maker = Maker(irc, wallet) + try: + print 'connecting to irc' + irc.run() + except: + debug('CRASHING, DUMPING EVERYTHING') + debug('wallet seed = ' + seed) + debug_dump_object(wallet, ['addr_cache']) + debug_dump_object(maker) + import traceback + traceback.print_exc() + +if __name__ == "__main__": + main() + print('done') diff --git a/lib/message_channel.py b/lib/message_channel.py new file mode 100644 index 00000000..a4c53006 --- /dev/null +++ b/lib/message_channel.py @@ -0,0 +1,79 @@ + +class CJPeerError(StandardError): + pass + +class MessageChannel(object): + ''' + Abstract class which implements a way for bots to communicate + ''' + + def __init__(self): + #all + self.on_welcome = None + self.on_set_topic = None + self.on_connect = None + self.on_disconnect = None + self.on_nick_leave = None + self.on_nick_change = None + #orderbook watch functions + self.on_order_seen = None + self.on_order_cancel = None + #taker functions + self.on_error = None + self.on_pubkey = None + self.on_ioauth = None + self.on_sig = None + #maker functions + self.on_orderbook_requested = None + self.on_order_fill = None + self.on_seen_auth = None + self.on_seen_tx = None + self.on_push_tx = None + + def run(self): pass + def shutdown(self): pass + def send_error(self, nick, errormsg): pass + + #callbacks for everyone + #some of these many not have meaning in a future channel, like bitmessage + def register_channel_callbacks(self, on_welcome=None, on_set_topic=None, + on_connect=None, on_disconnect=None, on_nick_leave=None, on_nick_change=None): + self.on_welcome = on_welcome + self.on_set_topic = on_set_topic + self.on_connect = on_connect + self.on_disconnect = on_disconnect + self.on_nick_leave = on_nick_leave + self.on_nick_change = on_nick_change + + #orderbook watcher commands + def register_orderbookwatch_callbacks(self, on_order_seen=None, + on_order_cancel=None): + self.on_order_seen = on_order_seen + self.on_order_cancel = on_order_cancel + def request_orderbook(self): pass + + #taker commands + def register_taker_callbacks(self, on_error=None, on_pubkey=None, on_ioauth=None, + on_sig=None): + self.on_error = on_error + self.on_pubkey = on_pubkey + self.on_ioauth = on_ioauth + self.on_sig = on_sig + def fill_orders(self, nickoid_dict, cj_amount, taker_pubkey): pass + def send_auth(self, nick, pubkey, sig): pass + def send_tx(self, nick_list, txhex): pass + def push_tx(self, nick, txhex): pass + + #maker commands + def register_maker_callbacks(self, on_orderbook_requested=None, on_order_fill=None, + on_seen_auth=None, on_seen_tx=None, on_push_tx=None): + self.on_orderbook_requested = on_orderbook_requested + self.on_order_fill = on_order_fill + self.on_seen_auth = on_seen_auth + self.on_seen_tx = on_seen_tx + self.on_push_tx = on_push_tx + def announce_orders(self, orderlist, nick=None): pass #nick=None means announce publicly + def cancel_orders(self, oid_list): pass + def send_pubkey(self, nick, pubkey): pass + def send_ioauth(self, nick, utxo_list, cj_pubkey, change_addr, sig): pass + def send_sigs(self, nick, sig_list): pass diff --git a/lib/old_mnemonic.py b/lib/old_mnemonic.py new file mode 100644 index 00000000..3a5449b8 --- /dev/null +++ b/lib/old_mnemonic.py @@ -0,0 +1,1691 @@ + +#!/usr/bin/env python +# +# Electrum - lightweight Bitcoin client +# Copyright (C) 2011 thomasv@gitorious +# +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see . + + + +# list of words from http://en.wiktionary.org/wiki/Wiktionary:Frequency_lists/Contemporary_poetry + +words = [ +"like", +"just", +"love", +"know", +"never", +"want", +"time", +"out", +"there", +"make", +"look", +"eye", +"down", +"only", +"think", +"heart", +"back", +"then", +"into", +"about", +"more", +"away", +"still", +"them", +"take", +"thing", +"even", +"through", +"long", +"always", +"world", +"too", +"friend", +"tell", +"try", +"hand", +"thought", +"over", +"here", +"other", +"need", +"smile", +"again", +"much", +"cry", +"been", +"night", +"ever", +"little", +"said", +"end", +"some", +"those", +"around", +"mind", +"people", +"girl", +"leave", +"dream", +"left", +"turn", +"myself", +"give", +"nothing", +"really", +"off", +"before", +"something", +"find", +"walk", +"wish", +"good", +"once", +"place", +"ask", +"stop", +"keep", +"watch", +"seem", +"everything", +"wait", +"got", +"yet", +"made", +"remember", +"start", +"alone", +"run", +"hope", +"maybe", +"believe", +"body", +"hate", +"after", +"close", +"talk", +"stand", +"own", +"each", +"hurt", +"help", +"home", +"god", +"soul", +"new", +"many", +"two", +"inside", +"should", +"true", +"first", +"fear", +"mean", +"better", +"play", +"another", +"gone", +"change", +"use", +"wonder", +"someone", +"hair", +"cold", +"open", +"best", +"any", +"behind", +"happen", +"water", +"dark", +"laugh", +"stay", +"forever", +"name", +"work", +"show", +"sky", +"break", +"came", +"deep", +"door", +"put", +"black", +"together", +"upon", +"happy", +"such", +"great", +"white", +"matter", +"fill", +"past", +"please", +"burn", +"cause", +"enough", +"touch", +"moment", +"soon", +"voice", +"scream", +"anything", +"stare", +"sound", +"red", +"everyone", +"hide", +"kiss", +"truth", +"death", +"beautiful", +"mine", +"blood", +"broken", +"very", +"pass", +"next", +"forget", +"tree", +"wrong", +"air", +"mother", +"understand", +"lip", +"hit", +"wall", +"memory", +"sleep", +"free", +"high", +"realize", +"school", +"might", +"skin", +"sweet", +"perfect", +"blue", +"kill", +"breath", +"dance", +"against", +"fly", +"between", +"grow", +"strong", +"under", +"listen", +"bring", +"sometimes", +"speak", +"pull", +"person", +"become", +"family", +"begin", +"ground", +"real", +"small", +"father", +"sure", +"feet", +"rest", +"young", +"finally", +"land", +"across", +"today", +"different", +"guy", +"line", +"fire", +"reason", +"reach", +"second", +"slowly", +"write", +"eat", +"smell", +"mouth", +"step", +"learn", +"three", +"floor", +"promise", +"breathe", +"darkness", +"push", +"earth", +"guess", +"save", +"song", +"above", +"along", +"both", +"color", +"house", +"almost", +"sorry", +"anymore", +"brother", +"okay", +"dear", +"game", +"fade", +"already", +"apart", +"warm", +"beauty", +"heard", +"notice", +"question", +"shine", +"began", +"piece", +"whole", +"shadow", +"secret", +"street", +"within", +"finger", +"point", +"morning", +"whisper", +"child", +"moon", +"green", +"story", +"glass", +"kid", +"silence", +"since", +"soft", +"yourself", +"empty", +"shall", +"angel", +"answer", +"baby", +"bright", +"dad", +"path", +"worry", +"hour", +"drop", +"follow", +"power", +"war", +"half", +"flow", +"heaven", +"act", +"chance", +"fact", +"least", +"tired", +"children", +"near", +"quite", +"afraid", +"rise", +"sea", +"taste", +"window", +"cover", +"nice", +"trust", +"lot", +"sad", +"cool", +"force", +"peace", +"return", +"blind", +"easy", +"ready", +"roll", +"rose", +"drive", +"held", +"music", +"beneath", +"hang", +"mom", +"paint", +"emotion", +"quiet", +"clear", +"cloud", +"few", +"pretty", +"bird", +"outside", +"paper", +"picture", +"front", +"rock", +"simple", +"anyone", +"meant", +"reality", +"road", +"sense", +"waste", +"bit", +"leaf", +"thank", +"happiness", +"meet", +"men", +"smoke", +"truly", +"decide", +"self", +"age", +"book", +"form", +"alive", +"carry", +"escape", +"damn", +"instead", +"able", +"ice", +"minute", +"throw", +"catch", +"leg", +"ring", +"course", +"goodbye", +"lead", +"poem", +"sick", +"corner", +"desire", +"known", +"problem", +"remind", +"shoulder", +"suppose", +"toward", +"wave", +"drink", +"jump", +"woman", +"pretend", +"sister", +"week", +"human", +"joy", +"crack", +"grey", +"pray", +"surprise", +"dry", +"knee", +"less", +"search", +"bleed", +"caught", +"clean", +"embrace", +"future", +"king", +"son", +"sorrow", +"chest", +"hug", +"remain", +"sat", +"worth", +"blow", +"daddy", +"final", +"parent", +"tight", +"also", +"create", +"lonely", +"safe", +"cross", +"dress", +"evil", +"silent", +"bone", +"fate", +"perhaps", +"anger", +"class", +"scar", +"snow", +"tiny", +"tonight", +"continue", +"control", +"dog", +"edge", +"mirror", +"month", +"suddenly", +"comfort", +"given", +"loud", +"quickly", +"gaze", +"plan", +"rush", +"stone", +"town", +"battle", +"ignore", +"spirit", +"stood", +"stupid", +"yours", +"brown", +"build", +"dust", +"hey", +"kept", +"pay", +"phone", +"twist", +"although", +"ball", +"beyond", +"hidden", +"nose", +"taken", +"fail", +"float", +"pure", +"somehow", +"wash", +"wrap", +"angry", +"cheek", +"creature", +"forgotten", +"heat", +"rip", +"single", +"space", +"special", +"weak", +"whatever", +"yell", +"anyway", +"blame", +"job", +"choose", +"country", +"curse", +"drift", +"echo", +"figure", +"grew", +"laughter", +"neck", +"suffer", +"worse", +"yeah", +"disappear", +"foot", +"forward", +"knife", +"mess", +"somewhere", +"stomach", +"storm", +"beg", +"idea", +"lift", +"offer", +"breeze", +"field", +"five", +"often", +"simply", +"stuck", +"win", +"allow", +"confuse", +"enjoy", +"except", +"flower", +"seek", +"strength", +"calm", +"grin", +"gun", +"heavy", +"hill", +"large", +"ocean", +"shoe", +"sigh", +"straight", +"summer", +"tongue", +"accept", +"crazy", +"everyday", +"exist", +"grass", +"mistake", +"sent", +"shut", +"surround", +"table", +"ache", +"brain", +"destroy", +"heal", +"nature", +"shout", +"sign", +"stain", +"choice", +"doubt", +"glance", +"glow", +"mountain", +"queen", +"stranger", +"throat", +"tomorrow", +"city", +"either", +"fish", +"flame", +"rather", +"shape", +"spin", +"spread", +"ash", +"distance", +"finish", +"image", +"imagine", +"important", +"nobody", +"shatter", +"warmth", +"became", +"feed", +"flesh", +"funny", +"lust", +"shirt", +"trouble", +"yellow", +"attention", +"bare", +"bite", +"money", +"protect", +"amaze", +"appear", +"born", +"choke", +"completely", +"daughter", +"fresh", +"friendship", +"gentle", +"probably", +"six", +"deserve", +"expect", +"grab", +"middle", +"nightmare", +"river", +"thousand", +"weight", +"worst", +"wound", +"barely", +"bottle", +"cream", +"regret", +"relationship", +"stick", +"test", +"crush", +"endless", +"fault", +"itself", +"rule", +"spill", +"art", +"circle", +"join", +"kick", +"mask", +"master", +"passion", +"quick", +"raise", +"smooth", +"unless", +"wander", +"actually", +"broke", +"chair", +"deal", +"favorite", +"gift", +"note", +"number", +"sweat", +"box", +"chill", +"clothes", +"lady", +"mark", +"park", +"poor", +"sadness", +"tie", +"animal", +"belong", +"brush", +"consume", +"dawn", +"forest", +"innocent", +"pen", +"pride", +"stream", +"thick", +"clay", +"complete", +"count", +"draw", +"faith", +"press", +"silver", +"struggle", +"surface", +"taught", +"teach", +"wet", +"bless", +"chase", +"climb", +"enter", +"letter", +"melt", +"metal", +"movie", +"stretch", +"swing", +"vision", +"wife", +"beside", +"crash", +"forgot", +"guide", +"haunt", +"joke", +"knock", +"plant", +"pour", +"prove", +"reveal", +"steal", +"stuff", +"trip", +"wood", +"wrist", +"bother", +"bottom", +"crawl", +"crowd", +"fix", +"forgive", +"frown", +"grace", +"loose", +"lucky", +"party", +"release", +"surely", +"survive", +"teacher", +"gently", +"grip", +"speed", +"suicide", +"travel", +"treat", +"vein", +"written", +"cage", +"chain", +"conversation", +"date", +"enemy", +"however", +"interest", +"million", +"page", +"pink", +"proud", +"sway", +"themselves", +"winter", +"church", +"cruel", +"cup", +"demon", +"experience", +"freedom", +"pair", +"pop", +"purpose", +"respect", +"shoot", +"softly", +"state", +"strange", +"bar", +"birth", +"curl", +"dirt", +"excuse", +"lord", +"lovely", +"monster", +"order", +"pack", +"pants", +"pool", +"scene", +"seven", +"shame", +"slide", +"ugly", +"among", +"blade", +"blonde", +"closet", +"creek", +"deny", +"drug", +"eternity", +"gain", +"grade", +"handle", +"key", +"linger", +"pale", +"prepare", +"swallow", +"swim", +"tremble", +"wheel", +"won", +"cast", +"cigarette", +"claim", +"college", +"direction", +"dirty", +"gather", +"ghost", +"hundred", +"loss", +"lung", +"orange", +"present", +"swear", +"swirl", +"twice", +"wild", +"bitter", +"blanket", +"doctor", +"everywhere", +"flash", +"grown", +"knowledge", +"numb", +"pressure", +"radio", +"repeat", +"ruin", +"spend", +"unknown", +"buy", +"clock", +"devil", +"early", +"false", +"fantasy", +"pound", +"precious", +"refuse", +"sheet", +"teeth", +"welcome", +"add", +"ahead", +"block", +"bury", +"caress", +"content", +"depth", +"despite", +"distant", +"marry", +"purple", +"threw", +"whenever", +"bomb", +"dull", +"easily", +"grasp", +"hospital", +"innocence", +"normal", +"receive", +"reply", +"rhyme", +"shade", +"someday", +"sword", +"toe", +"visit", +"asleep", +"bought", +"center", +"consider", +"flat", +"hero", +"history", +"ink", +"insane", +"muscle", +"mystery", +"pocket", +"reflection", +"shove", +"silently", +"smart", +"soldier", +"spot", +"stress", +"train", +"type", +"view", +"whether", +"bus", +"energy", +"explain", +"holy", +"hunger", +"inch", +"magic", +"mix", +"noise", +"nowhere", +"prayer", +"presence", +"shock", +"snap", +"spider", +"study", +"thunder", +"trail", +"admit", +"agree", +"bag", +"bang", +"bound", +"butterfly", +"cute", +"exactly", +"explode", +"familiar", +"fold", +"further", +"pierce", +"reflect", +"scent", +"selfish", +"sharp", +"sink", +"spring", +"stumble", +"universe", +"weep", +"women", +"wonderful", +"action", +"ancient", +"attempt", +"avoid", +"birthday", +"branch", +"chocolate", +"core", +"depress", +"drunk", +"especially", +"focus", +"fruit", +"honest", +"match", +"palm", +"perfectly", +"pillow", +"pity", +"poison", +"roar", +"shift", +"slightly", +"thump", +"truck", +"tune", +"twenty", +"unable", +"wipe", +"wrote", +"coat", +"constant", +"dinner", +"drove", +"egg", +"eternal", +"flight", +"flood", +"frame", +"freak", +"gasp", +"glad", +"hollow", +"motion", +"peer", +"plastic", +"root", +"screen", +"season", +"sting", +"strike", +"team", +"unlike", +"victim", +"volume", +"warn", +"weird", +"attack", +"await", +"awake", +"built", +"charm", +"crave", +"despair", +"fought", +"grant", +"grief", +"horse", +"limit", +"message", +"ripple", +"sanity", +"scatter", +"serve", +"split", +"string", +"trick", +"annoy", +"blur", +"boat", +"brave", +"clearly", +"cling", +"connect", +"fist", +"forth", +"imagination", +"iron", +"jock", +"judge", +"lesson", +"milk", +"misery", +"nail", +"naked", +"ourselves", +"poet", +"possible", +"princess", +"sail", +"size", +"snake", +"society", +"stroke", +"torture", +"toss", +"trace", +"wise", +"bloom", +"bullet", +"cell", +"check", +"cost", +"darling", +"during", +"footstep", +"fragile", +"hallway", +"hardly", +"horizon", +"invisible", +"journey", +"midnight", +"mud", +"nod", +"pause", +"relax", +"shiver", +"sudden", +"value", +"youth", +"abuse", +"admire", +"blink", +"breast", +"bruise", +"constantly", +"couple", +"creep", +"curve", +"difference", +"dumb", +"emptiness", +"gotta", +"honor", +"plain", +"planet", +"recall", +"rub", +"ship", +"slam", +"soar", +"somebody", +"tightly", +"weather", +"adore", +"approach", +"bond", +"bread", +"burst", +"candle", +"coffee", +"cousin", +"crime", +"desert", +"flutter", +"frozen", +"grand", +"heel", +"hello", +"language", +"level", +"movement", +"pleasure", +"powerful", +"random", +"rhythm", +"settle", +"silly", +"slap", +"sort", +"spoken", +"steel", +"threaten", +"tumble", +"upset", +"aside", +"awkward", +"bee", +"blank", +"board", +"button", +"card", +"carefully", +"complain", +"crap", +"deeply", +"discover", +"drag", +"dread", +"effort", +"entire", +"fairy", +"giant", +"gotten", +"greet", +"illusion", +"jeans", +"leap", +"liquid", +"march", +"mend", +"nervous", +"nine", +"replace", +"rope", +"spine", +"stole", +"terror", +"accident", +"apple", +"balance", +"boom", +"childhood", +"collect", +"demand", +"depression", +"eventually", +"faint", +"glare", +"goal", +"group", +"honey", +"kitchen", +"laid", +"limb", +"machine", +"mere", +"mold", +"murder", +"nerve", +"painful", +"poetry", +"prince", +"rabbit", +"shelter", +"shore", +"shower", +"soothe", +"stair", +"steady", +"sunlight", +"tangle", +"tease", +"treasure", +"uncle", +"begun", +"bliss", +"canvas", +"cheer", +"claw", +"clutch", +"commit", +"crimson", +"crystal", +"delight", +"doll", +"existence", +"express", +"fog", +"football", +"gay", +"goose", +"guard", +"hatred", +"illuminate", +"mass", +"math", +"mourn", +"rich", +"rough", +"skip", +"stir", +"student", +"style", +"support", +"thorn", +"tough", +"yard", +"yearn", +"yesterday", +"advice", +"appreciate", +"autumn", +"bank", +"beam", +"bowl", +"capture", +"carve", +"collapse", +"confusion", +"creation", +"dove", +"feather", +"girlfriend", +"glory", +"government", +"harsh", +"hop", +"inner", +"loser", +"moonlight", +"neighbor", +"neither", +"peach", +"pig", +"praise", +"screw", +"shield", +"shimmer", +"sneak", +"stab", +"subject", +"throughout", +"thrown", +"tower", +"twirl", +"wow", +"army", +"arrive", +"bathroom", +"bump", +"cease", +"cookie", +"couch", +"courage", +"dim", +"guilt", +"howl", +"hum", +"husband", +"insult", +"led", +"lunch", +"mock", +"mostly", +"natural", +"nearly", +"needle", +"nerd", +"peaceful", +"perfection", +"pile", +"price", +"remove", +"roam", +"sanctuary", +"serious", +"shiny", +"shook", +"sob", +"stolen", +"tap", +"vain", +"void", +"warrior", +"wrinkle", +"affection", +"apologize", +"blossom", +"bounce", +"bridge", +"cheap", +"crumble", +"decision", +"descend", +"desperately", +"dig", +"dot", +"flip", +"frighten", +"heartbeat", +"huge", +"lazy", +"lick", +"odd", +"opinion", +"process", +"puzzle", +"quietly", +"retreat", +"score", +"sentence", +"separate", +"situation", +"skill", +"soak", +"square", +"stray", +"taint", +"task", +"tide", +"underneath", +"veil", +"whistle", +"anywhere", +"bedroom", +"bid", +"bloody", +"burden", +"careful", +"compare", +"concern", +"curtain", +"decay", +"defeat", +"describe", +"double", +"dreamer", +"driver", +"dwell", +"evening", +"flare", +"flicker", +"grandma", +"guitar", +"harm", +"horrible", +"hungry", +"indeed", +"lace", +"melody", +"monkey", +"nation", +"object", +"obviously", +"rainbow", +"salt", +"scratch", +"shown", +"shy", +"stage", +"stun", +"third", +"tickle", +"useless", +"weakness", +"worship", +"worthless", +"afternoon", +"beard", +"boyfriend", +"bubble", +"busy", +"certain", +"chin", +"concrete", +"desk", +"diamond", +"doom", +"drawn", +"due", +"felicity", +"freeze", +"frost", +"garden", +"glide", +"harmony", +"hopefully", +"hunt", +"jealous", +"lightning", +"mama", +"mercy", +"peel", +"physical", +"position", +"pulse", +"punch", +"quit", +"rant", +"respond", +"salty", +"sane", +"satisfy", +"savior", +"sheep", +"slept", +"social", +"sport", +"tuck", +"utter", +"valley", +"wolf", +"aim", +"alas", +"alter", +"arrow", +"awaken", +"beaten", +"belief", +"brand", +"ceiling", +"cheese", +"clue", +"confidence", +"connection", +"daily", +"disguise", +"eager", +"erase", +"essence", +"everytime", +"expression", +"fan", +"flag", +"flirt", +"foul", +"fur", +"giggle", +"glorious", +"ignorance", +"law", +"lifeless", +"measure", +"mighty", +"muse", +"north", +"opposite", +"paradise", +"patience", +"patient", +"pencil", +"petal", +"plate", +"ponder", +"possibly", +"practice", +"slice", +"spell", +"stock", +"strife", +"strip", +"suffocate", +"suit", +"tender", +"tool", +"trade", +"velvet", +"verse", +"waist", +"witch", +"aunt", +"bench", +"bold", +"cap", +"certainly", +"click", +"companion", +"creator", +"dart", +"delicate", +"determine", +"dish", +"dragon", +"drama", +"drum", +"dude", +"everybody", +"feast", +"forehead", +"former", +"fright", +"fully", +"gas", +"hook", +"hurl", +"invite", +"juice", +"manage", +"moral", +"possess", +"raw", +"rebel", +"royal", +"scale", +"scary", +"several", +"slight", +"stubborn", +"swell", +"talent", +"tea", +"terrible", +"thread", +"torment", +"trickle", +"usually", +"vast", +"violence", +"weave", +"acid", +"agony", +"ashamed", +"awe", +"belly", +"blend", +"blush", +"character", +"cheat", +"common", +"company", +"coward", +"creak", +"danger", +"deadly", +"defense", +"define", +"depend", +"desperate", +"destination", +"dew", +"duck", +"dusty", +"embarrass", +"engine", +"example", +"explore", +"foe", +"freely", +"frustrate", +"generation", +"glove", +"guilty", +"health", +"hurry", +"idiot", +"impossible", +"inhale", +"jaw", +"kingdom", +"mention", +"mist", +"moan", +"mumble", +"mutter", +"observe", +"ode", +"pathetic", +"pattern", +"pie", +"prefer", +"puff", +"rape", +"rare", +"revenge", +"rude", +"scrape", +"spiral", +"squeeze", +"strain", +"sunset", +"suspend", +"sympathy", +"thigh", +"throne", +"total", +"unseen", +"weapon", +"weary" +] + + + +n = 1626 + +# Note about US patent no 5892470: Here each word does not represent a given digit. +# Instead, the digit represented by a word is variable, it depends on the previous word. + +def mn_encode( message ): + assert len(message) % 8 == 0 + out = [] + for i in range(len(message)/8): + word = message[8*i:8*i+8] + x = int(word, 16) + w1 = (x%n) + w2 = ((x/n) + w1)%n + w3 = ((x/n/n) + w2)%n + out += [ words[w1], words[w2], words[w3] ] + return out + +def mn_decode( wlist ): + out = '' + for i in range(len(wlist)/3): + word1, word2, word3 = wlist[3*i:3*i+3] + w1 = words.index(word1) + w2 = (words.index(word2))%n + w3 = (words.index(word3))%n + x = w1 +n*((w2-w1)%n) +n*n*((w3-w2)%n) + out += '%08x'%x + return out + + +if __name__ == '__main__': + import sys + if len( sys.argv ) == 1: + print 'I need arguments: a hex string to encode, or a list of words to decode' + elif len( sys.argv ) == 2: + print ' '.join(mn_encode(sys.argv[1])) + else: + print mn_decode(sys.argv[1:]) diff --git a/lib/slowaes.py b/lib/slowaes.py new file mode 100644 index 00000000..8a621151 --- /dev/null +++ b/lib/slowaes.py @@ -0,0 +1,658 @@ +#!/usr/bin/python +# +# aes.py: implements AES - Advanced Encryption Standard +# from the SlowAES project, http://code.google.com/p/slowaes/ +# +# Copyright (c) 2008 Josh Davis ( http://www.josh-davis.org ), +# Alex Martelli ( http://www.aleax.it ) +# +# Ported from C code written by Laurent Haan ( http://www.progressive-coding.com ) +# +# Licensed under the Apache License, Version 2.0 +# http://www.apache.org/licenses/ +# +import os +import sys +import math + +def append_PKCS7_padding(s): + """return s padded to a multiple of 16-bytes by PKCS7 padding""" + numpads = 16 - (len(s)%16) + return s + numpads*chr(numpads) + +def strip_PKCS7_padding(s): + """return s stripped of PKCS7 padding""" + if len(s)%16 or not s: + raise ValueError("String of len %d can't be PCKS7-padded" % len(s)) + numpads = ord(s[-1]) + if numpads > 16: + raise ValueError("String ending with %r can't be PCKS7-padded" % s[-1]) + if not all(numpads == x for x in map(ord,s[-numpads:-1])): + raise ValueError("Invalid PKCS7 padding") + return s[:-numpads] + +class AES(object): + # valid key sizes + keySize = dict(SIZE_128=16, SIZE_192=24, SIZE_256=32) + + # Rijndael S-box + sbox = [0x63, 0x7c, 0x77, 0x7b, 0xf2, 0x6b, 0x6f, 0xc5, 0x30, 0x01, 0x67, + 0x2b, 0xfe, 0xd7, 0xab, 0x76, 0xca, 0x82, 0xc9, 0x7d, 0xfa, 0x59, + 0x47, 0xf0, 0xad, 0xd4, 0xa2, 0xaf, 0x9c, 0xa4, 0x72, 0xc0, 0xb7, + 0xfd, 0x93, 0x26, 0x36, 0x3f, 0xf7, 0xcc, 0x34, 0xa5, 0xe5, 0xf1, + 0x71, 0xd8, 0x31, 0x15, 0x04, 0xc7, 0x23, 0xc3, 0x18, 0x96, 0x05, + 0x9a, 0x07, 0x12, 0x80, 0xe2, 0xeb, 0x27, 0xb2, 0x75, 0x09, 0x83, + 0x2c, 0x1a, 0x1b, 0x6e, 0x5a, 0xa0, 0x52, 0x3b, 0xd6, 0xb3, 0x29, + 0xe3, 0x2f, 0x84, 0x53, 0xd1, 0x00, 0xed, 0x20, 0xfc, 0xb1, 0x5b, + 0x6a, 0xcb, 0xbe, 0x39, 0x4a, 0x4c, 0x58, 0xcf, 0xd0, 0xef, 0xaa, + 0xfb, 0x43, 0x4d, 0x33, 0x85, 0x45, 0xf9, 0x02, 0x7f, 0x50, 0x3c, + 0x9f, 0xa8, 0x51, 0xa3, 0x40, 0x8f, 0x92, 0x9d, 0x38, 0xf5, 0xbc, + 0xb6, 0xda, 0x21, 0x10, 0xff, 0xf3, 0xd2, 0xcd, 0x0c, 0x13, 0xec, + 0x5f, 0x97, 0x44, 0x17, 0xc4, 0xa7, 0x7e, 0x3d, 0x64, 0x5d, 0x19, + 0x73, 0x60, 0x81, 0x4f, 0xdc, 0x22, 0x2a, 0x90, 0x88, 0x46, 0xee, + 0xb8, 0x14, 0xde, 0x5e, 0x0b, 0xdb, 0xe0, 0x32, 0x3a, 0x0a, 0x49, + 0x06, 0x24, 0x5c, 0xc2, 0xd3, 0xac, 0x62, 0x91, 0x95, 0xe4, 0x79, + 0xe7, 0xc8, 0x37, 0x6d, 0x8d, 0xd5, 0x4e, 0xa9, 0x6c, 0x56, 0xf4, + 0xea, 0x65, 0x7a, 0xae, 0x08, 0xba, 0x78, 0x25, 0x2e, 0x1c, 0xa6, + 0xb4, 0xc6, 0xe8, 0xdd, 0x74, 0x1f, 0x4b, 0xbd, 0x8b, 0x8a, 0x70, + 0x3e, 0xb5, 0x66, 0x48, 0x03, 0xf6, 0x0e, 0x61, 0x35, 0x57, 0xb9, + 0x86, 0xc1, 0x1d, 0x9e, 0xe1, 0xf8, 0x98, 0x11, 0x69, 0xd9, 0x8e, + 0x94, 0x9b, 0x1e, 0x87, 0xe9, 0xce, 0x55, 0x28, 0xdf, 0x8c, 0xa1, + 0x89, 0x0d, 0xbf, 0xe6, 0x42, 0x68, 0x41, 0x99, 0x2d, 0x0f, 0xb0, + 0x54, 0xbb, 0x16] + + # Rijndael Inverted S-box + rsbox = [0x52, 0x09, 0x6a, 0xd5, 0x30, 0x36, 0xa5, 0x38, 0xbf, 0x40, 0xa3, + 0x9e, 0x81, 0xf3, 0xd7, 0xfb , 0x7c, 0xe3, 0x39, 0x82, 0x9b, 0x2f, + 0xff, 0x87, 0x34, 0x8e, 0x43, 0x44, 0xc4, 0xde, 0xe9, 0xcb , 0x54, + 0x7b, 0x94, 0x32, 0xa6, 0xc2, 0x23, 0x3d, 0xee, 0x4c, 0x95, 0x0b, + 0x42, 0xfa, 0xc3, 0x4e , 0x08, 0x2e, 0xa1, 0x66, 0x28, 0xd9, 0x24, + 0xb2, 0x76, 0x5b, 0xa2, 0x49, 0x6d, 0x8b, 0xd1, 0x25 , 0x72, 0xf8, + 0xf6, 0x64, 0x86, 0x68, 0x98, 0x16, 0xd4, 0xa4, 0x5c, 0xcc, 0x5d, + 0x65, 0xb6, 0x92 , 0x6c, 0x70, 0x48, 0x50, 0xfd, 0xed, 0xb9, 0xda, + 0x5e, 0x15, 0x46, 0x57, 0xa7, 0x8d, 0x9d, 0x84 , 0x90, 0xd8, 0xab, + 0x00, 0x8c, 0xbc, 0xd3, 0x0a, 0xf7, 0xe4, 0x58, 0x05, 0xb8, 0xb3, + 0x45, 0x06 , 0xd0, 0x2c, 0x1e, 0x8f, 0xca, 0x3f, 0x0f, 0x02, 0xc1, + 0xaf, 0xbd, 0x03, 0x01, 0x13, 0x8a, 0x6b , 0x3a, 0x91, 0x11, 0x41, + 0x4f, 0x67, 0xdc, 0xea, 0x97, 0xf2, 0xcf, 0xce, 0xf0, 0xb4, 0xe6, + 0x73 , 0x96, 0xac, 0x74, 0x22, 0xe7, 0xad, 0x35, 0x85, 0xe2, 0xf9, + 0x37, 0xe8, 0x1c, 0x75, 0xdf, 0x6e , 0x47, 0xf1, 0x1a, 0x71, 0x1d, + 0x29, 0xc5, 0x89, 0x6f, 0xb7, 0x62, 0x0e, 0xaa, 0x18, 0xbe, 0x1b , + 0xfc, 0x56, 0x3e, 0x4b, 0xc6, 0xd2, 0x79, 0x20, 0x9a, 0xdb, 0xc0, + 0xfe, 0x78, 0xcd, 0x5a, 0xf4 , 0x1f, 0xdd, 0xa8, 0x33, 0x88, 0x07, + 0xc7, 0x31, 0xb1, 0x12, 0x10, 0x59, 0x27, 0x80, 0xec, 0x5f , 0x60, + 0x51, 0x7f, 0xa9, 0x19, 0xb5, 0x4a, 0x0d, 0x2d, 0xe5, 0x7a, 0x9f, + 0x93, 0xc9, 0x9c, 0xef , 0xa0, 0xe0, 0x3b, 0x4d, 0xae, 0x2a, 0xf5, + 0xb0, 0xc8, 0xeb, 0xbb, 0x3c, 0x83, 0x53, 0x99, 0x61 , 0x17, 0x2b, + 0x04, 0x7e, 0xba, 0x77, 0xd6, 0x26, 0xe1, 0x69, 0x14, 0x63, 0x55, + 0x21, 0x0c, 0x7d] + + def getSBoxValue(self,num): + """Retrieves a given S-Box Value""" + return self.sbox[num] + + def getSBoxInvert(self,num): + """Retrieves a given Inverted S-Box Value""" + return self.rsbox[num] + + def rotate(self, word): + """ Rijndael's key schedule rotate operation. + + Rotate a word eight bits to the left: eg, rotate(1d2c3a4f) == 2c3a4f1d + Word is an char list of size 4 (32 bits overall). + """ + return word[1:] + word[:1] + + # Rijndael Rcon + Rcon = [0x8d, 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, + 0x6c, 0xd8, 0xab, 0x4d, 0x9a, 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, + 0x35, 0x6a, 0xd4, 0xb3, 0x7d, 0xfa, 0xef, 0xc5, 0x91, 0x39, 0x72, + 0xe4, 0xd3, 0xbd, 0x61, 0xc2, 0x9f, 0x25, 0x4a, 0x94, 0x33, 0x66, + 0xcc, 0x83, 0x1d, 0x3a, 0x74, 0xe8, 0xcb, 0x8d, 0x01, 0x02, 0x04, + 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, 0x6c, 0xd8, 0xab, 0x4d, + 0x9a, 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, 0x35, 0x6a, 0xd4, 0xb3, + 0x7d, 0xfa, 0xef, 0xc5, 0x91, 0x39, 0x72, 0xe4, 0xd3, 0xbd, 0x61, + 0xc2, 0x9f, 0x25, 0x4a, 0x94, 0x33, 0x66, 0xcc, 0x83, 0x1d, 0x3a, + 0x74, 0xe8, 0xcb, 0x8d, 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, + 0x80, 0x1b, 0x36, 0x6c, 0xd8, 0xab, 0x4d, 0x9a, 0x2f, 0x5e, 0xbc, + 0x63, 0xc6, 0x97, 0x35, 0x6a, 0xd4, 0xb3, 0x7d, 0xfa, 0xef, 0xc5, + 0x91, 0x39, 0x72, 0xe4, 0xd3, 0xbd, 0x61, 0xc2, 0x9f, 0x25, 0x4a, + 0x94, 0x33, 0x66, 0xcc, 0x83, 0x1d, 0x3a, 0x74, 0xe8, 0xcb, 0x8d, + 0x01, 0x02, 0x04, 0x08, 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, 0x6c, + 0xd8, 0xab, 0x4d, 0x9a, 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, 0x35, + 0x6a, 0xd4, 0xb3, 0x7d, 0xfa, 0xef, 0xc5, 0x91, 0x39, 0x72, 0xe4, + 0xd3, 0xbd, 0x61, 0xc2, 0x9f, 0x25, 0x4a, 0x94, 0x33, 0x66, 0xcc, + 0x83, 0x1d, 0x3a, 0x74, 0xe8, 0xcb, 0x8d, 0x01, 0x02, 0x04, 0x08, + 0x10, 0x20, 0x40, 0x80, 0x1b, 0x36, 0x6c, 0xd8, 0xab, 0x4d, 0x9a, + 0x2f, 0x5e, 0xbc, 0x63, 0xc6, 0x97, 0x35, 0x6a, 0xd4, 0xb3, 0x7d, + 0xfa, 0xef, 0xc5, 0x91, 0x39, 0x72, 0xe4, 0xd3, 0xbd, 0x61, 0xc2, + 0x9f, 0x25, 0x4a, 0x94, 0x33, 0x66, 0xcc, 0x83, 0x1d, 0x3a, 0x74, + 0xe8, 0xcb ] + + def getRconValue(self, num): + """Retrieves a given Rcon Value""" + return self.Rcon[num] + + def core(self, word, iteration): + """Key schedule core.""" + # rotate the 32-bit word 8 bits to the left + word = self.rotate(word) + # apply S-Box substitution on all 4 parts of the 32-bit word + for i in range(4): + word[i] = self.getSBoxValue(word[i]) + # XOR the output of the rcon operation with i to the first part + # (leftmost) only + word[0] = word[0] ^ self.getRconValue(iteration) + return word + + def expandKey(self, key, size, expandedKeySize): + """Rijndael's key expansion. + + Expands an 128,192,256 key into an 176,208,240 bytes key + + expandedKey is a char list of large enough size, + key is the non-expanded key. + """ + # current expanded keySize, in bytes + currentSize = 0 + rconIteration = 1 + expandedKey = [0] * expandedKeySize + + # set the 16, 24, 32 bytes of the expanded key to the input key + for j in range(size): + expandedKey[j] = key[j] + currentSize += size + + while currentSize < expandedKeySize: + # assign the previous 4 bytes to the temporary value t + t = expandedKey[currentSize-4:currentSize] + + # every 16,24,32 bytes we apply the core schedule to t + # and increment rconIteration afterwards + if currentSize % size == 0: + t = self.core(t, rconIteration) + rconIteration += 1 + # For 256-bit keys, we add an extra sbox to the calculation + if size == self.keySize["SIZE_256"] and ((currentSize % size) == 16): + for l in range(4): t[l] = self.getSBoxValue(t[l]) + + # We XOR t with the four-byte block 16,24,32 bytes before the new + # expanded key. This becomes the next four bytes in the expanded + # key. + for m in range(4): + expandedKey[currentSize] = expandedKey[currentSize - size] ^ \ + t[m] + currentSize += 1 + + return expandedKey + + def addRoundKey(self, state, roundKey): + """Adds (XORs) the round key to the state.""" + for i in range(16): + state[i] ^= roundKey[i] + return state + + def createRoundKey(self, expandedKey, roundKeyPointer): + """Create a round key. + Creates a round key from the given expanded key and the + position within the expanded key. + """ + roundKey = [0] * 16 + for i in range(4): + for j in range(4): + roundKey[j*4+i] = expandedKey[roundKeyPointer + i*4 + j] + return roundKey + + def galois_multiplication(self, a, b): + """Galois multiplication of 8 bit characters a and b.""" + p = 0 + for counter in range(8): + if b & 1: p ^= a + hi_bit_set = a & 0x80 + a <<= 1 + # keep a 8 bit + a &= 0xFF + if hi_bit_set: + a ^= 0x1b + b >>= 1 + return p + + # + # substitute all the values from the state with the value in the SBox + # using the state value as index for the SBox + # + def subBytes(self, state, isInv): + if isInv: getter = self.getSBoxInvert + else: getter = self.getSBoxValue + for i in range(16): state[i] = getter(state[i]) + return state + + # iterate over the 4 rows and call shiftRow() with that row + def shiftRows(self, state, isInv): + for i in range(4): + state = self.shiftRow(state, i*4, i, isInv) + return state + + # each iteration shifts the row to the left by 1 + def shiftRow(self, state, statePointer, nbr, isInv): + for i in range(nbr): + if isInv: + state[statePointer:statePointer+4] = \ + state[statePointer+3:statePointer+4] + \ + state[statePointer:statePointer+3] + else: + state[statePointer:statePointer+4] = \ + state[statePointer+1:statePointer+4] + \ + state[statePointer:statePointer+1] + return state + + # galois multiplication of the 4x4 matrix + def mixColumns(self, state, isInv): + # iterate over the 4 columns + for i in range(4): + # construct one column by slicing over the 4 rows + column = state[i:i+16:4] + # apply the mixColumn on one column + column = self.mixColumn(column, isInv) + # put the values back into the state + state[i:i+16:4] = column + + return state + + # galois multiplication of 1 column of the 4x4 matrix + def mixColumn(self, column, isInv): + if isInv: mult = [14, 9, 13, 11] + else: mult = [2, 1, 1, 3] + cpy = list(column) + g = self.galois_multiplication + + column[0] = g(cpy[0], mult[0]) ^ g(cpy[3], mult[1]) ^ \ + g(cpy[2], mult[2]) ^ g(cpy[1], mult[3]) + column[1] = g(cpy[1], mult[0]) ^ g(cpy[0], mult[1]) ^ \ + g(cpy[3], mult[2]) ^ g(cpy[2], mult[3]) + column[2] = g(cpy[2], mult[0]) ^ g(cpy[1], mult[1]) ^ \ + g(cpy[0], mult[2]) ^ g(cpy[3], mult[3]) + column[3] = g(cpy[3], mult[0]) ^ g(cpy[2], mult[1]) ^ \ + g(cpy[1], mult[2]) ^ g(cpy[0], mult[3]) + return column + + # applies the 4 operations of the forward round in sequence + def aes_round(self, state, roundKey): + state = self.subBytes(state, False) + state = self.shiftRows(state, False) + state = self.mixColumns(state, False) + state = self.addRoundKey(state, roundKey) + return state + + # applies the 4 operations of the inverse round in sequence + def aes_invRound(self, state, roundKey): + state = self.shiftRows(state, True) + state = self.subBytes(state, True) + state = self.addRoundKey(state, roundKey) + state = self.mixColumns(state, True) + return state + + # Perform the initial operations, the standard round, and the final + # operations of the forward aes, creating a round key for each round + def aes_main(self, state, expandedKey, nbrRounds): + state = self.addRoundKey(state, self.createRoundKey(expandedKey, 0)) + i = 1 + while i < nbrRounds: + state = self.aes_round(state, + self.createRoundKey(expandedKey, 16*i)) + i += 1 + state = self.subBytes(state, False) + state = self.shiftRows(state, False) + state = self.addRoundKey(state, + self.createRoundKey(expandedKey, 16*nbrRounds)) + return state + + # Perform the initial operations, the standard round, and the final + # operations of the inverse aes, creating a round key for each round + def aes_invMain(self, state, expandedKey, nbrRounds): + state = self.addRoundKey(state, + self.createRoundKey(expandedKey, 16*nbrRounds)) + i = nbrRounds - 1 + while i > 0: + state = self.aes_invRound(state, + self.createRoundKey(expandedKey, 16*i)) + i -= 1 + state = self.shiftRows(state, True) + state = self.subBytes(state, True) + state = self.addRoundKey(state, self.createRoundKey(expandedKey, 0)) + return state + + # encrypts a 128 bit input block against the given key of size specified + def encrypt(self, iput, key, size): + output = [0] * 16 + # the number of rounds + nbrRounds = 0 + # the 128 bit block to encode + block = [0] * 16 + # set the number of rounds + if size == self.keySize["SIZE_128"]: nbrRounds = 10 + elif size == self.keySize["SIZE_192"]: nbrRounds = 12 + elif size == self.keySize["SIZE_256"]: nbrRounds = 14 + else: return None + + # the expanded keySize + expandedKeySize = 16*(nbrRounds+1) + + # Set the block values, for the block: + # a0,0 a0,1 a0,2 a0,3 + # a1,0 a1,1 a1,2 a1,3 + # a2,0 a2,1 a2,2 a2,3 + # a3,0 a3,1 a3,2 a3,3 + # the mapping order is a0,0 a1,0 a2,0 a3,0 a0,1 a1,1 ... a2,3 a3,3 + # + # iterate over the columns + for i in range(4): + # iterate over the rows + for j in range(4): + block[(i+(j*4))] = iput[(i*4)+j] + + # expand the key into an 176, 208, 240 bytes key + # the expanded key + expandedKey = self.expandKey(key, size, expandedKeySize) + + # encrypt the block using the expandedKey + block = self.aes_main(block, expandedKey, nbrRounds) + + # unmap the block again into the output + for k in range(4): + # iterate over the rows + for l in range(4): + output[(k*4)+l] = block[(k+(l*4))] + return output + + # decrypts a 128 bit input block against the given key of size specified + def decrypt(self, iput, key, size): + output = [0] * 16 + # the number of rounds + nbrRounds = 0 + # the 128 bit block to decode + block = [0] * 16 + # set the number of rounds + if size == self.keySize["SIZE_128"]: nbrRounds = 10 + elif size == self.keySize["SIZE_192"]: nbrRounds = 12 + elif size == self.keySize["SIZE_256"]: nbrRounds = 14 + else: return None + + # the expanded keySize + expandedKeySize = 16*(nbrRounds+1) + + # Set the block values, for the block: + # a0,0 a0,1 a0,2 a0,3 + # a1,0 a1,1 a1,2 a1,3 + # a2,0 a2,1 a2,2 a2,3 + # a3,0 a3,1 a3,2 a3,3 + # the mapping order is a0,0 a1,0 a2,0 a3,0 a0,1 a1,1 ... a2,3 a3,3 + + # iterate over the columns + for i in range(4): + # iterate over the rows + for j in range(4): + block[(i+(j*4))] = iput[(i*4)+j] + # expand the key into an 176, 208, 240 bytes key + expandedKey = self.expandKey(key, size, expandedKeySize) + # decrypt the block using the expandedKey + block = self.aes_invMain(block, expandedKey, nbrRounds) + # unmap the block again into the output + for k in range(4): + # iterate over the rows + for l in range(4): + output[(k*4)+l] = block[(k+(l*4))] + return output + + +class AESModeOfOperation(object): + + aes = AES() + + # structure of supported modes of operation + modeOfOperation = dict(OFB=0, CFB=1, CBC=2) + + # converts a 16 character string into a number array + def convertString(self, string, start, end, mode): + if end - start > 16: end = start + 16 + if mode == self.modeOfOperation["CBC"]: ar = [0] * 16 + else: ar = [] + + i = start + j = 0 + while len(ar) < end - start: + ar.append(0) + while i < end: + ar[j] = ord(string[i]) + j += 1 + i += 1 + return ar + + # Mode of Operation Encryption + # stringIn - Input String + # mode - mode of type modeOfOperation + # hexKey - a hex key of the bit length size + # size - the bit length of the key + # hexIV - the 128 bit hex Initilization Vector + def encrypt(self, stringIn, mode, key, size, IV): + if len(key) % size: + return None + if len(IV) % 16: + return None + # the AES input/output + plaintext = [] + iput = [0] * 16 + output = [] + ciphertext = [0] * 16 + # the output cipher string + cipherOut = [] + # char firstRound + firstRound = True + if stringIn != None: + for j in range(int(math.ceil(float(len(stringIn))/16))): + start = j*16 + end = j*16+16 + if end > len(stringIn): + end = len(stringIn) + plaintext = self.convertString(stringIn, start, end, mode) + # print 'PT@%s:%s' % (j, plaintext) + if mode == self.modeOfOperation["CFB"]: + if firstRound: + output = self.aes.encrypt(IV, key, size) + firstRound = False + else: + output = self.aes.encrypt(iput, key, size) + for i in range(16): + if len(plaintext)-1 < i: + ciphertext[i] = 0 ^ output[i] + elif len(output)-1 < i: + ciphertext[i] = plaintext[i] ^ 0 + elif len(plaintext)-1 < i and len(output) < i: + ciphertext[i] = 0 ^ 0 + else: + ciphertext[i] = plaintext[i] ^ output[i] + for k in range(end-start): + cipherOut.append(ciphertext[k]) + iput = ciphertext + elif mode == self.modeOfOperation["OFB"]: + if firstRound: + output = self.aes.encrypt(IV, key, size) + firstRound = False + else: + output = self.aes.encrypt(iput, key, size) + for i in range(16): + if len(plaintext)-1 < i: + ciphertext[i] = 0 ^ output[i] + elif len(output)-1 < i: + ciphertext[i] = plaintext[i] ^ 0 + elif len(plaintext)-1 < i and len(output) < i: + ciphertext[i] = 0 ^ 0 + else: + ciphertext[i] = plaintext[i] ^ output[i] + for k in range(end-start): + cipherOut.append(ciphertext[k]) + iput = output + elif mode == self.modeOfOperation["CBC"]: + for i in range(16): + if firstRound: + iput[i] = plaintext[i] ^ IV[i] + else: + iput[i] = plaintext[i] ^ ciphertext[i] + # print 'IP@%s:%s' % (j, iput) + firstRound = False + ciphertext = self.aes.encrypt(iput, key, size) + # always 16 bytes because of the padding for CBC + for k in range(16): + cipherOut.append(ciphertext[k]) + return mode, len(stringIn), cipherOut + + # Mode of Operation Decryption + # cipherIn - Encrypted String + # originalsize - The unencrypted string length - required for CBC + # mode - mode of type modeOfOperation + # key - a number array of the bit length size + # size - the bit length of the key + # IV - the 128 bit number array Initilization Vector + def decrypt(self, cipherIn, originalsize, mode, key, size, IV): + # cipherIn = unescCtrlChars(cipherIn) + if len(key) % size: + return None + if len(IV) % 16: + return None + # the AES input/output + ciphertext = [] + iput = [] + output = [] + plaintext = [0] * 16 + # the output plain text string + stringOut = '' + # char firstRound + firstRound = True + if cipherIn != None: + for j in range(int(math.ceil(float(len(cipherIn))/16))): + start = j*16 + end = j*16+16 + if j*16+16 > len(cipherIn): + end = len(cipherIn) + ciphertext = cipherIn[start:end] + if mode == self.modeOfOperation["CFB"]: + if firstRound: + output = self.aes.encrypt(IV, key, size) + firstRound = False + else: + output = self.aes.encrypt(iput, key, size) + for i in range(16): + if len(output)-1 < i: + plaintext[i] = 0 ^ ciphertext[i] + elif len(ciphertext)-1 < i: + plaintext[i] = output[i] ^ 0 + elif len(output)-1 < i and len(ciphertext) < i: + plaintext[i] = 0 ^ 0 + else: + plaintext[i] = output[i] ^ ciphertext[i] + for k in range(end-start): + stringOut += chr(plaintext[k]) + iput = ciphertext + elif mode == self.modeOfOperation["OFB"]: + if firstRound: + output = self.aes.encrypt(IV, key, size) + firstRound = False + else: + output = self.aes.encrypt(iput, key, size) + for i in range(16): + if len(output)-1 < i: + plaintext[i] = 0 ^ ciphertext[i] + elif len(ciphertext)-1 < i: + plaintext[i] = output[i] ^ 0 + elif len(output)-1 < i and len(ciphertext) < i: + plaintext[i] = 0 ^ 0 + else: + plaintext[i] = output[i] ^ ciphertext[i] + for k in range(end-start): + stringOut += chr(plaintext[k]) + iput = output + elif mode == self.modeOfOperation["CBC"]: + output = self.aes.decrypt(ciphertext, key, size) + for i in range(16): + if firstRound: + plaintext[i] = IV[i] ^ output[i] + else: + plaintext[i] = iput[i] ^ output[i] + firstRound = False + if originalsize is not None and originalsize < end: + for k in range(originalsize-start): + stringOut += chr(plaintext[k]) + else: + for k in range(end-start): + stringOut += chr(plaintext[k]) + iput = ciphertext + return stringOut + + +def encryptData(key, data, mode=AESModeOfOperation.modeOfOperation["CBC"]): + """encrypt `data` using `key` + + `key` should be a string of bytes. + + returned cipher is a string of bytes prepended with the initialization + vector. + + """ + key = map(ord, key) + if mode == AESModeOfOperation.modeOfOperation["CBC"]: + data = append_PKCS7_padding(data) + keysize = len(key) + assert keysize in AES.keySize.values(), 'invalid key size: %s' % keysize + # create a new iv using random data + iv = [ord(i) for i in os.urandom(16)] + moo = AESModeOfOperation() + (mode, length, ciph) = moo.encrypt(data, mode, key, keysize, iv) + # With padding, the original length does not need to be known. It's a bad + # idea to store the original message length. + # prepend the iv. + return ''.join(map(chr, iv)) + ''.join(map(chr, ciph)) + +def decryptData(key, data, mode=AESModeOfOperation.modeOfOperation["CBC"]): + """decrypt `data` using `key` + + `key` should be a string of bytes. + + `data` should have the initialization vector prepended as a string of + ordinal values. + + """ + + key = map(ord, key) + keysize = len(key) + assert keysize in AES.keySize.values(), 'invalid key size: %s' % keysize + # iv is first 16 bytes + iv = map(ord, data[:16]) + data = map(ord, data[16:]) + moo = AESModeOfOperation() + decr = moo.decrypt(data, None, mode, key, keysize, iv) + if mode == AESModeOfOperation.modeOfOperation["CBC"]: + decr = strip_PKCS7_padding(decr) + return decr + +def generateRandomKey(keysize): + """Generates a key from random data of length `keysize`. + + The returned key is a string of bytes. + + """ + if keysize not in (16, 24, 32): + emsg = 'Invalid keysize, %s. Should be one of (16, 24, 32).' + raise ValueError, emsg % keysize + return os.urandom(keysize) + +if __name__ == "__main__": + moo = AESModeOfOperation() + cleartext = "This is a test!" + cypherkey = [143,194,34,208,145,203,230,143,177,246,97,206,145,92,255,84] + iv = [103,35,148,239,76,213,47,118,255,222,123,176,106,134,98,92] + mode, orig_len, ciph = moo.encrypt(cleartext, moo.modeOfOperation["CBC"], + cypherkey, moo.aes.keySize["SIZE_128"], iv) + print 'm=%s, ol=%s (%s), ciph=%s' % (mode, orig_len, len(cleartext), ciph) + decr = moo.decrypt(ciph, orig_len, mode, cypherkey, + moo.aes.keySize["SIZE_128"], iv) + print decr diff --git a/lib/socks.py b/lib/socks.py new file mode 100644 index 00000000..bc3138a5 --- /dev/null +++ b/lib/socks.py @@ -0,0 +1,384 @@ +"""SocksiPy - Python SOCKS module. +Version 1.00 + +Copyright 2006 Dan-Haim. All rights reserved. + +Redistribution and use in source and binary forms, with or without modification, +are permitted provided that the following conditions are met: +1. Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. +3. Neither the name of Dan Haim nor the names of his contributors may be used + to endorse or promote products derived from this software without specific + prior written permission. + +THIS SOFTWARE IS PROVIDED BY DAN HAIM "AS IS" AND ANY EXPRESS OR IMPLIED +WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO +EVENT SHALL DAN HAIM OR HIS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT +LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA +OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF +LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT +OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMANGE. + + +This module provides a standard socket-like interface for Python +for tunneling connections through SOCKS proxies. + +""" + +import socket +import struct + +PROXY_TYPE_SOCKS4 = 1 +PROXY_TYPE_SOCKS5 = 2 +PROXY_TYPE_HTTP = 3 + +_defaultproxy = None +_orgsocket = socket.socket + +class ProxyError(Exception): + def __init__(self, value): + self.value = value + def __str__(self): + return repr(self.value) + +class GeneralProxyError(ProxyError): + def __init__(self, value): + self.value = value + def __str__(self): + return repr(self.value) + +class Socks5AuthError(ProxyError): + def __init__(self, value): + self.value = value + def __str__(self): + return repr(self.value) + +class Socks5Error(ProxyError): + def __init__(self, value): + self.value = value + def __str__(self): + return repr(self.value) + +class Socks4Error(ProxyError): + def __init__(self, value): + self.value = value + def __str__(self): + return repr(self.value) + +class HTTPError(ProxyError): + def __init__(self, value): + self.value = value + def __str__(self): + return repr(self.value) + +_generalerrors = ("success", + "invalid data", + "not connected", + "not available", + "bad proxy type", + "bad input") + +_socks5errors = ("succeeded", + "general SOCKS server failure", + "connection not allowed by ruleset", + "Network unreachable", + "Host unreachable", + "Connection refused", + "TTL expired", + "Command not supported", + "Address type not supported", + "Unknown error") + +_socks5autherrors = ("succeeded", + "authentication is required", + "all offered authentication methods were rejected", + "unknown username or invalid password", + "unknown error") + +_socks4errors = ("request granted", + "request rejected or failed", + "request rejected because SOCKS server cannot connect to identd on the client", + "request rejected because the client program and identd report different user-ids", + "unknown error") + +def setdefaultproxy(proxytype=None,addr=None,port=None,rdns=True,username=None,password=None): + """setdefaultproxy(proxytype, addr[, port[, rdns[, username[, password]]]]) + Sets a default proxy which all further socksocket objects will use, + unless explicitly changed. + """ + global _defaultproxy + _defaultproxy = (proxytype,addr,port,rdns,username,password) + +class socksocket(socket.socket): + """socksocket([family[, type[, proto]]]) -> socket object + + Open a SOCKS enabled socket. The parameters are the same as + those of the standard socket init. In order for SOCKS to work, + you must specify family=AF_INET, type=SOCK_STREAM and proto=0. + """ + + def __init__(self, family=socket.AF_INET, type=socket.SOCK_STREAM, proto=0, _sock=None): + _orgsocket.__init__(self,family,type,proto,_sock) + if _defaultproxy != None: + self.__proxy = _defaultproxy + else: + self.__proxy = (None, None, None, None, None, None) + self.__proxysockname = None + self.__proxypeername = None + + def __recvall(self, bytes): + """__recvall(bytes) -> data + Receive EXACTLY the number of bytes requested from the socket. + Blocks until the required number of bytes have been received. + """ + data = "" + while len(data) < bytes: + data = data + self.recv(bytes-len(data)) + return data + + def setproxy(self,proxytype=None,addr=None,port=None,rdns=True,username=None,password=None): + """setproxy(proxytype, addr[, port[, rdns[, username[, password]]]]) + Sets the proxy to be used. + proxytype - The type of the proxy to be used. Three types + are supported: PROXY_TYPE_SOCKS4 (including socks4a), + PROXY_TYPE_SOCKS5 and PROXY_TYPE_HTTP + addr - The address of the server (IP or DNS). + port - The port of the server. Defaults to 1080 for SOCKS + servers and 8080 for HTTP proxy servers. + rdns - Should DNS queries be preformed on the remote side + (rather than the local side). The default is True. + Note: This has no effect with SOCKS4 servers. + username - Username to authenticate with to the server. + The default is no authentication. + password - Password to authenticate with to the server. + Only relevant when username is also provided. + """ + self.__proxy = (proxytype,addr,port,rdns,username,password) + + def __negotiatesocks5(self,destaddr,destport): + """__negotiatesocks5(self,destaddr,destport) + Negotiates a connection through a SOCKS5 server. + """ + # First we'll send the authentication packages we support. + if (self.__proxy[4]!=None) and (self.__proxy[5]!=None): + # The username/password details were supplied to the + # setproxy method so we support the USERNAME/PASSWORD + # authentication (in addition to the standard none). + self.sendall("\x05\x02\x00\x02") + else: + # No username/password were entered, therefore we + # only support connections with no authentication. + self.sendall("\x05\x01\x00") + # We'll receive the server's response to determine which + # method was selected + chosenauth = self.__recvall(2) + if chosenauth[0] != "\x05": + self.close() + raise GeneralProxyError((1,_generalerrors[1])) + # Check the chosen authentication method + if chosenauth[1] == "\x00": + # No authentication is required + pass + elif chosenauth[1] == "\x02": + # Okay, we need to perform a basic username/password + # authentication. + self.sendall("\x01" + chr(len(self.__proxy[4])) + self.__proxy[4] + chr(len(self.proxy[5])) + self.__proxy[5]) + authstat = self.__recvall(2) + if authstat[0] != "\x01": + # Bad response + self.close() + raise GeneralProxyError((1,_generalerrors[1])) + if authstat[1] != "\x00": + # Authentication failed + self.close() + raise Socks5AuthError,((3,_socks5autherrors[3])) + # Authentication succeeded + else: + # Reaching here is always bad + self.close() + if chosenauth[1] == "\xFF": + raise Socks5AuthError((2,_socks5autherrors[2])) + else: + raise GeneralProxyError((1,_generalerrors[1])) + # Now we can request the actual connection + req = "\x05\x01\x00" + # If the given destination address is an IP address, we'll + # use the IPv4 address request even if remote resolving was specified. + try: + ipaddr = socket.inet_aton(destaddr) + req = req + "\x01" + ipaddr + except socket.error: + # Well it's not an IP number, so it's probably a DNS name. + if self.__proxy[3]==True: + # Resolve remotely + ipaddr = None + req = req + "\x03" + chr(len(destaddr)) + destaddr + else: + # Resolve locally + ipaddr = socket.inet_aton(socket.gethostbyname(destaddr)) + req = req + "\x01" + ipaddr + req = req + struct.pack(">H",destport) + self.sendall(req) + # Get the response + resp = self.__recvall(4) + if resp[0] != "\x05": + self.close() + raise GeneralProxyError((1,_generalerrors[1])) + elif resp[1] != "\x00": + # Connection failed + self.close() + raise Socks5Error(_socks5errors[min(9,ord(resp[1]))]) + # Get the bound address/port + elif resp[3] == "\x01": + boundaddr = self.__recvall(4) + elif resp[3] == "\x03": + resp = resp + self.recv(1) + boundaddr = self.__recvall(resp[4]) + else: + self.close() + raise GeneralProxyError((1,_generalerrors[1])) + boundport = struct.unpack(">H",self.__recvall(2))[0] + self.__proxysockname = (boundaddr,boundport) + if ipaddr != None: + self.__proxypeername = (socket.inet_ntoa(ipaddr),destport) + else: + self.__proxypeername = (destaddr,destport) + + def getproxysockname(self): + """getsockname() -> address info + Returns the bound IP address and port number at the proxy. + """ + return self.__proxysockname + + def getproxypeername(self): + """getproxypeername() -> address info + Returns the IP and port number of the proxy. + """ + return _orgsocket.getpeername(self) + + def getpeername(self): + """getpeername() -> address info + Returns the IP address and port number of the destination + machine (note: getproxypeername returns the proxy) + """ + return self.__proxypeername + + def __negotiatesocks4(self,destaddr,destport): + """__negotiatesocks4(self,destaddr,destport) + Negotiates a connection through a SOCKS4 server. + """ + # Check if the destination address provided is an IP address + rmtrslv = False + try: + ipaddr = socket.inet_aton(destaddr) + except socket.error: + # It's a DNS name. Check where it should be resolved. + if self.__proxy[3]==True: + ipaddr = "\x00\x00\x00\x01" + rmtrslv = True + else: + ipaddr = socket.inet_aton(socket.gethostbyname(destaddr)) + # Construct the request packet + req = "\x04\x01" + struct.pack(">H",destport) + ipaddr + # The username parameter is considered userid for SOCKS4 + if self.__proxy[4] != None: + req = req + self.__proxy[4] + req = req + "\x00" + # DNS name if remote resolving is required + # NOTE: This is actually an extension to the SOCKS4 protocol + # called SOCKS4A and may not be supported in all cases. + if rmtrslv==True: + req = req + destaddr + "\x00" + self.sendall(req) + # Get the response from the server + resp = self.__recvall(8) + if resp[0] != "\x00": + # Bad data + self.close() + raise GeneralProxyError((1,_generalerrors[1])) + if resp[1] != "\x5A": + # Server returned an error + self.close() + if ord(resp[1]) in (91,92,93): + self.close() + raise Socks4Error((ord(resp[1]),_socks4errors[ord(resp[1])-90])) + else: + raise Socks4Error((94,_socks4errors[4])) + # Get the bound address/port + self.__proxysockname = (socket.inet_ntoa(resp[4:]),struct.unpack(">H",resp[2:4])[0]) + if rmtrslv != None: + self.__proxypeername = (socket.inet_ntoa(ipaddr),destport) + else: + self.__proxypeername = (destaddr,destport) + + def __negotiatehttp(self,destaddr,destport): + """__negotiatehttp(self,destaddr,destport) + Negotiates a connection through an HTTP server. + """ + # If we need to resolve locally, we do this now + if self.__proxy[3] == False: + addr = socket.gethostbyname(destaddr) + else: + addr = destaddr + self.sendall("CONNECT " + addr + ":" + str(destport) + " HTTP/1.1\r\n" + "Host: " + destaddr + "\r\n\r\n") + # We read the response until we get the string "\r\n\r\n" + resp = self.recv(1) + while resp.find("\r\n\r\n")==-1: + resp = resp + self.recv(1) + # We just need the first line to check if the connection + # was successful + statusline = resp.splitlines()[0].split(" ",2) + if statusline[0] not in ("HTTP/1.0","HTTP/1.1"): + self.close() + raise GeneralProxyError((1,_generalerrors[1])) + try: + statuscode = int(statusline[1]) + except ValueError: + self.close() + raise GeneralProxyError((1,_generalerrors[1])) + if statuscode != 200: + self.close() + raise HTTPError((statuscode,statusline[2])) + self.__proxysockname = ("0.0.0.0",0) + self.__proxypeername = (addr,destport) + + def connect(self,destpair): + """connect(self,despair) + Connects to the specified destination through a proxy. + destpar - A tuple of the IP/DNS address and the port number. + (identical to socket's connect). + To select the proxy server use setproxy(). + """ + # Do a minimal input check first + if (type(destpair) in (list,tuple)==False) or (len(destpair)<2) or (type(destpair[0])!=str) or (type(destpair[1])!=int): + raise GeneralProxyError((5,_generalerrors[5])) + if self.__proxy[0] == PROXY_TYPE_SOCKS5: + if self.__proxy[2] != None: + portnum = self.__proxy[2] + else: + portnum = 1080 + _orgsocket.connect(self,(self.__proxy[1],portnum)) + self.__negotiatesocks5(destpair[0],destpair[1]) + elif self.__proxy[0] == PROXY_TYPE_SOCKS4: + if self.__proxy[2] != None: + portnum = self.__proxy[2] + else: + portnum = 1080 + _orgsocket.connect(self,(self.__proxy[1],portnum)) + self.__negotiatesocks4(destpair[0],destpair[1]) + elif self.__proxy[0] == PROXY_TYPE_HTTP: + if self.__proxy[2] != None: + portnum = self.__proxy[2] + else: + portnum = 8080 + _orgsocket.connect(self,(self.__proxy[1],portnum)) + self.__negotiatehttp(destpair[0],destpair[1]) + elif self.__proxy[0] == None: + _orgsocket.connect(self,(destpair[0],destpair[1])) + else: + raise GeneralProxyError((4,_generalerrors[4])) diff --git a/lib/taker.py b/lib/taker.py new file mode 100644 index 00000000..c76b1470 --- /dev/null +++ b/lib/taker.py @@ -0,0 +1,411 @@ +#! /usr/bin/env python + +from common import * +import common +import enc_wrapper +import bitcoin as btc + +import sqlite3, base64, threading, time, random, pprint + +class CoinJoinTX(object): + #soon the taker argument will be removed and just be replaced by wallet or some other interface + def __init__(self, msgchan, wallet, db, cj_amount, orders, input_utxos, my_cj_addr, + my_change_addr, my_txfee, finishcallback, choose_orders_recover, auth_addr=None): + ''' + if my_change is None then there wont be a change address + thats used if you want to entirely coinjoin one utxo with no change left over + orders is the orders you want to fill {'counterpartynick': oid, 'cp2': oid2} + ''' + debug('starting cj to ' + my_cj_addr + ' with change at ' + str(my_change_addr)) + #parameters + self.msgchan = msgchan + self.wallet = wallet + self.db = db + self.cj_amount = cj_amount + self.active_orders = dict(orders) + self.input_utxos = input_utxos + self.finishcallback = finishcallback + self.my_txfee = my_txfee + self.my_cj_addr = my_cj_addr + self.my_change_addr = my_change_addr + self.choose_orders_recover = choose_orders_recover + self.auth_addr = auth_addr + self.timeout_lock = threading.Condition() #used to wait() and notify() + #used to restrict access to certain variables across threads + self.timeout_thread_lock = threading.Condition() + self.end_timeout_thread = False + CoinJoinTX.TimeoutThread(self).start() + #state variables + self.txid = None + self.cjfee_total = 0 + self.nonrespondants = list(self.active_orders.keys()) + self.all_responded = False + self.latest_tx = None + self.utxos = {None: self.input_utxos.keys()} #None means they belong to me + self.outputs = [{'address': self.my_cj_addr, 'value': self.cj_amount}] + #create DH keypair on the fly for this Tx object + self.kp = enc_wrapper.init_keypair() + self.crypto_boxes = {} + self.msgchan.fill_orders(self.active_orders, self.cj_amount, self.kp.hex_pk()) + + def start_encryption(self, nick, maker_pk): + if nick not in self.active_orders.keys(): + debug("Counterparty not part of this transaction. Ignoring") + return + self.crypto_boxes[nick] = [maker_pk, enc_wrapper.as_init_encryption(\ + self.kp, enc_wrapper.init_pubkey(maker_pk))] + #send authorisation request + if self.auth_addr: + my_btc_addr = self.auth_addr + else: + my_btc_addr = self.input_utxos.itervalues().next()['address'] + my_btc_priv = self.wallet.get_key_from_addr(my_btc_addr) + my_btc_pub = btc.privtopub(my_btc_priv) + my_btc_sig = btc.ecdsa_sign(self.kp.hex_pk(), my_btc_priv) + self.msgchan.send_auth(nick, my_btc_pub, my_btc_sig) + + def auth_counterparty(self, nick, btc_sig, cj_pub): + '''Validate the counterpartys claim to own the btc + address/pubkey that will be used for coinjoining + with an ecdsa verification.''' + #crypto_boxes[nick][0] = maker_pubkey + if not btc.ecdsa_verify(self.crypto_boxes[nick][0], btc_sig, cj_pub): + debug('signature didnt match pubkey and message') + return False + return True + + def recv_txio(self, nick, utxo_list, cj_pub, change_addr): + if nick not in self.nonrespondants: + debug('recv_txio => nick=' + nick + ' not in nonrespondants ' + str(self.nonrespondants)) + return + self.utxos[nick] = utxo_list + order = self.db.execute('SELECT ordertype, txfee, cjfee FROM ' + 'orderbook WHERE oid=? AND counterparty=?', + (self.active_orders[nick], nick)).fetchone() + utxo_data = common.bc_interface.query_utxo_set(self.utxos[nick]) + if None in utxo_data: + common.debug('ERROR outputs unconfirmed or already spent. utxo_data=' + + pprint.pformat(utxo_data)) + #when internal reviewing of makers is created, add it here to immediately quit + return #ignore this message, eventually the timeout thread will recover + total_input = sum([d['value'] for d in utxo_data]) + real_cjfee = calc_cj_fee(order['ordertype'], order['cjfee'], self.cj_amount) + self.outputs.append({'address': change_addr, 'value': + total_input - self.cj_amount - order['txfee'] + real_cjfee}) + debug('fee breakdown for %s totalin=%d cjamount=%d txfee=%d realcjfee=%d' % (nick, + total_input, self.cj_amount, order['txfee'], real_cjfee)) + cj_addr = btc.pubtoaddr(cj_pub, get_p2pk_vbyte()) + self.outputs.append({'address': cj_addr, 'value': self.cj_amount}) + self.cjfee_total += real_cjfee + self.nonrespondants.remove(nick) + if len(self.nonrespondants) > 0: + debug('nonrespondants = ' + str(self.nonrespondants)) + return + self.all_responded = True + with self.timeout_lock: + self.timeout_lock.notify() + debug('got all parts, enough to build a tx cjfeetotal=' + str(self.cjfee_total)) + self.nonrespondants = list(self.active_orders.keys()) + + my_total_in = 0 + for u, va in self.input_utxos.iteritems(): + my_total_in += va['value'] + #my_total_in = sum([va['value'] for u, va in self.input_utxos.iteritems()]) + + my_change_value = my_total_in - self.cj_amount - self.cjfee_total - self.my_txfee + debug('fee breakdown for me totalin=%d txfee=%d cjfee_total=%d => changevalue=%d' % (my_total_in, + self.my_txfee, self.cjfee_total, my_change_value)) + if self.my_change_addr == None: + if my_change_value != 0 and abs(my_change_value) != 1: + #seems you wont always get exactly zero because of integer rounding + # so 1 satoshi extra or fewer being spent as miner fees is acceptable + debug('WARNING CHANGE NOT BEING USED\nCHANGEVALUE = ' + str(my_change_value)) + else: + self.outputs.append({'address': self.my_change_addr, 'value': my_change_value}) + utxo_tx = [dict([('output', u)]) for u in sum(self.utxos.values(), [])] + random.shuffle(utxo_tx) + random.shuffle(self.outputs) + tx = btc.mktx(utxo_tx, self.outputs) + debug('obtained tx\n' + pprint.pformat(btc.deserialize(tx))) + self.msgchan.send_tx(self.active_orders.keys(), tx) + + self.latest_tx = btc.deserialize(tx) + for index, ins in enumerate(self.latest_tx['ins']): + utxo = ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index']) + if utxo not in self.input_utxos.keys(): + continue + #placeholders required + ins['script'] = 'deadbeef' + + def add_signature(self, nick, sigb64): + if nick not in self.nonrespondants: + debug('add_signature => nick=' + nick + ' not in nonrespondants ' + str(self.nonrespondants)) + return + sig = base64.b64decode(sigb64).encode('hex') + inserted_sig = False + txhex = btc.serialize(self.latest_tx) + + #batch retrieval of utxo data + utxo = {} + ctr = 0 + for index, ins in enumerate(self.latest_tx['ins']): + utxo_for_checking = ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index']) + if ins['script'] != '' or utxo_for_checking in self.input_utxos.keys(): + continue + utxo[ctr] = [index, utxo_for_checking] + ctr += 1 + utxo_data = common.bc_interface.query_utxo_set([x[1] for x in utxo.values()]) + #insert signatures + for i,u in utxo.iteritems(): + if utxo_data[i] == None: + continue + sig_good = btc.verify_tx_input(txhex, u[0], utxo_data[i]['script'], *btc.deserialize_script(sig)) + if sig_good: + debug('found good sig at index=%d' % (u[0])) + self.latest_tx['ins'][u[0]]['script'] = sig + inserted_sig = True + #check if maker has sent everything possible + self.utxos[nick].remove(u[1]) + if len(self.utxos[nick]) == 0: + debug('nick = ' + nick + ' sent all sigs, removing from nonrespondant list') + self.nonrespondants.remove(nick) + break + if not inserted_sig: + debug('signature did not match anything in the tx') + #TODO what if the signature doesnt match anything + # nothing really to do except drop it, carry on and wonder why the + # other guy sent a failed signature + + tx_signed = True + for ins in self.latest_tx['ins']: + if ins['script'] == '': + tx_signed = False + if not tx_signed: + return + self.end_timeout_thread = True + self.all_responded = True + with self.timeout_lock: + self.timeout_lock.notify() + debug('all makers have sent their signatures') + for index, ins in enumerate(self.latest_tx['ins']): + #remove placeholders + if ins['script'] == 'deadbeef': + ins['script'] = '' + if self.finishcallback != None: + self.finishcallback(self) + + def self_sign(self): + #now sign it ourselves + tx = btc.serialize(self.latest_tx) + for index, ins in enumerate(self.latest_tx['ins']): + utxo = ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index']) + if utxo not in self.input_utxos.keys(): + continue + addr = self.input_utxos[utxo]['address'] + tx = btc.sign(tx, index, self.wallet.get_key_from_addr(addr)) + self.latest_tx = btc.deserialize(tx) + + def push(self, txd): + tx = btc.serialize(txd) + debug('\n' + tx) + debug('txid = ' + btc.txhash(tx)) + #TODO send to a random maker or push myself + #TODO need to check whether the other party sent it + #self.msgchan.push_tx(self.active_orders.keys()[0], txhex) + self.txid = common.bc_interface.pushtx(tx) + if self.txid == None: + debug('unable to pushtx') + + def self_sign_and_push(self): + self.self_sign() + self.push(self.latest_tx) + + def recover_from_nonrespondants(self): + debug('nonresponding makers = ' + str(self.nonrespondants)) + #if there is no choose_orders_recover then end and call finishcallback + # so the caller can handle it in their own way, notable for sweeping + # where simply replacing the makers wont work + if not self.choose_orders_recover: + self.end_timeout_thread = True + if self.finishcallback != None: + self.finishcallback(self) + return + + if self.latest_tx == None: + #nonresponding to !fill, recover by finding another maker + debug('nonresponse to !fill') + for nr in self.nonrespondants: + del self.active_orders[nr] + new_orders, new_makers_fee = self.choose_orders_recover(self.cj_amount, + len(self.nonrespondants), self.nonrespondants, self.active_orders.keys()) + for nick, order in new_orders.iteritems(): + self.active_orders[nick] = order + self.nonrespondants = list(new_orders.keys()) + debug('new active_orders = ' + pprint.pformat(self.active_orders) + + '\nnew nonrespondants = ' + pprint.pformat(self.nonrespondants)) + self.msgchan.fill_orders(new_orders, self.cj_amount, self.kp.hex_pk()) + else: + debug('nonresponse to !sig') + #nonresponding to !sig, have to restart tx from the beginning + self.end_timeout_thread = True + if self.finishcallback != None: + self.finishcallback(self) + #finishcallback will check if self.all_responded is True and will know it came from here + + class TimeoutThread(threading.Thread): + def __init__(self, cjtx): + threading.Thread.__init__(self) + self.cjtx = cjtx + + def run(self): + debug('started timeout thread for coinjoin of amount ' + + str(self.cjtx.cj_amount) + ' to addr ' + self.cjtx.my_cj_addr) + + #how the threading to check for nonresponding makers works like this + #there is a Condition object + #in a loop, call cond.wait(timeout) + # after it returns, check a boolean + # to see if if the messages have arrived + while not self.cjtx.end_timeout_thread: + debug('waiting for all replies.. timeout=' + str(common.maker_timeout_sec)) + with self.cjtx.timeout_lock: + self.cjtx.timeout_lock.wait(common.maker_timeout_sec) + if self.cjtx.all_responded: + debug('timeout thread woken by notify(), makers responded in time') + self.cjtx.all_responded = False + else: + debug('timeout thread woken by timeout, makers didnt respond') + with self.cjtx.timeout_thread_lock: + self.cjtx.recover_from_nonrespondants() + +class CoinJoinerPeer(object): + def __init__(self, msgchan): + self.msgchan = msgchan + + def get_crypto_box_from_nick(self, nick): + raise Exception() + + def on_set_topic(self, newtopic): + chunks = newtopic.split('|') + for msg in chunks[1:]: + try: + msg = msg.strip() + params = msg.split(' ') + min_version = int(params[0]) + max_version = int(params[1]) + alert = msg[msg.index(params[1]) + len(params[1]):].strip() + except ValueError, IndexError: + continue + if min_version < common.JM_VERSION and max_version > common.JM_VERSION: + print '=' * 60 + print 'JOINMARKET ALERT' + print alert + print '=' * 60 + common.joinmarket_alert = alert + + +class OrderbookWatch(CoinJoinerPeer): + def __init__(self, msgchan): + CoinJoinerPeer.__init__(self, msgchan) + self.msgchan.register_orderbookwatch_callbacks(self.on_order_seen, + self.on_order_cancel) + self.msgchan.register_channel_callbacks(self.on_welcome, self.on_set_topic, + None, self.on_disconnect, self.on_nick_leave, None) + + con = sqlite3.connect(":memory:", check_same_thread=False) + con.row_factory = sqlite3.Row + self.db = con.cursor() + self.db.execute("CREATE TABLE orderbook(counterparty TEXT, oid INTEGER, ordertype TEXT, " + + "minsize INTEGER, maxsize INTEGER, txfee INTEGER, cjfee TEXT);") + + def on_order_seen(self, counterparty, oid, ordertype, minsize, maxsize, txfee, cjfee): + try: + if int(oid) < 0 or int(oid) > sys.maxint: + debug("Got invalid order ID: " + oid + " from " + counterparty) + return + # delete orders eagerly, so in case a buggy maker sends an invalid offer, + # we won't accidentally !fill based on the ghost of its previous message. + self.db.execute("DELETE FROM orderbook WHERE counterparty=? AND oid=?;", + (counterparty, oid)) + # now validate the remaining fields + if int(minsize) < 0 or int(minsize) > 21*10**14: + debug("Got invalid minsize: " + minsize + " from " + counterparty) + return + if int(maxsize) < 0 or int(maxsize) > 21*10**14: + debug("Got invalid maxsize: " + maxsize + " from " + counterparty) + return + if int(txfee) < 0: + debug("Got invalid txfee: " + txfee + " from " + counterparty) + return + if int(minsize) > int(maxsize): + debug("Got minsize bigger than maxsize: " + minsize + + " - " + maxsize + " from " + counterparty) + return + self.db.execute('INSERT INTO orderbook VALUES(?, ?, ?, ?, ?, ?, ?);', + (counterparty, oid, ordertype, minsize, maxsize, txfee, + str(Decimal(cjfee)))) # any parseable Decimal is a valid cjfee + except InvalidOperation: + debug("Got invalid cjfee: " + cjfee + " from " + counterparty) + except: + debug("Error parsing order " + oid + " from " + counterparty) + + def on_order_cancel(self, counterparty, oid): + self.db.execute("DELETE FROM orderbook WHERE counterparty=? AND oid=?;", + (counterparty, oid)) + + def on_welcome(self): + self.msgchan.request_orderbook() + + def on_nick_leave(self, nick): + self.db.execute('DELETE FROM orderbook WHERE counterparty=?;', (nick,)) + + def on_disconnect(self): + self.db.execute('DELETE FROM orderbook;') + +#assume this only has one open cj tx at a time +class Taker(OrderbookWatch): + def __init__(self, msgchan): + OrderbookWatch.__init__(self, msgchan) + msgchan.register_taker_callbacks(self.on_error, self.on_pubkey, + self.on_ioauth, self.on_sig) + msgchan.cjpeer = self + self.cjtx = None + self.maker_pks = {} + #TODO have a list of maker's nick we're coinjoining with, so + # that some other guy doesnt send you confusing stuff + + def get_crypto_box_from_nick(self, nick): + if nick in self.cjtx.crypto_boxes: + return self.cjtx.crypto_boxes[nick][1] #libsodium encryption object + else: + debug('something wrong, no crypto object, nick=' + nick + ', message will be dropped') + return None + + def start_cj(self, wallet, cj_amount, orders, input_utxos, my_cj_addr, my_change_addr, + my_txfee, finishcallback=None, choose_orders_recover=None, auth_addr=None): + self.cjtx = CoinJoinTX(self.msgchan, wallet, self.db, cj_amount, orders, + input_utxos, my_cj_addr, my_change_addr, my_txfee, finishcallback, + choose_orders_recover, auth_addr) + + def on_error(self): + pass #TODO implement + + def on_pubkey(self, nick, maker_pubkey): + self.cjtx.start_encryption(nick, maker_pubkey) + + def on_ioauth(self, nick, utxo_list, cj_pub, change_addr, btc_sig): + if not self.cjtx.auth_counterparty(nick, btc_sig, cj_pub): + debug('Authenticated encryption with counterparty: ' + nick + \ + ' not established. TODO: send rejection message') + return + with self.cjtx.timeout_thread_lock: + self.cjtx.recv_txio(nick, utxo_list, cj_pub, change_addr) + + def on_sig(self, nick, sig): + with self.cjtx.timeout_thread_lock: + self.cjtx.add_signature(nick, sig) + +if __name__ == "__main__": + main() + print('done') + diff --git a/logs/.gitignore b/logs/.gitignore new file mode 100644 index 00000000..86d0cb27 --- /dev/null +++ b/logs/.gitignore @@ -0,0 +1,4 @@ +# Ignore everything in this directory +* +# Except this file +!.gitignore \ No newline at end of file diff --git a/maker.py b/maker.py deleted file mode 100644 index 98ee11b1..00000000 --- a/maker.py +++ /dev/null @@ -1,228 +0,0 @@ -#! /usr/bin/env python - -from common import * -import irclib -import bitcoin as btc -import sys -import sqlite3 -import base64 - -nickname = 'cj-maker' -seed = btc.sha256('dont use brainwallets except for holding testnet coins') - -class CoinJoinOrder(object): - def __init__(self, irc, nick, oid, amount): - self.oid = oid - self.cj_amount = amount - order = db.execute('SELECT * FROM myorders WHERE oid=?;', (oid,)).fetchone() - if amount <= order['minsize'] or amount >= order['maxsize']: - irc.privmsg(nick, command_prefix + 'error Amount out of range') - #TODO logic for this error causing the order to be removed from list of open orders - self.utxos, self.mixing_depth = oid_to_order(oid, amount) - self.ordertype = order['ordertype'] - self.txfee = order['txfee'] - self.cjfee = order['cjfee'] - self.cj_addr = wallet.get_receive_addr(self.mixing_depth) - self.change_addr = wallet.get_change_addr(self.mixing_depth - 1) - self.b64txparts = [] - #even if the order ends up never being furfilled, you dont want someone - # pretending to fill all your orders to find out which addresses you use - irc.privmsg(nick, command_prefix + 'myparts ' + ','.join(self.utxos) + ' ' + - self.cj_addr + ' ' + self.change_addr) - def recv_tx_part(self, b64txpart): - self.b64txparts.append(b64txpart) - #TODO this is a dos opportunity, flood someone with !txpart - #repeatedly to fill up their memory - - def recv_tx(self, irc, nick, b64txpart): - self.b64txparts.append(b64txpart) - tx = base64.b64decode(''.join(self.b64txparts)).encode('hex') - txd = btc.deserialize(tx) - goodtx, errmsg = self.verify_unsigned_tx(txd) - if not goodtx: - irc.privmsg(nick, command_prefix + 'error ' + errmsg) - return False - sigs = [] - for index, ins in enumerate(txd['ins']): - utxo = ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index']) - if utxo not in wallet.unspent: - continue - addr = wallet.unspent[utxo]['address'] - txs = btc.sign(tx, index, wallet.get_key_from_addr(addr)) - sigs.append(base64.b64encode(btc.deserialize(txs)['ins'][index]['script'].decode('hex'))) - if len(sigs) == 0: - print 'ERROR no private keys found' - - #TODO make this a function in irclib.py - sigline = '' - for sig in sigs: - prev_sigline = sigline - sigline = sigline + command_prefix + 'sig ' + sig - if len(sigline) > MAX_PRIVMSG_LEN: - irc.privmsg(nick, prev_sigline) - sigline = command_prefix + 'sig ' + sig - if len(sigline) > 0: - irc.privmsg(nick, sigline) - return True - - def verify_unsigned_tx(self, txd): - tx_utxos = set([ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index']) for ins in txd['ins']]) - if not tx_utxos.issuperset(set(self.utxos)): - return False, 'my utxos are not contained' - my_total_in = 0 - for u in self.utxos: - usvals = wallet.unspent[u] - my_total_in += int(usvals['value']) - - real_cjfee = calc_cj_fee(self.ordertype, self.cjfee, self.cj_amount) - expected_change_value = (my_total_in - self.cj_amount - - self.txfee + real_cjfee) - debug('earned fee = ' + str(real_cjfee)) - debug('mycjaddr, mychange = ' + self.cj_addr + ', ' + self.change_addr) - - times_seen_cj_addr = 0 - times_seen_change_addr = 0 - for outs in txd['outs']: - addr = btc.script_to_address(outs['script'], get_vbyte()) - if addr == self.cj_addr: - times_seen_cj_addr += 1 - if outs['value'] != self.cj_amount: - return False, 'Wrong cj_amount. I expect ' + str(cj_amount) - if addr == self.change_addr: - times_seen_change_addr += 1 - if outs['value'] != expected_change_value: - return False, 'wrong change, i expect ' + str(expected_change_address) - if times_seen_cj_addr != 1 or times_seen_change_addr != 1: - return False, 'cj or change addr not in tx outputs exactly once' - return True, None - -wallet = Wallet(seed) -active_orders = {} - -#these two functions create_my_orders() and oid_to_uxto() define the -# sell-side pricing algorithm of this bot -def create_my_orders(): - db.execute("CREATE TABLE myorders(oid INTEGER, ordertype TEXT, " - + "minsize INTEGER, maxsize INTEGER, txfee INTEGER, cjfee TEXT);") - - #tells the highest value possible made by combining all utxos - #fee is 0.2% of the cj amount - total_value = 0 - for utxo, addrvalue in wallet.unspent.iteritems(): - total_value += addrvalue['value'] - db.execute('INSERT INTO myorders VALUES(?, ?, ?, ?, ?, ?);', - (0, 'relorder', 0, total_value, 10000, '0.002')) - - ''' - #simple algorithm where each utxo we have becomes an order - oid = 0 - for un in db.execute('SELECT * FROM unspent;').fetchall(): - db.execute('INSERT INTO myorders VALUES(?, ?, ?, ?, ?, ?);', - (oid, 'absorder', 0, un['value'], 10000, '100000')) - oid += 1 - ''' - -def oid_to_order(oid, amount): - unspent = [] - for utxo, addrvalue in wallet.unspent.iteritems(): - unspent.append({'value': addrvalue['value'], 'utxo': utxo}) - inputs = btc.select(unspent, amount) - #TODO this raises an exception if you dont have enough money, id rather it just returned None - mixing_depth = 1 - return [i['utxo'] for i in inputs], mixing_depth - ''' - unspent = db.execute('SELECT * FROM unspent WHERE value > ?;', (amount,)).fetchone() - return [unspent['utxo']] - ''' - -#TODO this belongs in irclib.py -def irc_privmsg_size_throttle(irc, target, lines, prefix=''): - line = '' - for l in lines: - line += l - if len(line) > MAX_PRIVMSG_LEN: - irc.privmsg(target, prefix + line) - line = '' - if len(line) > 0: - irc.privmsg(target, prefix + line) - -def privmsg_all_orders(irc, target): - orderdb_keys = ['ordertype', 'oid', 'minsize', 'maxsize', 'txfee', 'cjfee'] - orderline = '' - for order in db.execute('SELECT * FROM myorders;').fetchall(): - elem_list = [str(order[k]) for k in orderdb_keys] - orderline += (command_prefix + ' '.join(elem_list)) - if len(orderline) > MAX_PRIVMSG_LEN: - irc.privmsg(target, orderline) - orderline = '' - if len(orderline) > 0: - irc.privmsg(target, orderline) - -def on_welcome(irc): - privmsg_all_orders(irc, channel) - -def on_privmsg(irc, nick, message): - #debug("privmsg nick=%s message=%s" % (nick, message)) - if message[0] != command_prefix: - return - command_lines = message.split(command_prefix) - for command_line in command_lines: - chunks = command_line.split(" ") - if chunks[0] == 'fill': - oid = chunks[1] - amount = int(chunks[2]) - active_orders[nick] = CoinJoinOrder(irc, nick, oid, amount) - elif chunks[0] == 'txpart': - b64txpart = chunks[1] #TODO check nick appears in active_orders - active_orders[nick].recv_tx_part(b64txpart) - elif chunks[0] == 'tx': - b64txpart = chunks[1] - active_orders[nick].recv_tx(irc, nick, b64txpart) - - -#each order has an id for referencing to and looking up -# using the same id again overwrites it, they'll be plenty of times when an order -# has to be modified and its better to just have !order rather than !cancelorder then !order -def on_pubmsg(irc, nick, message): - #debug("pubmsg nick=%s message=%s" % (nick, message)) - if message[0] == command_prefix: - chunks = message[1:].split(" ") - if chunks[0] == '%quit' or chunks[0] == '%makerquit': - irc.shutdown() - elif chunks[0] == '%say': #% is a way to remind me its a testing cmd - irc.pubmsg(message[6:]) - elif chunks[0] == '%rm': - irc.pubmsg('!cancel ' + chunks[1]) - elif chunks[0] == 'orderbook': - privmsg_all_orders(irc, nick) - -def on_set_topic(irc, newtopic): - chunks = newtopic.split('|') - try: - print chunks[1] - print chunks[3] - except IndexError: - pass - -def main(): - #TODO using sqlite3 to store my own orders is overkill, just - # use a python data structure - global db - con = sqlite3.connect(":memory:") - con.row_factory = sqlite3.Row - db = con.cursor() - wallet.download_wallet_history() - wallet.find_unspent_addresses() - create_my_orders() - - print 'starting irc' - irc = irclib.IRCClient() - irc.on_privmsg = on_privmsg - irc.on_pubmsg = on_pubmsg - irc.on_welcome = on_welcome - irc.on_set_topic = on_set_topic - irc.run(server, port, nickname, channel) - -if __name__ == "__main__": - main() - print('done') diff --git a/ob-watcher.py b/ob-watcher.py new file mode 100644 index 00000000..ada179b0 --- /dev/null +++ b/ob-watcher.py @@ -0,0 +1,283 @@ +import BaseHTTPServer, SimpleHTTPServer, threading +import urllib2 +import io, base64, time, sys, os +data_dir = os.path.dirname(os.path.realpath(__file__)) +sys.path.insert(0, os.path.join(data_dir, 'lib')) + +import taker +from irc import IRCMessageChannel, random_nick +from common import * +import common + +# ['counterparty', 'oid', 'ordertype', 'minsize', 'maxsize', 'txfee', 'cjfee'] +col = ' {1}\n' # .format(field,label) + +tableheading = '\n ' + ''.join([ + col.format('ordertype','Type'), + col.format('counterparty','Counterparty'), + col.format('oid','Order ID'), + col.format('cjfee','Fee'), + col.format('txfee','Miner Fee Contribution'), + col.format('minsize','Minimum Size'), + col.format('maxsize','Maximum Size')]) + ' ' + +shutdownform = '' +shutdownpage = '

Successfully Shut down

' + +refresh_orderbook_form = '' + +def calc_depth_data(db, value): + pass + +def calc_order_size_data(db): + return ordersizes + +def create_depth_chart(db, cj_amount): + try: + import matplotlib.pyplot as plt + except ImportError: + return 'Install matplotlib to see graphs' + sqlorders = db.execute('SELECT * FROM orderbook;').fetchall() + orderfees = [calc_cj_fee(o['ordertype'], o['cjfee'], cj_amount)/1e8 + for o in sqlorders if cj_amount >= o['minsize'] and cj_amount <= o['maxsize']] + + if len(orderfees) == 0: + return 'No orders at amount ' + str(cj_amount/1e8) + fig = plt.figure() + if len(orderfees) == 1: + plt.hist(orderfees, 30, rwidth=0.8, range=(orderfees[0]/2, orderfees[0]*2)) + else: + plt.hist(orderfees, 30, rwidth=0.8) + plt.grid() + plt.title('CoinJoin Orderbook Depth Chart for amount=' + str(cj_amount/1e8) + 'btc') + plt.xlabel('CoinJoin Fee / btc') + plt.ylabel('Frequency') + return get_graph_html(fig) + +def create_size_histogram(db, args): + try: + import matplotlib.pyplot as plt + except ImportError: + return 'Install matplotlib to see graphs' + rows = db.execute('SELECT maxsize FROM orderbook;').fetchall() + ordersizes = sorted([r['maxsize']/1e8 for r in rows]) + + fig = plt.figure() + scale = args.get("scale") + if (scale is not None) and (scale[0] == "log"): + ratio = ordersizes[-1] / ordersizes[0] + step = ratio ** 0.0333 # 1/30 + bins = [ordersizes[0] * (step ** i) for i in range(30)] + else: + bins = 30 + plt.hist(ordersizes, bins, histtype='bar', rwidth=0.8) + if bins is not 30: + fig.axes[0].set_xscale('log') + plt.grid() + plt.xlabel('Order sizes / btc') + plt.ylabel('Frequency') + return get_graph_html(fig) + ("
log scale" + if bins == 30 else "
linear") + +def get_graph_html(fig): + imbuf = io.BytesIO() + fig.savefig(imbuf, format='png') + b64 = base64.b64encode(imbuf.getvalue()) + return '' + +def do_nothing(arg, order): + return arg + +def ordertype_display(ordertype, order): + ordertypes = {'absorder': 'Absolute Fee', 'relorder': 'Relative Fee'} + return ordertypes[ordertype] + +def cjfee_display(cjfee, order): + if order['ordertype'] == 'absorder': + return satoshi_to_unit(cjfee, order) + elif order['ordertype'] == 'relorder': + return str(float(cjfee) * 100) + '%' + +def satoshi_to_unit(sat, order): + return "%.8f" % float(Decimal(sat) / Decimal(1e8)) + +def order_str(s, order): + return str(s) + +class OrderbookPageRequestHeader(SimpleHTTPServer.SimpleHTTPRequestHandler): + def __init__(self, request, client_address, base_server): + self.taker = base_server.taker + self.base_server = base_server + SimpleHTTPServer.SimpleHTTPRequestHandler.__init__(self, request, client_address, base_server) + + def create_orderbook_table(self, orderby, desc): + result = '' + rows = self.taker.db.execute('SELECT * FROM orderbook;').fetchall() + if not rows: + return 0, result + if orderby: + orderby = orderby[0] + ordersorder = ['absorder','relorder'] + if orderby not in rows[0].keys() or orderby == 'cjfee': + orderby = 'cjfee' + orderby_cmp = lambda x,y: cmp(Decimal(x['cjfee']),Decimal(y['cjfee'])) if x['ordertype']==y['ordertype'] \ + else cmp(ordersorder.index(x['ordertype']),ordersorder.index(y['ordertype'])) + else: + orderby_cmp = lambda x,y: cmp(x[orderby],y[orderby]) + if desc: + orderby_cmp_wrapper = orderby_cmp + orderby_cmp = lambda x,y: orderby_cmp_wrapper(y,x) + order_keys_display = (('ordertype', ordertype_display), ('counterparty', do_nothing), + ('oid', order_str), ('cjfee', cjfee_display), ('txfee', satoshi_to_unit), + ('minsize', satoshi_to_unit), ('maxsize', satoshi_to_unit)) + + for o in sorted(rows,cmp=orderby_cmp): + result += ' \n' + for key, displayer in order_keys_display: + result += ' \n' + result += ' \n' + return len(rows), result + + def create_orderbook_obj(self): + rows = self.taker.db.execute('SELECT * FROM orderbook;').fetchall() + if not rows: + return [] + + result = [] + for row in rows: + o = dict(row) + if 'cjfee' in o: + o['cjfee'] = int(o['cjfee']) if o['ordertype']=='absorder' else float(o['cjfee']) + result.append(o) + return result + + def get_counterparty_count(self): + counterparties = self.taker.db.execute('SELECT DISTINCT counterparty FROM orderbook;').fetchall() + return str(len(counterparties)) + + def do_GET(self): + #SimpleHTTPServer.SimpleHTTPRequestHandler.do_GET(self) + #print 'httpd received ' + self.path + ' request' + self.path, query = self.path.split('?',1) if '?' in self.path else (self.path,'') + args = urllib2.urlparse.parse_qs(query) + pages = ['/', '/ordersize', '/depth', '/orderbook.json'] + if self.path not in pages: + return + fd = open('orderbook.html', 'r') + orderbook_fmt = fd.read() + fd.close() + alert_msg = '' + if common.joinmarket_alert: + alert_msg = '
JoinMarket Alert Message:
' + common.joinmarket_alert + if self.path == '/': + ordercount, ordertable = self.create_orderbook_table(args.get('orderby'),'desc' in args) + replacements = { + 'PAGETITLE': 'JoinMarket Browser Interface', + 'MAINHEADING': 'JoinMarket Orderbook', + 'SECONDHEADING': (str(ordercount) + ' orders found by ' + + self.get_counterparty_count() + ' counterparties' + alert_msg), + 'MAINBODY': refresh_orderbook_form + shutdownform + tableheading + ordertable + '
' + displayer(o[key], o) + '
\n' + } + elif self.path == '/ordersize': + replacements = { + 'PAGETITLE': 'JoinMarket Browser Interface', + 'MAINHEADING': 'Order Sizes', + 'SECONDHEADING': 'Order Size Histogram' + alert_msg, + 'MAINBODY': create_size_histogram(self.taker.db, args) + } + elif self.path.startswith('/depth'): + #if self.path[6] == '?': + # quantity = + cj_amounts = [10**cja for cja in range(4, 10, 1)] + mainbody = [create_depth_chart(self.taker.db, cja) for cja in cj_amounts] + replacements = { + 'PAGETITLE': 'JoinMarket Browser Interface', + 'MAINHEADING': 'Depth Chart', + 'SECONDHEADING': 'Orderbook Depth' + alert_msg, + 'MAINBODY': '
'.join(mainbody) + } + elif self.path == '/orderbook.json': + replacements = {} + orderbook_fmt = json.dumps(self.create_orderbook_obj()) + orderbook_page = orderbook_fmt + for key, rep in replacements.iteritems(): + orderbook_page = orderbook_page.replace(key, rep) + self.send_response(200) + if self.path.endswith('.json'): + self.send_header('Content-Type', 'application/json') + else: + self.send_header('Content-Type', 'text/html') + self.send_header('Content-Length', len(orderbook_page)) + self.end_headers() + self.wfile.write(orderbook_page) + + def do_POST(self): + pages = ['/shutdown', '/refreshorderbook'] + if self.path not in pages: + return + if self.path == '/shutdown': + self.taker.msgchan.shutdown() + self.send_response(200) + self.send_header('Content-Type', 'text/html') + self.send_header('Content-Length', len(shutdownpage)) + self.end_headers() + self.wfile.write(shutdownpage) + self.base_server.__shutdown_request = True + elif self.path == '/refreshorderbook': + self.taker.msgchan.request_orderbook() + time.sleep(5) + self.path = '/' + self.do_GET() + +class HTTPDThread(threading.Thread): + def __init__(self, taker, hostport): + threading.Thread.__init__(self) + self.daemon = True + self.taker = taker + self.hostport = hostport + + def run(self): + #hostport = ('localhost', 62601) + httpd = BaseHTTPServer.HTTPServer(self.hostport, OrderbookPageRequestHeader) + httpd.taker = self.taker + print('\nstarted http server, visit http://{0}:{1}/\n'.format(*self.hostport)) + httpd.serve_forever() + + +class GUITaker(taker.OrderbookWatch): + def __init__(self, msgchan, hostport): + self.hostport = hostport + super(GUITaker, self).__init__(msgchan) + + def on_welcome(self): + taker.OrderbookWatch.on_welcome(self) + HTTPDThread(self, self.hostport).start() + +def main(): + import bitcoin as btc + import common + import binascii, os + from optparse import OptionParser + + common.nickname =random_nick() #watcher' +binascii.hexlify(os.urandom(4)) + common.load_program_config() + + parser = OptionParser(usage='usage: %prog [options]', + description='Runs a webservice which shows the orderbook.') + parser.add_option('-H', '--host', action='store', type='string', dest='host', + default='localhost', help='hostname or IP to bind to, default=localhost') + parser.add_option('-p', '--port', action='store', type='int', dest='port', + help='port to listen on, default=62601', default=62601) + (options, args) = parser.parse_args() + + hostport = (options.host, options.port) + + irc = IRCMessageChannel(common.nickname) + taker = GUITaker(irc, hostport) + print('starting irc') + + irc.run() + +if __name__ == "__main__": + main() + print('done') diff --git a/orderbook.html b/orderbook.html new file mode 100644 index 00000000..f86650ed --- /dev/null +++ b/orderbook.html @@ -0,0 +1,105 @@ + + + + + + PAGETITLE + + + + + + + + + + + + + + +
+

MAINHEADING

+

SECONDHEADING

+ + MAINBODY +
+
+ + + diff --git a/patientsendpayment.py b/patientsendpayment.py new file mode 100644 index 00000000..00f153fb --- /dev/null +++ b/patientsendpayment.py @@ -0,0 +1,180 @@ + +from optparse import OptionParser +from datetime import timedelta +import threading, time, binascii, os, sys +data_dir = os.path.dirname(os.path.realpath(__file__)) +sys.path.insert(0, os.path.join(data_dir, 'lib')) + +from common import * +import common +import taker +import maker +from irc import IRCMessageChannel, random_nick +import bitcoin as btc + +class TakerThread(threading.Thread): + def __init__(self, tmaker): + threading.Thread.__init__(self) + self.daemon = True + self.tmaker = tmaker + self.finished = False + + def finishcallback(self, coinjointx): + self.tmaker.msgchan.shutdown() + + def run(self): + #TODO this thread doesnt wake up for what could be hours + # need a loop that periodically checks self.finished + #TODO another issue is, what if the bot has run out of utxos and + # needs to wait for some tx to confirm before it can trade + # presumably it needs to wait here until the tx confirms + time.sleep(self.tmaker.waittime) + if self.finished: + return + print 'giving up waiting' + #cancel the remaining order + self.tmaker.modify_orders([0], []) + orders, total_cj_fee = choose_orders(self.tmaker.db, self.tmaker.amount, self.tmaker.makercount, weighted_order_choose) + print 'chosen orders to fill ' + str(orders) + ' totalcjfee=' + str(total_cj_fee) + total_amount = self.tmaker.amount + total_cj_fee + self.tmaker.txfee + print 'total amount spent = ' + str(total_amount) + + utxos = self.tmaker.wallet.select_utxos(self.tmaker.mixdepth, total_amount) + self.tmaker.start_cj(self.tmaker.wallet, self.tmaker.amount, orders, utxos, + self.tmaker.destaddr, self.tmaker.wallet.get_change_addr(self.tmaker.mixdepth), + self.tmaker.txfee, self.finishcallback) + +class PatientSendPayment(maker.Maker, taker.Taker): + def __init__(self, msgchan, wallet, destaddr, amount, makercount, txfee, cjfee, + waittime, mixdepth): + self.destaddr = destaddr + self.amount = amount + self.makercount = makercount + self.txfee = txfee + self.cjfee = cjfee + self.waittime = waittime + self.mixdepth = mixdepth + maker.Maker.__init__(self, msgchan, wallet) + taker.Taker.__init__(self, msgchan) + + def get_crypto_box_from_nick(self, nick): + if self.cjtx: + return taker.Taker.get_crypto_box_from_nick(self, nick) + else: + return maker.Maker.get_crypto_box_from_nick(self, nick) + + def on_welcome(self): + maker.Maker.on_welcome(self) + taker.Taker.on_welcome(self) + self.takerthread = TakerThread(self) + self.takerthread.start() + + def create_my_orders(self): + #choose an absolute fee order to discourage people from + # mixing smaller amounts + order = {'oid': 0, 'ordertype': 'absorder', 'minsize': 0, + 'maxsize': self.amount, 'txfee': self.txfee, 'cjfee': self.cjfee} + return [order] + + def oid_to_order(self, cjorder, oid, amount): + #TODO race condition (kinda) + #if an order arrives and before it finishes another order arrives + # its possible this bot will end up paying to the destaddr more than it + # intended + utxos = self.wallet.select_utxos(self.mixdepth, amount) + return utxos, self.destaddr, self.wallet.get_change_addr(self.mixdepth) + + def on_tx_unconfirmed(self, cjorder, balance, removed_utxos): + self.amount -= cjorder.cj_amount + if self.amount == 0: + self.takerthread.finished = True + print 'finished sending, exiting..' + self.msgchan.shutdown() + return ([], []) + available_balance = self.wallet.get_balance_by_mixdepth()[self.mixdepth] + if available_balance >= self.amount: + order = {'oid': 0, 'ordertype': 'absorder', 'minsize': 0, + 'maxsize': self.amount, 'txfee': self.txfee, 'cjfee': self.cjfee} + return ([], [order]) + else: + debug('not enough money left, have to wait until tx confirms') + return ([0], []) + + def on_tx_confirmed(self, cjorder, confirmations, txid, balance): + if len(self.orderlist) == 0: + order = {'oid': 0, 'ordertype': 'absorder', 'minsize': 0, + 'maxsize': self.amount, 'txfee': self.txfee, 'cjfee': self.cjfee} + return ([], [order]) + else: + return ([], []) + + +def main(): + parser = OptionParser(usage='usage: %prog [options] [wallet file / fromaccount] [amount] [destaddr]', + description='Sends a payment from your wallet to an given address' + + ' using coinjoin. First acts as a maker, announcing an order and ' + + 'waiting for someone to fill it. After a set period of time, gives' + + ' up waiting and acts as a taker and coinjoins any remaining coins') + parser.add_option('-f', '--txfee', action='store', type='int', dest='txfee', + default=10000, help='miner fee contribution, in satoshis, default=10000') + parser.add_option('-N', '--makercount', action='store', type='int', dest='makercount', + help='how many makers to coinjoin with when taking liquidity, default=2', default=2) + parser.add_option('-w', '--wait-time', action='store', type='float', dest='waittime', + help='wait time in hours as a maker before becoming a taker, default=8', default=8) + parser.add_option('-c', '--cjfee', action='store', type='int', dest='cjfee', + help='coinjoin fee asked for when being a maker, in satoshis per order filled, default=50000', default=50000) + parser.add_option('-m', '--mixdepth', action='store', type='int', dest='mixdepth', + help='mixing depth to spend from, default=0', default=0) + parser.add_option('--rpcwallet', action='store_true', dest='userpcwallet', default=False, + help='Use the Bitcoin Core wallet through json rpc, instead of the internal joinmarket ' + + 'wallet. Requires blockchain_source=json-rpc') + (options, args) = parser.parse_args() + + if len(args) < 3: + parser.error('Needs a wallet, amount and destination address') + sys.exit(0) + wallet_name = args[0] + amount = int(args[1]) + destaddr = args[2] + + load_program_config() + addr_valid, errormsg = validate_address(destaddr) + if not addr_valid: + print 'ERROR: Address invalid. ' + errormsg + return + + waittime = timedelta(hours=options.waittime).total_seconds() + print 'txfee=%d cjfee=%d waittime=%s makercount=%d' % (options.txfee, options.cjfee, + str(timedelta(hours=options.waittime)), options.makercount) + + if not options.userpcwallet: + wallet = Wallet(wallet_name, options.mixdepth + 1) + else: + print 'not implemented yet' + sys.exit(0) + wallet = BitcoinCoreWallet(fromaccount = wallet_name) + common.bc_interface.sync_wallet(wallet) + + available_balance = wallet.get_balance_by_mixdepth()[options.mixdepth] + if available_balance < amount: + print 'not enough money at mixdepth=%d, exiting' % (options.mixdepth) + return + + common.nickname = random_nick() + debug('Running patient sender of a payment') + + irc = IRCMessageChannel(common.nickname) + bot = PatientSendPayment(irc, wallet, destaddr, amount, options.makercount, + options.txfee, options.cjfee, waittime, options.mixdepth) + try: + irc.run() + except: + debug('CRASHING, DUMPING EVERYTHING') + debug_dump_object(wallet, ['addr_cache', 'keys', 'seed']) + debug_dump_object(taker) + import traceback + traceback.print_exc() + +if __name__ == "__main__": + main() + print('done') diff --git a/sendpayment.py b/sendpayment.py new file mode 100644 index 00000000..2494f6f3 --- /dev/null +++ b/sendpayment.py @@ -0,0 +1,204 @@ +#! /usr/bin/env python + +from optparse import OptionParser +import threading, pprint, sys, os +data_dir = os.path.dirname(os.path.realpath(__file__)) +sys.path.insert(0, os.path.join(data_dir, 'lib')) + +from common import * +import common +import taker as takermodule +from irc import IRCMessageChannel, random_nick +import bitcoin as btc + +def check_high_fee(total_fee_pc): + WARNING_THRESHOLD = 0.02 # 2% + if total_fee_pc > WARNING_THRESHOLD: + print '\n'.join(['='* 60]*3) + print 'WARNING ' * 6 + print '\n'.join(['='* 60]*1) + print 'OFFERED COINJOIN FEE IS UNUSUALLY HIGH. DOUBLE/TRIPLE CHECK.' + print '\n'.join(['='* 60]*1) + print 'WARNING ' * 6 + print '\n'.join(['='* 60]*3) + +#thread which does the buy-side algorithm +# chooses which coinjoins to initiate and when +class PaymentThread(threading.Thread): + def __init__(self, taker): + threading.Thread.__init__(self) + self.daemon = True + self.taker = taker + self.ignored_makers = [] + + def create_tx(self): + crow = self.taker.db.execute('SELECT COUNT(DISTINCT counterparty) FROM orderbook;').fetchone() + counterparty_count = crow['COUNT(DISTINCT counterparty)'] + counterparty_count -= len(self.ignored_makers) + if counterparty_count < self.taker.makercount: + print 'not enough counterparties to fill order, ending' + self.taker.msgchan.shutdown() + return + + utxos = None + orders = None + cjamount = 0 + change_addr = None + choose_orders_recover = None + if self.taker.amount == 0: + utxos = self.taker.wallet.get_utxos_by_mixdepth()[self.taker.mixdepth] + total_value = sum([va['value'] for va in utxos.values()]) + orders, cjamount = choose_sweep_orders(self.taker.db, total_value, + self.taker.txfee, self.taker.makercount, + self.taker.chooseOrdersFunc, self.ignored_makers) + if not self.taker.answeryes: + total_cj_fee = total_value - cjamount - self.taker.txfee + debug('total cj fee = ' + str(total_cj_fee)) + total_fee_pc = 1.0*total_cj_fee / cjamount + debug('total coinjoin fee = ' + str(float('%.3g' % (100.0 * total_fee_pc))) + '%') + check_high_fee(total_fee_pc) + if raw_input('send with these orders? (y/n):')[0] != 'y': + self.finishcallback(None) + return + else: + orders, total_cj_fee = self.sendpayment_choose_orders(self.taker.amount, + self.taker.makercount) + if not orders: + debug('ERROR not enough liquidity in the orderbook, exiting') + return + total_amount = self.taker.amount + total_cj_fee + self.taker.txfee + print 'total amount spent = ' + str(total_amount) + utxos = self.taker.wallet.select_utxos(self.taker.mixdepth, total_amount) + cjamount = self.taker.amount + change_addr = self.taker.wallet.get_change_addr(self.taker.mixdepth) + choose_orders_recover = self.sendpayment_choose_orders + + self.taker.start_cj(self.taker.wallet, cjamount, orders, utxos, + self.taker.destaddr, change_addr, self.taker.txfee, + self.finishcallback, choose_orders_recover) + + def finishcallback(self, coinjointx): + if coinjointx.all_responded: + coinjointx.self_sign_and_push() + debug('created fully signed tx, ending') + self.taker.msgchan.shutdown() + return + self.ignored_makers += coinjointx.nonrespondants + debug('recreating the tx, ignored_makers=' + str(self.ignored_makers)) + self.create_tx() + + def sendpayment_choose_orders(self, cj_amount, makercount, nonrespondants=[], active_nicks=[]): + self.ignored_makers += nonrespondants + orders, total_cj_fee = choose_orders(self.taker.db, cj_amount, makercount, + self.taker.chooseOrdersFunc, self.ignored_makers + active_nicks) + if not orders: + return None, 0 + print 'chosen orders to fill ' + str(orders) + ' totalcjfee=' + str(total_cj_fee) + if not self.taker.answeryes: + if len(self.ignored_makers) > 0: + noun = 'total' + else: + noun = 'additional' + total_fee_pc = 1.0*total_cj_fee / cj_amount + debug(noun + ' coinjoin fee = ' + str(float('%.3g' % (100.0 * total_fee_pc))) + '%') + check_high_fee(total_fee_pc) + if raw_input('send with these orders? (y/n):')[0] != 'y': + debug('ending') + self.taker.msgchan.shutdown() + return None, -1 + return orders, total_cj_fee + + def run(self): + print 'waiting for all orders to certainly arrive' + time.sleep(self.taker.waittime) + self.create_tx() + + +class SendPayment(takermodule.Taker): + def __init__(self, msgchan, wallet, destaddr, amount, makercount, txfee, waittime, mixdepth, answeryes, chooseOrdersFunc): + takermodule.Taker.__init__(self, msgchan) + self.wallet = wallet + self.destaddr = destaddr + self.amount = amount + self.makercount = makercount + self.txfee = txfee + self.waittime = waittime + self.mixdepth = mixdepth + self.answeryes = answeryes + self.chooseOrdersFunc = chooseOrdersFunc + + def on_welcome(self): + takermodule.Taker.on_welcome(self) + PaymentThread(self).start() + +def main(): + parser = OptionParser(usage='usage: %prog [options] [wallet file / fromaccount] [amount] [destaddr]', + description='Sends a single payment from a given mixing depth of your ' + + 'wallet to an given address using coinjoin and then switches off. Also sends from bitcoinqt. ' + + 'Setting amount to zero will do a sweep, where the entire mix depth is emptied') + parser.add_option('-f', '--txfee', action='store', type='int', dest='txfee', + default=10000, help='miner fee contribution, in satoshis, default=10000') + parser.add_option('-w', '--wait-time', action='store', type='float', dest='waittime', + help='wait time in seconds to allow orders to arrive, default=5', default=5) + parser.add_option('-N', '--makercount', action='store', type='int', dest='makercount', + help='how many makers to coinjoin with, default=2', default=2) + parser.add_option('-C','--choose-cheapest', action='store_true', dest='choosecheapest', default=False, + help='override weightened offers picking and choose cheapest') + parser.add_option('-P','--pick-orders', action='store_true', dest='pickorders', default=False, + help='manually pick which orders to take. doesn\'t work while sweeping.') + parser.add_option('-m', '--mixdepth', action='store', type='int', dest='mixdepth', + help='mixing depth to spend from, default=0', default=0) + parser.add_option('--yes', action='store_true', dest='answeryes', default=False, + help='answer yes to everything') + parser.add_option('--rpcwallet', action='store_true', dest='userpcwallet', default=False, + help='Use the Bitcoin Core wallet through json rpc, instead of the internal joinmarket ' + + 'wallet. Requires blockchain_source=json-rpc') + (options, args) = parser.parse_args() + + if len(args) < 3: + parser.error('Needs a wallet, amount and destination address') + sys.exit(0) + wallet_name = args[0] + amount = int(args[1]) + destaddr = args[2] + + load_program_config() + addr_valid, errormsg = validate_address(destaddr) + if not addr_valid: + print 'ERROR: Address invalid. ' + errormsg + return + + chooseOrdersFunc = None + if options.pickorders and amount != 0: #cant use for sweeping + chooseOrdersFunc = pick_order + elif options.choosecheapest: + chooseOrdersFunc = cheapest_order_choose + else: #choose randomly (weighted) + chooseOrdersFunc = weighted_order_choose + + common.nickname = random_nick() + debug('starting sendpayment') + + if not options.userpcwallet: + wallet = Wallet(wallet_name, options.mixdepth + 1) + else: + wallet = BitcoinCoreWallet(fromaccount = wallet_name) + common.bc_interface.sync_wallet(wallet) + + irc = IRCMessageChannel(common.nickname) + taker = SendPayment(irc, wallet, destaddr, amount, options.makercount, options.txfee, + options.waittime, options.mixdepth, options.answeryes, chooseOrdersFunc) + try: + debug('starting irc') + irc.run() + except: + debug('CRASHING, DUMPING EVERYTHING') + debug_dump_object(wallet, ['addr_cache', 'keys', 'wallet_name', 'seed']) + debug_dump_object(taker) + import traceback + debug(traceback.format_exc()) + +if __name__ == "__main__": + main() + print('done') + diff --git a/taker.py b/taker.py deleted file mode 100644 index 8badb809..00000000 --- a/taker.py +++ /dev/null @@ -1,300 +0,0 @@ -#! /usr/bin/env python - -from common import * -import irclib -import bitcoin as btc - -import sqlite3, sys, base64 -import threading, time - -nickname = 'cj-taker' -seed = btc.sha256('your brainwallet goes here') -my_utxo = '5cf68d4c42132f8f0bef8573454036953ddb3ba77a3bf3797d9862b7102d65cd:0' - -my_tx_fee_contribution = 10000 - -class CoinJoinTX(object): - def __init__(self, irc, cj_amount, counterparties, oids, my_utxos, my_cj_addr, - my_change_addr, my_txfee, finishcallback=None): - ''' - if my_change is None then there wont be a change address - thats used if you want to entirely coinjoin one utxo with no change left over - ''' - self.cj_amount = cj_amount - self.active_orders = dict(zip(counterparties, oids)) - self.nonrespondants = list(counterparties) - self.my_utxos = my_utxos - self.utxos = {irc.nick: my_utxos} - self.finishcallback = finishcallback - self.my_txfee = my_txfee - self.outputs = [{'address': my_cj_addr, 'value': self.cj_amount}] - self.my_change_addr = my_change_addr - self.cjfee_total = 0 - self.latest_tx = None - for c, oid in zip(counterparties, oids): - irc.privmsg(c, command_prefix + 'fill ' + str(oid) + ' ' + str(cj_amount)) - - def recv_tx_parts(self, irc, nick, utxo_list, cj_addr, change_addr): - if nick not in self.nonrespondants: - debug('nick(' + nick + ') not in nonrespondants ' + str(self.nonrespondants)) - return - self.utxos[nick] = utxo_list - self.nonrespondants.remove(nick) - order = db.execute('SELECT ordertype, txfee, cjfee FROM ' - 'orderbook WHERE oid=? AND counterparty=?', - (self.active_orders[nick], nick)).fetchone() - real_cjfee = calc_cj_fee(order['ordertype'], order['cjfee'], self.cj_amount) - self.outputs.append({'address': change_addr, 'value': - calc_total_input_value(self.utxos[nick]) - self.cj_amount - - order['txfee'] + real_cjfee}) - self.outputs.append({'address': cj_addr, 'value': self.cj_amount}) - self.cjfee_total += real_cjfee - if len(self.nonrespondants) > 0: - return - debug('got all parts, enough to build a tx cjfeetotal=' + str(self.cjfee_total)) - - my_total_in = 0 - for u in self.my_utxos: - usvals = wallet.unspent[u] - my_total_in += int(usvals['value']) - - my_change_value = my_total_in - self.cj_amount - self.cjfee_total - self.my_txfee - if self.my_change_addr == None: - if my_change_value != 0: - print 'WARNING CHANGE NOT BEING USED\nCHANGEVALUE = ' + str(my_change_value) - else: - self.outputs.append({'address': self.my_change_addr, 'value': my_change_value}) - utxo_tx = [dict([('output', u)]) for u in sum(self.utxos.values(), [])] - tx = btc.mktx(utxo_tx, self.outputs) - txb64 = base64.b64encode(tx.decode('hex')) - n = MAX_PRIVMSG_LEN - txparts = [txb64[i:i+n] for i in range(0, len(txb64), n)] - for p in txparts[:-1]: - for nickk in self.active_orders.keys(): - irc.privmsg(nickk, command_prefix + 'txpart' + p) - for nickk in self.active_orders.keys(): - irc.privmsg(nickk, command_prefix + 'tx ' + txparts[-1]) - #now sign it ourselves here - - for index, ins in enumerate(btc.deserialize(tx)['ins']): - utxo = ins['outpoint']['hash'] + ':' + str(ins['outpoint']['index']) - if utxo not in self.my_utxos: - continue - if utxo not in wallet.unspent: - continue - addr = wallet.unspent[utxo]['address'] - tx = btc.sign(tx, index, wallet.get_key_from_addr(addr)) - self.latest_tx = btc.deserialize(tx) - - def add_signature(self, sigb64): - sig = base64.b64decode(sigb64).encode('hex') - - inserted_sig = False - tx = btc.serialize(self.latest_tx) - for index, ins in enumerate(self.latest_tx['ins']): - ftx = btc.blockr_fetchtx(ins['outpoint']['hash'], get_network()) - src_val = btc.deserialize(ftx)['outs'][ ins['outpoint']['index'] ] - sig_good = btc.verify_tx_input(tx, index, src_val['script'], *btc.deserialize_script(sig)) - if sig_good: - debug('found good sig at index=%d' % (index)) - ins['script'] = sig - inserted_sig = True - break - if not inserted_sig: - debug('signature did not match anything in the tx') - #TODO what if the signature doesnt match anything - # nothing really to do except drop it, carry on and wonder why the - # other guy sent a failed signature - - tx_signed = True - for ins in self.latest_tx['ins']: - if ins['script'] == '': - tx_signed = False - if not tx_signed: - return - debug('the entire tx is signed, ready to pushtx()') - print btc.serialize(self.latest_tx) - #ret = btc.blockr_pushtx(btc.serialize(self.latest_tx), get_network()) - #print 'pushed tx ' + str(ret) - if self.finishcallback != None: - self.finishcallback() - -wallet = Wallet(seed) -cjtx = None - -algo_thread = None - -#how long to wait for all the orders to arrive before starting to do coinjoins -ORDER_ARRIVAL_WAIT_TIME = 3 - -def choose_order(cj_amount): - - sqlorders = db.execute('SELECT * FROM orderbook;').fetchall() - #for o in sqlorders: - # print '(%s %s %d %d-%d %d %s)' % (o['counterparty'], o['ordertype'], o['oid'], - # o['minsize'], o['maxsize'], o['txfee'], o['cjfee']) - - orders = [(o['counterparty'], o['oid'], o['txfee'] + - calc_cj_fee(o['ordertype'], o['cjfee'], cj_amount)) - for o in sqlorders if cj_amount >= o['minsize'] or cj_amount <= o['maxsize']] - orders = sorted(orders, key=lambda k: k[2]) - print 'orders = ' + str(orders) - return orders[0] #choose the cheapest - -#thread which does the buy-side algorithm -# chooses which coinjoins to initiate and when -class AlgoThread(threading.Thread): - def __init__(self, irc, initial_unspents): - threading.Thread.__init__(self) - self.daemon = True - self.irc = irc - self.initial_unspents = initial_unspents - self.finished_cj = False - - def finished_cj_callback(self): - self.finished_cj = True - print 'finished cj' - - def run(self): - global cjtx - time.sleep(ORDER_ARRIVAL_WAIT_TIME) - #while True: - if 1: - #wait for orders to arrive - #TODO just make this do one tx and then stop - if len(self.initial_unspents) == 0: - print 'finished mixing, closing...' - self.irc.shutdown() - #break - - utxo, addrvalue = self.initial_unspents.popitem() - counterparty, oid, fee = choose_order(addrvalue['value']) - cj_amount = addrvalue['value'] - fee - self.finished_cj = False - cjtx = CoinJoinTX(self.irc, cj_amount, [counterparty], [int(oid)], - [utxo], wallet.get_receive_addr(mixing_depth=1), None, my_tx_fee_contribution, self.finished_cj_callback) - #algorithm for making - ''' - single_cj_amount = 112000000 - unspent = [] - for utxo, addrvalue in self.initial_unspents.iteritems(): - unspent.append({'value': addrvalue['value'], 'utxo': utxo}) - inputs = btc.select(unspent, single_cj_amount) - my_utxos = [i['utxo'] for i in inputs] - counterparty, oid = choose_order(single_cj_amount) - cjtx = CoinJoinTX(self.irc, int(single_cj_amount), [counterparty], [int(oid)], - my_utxos, wallet.get_receive_addr(mixing_depth=1), wallet.get_change_addr(mixing_depth=0)) - ''' - while not self.finished_cj: - time.sleep(5) - print 'woken algo thread' - - -def add_order(nick, chunks): - db.execute('INSERT INTO orderbook VALUES(?, ?, ?, ?, ?, ?, ?);', - (nick, chunks[1], chunks[0], chunks[2], chunks[3], chunks[4], chunks[5])) - -def on_privmsg(irc, nick, message): - #debug("privmsg nick=%s message=%s" % (nick, message)) - if message[0] != command_prefix: - return - - for command in message[1:].split(command_prefix): - chunks = command.split(" ") - if chunks[0] in ordername_list: - add_order(nick, chunks) - elif chunks[0] == 'myparts': - utxo_list = chunks[1].split(',') - cj_addr = chunks[2] - change_addr = chunks[3] - cjtx.recv_tx_parts(irc, nick, utxo_list, cj_addr, change_addr) - elif chunks[0] == 'sig': - sig = chunks[1] - cjtx.add_signature(sig) - -#each order has an id for referencing to and looking up -# using the same id again overwrites it, they'll be plenty of times when an order -# has to be modified and its better to just have !order rather than !cancelorder then !order -def on_pubmsg(irc, nick, message): - global cjtx - print("pubmsg nick=%s message=%s" % (nick, message)) - if message[0] != command_prefix: - return - - for command in message[1:].split(command_prefix): - #commands starting with % are for testing and will be removed in the final version - chunks = command.split(" ") - if chunks[0] == '%quit' or chunks[0] == '%takerquit': - irc.shutdown() - elif chunks[0] == 'cancel': - #!cancel [oid] - try: - oid = int(chunks[1]) - db.execute("DELETE FROM orderbook WHERE counterparty=? AND oid=?;", - (nick, oid)) - except ValueError as e: - debug("!cancel " + repr(e)) - return - elif chunks[0] in ordername_list: - add_order(nick, chunks) - elif chunks[0] == '%showob': - print('printing orderbook') - for o in db.execute('SELECT * FROM orderbook;').fetchall(): - print '(%s %s %d %d-%d %d %s)' % (o['counterparty'], o['ordertype'], o['oid'], - o['minsize'], o['maxsize'], o['txfee'], o['cjfee']) - print('done') - elif chunks[0] == '%fill': - counterparty = chunks[1] - oid = chunks[2] - amount = chunks[3] - #!fill [counterparty] [oid] [amount] - cjtx = CoinJoinTX(irc, int(amount), [counterparty], [int(oid)], - [my_utxo], wallet.get_receive_addr(mixing_depth=1), - wallet.get_change_addr(mixing_depth=0), my_tx_fee_contribution) - - #self.connection.quit("Using irc.client.py") - -def on_welcome(irc): - global algo_thread - irc.pubmsg(command_prefix + 'orderbook') - algo_thread = AlgoThread(irc, wallet.unspent.copy()) - #algo_thread.start() - -def on_set_topic(irc, newtopic): - chunks = newtopic.split('|') - try: - print chunks[1] - print chunks[2] - except IndexError: - pass - -''' -for m in range(2): - print 'mixing depth ' + str(m) - for forchange in range(2): - print ' forchange=' + str(forchange) - for n in range(3): - #print ' ' + str(n) + ' ' + btc.privtoaddr(wallet.get_key(m, forchange, n), 0x6f) -''' - -def main(): - global db - con = sqlite3.connect(":memory:", check_same_thread=False) - con.row_factory = sqlite3.Row - db = con.cursor() - db.execute("CREATE TABLE orderbook(counterparty TEXT, oid INTEGER, ordertype TEXT, " - + "minsize INTEGER, maxsize INTEGER, txfee INTEGER, cjfee TEXT);") - wallet.download_wallet_history() - wallet.find_unspent_addresses() - - print 'starting irc' - irc = irclib.IRCClient() - irc.on_privmsg = on_privmsg - irc.on_pubmsg = on_pubmsg - irc.on_welcome = on_welcome - irc.on_set_topic = on_set_topic - irc.run(server, port, nickname, channel) - -if __name__ == "__main__": - main() - print('done') diff --git a/test/bitcoin.conf b/test/bitcoin.conf new file mode 100644 index 00000000..ca22075f --- /dev/null +++ b/test/bitcoin.conf @@ -0,0 +1,5 @@ +rpcuser=bitcoinrpc +rpcpassword=123456abcdef +walletnotify=/usr/bin/wget -q --spider --timeout=0.5 --tries=1 http://localhost:62612/walletnotify?%s +alertnotify=/usr/bin/wget -q --spider --timeout=0.5 --tries=1 http://localhost:62612/alertnotify?%s + diff --git a/test/commontest.py b/test/commontest.py new file mode 100644 index 00000000..0c2c86d6 --- /dev/null +++ b/test/commontest.py @@ -0,0 +1,72 @@ +import sys +import os, time +data_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) +sys.path.insert(0, os.path.join(data_dir, 'lib')) +import subprocess +import unittest +import common +from blockchaininterface import * +import bitcoin as btc +import binascii +import pexpect +import random + +'''Some helper functions for testing''' + + +'''This code is intended to provide +subprocess startup cross-platform with +some useful options; it could do with +some simplification/improvement.''' +import platform +OS = platform.system() +PINL = '\r\n' if OS == 'Windows' else '\n' + +def local_command(command, bg=False, redirect=''): + if redirect=='NULL': + if OS=='Windows': + command.append(' > NUL 2>&1') + elif OS=='Linux': + command.extend(['>', '/dev/null', '2>&1']) + else: + print "OS not recognised, quitting." + elif redirect: + command.extend(['>', redirect]) + + if bg: + #using subprocess.PIPE seems to cause problems + FNULL = open(os.devnull,'w') + return subprocess.Popen(command, stdout=FNULL, stderr=subprocess.STDOUT, close_fds=True) + else: + #in case of foreground execution, we can use the output; if not + #it doesn't matter + return subprocess.check_output(command) + +def make_wallets(n, wallet_structures = None, mean_amt=1, sdev_amt=0): + '''n: number of wallets to be created + wallet_structure: array of n arrays , each subarray + specifying the number of addresses to be populated with coins + at each depth (for now, this will only populate coins into 'receive' addresses) + mean_amt: the number of coins (in btc units) in each address as above + sdev_amt: if randomness in amouts is desired, specify here. + Returns: a dict of dicts of form {0:{'seed':seed,'wallet':Wallet object},1:..,}''' + if len(wallet_structures) != n: + raise Exception("Number of wallets doesn't match wallet structures") + seeds = common.chunks(binascii.hexlify(os.urandom(15*n)),n) + wallets = {} + for i in range(n): + wallets[i] = {'seed':seeds[i], 'wallet': common.Wallet(seeds[i], max_mix_depth=5)} + for j in range(5): + for k in range(wallet_structures[i][j]): + deviation = sdev_amt*random.random() + amt = mean_amt - sdev_amt/2.0 + deviation + if amt < 0: amt = 0.001 + common.bc_interface.grab_coins(wallets[i]['wallet'].get_receive_addr(j),amt) + return wallets + +def interact(process, inputs, expected): + if len(inputs) != len(expected): + raise Exception("Invalid inputs to interact()") + for i, inp in enumerate(inputs): + process.expect(expected[i]) + process.sendline(inp) \ No newline at end of file diff --git a/test/randomfunc-test.py b/test/randomfunc-test.py new file mode 100644 index 00000000..28dc2bfb --- /dev/null +++ b/test/randomfunc-test.py @@ -0,0 +1,75 @@ + +import sys, os +data_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) +sys.path.insert(0, os.path.join(data_dir, 'lib')) + +import common + +try: + import matplotlib.pyplot as plt + import numpy as np +except ImportError: + print 'Install matplotlib and numpy to run this test' + sys.exit(0) + +def test_power(): + a = 5. # shape + samples = 10000 + s1 = np.random.power(a, samples) + s2 = common.rand_pow_array(a, samples) + + plt.figure('power test') + count1, bins1, ignored1 = plt.hist(s1, bins=30, label='numpy', histtype='step') + x = np.linspace(0, 1, 100) + y = a*x**(a-1.0) + normed_y1 = samples*np.diff(bins1)[0]*y + plt.plot(x, normed_y1, label='numpy.random.power fit') + + count2, bins2, ignored2 = plt.hist(s2, bins=30, label='joinmarket', histtype='step') + normed_y2 = samples*np.diff(bins2)[0]*y + plt.plot(x, normed_y2, label='common.rand_pow_array fit') + plt.title('testing power distribution') + plt.legend(loc='upper left') + plt.show() + +def test_choice(): + + xaxis_divisions = 100 + sinp = np.sin(np.arange(xaxis_divisions) * 2*np.pi / xaxis_divisions)**2 + sinp /= sum(sinp) + sinp = list(sinp) + + sincp = np.sinc((np.arange(xaxis_divisions) - 2*xaxis_divisions/3) * 2*np.pi / xaxis_divisions)**2 + sincp /= sum(sincp) + sincp = list(sincp) + + x = np.arange(xaxis_divisions) * 2*np.pi / xaxis_divisions + gamma2p = x**2 * np.exp(-2*x) + gamma2p /= sum(gamma2p) + gamma2p = list(gamma2p) + + plt.figure('choice test') + for p, name in ((sinp, 'sin'), (sincp, 'sinc'), (gamma2p, 'gamma(2, 2)')): + #for p, name in ((sincp, 'sincp'), ): + samples = 50000 + common_data = [] + numpy_data = [] + for i in range(samples): + cpoint = common.rand_weighted_choice(xaxis_divisions, p) + common_data.append(cpoint) + nppoint = np.random.choice(xaxis_divisions, p=p) + numpy_data.append(nppoint) + + count1, bins1, ignored1 = plt.hist(common_data, bins=xaxis_divisions, label=name + 'joinmarket', histtype='step') + count2, bins2, ignored2 = plt.hist(numpy_data, bins=xaxis_divisions, label=name + 'numpy', histtype='step') + + plt.title('testing choice') + plt.legend(loc='upper left') + plt.show() + +def main(): + test_power() + test_choice() + +if __name__ == '__main__': + main() \ No newline at end of file diff --git a/test/regtest.py b/test/regtest.py new file mode 100644 index 00000000..80b54ef1 --- /dev/null +++ b/test/regtest.py @@ -0,0 +1,162 @@ +import sys +import os, time +data_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) +sys.path.insert(0, os.path.join(data_dir, 'lib')) +import subprocess +import unittest +import common +import commontest +from blockchaininterface import * +import bitcoin as btc +import binascii + +''' Just some random thoughts to motivate possible tests; +almost none of this has really been done: + +Expectations +1. Any bot should run indefinitely irrespective of the input +messages it receives, except bots which perform a finite action + +2. A bot must never spend an unacceptably high transaction fee. + +3. A bot must explicitly reject interactions with another bot not +respecting the JoinMarket protocol for its version. + +4. Bots must never send bitcoin data in the clear over the wire. +''' + +'''helper functions put here to avoid polluting the main codebase.''' + +import platform +OS = platform.system() +PINL = '\r\n' if OS == 'Windows' else '\n' + +def local_command(command, bg=False, redirect=''): + if redirect=='NULL': + if OS=='Windows': + command.append(' > NUL 2>&1') + elif OS=='Linux': + command.extend(['>', '/dev/null', '2>&1']) + else: + print "OS not recognised, quitting." + elif redirect: + command.extend(['>', redirect]) + + if bg: + FNULL = open(os.devnull,'w') + return subprocess.Popen(command, stdout=FNULL, stderr=subprocess.STDOUT, close_fds=True) + else: + #in case of foreground execution, we can use the output; if not + #it doesn't matter + return subprocess.check_output(command) + + + +class Join2PTests(unittest.TestCase): + '''This test case intends to simulate + a single join with a single counterparty. In that sense, + it's not realistic, because nobody (should) do joins with only 1 maker, + but this test has the virtue of being the simplest possible thing + that JoinMarket can do. ''' + def setUp(self): + #create 2 new random wallets. + #put 10 coins into the first receive address + #to allow that bot to start. + self.wallets = commontest.make_wallets(2, + wallet_structures=[[1,0,0,0,0],[1,0,0,0,0]], mean_amt=10) + + + def run_simple_send(self, n, m): + #start yield generator with wallet1 + yigen_proc = local_command(['python','yield-generator.py', + str(self.wallets[0]['seed'])],bg=True) + + #A significant delay is needed to wait for the yield generator to sync its wallet + time.sleep(30) + + #run a single sendpayment call with wallet2 + amt = n*100000000 #in satoshis + dest_address = btc.privkey_to_address(os.urandom(32), common.get_p2pk_vbyte()) + try: + for i in range(m): + sp_proc = local_command(['python','sendpayment.py','--yes','-N','1', self.wallets[1]['seed'],\ + str(amt), dest_address]) + except subprocess.CalledProcessError, e: + if yigen_proc: + yigen_proc.terminate() + print e.returncode + print e.message + raise + + if yigen_proc: + yigen_proc.terminate() + + received = common.bc_interface.get_received_by_addr([dest_address], None)['data'][0]['balance'] + if received != amt*m: + common.debug('received was: '+str(received)+ ' but amount was: '+str(amt)) + return False + return True + + def test_simple_send(self): + self.failUnless(self.run_simple_send(2, 2)) + + +class JoinNPTests(unittest.TestCase): + + def setUp(self): + self.n = 2 + #create n+1 new random wallets. + #put 10 coins into the first receive address + #to allow that bot to start. + wallet_structures = [[1,0,0,0,0]]*3 + self.wallets = commontest.make_wallets(3, wallet_structures=wallet_structures, + mean_amt=10) + #the sender is wallet (n+1), i.e. index wallets[n] + + + def test_n_partySend(self): + self.failUnless(self.run_nparty_join()) + + def run_nparty_join(self): + yigen_procs = [] + for i in range(self.n): + ygp = local_command(['python','yield-generator.py',\ + str(self.wallets[i]['seed'])], bg=True) + time.sleep(2) #give it a chance + yigen_procs.append(ygp) + + #A significant delay is needed to wait for the yield generators to sync + time.sleep(60) + + #run a single sendpayment call + amt = 100000000 #in satoshis + dest_address = btc.privkey_to_address(os.urandom(32), common.get_p2pk_vbyte()) + try: + sp_proc = local_command(['python','sendpayment.py','--yes','-N', str(self.n),\ + self.wallets[self.n]['seed'], str(amt), dest_address]) + except subprocess.CalledProcessError, e: + for ygp in yigen_procs: + ygp.kill() + print e.returncode + print e.message + raise + + if any(yigen_procs): + for ygp in yigen_procs: + ygp.kill() + + received = common.bc_interface.get_received_by_addr([dest_address], None)['data'][0]['balance'] + if received != amt: + return False + return True + + +def main(): + os.chdir(data_dir) + common.load_program_config() + unittest.main() + +if __name__ == '__main__': + main() + + diff --git a/test/regtest_joinmarket.cfg b/test/regtest_joinmarket.cfg new file mode 100644 index 00000000..c5337b19 --- /dev/null +++ b/test/regtest_joinmarket.cfg @@ -0,0 +1,20 @@ +#NOTE: This configuration file is for testing with regtest only +#For mainnet usage, running a JoinMarket script will create the default file +[BLOCKCHAIN] +blockchain_source = regtest +rpc_host = localhost +rpc_port = 18332 +rpc_user = bitcoinrpc +rpc_password = 123456abcdef +network = testnet +bitcoin_cli_cmd = bitcoin-cli +notify_port = 62612 +[MESSAGING] +host = chat.freenode.net +channel = joinmarket-pit +port = 6667 +usessl = false +socks5 = false +socks5_host = localhost +socks5_port = 9150 + diff --git a/test/tumbler-test.py b/test/tumbler-test.py new file mode 100644 index 00000000..a9374029 --- /dev/null +++ b/test/tumbler-test.py @@ -0,0 +1,123 @@ +import sys +import os, time, random +data_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) +sys.path.insert(0, os.path.join(data_dir, 'lib')) +import subprocess +import unittest +import common +from blockchaininterface import * +import bitcoin as btc +import binascii +import pexpect +import platform +OS = platform.system() +PINL = '\r\n' if OS == 'Windows' else '\n' + +def local_command(command, bg=False, redirect=''): + if redirect=='NULL': + if OS=='Windows': + command.append(' > NUL 2>&1') + elif OS=='Linux': + command.extend(['>', '/dev/null', '2>&1']) + else: + print "OS not recognised, quitting." + elif redirect: + command.extend(['>', redirect]) + + if bg: + FNULL = open(os.devnull,'w') + return subprocess.Popen(command, stdout=FNULL, stderr=subprocess.STDOUT, close_fds=True) + #return subprocess.Popen(command, stdout=subprocess.PIPE, + # stderr=subprocess.PIPE, stdin=subprocess.PIPE) + else: + #in case of foreground execution, we can use the output; if not + #it doesn't matter + return subprocess.check_output(command) + + +def interact(process, inputs, expected): + if len(inputs) != len(expected): + raise Exception("Invalid inputs to interact()") + for i, inp in enumerate(inputs): + process.expect(expected[i]) + process.sendline(inp) + +class TumblerTests(unittest.TestCase): + def setUp(self): + #create 7 new random wallets. + #put about 10 coins in each, spread over random mixdepths + #in units of 0.5 + + seeds = common.chunks(binascii.hexlify(os.urandom(15*7)),7) + self.wallets = {} + for i in range(7): + self.wallets[i] = {'seed':seeds[i], 'wallet': common.Wallet(seeds[i], max_mix_depth=5)} + #adding coins somewhat randomly, spread over all 5 depths + for i in range(7): + w = self.wallets[i]['wallet'] + for j in range(5): + for k in range(4): + base = 0.001 if i==6 else 1.0 + amt = base + random.random() #average is 0.5 for tumbler, else 1.5 + common.bc_interface.grab_coins(w.get_receive_addr(j),amt) + + def run_tumble(self, amt): + yigen_procs = [] + for i in range(6): + ygp = local_command(['python','yield-generator.py',\ + str(self.wallets[i]['seed'])], bg=True) + time.sleep(2) #give it a chance + yigen_procs.append(ygp) + + #A significant delay is needed to wait for the yield generators to sync + time.sleep(60) + + #start a tumbler + amt = amt*1e8 #in satoshis + #send to any old address + dest_address = btc.privkey_to_address(os.urandom(32), common.get_addr_vbyte()) + try: + #default mixdepth source is zero, so will take coins from m 0. + #see tumbler.py --h for details + expected = ['tumble with these tx'] + test_in = ['y'] + p = pexpect.spawn('python',['tumbler.py', '-N', '1', '0', + '-a', '0', '-M', '5', + self.wallets[6]['seed'], dest_address]) + interact(p, test_in, expected) + p.expect(pexpect.EOF, timeout=100000) + p.close() + if p.exitstatus != 0: + print 'failed due to exit status: '+str(p.exitstatus) + return False + #print('use seed: '+self.wallets[6]['seed']) + #print('use dest addr: '+dest_address) + #ret = raw_input('quit?') + except subprocess.CalledProcessError, e: + for ygp in yigen_procs: + ygp.kill() + print e.returncode + print e.message + raise + + if any(yigen_procs): + for ygp in yigen_procs: + ygp.kill() + + received = common.bc_interface.get_received_by_addr([dest_address], None)['data'][0]['balance'] + print('received: '+str(received)) + return True + + def test_simple_send(self): + self.failUnless(self.run_tumble(1)) + + +def main(): + os.chdir(data_dir) + common.load_program_config() + unittest.main() + +if __name__ == '__main__': + main() + + diff --git a/test/wallet-test.py b/test/wallet-test.py new file mode 100644 index 00000000..84d5b4f7 --- /dev/null +++ b/test/wallet-test.py @@ -0,0 +1,96 @@ +import sys +import os, time +data_dir = os.path.dirname(os.path.dirname(os.path.realpath(__file__))) +sys.path.insert(0, os.path.join(data_dir, 'lib')) +import subprocess +import unittest +import common +import commontest +from blockchaininterface import * +import bitcoin as btc +import binascii +import pexpect + +class TestWalletCreation(unittest.TestCase): + + def test_generate(self): + print 'wallet generation and encryption password tests' + #testing a variety of passwords + self.failUnless(self.run_generate('abc123')) + self.failUnless(self.run_generate('dddddddddddddddddddddddddddddddddddddddddddd')) + #null password is accepted + self.failUnless(self.run_generate('')) + #binary password is accepted; good luck with that! + self.failUnless(self.run_generate('\x01'*10)) + #password with NULL bytes is *not* accepted + self.failIf(self.run_generate('\x00'*10)) + + def run_generate(self, pwd): + try: + test_in = [pwd,pwd,'testwallet.json'] + expected = ['Enter wallet encryption passphrase:', + 'Reenter wallet encryption passphrase:', + 'Input wallet file name'] + testlog = open('test/testlog-'+pwd, 'wb') + p = pexpect.spawn('python wallet-tool.py generate', logfile=testlog) + commontest.interact(p, test_in, expected) + p.expect('saved to') + #time.sleep(2) + p.close() + testlog.close() + #anything to check in the log? + if p.exitstatus != 0: + print 'failed due to exit status: '+str(p.exitstatus) + return False + #check the wallet exists (and contains appropriate json?) + if not os.path.isfile('wallets/testwallet.json'): + print 'failed due to wallet missing' + return False + os.remove('wallets/testwallet.json') + except: + return False + return True + + +class TestWalletRecovery(unittest.TestCase): + + def setUp(self): + self.testseed = 'earth gentle mouth circle despite pocket adore student board dress blanket worthless' + + def test_recover(self): + print 'wallet recovery from seed test' + self.failUnless(self.run_recover(self.testseed)) + #try using an invalid word list; can add more variants + wrongseed = 'oops '+self.testseed + self.failIf(self.run_recover(wrongseed)) + + def run_recover(self, seed): + try: + testlog = open('test_recover','wb') + p = pexpect.spawn('python wallet-tool.py recover', logfile = testlog) + expected = ['Input 12 word recovery seed', + 'Enter wallet encryption passphrase:', + 'Reenter wallet encryption passphrase:', + 'Input wallet file name'] + test_in = [seed, 'abc123', 'abc123', 'test_recover_wallet.json'] + commontest.interact(p, test_in, expected) + p.expect('saved to') + p.close() + testlog.close() + #anything to check in the log? + if p.exitstatus != 0: + print 'failed due to exit status: '+str(p.exitstatus) + return False + #check the wallet exists (and contains appropriate json? todo) + if not os.path.isfile('wallets/test_recover_wallet.json'): + print 'failed due to wallet missing' + return False + os.remove('wallets/test_recover_wallet.json') + except: + return False + return True + +if __name__ == '__main__': + os.chdir(data_dir) + common.load_program_config() + unittest.main() \ No newline at end of file diff --git a/test/wnb.sh b/test/wnb.sh new file mode 100755 index 00000000..2948c75a --- /dev/null +++ b/test/wnb.sh @@ -0,0 +1,6 @@ +#!/bin/bash +for i in {62612..62619} +do +curl -sI --max-time 0.6 http://localhost:$i/walletnotify?$1 +done + diff --git a/tumbler.py b/tumbler.py new file mode 100644 index 00000000..0150ea62 --- /dev/null +++ b/tumbler.py @@ -0,0 +1,394 @@ + +import datetime, threading, binascii, sys, os, copy +data_dir = os.path.dirname(os.path.realpath(__file__)) +sys.path.insert(0, os.path.join(data_dir, 'lib')) + +import taker as takermodule +import common +from common import * +from irc import IRCMessageChannel, random_nick + +from optparse import OptionParser +from pprint import pprint + +orderwaittime = 10 + +def lower_bounded_int(thelist, lowerbound): + return [int(l) if int(l) >= lowerbound else lowerbound for l in thelist] + +def generate_tumbler_tx(destaddrs, options): + #sends the coins up through a few mixing depths + #send to the destination addresses from different mixing depths + + #simple algo, move coins completely from one mixing depth to the next + # until you get to the end, then send to destaddrs + + #txcounts for going completely from one mixdepth to the next + # follows a normal distribution + txcounts = rand_norm_array(options.txcountparams[0], + options.txcountparams[1], options.mixdepthcount) + txcounts = lower_bounded_int(txcounts, 1) + tx_list = [] + for m, txcount in enumerate(txcounts): + #assume that the sizes of outputs will follow a power law + amount_fractions = rand_pow_array(options.amountpower, txcount) + amount_fractions = [1.0 - x for x in amount_fractions] + amount_fractions = [x/sum(amount_fractions) for x in amount_fractions] + #transaction times are uncorrelated + #time between events in a poisson process followed exp + waits = rand_exp_array(options.timelambda, txcount) + #number of makers to use follows a normal distribution + makercounts = rand_norm_array(options.makercountrange[0], options.makercountrange[1], txcount) + makercounts = lower_bounded_int(makercounts, 2) + for amount_fraction, wait, makercount in zip(amount_fractions, waits, makercounts): + tx = {'amount_fraction': amount_fraction, 'wait': round(wait, 2), + 'srcmixdepth': m + options.mixdepthsrc, 'makercount': makercount, 'destination': 'internal'} + tx_list.append(tx) + + addrask = options.addrcount - len(destaddrs) + external_dest_addrs = ['addrask']*addrask + destaddrs + for mix_offset in range(options.addrcount): + srcmix = options.mixdepthsrc + options.mixdepthcount - mix_offset - 1 + for tx in reversed(tx_list): + if tx['srcmixdepth'] == srcmix: + tx['destination'] = external_dest_addrs[mix_offset] + break + if mix_offset == 0: + #setting last mixdepth to send all to dest + tx_list_remove = [] + for tx in tx_list: + if tx['srcmixdepth'] == srcmix: + if tx['destination'] == 'internal': + tx_list_remove.append(tx) + else: + tx['amount_fraction'] = 1.0 + [tx_list.remove(t) for t in tx_list_remove] + return tx_list + +#thread which does the buy-side algorithm +# chooses which coinjoins to initiate and when +class TumblerThread(threading.Thread): + def __init__(self, taker): + threading.Thread.__init__(self) + self.daemon = True + self.taker = taker + self.ignored_makers = [] + self.sweeping = False + + def unconfirm_callback(self, txd, txid): + debug('that was %d tx out of %d' % (self.current_tx+1, len(self.taker.tx_list))) + + def confirm_callback(self, txd, txid, confirmations): + self.taker.wallet.add_new_utxos(txd, txid) + self.lockcond.acquire() + self.lockcond.notify() + self.lockcond.release() + + def finishcallback(self, coinjointx): + if coinjointx.all_responded: + coinjointx.self_sign_and_push() + common.bc_interface.add_tx_notify(coinjointx.latest_tx, + self.unconfirm_callback, self.confirm_callback, coinjointx.my_cj_addr) + self.taker.wallet.remove_old_utxos(coinjointx.latest_tx) + else: + self.ignored_makers += coinjointx.nonrespondants + debug('recreating the tx, ignored_makers=' + str(self.ignored_makers)) + self.create_tx() + + def tumbler_choose_orders(self, cj_amount, makercount, nonrespondants=[], active_nicks=[]): + self.ignored_makers += nonrespondants + while True: + orders, total_cj_fee = choose_orders(self.taker.db, cj_amount, + makercount, weighted_order_choose, self.ignored_makers + active_nicks) + abs_cj_fee = 1.0*total_cj_fee / makercount + rel_cj_fee = abs_cj_fee / cj_amount + debug('rel/abs average fee = ' + str(rel_cj_fee) + ' / ' + str(abs_cj_fee)) + + if rel_cj_fee > self.taker.maxcjfee[0] and abs_cj_fee > self.taker.maxcjfee[1]: + debug('cj fee higher than maxcjfee, waiting 60 seconds') + time.sleep(60) + continue + if orders == None: + debug('waiting for liquidity 1min, hopefully more orders should come in') + time.sleep(60) + continue + break + debug('chosen orders to fill ' + str(orders) + ' totalcjfee=' + str(total_cj_fee)) + return orders, total_cj_fee + + def create_tx(self): + utxos = None + orders = None + cj_amount = 0 + change_addr = None + choose_orders_recover = None + if self.sweep: + debug('sweeping') + utxos = self.taker.wallet.get_utxos_by_mixdepth()[self.tx['srcmixdepth']] + total_value = sum([addrval['value'] for addrval in utxos.values()]) + while True: + orders, cj_amount = choose_sweep_orders(self.taker.db, total_value, + self.taker.txfee, self.tx['makercount'], weighted_order_choose, + self.ignored_makers) + if orders == None: + debug('waiting for liquidity 1min, hopefully more orders should come in') + time.sleep(60) + continue + abs_cj_fee = 1.0*(total_value - cj_amount) / self.tx['makercount'] + rel_cj_fee = abs_cj_fee / cj_amount + debug('rel/abs average fee = ' + str(rel_cj_fee) + ' / ' + str(abs_cj_fee)) + if rel_cj_fee > self.taker.maxcjfee[0] and abs_cj_fee > self.taker.maxcjfee[1]: + debug('cj fee higher than maxcjfee, waiting 60 seconds') + time.sleep(60) + continue + break + else: + cj_amount = int(self.tx['amount_fraction'] * self.balance) + if cj_amount < self.taker.mincjamount: + debug('cj amount too low, bringing up') + cj_amount = self.taker.mincjamount + change_addr = self.taker.wallet.get_change_addr(self.tx['srcmixdepth']) + debug('coinjoining ' + str(cj_amount) + ' satoshi') + orders, total_cj_fee = self.tumbler_choose_orders(cj_amount, self.tx['makercount']) + total_amount = cj_amount + total_cj_fee + self.taker.txfee + debug('total amount spent = ' + str(total_amount)) + utxos = self.taker.wallet.select_utxos(self.tx['srcmixdepth'], total_amount) + choose_orders_recover = self.tumbler_choose_orders + + self.taker.start_cj(self.taker.wallet, cj_amount, orders, utxos, + self.destaddr, change_addr, self.taker.txfee, + self.finishcallback, choose_orders_recover) + + def init_tx(self, tx, balance, sweep): + destaddr = None + if tx['destination'] == 'internal': + destaddr = self.taker.wallet.get_receive_addr(tx['srcmixdepth'] + 1) + elif tx['destination'] == 'addrask': + common.debug_silence = True + while True: + destaddr = raw_input('insert new address: ') + addr_valid, errormsg = validate_address(destaddr) + if addr_valid: + break + print 'Address ' + destaddr + ' invalid. ' + errormsg + ' try again' + common.debug_silence = False + else: + destaddr = tx['destination'] + self.sweep = sweep + self.balance = balance + self.tx = tx + self.destaddr = destaddr + self.create_tx() + self.lockcond.acquire() + self.lockcond.wait() + self.lockcond.release() + debug('tx confirmed, waiting for ' + str(tx['wait']) + ' minutes') + time.sleep(tx['wait'] * 60) + debug('woken') + + def run(self): + debug('waiting for all orders to certainly arrive') + time.sleep(orderwaittime) + + sqlorders = self.taker.db.execute('SELECT cjfee, ordertype FROM orderbook;').fetchall() + orders = [o['cjfee'] for o in sqlorders if o['ordertype'] == 'relorder'] + orders = sorted(orders) + if len(orders) == 0: + debug('There are no orders at all in the orderbook! Is the bot connecting to the right server?') + return + relorder_fee = float(orders[0]) + debug('relorder fee = ' + str(relorder_fee)) + maker_count = sum([tx['makercount'] for tx in self.taker.tx_list]) + debug('uses ' + str(maker_count) + ' makers, at ' + str(relorder_fee*100) + '% per maker, estimated total cost ' + + str(round((1 - (1 - relorder_fee)**maker_count) * 100, 3)) + '%') + debug('waiting for orders to arrive') + time.sleep(orderwaittime) + debug('starting') + self.lockcond = threading.Condition() + + self.balance_by_mixdepth = {} + for i, tx in enumerate(self.taker.tx_list): + if tx['srcmixdepth'] not in self.balance_by_mixdepth: + self.balance_by_mixdepth[tx['srcmixdepth']] = self.taker.wallet.get_balance_by_mixdepth()[tx['srcmixdepth']] + sweep = True + for later_tx in self.taker.tx_list[i + 1:]: + if later_tx['srcmixdepth'] == tx['srcmixdepth']: + sweep = False + self.current_tx = i + self.init_tx(tx, self.balance_by_mixdepth[tx['srcmixdepth']], sweep) + + debug('total finished') + self.taker.msgchan.shutdown() + + ''' + crow = self.taker.db.execute('SELECT COUNT(DISTINCT counterparty) FROM orderbook;').fetchone() + counterparty_count = crow['COUNT(DISTINCT counterparty)'] + if counterparty_count < self.taker.makercount: + print 'not enough counterparties to fill order, ending' + self.taker.msgchan.shutdown() + return + ''' + + +class Tumbler(takermodule.Taker): + def __init__(self, msgchan, wallet, tx_list, txfee, maxcjfee, mincjamount): + takermodule.Taker.__init__(self, msgchan) + self.wallet = wallet + self.tx_list = tx_list + self.maxcjfee = maxcjfee + self.txfee = txfee + self.mincjamount = mincjamount + self.tumbler_thread = None + + def on_welcome(self): + takermodule.Taker.on_welcome(self) + if not self.tumbler_thread: + self.tumbler_thread = TumblerThread(self) + self.tumbler_thread.start() + +def main(): + parser = OptionParser(usage='usage: %prog [options] [wallet file] [destaddr(s)...]', + description='Sends bitcoins to many different addresses using coinjoin in' + ' an attempt to break the link between them. Sending to multiple ' + ' addresses is highly recommended for privacy. This tumbler can' + ' be configured to ask for more address mid-run, giving the user' + ' a chance to click `Generate New Deposit Address` on whatever service' + ' they are using.') + parser.add_option('-m', '--mixdepthsource', type='int', dest='mixdepthsrc', + help='mixing depth to spend from, default=0', default=0) + parser.add_option('-f', '--txfee', type='int', dest='txfee', + default=10000, help='miner fee contribution, in satoshis, default=10000') + parser.add_option('-x', '--maxcjfee', type='float', dest='maxcjfee', nargs=2, + default=(0.01, 10000), help='maximum coinjoin fee and bitcoin value the tumbler is ' + 'willing to pay to a single market maker. Both values need to be exceeded, so if ' + 'the fee is 30% but only 500satoshi is paid the tx will go ahead. default=0.01, 10000 (1%, 10000satoshi)') + parser.add_option('-a', '--addrcount', type='int', dest='addrcount', + default=3, help='How many destination addresses in total should be used. If not enough are given' + ' as command line arguments, the script will ask for more, default=3') + parser.add_option('-N', '--makercountrange', type='float', nargs=2, action='store', + dest='makercountrange', + help='Input the mean and spread of number of makers to use. e.g. 3 1.5 will be a normal distribution ' + 'with mean 3 and standard deveation 1.5 inclusive, default=3 1.5', default=(3, 1.5)) + parser.add_option('-M', '--mixdepthcount', type='int', dest='mixdepthcount', + help='How many mixing depths to mix through', default=4) + parser.add_option('-c', '--txcountparams', type='float', nargs=2, dest='txcountparams', default=(4, 1), + help='The number of transactions to take coins from one mixing depth to the next, it is' + ' randomly chosen following a normal distribution. Should be similar to --addrask. ' + 'This option controlls the parameters of that normal curve. (mean, standard deviation). default=(4, 1)') + parser.add_option('--amountpower', type='float', dest='amountpower', default=100.0, + help='The output amounts follow a power law distribution, this is the power, default=100.0') + parser.add_option('-l', '--timelambda', type='float', dest='timelambda', default=20, + help='Average the number of minutes to wait between transactions. Randomly chosen ' + ' following an exponential distribution, which describes the time between uncorrelated' + ' events. default=20') + parser.add_option('-w', '--wait-time', action='store', type='float', dest='waittime', + help='wait time in seconds to allow orders to arrive, default=5', default=5) + parser.add_option('-s', '--mincjamount', type='int', dest='mincjamount', default=100000, + help='minimum coinjoin amount in transaction in satoshi, default 100k') + (options, args) = parser.parse_args() + #TODO somehow implement a lower limit + + if len(args) < 1: + parser.error('Needs a wallet file') + sys.exit(0) + wallet_file = args[0] + destaddrs = args[1:] + + common.load_program_config() + for addr in destaddrs: + addr_valid, errormsg = validate_address(addr) + if not addr_valid: + print 'ERROR: Address ' + addr + ' invalid. ' + errormsg + return + + if len(destaddrs) > options.addrcount: + options.addrcount = len(destaddrs) + if options.addrcount+1 > options.mixdepthcount: + print 'not enough mixing depths to pay to all destination addresses, increasing mixdepthcount' + options.mixdepthcount = options.addrcount+1 + if options.addrcount <= 1: + print '='*50 + print 'WARNING: You are only using one destination address' + print 'this is very bad for privacy' + print '='*50 + + print str(options) + tx_list = generate_tumbler_tx(destaddrs, options) + if not tx_list: + return + + tx_list2 = copy.deepcopy(tx_list) + tx_dict = {} + for tx in tx_list2: + srcmixdepth = tx['srcmixdepth'] + tx.pop('srcmixdepth') + if srcmixdepth not in tx_dict: + tx_dict[srcmixdepth] = [] + tx_dict[srcmixdepth].append(tx) + dbg_tx_list = [] + for srcmixdepth, txlist in tx_dict.iteritems(): + dbg_tx_list.append({'srcmixdepth': srcmixdepth, 'tx': txlist}) + debug('tumbler transaction list') + pprint(dbg_tx_list) + + total_wait = sum([tx['wait'] for tx in tx_list]) + print 'creates ' + str(len(tx_list)) + ' transactions in total' + print 'waits in total for ' + str(len(tx_list)) + ' blocks and ' + str(total_wait) + ' minutes' + total_block_and_wait = len(tx_list)*10 + total_wait + print('estimated time taken ' + str(total_block_and_wait) + + ' minutes or ' + str(round(total_block_and_wait/60.0, 2)) + ' hours') + + ret = raw_input('tumble with these tx? (y/n):') + if ret[0] != 'y': + return + + #NOTE: possibly out of date documentation + #a couple of modes + #im-running-from-the-nsa, takes about 80 hours, costs a lot + #python tumbler.py -a 10 -N 10 5 -c 10 5 -l 50 -M 10 wallet_file 1xxx + # + #quick and cheap, takes about 90 minutes + #python tumbler.py -N 2 1 -c 3 0.001 -l 10 -M 3 -a 1 wallet_file 1xxx + # + #default, good enough for most, takes about 5 hours + #python tumbler.py wallet_file 1xxx + # + #for quick testing + #python tumbler.py -N 2 1 -c 3 0.001 -l 0.1 -M 3 -a 0 wallet_file 1xxx 1yyy + wallet = Wallet(wallet_file, max_mix_depth = options.mixdepthsrc + options.mixdepthcount) + common.bc_interface.sync_wallet(wallet) + + #check if there are actually any coins at the mixdepthsrc level + #if not, allow user to start at minimum used level + used_depths = [k for k,v in wallet.get_utxos_by_mixdepth().iteritems() if v != {}] + if options.mixdepthsrc not in used_depths: + ret = raw_input("no coins in chosen src level. Use lowest possible level? (y/n):") + if ret[0] !='y': + return + options.mixdepthsrc = sorted(used_depths.keys())[0] + print "starting with depth: " + str(options.mixdepthsrc) + wallet = Wallet(wallet_file,max_mix_depth = options.mixdepthsrc + options.mixdepthcount) + common.bc_interface.sync_wallet(wallet) + + + + common.nickname = random_nick() + debug('starting tumbler') + irc = IRCMessageChannel(common.nickname) + tumbler = Tumbler(irc, wallet, tx_list, options.txfee, options.maxcjfee, options.mincjamount) + try: + debug('connecting to irc') + irc.run() + except: + debug('CRASHING, DUMPING EVERYTHING') + debug_dump_object(wallet, ['addr_cache', 'keys', 'seed']) + debug_dump_object(tumbler) + debug_dump_object(tumbler.cjtx) + import traceback + debug(traceback.format_exc()) + + +if __name__ == "__main__": + main() + print('done') + diff --git a/wallet-tool.py b/wallet-tool.py new file mode 100644 index 00000000..3f13c692 --- /dev/null +++ b/wallet-tool.py @@ -0,0 +1,133 @@ + +import sys, os +import getpass, json, datetime +from optparse import OptionParser +data_dir = os.path.dirname(os.path.realpath(__file__)) +sys.path.insert(0, os.path.join(data_dir, 'lib')) + +import bitcoin as btc +from common import Wallet, load_program_config, get_p2pk_vbyte +import common +import old_mnemonic, slowaes + +#structure for cj market wallet +# m/0/ root key +# m/0/n/ nth mixing depth, where n=0 is unmixed, n=1 is coinjoined once, etc +# pay in coins to mix at n=0 addresses +# coins move up a level when they are cj'd and stay at same level if they're the change from a coinjoin +# using coins from different levels as inputs to the same tx is probably detrimental to privacy +# m/0/n/0/k kth receive address, for mixing depth n +# m/0/n/1/k kth change address, for mixing depth n + + +parser = OptionParser(usage='usage: %prog [options] [wallet file] [method]', + description='Does useful little tasks involving your bip32 wallet. The' + + ' method is one of the following: display- shows addresses and balances.' + + ' displayall - shows ALL addresses and balances.' + + ' summary - shows a summary of mixing depth balances.' + + ' generate - generates a new wallet.' + + ' recover - recovers a wallet from the 12 word recovery seed.' + + ' showseed - shows the wallet recovery seed and hex seed.') +parser.add_option('-p', '--privkey', action='store_true', dest='showprivkey', + help='print private key along with address, default false') +parser.add_option('-m', '--maxmixdepth', action='store', type='int', dest='maxmixdepth', + help='maximum mixing depth to look for, default=5') +parser.add_option('-g', '--gap-limit', type="int", action='store', dest='gaplimit', + help='gap limit for wallet, default=6', default=6) +(options, args) = parser.parse_args() + +#if the index_cache stored in wallet.json is longer than the default +#then set maxmixdepth to the length of index_cache +maxmixdepth_configured = True +if not options.maxmixdepth: + maxmixdepth_configured = False + options.maxmixdepth = 5 + +noseed_methods = ['generate', 'recover'] +methods = ['display', 'displayall', 'summary'] + noseed_methods + +if len(args) < 1: + parser.error('Needs a wallet file or method') + sys.exit(0) +load_program_config() + +if args[0] in noseed_methods: + method = args[0] +else: + seed = args[0] + method = ('display' if len(args) == 1 else args[1].lower()) + wallet = Wallet(seed, options.maxmixdepth, options.gaplimit, extend_mixdepth=not maxmixdepth_configured) + if method != 'showseed': + common.bc_interface.sync_wallet(wallet) + +if method == 'display' or method == 'displayall': + total_balance = 0 + for m in range(wallet.max_mix_depth): + print 'mixing depth %d m/0/%d/' % (m, m) + balance_depth = 0 + for forchange in [0, 1]: + print(' ' + ('receive' if forchange==0 else 'change') + + ' addresses m/0/%d/%d/' % (m, forchange)) + for k in range(wallet.index[m][forchange] + options.gaplimit): + addr = wallet.get_addr(m, forchange, k) + balance = 0.0 + for addrvalue in wallet.unspent.values(): + if addr == addrvalue['address']: + balance += addrvalue['value'] + balance_depth += balance + used = ('used' if k < wallet.index[m][forchange] else ' new') + privkey = btc.encode_privkey(wallet.get_key(m, forchange, k), 'wif_compressed', + get_p2pk_vbyte()) if options.showprivkey else '' + if method == 'displayall' or balance > 0 or (used == ' new' and forchange==0): + print ' m/0/%d/%d/%03d %-35s%s %.8f btc %s' % (m, forchange, k, addr, used, balance/1e8, privkey) + print 'for mixdepth=%d balance=%.8fbtc' % (m, balance_depth/1e8) + total_balance += balance_depth + print 'total balance = %.8fbtc' % (total_balance/1e8) +elif method == 'summary': + total_balance = 0 + for m in range(wallet.max_mix_depth): + balance_depth = 0 + for forchange in [0, 1]: + for k in range(wallet.index[m][forchange]): + addr = wallet.get_addr(m, forchange, k) + for addrvalue in wallet.unspent.values(): + if addr == addrvalue['address']: + balance_depth += addrvalue['value'] + print 'for mixdepth=%d balance=%.8fbtc' % (m, balance_depth/1e8) + total_balance += balance_depth + print 'total balance = %.8fbtc' % (total_balance/1e8) +elif method == 'generate' or method == 'recover': + if method == 'generate': + seed = btc.sha256(os.urandom(64))[:32] + words = old_mnemonic.mn_encode(seed) + print 'Write down this wallet recovery seed\n\n' + ' '.join(words) + '\n' + elif method == 'recover': + words = raw_input('Input 12 word recovery seed: ') + words = words.split() #default for split is 1 or more whitespace chars + if len(words) != 12: + print 'ERROR. Recovery seed phrase must be exactly 12 words.' + sys.exit(0) + seed = old_mnemonic.mn_decode(words) + print seed + password = getpass.getpass('Enter wallet encryption passphrase: ') + password2 = getpass.getpass('Reenter wallet encryption passphrase: ') + if password != password2: + print 'ERROR. Passwords did not match' + sys.exit(0) + password_key = btc.bin_dbl_sha256(password) + encrypted_seed = slowaes.encryptData(password_key, seed.decode('hex')) + timestamp = datetime.datetime.now().strftime("%Y/%m/%d %H:%M:%S") + walletfile = json.dumps({'creator': 'joinmarket project', 'creation_time': timestamp, + 'encrypted_seed': encrypted_seed.encode('hex'), 'network': common.get_network()}) + walletname = raw_input('Input wallet file name (default: wallet.json): ') + if len(walletname) == 0: + walletname = 'wallet.json' + fd = open(os.path.join('wallets', walletname), 'w') + fd.write(walletfile) + fd.close() + print 'saved to ' + walletname +elif method == 'showseed': + hexseed = wallet.seed + print 'hexseed = ' + hexseed + words = old_mnemonic.mn_encode(hexseed) + print 'Wallet recovery seed\n\n' + ' '.join(words) + '\n' diff --git a/wallets/.gitignore b/wallets/.gitignore new file mode 100644 index 00000000..86d0cb27 --- /dev/null +++ b/wallets/.gitignore @@ -0,0 +1,4 @@ +# Ignore everything in this directory +* +# Except this file +!.gitignore \ No newline at end of file diff --git a/yield-generator.py b/yield-generator.py new file mode 100644 index 00000000..caf65843 --- /dev/null +++ b/yield-generator.py @@ -0,0 +1,158 @@ +#! /usr/bin/env python + +import time, os, binascii, sys, datetime +import pprint +data_dir = os.path.dirname(os.path.realpath(__file__)) +sys.path.insert(0, os.path.join(data_dir, 'lib')) + +from maker import * +from irc import IRCMessageChannel, random_nick +import bitcoin as btc +import common, blockchaininterface + +from socket import gethostname + +txfee = 1000 +cjfee = '0.002' # 0.2% fee +nickname = random_nick() +nickserv_password = '' +minsize = int(1.2 * txfee / float(cjfee)) #minimum size is such that you always net profit at least 20% of the miner fee +mix_levels = 5 + + + +#is a maker for the purposes of generating a yield from held +# bitcoins without ruining privacy for the taker, the taker could easily check +# the history of the utxos this bot sends, so theres not much incentive +# to ruin the privacy for barely any more yield +#sell-side algorithm: +#add up the value of each utxo for each mixing depth, +# announce a relative-fee order of the highest balance +#spent from utxos that try to make the highest balance even higher +# so try to keep coins concentrated in one mixing depth +class YieldGenerator(Maker): + statement_file = os.path.join('logs', 'yigen-statement.csv') + + def __init__(self, msgchan, wallet): + Maker.__init__(self, msgchan, wallet) + self.msgchan.register_channel_callbacks(self.on_welcome, self.on_set_topic, + None, None, self.on_nick_leave, None) + self.tx_unconfirm_timestamp = {} + + def log_statement(self, data): + if common.get_network() == 'testnet': + return + + data = [str(d) for d in data] + self.income_statement = open(self.statement_file, 'a') + self.income_statement.write(','.join(data) + '\n') + self.income_statement.close() + + def on_welcome(self): + Maker.on_welcome(self) + if not os.path.isfile(self.statement_file): + self.log_statement(['timestamp', 'cj amount/satoshi', 'my input count', + 'my input value/satoshi', 'cjfee/satoshi', 'earned/satoshi', + 'confirm time/min', 'notes']) + + timestamp = datetime.datetime.now().strftime("%Y/%m/%d %H:%M:%S") + self.log_statement([timestamp, '', '', '', '', '', '', 'Connected']) + + def create_my_orders(self): + mix_balance = self.wallet.get_balance_by_mixdepth() + if len([b for m, b in mix_balance.iteritems() if b > 0]) == 0: + debug('do not have any coins left') + return [] + + #print mix_balance + max_mix = max(mix_balance, key=mix_balance.get) + order = {'oid': 0, 'ordertype': 'relorder', 'minsize': minsize, + 'maxsize': mix_balance[max_mix] - common.DUST_THRESHOLD, 'txfee': txfee, 'cjfee': cjfee} + return [order] + + def oid_to_order(self, cjorder, oid, amount): + mix_balance = self.wallet.get_balance_by_mixdepth() + max_mix = max(mix_balance, key=mix_balance.get) + + #algo attempts to make the largest-balance mixing depth get an even larger balance + debug('finding suitable mixdepth') + mixdepth = (max_mix - 1) % self.wallet.max_mix_depth + while True: + if mixdepth in mix_balance and mix_balance[mixdepth] >= amount: + break + mixdepth = (mixdepth - 1) % self.wallet.max_mix_depth + #mixdepth is the chosen depth we'll be spending from + cj_addr = self.wallet.get_receive_addr((mixdepth + 1) % self.wallet.max_mix_depth) + change_addr = self.wallet.get_change_addr(mixdepth) + + utxos = self.wallet.select_utxos(mixdepth, amount) + my_total_in = sum([va['value'] for va in utxos.values()]) + real_cjfee = calc_cj_fee(cjorder.ordertype, cjorder.cjfee, amount) + change_value = my_total_in - amount - cjorder.txfee + real_cjfee + if change_value <= common.DUST_THRESHOLD: + debug('change value=%d below dust threshold, finding new utxos' % (change_value)) + try: + utxos = self.wallet.select_utxos(mixdepth, amount + common.DUST_THRESHOLD) + except Exception: + debug('dont have the required UTXOs to make a output above the dust threshold, quitting') + return None, None, None + + return utxos, cj_addr, change_addr + + def on_tx_unconfirmed(self, cjorder, txid, removed_utxos): + self.tx_unconfirm_timestamp[cjorder.cj_addr] = int(time.time()) + #if the balance of the highest-balance mixing depth change then reannounce it + oldorder = self.orderlist[0] if len(self.orderlist) > 0 else None + neworders = self.create_my_orders() + if len(neworders) == 0: + return ([0], []) #cancel old order + if oldorder: #oldorder may not exist when this is called from on_tx_confirmed + if oldorder['maxsize'] == neworders[0]['maxsize']: + return ([], []) #change nothing + #announce new order, replacing the old order + return ([], [neworders[0]]) + + def on_tx_confirmed(self, cjorder, confirmations, txid): + confirm_time = int(time.time()) - self.tx_unconfirm_timestamp[cjorder.cj_addr] + timestamp = datetime.datetime.now().strftime("%Y/%m/%d %H:%M:%S") + self.log_statement([timestamp, cjorder.cj_amount, len(cjorder.utxos), + sum([av['value'] for av in cjorder.utxos.values()]), cjorder.real_cjfee, + cjorder.real_cjfee - cjorder.txfee, round(confirm_time / 60.0, 2), '']) + return self.on_tx_unconfirmed(cjorder, txid, None) + +def main(): + common.load_program_config() + import sys + seed = sys.argv[1] + if isinstance(common.bc_interface, blockchaininterface.BlockrInterface): + print '\nYou are running a yield generator by polling the blockr.io website' + print 'This is quite bad for privacy. That site is owned by coinbase.com' + print 'Also your bot will run faster and more efficently, you can be immediately notified of new bitcoin network' + print ' information so your money will be working for you as hard as possible' + print 'Learn how to setup JoinMarket with Bitcoin Core: https://github.com/chris-belcher/joinmarket/wiki/Running-JoinMarket-with-Bitcoin-Core-full-node' + ret = raw_input('\nContinue? (y/n):') + if ret[0] != 'y': + return + + wallet = Wallet(seed, max_mix_depth = mix_levels) + common.bc_interface.sync_wallet(wallet) + + common.nickname = nickname + debug('starting yield generator') + irc = IRCMessageChannel(common.nickname, realname='btcint=' + common.config.get("BLOCKCHAIN", "blockchain_source"), + password=nickserv_password) + maker = YieldGenerator(irc, wallet) + try: + debug('connecting to irc') + irc.run() + except: + debug('CRASHING, DUMPING EVERYTHING') + debug_dump_object(wallet, ['addr_cache', 'keys', 'seed']) + debug_dump_object(maker) + debug_dump_object(irc) + import traceback + debug(traceback.format_exc()) + +if __name__ == "__main__": + main() + print('done')