How can I get the list of all bitcoin transactions?
I am currently thinking of doing a research project on Bitcoins. I want to download the list of all transactions of bitcoins. I was wondering what are the ways to do this? I believe I can scrape, but is there a better and faster way to do this?
5 Answers 5
The list of all transactions is the blockchain. The fastest way to get the blockchain is to run a full node which will download the blockchain for you and provide ways for you to access the data stored within the blockchain.
You can use Bitcoin Core and enable the transaction index so that you can get all transaction information. The transaction index is enabled by adding txindex=1 to your bitcoin.conf file.
If you’re looking for all BTC balances, this github repository looks useful. It would likely crash due to memory issues, but could show you all transactions using the «transactions» argument. https://github.com/znort987/blockparser
There is a new tool by a Princeton research group that does exactly this: https://github.com/citp/BlockSci
If you already download (sync) the full blockchain database with Bitcoin Core, the fastest way to get the full list of all transactions is to use blockchain parser.
This script parses raw blockchain database which is stored in blk******.dat files and can extract all the data. If you need just TX hashes, you can leave only them as output information and just comment all the fields that you don’t need to be in the output text files.
This is the most easiest way to do what you need.
Download the bitcoin core bootstrap.dat
Most downloads are compressed to save on download time.
Источник
How can I get the list of all bitcoin transactions?
I am currently thinking of doing a research project on Bitcoins. I want to download the list of all transactions of bitcoins. I was wondering what are the ways to do this? I believe I can scrape, but is there a better and faster way to do this?
5 Answers 5
The list of all transactions is the blockchain. The fastest way to get the blockchain is to run a full node which will download the blockchain for you and provide ways for you to access the data stored within the blockchain.
You can use Bitcoin Core and enable the transaction index so that you can get all transaction information. The transaction index is enabled by adding txindex=1 to your bitcoin.conf file.
If you’re looking for all BTC balances, this github repository looks useful. It would likely crash due to memory issues, but could show you all transactions using the «transactions» argument. https://github.com/znort987/blockparser
There is a new tool by a Princeton research group that does exactly this: https://github.com/citp/BlockSci
If you already download (sync) the full blockchain database with Bitcoin Core, the fastest way to get the full list of all transactions is to use blockchain parser.
This script parses raw blockchain database which is stored in blk******.dat files and can extract all the data. If you need just TX hashes, you can leave only them as output information and just comment all the fields that you don’t need to be in the output text files.
This is the most easiest way to do what you need.
Download the bitcoin core bootstrap.dat
Most downloads are compressed to save on download time.
Источник
Database Dumps
For some tasks like extracting lots of blockchain data (e.g. all transactions over a 2 month period) it’s better to use our dumps feature instead of using our API or other services. It’s possible to download the entire database dumps in TSV format and insert the data onto your own database server (like Postgresql or whatever) to further analyze it. Dumps are updated daily. The total amount of data available is over 1 Tb (compressed)!
Here’s the list of available dumps:
Our database dumps are available here: https://gz.blockchair.com, field descriptions are available here.
You can download 1 file at a time and the speed is limited to 10 kB/s. If you are a business or an academia/non-profit organisation that requires a higher speed, please contact us at .
Feel free to use our API as well. It allows to filter and sort samples and get aggregated data (for example, the average fee for transaction grouped by day or month) without downloading the entire set.
Full Node Dumps
Since syncing a full node can take quite some time (for example, syncing a full archival Ethereum node takes more than a month) we decided to make it possible to download data snapshots. Once you’ve downloaded a snapshot you’d need just to put it to a correct data folder and run the node.
Here is the list of snapshots we’re providing:
Our node dumps are available here: https://tar.blockchair.com
You can download 1 file at a time and the speed is limited to 10 kB/s. If you are a business or an academia/non-profit organisation that requires a higher speed, please contact us at .
Источник
Download all bitcoin transactions
bitcoin-transactions — Discover and move your coins by yourself
Javascript implementation of the Bitcoin protocol for any Bitcoin based coins, on server and inside browsers as a standalone offline webapp, this module offers all the tools to manage your addresses and transactions by yourself without the need to sync a full node and/or use a wallet that you don’t trust
Funding and license
Therefore the initial dev fees have been removed and the code is open source and provided in clear under a MIT license
This module is using the very good Elliptic, BS58, BECH32, Forge SHA256, Browserify, Terser and other modules from us under a MIT license
The browser version is here
In case you don’t see how to use it for some parts, please refer to the documentation and examples here
Installation (nodejs version)
Install nodejs for Windows,Mac,Linux, usually this is easy
Create a bitcoin-transactions directory, download and unzip master
Support and supported coins
If you experience some issues with this module or don’t feel comfortable to use it by yourself, or just don’t know how to use it, please email at our github address or email contact peersm com, or post an issue
If you don’t know very well how to find your transactions, your can refer to the initial doc
The list of supported coins is here, as you can see the tool does support a lot of coins, most of them not being serious, we do not intend to support additional non serious coins
Example: node tx.js BTC createwallet 4ecf2e71d567072fe2f9cda40873afcaae4224e3f249018621a90dd43e88f8de 100 «m/44’/0’/0’/0/500» «>
is the type of coin, is a BIP32 seed (32 bytes) or a xprv derived seed, nb is the number of addresses to be generated, path is the derivation path (it defaults to the standard path or BIP44 path), the «‘» stands for hardened addresses
You can also use create_wallet where secret is an already generated hd object to iterate on it
Since we did implement it before Zcash team, we kept our implementation for z-addresses, now please note that this is probably not following the BIP32 Zcash implementation
For a really detailed explanation of how transactions work please refer to the initial documentation above
The general syntax is:
node tx.js create prevtx=tx1_tx2. prevaddr=prevaddr1_prevaddr2_. prevamount=amount1_amount2_. previndex=index1_index2. privkey=privkey1_privKey2 addr=addr1_addr2. fees=fees amount=amount1_amount2 optional
- prevtx are the transactions ids of the outputs to be spent
- prevaddr are the addresses of the outputs to be spent
- amount are the amounts of the outputs to be spent
- previndex are the indexes of the outputs to be spent (warning: it starts with zero)
- privkey are the private keys or redeem scripts corresponding to prevaddr
- addr are the destination addresses
- fees are the network fees
- amount are the amounts to be sent, the delta will be refunded to the first prevaddr address, if omitted the total of prevamounts minus the network fees will go to the destination address, if there is several destination addresses, there must be the same number of amounts, same as before in that case if there is a delta for the sum of amounts + fees compared to the total of prevamounts, it will be refunded to the first prevaddr
If prevaddr corresponds to a segwit address, you must use prevaddr=prevaddr1-segwit_prevaddr2-segwit_.
In that case, if prevaddr is not a bech32 address it will create a segwit «nested into p2sh» transaction, if not it will create a legacy segwit transaction
Same thing applies if a destination address is a bech32 one or not
The best is to refer to all the examples in test vectors and test vectors multiple addresses
The calculation for the fees is:
You must choose the fees according to the rules of the coin used, do not go below
1 satoshi per byte or your transaction will not be accepted by the network, due to rounding issues there can be one satoshi floating around
If your coins are on a segwit address you must add the -segwit string in prevaddr:
Same as above with a _ separator for all fields, if the data is the same (for example all outputs corresponding to the same address), then just put it once not using _
You can mix all kind of transactions using the _ separator, please see the last example in test vectors
Sending your transaction
The difference between complete transaction and body in the output of the create command is just that complete transaction includes the body with the network headers:
node tx.js send complete transaction advised full node (as shown in here section)
paste the body (and not the complete transaction ) in an explorer (example: https://btgexplorer.com/tx/send or https://explorer.btcprivate.org/tx/send)
If you used the first method you can also check that your transaction is in mempool (see in the code and uncomment)
The tool double checks many times many things, now if you make a mistake with the destination address it can’t detect it, be sure to use an address that you master
The tool can detect mistakes made with the amounts also but not always, be carefull for example not to send everything as network fees
Check carefully the output of the create command, you will see the details of the transaction, you MUST at the end get «Transaction verified» AND «serialize/deserialize OK», if you don’t get this then something is wrong with your parameters
Note: once an output corresponding to an address is spent then the corresponding public key or redeem script becomes public, this feature allows you to see it, this eliminates one check to validate further transactions involving this address, that’s why it is not recommended to reuse same addresses
See Sign/Verify messages below also
This will convert the address from coin1 format to coin2 format
Decode redeem scripts
To find the keys corresponding to a m of n redeem script, you can run:
Create redeem scripts
Probably of no use except for testing purposes:
BIP32: defines how to derive P2PK keys from a 32 bytes seed according to a path (see createwallet above)
BIP39: allows you to derive a seed from n words (where n is a multiple of 3) chosen from a list, there is a non mandatory checksum but probably enforced by every wallets, the seed is derived first according to BIP32, then derived from the path chosen
BIP44: defines the path to derive a BIP39 seed for each coin
BIP49: defines the path to derive a BIP39 seed to segwit nested addresses for each coin
BIP84: defines the path to derive a BIP39 seed to bech32 addresses for each coin
BIP141: segwit specs
The legacy derivation path m/0’/0’/0′ defined by bitcoin core is used by default for BIP32 and BIP141, use the ‘bip44’ tag to change it to bip44 path
See the test vectors in BIPs
BIP39 words recovery
The tool allows you to recover one or two words in a BIP39 sequence in case you lost them (or to get the right words to have a correct checksum), two words still makes too many possibilities so most likely you should use this feature with only one word
m is the position of the first missing word, n the one of the second one (if relevant)
BIP39 seeds generation
We do not advise to use this feature since javascript prng still can be weak, instead we would recommend to use a BIP32 seed from for example QRNG go to Live Numbers—>Live streams—>Hex and copy 64 digits (ie a 32B hex seed)
nb is the number of words
Get public key from private
Get private from WIF and vice-versa
Public to addresses and hash
This is used to prove ownership of a given address
Be careful anyway not to over use it, or do it privately, because once you have proven the ownership of a given address, then the corresponding public key becomes public, see decode transactions above
Each coin uses a prefix appended to the message to be signed (generally the string ‘Bitcoin signed message:\n’), we have set the correct prefix for the main coins but did not check for all of them
type is: ‘n’ for compressed key, ‘s’ for segwit, ‘b’ for bech32 type is in fact of no use and not relevant for this module, therefore ignored by verify (but set correctly for signing), but it can be checked by other tools node tx.js BTC signmessage ‘Thanks Ayms this module is great!’ Kyib9iMhJxL6Zh1srtz3caTAqXhP5gsETuUBiQZEmFBrU3KAkiAg n Signature : Hw/PE+L4GW8C4S+V/6rgDZx9UHHU8hVXe8knNQQIFl73b98EWD8C1/lXA6uMOS5jJTLTXDf2t2a5zUkvOdTEQt4= node tx.js BTC verifymessage ‘Thanks Ayms this module is great!’ ‘Hw/PE+L4GW8C4S+V/6rgDZx9UHHU8hVXe8knNQQIFl73b98EWD8C1/lXA6uMOS5jJTLTXDf2t2a5zUkvOdTEQt4=’ 18v29GccczH8nxPVm3zQ1xVaxV5Wh4Yz9v Signature verified — Public key 038c71760a4aac81afd4a314dc989c1a0621d28aac584992a26d1993709bdafddb node tx.js BTC verifymessage ‘Thanks Ayms this module is great!’ ‘Hw/PE+L4GW8C4S+V/6rgDZx9UHHU8hVXe8knNQQIFl73b98EWD8C1/lXA6uMOS5jJTLTXDf2t2a5zUkvOdTEQt4=’ 3NqxkcqbasHnTxQ6hHyMrJv5kC1D4o3jTC Signature verified — Public key 038c71760a4aac81afd4a314dc989c1a0621d28aac584992a26d1993709bdafddb node tx.js BTC verifymessage ‘Thanks Ayms this module is great!’ ‘Hw/PE+L4GW8C4S+V/6rgDZx9UHHU8hVXe8knNQQIFl73b98EWD8C1/lXA6uMOS5jJTLTXDf2t2a5zUkvOdTEQt4=’ bc1q2m87z4h0mwxdqjuetmgf95tdc9wmdqz7tg4540 Signature verified — Public key 038c71760a4aac81afd4a314dc989c1a0621d28aac584992a26d1993709bdafddb node tx.js BCH verifymessage ‘Thanks Ayms this module is great!’ ‘Hw/PE+L4GW8C4S+V/6rgDZx9UHHU8hVXe8knNQQIFl73b98EWD8C1/lXA6uMOS5jJTLTXDf2t2a5zUkvOdTEQt4=’ qptvlc2kaldce5ztn90dpyk3dhq4md5qtc6jk64fjm Signature verified — Public key 038c71760a4aac81afd4a314dc989c1a0621d28aac584992a26d1993709bdafddb node tx.js ZEC verifymessage ‘I am a signed Zcash message’ ‘IO5jX5/RhpIqbo52uOs1d9g0D5+Al4cpzyWyEitZYO24HFZbb6lI94k+X7LwhqmaEi6eorUzpDSG2JPArKTN1EU=’ t1LF5Se66f3uRFPFLC86FSRBasbLqTbz86k Signature verified — Public key 0259a77bc3c5621b65e557a6cc83a45a8fc4a9aa5a8908ae8e9664f41cd345842f «>
Notes for the devs
The module is not trivial, any slight change can cause everything not to work any longer and can become difficult to identify at a certain point of time, therefore it is advised to periodically test the latest example in the multisig test vectors while making changes (if this one passes it’s unlikely that something is wrong). In addition serialize/deserialize is not symetrical then true must be used when not coming from a Tx constructor transaction, initially this was a mistake due to misreading of segwit specs, we decided to keep it like this because double checking symetrical potential wrong things is just useless, therefore this adds a last serialize/deserialize independant check
Browserification process and bugs
The User Interface code and the complete browserified javascript code is here
There are probably some details to fix in the UI, for minor ones please email us, for others please post a bug or email
Before posting or emailing something please open the browser javascript console and send the content to us at the same time if it shows some js errors
Related external bugs:
- https://github.com/indutny/bn.js/issues/227 this causes BN code to be included twice by browserify
- https://github.com/mishoo/UglifyJS2/issues/3443 surprisingly uglify does not support ES6
About
Javascript implementation of the Bitcoin protocol for any Bitcoin based coins, on server and inside browsers, discover and move/manage your coins by yourself: Bitcoin transactions made simple for standard or multisig wallets, segwit and bech32 are supported, create and send by your own your Bitcoin, Bitcoin Cash, Zcash, Litecoin, DOGE, Dash, etc…
Источник