yGTHn9Mq

1Ethereum programming for web developers
2by Jon Evans, CTO, HappyFunCorp
3jon@happyfuncorp.com
4Hello, fellow web developer! If you're reading this, you're probably
5interested in blockchains, smart contracts, etc., as someone who
6actually wants to write some smart-contract code. I'm going to walk
7you through setting up, writing, and deploying a smart contract to a
8real live Ethereum blockchain, and then interacting with that contract
9in a browser via a web service.
10The Ethereum Virtual Machine
11I'm not going to explain Blockchains 101 or Ethereum 101: there are
12many other places to go for that. But it's probably worth discussing
13Ethereum at a very high level from a developer's perspective.
14You don't need to care about mining or Proof-of-Work vs. Proof-ofStake,
15or anything like that. But you should know that Ethereum is a
16decentralized virtual machine that runs on many nodes scattered
17around the world, and so-called "smart contracts" are code which runs
18(along with data which is stored) within that virtual machine, i.e. on
19every single node.
20This is obviously hugely inefficient, but it has advantages; everyone in
21the world can rely on this code/data, because no central service or
22system can tamper with it; and anyone can submit code/data to this
23machine without the registering or asking permission. They do,
24however, need to pay. Every line of code and byte of storage in
25Ethereum has a price.
26Ethereum, like Bitcoin, has a native currency, called "ether"; this is the
27same Ether currency that is traded on exchanges like Coinbase. When
28used to pay for Ethereum computing/storage, it is called "gas." For any
29given smart contract, gas has a "limit" and a "price." This is pretty
30confusing at first, but don't worry, you'll wrap your head around it
31eventually, and anyway this tutorial uses free fake money on a socalled
32"testnet" Ethereum blockchain.
33Blockchain Languages
34In principle many languages can be compiled down to the bytecode
35used by the Ethereum VM, but in practice almost all smart contracts
36are written in the "Ethereum-native" language called Solidity. Solidity
37is still arguably somewhere between alpha-release and beta-release
38quality, and has lots of ... idiosyncracies. (See this scathing
39commentary from six months ago: https://news.ycombinator.com/item?
40id=14691212) Still, it remains the de facto state of the art.
41Solidity reads a bit like Javascript. It has a lot of quirks and pitfalls,
42though, especially when it comes to moving money around. Be very
43careful if writing real money-transfer code; do your security homework,
44get others to review your code, and seriously consider an official
45security audit or even formal verification. Again, this tutorial uses
46fake/test money, not real money.
47Solidity code runs on the blockchain. But talking to the Ethereum
48blockchain from external code -- like, say, a web server -- is another
49matter. In theory, you could roll your own JSON-RPC calls; in practice,
50you want to use an existing library. The de facto state of the art here is
51a Javascript library called "web3". I'm more of a Ruby / Python / Go
52server-sider myself, but (for better or worse) JS is widely used and
53understood by web developers everywhere, so much of this tutorial is
54written in Node. Reminder: "the web framework named Node" and "an
55Ethereum node" are two completely different uses of the same word.
56You have been warned.
57Most Solidity tutorials assume you're running an Ethereum node on
58your machine, and/or one or more browser plugins. These steps add
59complexity & cognitive overhead and are not actually necessary. This
60tutorial is for web developers, who are accustomed to talking to APIs
61running on external servers; it will get your smart contracts up and
62running, on a real live blockchain, without ever needing to run an
63Ethereum node yourself.
64Overview
65Let's quickly go over what we're about to cover:
661. Selecting an Ethereum testnet; creating an address; storing its
67private key; financing it with fake money
682.Installing and understanding the Truffle development environment for
69Solidity
703.Writing your first smart contract
714. Testing your first smart contract
725. Deploying your first smart contract
736. Talking to your smart contract from a Node web service
747. End-to-end browser <> Node <> blockchain <> Node <> database data
75flow.
76Also, I'm a believer in entire tutorials contained within a single
77document, which this is. That said, you can also go look at the Git
78repository containing the final code at
79https://github.com/HappyFunCorp/Webthereum and witness it up and
80running (crudely) on Heroku at https://webthereum.herokuapp.com/ --
81though please please read the security discussion at the very end of
82this document first, or, at least, don't enter real private keys into that
83app. (It would probably be safe to do so -- enforced HTTPS connection,
84they're never saved on the server side, etc. -- but that's not the point.)
851. Testnet setup
86The "real" Ethereum blockchain is protected by a vast network of
87miners securing its transactions with billions of hashes per second.
88However there are also "testnet" blockchains, which are either less
89secure or are "private" i.e. controlled by a small subset of
90permissioned miners. We will use the Rinkeby (https://www.rinkeby.io/)
91testnet, an Ethereum blockchain that anyone can connect to remotely,
92courtesy of the fine folks at Infura (http://infura.io/).
93But first you need an address and private key. You can go make these
94online too. Just head over to MyEtherWallet
95(https://www.myetherwallet.com/) and enter a password. You won't use
96this for real money, so don't worry too much about security. If you
97were using this for real money, you'd be much, much more careful
98about this process, though, right?
99Once you've entered your password, you will be offered the opportunity
100to "Download Keystore File." Do so.
101Then go back to MyEtherWallet, click on "View Wallet Info", select
102"Keystore / JSON file," re-upload the file you just downloaded, and reenter
103the password you just entered. Two important strings of
104alphanumeric characters will be revealed to you: your address and
105your private key. (Click on the little eye icon to reveal the latter.)
106Again, if you were dealing with real amounts of real money, and/or
107important smart contracts, you would take significant measures to
108protect and secure your private key. You aren't -- but, still, you don't
109want to get into the habit of hardcoding your private key into your code
110and/or your repo.
111I strongly suggest that you make the address and the private key
112environment variables. Then later you can deploy your code to AWS or
113Heroku or GCE and simply set the appropriate environment variables
114there. This also makes it easy to have different testing / staging /
115production environments.
116In a Bash environment like OS X, you can do this by appending these
117two lines to your .bash_profile in your home directory:
118export RINKEBY_ADDRESS="0x1234567890ABCDEF..."
119export RINKEBY_KEY="ABCDEF123456789..."
120Setting environment variables in other environments is left as an
121exercise for the reader. Note that you'll have to close and re-open your
122terminal window to refresh your environment. (Check with "printenv"
123in bash.)
124Voila: you now have a valid Ethereum address, and the private key used
125to sign its transactions, available for use in your code. However, to
126actually deploy and run transactions, you'll need (fake) money.
127Conveniently you can get this online, too, via the Rinkeby faucet:
128https://faucet.rinkeby.io/
129All you need to do is post your new Ethereum address (not your private
130key!) to Facebook or Twitter or Google+ (publicly): paste a link to that
131tweet/post into the input box at that faucet's web page; and wait at
132most a minute or two. The minimum amount of ether you'll receive (3)
133is already more than you need for basic development, but still, might
134as well get as much as you can.
135The faucet transaction might take a minute to complete. You can
136check your balance via Infura's Etherscan service:
137https://rinkeby.etherscan.io/address/[your-address-goes-here]
138Success? Congratulations! It is now almost time to move on to writing
139some code. First, though, we have one more thing to install and
140configure. Don't worry, it's easy; trust me, it's worth it.
1412. Truffle
142First -- you're familiar with Javascript, right? If not ... this may not be
143the tutorial for you. OK, you're vaguely familiar with Node and NPM,
144right? If not, get thee first to a different tutorial (eg
145http://nodesource.com/blog/an-absolute-beginners-guide-to-using-npm/)
146and study up.
147Know your away around those basics? Great. On to the Truffle
148Framework (http://truffleframework.com/docs/), "your Ethereum Swiss
149Army Knife." You'll use it to compile and test your Solidity code. You
150can also use Truffle to deploy your code, but it needs a local Ethereum
151node, so we'll deploy in a different way, described below.
152Installing Truffle via NPM is as easy as
153npm install -g truffle
154It then offers various commands, the most relevant of which for this
155tutorial are "truffle init", "truffle compile", and "truffle test." Once
156installed, open up a terminal window, create a new directory for your
157new Ethereum project -- I suggest the name "webthereum" -- go to it,
158create a "truffle" subdirectory, go to it, and type
159truffle init
160Voila: a scaffolding of files and directories will be created for you.
161Now, open up the text editor of your choice and create a new file
162called "MyPrize.sol" in the "contracts" sub-subdirectory that Truffle
163just created for you. It's finally time to start writing yourself some
164smart contracts.
1653. Baby's First Smart Contract
166Our smart contract, MyPrize, is going to manage ownership of 10
167geolocated augmented-reality entities, called "prizes." Each prize has
168an owner; a location; and a "metadata" field for a URL which points to
169the metadata that describes its contents, i.e. what it looks like in AR.
170For our smart-contract purposes, all we care about is that there's a
171string called "metadata."
172After a prize has been placed, it stays where it is until 1000 new
173blocks have been added to the blockchain (i.e. about 4 hours), and then
174anyone else can claim it, rewrite the metadata (e.g. what this entity
175looks like you point a custom AR app at the location where it has been
176placed) and then place it somewhere else. It's up to some external
177system (e.g. a custom Android/iOS/web app talking to a web service) to
178handle parsing the metadata, showing the AR object, etc. Our smart
179contract just handles ownership and location. Simple enough. Right?
180There can be many subtle complexities when writing Solidity smart
181contracts. Contracts can send and receive money. Contracts can
182spawn, own, and link to other contracts. Issues of identity, ownership,
183and versioning come up. It's common for whoever first deploys a
184contract to be its "owner," and for important functionality to be calved
185off to separate subcontracts, which the owner can replace if a bug
186needs to be fixed or when a new version is ready.
187Does this "code written so that its owner, and/or a set of other
188designated admins, can control, replace, and disable it" pattern play a
189little awkwardly with the "smart contracts are permissionless and
190irrevocable with no central control" hype? You betcha! But we're not 
191going to worry about theoretical philosophical concerns; we're just
192going to write some running code. And here it is:
193pragma solidity ^0.4.18;
194contract MyPrize {
195 struct Prize {
196 uint id;
197 address owner; // current owner
198 bytes10 geohash; // current unclaimed location
199 string metadata; // probably a URL pointing to the full metadata
200 uint placedAt; // block count when it was placed
201 }
202
203 mapping (uint => Prize) public prizes;
204 // Constructor
205 function MyPrize() public {
206 for (uint i=1; i <= 10; i++) {
207 Prize memory prize = Prize({id: i, owner: msg.sender, geohash: "",
208metadata: "", placedAt: 0});
209 prizes[i] = prize;
210 }
211 }
212
213 function placePrize (uint prizeId, bytes10 geohash, string _metadata) public
214{
215 var prize = prizes[prizeId];
216 require (prize.id != 0);
217 require (geohash != bytes10(""));
218 require (prize.owner == address(0) || msg.sender == prize.owner);
219 prize.owner = address(0);
220 prize.geohash = geohash;
221 prize.metadata = _metadata;
222 prize.placedAt = block.number;
223 }
224
225 function claimPrize (uint prizeId, bytes10 geohash) public {
226 var prize = prizes[prizeId];
227 require (prize.id != 0);
228 require (geohash != bytes10("") && prize.geohash == geohash);
229 require (block.number - prize.placedAt > 1000);
230 prize.owner = msg.sender;
231 prize.geohash = "";
232 prize.metadata = "";
233 prize.placedAt = 0;
234 }
235}
236Save that file, go back to your command line, go back to the "truffle"
237subdirectory beneath your root "webthereum" directory, and type
238truffle compile
239You should see something like:
240Compiling ./contracts/MyPrize.sol...
241Writing artifacts to ./build/contracts
2424. Testing Baby's First Smart Contract
243Writing automated tests for your software is always a good idea; but
244it's a really, really excellently awesome and important idea for Solidity
245code. If you don't find a bug until the code is on the blockchain, not
246only have you wasted a ton of time and a slew of money, your
247diagnosis and debugging options are very, very limited. Did I mention
248that they are expensive and time-consuming, too?
249So test. To its credit, Truffle lets you write test in both Solidity and
250Javascript. You could make a good case for writing both kinds of tests
251(so that you test both the VM operations, with the former, and the JSblockchain
252interface) but in the interests of expediency let's just whip
253up a Solidity test for MyPrize, called TestMyPrize.sol, within the "test"
254sub-subdirectory Truffle created for us:
255pragma solidity ^0.4.18;
256import "truffle/Assert.sol";
257import "../contracts/MyPrize.sol";
258contract TestMyPrize {
259 MyPrize mp = new MyPrize();
260 function testMyPrize() public {
261 mp.placePrize(1, bytes10("abcdefghij"), "http://test.example.com/");
262 var (id, owner, geohash, metadata, placedAt) = mp.prizes(uint(1));
263 Assert.equal (bytes10("abcdefghij"), geohash, "Geohash was successfully
264set");
265 Assert.equal (address(0), owner, "Ownership was successfully abdicated");
266 Assert.equal (block.number, placedAt, "PlacedAt successfully set");
267 // mp.claimPrize(1, bytes10("abcdefghij"));
268 // var (id2, owner2, geohash2, metadata2, placedAt2) =
269mp.prizes(uint(1));
270 // Assert.equal (bytes10(""), geohash2, "Geohash was successfully
271cleared");
272 // Assert.equal (msg.sender, owner2, "Ownership was successfully
273claimed");
274 }
275}
276You'll notice that we don't test the metadata, and that the "claim"
277tests are commented out. This is because we can't actually test either
278of those things; Truffle's Assert.equal doesn't work with strings
279returned from contract calls, and because (as far as I can tell) we can't
280easily advance the block number within Truffle. So the claim call will
281fail ungracefully if we try, teaching us nothing. I did mention that the
282whole ecosystem is still somewhere between alpha and beta software,
283right?
284But we test what we can. Back to the command line, and
285truffle test
286Et voila!
287 TestMyPrize
288 ✓ testMyPrize (105ms)
289 1 passing (691ms)
2905. Deploying Baby's First Smart Contract
291So you wrote (well, you copy-and-pasted, but it's a start) your first
292smart contract. You tested your first smart contract. You've got a
293blockchain address and a private key. Now how do all these
294ingredients bake together into a delicious cake running in production?
295(Pardon the mixed metaphor; I'm hungry as I type this.)
296You could run an Ethereum client like Geth or Parity or MetaMask on
297your local machine, point Truffle to it, and simply type "truffle deploy".
298But let's go a little more low-level, and at the same time, keep you from
299having to run your own local node. Infura lets you submit signed
300Ethereum transactions to the Rinkeby testnet via the Internet, and
301contract deployment is just another kind of Ethereum transaction.
302But you don't want to have to deal with a slew of cryptic bash
303commands either. Good news: you don't need to. You can compile and
304deploy entirely via Javascript, courtesy of the web3 library and the
305solc compiler. Let's take a look at that in a little more detail:
306First off, create a new "node" subdirectory under your root
307"webthereum" directory. Then go there, run
308npm init
309and tap ENTER through all the default values, or tweak them as you
310like. (You do have NPM installed and running on your system, right?)
311Next, install a few new NPM packages:
312npm install web3
313npm install solc
314npm install fs
315npm install ethereumjs-tx
316And now, save the following code to "deploy.js" in that newly created
317"node" directory:
318var Web3 = require('web3');
319var solc = require('solc');
320var fs = require('fs');
321var tx = require('ethereumjs-tx');
322const web3 = new Web3(new
323Web3.providers.HttpProvider("https://rinkeby.infura.io/"));
324deployContract();
325// showABI();
326function deployContract() {
327
328web3.eth.getTransactionCount(process.env.RINKEBY_ADDRESS).then((txnCou
329nt) => {
330 console.log("txn count", txnCount);
331 const source =
332fs.readFileSync(__dirname+'/../truffle/contracts/MyPrize.sol');
333 const compiled = solc.compile(source.toString(), 1);
334 const bytecode = compiled.contracts[':MyPrize'].bytecode;
335 const rawContractTx = {
336 from: process.env.RINKEBY_ADDRESS,
337 nonce: web3.utils.toHex(txnCount),
338 gasLimit: web3.utils.toHex(4000000),
339 gasPrice: web3.utils.toHex(20000000000),
340 data: '0x' + bytecode,
341 };
342 console.log("contract deploying...");
343 sendRaw(rawContractTx);
344 });
345}
346function showABI() {
347 const source =
348fs.readFileSync(__dirname+'/../truffle/contracts/MyPrize.sol');
349 const compiled = solc.compile(source.toString(), 1);
350 const abi = compiled.contracts[':MyPrize'].interface;
351 console.log("abi", abi);
352}
353function sendRaw(rawTx) {
354 var privateKey = new Buffer(process.env.RINKEBY_KEY, 'hex');
355 var transaction = new tx(rawTx);
356 transaction.sign(privateKey);
357 var serializedTx = transaction.serialize().toString('hex');
358 web3.eth.sendSignedTransaction(
359 '0x' + serializedTx, function(err, result) {
360 if(err) {
361 console.log("txn err", err);
362 } else {
363 console.log("txn result", result);
364 }
365 });
366}
367Note that this code depends on the RINKEBY_KEY and
368RINKEBY_ADDRESS environment variables discussed above, so make 
369sure those are set and available. Once that's done, simply return to
370your command line and type
371node deploy.js
372If you have followed all the above faithfully, and if the tutorial gods
373smile upon you, you will see a result like:
374txn count 0
375contract deploying...
376txn result
3770x935c3a2849064d45673b6afd5989e656d5adaff0dc6644b08e33fa2cee57acd9
378Congratulations! You have successfully submitted your smart contract
379to the Rinkeby blockchain. (And unless there's some unforeseen
380problem it will soon be mined and deployed.)
381Let's go over what we just did. First we built a web3 Ethereum client
382as a Javascript object, and pointed it to the Rinkeby testnet
383blockchain. Then, in deployContract(), we connected to that blockchain
384and asked it how many transactions our address had performed (and in
385passing verified that this connection worked.) We subsequently used
386that number as the "nonce" for our new transaction, which, to vastly
387oversimplify, ensures that each transaction is unique.
388We then used the "solc" library to compile the Solidity code we
389previously wrote -- yes, that's right, we ran a Solidity compiler inside of
390Javascript -- and built a dictionary with all the values required for our
391new Ethereum transaction. This included the address of our Ethereum
392account, which we previously set as an environment variable, and the
393compiled bytecode as its payload. We also set the gasLimit and
394gasPrice to reasonably generous values, to ensure that we could pay
395for this transaction out of the money in our account, and finally sent
396the assembled dictionary off to the "sendRaw" method.
397"sendRaw," in turn, converted the dictionary into a transaction object
398from the "ethereumjs-tx" library, signed it using the private key we
399created and set as an environment variable above, serialized it, and
400used our web3 client to actually submit the transaction to the
401blockchain via a JSON-RPC call over the Internet. Once accepted and
402mined / paid for, our MyPrize contract will be there on the blockchain 
403forevermore, its code eternally available for anyone to call.
404You can doublecheck the transaction's status by heading over to the
405Etherscan service for Rinkeby (https://rinkeby.etherscan.io/) and
406entering into its search box the transaction hash that was logged to
407your console. You can also use your Ethereum address to get a list of
408all transactions you've submitted. Don't panic if this transaction
409doesn't show up immediately -- sometime it can take a minute or so.
410But it shouldn't take longer than that.
411You can get a wealth of in-depth data about your transactions from
412Etherscan. I encourage you to go through it in some detail. Much may
413not make sense at first, but, again, you'll wrap your head around it all
414eventually.
4156. Node, Meet Blockchain; Blockchain, Meet Node
416Now it's time to move from the command line to the browser. A caveat:
417as mentioned, I'm generally a Ruby/Rails or Python/Django or Go server
418guy, so what follows is probably not especially elegant or idiomatic
419Node code. It should get the job done though.
420The principle is simple: get Node to use web3 to connect to our
421blockchain smart contract, so that anybody in the world can interact
422with it via our web service, without them having to install a browser
423plugin or run an Ethereum node. In fact we won't even run a node on
424our server, though you would if you wanted to scale; instead we'll just
425keep talking to Rinkeby via Infura.
426We're mostly just going to build a JSON API, suitable for use by a
427smartphone app or a modern Javascript framework like React or
428Angular. We will, however, also create a very simple HTML page for
429browser consumption, which (for now) just lets users check on the
430state of our ten prizes.
431We're also going to use Express and Helmet to make Node easier /
432better, so make sure you run
433npm install express
434npm install helmet
435before pointing your text editor to "index1.js" within the "node"
436directory and dumping the following code into it:
437const express = require('express');
438const helmet = require('helmet');
439const Web3 = require('web3');
440const contractAddress = undefined; // TODO
441const abi = [{"constant":false,"inputs":[{"name":"prizeId","type":"uint256"},
442{"name":"geohash","type":"bytes10"},
443{"name":"_metadata","type":"string"}],"name":"placePrize","outputs":
444[],"payable":false,"stateMutability":"nonpayable","type":"function"},
445{"constant":false,"inputs":[{"name":"prizeId","type":"uint256"},
446{"name":"geohash","type":"bytes10"}],"name":"claimPrize","outputs":
447[],"payable":false,"stateMutability":"nonpayable","type":"function"},
448{"constant":true,"inputs":
449[{"name":"","type":"uint256"}],"name":"prizes","outputs":
450[{"name":"id","type":"uint256"},{"name":"owner","type":"address"},
451{"name":"geohash","type":"bytes10"},{"name":"metadata","type":"string"},
452{"name":"placedAt","type":"uint256"}],"payable":false,"stateMutability":"vie
453w","type":"function"},{"inputs":
454[],"payable":false,"stateMutability":"nonpayable","type":"constructor"}];
455const app = express();
456const port = process.env.PORT || 8801;
457app.use(helmet());
458app.get('/', (req, response) => {
459 var html = "<HTML><HEAD><TITLE>Webthereum
460Tutorial</TITLE></HEAD><BODY>";
461 html += "<FORM action='prize'><SELECT name='prizeId'>";
462 for (var i=1; i<=10; i++) {
463 html += "<OPTION>"+i;
464 }
465 html += "</SELECT><INPUT type='submit'></FORM>";
466 html += "</BODY></HTML>";
467 response.send(html);
468});
469// Display the prize data, if any
470app.get('/prize', (req, response) => {
471 get_prize_data(req.query.prizeId, function(err, res) {
472 if (err) {
473 response.json( {"err": err} );
474 } else {
475 response.json( {"res": res} );
476 }
477 });
478});
479// error handling: for now just console.log
480app.use((err, request, response, next) => {
481 console.log(err);
482 response.status(500).send('Something broke! '+ JSON.stringify(err));
483});
484app.listen(port, (err) => {
485 if (err) {
486 return console.log('something bad happened', err);
487 }
488 console.log(`server is listening on ${port}`);
489});
490// Get prize data
491function get_prize_data(prizeId, callback) {
492 web3 = new Web3(new
493Web3.providers.HttpProvider("https://rinkeby.infura.io/"));
494 var contract = new web3.eth.Contract(abi, contractAddress);
495 contract.methods.prizes(prizeId).call(function(error, result) {
496 if (error) {
497 console.log('error', error);
498 callback(error, null);
499 } else {
500 console.log('result', result);
501 callback(null, result);
502 }
503 })
504 .catch(function(error) {
505 console.log('call error ' + error);
506 callback(error, null);
507 });
508};
509This time, however, you cannot run the code out of the box. You have to
510fill in the "contractAddress" and "abi" values at the top first. The
511contract address is straightforward enough: head to Etherscan, go to
512the details of the transaction you submitted, and you should see a line
513to the effect of 
514[Contract "0x8268205e3e22ccf75615b997ef91e77eeed181aa" Created]
515That's your contract address -- ie replace the current line with
516something like
517const contractAddress =
518"0x8268205e3e22ccf75615b997ef91e77eeed181aa";
519Note that it's a string, i.e. must be between quotes. You could also set
520and use it as an environment variable, as before, but note that the
521Ethereum contract address is different from your Ethereum account
522address.
523You should now just need to run
524node index1.js
525and point your browser to localhost:8801. A very very simple HTML
526form should appear. Hit "submit" and you should get some raw JSON
527back, such as --
528{"res":
529{"0":"1","1":"0xC7679A8C55817EFA649961C6d9DF7596e4bd8C51","2":"0x0
5300000000000000000000","3":"","4":"0","id":"1","owner":"0xC7679A8C55817E
531FA649961C6d9DF7596e4bd8C51","geohash":"0x00000000000000000000","
532metadata":"","placedAt":"0"}}
533Not super-interesting, yet -- but what's important is that this is real live
534data coming from your Solidity smart contract, running on the Rinkeby
535blockchain, to the browser, via your Node service! Give yourself a pat
536on the back for making this happen; you've earned it.
537A Quick Aside
538In order to call the contract, your Node code needs to know its
539interface, aka its ABI, which is hardcoded above. In theory, you should
540be able to recompile your Solidity contract from source and use its
541.interface field, rather than hardcoding it. In theory. In practice that
542seems to not actually work.
543So, if you change the MyPrize contract, do the following: Re-open
544deploy.js. Uncomment the "showABI()" line. Save the file. Then run
545"node deploy.js" again. The new contract will be deployed, to a new
546address, and the ABI will be logged to the console. Now, copy-andpaste
547that ABI back to index1.js; update the contract address (don't
548use the transaction hash!) in that same file; and you're good to go.
5497. End-To-End It, Baby
550OK, let's put it all together; let's run a Node web service that you can
551use to place prizes, to claim prizes, and, what's more, let's add in a
552Postgres database that you can use to track prize data and transaction
553information.
554Why a database? Because a blockchain is hugely inefficient with
555respect to both ease-of-development and performance. The idea is to
556use the blockchain as a source of master truth for the data which
557needs to be decentralized (such as ownership and location data) and a
558database as a scalable read cache, and/or for data that doesn't need to
559be in the blockchain.
560Granted, this raises unpleasant issues like "when do you invalidate the
561cache / update the database?" but such is the price we pay. (One
562answer which springs to mind: refresh the database every time you
563write a value to the blockchain, and also via a daily cron job.)
564Enough musing; let's code. But first, for the below to work you'll need
565to
566npm install pg
567npm install express-enforces-ssl
568-- and, more to the point, get PostgreSQL running locally on your
569machine, and create a database "webthereum" within it. Explaining
570Postgres is way out of the scope of this tutorial, I'm afraid, so this
571section assumes a basic degree of comfort with databases in general
572and Postgres specifically.
573Once you've done that, here is a new and substantially larger file,
574index.js, for your node directory:
575// Set up web server
576const express = require('express');
577const helmet = require('helmet');
578const app = express();
579const port = process.env.PORT || 8801;
580app.use(helmet());
581app.use(express.json());
582app.use(express.urlencoded({ extended: true }));
583const express_enforces_ssl = require('express-enforces-ssl');
584// TODO: uncomment in production!
585// app.enable('trust proxy'); // To force SSL on eg Heroku
586// TODO: uncomment in production!
587// app.use(express_enforces_ssl());
588// Ethereum client setup / data
589const contractAddress = undefined; // TODO
590const abi = [{"constant":false,"inputs":[{"name":"prizeId","type":"uint256"},
591{"name":"geohash","type":"bytes10"},
592{"name":"_metadata","type":"string"}],"name":"placePrize","outputs":
593[],"payable":false,"stateMutability":"nonpayable","type":"function"},
594{"constant":false,"inputs":[{"name":"prizeId","type":"uint256"},
595{"name":"geohash","type":"bytes10"}],"name":"claimPrize","outputs":
596[],"payable":false,"stateMutability":"nonpayable","type":"function"},
597{"constant":true,"inputs":
598[{"name":"","type":"uint256"}],"name":"prizes","outputs":
599[{"name":"id","type":"uint256"},{"name":"owner","type":"address"},
600{"name":"geohash","type":"bytes10"},{"name":"metadata","type":"string"},
601{"name":"placedAt","type":"uint256"}],"payable":false,"stateMutability":"vie
602w","type":"function"},{"inputs":
603[],"payable":false,"stateMutability":"nonpayable","type":"constructor"}];
604const Web3 = require('web3');
605const web3 = new Web3(new
606Web3.providers.HttpProvider("https://rinkeby.infura.io/"));
607// Set up database
608const { Pool } = require('pg');
609const local_db_url = 'postgres://localhost:5432/webthereum';
610const db_url = process.env.DATABASE_URL || local_db_url;
611require('pg').defaults.ssl = db_url != local_db_url;
612const pool = new Pool({ connectionString: db_url });
613pool.on('error', (err, client) => {
614 console.error('Unexpected error on idle client', err);
615 console.error('client is', client);
616 process.exit(-1);
617});
618const db_creation_string = `
619 CREATE TABLE IF NOT EXISTS prizes(id SERIAL PRIMARY KEY, created_at
620timestamp with time zone NOT NULL DEFAULT current_timestamp,
621updated_at timestamp NOT NULL DEFAULT current_timestamp, latestTxn
622CHAR(66), prizeData TEXT);
623 CREATE TABLE IF NOT EXISTS transactions(id SERIAL PRIMARY KEY,
624created_at timestamp with time zone NOT NULL DEFAULT current_timestamp,
625updated_at timestamp NOT NULL DEFAULT current_timestamp, prizeId INT,
626txnState INT not null, txnHash CHAR(66) not null, arguments TEXT, txnData
627TEXT, txnReceipt TEXT);
628 CREATE INDEX IF NOT EXISTS idTxnHash ON transactions(txnHash);
629`;
630// Enforce HTTPS in non-local environments
631app.use((request, response, next) => {
632 if (request.hostname == "localhost" || request.hostname == "127.0.0.1" ||
633request.secure) {
634 next();
635 } else {
636 response.send("Insecure request on non-localhost server! Uncomment
637express_enforces_ssl usage.");
638 }
639});
640// Some minimal error handling
641app.use((err, request, response, next) => {
642 console.log(err);
643 response.status(500).send('Something broke! '+ JSON.stringify(err));
644 next();
645});
646// Create database. Does nothing if already created.
647app.get('/create', (req, response) => {
648 pool.query(db_creation_string, (err, res) => {
649 if (err) {
650 response.json( {"err": ""+err} );
651 } else {
652 response.send( "Database created: " + res );
653 }
654 });
655});
656// Home page
657app.get('/', (req, response) => {
658 var html = "<HTML><HEAD><TITLE>Webthereum
659Tutorial</TITLE></HEAD><BODY>";
660 html += "<H3>View</H3>";
661 html += "<FORM action='prize'><SELECT name='prizeId'>";
662 for (var i=1; i<=10; i++) {
663 html += "<OPTION>"+i;
664 }
665 html += "</SELECT><INPUT type='submit'></FORM>";
666 html += "<HR/>";
667 html += "<H3>Place</H3>";
668 html += "<FORM action='place' method='post'><SELECT
669name='prizeId'>";
670 for (var j=1; j<=10; j++) {
671 html += "<OPTION>"+j;
672 }
673 html += "</SELECT>";
674 html += "<INPUT type='text' name='senderAddress' placeholder='Sender
675Address'>";
676 html += "<INPUT type='password' name='privateKey' placeholder='Private
677Key'>";
678 html += "<INPUT type='text' name='geohash' placeholder='Geohash'>";
679 html += "<INPUT type='text' name='metadata' placeholder='Metadata'>";
680 html += "<INPUT type='submit'></FORM>";
681 html += "<HR/>";
682 html += "<H3>Claim</H3>";
683 html += "<FORM action='claim' method='post'><SELECT
684name='prizeId'>";
685 for (var k=1; k<=10; k++) {
686 html += "<OPTION>"+k;
687 }
688 html += "</SELECT>";
689 html += "<INPUT type='text' name='senderAddress' placeholder='Sender
690Address'>";
691 html += "<INPUT type='password' name='privateKey' placeholder='Private
692Key'>";
693 html += "<INPUT type='text' name='geohash' placeholder='Geohash'>";
694 html += "<INPUT type='submit'></FORM>";
695 html += "<HR/>";
696 html += "<a href='/prizes'>View Current Prize Data Cache</a>";
697 html += "</BODY></HTML>";
698 response.send(html);
699});
700// List all the prizes that the database knows about, with their raw JSON data.
701app.get('/prizes', (req, response) => {
702 var html = "<HTML><HEAD><TITLE>Webthereum
703Tutorial</TITLE></HEAD><BODY>";
704 html += "<H3>Prizes</H3>";
705 html += "<P>This shows the contents of the local database. Actual
706blockchain data may now vary.</P>";
707 html +=
708"<TABLE><TR><TH>ID</TH><TH>Owner</TH><TH>Geohash</TH><TH
709>Placed At</TH><TH>Metadata</TH><TH>Latest DB
710Txn</TH><TH>State</TH></TR>";
711 pool.query("SELECT * FROM Prizes p LEFT OUTER JOIN Transactions t ON
712p.latestTxn = t.txnHash ORDER BY p.id", [], (err, res) => {
713 if (err) {
714 console.log("select prizes error", err);
715 html += "Could not get prizes from database";
716 response.send(html);
717 return;
718 }
719 for (var i = 0; i < res.rows.length; i++) {
720 var row = res.rows[i];
721 var prize = JSON.parse(row.prizedata);
722 var owner = prize.owner ==
723'0x0000000000000000000000000000000000000000' ? "" : prize.owner;
724 var txnLink = row.latesttxn === null ? '' : "<a href='/checkTxn?
725txnHash=" + row.latesttxn + "'>view</a>";
726 var state = res.rows[i].txnstate;
727 state = (!state || state == -1) ? "Undefined" : state === 0 ? "Pending" :
728state == 1 ? "Mined" : state == 2 ? "Failed" : "Success";
729 html += "<TR><TD>" + prize.id + "</TD><TD>" + owner +
730"</TD><TD>" + web3.utils.hexToAscii(prize.geohash);
731 html += "</TD><TD>" + prize.placedAt + "</TD><TD>" +
732prize.metadata + "</TD><TD>" + txnLink + "</TD><TD>" + state +
733"</TD></TR>";
734 }
735 html += "</TABLE></BODY></HTML>";
736 response.send(html);
737 });
738});
739// Display the prize data, if any
740app.get('/prize', (req, response) => {
741 get_prize_data(req.query.prizeId, function(err, res) {
742 if (err) {
743 response.json( {"err": err} );
744 } else {
745 var dbId = req.query.prizeId;
746 var dbData = JSON.stringify(res);
747 pool.query("INSERT INTO Prizes (id, prizeData) VALUES ($1, $2) ON
748CONFLICT (id) DO UPDATE SET prizeData = $2;", [dbId, dbData], (dberr) => {
749 if (dberr) {
750 response.json( {"err": ""+dberr, "res" : res} );
751 return;
752 }
753 response.json( {"prize" : res} );
754 });
755 }
756 });
757});
758// Place a prize
759app.post('/place', (req, response) => {
760 console.log("req", JSON.stringify(req.body));
761 place_prize(req.body.prizeId, req.body.geohash, req.body.metadata,
762req.body.senderAddress, req.body.privateKey, function(err, txnHash) {
763 if (err) {
764 response.json( {"err": ""+err, "txnHash" : txnHash} );
765 return;
766 }
767 response.send("Placement submitted: txn "+txnHash);
768 });
769});
770// Register a site as taken, with image URL
771app.post('/claim', (req, response) => {
772 claim_prize(req.body.prizeId, req.body.geohash, req.body.senderAddress,
773req.body.privateKey, function(err, txnHash) {
774 if (err) {
775 response.json( {"err": ""+err, "txnHash" : txnHash} );
776 return;
777 }
778 response.send("Claim submitted: txn "+txnHash);
779 });
780});
781// Check a transaction
782app.get('/checkTxn', (req, response) => {
783 get_eth_txn_receipt(req.query.txnHash, function(err, result) {
784 if (result) {
785 response.json({"txnHash" : req.query.txnHash, "data" : result});
786 } else {
787 get_eth_txn_data(req.query.txnHash, function(err2, result) {
788 if (result) {
789 response.json({"txnHash" : req.query.txnHash, "data" : result});
790 } else {
791 response.json({"err" : err2, "receiptErr" : err});
792 }
793 });
794 }
795 });
796});
797app.listen(port, (err) => {
798 if (err) {
799 return console.log('something bad happened', err);
800 }
801 console.log(`server is listening on ${port}`);
802});
803//
804// Ethereum calls
805//
806const STATE_UNDEFINED = -1;
807const STATE_PENDING = 0;
808const STATE_MINED = 1;
809const STATE_FAILED = 2;
810const STATE_SUCCESS = 3;
811// Get prize data
812var get_prize_data = function(prizeId, callback) {
813 var contract = new web3.eth.Contract(abi, contractAddress);
814 contract.methods.prizes(prizeId).call(function(error, result) {
815 if (error) {
816 console.log('error', error);
817 callback(error, null);
818 } else {
819 console.log('result', result);
820 callback(null, result);
821 }
822 })
823 .catch(function(error) {
824 console.log('call error ' + error);
825 callback(error, null);
826 });
827};
828// Get initial ethereum transaction data
829function get_eth_txn_data(txnHash, callback) {
830 // do we have the data?
831 pool.query("SELECT txnData, txnState FROM Transactions WHERE txnHash
832= $1", [txnHash], (err, res) => {
833 if (err) {
834 console.log("select error", err);
835 callback(err, null);
836 return;
837 }
838 if (res.rows.length === 0) {
839 console.log("transaction not found", txnHash);
840 callback("Transaction not found!", null);
841 return;
842 }
843 var dbState = res.rows[0].txnstate ? res.rows[0].txnstate :
844STATE_UNDEFINED;
845 var dbData = res.rows[0].txndata ? JSON.parse(res.rows[0].txndata) : null;
846 if (dbState >= STATE_MINED && dbData) {
847 callback(null, dbData);
848 return;
849 }
850 web3.eth.getTransaction(txnHash).then((txnData) => {
851 var txnState = dbState;
852 if (dbState <= STATE_PENDING) {
853 txnState = txnData.blockNumber === null ? STATE_PENDING :
854STATE_MINED;
855 }
856 pool.query("UPDATE Transactions SET txnData = $1, txnState = $2
857WHERE txnHash = $3", [JSON.stringify(txnData), txnState, txnHash], (err) =>
858{
859 if (err) {
860 console.log("update error", err);
861 callback(err, txnData);
862 } else {
863 callback(null, txnData);
864 }
865 });
866 }).catch((error) => {
867 console.log("web3 txn data error", error);
868 callback(error, null);
869 });
870 });
871}
872// Get processed ethereum transaction receipt
873function get_eth_txn_receipt(txnHash, callback) {
874 // do we have the receipt?
875 pool.query("SELECT txnData, txnReceipt, txnState FROM Transactions
876WHERE txnHash = $1", [txnHash], (err, res) => {
877 if (err) {
878 console.log("select err", err);
879 callback(err, null);
880 return;
881 }
882 if (res.rows.length === 0) {
883 console.log("txn not found", txnHash);
884 callback("Transaction not found!", null);
885 return;
886 }
887 var dbReceipt = res.rows[0].txnreceipt ?
888JSON.parse(res.rows[0].txnreceipt) : null;
889 if (dbReceipt) {
890 var dbData = res.rows[0].txndata ? JSON.parse(res.rows[0].txndata) :
891null;
892 var dbState = res.rows[0].txnstate ? res.rows[0].txnstate :
893STATE_UNDEFINED;
894 callback(null, { "state" : dbState, "receipt" : dbReceipt, "initialData" :
895dbData });
896 return;
897 }
898 web3.eth.getTransactionReceipt(txnHash).then((receipt) => {
899 var txnState = receipt.status === "0x1" ? STATE_SUCCESS :
900STATE_FAILED;
901 pool.query("UPDATE Transactions SET txnReceipt = $1, txnState = $2
902WHERE txnHash = $3", [JSON.stringify(receipt), txnState, txnHash], (err) =>
903{
904 if (err) {
905 console.log("update err", err);
906 callback(err, receipt);
907 } else {
908 callback(null, {"receipt" : receipt});
909 }
910 });
911 }).catch((error) => {
912 console.log("web3 receipt error", error);
913 callback(error, null);
914 });
915 });
916}
917//
918// Ethereum transactions
919//
920const GAS_LIMIT = 4000000; // should not be a constant if using real money
921const GAS_PRICE = 20000000000; // should not be a constant if using real
922money
923// Place a prize
924function place_prize(prizeId, geohash, metadata, senderAddress, privateKey,
925callback) {
926 web3.eth.getTransactionCount(senderAddress).then((txnCount) => {
927 console.log("prizeId", prizeId);
928 var contract = new web3.eth.Contract(abi, contractAddress);
929 var geohashBytes = web3.utils.asciiToHex(geohash);
930 var placeMethod = contract.methods.placePrize(prizeId, geohashBytes,
931metadata);
932 var encodedABI = placeMethod.encodeABI();
933 var placeTx = {
934 from: senderAddress,
935 to: contractAddress,
936 nonce: web3.utils.toHex(txnCount),
937 gasLimit: web3.utils.toHex(GAS_LIMIT),
938 gasPrice: web3.utils.toHex(GAS_PRICE),
939 data: encodedABI,
940 };
941 sendTxn(privateKey, placeTx, prizeId, {"args" : { "geohash" : geohash,
942"metadata" : metadata } }, callback );
943 }).catch((err) => {
944 console.log("web3 err", err);
945 callback(err, null);
946 });
947}
948// Claim a prize
949function claim_prize(prizeId, geohash, senderAddress, privateKey, callback) {
950 web3.eth.getTransactionCount(senderAddress).then((txnCount) => {
951 var contract = new web3.eth.Contract(abi, contractAddress);
952 var geohashBytes = web3.utils.asciiToHex(geohash);
953 var claimMethod = contract.methods.claimPrize(prizeId, geohashBytes);
954 var encodedABI = claimMethod.encodeABI();
955 var claimTx = {
956 from: senderAddress,
957 to: contractAddress,
958 nonce: web3.utils.toHex(txnCount),
959 gasLimit: web3.utils.toHex(GAS_LIMIT),
960 gasPrice: web3.utils.toHex(GAS_PRICE),
961 data: encodedABI,
962 };
963 sendTxn(privateKey, claimTx, prizeId, {"args" : { "geohash" : geohash } },
964callback );
965 }).catch((err) => {
966 console.log("web3 err", err);
967 callback(err, null);
968 });
969}
970function sendTxn(privateKey, rawTx, prizeId, args, callback) {
971 var tx = require('ethereumjs-tx');
972 var privateKeyBuffer = new Buffer(privateKey, 'hex');
973 var transaction = new tx(rawTx);
974 transaction.sign(privateKeyBuffer);
975 var serializedTx = transaction.serialize().toString('hex');
976 web3.eth.sendSignedTransaction(
977 '0x' + serializedTx, function(err, txnHash) {
978 if(err) {
979 console.log("txn err", err);
980 callback(err, null);
981 } else {
982 console.log("txn result", txnHash);
983 pool.query("INSERT INTO Transactions (prizeId, txnHash, txnState,
984arguments) VALUES ($1, $2, $3, $4)", [prizeId, txnHash, STATE_PENDING,
985args], (err) => {
986 if (err) {
987 console.log("insert err", err);
988 callback(err, txnHash);
989 } else {
990 pool.query("INSERT INTO Prizes (id, latestTxn) VALUES ($1, $2) ON
991CONFLICT (id) DO UPDATE SET latestTxn = $2;", [prizeId, txnHash], (dberr)
992=> {
993 if (dberr) {
994 callback(dberr, null);
995 } else {
996 callback(null, txnHash);
997 }
998 });
999 }
1000 });
1001 }
1002 }).catch((err) => {
1003 callback(err, null);
1004 });
1005}
1006Save that. Replace the "contractAddress = undefined" with your real
1007address, as before. Then fire up
1008node index.js
1009and point your browser at localhost:8801, and you should see a slightly
1010more full-featured (if extremely spartan and ugly) web page -- one that
1011lets you not just view prize data, but also place or claim prizes. You
1012can also view the data stored in the local database, and check the
1013details of what the database thinks is the most recent blockchain
1014transaction relating to that prize.
1015Note however that this is highly asynchronous. When you "place" or
1016"claim" a prize, you're really just submitting a transaction to the
1017blockchain; it takes a minute or so, an eternity in web time, for that
1018transaction to be accepted or rejected. (It might be rejected because
1019e.g. there isn't enough ether in the sender's account to pay for it.) In a
1020real web app this delay must be handled by the server code, and
1021communicated to the user, in an elegant manner.
1022Otherwise there isn't a lot of conceptually new code here. As with the
1023previous section, you use the Node service as an intermediary
1024between the blockchain and the browser. As with deploying the
1025contract in the first place, to write data to the blockchain you build
1026Ethereum transactions and sign them. The big difference here is that
1027we aren't restricted to a single address stored in an environment
1028variable; we let anyone with a blockchain address and private key
1029perform these transactions.
1030On the one hand this approach arguably combines the decentralized
1031power of blockchain apps with the accessibility and scalability of web 
1032applications. On the other, though --
1033Security and the Browser <> Server <> Blockchain Pipeline
1034Private keys are called that for a reason. Ethereum addresses can hold
1035very, very significant amounts of money, and their private keys should
1036be guarded accordingly. This means that as a general rule you should
1037not ever be pasting them into a browser. You face potential client-side
1038plugin attacks, Man-in-the-Middle attacks, cross-site scripting attacks,
1039server-side hack attacks, etc etc etc.
1040It's fine for this tutorial which uses fake money, of course. (Though
1041even this tutorial makes a conscious point of enforcing HTTPS if the
1042server is not running locally, using a password HTML field for the
1043private key, etc.) But if you want to build a real blockchain-powered
1044web service, you're either going to have to be smarter about security
1045than this, or you'll have to outsource being smarter to your users, e.g.
1046by telling them to only use throwaway, low-value Ethereum addresses
1047for your service.
1048That last may sound callous and insecure, and it is, but in the long run
1049maybe part of the correct security solution is for users to have
1050"saving" Ethereum addresses, with real money, which they keep very
1051secure, from which they occasionally transfer money to "spending"
1052addresses which they actually use for application transactions.
1053Note: part of. Even then you shouldn't be pasting private keys into web
1054forms. Perhaps private keys should be decomposed into two halves,
1055unique to a given service, when a user registers: the server maintains
1056one half, the user enters the other half (which the server never stores)
1057to validate each transaction, and if either gets hacked, well, each half
1058is useless without the other. Perhaps some centralized (or even
1059decentralized) private-key manager such as 1Password or LastPass
1060will arise to solve this problem in general.
1061Regardless, the thing to remember is that the considerable value of
1062ether currency, combined with the irrevocable nature of Ethereum
1063transactions, means that if you're building a real Ethereum web app,
1064you have to take private-key security very seriously. You have been 
1065warned.
1066That's all, folks!
1067Thus endeth this tutorial. Your thoughts and feedback are (probably)
1068welcome, so feel free to email me or @ me on Twitter; details below.
1069Thanks in advance --
1070Jon Evans
1071jon@happyfuncorp.com
1072twitter.com/rezendi