· 6 years ago · Sep 05, 2019, 12:07 AM
1{
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "metadata": {},
6 "source": [
7 "# Understanding neural nets' _output shape_ and _number of parameters_ layer by layer\n",
8 "## <font color='green'>Sequential class version</font>\n",
9 "> ... understanding how each layer of a neural network changes the shape of the data as it flows through the network is a key part of truly understanding the mechanics of deep learning ~ _David Foster_, _Generative Deep Learning_"
10 ]
11 },
12 {
13 "cell_type": "markdown",
14 "metadata": {},
15 "source": [
16 "In this notebook we look at 3 ways of finding the _output shape_ and _number of parameters_ layer by layer:\n",
17 "1. Keras' built-in method: <font color=blue>model.summary()</font>;\n",
18 "2. our home-made method: <font color=blue>mysummary(model)</font>;\n",
19 "3. manual back-of-the-envelope calculation.\n",
20 "\n",
21 "We beginning by <font color=green>def</font>ining our home-made method, <font color=blue>mysummary(model)</font>. \n",
22 "Worked examples of different network architectures, of fully connected and 2D convolution layers, will follow later.\n",
23 "\n",
24 "This notebook is the <font color=green>sequential class version</font>, which matches cell-by-cell with the companion <font color=green>function API version</font>. Sequential class and functional API are two different ways of defining a neural network architecture in Keras."
25 ]
26 },
27 {
28 "cell_type": "code",
29 "execution_count": 1,
30 "metadata": {},
31 "outputs": [
32 {
33 "name": "stderr",
34 "output_type": "stream",
35 "text": [
36 "Using TensorFlow backend.\n"
37 ]
38 }
39 ],
40 "source": [
41 "def mysummary(model):\n",
42 " PASS0 = True\n",
43 " print('OUTPUT FROM mysummary(model):')\n",
44 " for layer in model.layers:\n",
45 " if PASS0: \n",
46 " if len(layer.input_shape)>2 and layer.input_shape[1] != layer.input_shape[2]:\n",
47 " print('warning: input_shape not square')\n",
48 "# take layer.input_shape[1:] only if this is the first layer, which happens at the first pass\n",
49 "# take just [1:] because [0] is always None\n",
50 " input_shape = layer.input_shape[1:]\n",
51 " PASS0 = False\n",
52 " param = 0\n",
53 " if 'Conv2D' in str(layer.build):\n",
54 " if layer.kernel_size[0] != layer.kernel_size[1]:\n",
55 " print('warning: kernel size not square')\n",
56 " if layer.strides[0] != layer.strides[1]:\n",
57 " print('warning: strides not square')\n",
58 "# strides defaults to 1 and padding defaults to none in Keras\n",
59 "# when strides=1, output dimensions are therefore reduced by 2\n",
60 "# when user specificies strides>1, output dimensions are then reduced by that user-specified factor\n",
61 " if layer.strides[0]==1 and layer.strides[1]==1 and layer.padding=='valid':\n",
62 " output_shape = int(input_shape[0] - 2), int(input_shape[1] - 2), layer.filters\n",
63 " else:\n",
64 " output_shape = int(input_shape[0] // layer.strides[0]), int(input_shape[1] // layer.strides[1]), layer.filters\n",
65 "# the number of parameters from a conv2d layer depends on \n",
66 "# - the shape of conv2d kernel\n",
67 "# - the number of channels of the previous layer (or the input data, if it's the first layer)\n",
68 "# - the number of conv2d filters \n",
69 "# the one in the 'plus one' is for the bias\n",
70 " param = (layer.kernel_size[0] * layer.kernel_size[1] * layer.input_shape[3] + 1 ) * layer.filters\n",
71 " elif 'MaxPooling2D'in str(layer.build):\n",
72 " if layer.pool_size[0] != layer.pool_size[1]:\n",
73 " print('warning: pool size not square')\n",
74 "# maxpooling reduces output_shape[:2] by the user-specified pool sizes, without changing output_shape[2]\n",
75 " output_shape = int(input_shape[0] / layer.pool_size[0]), int(input_shape[1] / layer.pool_size[1]), input_shape[2]\n",
76 " elif 'Flatten' in str(layer.build):\n",
77 " output_shape = (input_shape[0] * input_shape[1] * input_shape[2], )\n",
78 " elif 'Dropout' in str(layer.build) or 'input' in str(layer.build):\n",
79 " output_shape = input_shape\n",
80 " elif 'Dense' in str(layer.build):\n",
81 "# output_shape from dense is the user-specified units\n",
82 " output_shape = (layer.units, )\n",
83 "# the number of parameters is units multiplied by (input shape plus 1), where 1 is for the bias\n",
84 " param = (input_shape[0] + 1) * layer.units\n",
85 "# verify that our formulae produce the same output as those calculated internally by Keras\n",
86 " assert output_shape == layer.output_shape[1:], (output_shape, layer.output_shape[1:])\n",
87 " assert param == layer.count_params()\n",
88 " print('{:29s}(None, '.format(layer.name), end='')\n",
89 " for t in output_shape[:-1]:\n",
90 " print(t, end=', ')\n",
91 "# some alignment cosmetics for printing\n",
92 " print('{})'.format(output_shape[-1]), end='')\n",
93 " if len(output_shape)==1:\n",
94 " print(' ', end='')\n",
95 " print(' '*(20 - len(str(output_shape))), param)\n",
96 " input_shape = output_shape\n",
97 "\n",
98 "from keras import models, layers"
99 ]
100 },
101 {
102 "cell_type": "markdown",
103 "metadata": {},
104 "source": [
105 "## Worked examples\n",
106 "In each cell that follows, we define a network architecture using the sequential class, and proceed to\n",
107 "1. call Keras' built-in method: <font color=blue>model.summary()</font>;\n",
108 "2. call <font color=blue>mysummary(model)</font> which we just <font color=green>def</font>ined;\n",
109 "3. markup inline with our manual back-of-the-envelop calculation of the _output shape_ and _number of parameters_ layer by layer."
110 ]
111 },
112 {
113 "cell_type": "code",
114 "execution_count": 2,
115 "metadata": {},
116 "outputs": [
117 {
118 "name": "stdout",
119 "output_type": "stream",
120 "text": [
121 "_________________________________________________________________\n",
122 "Layer (type) Output Shape Param # \n",
123 "=================================================================\n",
124 "dense_1 (Dense) (None, 32) 25120 \n",
125 "_________________________________________________________________\n",
126 "dense_2 (Dense) (None, 32) 1056 \n",
127 "=================================================================\n",
128 "Total params: 26,176\n",
129 "Trainable params: 26,176\n",
130 "Non-trainable params: 0\n",
131 "_________________________________________________________________\n",
132 "OUTPUT FROM mysummary(model):\n",
133 "dense_1 (None, 32) 25120\n",
134 "dense_2 (None, 32) 1056\n"
135 ]
136 }
137 ],
138 "source": [
139 "model = models.Sequential()\n",
140 "model.add(layers.Dense(32, input_shape=(784,)))\n",
141 "# Output dimension of a dense layer is the user-defined units, which is 32 in this case\n",
142 "# Number of parameters = (784 + 1) * 32 = 25120\n",
143 "model.add(layers.Dense(32))\n",
144 "# Output dimension is again 32\n",
145 "# Number of parameters = (32 + 1) * 32 = 1056, where the first '32' is from layer dense_1, and the second '32' is from current layer, dense_2\n",
146 "model.summary()\n",
147 "mysummary(model)"
148 ]
149 },
150 {
151 "cell_type": "code",
152 "execution_count": 3,
153 "metadata": {},
154 "outputs": [
155 {
156 "name": "stdout",
157 "output_type": "stream",
158 "text": [
159 "_________________________________________________________________\n",
160 "Layer (type) Output Shape Param # \n",
161 "=================================================================\n",
162 "dense_3 (Dense) (None, 16) 160016 \n",
163 "_________________________________________________________________\n",
164 "dense_4 (Dense) (None, 16) 272 \n",
165 "_________________________________________________________________\n",
166 "dense_5 (Dense) (None, 1) 17 \n",
167 "=================================================================\n",
168 "Total params: 160,305\n",
169 "Trainable params: 160,305\n",
170 "Non-trainable params: 0\n",
171 "_________________________________________________________________\n",
172 "OUTPUT FROM mysummary(model):\n",
173 "dense_3 (None, 16) 160016\n",
174 "dense_4 (None, 16) 272\n",
175 "dense_5 (None, 1) 17\n"
176 ]
177 }
178 ],
179 "source": [
180 "model = models.Sequential()\n",
181 "# activations affect neither the output dimensions nor the number of parameters\n",
182 "model.add(layers.Dense(16, activation='relu', input_shape=(10000,)))\n",
183 "# Output shape = (None, user-specified units) = (None, 16)\n",
184 "# Number of parameters = (10000 + 1) * 16 = 160016\n",
185 "model.add(layers.Dense(16, activation='relu'))\n",
186 "# Output shape = (None, user-specified units) = (None, 16)\n",
187 "# Number of parameters = (16 + 1) * 16 = 272, where the first '16' is from layer dense_3 and the second '16' is from the current layer, dense_4\n",
188 "model.add(layers.Dense(1, activation='sigmoid'))\n",
189 "# Output shape = (None, user-specified units) = (None, 1)\n",
190 "# Number of parameters = (16 + 1) * 1 = 17, where the 16 is from the current layer, dense_5, itself\n",
191 "model.summary()\n",
192 "mysummary(model)"
193 ]
194 },
195 {
196 "cell_type": "code",
197 "execution_count": 4,
198 "metadata": {},
199 "outputs": [
200 {
201 "name": "stdout",
202 "output_type": "stream",
203 "text": [
204 "_________________________________________________________________\n",
205 "Layer (type) Output Shape Param # \n",
206 "=================================================================\n",
207 "dense_6 (Dense) (None, 64) 640064 \n",
208 "_________________________________________________________________\n",
209 "dense_7 (Dense) (None, 64) 4160 \n",
210 "_________________________________________________________________\n",
211 "dense_8 (Dense) (None, 46) 2990 \n",
212 "=================================================================\n",
213 "Total params: 647,214\n",
214 "Trainable params: 647,214\n",
215 "Non-trainable params: 0\n",
216 "_________________________________________________________________\n",
217 "OUTPUT FROM mysummary(model):\n",
218 "dense_6 (None, 64) 640064\n",
219 "dense_7 (None, 64) 4160\n",
220 "dense_8 (None, 46) 2990\n"
221 ]
222 }
223 ],
224 "source": [
225 "model = models.Sequential()\n",
226 "model.add(layers.Dense(64, activation='relu', input_shape=(10000,)))\n",
227 "# Output shape = (None, user-specified units) = (None, 64)\n",
228 "# Number of parameters = (10000 + 1) * 64 = 640064\n",
229 "model.add(layers.Dense(64, activation='relu'))\n",
230 "# Output shape = (None, user-specified units) = (None, 64)\n",
231 "# Number of parameters = (64 + 1) * 64 = 4160, where the first 64 is from layer dense_6 and the second '64' is from the current layer, dense_7\n",
232 "model.add(layers.Dense(46, activation='softmax'))\n",
233 "# Output shape = (None, user-specified units) = (None, 64)\n",
234 "# Number of parameters = (64 + 1) * 46 = 2990\n",
235 "model.summary()\n",
236 "mysummary(model)"
237 ]
238 },
239 {
240 "cell_type": "code",
241 "execution_count": 5,
242 "metadata": {},
243 "outputs": [
244 {
245 "name": "stdout",
246 "output_type": "stream",
247 "text": [
248 "_________________________________________________________________\n",
249 "Layer (type) Output Shape Param # \n",
250 "=================================================================\n",
251 "dense_9 (Dense) (None, 512) 401920 \n",
252 "_________________________________________________________________\n",
253 "dense_10 (Dense) (None, 10) 5130 \n",
254 "=================================================================\n",
255 "Total params: 407,050\n",
256 "Trainable params: 407,050\n",
257 "Non-trainable params: 0\n",
258 "_________________________________________________________________\n",
259 "OUTPUT FROM mysummary(model):\n",
260 "dense_9 (None, 512) 401920\n",
261 "dense_10 (None, 10) 5130\n"
262 ]
263 }
264 ],
265 "source": [
266 "model = models.Sequential()\n",
267 "model.add(layers.Dense(512, activation='relu', input_shape=(28 * 28,)))\n",
268 "# Output shape = (None, user-specified units) = (None, 512)\n",
269 "# Number of parameters = (28*28 + 1) * 512 = 401920\n",
270 "model.add(layers.Dense(10, activation='softmax'))\n",
271 "# Output dimension = (None, user-specified units) = (None, 10)\n",
272 "# Number of parameters = (512 + 1) * 10 = 5130\n",
273 "model.summary()\n",
274 "mysummary(model)"
275 ]
276 },
277 {
278 "cell_type": "code",
279 "execution_count": 6,
280 "metadata": {},
281 "outputs": [
282 {
283 "name": "stdout",
284 "output_type": "stream",
285 "text": [
286 "_________________________________________________________________\n",
287 "Layer (type) Output Shape Param # \n",
288 "=================================================================\n",
289 "conv2d_1 (Conv2D) (None, 26, 26, 32) 320 \n",
290 "_________________________________________________________________\n",
291 "conv2d_2 (Conv2D) (None, 24, 24, 64) 18496 \n",
292 "_________________________________________________________________\n",
293 "conv2d_3 (Conv2D) (None, 22, 22, 64) 36928 \n",
294 "=================================================================\n",
295 "Total params: 55,744\n",
296 "Trainable params: 55,744\n",
297 "Non-trainable params: 0\n",
298 "_________________________________________________________________\n",
299 "OUTPUT FROM mysummary(model):\n",
300 "conv2d_1 (None, 26, 26, 32) 320\n",
301 "conv2d_2 (None, 24, 24, 64) 18496\n",
302 "conv2d_3 (None, 22, 22, 64) 36928\n"
303 ]
304 }
305 ],
306 "source": [
307 "model = models.Sequential()\n",
308 "model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))\n",
309 "# Output shape = (None, 28-2, 28-2, 32)\n",
310 "# Number of parameters = (3*3*1 + 1) * 32 = 320\n",
311 "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
312 "# Output shape = (None, 26-2, 26-2, 64\n",
313 "# Number of parameters = (3*3*32 + 1) * 64 = 18496\n",
314 "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
315 "# Output shape = (None, 24-2, 24-2, 64\n",
316 "# Number of parameters = (3*3*64 + 1) * 64 = 36928\n",
317 "model.summary()\n",
318 "mysummary(model)"
319 ]
320 },
321 {
322 "cell_type": "code",
323 "execution_count": 7,
324 "metadata": {},
325 "outputs": [
326 {
327 "name": "stdout",
328 "output_type": "stream",
329 "text": [
330 "_________________________________________________________________\n",
331 "Layer (type) Output Shape Param # \n",
332 "=================================================================\n",
333 "conv2d_4 (Conv2D) (None, 26, 26, 32) 320 \n",
334 "_________________________________________________________________\n",
335 "max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32) 0 \n",
336 "_________________________________________________________________\n",
337 "conv2d_5 (Conv2D) (None, 11, 11, 64) 18496 \n",
338 "_________________________________________________________________\n",
339 "max_pooling2d_2 (MaxPooling2 (None, 5, 5, 64) 0 \n",
340 "_________________________________________________________________\n",
341 "conv2d_6 (Conv2D) (None, 3, 3, 64) 36928 \n",
342 "_________________________________________________________________\n",
343 "flatten_1 (Flatten) (None, 576) 0 \n",
344 "_________________________________________________________________\n",
345 "dense_11 (Dense) (None, 64) 36928 \n",
346 "_________________________________________________________________\n",
347 "dense_12 (Dense) (None, 10) 650 \n",
348 "=================================================================\n",
349 "Total params: 93,322\n",
350 "Trainable params: 93,322\n",
351 "Non-trainable params: 0\n",
352 "_________________________________________________________________\n",
353 "OUTPUT FROM mysummary(model):\n",
354 "conv2d_4 (None, 26, 26, 32) 320\n",
355 "max_pooling2d_1 (None, 13, 13, 32) 0\n",
356 "conv2d_5 (None, 11, 11, 64) 18496\n",
357 "max_pooling2d_2 (None, 5, 5, 64) 0\n",
358 "conv2d_6 (None, 3, 3, 64) 36928\n",
359 "flatten_1 (None, 576) 0\n",
360 "dense_11 (None, 64) 36928\n",
361 "dense_12 (None, 10) 650\n"
362 ]
363 }
364 ],
365 "source": [
366 "model = models.Sequential()\n",
367 "model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))\n",
368 "# Output shape = (None, 28-2, 28-2, 32)\n",
369 "# Number of parameters = (3*3*1 + 1) * 32 = 320\n",
370 "model.add(layers.MaxPooling2D(2, 2))\n",
371 "# Output dimensions = 26/2, 26/2, 32\n",
372 "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
373 "# Output shape = (None, 13-2, 13-2, 64)\n",
374 "# Number of parameters = (3*3*32 + 1) * 64 = 18496\n",
375 "model.add(layers.MaxPooling2D(2, 2))\n",
376 "# Output dimensions = floor(11/2), floor(11/2), 64\n",
377 "model.add(layers.Conv2D(64, (3, 3), activation='relu'))\n",
378 "# Output shape = (None, 5-2, 5-2, 64)\n",
379 "# Number of parameters = (3*3*64 + 1) * 64 = 36928\n",
380 "model.add(layers.Flatten())\n",
381 "# Output shape = (None, 3 * 3 * 64) = (None, 576)\n",
382 "model.add(layers.Dense(64, activation='relu'))\n",
383 "# Output shape = (None, user-specified units) = (None, 64)\n",
384 "# Number of parameters = (576 + 1) * 64 = 36928\n",
385 "model.add(layers.Dense(10, activation='relu'))\n",
386 "# Output shape = (None, user-specified units) = (None, 10)\n",
387 "# Number of parameters = (64 + 1) * 10 = 650\n",
388 "model.summary()\n",
389 "mysummary(model)"
390 ]
391 },
392 {
393 "cell_type": "code",
394 "execution_count": 8,
395 "metadata": {},
396 "outputs": [
397 {
398 "name": "stdout",
399 "output_type": "stream",
400 "text": [
401 "_________________________________________________________________\n",
402 "Layer (type) Output Shape Param # \n",
403 "=================================================================\n",
404 "flatten_2 (Flatten) (None, 3072) 0 \n",
405 "_________________________________________________________________\n",
406 "dense_13 (Dense) (None, 200) 614600 \n",
407 "_________________________________________________________________\n",
408 "dense_14 (Dense) (None, 150) 30150 \n",
409 "_________________________________________________________________\n",
410 "dense_15 (Dense) (None, 10) 1510 \n",
411 "=================================================================\n",
412 "Total params: 646,260\n",
413 "Trainable params: 646,260\n",
414 "Non-trainable params: 0\n",
415 "_________________________________________________________________\n",
416 "OUTPUT FROM mysummary(model):\n",
417 "flatten_2 (None, 3072) 0\n",
418 "dense_13 (None, 200) 614600\n",
419 "dense_14 (None, 150) 30150\n",
420 "dense_15 (None, 10) 1510\n"
421 ]
422 }
423 ],
424 "source": [
425 "model = models.Sequential()\n",
426 "model.add(layers.InputLayer(input_shape=(32, 32, 3)))\n",
427 "model.add(layers.Flatten())\n",
428 "# Output shape = (None, 32 * 32 * 3)\n",
429 "model.add(layers.Dense(200, activation='relu'))\n",
430 "# Output shape = (None, user-defined units) = (None, 200)\n",
431 "# Number of parameters = (3072 + 1 ) * 200 = 614600\n",
432 "model.add(layers.Dense(150, activation='relu'))\n",
433 "# Output shape = (None, user-defined units) = (None, 150)\n",
434 "# Number of parameters = (200 + 1) * 150 = 30150\n",
435 "model.add(layers.Dense(10, activation='softmax'))\n",
436 "# Output shape = (None, user-defined units) = (None, 10)\n",
437 "# Number of parameters = (150 + 1) * 10 = 1510\n",
438 "model.summary()\n",
439 "mysummary(model)"
440 ]
441 },
442 {
443 "cell_type": "code",
444 "execution_count": 9,
445 "metadata": {},
446 "outputs": [
447 {
448 "name": "stdout",
449 "output_type": "stream",
450 "text": [
451 "_________________________________________________________________\n",
452 "Layer (type) Output Shape Param # \n",
453 "=================================================================\n",
454 "conv2d_7 (Conv2D) (None, 16, 16, 10) 490 \n",
455 "_________________________________________________________________\n",
456 "conv2d_8 (Conv2D) (None, 8, 8, 20) 1820 \n",
457 "_________________________________________________________________\n",
458 "flatten_3 (Flatten) (None, 1280) 0 \n",
459 "_________________________________________________________________\n",
460 "dense_16 (Dense) (None, 10) 12810 \n",
461 "=================================================================\n",
462 "Total params: 15,120\n",
463 "Trainable params: 15,120\n",
464 "Non-trainable params: 0\n",
465 "_________________________________________________________________\n",
466 "OUTPUT FROM mysummary(model):\n",
467 "conv2d_7 (None, 16, 16, 10) 490\n",
468 "conv2d_8 (None, 8, 8, 20) 1820\n",
469 "flatten_3 (None, 1280) 0\n",
470 "dense_16 (None, 10) 12810\n"
471 ]
472 }
473 ],
474 "source": [
475 "model = models.Sequential()\n",
476 "model.add(layers.Conv2D(10, (4, 4), strides = 2, padding = 'same', input_shape=(32, 32, 3)))\n",
477 "# Output shape = (None, 32/2, 32/2, 10)\n",
478 "# Number of parameters = ( 4*4*3 + 1 ) * 10\n",
479 "model.add(layers.Conv2D(20, (3, 3), strides = 2, padding = 'same'))\n",
480 "# Output shape = (None, 16/2, 16/2, 20)\n",
481 "# Number of parameters = ( 3*3*10 + 1 ) * 20\n",
482 "model.add(layers.Flatten())\n",
483 "# Output shape = (None, 8 * 8 * 20)\n",
484 "model.add(layers.Dense(10, activation='softmax'))\n",
485 "# Output shape = (None, user-defined units) = (None, 10)\n",
486 "# Number of parameters = ( 1280 + 1 ) * 10\n",
487 "model.summary()\n",
488 "mysummary(model)"
489 ]
490 },
491 {
492 "cell_type": "code",
493 "execution_count": 10,
494 "metadata": {},
495 "outputs": [
496 {
497 "name": "stdout",
498 "output_type": "stream",
499 "text": [
500 "_________________________________________________________________\n",
501 "Layer (type) Output Shape Param # \n",
502 "=================================================================\n",
503 "conv2d_9 (Conv2D) (None, 148, 148, 32) 896 \n",
504 "_________________________________________________________________\n",
505 "max_pooling2d_3 (MaxPooling2 (None, 74, 74, 32) 0 \n",
506 "_________________________________________________________________\n",
507 "conv2d_10 (Conv2D) (None, 72, 72, 64) 18496 \n",
508 "_________________________________________________________________\n",
509 "max_pooling2d_4 (MaxPooling2 (None, 36, 36, 64) 0 \n",
510 "_________________________________________________________________\n",
511 "conv2d_11 (Conv2D) (None, 34, 34, 128) 73856 \n",
512 "_________________________________________________________________\n",
513 "max_pooling2d_5 (MaxPooling2 (None, 17, 17, 128) 0 \n",
514 "_________________________________________________________________\n",
515 "conv2d_12 (Conv2D) (None, 15, 15, 128) 147584 \n",
516 "_________________________________________________________________\n",
517 "max_pooling2d_6 (MaxPooling2 (None, 7, 7, 128) 0 \n",
518 "_________________________________________________________________\n",
519 "flatten_4 (Flatten) (None, 6272) 0 \n",
520 "_________________________________________________________________\n",
521 "dropout_1 (Dropout) (None, 6272) 0 \n",
522 "_________________________________________________________________\n",
523 "dense_17 (Dense) (None, 512) 3211776 \n",
524 "_________________________________________________________________\n",
525 "dense_18 (Dense) (None, 1) 513 \n",
526 "=================================================================\n",
527 "Total params: 3,453,121\n",
528 "Trainable params: 3,453,121\n",
529 "Non-trainable params: 0\n",
530 "_________________________________________________________________\n",
531 "OUTPUT FROM mysummary(model):\n",
532 "conv2d_9 (None, 148, 148, 32) 896\n",
533 "max_pooling2d_3 (None, 74, 74, 32) 0\n",
534 "conv2d_10 (None, 72, 72, 64) 18496\n",
535 "max_pooling2d_4 (None, 36, 36, 64) 0\n",
536 "conv2d_11 (None, 34, 34, 128) 73856\n",
537 "max_pooling2d_5 (None, 17, 17, 128) 0\n",
538 "conv2d_12 (None, 15, 15, 128) 147584\n",
539 "max_pooling2d_6 (None, 7, 7, 128) 0\n",
540 "flatten_4 (None, 6272) 0\n",
541 "dropout_1 (None, 6272) 0\n",
542 "dense_17 (None, 512) 3211776\n",
543 "dense_18 (None, 1) 513\n"
544 ]
545 }
546 ],
547 "source": [
548 "model = models.Sequential()\n",
549 "model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(150, 150, 3))) \n",
550 "# Output shape = (None, 150-2, 150-2, 32)\n",
551 "# Number of parameters = (3*3*3 + 1) * 32 = 896\n",
552 "model.add(layers.MaxPooling2D(2, 2)) \n",
553 "# Output shape = (None, 148/2, 148/2, 32) \n",
554 "model.add(layers.Conv2D(64, (3, 3), activation='relu')) \n",
555 "# Output shape = (None, 74-2, 74-2, 64)\n",
556 "# Number of parameters = (3*3*32 + 1) * 64 = 18496\n",
557 "model.add(layers.MaxPooling2D(2, 2)) \n",
558 "# Output shape = (None, 72/2, 72/2, 64)\n",
559 "model.add(layers.Conv2D(128, (3, 3), activation='relu'))\n",
560 "# Output shape = (None, 36-2, 36-2, 128)\n",
561 "# Number of parameters = (3*3*64 + 1) * 128 = 73856\n",
562 "model.add(layers.MaxPooling2D(2, 2))\n",
563 "# Output shape = (None, 34/2, 34/2, 128)\n",
564 "model.add(layers.Conv2D(128, (3, 3), activation='relu'))\n",
565 "# Output shape = (None, 17-2, 17-2, 128)\n",
566 "# Number of parameters = (3*3*128 + 1) * 128 = 147584\n",
567 "model.add(layers.MaxPooling2D(2, 2))\n",
568 "# Output shape = (None, floor(15/2), floor(15/2), 128)\n",
569 "model.add(layers.Flatten())\n",
570 "# Output shape = (None, 7 * 7 * 128) = (None, 6272)\n",
571 "model.add(layers.Dropout(.5))\n",
572 "# Output shape unchanged\n",
573 "model.add(layers.Dense(512, activation='relu'))\n",
574 "# Output shape = (None, user-specified units) = (None, 512)\n",
575 "# Number of parameters = (6272 + 1) * 512 = 3211776\n",
576 "model.add(layers.Dense(1, activation='sigmoid'))\n",
577 "# Output shape = (None, user-specified units) = (None, 1)\n",
578 "# Number of parameters = (512 + 1) * 1 = 513\n",
579 "model.summary()\n",
580 "mysummary(model)"
581 ]
582 },
583 {
584 "cell_type": "code",
585 "execution_count": 11,
586 "metadata": {},
587 "outputs": [
588 {
589 "name": "stdout",
590 "output_type": "stream",
591 "text": [
592 "_________________________________________________________________\n",
593 "Layer (type) Output Shape Param # \n",
594 "=================================================================\n",
595 "block1_conv2d_1 (Conv2D) (None, 150, 150, 64) 1792 \n",
596 "_________________________________________________________________\n",
597 "block1_conv2d_2 (Conv2D) (None, 150, 150, 64) 36928 \n",
598 "_________________________________________________________________\n",
599 "block1_max_pooling2d (MaxPoo (None, 75, 75, 64) 0 \n",
600 "_________________________________________________________________\n",
601 "block2_conv2d_1 (Conv2D) (None, 75, 75, 128) 73856 \n",
602 "_________________________________________________________________\n",
603 "block2_conv2d_2 (Conv2D) (None, 75, 75, 128) 147584 \n",
604 "_________________________________________________________________\n",
605 "block2_max_pooling2d (MaxPoo (None, 37, 37, 128) 0 \n",
606 "_________________________________________________________________\n",
607 "block3_conv2d_1 (Conv2D) (None, 37, 37, 256) 295168 \n",
608 "_________________________________________________________________\n",
609 "block3_conv2d_2 (Conv2D) (None, 37, 37, 256) 590080 \n",
610 "_________________________________________________________________\n",
611 "block3_conv2d_3 (Conv2D) (None, 37, 37, 256) 590080 \n",
612 "_________________________________________________________________\n",
613 "block3_max_pooling2d (MaxPoo (None, 18, 18, 256) 0 \n",
614 "_________________________________________________________________\n",
615 "block4_conv2d_1 (Conv2D) (None, 18, 18, 512) 1180160 \n",
616 "_________________________________________________________________\n",
617 "block4_conv2d_2 (Conv2D) (None, 18, 18, 512) 2359808 \n",
618 "_________________________________________________________________\n",
619 "block4_conv2d_3 (Conv2D) (None, 18, 18, 512) 2359808 \n",
620 "_________________________________________________________________\n",
621 "block4_max_pooling2d (MaxPoo (None, 9, 9, 512) 0 \n",
622 "_________________________________________________________________\n",
623 "block5_conv2d_1 (Conv2D) (None, 9, 9, 512) 2359808 \n",
624 "_________________________________________________________________\n",
625 "block5_conv2d_2 (Conv2D) (None, 9, 9, 512) 2359808 \n",
626 "_________________________________________________________________\n",
627 "block5_conv2d_3 (Conv2D) (None, 9, 9, 512) 2359808 \n",
628 "_________________________________________________________________\n",
629 "block5_max_pooling2d (MaxPoo (None, 4, 4, 512) 0 \n",
630 "=================================================================\n",
631 "Total params: 14,714,688\n",
632 "Trainable params: 14,714,688\n",
633 "Non-trainable params: 0\n",
634 "_________________________________________________________________\n",
635 "OUTPUT FROM mysummary(model):\n",
636 "block1_conv2d_1 (None, 150, 150, 64) 1792\n",
637 "block1_conv2d_2 (None, 150, 150, 64) 36928\n",
638 "block1_max_pooling2d (None, 75, 75, 64) 0\n",
639 "block2_conv2d_1 (None, 75, 75, 128) 73856\n",
640 "block2_conv2d_2 (None, 75, 75, 128) 147584\n",
641 "block2_max_pooling2d (None, 37, 37, 128) 0\n",
642 "block3_conv2d_1 (None, 37, 37, 256) 295168\n",
643 "block3_conv2d_2 (None, 37, 37, 256) 590080\n",
644 "block3_conv2d_3 (None, 37, 37, 256) 590080\n",
645 "block3_max_pooling2d (None, 18, 18, 256) 0\n",
646 "block4_conv2d_1 (None, 18, 18, 512) 1180160\n",
647 "block4_conv2d_2 (None, 18, 18, 512) 2359808\n",
648 "block4_conv2d_3 (None, 18, 18, 512) 2359808\n",
649 "block4_max_pooling2d (None, 9, 9, 512) 0\n",
650 "block5_conv2d_1 (None, 9, 9, 512) 2359808\n",
651 "block5_conv2d_2 (None, 9, 9, 512) 2359808\n",
652 "block5_conv2d_3 (None, 9, 9, 512) 2359808\n",
653 "block5_max_pooling2d (None, 4, 4, 512) 0\n"
654 ]
655 }
656 ],
657 "source": [
658 "# VGG16 coded manually\n",
659 "model = models.Sequential()\n",
660 "model.add(layers.Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2d_1', input_shape=(150, 150, 3)))\n",
661 "model.add(layers.Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2d_2'))\n",
662 "model.add(layers.MaxPooling2D((2, 2), strides=(2, 2), name='block1_max_pooling2d'))\n",
663 "model.add(layers.Conv2D(128, (3, 3), activation='relu', padding='same', name='block2_conv2d_1'))\n",
664 "model.add(layers.Conv2D(128, (3, 3), activation='relu', padding='same', name='block2_conv2d_2'))\n",
665 "model.add(layers.MaxPooling2D((2, 2), strides=(2, 2), name='block2_max_pooling2d'))\n",
666 "model.add(layers.Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv2d_1'))\n",
667 "model.add(layers.Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv2d_2'))\n",
668 "model.add(layers.Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv2d_3'))\n",
669 "model.add(layers.MaxPooling2D((2, 2), strides=(2, 2), name='block3_max_pooling2d'))\n",
670 "model.add(layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv2d_1'))\n",
671 "model.add(layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv2d_2'))\n",
672 "model.add(layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv2d_3'))\n",
673 "model.add(layers.MaxPooling2D((2, 2), strides=(2, 2), name='block4_max_pooling2d'))\n",
674 "model.add(layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv2d_1'))\n",
675 "model.add(layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv2d_2'))\n",
676 "model.add(layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv2d_3'))\n",
677 "model.add(layers.MaxPooling2D((2, 2), strides=(2, 2), name='block5_max_pooling2d'))\n",
678 "model.summary()\n",
679 "mysummary(model)"
680 ]
681 },
682 {
683 "cell_type": "code",
684 "execution_count": 12,
685 "metadata": {},
686 "outputs": [
687 {
688 "name": "stdout",
689 "output_type": "stream",
690 "text": [
691 "_________________________________________________________________\n",
692 "Layer (type) Output Shape Param # \n",
693 "=================================================================\n",
694 "input_2 (InputLayer) (None, 150, 150, 3) 0 \n",
695 "_________________________________________________________________\n",
696 "block1_conv1 (Conv2D) (None, 150, 150, 64) 1792 \n",
697 "_________________________________________________________________\n",
698 "block1_conv2 (Conv2D) (None, 150, 150, 64) 36928 \n",
699 "_________________________________________________________________\n",
700 "block1_pool (MaxPooling2D) (None, 75, 75, 64) 0 \n",
701 "_________________________________________________________________\n",
702 "block2_conv1 (Conv2D) (None, 75, 75, 128) 73856 \n",
703 "_________________________________________________________________\n",
704 "block2_conv2 (Conv2D) (None, 75, 75, 128) 147584 \n",
705 "_________________________________________________________________\n",
706 "block2_pool (MaxPooling2D) (None, 37, 37, 128) 0 \n",
707 "_________________________________________________________________\n",
708 "block3_conv1 (Conv2D) (None, 37, 37, 256) 295168 \n",
709 "_________________________________________________________________\n",
710 "block3_conv2 (Conv2D) (None, 37, 37, 256) 590080 \n",
711 "_________________________________________________________________\n",
712 "block3_conv3 (Conv2D) (None, 37, 37, 256) 590080 \n",
713 "_________________________________________________________________\n",
714 "block3_pool (MaxPooling2D) (None, 18, 18, 256) 0 \n",
715 "_________________________________________________________________\n",
716 "block4_conv1 (Conv2D) (None, 18, 18, 512) 1180160 \n",
717 "_________________________________________________________________\n",
718 "block4_conv2 (Conv2D) (None, 18, 18, 512) 2359808 \n",
719 "_________________________________________________________________\n",
720 "block4_conv3 (Conv2D) (None, 18, 18, 512) 2359808 \n",
721 "_________________________________________________________________\n",
722 "block4_pool (MaxPooling2D) (None, 9, 9, 512) 0 \n",
723 "_________________________________________________________________\n",
724 "block5_conv1 (Conv2D) (None, 9, 9, 512) 2359808 \n",
725 "_________________________________________________________________\n",
726 "block5_conv2 (Conv2D) (None, 9, 9, 512) 2359808 \n",
727 "_________________________________________________________________\n",
728 "block5_conv3 (Conv2D) (None, 9, 9, 512) 2359808 \n",
729 "_________________________________________________________________\n",
730 "block5_pool (MaxPooling2D) (None, 4, 4, 512) 0 \n",
731 "=================================================================\n",
732 "Total params: 14,714,688\n",
733 "Trainable params: 14,714,688\n",
734 "Non-trainable params: 0\n",
735 "_________________________________________________________________\n",
736 "OUTPUT FROM mysummary(model):\n",
737 "input_2 (None, 150, 150, 3) 0\n",
738 "block1_conv1 (None, 150, 150, 64) 1792\n",
739 "block1_conv2 (None, 150, 150, 64) 36928\n",
740 "block1_pool (None, 75, 75, 64) 0\n",
741 "block2_conv1 (None, 75, 75, 128) 73856\n",
742 "block2_conv2 (None, 75, 75, 128) 147584\n",
743 "block2_pool (None, 37, 37, 128) 0\n",
744 "block3_conv1 (None, 37, 37, 256) 295168\n",
745 "block3_conv2 (None, 37, 37, 256) 590080\n",
746 "block3_conv3 (None, 37, 37, 256) 590080\n",
747 "block3_pool (None, 18, 18, 256) 0\n",
748 "block4_conv1 (None, 18, 18, 512) 1180160\n",
749 "block4_conv2 (None, 18, 18, 512) 2359808\n",
750 "block4_conv3 (None, 18, 18, 512) 2359808\n",
751 "block4_pool (None, 9, 9, 512) 0\n",
752 "block5_conv1 (None, 9, 9, 512) 2359808\n",
753 "block5_conv2 (None, 9, 9, 512) 2359808\n",
754 "block5_conv3 (None, 9, 9, 512) 2359808\n",
755 "block5_pool (None, 4, 4, 512) 0\n"
756 ]
757 }
758 ],
759 "source": [
760 "# VGG16 loaded directly from keras\n",
761 "from keras.applications import VGG16\n",
762 "model = VGG16(weights='imagenet',\n",
763 " include_top=False,\n",
764 " input_shape=(150, 150, 3))\n",
765 "model.summary()\n",
766 "mysummary(model)"
767 ]
768 }
769 ],
770 "metadata": {
771 "kernelspec": {
772 "display_name": "fastai",
773 "language": "python",
774 "name": "fastai"
775 },
776 "language_info": {
777 "codemirror_mode": {
778 "name": "ipython",
779 "version": 3
780 },
781 "file_extension": ".py",
782 "mimetype": "text/x-python",
783 "name": "python",
784 "nbconvert_exporter": "python",
785 "pygments_lexer": "ipython3",
786 "version": "3.6.8"
787 }
788 },
789 "nbformat": 4,
790 "nbformat_minor": 2
791}