· 6 years ago · Sep 05, 2019, 12:04 AM
1{
2 "cells": [
3 {
4 "cell_type": "markdown",
5 "metadata": {},
6 "source": [
7 "# Understanding neural nets' _output shape_ and _number of parameters_ layer by layer\n",
8 "## <font color=green>functional API version</font>\n",
9 "> ... understanding how each layer of a neural network changes the shape of the data as it flows through the network is a key part of truly understanding the mechanics of deep learning ~ _David Foster_, _Generative Deep Learning_"
10 ]
11 },
12 {
13 "cell_type": "markdown",
14 "metadata": {},
15 "source": [
16 "In this notebook we look at 3 ways of finding the _output shape_ and _number of parameters_ layer by layer:\n",
17 "1. Keras' built-in method: <font color=blue>model.summary()</font>;\n",
18 "2. our home-made method: <font color=blue>mysummary(model)</font>;\n",
19 "3. manual back-of-the-envelope calculation.\n",
20 "\n",
21 "We beginning by <font color=green>def</font>ining our home-made method, <font color=blue>mysummary(model)</font>. \n",
22 "Worked examples of different network architectures, of fully connected and 2D convolution layers, will follow later.\n",
23 "\n",
24 "This notebook is the <font color=green>function API version</font>, which matches cell-by-cell with the companion <font color=green>sequential class version</font>. Sequential class and functional API are two different ways of defining a neural network architecture in Keras."
25 ]
26 },
27 {
28 "cell_type": "code",
29 "execution_count": 1,
30 "metadata": {},
31 "outputs": [
32 {
33 "name": "stderr",
34 "output_type": "stream",
35 "text": [
36 "Using TensorFlow backend.\n"
37 ]
38 }
39 ],
40 "source": [
41 "def mysummary(model):\n",
42 " PASS0 = True\n",
43 " print('OUTPUT FROM mysummary(model):')\n",
44 " for layer in model.layers:\n",
45 " if PASS0: \n",
46 " if len(layer.input_shape)>2 and layer.input_shape[1] != layer.input_shape[2]:\n",
47 " print('warning: input_shape not square')\n",
48 "# take layer.input_shape[1:] only if this is the first layer, which happens at the first pass\n",
49 "# take just [1:] because [0] is always None\n",
50 " input_shape = layer.input_shape[1:]\n",
51 " PASS0 = False\n",
52 " param = 0\n",
53 " if 'Conv2D' in str(layer.build):\n",
54 " if layer.kernel_size[0] != layer.kernel_size[1]:\n",
55 " print('warning: kernel size not square')\n",
56 " if layer.strides[0] != layer.strides[1]:\n",
57 " print('warning: strides not square')\n",
58 "# strides defaults to 1 and padding defaults to none in Keras\n",
59 "# when strides=1, output dimensions are therefore reduced by 2\n",
60 "# when user specificies strides>1, output dimensions are then reduced by that user-specified factor\n",
61 " if layer.strides[0]==1 and layer.strides[1]==1 and layer.padding=='valid':\n",
62 " output_shape = int(input_shape[0] - 2), int(input_shape[1] - 2), layer.filters\n",
63 " else:\n",
64 " output_shape = int(input_shape[0] // layer.strides[0]), int(input_shape[1] // layer.strides[1]), layer.filters\n",
65 "# the number of parameters from a conv2d layer depends on \n",
66 "# - the shape of conv2d kernel\n",
67 "# - the number of channels of the previous layer (or the input data, if it's the first layer)\n",
68 "# - the number of conv2d filters \n",
69 "# the one in the 'plus one' is for the bias\n",
70 " param = (layer.kernel_size[0] * layer.kernel_size[1] * layer.input_shape[3] + 1 ) * layer.filters\n",
71 " elif 'MaxPooling2D'in str(layer.build):\n",
72 " if layer.pool_size[0] != layer.pool_size[1]:\n",
73 " print('warning: pool size not square')\n",
74 "# maxpooling reduces output_shape[:2] by the user-specified pool sizes, without changing output_shape[2]\n",
75 " output_shape = int(input_shape[0] / layer.pool_size[0]), int(input_shape[1] / layer.pool_size[1]), input_shape[2]\n",
76 " elif 'Flatten' in str(layer.build):\n",
77 " output_shape = (input_shape[0] * input_shape[1] * input_shape[2], )\n",
78 " elif 'Dropout' in str(layer.build) or 'input' in str(layer.build):\n",
79 " output_shape = input_shape\n",
80 " elif 'Dense' in str(layer.build):\n",
81 "# output_shape from dense is the user-specified units\n",
82 " output_shape = (layer.units, )\n",
83 "# the number of parameters is units multiplied by (input shape plus 1), where 1 is for the bias\n",
84 " param = (input_shape[0] + 1) * layer.units\n",
85 "# verify that our formulae produce the same output as those calculated internally by Keras\n",
86 " assert output_shape == layer.output_shape[1:], (output_shape, layer.output_shape[1:])\n",
87 " assert param == layer.count_params()\n",
88 " print('{:29s}(None, '.format(layer.name), end='')\n",
89 " for t in output_shape[:-1]:\n",
90 " print(t, end=', ')\n",
91 "# some alignment cosmetics for printing\n",
92 " print('{})'.format(output_shape[-1]), end='')\n",
93 " if len(output_shape)==1:\n",
94 " print(' ', end='')\n",
95 " print(' '*(20 - len(str(output_shape))), param)\n",
96 " input_shape = output_shape\n",
97 "\n",
98 "from keras import models, layers"
99 ]
100 },
101 {
102 "cell_type": "markdown",
103 "metadata": {},
104 "source": [
105 "## Worked examples\n",
106 "In each cell that follows, we define a network architecture using functional API, and proceed to\n",
107 "1. call Keras' built-in method: <font color=blue>model.summary()</font>;\n",
108 "2. call <font color=blue>mysummary(model)</font> which we just <font color=green>def</font>ined;\n",
109 "3. markup inline with our manual back-of-the-envelop calculation of the _output shape_ and _number of parameters_ layer by layer."
110 ]
111 },
112 {
113 "cell_type": "code",
114 "execution_count": 2,
115 "metadata": {},
116 "outputs": [
117 {
118 "name": "stdout",
119 "output_type": "stream",
120 "text": [
121 "_________________________________________________________________\n",
122 "Layer (type) Output Shape Param # \n",
123 "=================================================================\n",
124 "input_1 (InputLayer) (None, 784) 0 \n",
125 "_________________________________________________________________\n",
126 "dense_1 (Dense) (None, 32) 25120 \n",
127 "_________________________________________________________________\n",
128 "dense_2 (Dense) (None, 32) 1056 \n",
129 "=================================================================\n",
130 "Total params: 26,176\n",
131 "Trainable params: 26,176\n",
132 "Non-trainable params: 0\n",
133 "_________________________________________________________________\n",
134 "OUTPUT FROM mysummary(model):\n",
135 "input_1 (None, 784) 0\n",
136 "dense_1 (None, 32) 25120\n",
137 "dense_2 (None, 32) 1056\n"
138 ]
139 }
140 ],
141 "source": [
142 "input_layer = layers.Input(shape=(784,))\n",
143 "x = layers.Dense(32)(input_layer)\n",
144 "# Output dimension of a dense layer is the user-defined units, which is 32 in this case\n",
145 "# Number of parameters = (784 + 1) * 32 = 25120\n",
146 "output_layer = layers.Dense(32)(x)\n",
147 "# Output dimension is again 32\n",
148 "# Number of parameters = (32 + 1) * 32 = 1056, where the first '32' is from layer dense_1, and the second '32' is from current layer, dense_2\n",
149 "model = models.Model(input_layer, output_layer)\n",
150 "model.summary()\n",
151 "mysummary(model)"
152 ]
153 },
154 {
155 "cell_type": "code",
156 "execution_count": 3,
157 "metadata": {
158 "scrolled": true
159 },
160 "outputs": [
161 {
162 "name": "stdout",
163 "output_type": "stream",
164 "text": [
165 "_________________________________________________________________\n",
166 "Layer (type) Output Shape Param # \n",
167 "=================================================================\n",
168 "input_2 (InputLayer) (None, 10000) 0 \n",
169 "_________________________________________________________________\n",
170 "dense_3 (Dense) (None, 16) 160016 \n",
171 "_________________________________________________________________\n",
172 "dense_4 (Dense) (None, 16) 272 \n",
173 "_________________________________________________________________\n",
174 "dense_5 (Dense) (None, 1) 17 \n",
175 "=================================================================\n",
176 "Total params: 160,305\n",
177 "Trainable params: 160,305\n",
178 "Non-trainable params: 0\n",
179 "_________________________________________________________________\n",
180 "OUTPUT FROM mysummary(model):\n",
181 "input_2 (None, 10000) 0\n",
182 "dense_3 (None, 16) 160016\n",
183 "dense_4 (None, 16) 272\n",
184 "dense_5 (None, 1) 17\n"
185 ]
186 }
187 ],
188 "source": [
189 "input_layer = layers.Input(shape=(10000,))\n",
190 "# activations affect neither the output dimensions nor the number of parameters\n",
191 "x = layers.Dense(16, activation='relu')(input_layer)\n",
192 "# Output shape = (None, user-specified units) = (None, 16)\n",
193 "# Number of parameters = (10000 + 1) * 16 = 160016\n",
194 "x = layers.Dense(16, activation='relu')(x)\n",
195 "# Output shape = (None, user-specified units) = (None, 16)\n",
196 "# Number of parameters = (16 + 1) * 16 = 272, where the first '16' is from layer dense_3 and the second '16' is from the current layer, dense_4\n",
197 "output_layer = layers.Dense(1, activation='sigmoid')(x)\n",
198 "# Output shape = (None, user-specified units) = (None, 1)\n",
199 "# Number of parameters = (16 + 1) * 1 = 17, where the 16 is from the current layer, dense_5, itself\n",
200 "model = models.Model(input_layer, output_layer)\n",
201 "model.summary()\n",
202 "mysummary(model)"
203 ]
204 },
205 {
206 "cell_type": "code",
207 "execution_count": 4,
208 "metadata": {},
209 "outputs": [
210 {
211 "name": "stdout",
212 "output_type": "stream",
213 "text": [
214 "_________________________________________________________________\n",
215 "Layer (type) Output Shape Param # \n",
216 "=================================================================\n",
217 "input_3 (InputLayer) (None, 10000) 0 \n",
218 "_________________________________________________________________\n",
219 "dense_6 (Dense) (None, 64) 640064 \n",
220 "_________________________________________________________________\n",
221 "dense_7 (Dense) (None, 64) 4160 \n",
222 "_________________________________________________________________\n",
223 "dense_8 (Dense) (None, 46) 2990 \n",
224 "=================================================================\n",
225 "Total params: 647,214\n",
226 "Trainable params: 647,214\n",
227 "Non-trainable params: 0\n",
228 "_________________________________________________________________\n",
229 "OUTPUT FROM mysummary(model):\n",
230 "input_3 (None, 10000) 0\n",
231 "dense_6 (None, 64) 640064\n",
232 "dense_7 (None, 64) 4160\n",
233 "dense_8 (None, 46) 2990\n"
234 ]
235 }
236 ],
237 "source": [
238 "input_layer = layers.Input(shape=(10000,))\n",
239 "x = layers.Dense(64, activation='relu')(input_layer)\n",
240 "# Output shape = (None, user-specified units) = (None, 64)\n",
241 "# Number of parameters = (10000 + 1) * 64 = 640064\n",
242 "x = layers.Dense(64, activation='relu')(x)\n",
243 "# Output shape = (None, user-specified units) = (None, 64)\n",
244 "# Number of parameters = (64 + 1) * 64 = 4160, where the first 64 is from layer dense_6 and the second '64' is from the current layer, dense_7\n",
245 "output_layer = layers.Dense(46, activation='softmax')(x)\n",
246 "# Output shape = (None, user-specified units) = (None, 64)\n",
247 "# Number of parameters = (64 + 1) * 46 = 2990\n",
248 "model = models.Model(input_layer, output_layer)\n",
249 "model.summary()\n",
250 "mysummary(model)"
251 ]
252 },
253 {
254 "cell_type": "code",
255 "execution_count": 5,
256 "metadata": {},
257 "outputs": [
258 {
259 "name": "stdout",
260 "output_type": "stream",
261 "text": [
262 "_________________________________________________________________\n",
263 "Layer (type) Output Shape Param # \n",
264 "=================================================================\n",
265 "input_4 (InputLayer) (None, 784) 0 \n",
266 "_________________________________________________________________\n",
267 "dense_9 (Dense) (None, 512) 401920 \n",
268 "_________________________________________________________________\n",
269 "dense_10 (Dense) (None, 10) 5130 \n",
270 "=================================================================\n",
271 "Total params: 407,050\n",
272 "Trainable params: 407,050\n",
273 "Non-trainable params: 0\n",
274 "_________________________________________________________________\n",
275 "OUTPUT FROM mysummary(model):\n",
276 "input_4 (None, 784) 0\n",
277 "dense_9 (None, 512) 401920\n",
278 "dense_10 (None, 10) 5130\n"
279 ]
280 }
281 ],
282 "source": [
283 "input_layer = layers.Input(shape=(28 * 28,))\n",
284 "x = layers.Dense(512, activation='relu')(input_layer)\n",
285 "# Output shape = (None, user-specified units) = (None, 512)\n",
286 "# Number of parameters = (28*28 + 1) * 512 = 401920\n",
287 "output_layer = layers.Dense(10, activation='softmax')(x)\n",
288 "# Output dimension = (None, user-specified units) = (None, 10)\n",
289 "# Number of parameters = (512 + 1) * 10 = 5130\n",
290 "model = models.Model(input_layer, output_layer)\n",
291 "model.summary()\n",
292 "mysummary(model)"
293 ]
294 },
295 {
296 "cell_type": "code",
297 "execution_count": 6,
298 "metadata": {},
299 "outputs": [
300 {
301 "name": "stdout",
302 "output_type": "stream",
303 "text": [
304 "_________________________________________________________________\n",
305 "Layer (type) Output Shape Param # \n",
306 "=================================================================\n",
307 "input_5 (InputLayer) (None, 28, 28, 1) 0 \n",
308 "_________________________________________________________________\n",
309 "conv2d_1 (Conv2D) (None, 26, 26, 32) 320 \n",
310 "_________________________________________________________________\n",
311 "conv2d_2 (Conv2D) (None, 24, 24, 64) 18496 \n",
312 "_________________________________________________________________\n",
313 "conv2d_3 (Conv2D) (None, 22, 22, 64) 36928 \n",
314 "=================================================================\n",
315 "Total params: 55,744\n",
316 "Trainable params: 55,744\n",
317 "Non-trainable params: 0\n",
318 "_________________________________________________________________\n",
319 "OUTPUT FROM mysummary(model):\n",
320 "input_5 (None, 28, 28, 1) 0\n",
321 "conv2d_1 (None, 26, 26, 32) 320\n",
322 "conv2d_2 (None, 24, 24, 64) 18496\n",
323 "conv2d_3 (None, 22, 22, 64) 36928\n"
324 ]
325 }
326 ],
327 "source": [
328 "input_layer = layers.Input(shape=(28, 28, 1))\n",
329 "x = layers.Conv2D(32, (3, 3), activation='relu')(input_layer)\n",
330 "# Output shape = (None, 28-2, 28-2, 32)\n",
331 "# Number of parameters = (3*3*1 + 1) * 32 = 320\n",
332 "x = layers.Conv2D(64, (3, 3), activation='relu')(x)\n",
333 "# Output shape = (None, 26-2, 26-2, 64\n",
334 "# Number of parameters = (3*3*32 + 1) * 64 = 18496\n",
335 "output_layer = layers.Conv2D(64, (3, 3), activation='relu')(x)\n",
336 "# Output shape = (None, 24-2, 24-2, 64\n",
337 "# Number of parameters = (3*3*64 + 1) * 64 = 36928\n",
338 "model = models.Model(input_layer, output_layer)\n",
339 "model.summary()\n",
340 "mysummary(model)"
341 ]
342 },
343 {
344 "cell_type": "code",
345 "execution_count": 7,
346 "metadata": {},
347 "outputs": [
348 {
349 "name": "stdout",
350 "output_type": "stream",
351 "text": [
352 "_________________________________________________________________\n",
353 "Layer (type) Output Shape Param # \n",
354 "=================================================================\n",
355 "input_6 (InputLayer) (None, 28, 28, 1) 0 \n",
356 "_________________________________________________________________\n",
357 "conv2d_4 (Conv2D) (None, 26, 26, 32) 320 \n",
358 "_________________________________________________________________\n",
359 "max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32) 0 \n",
360 "_________________________________________________________________\n",
361 "conv2d_5 (Conv2D) (None, 11, 11, 64) 18496 \n",
362 "_________________________________________________________________\n",
363 "max_pooling2d_2 (MaxPooling2 (None, 5, 5, 64) 0 \n",
364 "_________________________________________________________________\n",
365 "conv2d_6 (Conv2D) (None, 3, 3, 64) 36928 \n",
366 "_________________________________________________________________\n",
367 "flatten_1 (Flatten) (None, 576) 0 \n",
368 "_________________________________________________________________\n",
369 "dense_11 (Dense) (None, 64) 36928 \n",
370 "_________________________________________________________________\n",
371 "dense_12 (Dense) (None, 10) 650 \n",
372 "=================================================================\n",
373 "Total params: 93,322\n",
374 "Trainable params: 93,322\n",
375 "Non-trainable params: 0\n",
376 "_________________________________________________________________\n",
377 "OUTPUT FROM mysummary(model):\n",
378 "input_6 (None, 28, 28, 1) 0\n",
379 "conv2d_4 (None, 26, 26, 32) 320\n",
380 "max_pooling2d_1 (None, 13, 13, 32) 0\n",
381 "conv2d_5 (None, 11, 11, 64) 18496\n",
382 "max_pooling2d_2 (None, 5, 5, 64) 0\n",
383 "conv2d_6 (None, 3, 3, 64) 36928\n",
384 "flatten_1 (None, 576) 0\n",
385 "dense_11 (None, 64) 36928\n",
386 "dense_12 (None, 10) 650\n"
387 ]
388 }
389 ],
390 "source": [
391 "input_layer = layers.Input(shape=(28, 28, 1))\n",
392 "x = layers.Conv2D(32, (3, 3), activation='relu')(input_layer)\n",
393 "# Output shape = (None, 28-2, 28-2, 32)\n",
394 "# Number of parameters = (3*3*1 + 1) * 32 = 320\n",
395 "x = layers.MaxPooling2D(2, 2)(x)\n",
396 "# Output dimensions = 26/2, 26/2, 32\n",
397 "x = layers.Conv2D(64, (3, 3), activation='relu')(x)\n",
398 "# Output shape = (None, 13-2, 13-2, 64)\n",
399 "# Number of parameters = (3*3*32 + 1) * 64 = 18496\n",
400 "x = layers.MaxPooling2D(2, 2)(x)\n",
401 "# Output dimensions = floor(11/2), floor(11/2), 64\n",
402 "x = layers.Conv2D(64, (3, 3), activation='relu')(x)\n",
403 "# Output shape = (None, 5-2, 5-2, 64)\n",
404 "# Number of parameters = (3*3*64 + 1) * 64 = 36928\n",
405 "x = layers.Flatten()(x)\n",
406 "# Output shape = (None, 3 * 3 * 64) = (None, 576)\n",
407 "x = layers.Dense(64, activation='relu')(x)\n",
408 "# Output shape = (None, user-specified units) = (None, 64)\n",
409 "# Number of parameters = (576 + 1) * 64 = 36928\n",
410 "output_layer = layers.Dense(10, activation='relu')(x)\n",
411 "# Output shape = (None, user-specified units) = (None, 10)\n",
412 "# Number of parameters = (64 + 1) * 10 = 650\n",
413 "model = models.Model(input_layer, output_layer)\n",
414 "model.summary()\n",
415 "mysummary(model)"
416 ]
417 },
418 {
419 "cell_type": "code",
420 "execution_count": 8,
421 "metadata": {},
422 "outputs": [
423 {
424 "name": "stdout",
425 "output_type": "stream",
426 "text": [
427 "_________________________________________________________________\n",
428 "Layer (type) Output Shape Param # \n",
429 "=================================================================\n",
430 "input_7 (InputLayer) (None, 32, 32, 3) 0 \n",
431 "_________________________________________________________________\n",
432 "flatten_2 (Flatten) (None, 3072) 0 \n",
433 "_________________________________________________________________\n",
434 "dense_13 (Dense) (None, 200) 614600 \n",
435 "_________________________________________________________________\n",
436 "dense_14 (Dense) (None, 150) 30150 \n",
437 "_________________________________________________________________\n",
438 "dense_15 (Dense) (None, 10) 1510 \n",
439 "=================================================================\n",
440 "Total params: 646,260\n",
441 "Trainable params: 646,260\n",
442 "Non-trainable params: 0\n",
443 "_________________________________________________________________\n",
444 "OUTPUT FROM mysummary(model):\n",
445 "input_7 (None, 32, 32, 3) 0\n",
446 "flatten_2 (None, 3072) 0\n",
447 "dense_13 (None, 200) 614600\n",
448 "dense_14 (None, 150) 30150\n",
449 "dense_15 (None, 10) 1510\n"
450 ]
451 }
452 ],
453 "source": [
454 "input_layer = layers.Input(shape=(32, 32, 3))\n",
455 "x = layers.Flatten()(input_layer)\n",
456 "# Output shape = (None, 32 * 32 * 3)\n",
457 "x = layers.Dense(200, activation='relu')(x)\n",
458 "# Output shape = (None, user-defined units) = (None, 200)\n",
459 "# Number of parameters = (3072 + 1 ) * 200 = 614600\n",
460 "x = layers.Dense(150, activation='relu')(x)\n",
461 "# Output shape = (None, user-defined units) = (None, 150)\n",
462 "# Number of parameters = (200 + 1) * 150 = 30150\n",
463 "output_layer = layers.Dense(10, activation='softmax')(x)\n",
464 "# Output shape = (None, user-defined units) = (None, 10)\n",
465 "# Number of parameters = (150 + 1) * 10 = 1510\n",
466 "model = models.Model(input_layer, output_layer)\n",
467 "model.summary()\n",
468 "mysummary(model)"
469 ]
470 },
471 {
472 "cell_type": "code",
473 "execution_count": 9,
474 "metadata": {},
475 "outputs": [
476 {
477 "name": "stdout",
478 "output_type": "stream",
479 "text": [
480 "_________________________________________________________________\n",
481 "Layer (type) Output Shape Param # \n",
482 "=================================================================\n",
483 "input_8 (InputLayer) (None, 32, 32, 3) 0 \n",
484 "_________________________________________________________________\n",
485 "conv2d_7 (Conv2D) (None, 16, 16, 10) 490 \n",
486 "_________________________________________________________________\n",
487 "conv2d_8 (Conv2D) (None, 8, 8, 20) 1820 \n",
488 "_________________________________________________________________\n",
489 "flatten_3 (Flatten) (None, 1280) 0 \n",
490 "_________________________________________________________________\n",
491 "dense_16 (Dense) (None, 10) 12810 \n",
492 "=================================================================\n",
493 "Total params: 15,120\n",
494 "Trainable params: 15,120\n",
495 "Non-trainable params: 0\n",
496 "_________________________________________________________________\n",
497 "OUTPUT FROM mysummary(model):\n",
498 "input_8 (None, 32, 32, 3) 0\n",
499 "conv2d_7 (None, 16, 16, 10) 490\n",
500 "conv2d_8 (None, 8, 8, 20) 1820\n",
501 "flatten_3 (None, 1280) 0\n",
502 "dense_16 (None, 10) 12810\n"
503 ]
504 }
505 ],
506 "source": [
507 "input_layer = layers.Input(shape=(32, 32, 3))\n",
508 "x = layers.Conv2D(10, (4, 4), strides = 2, padding = 'same')(input_layer)\n",
509 "# Output shape = (None, 32/2, 32/2, 10)\n",
510 "# Number of parameters = ( 4*4*3 + 1 ) * 10\n",
511 "x = layers.Conv2D(20, (3, 3), strides = 2, padding = 'same')(x)\n",
512 "# Output shape = (None, 16/2, 16/2, 20)\n",
513 "# Number of parameters = ( 3*3*10 + 1 ) * 20\n",
514 "x = layers.Flatten()(x)\n",
515 "# Output shape = (None, 8 * 8 * 20)\n",
516 "output_layer = layers.Dense(10, activation='softmax')(x)\n",
517 "# Output shape = (None, user-defined units) = (None, 10)\n",
518 "# Number of parameters = ( 1280 + 1 ) * 10\n",
519 "model = models.Model(input_layer, output_layer)\n",
520 "model.summary()\n",
521 "mysummary(model)"
522 ]
523 },
524 {
525 "cell_type": "code",
526 "execution_count": 10,
527 "metadata": {},
528 "outputs": [
529 {
530 "name": "stdout",
531 "output_type": "stream",
532 "text": [
533 "_________________________________________________________________\n",
534 "Layer (type) Output Shape Param # \n",
535 "=================================================================\n",
536 "input_9 (InputLayer) (None, 150, 150, 3) 0 \n",
537 "_________________________________________________________________\n",
538 "conv2d_9 (Conv2D) (None, 148, 148, 32) 896 \n",
539 "_________________________________________________________________\n",
540 "max_pooling2d_3 (MaxPooling2 (None, 74, 74, 32) 0 \n",
541 "_________________________________________________________________\n",
542 "conv2d_10 (Conv2D) (None, 72, 72, 64) 18496 \n",
543 "_________________________________________________________________\n",
544 "max_pooling2d_4 (MaxPooling2 (None, 36, 36, 64) 0 \n",
545 "_________________________________________________________________\n",
546 "conv2d_11 (Conv2D) (None, 34, 34, 128) 73856 \n",
547 "_________________________________________________________________\n",
548 "max_pooling2d_5 (MaxPooling2 (None, 17, 17, 128) 0 \n",
549 "_________________________________________________________________\n",
550 "conv2d_12 (Conv2D) (None, 15, 15, 128) 147584 \n",
551 "_________________________________________________________________\n",
552 "max_pooling2d_6 (MaxPooling2 (None, 7, 7, 128) 0 \n",
553 "_________________________________________________________________\n",
554 "flatten_4 (Flatten) (None, 6272) 0 \n",
555 "_________________________________________________________________\n",
556 "dropout_1 (Dropout) (None, 6272) 0 \n",
557 "_________________________________________________________________\n",
558 "dense_17 (Dense) (None, 512) 3211776 \n",
559 "_________________________________________________________________\n",
560 "dense_18 (Dense) (None, 1) 513 \n",
561 "=================================================================\n",
562 "Total params: 3,453,121\n",
563 "Trainable params: 3,453,121\n",
564 "Non-trainable params: 0\n",
565 "_________________________________________________________________\n",
566 "OUTPUT FROM mysummary(model):\n",
567 "input_9 (None, 150, 150, 3) 0\n",
568 "conv2d_9 (None, 148, 148, 32) 896\n",
569 "max_pooling2d_3 (None, 74, 74, 32) 0\n",
570 "conv2d_10 (None, 72, 72, 64) 18496\n",
571 "max_pooling2d_4 (None, 36, 36, 64) 0\n",
572 "conv2d_11 (None, 34, 34, 128) 73856\n",
573 "max_pooling2d_5 (None, 17, 17, 128) 0\n",
574 "conv2d_12 (None, 15, 15, 128) 147584\n",
575 "max_pooling2d_6 (None, 7, 7, 128) 0\n",
576 "flatten_4 (None, 6272) 0\n",
577 "dropout_1 (None, 6272) 0\n",
578 "dense_17 (None, 512) 3211776\n",
579 "dense_18 (None, 1) 513\n"
580 ]
581 }
582 ],
583 "source": [
584 "input_layer = layers.Input(shape=(150, 150, 3))\n",
585 "x = layers.Conv2D(32, (3, 3), activation='relu')(input_layer)\n",
586 "# Output shape = (None, 150-2, 150-2, 32)\n",
587 "# Number of parameters = (3*3*3 + 1) * 32 = 896\n",
588 "x = layers.MaxPooling2D(2, 2)(x) \n",
589 "# Output shape = (None, 148/2, 148/2, 32) \n",
590 "x = layers.Conv2D(64, (3, 3), activation='relu')(x) \n",
591 "# Output shape = (None, 74-2, 74-2, 64)\n",
592 "# Number of parameters = (3*3*32 + 1) * 64 = 18496\n",
593 "x = layers.MaxPooling2D(2, 2)(x)\n",
594 "# Output shape = (None, 72/2, 72/2, 64)\n",
595 "x = layers.Conv2D(128, (3, 3), activation='relu')(x)\n",
596 "# Output shape = (None, 36-2, 36-2, 128)\n",
597 "# Number of parameters = (3*3*64 + 1) * 128 = 73856\n",
598 "x = layers.MaxPooling2D(2, 2)(x)\n",
599 "# Output shape = (None, 34/2, 34/2, 128)\n",
600 "x = layers.Conv2D(128, (3, 3), activation='relu')(x)\n",
601 "# Output shape = (None, 17-2, 17-2, 128)\n",
602 "# Number of parameters = (3*3*128 + 1) * 128 = 147584\n",
603 "x = layers.MaxPooling2D(2, 2)(x)\n",
604 "# Output shape = (None, floor(15/2), floor(15/2), 128)\n",
605 "x = layers.Flatten()(x)\n",
606 "# Output shape = (None, 7 * 7 * 128) = (None, 6272)\n",
607 "x = layers.Dropout(.5)(x)\n",
608 "# Output shape unchanged\n",
609 "x = layers.Dense(512, activation='relu')(x)\n",
610 "# Output shape = (None, user-specified units) = (None, 512)\n",
611 "# Number of parameters = (6272 + 1) * 512 = 3211776\n",
612 "output_layer = layers.Dense(1, activation='sigmoid')(x)\n",
613 "# Output shape = (None, user-specified units) = (None, 1)\n",
614 "# Number of parameters = (512 + 1) * 1 = 513\n",
615 "model = models.Model(input_layer, output_layer)\n",
616 "model.summary()\n",
617 "mysummary(model)"
618 ]
619 },
620 {
621 "cell_type": "code",
622 "execution_count": 11,
623 "metadata": {},
624 "outputs": [
625 {
626 "name": "stdout",
627 "output_type": "stream",
628 "text": [
629 "_________________________________________________________________\n",
630 "Layer (type) Output Shape Param # \n",
631 "=================================================================\n",
632 "input_10 (InputLayer) (None, 150, 150, 3) 0 \n",
633 "_________________________________________________________________\n",
634 "block1_conv2d_1 (Conv2D) (None, 150, 150, 64) 1792 \n",
635 "_________________________________________________________________\n",
636 "block1_conv2d_2 (Conv2D) (None, 150, 150, 64) 36928 \n",
637 "_________________________________________________________________\n",
638 "block1_max_pooling2d (MaxPoo (None, 75, 75, 64) 0 \n",
639 "_________________________________________________________________\n",
640 "block2_conv2d_1 (Conv2D) (None, 75, 75, 128) 73856 \n",
641 "_________________________________________________________________\n",
642 "block2_conv2d_2 (Conv2D) (None, 75, 75, 128) 147584 \n",
643 "_________________________________________________________________\n",
644 "block2_max_pooling2d (MaxPoo (None, 37, 37, 128) 0 \n",
645 "_________________________________________________________________\n",
646 "block3_conv2d_1 (Conv2D) (None, 37, 37, 256) 295168 \n",
647 "_________________________________________________________________\n",
648 "block3_conv2d_2 (Conv2D) (None, 37, 37, 256) 590080 \n",
649 "_________________________________________________________________\n",
650 "block3_conv2d_3 (Conv2D) (None, 37, 37, 256) 590080 \n",
651 "_________________________________________________________________\n",
652 "block3_max_pooling2d (MaxPoo (None, 18, 18, 256) 0 \n",
653 "_________________________________________________________________\n",
654 "block4_conv2d_1 (Conv2D) (None, 18, 18, 512) 1180160 \n",
655 "_________________________________________________________________\n",
656 "block4_conv2d_2 (Conv2D) (None, 18, 18, 512) 2359808 \n",
657 "_________________________________________________________________\n",
658 "block4_conv2d_3 (Conv2D) (None, 18, 18, 512) 2359808 \n",
659 "_________________________________________________________________\n",
660 "block4_max_pooling2d (MaxPoo (None, 9, 9, 512) 0 \n",
661 "_________________________________________________________________\n",
662 "block5_conv2d_1 (Conv2D) (None, 9, 9, 512) 2359808 \n",
663 "_________________________________________________________________\n",
664 "block5_conv2d_2 (Conv2D) (None, 9, 9, 512) 2359808 \n",
665 "_________________________________________________________________\n",
666 "block5_conv2d_3 (Conv2D) (None, 9, 9, 512) 2359808 \n",
667 "_________________________________________________________________\n",
668 "block5_max_pooling2d (MaxPoo (None, 4, 4, 512) 0 \n",
669 "=================================================================\n",
670 "Total params: 14,714,688\n",
671 "Trainable params: 14,714,688\n",
672 "Non-trainable params: 0\n",
673 "_________________________________________________________________\n",
674 "OUTPUT FROM mysummary(model):\n",
675 "input_10 (None, 150, 150, 3) 0\n",
676 "block1_conv2d_1 (None, 150, 150, 64) 1792\n",
677 "block1_conv2d_2 (None, 150, 150, 64) 36928\n",
678 "block1_max_pooling2d (None, 75, 75, 64) 0\n",
679 "block2_conv2d_1 (None, 75, 75, 128) 73856\n",
680 "block2_conv2d_2 (None, 75, 75, 128) 147584\n",
681 "block2_max_pooling2d (None, 37, 37, 128) 0\n",
682 "block3_conv2d_1 (None, 37, 37, 256) 295168\n",
683 "block3_conv2d_2 (None, 37, 37, 256) 590080\n",
684 "block3_conv2d_3 (None, 37, 37, 256) 590080\n",
685 "block3_max_pooling2d (None, 18, 18, 256) 0\n",
686 "block4_conv2d_1 (None, 18, 18, 512) 1180160\n",
687 "block4_conv2d_2 (None, 18, 18, 512) 2359808\n",
688 "block4_conv2d_3 (None, 18, 18, 512) 2359808\n",
689 "block4_max_pooling2d (None, 9, 9, 512) 0\n",
690 "block5_conv2d_1 (None, 9, 9, 512) 2359808\n",
691 "block5_conv2d_2 (None, 9, 9, 512) 2359808\n",
692 "block5_conv2d_3 (None, 9, 9, 512) 2359808\n",
693 "block5_max_pooling2d (None, 4, 4, 512) 0\n"
694 ]
695 }
696 ],
697 "source": [
698 "# VGG16 coded manually\n",
699 "input_layer = layers.Input(shape=(150, 150, 3))\n",
700 "x = layers.Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2d_1')(input_layer)\n",
701 "x = layers.Conv2D(64, (3, 3), activation='relu', padding='same', name='block1_conv2d_2')(x)\n",
702 "x = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block1_max_pooling2d')(x)\n",
703 "x = layers.Conv2D(128, (3, 3), activation='relu', padding='same', name='block2_conv2d_1')(x)\n",
704 "x = layers.Conv2D(128, (3, 3), activation='relu', padding='same', name='block2_conv2d_2')(x)\n",
705 "x = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block2_max_pooling2d')(x)\n",
706 "x = layers.Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv2d_1')(x)\n",
707 "x = layers.Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv2d_2')(x)\n",
708 "x = layers.Conv2D(256, (3, 3), activation='relu', padding='same', name='block3_conv2d_3')(x)\n",
709 "x = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block3_max_pooling2d')(x)\n",
710 "x = layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv2d_1')(x)\n",
711 "x = layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv2d_2')(x)\n",
712 "x = layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block4_conv2d_3')(x)\n",
713 "x = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block4_max_pooling2d')(x)\n",
714 "x = layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv2d_1')(x)\n",
715 "x = layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv2d_2')(x)\n",
716 "x = layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block5_conv2d_3')(x)\n",
717 "output_layer = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block5_max_pooling2d')(x)\n",
718 "'''\n",
719 "x = layers.Flatten()(x)\n",
720 "x = layers.Dense(4096, activation='relu')(x)\n",
721 "x = layers.Dense(4096, activation='relu')(x)\n",
722 "output_layer = layers.Dense(10, activation='softmax')(x)\n",
723 "'''\n",
724 "model = models.Model(input_layer, output_layer)\n",
725 "model.summary()\n",
726 "mysummary(model)"
727 ]
728 },
729 {
730 "cell_type": "code",
731 "execution_count": null,
732 "metadata": {},
733 "outputs": [],
734 "source": []
735 }
736 ],
737 "metadata": {
738 "kernelspec": {
739 "display_name": "fastai",
740 "language": "python",
741 "name": "fastai"
742 },
743 "language_info": {
744 "codemirror_mode": {
745 "name": "ipython",
746 "version": 3
747 },
748 "file_extension": ".py",
749 "mimetype": "text/x-python",
750 "name": "python",
751 "nbconvert_exporter": "python",
752 "pygments_lexer": "ipython3",
753 "version": "3.6.8"
754 }
755 },
756 "nbformat": 4,
757 "nbformat_minor": 2
758}