AIFFEL 아이펠 26일차

딥네트워크, 서로 뭐가 다른 거죠?

ImageNet: 대량의 이미지 데이터를 포함하는 데이터셋

https://www.ted.com/talks/fei_fei_li_how_we_re_teaching_computers_to_understand_pictures

How we're teaching computers to understand pictures

AlexNet

2012년 SuperVision팀이 오류율 16%로 이전에 비해서 오류율이 훨씬 감소한 결과를 보여줬다. 오류율을 낮출 수 있었던 이유는 네트워크를 깊게 쌓았기 때문이다. 이 네트워크의 이름은 AlexNet으로 지었다. MNIST에 CNN과 클래스가 많이 있는 네트워크로 볼 수 있으며, 여기에 ReLU 활성화 함수와 드롭아웃, 그리고 오버래핑 풀링 등이 적용되었다.

https://bskyvision.com/421

[CNN 알고리즘들] AlexNet의 구조

VGG

AlexNet과 같이 이미지넷 챌린지에서 공개된 모델이다. 2014년 준우승을 거뒀는데, 간결한 구조로 많은 활용이 이뤄졌다. 이전에 우승한 네트워크들이 10개가 안되는 CNN층을 가진 반면, VGG는 16개, 19개의 층으로 이뤄졌다.(이름 뒤에 VGG16, VGG19처럼 숫자가 붙음)

VGG는 3x3 커널을 사용해 더 많은 레이어를 쌓고 이미지의 비선형적 특성을 더 잘 잡아낼 수 있게 만들었다. 레이어가 많아지면 연산이 많아진다. 입/출력 채널의 수를 C라고 했을 때, 커널의 크기가 7x7인 CNN레이어를 한 번 쌓으면 7x7xC^2번 연산을 해야한다. 커널의 크기가 5x5인 CNN은 5x5xC^2의 연산이 필요하다. 3x3은? 레이어가 N개 일 때, Nx3x3xC^2의 연산으로 레이어가 3개더라도 5x5인 레이어 1개와 비슷한 연산수를 유지하게 한다.

https://bskyvision.com/504

[CNN 알고리즘들] VGGNet의 구조 (VGG16)

https://m.blog.naver.com/laonple/220686328027

[Part Ⅴ. Best CNN Architecture] 5. GoogLeNet [1] - 라온피플 머신러닝 아카데미 -

https://ratsgo.github.io/deep%20learning/2017/10/09/CNNs/

CNN 주요 모델들 · ratsgo's blog

네트워크를 깊게 쌓는 것은 모델이 깊어질수록 모델 학습을 위한 Gradient가 사라지는 현상이 발생하기 때문이다. 

네트워크는 Gradient descent를 통해서 기울기를 학습하는데 깊은 레이어에는 데이터에 따른 차이가 충분하게 반영되지 못한다.

기울기 소실(경사소실, Vanishing Gadient): Gradient가 너무 작아져서 레이어를 학습시키기 위해 충분한 값을 표현하지 못하는 경우

레이어가 깊어지면서 Gradient가 매우 커지거나 작아진다. 레이어의 가중치가 반복해서 곱해지면, 1보다 작을 때는 0에 너무 가까워지고, 1보다 클 때는 그 값이 기하급수적으로 커진다.

https://www.youtube.com/watch?v=qhXZsFVxGKo 

ResNet

Vanishing/Exploding Gradient문제를 해결하기 위해 ResNet은 간단한 방법을 사용했다. ResNet은 152개의 레이어를 사용했는데, Skip Connection이라는 구조를 사용해서 Vanishing Gradient문제를 해결했다.

Skip Connection은 레이어의 입력을 다른 곳에 이어서 Gradient가 깊은 곳까지 이어지도록 한다. 아래 그림처럼 레이어와 Skip Connection이 있는 블록을 Residual Block이라고 한다.

https://www.youtube.com/watch?v=ZILIbUvp5lk 

---

분류기 모델을 VGG16으로 바꿔보자!

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers

# CIFAR100 데이터셋을 가져옵시다. 
cifar100 = keras.datasets.cifar100

(x_train, y_train), (x_test, y_test) = cifar100.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

print("x_train:", len(x_train), "x_test:", len(x_test))
>>> x_train: 50000 x_test: 10000

img_input = keras.Input(shape=(32, 32, 3))

x = keras.layers.Conv2D(16, 3, activation='relu')(img_input)
x = keras.layers.MaxPool2D((2,2))(x)
x = keras.layers.Conv2D(32, 3, activation='relu')(x)
x = keras.layers.MaxPool2D((2,2))(x)
x = keras.layers.Flatten()(x)
x = keras.layers.Dense(256, activation='relu')(x)
predictions = keras.layers.Dense(100, activation='softmax')(x)

model = keras.Model(inputs=img_input, outputs=predictions)

model.summary()

# 모델 학습! 구조를 파악하기 위한 것이기 때문에 1Epoch만 학습한다
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=1)

https://github.com/keras-team/keras-applications/blob/master/keras_applications/vgg16.py

GitHub - keras-team/keras-applications: Reference implementations of popular deep learning models.

여기서 구현 코드를 보고 각 블록별 코드를 짠다.

# 첫 번째 블록
x = layers.Conv2D(64, (3, 3),
                  activation='relu',
                  padding='same',
                  name='block1_conv1')(img_input)
x = layers.Conv2D(64, (3, 3),
                  activation='relu',
                  padding='same',
                  name='block1_conv2')(x)
x = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block1_pool')(x)

# 두 번째 블록

# [[YOUR CODE]]
x = layers.Conv2D(128, (3, 3), activation='relu', padding='same', name='block2conv1')(x)
x = layers.Conv2D(128, (3, 3), activation='relu', padding='same', name='block2conv2')(x)
x = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block2_pool')(x)

# 세 번째 블록

# [[YOUR CODE]]
x = layers.Conv2D(256, (3, 3), activation='relu', padding='same', name='block3conv1')(x)
x = layers.Conv2D(256, (3, 3), activation='relu', padding='same', name='block3conv2')(x)
x = layers.Conv2D(256, (3, 3), activation='relu', padding='same', name='block3conv3')(x)
x = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block3pool')(x)

# 네 번째 블록

# [[YOUR CODE]]
x = layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block4conv1')(x)
x = layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block4conv2')(x)
x = layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block4conv3')(x)
x = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block4pool')(x)

# 다섯 번째 블록

# [[YOUR CODE]]
x = layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block5conv1')(x)
x = layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block5conv2')(x)
x = layers.Conv2D(512, (3, 3), activation='relu', padding='same', name='block5conv3')(x)
x = layers.MaxPooling2D((2, 2), strides=(2, 2), name='block5pool')(x)

# 여섯 번째 블록
# [Keras VGG16 코드 구현] 링크의 if include_top: 부분을 유심히 보세요 

# [[YOUR CODE]]
x = layers.Flatten(name='flatten')(x)
x = layers.Dense(4096, activation='relu', name='fc1')(x)
x = layers.Dense(4096, activation='relu', name='fc2')(x)

classes=100
x = layers.Dense(classes, activation='softmax', name='predictions')(x)    # CIFAR100을 위한 모델 Output

model = keras.Model(name="VGG-16", inputs=img_input, outputs=x)

model.summary()

Model: "VGG-16"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_3 (InputLayer)         [(None, 32, 32, 3)]       0         
_________________________________________________________________
block1_conv1 (Conv2D)        (None, 32, 32, 64)        1792      
_________________________________________________________________
block1_conv2 (Conv2D)        (None, 32, 32, 64)        36928     
_________________________________________________________________
block1_pool (MaxPooling2D)   (None, 16, 16, 64)        0         
_________________________________________________________________
block2conv1 (Conv2D)         (None, 16, 16, 128)       73856     
_________________________________________________________________
block2conv2 (Conv2D)         (None, 16, 16, 128)       147584    
_________________________________________________________________
block2_pool (MaxPooling2D)   (None, 8, 8, 128)         0         
_________________________________________________________________
block3conv1 (Conv2D)         (None, 8, 8, 256)         295168    
_________________________________________________________________
block3conv2 (Conv2D)         (None, 8, 8, 256)         590080    
_________________________________________________________________
block3conv3 (Conv2D)         (None, 8, 8, 256)         590080    
_________________________________________________________________
block3pool (MaxPooling2D)    (None, 4, 4, 256)         0         
_________________________________________________________________
block4conv1 (Conv2D)         (None, 4, 4, 512)         1180160   
_________________________________________________________________
block4conv2 (Conv2D)         (None, 4, 4, 512)         2359808   
_________________________________________________________________
block4conv3 (Conv2D)         (None, 4, 4, 512)         2359808   
_________________________________________________________________
block4pool (MaxPooling2D)    (None, 2, 2, 512)         0         
_________________________________________________________________
block5conv1 (Conv2D)         (None, 2, 2, 512)         2359808   
_________________________________________________________________
block5conv2 (Conv2D)         (None, 2, 2, 512)         2359808   
_________________________________________________________________
block5conv3 (Conv2D)         (None, 2, 2, 512)         2359808   
_________________________________________________________________
block5pool (MaxPooling2D)    (None, 1, 1, 512)         0         
_________________________________________________________________
flatten (Flatten)            (None, 512)               0         
_________________________________________________________________
fc1 (Dense)                  (None, 4096)              2101248   
_________________________________________________________________
fc2 (Dense)                  (None, 4096)              16781312  
_________________________________________________________________
predictions (Dense)          (None, 100)               409700    
=================================================================
Total params: 34,006,948
Trainable params: 34,006,948
Non-trainable params: 0
_________________________________________________________________

# 모델 학습
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=1)

---

ResNet 으로 바꿔보자!

아래에서 구현 코드를 참고하자.

https://git.dst.etit.tu-chemnitz.de/external/tf-models/-/blob/1d057dfc32f515a63ab1e23fd72052ab2a954952/official/vision/image_classification/resnet_model.py

official/vision/image_classification/resnet_model.py · 1d057dfc32f515a63ab1e23fd72052ab2a954952 · external / TensorFlow Models

# 추가로 import해야 할 패키지들을 먼저 가져온다
from tensorflow.keras import backend, regularizers, initializers, models

# block 안에 반복적으로 활용되는 L2 regularizer를 선언해 줍니다.
def _gen_l2_regularizer(use_l2_regularizer=True, l2_weight_decay=1e-4):
  return regularizers.l2(l2_weight_decay) if use_l2_regularizer else None

def conv_block(input_tensor,
               kernel_size,
               filters,
               stage,
               block,
               strides=(2, 2),
               use_l2_regularizer=True,
               batch_norm_decay=0.9,
               batch_norm_epsilon=1e-5):
  """A block that has a conv layer at shortcut.
  Note that from stage 3,
  the second conv layer at main path is with strides=(2, 2)
  And the shortcut should have strides=(2, 2) as well
  Args:
    input_tensor: input tensor
    kernel_size: default 3, the kernel size of middle conv layer at main path
    filters: list of integers, the filters of 3 conv layer at main path
    stage: integer, current stage label, used for generating layer names
    block: 'a','b'..., current block label, used for generating layer names
    strides: Strides for the second conv layer in the block.
    use_l2_regularizer: whether to use L2 regularizer on Conv layer.
    batch_norm_decay: Moment of batch norm layers.
    batch_norm_epsilon: Epsilon of batch borm layers.
  Returns:
    Output tensor for the block.
  """
  filters1, filters2, filters3 = filters
  if backend.image_data_format() == 'channels_last':
    bn_axis = 3
  else:
    bn_axis = 1
  conv_name_base = 'res' + str(stage) + block + '_branch'
  bn_name_base = 'bn' + str(stage) + block + '_branch'

  x = layers.Conv2D(
      filters1, (1, 1),
      use_bias=False,
      kernel_initializer='he_normal',
      kernel_regularizer=_gen_l2_regularizer(use_l2_regularizer),
      name=conv_name_base + '2a')(
          input_tensor)
  x = layers.BatchNormalization(
      axis=bn_axis,
      momentum=batch_norm_decay,
      epsilon=batch_norm_epsilon,
      name=bn_name_base + '2a')(
          x)
  x = layers.Activation('relu')(x)

  x = layers.Conv2D(
      filters2,
      kernel_size,
      strides=strides,
      padding='same',
      use_bias=False,
      kernel_initializer='he_normal',
      kernel_regularizer=_gen_l2_regularizer(use_l2_regularizer),
      name=conv_name_base + '2b')(
          x)
  x = layers.BatchNormalization(
      axis=bn_axis,
      momentum=batch_norm_decay,
      epsilon=batch_norm_epsilon,
      name=bn_name_base + '2b')(
          x)
  x = layers.Activation('relu')(x)

  x = layers.Conv2D(
      filters3, (1, 1),
      use_bias=False,
      kernel_initializer='he_normal',
      kernel_regularizer=_gen_l2_regularizer(use_l2_regularizer),
      name=conv_name_base + '2c')(
          x)
  x = layers.BatchNormalization(
      axis=bn_axis,
      momentum=batch_norm_decay,
      epsilon=batch_norm_epsilon,
      name=bn_name_base + '2c')(
          x)

  shortcut = layers.Conv2D(
      filters3, (1, 1),
      strides=strides,
      use_bias=False,
      kernel_initializer='he_normal',
      kernel_regularizer=_gen_l2_regularizer(use_l2_regularizer),
      name=conv_name_base + '1')(
          input_tensor)
  shortcut = layers.BatchNormalization(
      axis=bn_axis,
      momentum=batch_norm_decay,
      epsilon=batch_norm_epsilon,
      name=bn_name_base + '1')(
          shortcut)

  x = layers.add([x, shortcut])
  x = layers.Activation('relu')(x)
  return x

def identity_block(input_tensor,
                   kernel_size,
                   filters,
                   stage,
                   block,
                   use_l2_regularizer=True,
                   batch_norm_decay=0.9,
                   batch_norm_epsilon=1e-5):
  """The identity block is the block that has no conv layer at shortcut.
  Args:
    input_tensor: input tensor
    kernel_size: default 3, the kernel size of middle conv layer at main path
    filters: list of integers, the filters of 3 conv layer at main path
    stage: integer, current stage label, used for generating layer names
    block: 'a','b'..., current block label, used for generating layer names
    use_l2_regularizer: whether to use L2 regularizer on Conv layer.
    batch_norm_decay: Moment of batch norm layers.
    batch_norm_epsilon: Epsilon of batch borm layers.
  Returns:
    Output tensor for the block.
  """
  filters1, filters2, filters3 = filters
  if backend.image_data_format() == 'channels_last':
    bn_axis = 3
  else:
    bn_axis = 1
  conv_name_base = 'res' + str(stage) + block + '_branch'
  bn_name_base = 'bn' + str(stage) + block + '_branch'

  x = layers.Conv2D(
      filters1, (1, 1),
      use_bias=False,
      kernel_initializer='he_normal',
      kernel_regularizer=_gen_l2_regularizer(use_l2_regularizer),
      name=conv_name_base + '2a')(
          input_tensor)
  x = layers.BatchNormalization(
      axis=bn_axis,
      momentum=batch_norm_decay,
      epsilon=batch_norm_epsilon,
      name=bn_name_base + '2a')(
          x)
  x = layers.Activation('relu')(x)

  x = layers.Conv2D(
      filters2,
      kernel_size,
      padding='same',
      use_bias=False,
      kernel_initializer='he_normal',
      kernel_regularizer=_gen_l2_regularizer(use_l2_regularizer),
      name=conv_name_base + '2b')(
          x)
  x = layers.BatchNormalization(
      axis=bn_axis,
      momentum=batch_norm_decay,
      epsilon=batch_norm_epsilon,
      name=bn_name_base + '2b')(
          x)
  x = layers.Activation('relu')(x)

  x = layers.Conv2D(
      filters3, (1, 1),
      use_bias=False,
      kernel_initializer='he_normal',
      kernel_regularizer=_gen_l2_regularizer(use_l2_regularizer),
      name=conv_name_base + '2c')(
          x)
  x = layers.BatchNormalization(
      axis=bn_axis,
      momentum=batch_norm_decay,
      epsilon=batch_norm_epsilon,
      name=bn_name_base + '2c')(
          x)

  x = layers.add([x, input_tensor])
  x = layers.Activation('relu')(x)
  return x

def resnet50(num_classes,
             batch_size=None,
             use_l2_regularizer=True,
             rescale_inputs=False,
             batch_norm_decay=0.9,
             batch_norm_epsilon=1e-5):
  """Instantiates the ResNet50 architecture.
  Args:
    num_classes: `int` number of classes for image classification.
    batch_size: Size of the batches for each step.
    use_l2_regularizer: whether to use L2 regularizer on Conv/Dense layer.
    rescale_inputs: whether to rescale inputs from 0 to 1.
    batch_norm_decay: Moment of batch norm layers.
    batch_norm_epsilon: Epsilon of batch borm layers.
  Returns:
      A Keras model instance.
  """

  input_shape = (32, 32, 3)  # CIFAR100을 위한 input_shape 조정입니다. 
  img_input = layers.Input(shape=input_shape, batch_size=batch_size)
  if rescale_inputs:
    # Hub image modules expect inputs in the range [0, 1]. This rescales these
    # inputs to the range expected by the trained model.
    x = layers.Lambda(
        lambda x: x * 255.0 - backend.constant(
            imagenet_preprocessing.CHANNEL_MEANS,
            shape=[1, 1, 3],
            dtype=x.dtype),
        name='rescale')(
            img_input)
  else:
    x = img_input

  if backend.image_data_format() == 'channels_first':
    x = layers.Permute((3, 1, 2))(x)
    bn_axis = 1
  else:  # channels_last
    bn_axis = 3

  block_config = dict(
      use_l2_regularizer=use_l2_regularizer,
      batch_norm_decay=batch_norm_decay,
      batch_norm_epsilon=batch_norm_epsilon)
  x = layers.ZeroPadding2D(padding=(3, 3), name='conv1_pad')(x)
  x = layers.Conv2D(
      64, (7, 7),
      strides=(2, 2),
      padding='valid',
      use_bias=False,
      kernel_initializer='he_normal',
      kernel_regularizer=_gen_l2_regularizer(use_l2_regularizer),
      name='conv1')(
          x)
  x = layers.BatchNormalization(
      axis=bn_axis,
      momentum=batch_norm_decay,
      epsilon=batch_norm_epsilon,
      name='bn_conv1')(
          x)
  x = layers.Activation('relu')(x)
  x = layers.MaxPooling2D((3, 3), strides=(2, 2), padding='same')(x)

  x = conv_block(
      x, 3, [64, 64, 256], stage=2, block='a', strides=(1, 1), **block_config)
  x = identity_block(x, 3, [64, 64, 256], stage=2, block='b', **block_config)
  x = identity_block(x, 3, [64, 64, 256], stage=2, block='c', **block_config)

  x = conv_block(x, 3, [128, 128, 512], stage=3, block='a', **block_config)
  x = identity_block(x, 3, [128, 128, 512], stage=3, block='b', **block_config)
  x = identity_block(x, 3, [128, 128, 512], stage=3, block='c', **block_config)
  x = identity_block(x, 3, [128, 128, 512], stage=3, block='d', **block_config)

  x = conv_block(x, 3, [256, 256, 1024], stage=4, block='a', **block_config)
  x = identity_block(x, 3, [256, 256, 1024], stage=4, block='b', **block_config)
  x = identity_block(x, 3, [256, 256, 1024], stage=4, block='c', **block_config)
  x = identity_block(x, 3, [256, 256, 1024], stage=4, block='d', **block_config)
  x = identity_block(x, 3, [256, 256, 1024], stage=4, block='e', **block_config)
  x = identity_block(x, 3, [256, 256, 1024], stage=4, block='f', **block_config)

  x = conv_block(x, 3, [512, 512, 2048], stage=5, block='a', **block_config)
  x = identity_block(x, 3, [512, 512, 2048], stage=5, block='b', **block_config)
  x = identity_block(x, 3, [512, 512, 2048], stage=5, block='c', **block_config)

  x = layers.GlobalAveragePooling2D()(x)
  x = layers.Dense(
      num_classes,
      kernel_initializer=initializers.RandomNormal(stddev=0.01),
      kernel_regularizer=_gen_l2_regularizer(use_l2_regularizer),
      bias_regularizer=_gen_l2_regularizer(use_l2_regularizer),
      name='fc1000')(
          x)

  # A softmax that is followed by the model loss must be done cannot be done
  # in float16 due to numeric issues. So we pass dtype=float32.
  x = layers.Activation('softmax', dtype='float32')(x)

  # Create model.
  return models.Model(img_input, x, name='resnet50')

model = resnet50(num_classes=100)

model.summary()

Model: "resnet50"
__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
input_5 (InputLayer)            [(None, 32, 32, 3)]  0                                            
__________________________________________________________________________________________________
conv1_pad (ZeroPadding2D)       (None, 38, 38, 3)    0           input_5[0][0]                    
__________________________________________________________________________________________________
conv1 (Conv2D)                  (None, 16, 16, 64)   9408        conv1_pad[0][0]                  
__________________________________________________________________________________________________
bn_conv1 (BatchNormalization)   (None, 16, 16, 64)   256         conv1[0][0]                      
__________________________________________________________________________________________________
activation (Activation)         (None, 16, 16, 64)   0           bn_conv1[0][0]                   
__________________________________________________________________________________________________
max_pooling2d_6 (MaxPooling2D)  (None, 8, 8, 64)     0           activation[0][0]                 
__________________________________________________________________________________________________
res2a_branch2a (Conv2D)         (None, 8, 8, 64)     4096        max_pooling2d_6[0][0]            
__________________________________________________________________________________________________
bn2a_branch2a (BatchNormalizati (None, 8, 8, 64)     256         res2a_branch2a[0][0]             
__________________________________________________________________________________________________
activation_1 (Activation)       (None, 8, 8, 64)     0           bn2a_branch2a[0][0]              
__________________________________________________________________________________________________
res2a_branch2b (Conv2D)         (None, 8, 8, 64)     36864       activation_1[0][0]               
__________________________________________________________________________________________________
bn2a_branch2b (BatchNormalizati (None, 8, 8, 64)     256         res2a_branch2b[0][0]             
__________________________________________________________________________________________________
activation_2 (Activation)       (None, 8, 8, 64)     0           bn2a_branch2b[0][0]              
__________________________________________________________________________________________________
res2a_branch2c (Conv2D)         (None, 8, 8, 256)    16384       activation_2[0][0]               
__________________________________________________________________________________________________
res2a_branch1 (Conv2D)          (None, 8, 8, 256)    16384       max_pooling2d_6[0][0]            
__________________________________________________________________________________________________
bn2a_branch2c (BatchNormalizati (None, 8, 8, 256)    1024        res2a_branch2c[0][0]             
__________________________________________________________________________________________________
bn2a_branch1 (BatchNormalizatio (None, 8, 8, 256)    1024        res2a_branch1[0][0]              
__________________________________________________________________________________________________
add (Add)                       (None, 8, 8, 256)    0           bn2a_branch2c[0][0]              
                                                                 bn2a_branch1[0][0]               
__________________________________________________________________________________________________
activation_3 (Activation)       (None, 8, 8, 256)    0           add[0][0]                        
__________________________________________________________________________________________________
res2b_branch2a (Conv2D)         (None, 8, 8, 64)     16384       activation_3[0][0]               
__________________________________________________________________________________________________
bn2b_branch2a (BatchNormalizati (None, 8, 8, 64)     256         res2b_branch2a[0][0]             
__________________________________________________________________________________________________
activation_4 (Activation)       (None, 8, 8, 64)     0           bn2b_branch2a[0][0]              
__________________________________________________________________________________________________
res2b_branch2b (Conv2D)         (None, 8, 8, 64)     36864       activation_4[0][0]               
__________________________________________________________________________________________________
bn2b_branch2b (BatchNormalizati (None, 8, 8, 64)     256         res2b_branch2b[0][0]             
__________________________________________________________________________________________________
activation_5 (Activation)       (None, 8, 8, 64)     0           bn2b_branch2b[0][0]              
__________________________________________________________________________________________________
res2b_branch2c (Conv2D)         (None, 8, 8, 256)    16384       activation_5[0][0]               
__________________________________________________________________________________________________
bn2b_branch2c (BatchNormalizati (None, 8, 8, 256)    1024        res2b_branch2c[0][0]             
__________________________________________________________________________________________________
add_1 (Add)                     (None, 8, 8, 256)    0           bn2b_branch2c[0][0]              
                                                                 activation_3[0][0]               
__________________________________________________________________________________________________
activation_6 (Activation)       (None, 8, 8, 256)    0           add_1[0][0]                      
__________________________________________________________________________________________________
res2c_branch2a (Conv2D)         (None, 8, 8, 64)     16384       activation_6[0][0]               
__________________________________________________________________________________________________
bn2c_branch2a (BatchNormalizati (None, 8, 8, 64)     256         res2c_branch2a[0][0]             
__________________________________________________________________________________________________
activation_7 (Activation)       (None, 8, 8, 64)     0           bn2c_branch2a[0][0]              
__________________________________________________________________________________________________
res2c_branch2b (Conv2D)         (None, 8, 8, 64)     36864       activation_7[0][0]               
__________________________________________________________________________________________________
bn2c_branch2b (BatchNormalizati (None, 8, 8, 64)     256         res2c_branch2b[0][0]             
__________________________________________________________________________________________________
activation_8 (Activation)       (None, 8, 8, 64)     0           bn2c_branch2b[0][0]              
__________________________________________________________________________________________________
res2c_branch2c (Conv2D)         (None, 8, 8, 256)    16384       activation_8[0][0]               
__________________________________________________________________________________________________
bn2c_branch2c (BatchNormalizati (None, 8, 8, 256)    1024        res2c_branch2c[0][0]             
__________________________________________________________________________________________________
add_2 (Add)                     (None, 8, 8, 256)    0           bn2c_branch2c[0][0]              
                                                                 activation_6[0][0]               
__________________________________________________________________________________________________
activation_9 (Activation)       (None, 8, 8, 256)    0           add_2[0][0]                      
__________________________________________________________________________________________________
res3a_branch2a (Conv2D)         (None, 8, 8, 128)    32768       activation_9[0][0]               
__________________________________________________________________________________________________
bn3a_branch2a (BatchNormalizati (None, 8, 8, 128)    512         res3a_branch2a[0][0]             
__________________________________________________________________________________________________
activation_10 (Activation)      (None, 8, 8, 128)    0           bn3a_branch2a[0][0]              
__________________________________________________________________________________________________
res3a_branch2b (Conv2D)         (None, 4, 4, 128)    147456      activation_10[0][0]              
__________________________________________________________________________________________________
bn3a_branch2b (BatchNormalizati (None, 4, 4, 128)    512         res3a_branch2b[0][0]             
__________________________________________________________________________________________________
activation_11 (Activation)      (None, 4, 4, 128)    0           bn3a_branch2b[0][0]              
__________________________________________________________________________________________________
res3a_branch2c (Conv2D)         (None, 4, 4, 512)    65536       activation_11[0][0]              
__________________________________________________________________________________________________
res3a_branch1 (Conv2D)          (None, 4, 4, 512)    131072      activation_9[0][0]               
__________________________________________________________________________________________________
bn3a_branch2c (BatchNormalizati (None, 4, 4, 512)    2048        res3a_branch2c[0][0]             
__________________________________________________________________________________________________
bn3a_branch1 (BatchNormalizatio (None, 4, 4, 512)    2048        res3a_branch1[0][0]              
__________________________________________________________________________________________________
add_3 (Add)                     (None, 4, 4, 512)    0           bn3a_branch2c[0][0]              
                                                                 bn3a_branch1[0][0]               
__________________________________________________________________________________________________
activation_12 (Activation)      (None, 4, 4, 512)    0           add_3[0][0]                      
__________________________________________________________________________________________________
res3b_branch2a (Conv2D)         (None, 4, 4, 128)    65536       activation_12[0][0]              
__________________________________________________________________________________________________
bn3b_branch2a (BatchNormalizati (None, 4, 4, 128)    512         res3b_branch2a[0][0]             
__________________________________________________________________________________________________
activation_13 (Activation)      (None, 4, 4, 128)    0           bn3b_branch2a[0][0]              
__________________________________________________________________________________________________
res3b_branch2b (Conv2D)         (None, 4, 4, 128)    147456      activation_13[0][0]              
__________________________________________________________________________________________________
bn3b_branch2b (BatchNormalizati (None, 4, 4, 128)    512         res3b_branch2b[0][0]             
__________________________________________________________________________________________________
activation_14 (Activation)      (None, 4, 4, 128)    0           bn3b_branch2b[0][0]              
__________________________________________________________________________________________________
res3b_branch2c (Conv2D)         (None, 4, 4, 512)    65536       activation_14[0][0]              
__________________________________________________________________________________________________
bn3b_branch2c (BatchNormalizati (None, 4, 4, 512)    2048        res3b_branch2c[0][0]             
__________________________________________________________________________________________________
add_4 (Add)                     (None, 4, 4, 512)    0           bn3b_branch2c[0][0]              
                                                                 activation_12[0][0]              
__________________________________________________________________________________________________
activation_15 (Activation)      (None, 4, 4, 512)    0           add_4[0][0]                      
__________________________________________________________________________________________________
res3c_branch2a (Conv2D)         (None, 4, 4, 128)    65536       activation_15[0][0]              
__________________________________________________________________________________________________
bn3c_branch2a (BatchNormalizati (None, 4, 4, 128)    512         res3c_branch2a[0][0]             
__________________________________________________________________________________________________
activation_16 (Activation)      (None, 4, 4, 128)    0           bn3c_branch2a[0][0]              
__________________________________________________________________________________________________
res3c_branch2b (Conv2D)         (None, 4, 4, 128)    147456      activation_16[0][0]              
__________________________________________________________________________________________________
bn3c_branch2b (BatchNormalizati (None, 4, 4, 128)    512         res3c_branch2b[0][0]             
__________________________________________________________________________________________________
activation_17 (Activation)      (None, 4, 4, 128)    0           bn3c_branch2b[0][0]              
__________________________________________________________________________________________________
res3c_branch2c (Conv2D)         (None, 4, 4, 512)    65536       activation_17[0][0]              
__________________________________________________________________________________________________
bn3c_branch2c (BatchNormalizati (None, 4, 4, 512)    2048        res3c_branch2c[0][0]             
__________________________________________________________________________________________________
add_5 (Add)                     (None, 4, 4, 512)    0           bn3c_branch2c[0][0]              
                                                                 activation_15[0][0]              
__________________________________________________________________________________________________
activation_18 (Activation)      (None, 4, 4, 512)    0           add_5[0][0]                      
__________________________________________________________________________________________________
res3d_branch2a (Conv2D)         (None, 4, 4, 128)    65536       activation_18[0][0]              
__________________________________________________________________________________________________
bn3d_branch2a (BatchNormalizati (None, 4, 4, 128)    512         res3d_branch2a[0][0]             
__________________________________________________________________________________________________
activation_19 (Activation)      (None, 4, 4, 128)    0           bn3d_branch2a[0][0]              
__________________________________________________________________________________________________
res3d_branch2b (Conv2D)         (None, 4, 4, 128)    147456      activation_19[0][0]              
__________________________________________________________________________________________________
bn3d_branch2b (BatchNormalizati (None, 4, 4, 128)    512         res3d_branch2b[0][0]             
__________________________________________________________________________________________________
activation_20 (Activation)      (None, 4, 4, 128)    0           bn3d_branch2b[0][0]              
__________________________________________________________________________________________________
res3d_branch2c (Conv2D)         (None, 4, 4, 512)    65536       activation_20[0][0]              
__________________________________________________________________________________________________
bn3d_branch2c (BatchNormalizati (None, 4, 4, 512)    2048        res3d_branch2c[0][0]             
__________________________________________________________________________________________________
add_6 (Add)                     (None, 4, 4, 512)    0           bn3d_branch2c[0][0]              
                                                                 activation_18[0][0]              
__________________________________________________________________________________________________
activation_21 (Activation)      (None, 4, 4, 512)    0           add_6[0][0]                      
__________________________________________________________________________________________________
res4a_branch2a (Conv2D)         (None, 4, 4, 256)    131072      activation_21[0][0]              
__________________________________________________________________________________________________
bn4a_branch2a (BatchNormalizati (None, 4, 4, 256)    1024        res4a_branch2a[0][0]             
__________________________________________________________________________________________________
activation_22 (Activation)      (None, 4, 4, 256)    0           bn4a_branch2a[0][0]              
__________________________________________________________________________________________________
res4a_branch2b (Conv2D)         (None, 2, 2, 256)    589824      activation_22[0][0]              
__________________________________________________________________________________________________
bn4a_branch2b (BatchNormalizati (None, 2, 2, 256)    1024        res4a_branch2b[0][0]             
__________________________________________________________________________________________________
activation_23 (Activation)      (None, 2, 2, 256)    0           bn4a_branch2b[0][0]              
__________________________________________________________________________________________________
res4a_branch2c (Conv2D)         (None, 2, 2, 1024)   262144      activation_23[0][0]              
__________________________________________________________________________________________________
res4a_branch1 (Conv2D)          (None, 2, 2, 1024)   524288      activation_21[0][0]              
__________________________________________________________________________________________________
bn4a_branch2c (BatchNormalizati (None, 2, 2, 1024)   4096        res4a_branch2c[0][0]             
__________________________________________________________________________________________________
bn4a_branch1 (BatchNormalizatio (None, 2, 2, 1024)   4096        res4a_branch1[0][0]              
__________________________________________________________________________________________________
add_7 (Add)                     (None, 2, 2, 1024)   0           bn4a_branch2c[0][0]              
                                                                 bn4a_branch1[0][0]               
__________________________________________________________________________________________________
activation_24 (Activation)      (None, 2, 2, 1024)   0           add_7[0][0]                      
__________________________________________________________________________________________________
res4b_branch2a (Conv2D)         (None, 2, 2, 256)    262144      activation_24[0][0]              
__________________________________________________________________________________________________
bn4b_branch2a (BatchNormalizati (None, 2, 2, 256)    1024        res4b_branch2a[0][0]             
__________________________________________________________________________________________________
activation_25 (Activation)      (None, 2, 2, 256)    0           bn4b_branch2a[0][0]              
__________________________________________________________________________________________________
res4b_branch2b (Conv2D)         (None, 2, 2, 256)    589824      activation_25[0][0]              
__________________________________________________________________________________________________
bn4b_branch2b (BatchNormalizati (None, 2, 2, 256)    1024        res4b_branch2b[0][0]             
__________________________________________________________________________________________________
activation_26 (Activation)      (None, 2, 2, 256)    0           bn4b_branch2b[0][0]              
__________________________________________________________________________________________________
res4b_branch2c (Conv2D)         (None, 2, 2, 1024)   262144      activation_26[0][0]              
__________________________________________________________________________________________________
bn4b_branch2c (BatchNormalizati (None, 2, 2, 1024)   4096        res4b_branch2c[0][0]             
__________________________________________________________________________________________________
add_8 (Add)                     (None, 2, 2, 1024)   0           bn4b_branch2c[0][0]              
                                                                 activation_24[0][0]              
__________________________________________________________________________________________________
activation_27 (Activation)      (None, 2, 2, 1024)   0           add_8[0][0]                      
__________________________________________________________________________________________________
res4c_branch2a (Conv2D)         (None, 2, 2, 256)    262144      activation_27[0][0]              
__________________________________________________________________________________________________
bn4c_branch2a (BatchNormalizati (None, 2, 2, 256)    1024        res4c_branch2a[0][0]             
__________________________________________________________________________________________________
activation_28 (Activation)      (None, 2, 2, 256)    0           bn4c_branch2a[0][0]              
__________________________________________________________________________________________________
res4c_branch2b (Conv2D)         (None, 2, 2, 256)    589824      activation_28[0][0]              
__________________________________________________________________________________________________
bn4c_branch2b (BatchNormalizati (None, 2, 2, 256)    1024        res4c_branch2b[0][0]             
__________________________________________________________________________________________________
activation_29 (Activation)      (None, 2, 2, 256)    0           bn4c_branch2b[0][0]              
__________________________________________________________________________________________________
res4c_branch2c (Conv2D)         (None, 2, 2, 1024)   262144      activation_29[0][0]              
__________________________________________________________________________________________________
bn4c_branch2c (BatchNormalizati (None, 2, 2, 1024)   4096        res4c_branch2c[0][0]             
__________________________________________________________________________________________________
add_9 (Add)                     (None, 2, 2, 1024)   0           bn4c_branch2c[0][0]              
                                                                 activation_27[0][0]              
__________________________________________________________________________________________________
activation_30 (Activation)      (None, 2, 2, 1024)   0           add_9[0][0]                      
__________________________________________________________________________________________________
res4d_branch2a (Conv2D)         (None, 2, 2, 256)    262144      activation_30[0][0]              
__________________________________________________________________________________________________
bn4d_branch2a (BatchNormalizati (None, 2, 2, 256)    1024        res4d_branch2a[0][0]             
__________________________________________________________________________________________________
activation_31 (Activation)      (None, 2, 2, 256)    0           bn4d_branch2a[0][0]              
__________________________________________________________________________________________________
res4d_branch2b (Conv2D)         (None, 2, 2, 256)    589824      activation_31[0][0]              
__________________________________________________________________________________________________
bn4d_branch2b (BatchNormalizati (None, 2, 2, 256)    1024        res4d_branch2b[0][0]             
__________________________________________________________________________________________________
activation_32 (Activation)      (None, 2, 2, 256)    0           bn4d_branch2b[0][0]              
__________________________________________________________________________________________________
res4d_branch2c (Conv2D)         (None, 2, 2, 1024)   262144      activation_32[0][0]              
__________________________________________________________________________________________________
bn4d_branch2c (BatchNormalizati (None, 2, 2, 1024)   4096        res4d_branch2c[0][0]             
__________________________________________________________________________________________________
add_10 (Add)                    (None, 2, 2, 1024)   0           bn4d_branch2c[0][0]              
                                                                 activation_30[0][0]              
__________________________________________________________________________________________________
activation_33 (Activation)      (None, 2, 2, 1024)   0           add_10[0][0]                     
__________________________________________________________________________________________________
res4e_branch2a (Conv2D)         (None, 2, 2, 256)    262144      activation_33[0][0]              
__________________________________________________________________________________________________
bn4e_branch2a (BatchNormalizati (None, 2, 2, 256)    1024        res4e_branch2a[0][0]             
__________________________________________________________________________________________________
activation_34 (Activation)      (None, 2, 2, 256)    0           bn4e_branch2a[0][0]              
__________________________________________________________________________________________________
res4e_branch2b (Conv2D)         (None, 2, 2, 256)    589824      activation_34[0][0]              
__________________________________________________________________________________________________
bn4e_branch2b (BatchNormalizati (None, 2, 2, 256)    1024        res4e_branch2b[0][0]             
__________________________________________________________________________________________________
activation_35 (Activation)      (None, 2, 2, 256)    0           bn4e_branch2b[0][0]              
__________________________________________________________________________________________________
res4e_branch2c (Conv2D)         (None, 2, 2, 1024)   262144      activation_35[0][0]              
__________________________________________________________________________________________________
bn4e_branch2c (BatchNormalizati (None, 2, 2, 1024)   4096        res4e_branch2c[0][0]             
__________________________________________________________________________________________________
add_11 (Add)                    (None, 2, 2, 1024)   0           bn4e_branch2c[0][0]              
                                                                 activation_33[0][0]              
__________________________________________________________________________________________________
activation_36 (Activation)      (None, 2, 2, 1024)   0           add_11[0][0]                     
__________________________________________________________________________________________________
res4f_branch2a (Conv2D)         (None, 2, 2, 256)    262144      activation_36[0][0]              
__________________________________________________________________________________________________
bn4f_branch2a (BatchNormalizati (None, 2, 2, 256)    1024        res4f_branch2a[0][0]             
__________________________________________________________________________________________________
activation_37 (Activation)      (None, 2, 2, 256)    0           bn4f_branch2a[0][0]              
__________________________________________________________________________________________________
res4f_branch2b (Conv2D)         (None, 2, 2, 256)    589824      activation_37[0][0]              
__________________________________________________________________________________________________
bn4f_branch2b (BatchNormalizati (None, 2, 2, 256)    1024        res4f_branch2b[0][0]             
__________________________________________________________________________________________________
activation_38 (Activation)      (None, 2, 2, 256)    0           bn4f_branch2b[0][0]              
__________________________________________________________________________________________________
res4f_branch2c (Conv2D)         (None, 2, 2, 1024)   262144      activation_38[0][0]              
__________________________________________________________________________________________________
bn4f_branch2c (BatchNormalizati (None, 2, 2, 1024)   4096        res4f_branch2c[0][0]             
__________________________________________________________________________________________________
add_12 (Add)                    (None, 2, 2, 1024)   0           bn4f_branch2c[0][0]              
                                                                 activation_36[0][0]              
__________________________________________________________________________________________________
activation_39 (Activation)      (None, 2, 2, 1024)   0           add_12[0][0]                     
__________________________________________________________________________________________________
res5a_branch2a (Conv2D)         (None, 2, 2, 512)    524288      activation_39[0][0]              
__________________________________________________________________________________________________
bn5a_branch2a (BatchNormalizati (None, 2, 2, 512)    2048        res5a_branch2a[0][0]             
__________________________________________________________________________________________________
activation_40 (Activation)      (None, 2, 2, 512)    0           bn5a_branch2a[0][0]              
__________________________________________________________________________________________________
res5a_branch2b (Conv2D)         (None, 1, 1, 512)    2359296     activation_40[0][0]              
__________________________________________________________________________________________________
bn5a_branch2b (BatchNormalizati (None, 1, 1, 512)    2048        res5a_branch2b[0][0]             
__________________________________________________________________________________________________
activation_41 (Activation)      (None, 1, 1, 512)    0           bn5a_branch2b[0][0]              
__________________________________________________________________________________________________
res5a_branch2c (Conv2D)         (None, 1, 1, 2048)   1048576     activation_41[0][0]              
__________________________________________________________________________________________________
res5a_branch1 (Conv2D)          (None, 1, 1, 2048)   2097152     activation_39[0][0]              
__________________________________________________________________________________________________
bn5a_branch2c (BatchNormalizati (None, 1, 1, 2048)   8192        res5a_branch2c[0][0]             
__________________________________________________________________________________________________
bn5a_branch1 (BatchNormalizatio (None, 1, 1, 2048)   8192        res5a_branch1[0][0]              
__________________________________________________________________________________________________
add_13 (Add)                    (None, 1, 1, 2048)   0           bn5a_branch2c[0][0]              
                                                                 bn5a_branch1[0][0]               
__________________________________________________________________________________________________
activation_42 (Activation)      (None, 1, 1, 2048)   0           add_13[0][0]                     
__________________________________________________________________________________________________
res5b_branch2a (Conv2D)         (None, 1, 1, 512)    1048576     activation_42[0][0]              
__________________________________________________________________________________________________
bn5b_branch2a (BatchNormalizati (None, 1, 1, 512)    2048        res5b_branch2a[0][0]             
__________________________________________________________________________________________________
activation_43 (Activation)      (None, 1, 1, 512)    0           bn5b_branch2a[0][0]              
__________________________________________________________________________________________________
res5b_branch2b (Conv2D)         (None, 1, 1, 512)    2359296     activation_43[0][0]              
__________________________________________________________________________________________________
bn5b_branch2b (BatchNormalizati (None, 1, 1, 512)    2048        res5b_branch2b[0][0]             
__________________________________________________________________________________________________
activation_44 (Activation)      (None, 1, 1, 512)    0           bn5b_branch2b[0][0]              
__________________________________________________________________________________________________
res5b_branch2c (Conv2D)         (None, 1, 1, 2048)   1048576     activation_44[0][0]              
__________________________________________________________________________________________________
bn5b_branch2c (BatchNormalizati (None, 1, 1, 2048)   8192        res5b_branch2c[0][0]             
__________________________________________________________________________________________________
add_14 (Add)                    (None, 1, 1, 2048)   0           bn5b_branch2c[0][0]              
                                                                 activation_42[0][0]              
__________________________________________________________________________________________________
activation_45 (Activation)      (None, 1, 1, 2048)   0           add_14[0][0]                     
__________________________________________________________________________________________________
res5c_branch2a (Conv2D)         (None, 1, 1, 512)    1048576     activation_45[0][0]              
__________________________________________________________________________________________________
bn5c_branch2a (BatchNormalizati (None, 1, 1, 512)    2048        res5c_branch2a[0][0]             
__________________________________________________________________________________________________
activation_46 (Activation)      (None, 1, 1, 512)    0           bn5c_branch2a[0][0]              
__________________________________________________________________________________________________
res5c_branch2b (Conv2D)         (None, 1, 1, 512)    2359296     activation_46[0][0]              
__________________________________________________________________________________________________
bn5c_branch2b (BatchNormalizati (None, 1, 1, 512)    2048        res5c_branch2b[0][0]             
__________________________________________________________________________________________________
activation_47 (Activation)      (None, 1, 1, 512)    0           bn5c_branch2b[0][0]              
__________________________________________________________________________________________________
res5c_branch2c (Conv2D)         (None, 1, 1, 2048)   1048576     activation_47[0][0]              
__________________________________________________________________________________________________
bn5c_branch2c (BatchNormalizati (None, 1, 1, 2048)   8192        res5c_branch2c[0][0]             
__________________________________________________________________________________________________
add_15 (Add)                    (None, 1, 1, 2048)   0           bn5c_branch2c[0][0]              
                                                                 activation_45[0][0]              
__________________________________________________________________________________________________
activation_48 (Activation)      (None, 1, 1, 2048)   0           add_15[0][0]                     
__________________________________________________________________________________________________
global_average_pooling2d (Globa (None, 2048)         0           activation_48[0][0]              
__________________________________________________________________________________________________
fc1000 (Dense)                  (None, 100)          204900      global_average_pooling2d[0][0]   
__________________________________________________________________________________________________
activation_49 (Activation)      (None, 100)          0           fc1000[0][0]                     
==================================================================================================
Total params: 23,766,052
Trainable params: 23,712,932
Non-trainable params: 53,120
__________________________________________________________________________________________________

# 모델 학습
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=1)