Data Science/AI

인공지능 #9 | 텍스트를 위한 인공 신경망(NLP, IMDB, LSTM, GRU)

토마토. 2023. 3. 14. 16:23

Chapter9

9. NLP - 텍스트를 위한 인공 신경망¶

9-1. 순차 데이터와 순환 신경망¶

순차 데이터 sequential data: 텍스트, 시계열 데이터처럼 순서에 의미가 있는 데이터
순차 데이터를 다루기 위해서는 이전에 입력한 데이터를 기억하는 기능이 필요하다
FFDD Feedforward neural network: 피드포워드 신경망, 데이터 흐름이 앞으로만 전달되는 신경망
RNN 순환 신경망 recurrent neural network: 순차 데이터에 잘 맞는 인공 신경망. 순환층을 1개 이상 사용한 신경망
셀: RNN에서 순환층을 셀이라고 부른다
은닉 상태: 셀의 출력 -> 은닉 상태가 다음 층으로 전달된다 + 셀이 타음 스텝 데이터를 처리할 때 재사용된다

9-2. 순환 신경망으로 IMDB 리뷰 분류하기¶

NLP Natural Language Processing
음성 인식, 기계 번역, 감성 분석 등
훈련 데이터를 "corpus"라고 부름
단어에 대응하는 고유한 정수를 부여한다, 이를 "token"이라고 함

데이터 불러오기

In [1]:

from tensorflow.keras.datasets import imdb
(train_input, train_target), (test_input, test_target) = imdb.load_data(num_words=500)

Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz
17464789/17464789 [==============================] - 20s 1us/step

In [2]:

print(train_input.shape, test_input.shape)

(25000,) (25000,)

In [3]:

print(train_input[0])

[1, 14, 22, 16, 43, 2, 2, 2, 2, 65, 458, 2, 66, 2, 4, 173, 36, 256, 5, 25, 100, 43, 2, 112, 50, 2, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 2, 2, 17, 2, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2, 19, 14, 22, 4, 2, 2, 469, 4, 22, 71, 87, 12, 16, 43, 2, 38, 76, 15, 13, 2, 4, 22, 17, 2, 17, 12, 16, 2, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2, 2, 16, 480, 66, 2, 33, 4, 130, 12, 16, 38, 2, 5, 25, 124, 51, 36, 135, 48, 25, 2, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 2, 15, 256, 4, 2, 7, 2, 5, 2, 36, 71, 43, 2, 476, 26, 400, 317, 46, 7, 4, 2, 2, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2, 56, 26, 141, 6, 194, 2, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 2, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 2, 88, 12, 16, 283, 5, 16, 2, 113, 103, 32, 15, 16, 2, 19, 178, 32]

In [4]:

print(train_target[:20]) # 0은 부정, 1은 긍정

[1 0 0 1 0 0 1 0 1 0 1 0 0 0 0 0 1 1 0 1]

훈련 셋과 검증 셋 분리하기

In [5]:

from sklearn.model_selection import train_test_split
train_input, val_input, train_target, val_target = train_test_split(
    train_input, train_target, test_size=0.2, random_state=42
)

In [6]:

import numpy as np
import matplotlib.pyplot as plt

lengths = np.array([len(x) for x in train_input])
print(np.mean(lengths), np.median(lengths))

plt.hist(lengths)
plt.xlabel("length")
plt.ylabel("frequency")
plt.show()

239.00925 178.0

100개 단어 이하인 리뷰만 사용하기

100보다 짧은 리뷰는 토큰 0으로 패딩을 채운다

100보다 긴 리뷰는 앞부분부터 자른다

In [9]:

from tensorflow.keras.preprocessing.sequence import pad_sequences
train_seq = pad_sequences(train_input, maxlen=100)
val_seq = pad_sequences(val_input, maxlen=100)
print(train_seq.shape)

(20000, 100)

순환 신경망 만들기¶

가장 간단한 순환층 클래스: SimpleRNN

In [10]:

from tensorflow import keras
model = keras.Sequential()
model.add(keras.layers.SimpleRNN(8, # 뉴련 개수
                                 input_shape=(100, 500), # input 차원(샘플 길이, 원핫인코딩 길이)
                                 activation="tanh")) # 활성화 함수 tanh
model.add(keras.layers.Dense(1, activation="sigmoid"))

원핫 인코딩

In [14]:

train_oh = keras.utils.to_categorical(train_seq)
val_oh = keras.utils.to_categorical(val_seq)
print(train_oh.shape)

(20000, 100, 500)

In [15]:

model.summary()

Model: "sequential"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 simple_rnn (SimpleRNN)      (None, 8)                 4072      
                                                                 
 dense (Dense)               (None, 1)                 9         
                                                                 
=================================================================
Total params: 4,081
Trainable params: 4,081
Non-trainable params: 0
_________________________________________________________________

학습시키기

In [16]:

rmsprop = keras.optimizers.RMSprop(learning_rate=1e-4)
model.compile(
    optimizer=rmsprop, 
    loss="binary_crossentropy",
    metrics=["accuracy"]
)
checkpoint_cb = keras.callbacks.ModelCheckpoint("best-simplernn-model.h5",
                                                save_best_only=True)
early_stopping_cb = keras.callbacks.EarlyStopping(patience=3, 
                                                  restore_best_weights=True)
history = model.fit(train_oh, train_target, 
                    epochs=100, 
                    batch_size=64, 
                    validation_data=(val_oh, val_target),
                    callbacks=[checkpoint_cb, early_stopping_cb])

Epoch 1/100
313/313 [==============================] - 10s 28ms/step - loss: 0.7023 - accuracy: 0.4967 - val_loss: 0.6996 - val_accuracy: 0.5046
Epoch 2/100
313/313 [==============================] - 9s 28ms/step - loss: 0.6966 - accuracy: 0.5051 - val_loss: 0.6960 - val_accuracy: 0.5090
Epoch 3/100
313/313 [==============================] - 9s 29ms/step - loss: 0.6937 - accuracy: 0.5169 - val_loss: 0.6938 - val_accuracy: 0.5144
Epoch 4/100
313/313 [==============================] - 11s 34ms/step - loss: 0.6891 - accuracy: 0.5375 - val_loss: 0.6865 - val_accuracy: 0.5480
Epoch 5/100
313/313 [==============================] - 11s 34ms/step - loss: 0.6784 - accuracy: 0.5874 - val_loss: 0.6769 - val_accuracy: 0.5878
Epoch 6/100
313/313 [==============================] - 15s 47ms/step - loss: 0.6678 - accuracy: 0.6219 - val_loss: 0.6677 - val_accuracy: 0.6100
Epoch 7/100
313/313 [==============================] - 13s 43ms/step - loss: 0.6566 - accuracy: 0.6517 - val_loss: 0.6550 - val_accuracy: 0.6584
Epoch 8/100
313/313 [==============================] - 11s 35ms/step - loss: 0.6418 - accuracy: 0.6845 - val_loss: 0.6432 - val_accuracy: 0.6702
Epoch 9/100
313/313 [==============================] - 9s 27ms/step - loss: 0.6294 - accuracy: 0.7006 - val_loss: 0.6346 - val_accuracy: 0.6796
Epoch 10/100
313/313 [==============================] - 11s 35ms/step - loss: 0.6168 - accuracy: 0.7168 - val_loss: 0.6213 - val_accuracy: 0.6970
Epoch 11/100
313/313 [==============================] - 9s 30ms/step - loss: 0.6047 - accuracy: 0.7281 - val_loss: 0.6077 - val_accuracy: 0.7194
Epoch 12/100
313/313 [==============================] - 10s 33ms/step - loss: 0.5919 - accuracy: 0.7360 - val_loss: 0.5947 - val_accuracy: 0.7280
Epoch 13/100
313/313 [==============================] - 12s 39ms/step - loss: 0.5786 - accuracy: 0.7444 - val_loss: 0.5837 - val_accuracy: 0.7296
Epoch 14/100
313/313 [==============================] - 11s 34ms/step - loss: 0.5657 - accuracy: 0.7488 - val_loss: 0.5728 - val_accuracy: 0.7294
Epoch 15/100
313/313 [==============================] - 10s 33ms/step - loss: 0.5530 - accuracy: 0.7520 - val_loss: 0.5595 - val_accuracy: 0.7348
Epoch 16/100
313/313 [==============================] - 17s 54ms/step - loss: 0.5375 - accuracy: 0.7578 - val_loss: 0.5440 - val_accuracy: 0.7436
Epoch 17/100
313/313 [==============================] - 14s 43ms/step - loss: 0.5230 - accuracy: 0.7667 - val_loss: 0.5305 - val_accuracy: 0.7510
Epoch 18/100
313/313 [==============================] - 10s 31ms/step - loss: 0.5093 - accuracy: 0.7721 - val_loss: 0.5186 - val_accuracy: 0.7554
Epoch 19/100
313/313 [==============================] - 10s 31ms/step - loss: 0.4986 - accuracy: 0.7771 - val_loss: 0.5078 - val_accuracy: 0.7660
Epoch 20/100
313/313 [==============================] - 9s 30ms/step - loss: 0.4888 - accuracy: 0.7837 - val_loss: 0.4965 - val_accuracy: 0.7738
Epoch 21/100
313/313 [==============================] - 10s 30ms/step - loss: 0.4796 - accuracy: 0.7876 - val_loss: 0.4892 - val_accuracy: 0.7764
Epoch 22/100
313/313 [==============================] - 13s 40ms/step - loss: 0.4720 - accuracy: 0.7907 - val_loss: 0.4860 - val_accuracy: 0.7814
Epoch 23/100
313/313 [==============================] - 12s 39ms/step - loss: 0.4659 - accuracy: 0.7930 - val_loss: 0.4755 - val_accuracy: 0.7828
Epoch 24/100
313/313 [==============================] - 12s 38ms/step - loss: 0.4590 - accuracy: 0.7973 - val_loss: 0.4706 - val_accuracy: 0.7858
Epoch 25/100
313/313 [==============================] - 12s 37ms/step - loss: 0.4544 - accuracy: 0.7974 - val_loss: 0.4683 - val_accuracy: 0.7868
Epoch 26/100
313/313 [==============================] - 11s 35ms/step - loss: 0.4496 - accuracy: 0.8005 - val_loss: 0.4674 - val_accuracy: 0.7852
Epoch 27/100
313/313 [==============================] - 11s 34ms/step - loss: 0.4456 - accuracy: 0.8016 - val_loss: 0.4625 - val_accuracy: 0.7900
Epoch 28/100
313/313 [==============================] - 10s 32ms/step - loss: 0.4412 - accuracy: 0.8025 - val_loss: 0.4651 - val_accuracy: 0.7848
Epoch 29/100
313/313 [==============================] - 10s 32ms/step - loss: 0.4383 - accuracy: 0.8059 - val_loss: 0.4627 - val_accuracy: 0.7856
Epoch 30/100
313/313 [==============================] - 10s 32ms/step - loss: 0.4348 - accuracy: 0.8069 - val_loss: 0.4571 - val_accuracy: 0.7882
Epoch 31/100
313/313 [==============================] - 10s 32ms/step - loss: 0.4325 - accuracy: 0.8067 - val_loss: 0.4549 - val_accuracy: 0.7924
Epoch 32/100
313/313 [==============================] - 10s 33ms/step - loss: 0.4298 - accuracy: 0.8088 - val_loss: 0.4533 - val_accuracy: 0.7946
Epoch 33/100
313/313 [==============================] - 10s 32ms/step - loss: 0.4279 - accuracy: 0.8091 - val_loss: 0.4518 - val_accuracy: 0.7912
Epoch 34/100
313/313 [==============================] - 12s 39ms/step - loss: 0.4268 - accuracy: 0.8090 - val_loss: 0.4499 - val_accuracy: 0.7922
Epoch 35/100
313/313 [==============================] - 13s 43ms/step - loss: 0.4250 - accuracy: 0.8110 - val_loss: 0.4540 - val_accuracy: 0.7912
Epoch 36/100
313/313 [==============================] - 12s 38ms/step - loss: 0.4239 - accuracy: 0.8099 - val_loss: 0.4526 - val_accuracy: 0.7938
Epoch 37/100
313/313 [==============================] - 11s 34ms/step - loss: 0.4223 - accuracy: 0.8117 - val_loss: 0.4476 - val_accuracy: 0.7948
Epoch 38/100
313/313 [==============================] - 11s 35ms/step - loss: 0.4202 - accuracy: 0.8137 - val_loss: 0.4590 - val_accuracy: 0.7882
Epoch 39/100
313/313 [==============================] - 10s 31ms/step - loss: 0.4198 - accuracy: 0.8112 - val_loss: 0.4490 - val_accuracy: 0.7930
Epoch 40/100
313/313 [==============================] - 11s 34ms/step - loss: 0.4178 - accuracy: 0.8134 - val_loss: 0.4499 - val_accuracy: 0.7914

훈련 손실과 검증 손실

In [17]:

plt.plot(history.history["loss"])
plt.plot(history.history["val_loss"])
plt.xlabel("epoch")
plt.ylabel("loss")
plt.legend(["train", "val"])
plt.show()

단어 임베딩 사용하기 word embedding
각 단어를 고정된 크기의 실수 벡터로 바꿈

In [18]:

model2 = keras.Sequential()
model2.add(keras.layers.Embedding(500, 16, input_length=100))
model2.add(keras.layers.SimpleRNN(8))
model2.add(keras.layers.Dense(1, activation="sigmoid"))

In [19]:

model2.summary()

Model: "sequential_1"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 embedding (Embedding)       (None, 100, 16)           8000      
                                                                 
 simple_rnn_1 (SimpleRNN)    (None, 8)                 200       
                                                                 
 dense_1 (Dense)             (None, 1)                 9         
                                                                 
=================================================================
Total params: 8,209
Trainable params: 8,209
Non-trainable params: 0
_________________________________________________________________

학습시키기

In [20]:

rmsprop = keras.optimizers.RMSprop(learning_rate=1e-4)
model2.compile(optimizer=rmsprop, loss="binary_crossentropy", metrics=["accuracy"])
checkpoint_cb = keras.callbacks.ModelCheckpoint("best-embedding-model.h5",
                                                save_best_only=True)
early_stopping_cb = keras.callbacks.EarlyStopping(patience=3, 
                                                  restore_best_weights=True)
history = model2.fit(train_seq, train_target, 
                     epochs=100, batch_size=64, 
                     validation_data=(val_seq, val_target), 
                     callbacks=[checkpoint_cb, early_stopping_cb])

Epoch 1/100
313/313 [==============================] - 9s 24ms/step - loss: 0.6947 - accuracy: 0.5099 - val_loss: 0.6892 - val_accuracy: 0.5406
Epoch 2/100
313/313 [==============================] - 8s 25ms/step - loss: 0.6749 - accuracy: 0.6126 - val_loss: 0.6651 - val_accuracy: 0.6582
Epoch 3/100
313/313 [==============================] - 10s 31ms/step - loss: 0.6553 - accuracy: 0.6882 - val_loss: 0.6513 - val_accuracy: 0.6948
Epoch 4/100
313/313 [==============================] - 11s 36ms/step - loss: 0.6410 - accuracy: 0.7179 - val_loss: 0.6404 - val_accuracy: 0.7160
Epoch 5/100
313/313 [==============================] - 11s 34ms/step - loss: 0.6283 - accuracy: 0.7362 - val_loss: 0.6292 - val_accuracy: 0.7338
Epoch 6/100
313/313 [==============================] - 8s 27ms/step - loss: 0.6170 - accuracy: 0.7466 - val_loss: 0.6189 - val_accuracy: 0.7404
Epoch 7/100
313/313 [==============================] - 7s 23ms/step - loss: 0.6057 - accuracy: 0.7561 - val_loss: 0.6114 - val_accuracy: 0.7450
Epoch 8/100
313/313 [==============================] - 7s 23ms/step - loss: 0.5942 - accuracy: 0.7661 - val_loss: 0.6002 - val_accuracy: 0.7560
Epoch 9/100
313/313 [==============================] - 7s 23ms/step - loss: 0.5839 - accuracy: 0.7704 - val_loss: 0.5902 - val_accuracy: 0.7582
Epoch 10/100
313/313 [==============================] - 7s 23ms/step - loss: 0.5728 - accuracy: 0.7752 - val_loss: 0.5827 - val_accuracy: 0.7588
Epoch 11/100
313/313 [==============================] - 7s 23ms/step - loss: 0.5628 - accuracy: 0.7823 - val_loss: 0.5766 - val_accuracy: 0.7582
Epoch 12/100
313/313 [==============================] - 7s 23ms/step - loss: 0.5528 - accuracy: 0.7836 - val_loss: 0.5690 - val_accuracy: 0.7596
Epoch 13/100
313/313 [==============================] - 8s 25ms/step - loss: 0.5432 - accuracy: 0.7860 - val_loss: 0.5625 - val_accuracy: 0.7606
Epoch 14/100
313/313 [==============================] - 7s 24ms/step - loss: 0.5342 - accuracy: 0.7890 - val_loss: 0.5575 - val_accuracy: 0.7582
Epoch 15/100
313/313 [==============================] - 7s 23ms/step - loss: 0.5264 - accuracy: 0.7892 - val_loss: 0.5509 - val_accuracy: 0.7584
Epoch 16/100
313/313 [==============================] - 7s 24ms/step - loss: 0.5176 - accuracy: 0.7926 - val_loss: 0.5439 - val_accuracy: 0.7608
Epoch 17/100
313/313 [==============================] - 7s 23ms/step - loss: 0.5107 - accuracy: 0.7948 - val_loss: 0.5413 - val_accuracy: 0.7608
Epoch 18/100
313/313 [==============================] - 7s 23ms/step - loss: 0.5027 - accuracy: 0.7969 - val_loss: 0.5382 - val_accuracy: 0.7580
Epoch 19/100
313/313 [==============================] - 8s 24ms/step - loss: 0.4970 - accuracy: 0.7970 - val_loss: 0.5336 - val_accuracy: 0.7558
Epoch 20/100
313/313 [==============================] - 7s 24ms/step - loss: 0.4912 - accuracy: 0.7972 - val_loss: 0.5272 - val_accuracy: 0.7620
Epoch 21/100
313/313 [==============================] - 7s 24ms/step - loss: 0.4852 - accuracy: 0.7977 - val_loss: 0.5249 - val_accuracy: 0.7628
Epoch 22/100
313/313 [==============================] - 7s 24ms/step - loss: 0.4806 - accuracy: 0.7997 - val_loss: 0.5209 - val_accuracy: 0.7626
Epoch 23/100
313/313 [==============================] - 7s 24ms/step - loss: 0.4749 - accuracy: 0.8009 - val_loss: 0.5192 - val_accuracy: 0.7588
Epoch 24/100
313/313 [==============================] - 7s 24ms/step - loss: 0.4698 - accuracy: 0.8026 - val_loss: 0.5182 - val_accuracy: 0.7620
Epoch 25/100
313/313 [==============================] - 7s 24ms/step - loss: 0.4649 - accuracy: 0.8015 - val_loss: 0.5153 - val_accuracy: 0.7624
Epoch 26/100
313/313 [==============================] - 8s 25ms/step - loss: 0.4626 - accuracy: 0.8035 - val_loss: 0.5155 - val_accuracy: 0.7672
Epoch 27/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4604 - accuracy: 0.8036 - val_loss: 0.5130 - val_accuracy: 0.7654
Epoch 28/100
313/313 [==============================] - 8s 24ms/step - loss: 0.4558 - accuracy: 0.8060 - val_loss: 0.5175 - val_accuracy: 0.7610
Epoch 29/100
313/313 [==============================] - 7s 22ms/step - loss: 0.4530 - accuracy: 0.8076 - val_loss: 0.5145 - val_accuracy: 0.7630
Epoch 30/100
313/313 [==============================] - 7s 23ms/step - loss: 0.4499 - accuracy: 0.8075 - val_loss: 0.5133 - val_accuracy: 0.7688

In [21]:

plt.plot(history.history["loss"])
plt.plot(history.history["val_loss"])
plt.xlabel("epoch")
plt.ylabel("loss")
plt.legend(["train", "val"])
plt.show()

9-3. LSTM과 GRU 셀¶

RNN에서 빼놓을 수 없는 기술인 LSTM과 GRU
LSTM은 Long Short-Term Memory의 약자
GRU는 Gated Recurrent Unit의 약자

LSTM Long Short-Term Memory¶

타임스탭이 긴 데이터를 효과적으로 학습하기 위해 고안된 순환층

데이터 불러오기

In [22]:

from tensorflow.keras.datasets import imdb
from sklearn.model_selection import train_test_split
(train_input, train_target), (test_input, test_target) = imdb.load_data(num_words=100)
train_input, val_input, train_target, val_target = train_test_split(
    train_input, train_target, test_size=0.2, random_state=42
)

데이터 전처리 -> 샘플 길이 맞추기

In [23]:

from tensorflow.keras.preprocessing.sequence import pad_sequences
train_seq = pad_sequences(train_input, maxlen=100)
val_seq = pad_sequences(val_input, maxlen=100)

모델 만들기

In [24]:

from tensorflow import keras
model = keras.Sequential()
model.add(keras.layers.Embedding(500, 16, input_length=100))
model.add(keras.layers.LSTM(8))
model.add(keras.layers.Dense(1, activation="sigmoid"))

In [25]:

model.summary()

Model: "sequential_2"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 embedding_1 (Embedding)     (None, 100, 16)           8000      
                                                                 
 lstm (LSTM)                 (None, 8)                 800       
                                                                 
 dense_2 (Dense)             (None, 1)                 9         
                                                                 
=================================================================
Total params: 8,809
Trainable params: 8,809
Non-trainable params: 0
_________________________________________________________________

훈련시키기

In [26]:

rmsprop = keras.optimizers.RMSprop(learning_rate=1e-4)
model.compile(optimizer=rmsprop, 
              loss="binary_crossentropy",
              metrics=["accuracy"])
checkpoint_cb = keras.callbacks.ModelCheckpoint("best-lstm-model.h5",
                                                save_best_only=True)
early_stopping_cb = keras.callbacks.EarlyStopping(patience=3, 
                                                  restore_best_weights=True)
history = model.fit(train_seq, train_target, 
                    epochs=100, 
                    batch_size=64, 
                    validation_data=(val_seq, val_target),
                    callbacks=[checkpoint_cb, early_stopping_cb])

Epoch 1/100
313/313 [==============================] - 15s 39ms/step - loss: 0.6930 - accuracy: 0.5115 - val_loss: 0.6927 - val_accuracy: 0.5284
Epoch 2/100
313/313 [==============================] - 12s 39ms/step - loss: 0.6919 - accuracy: 0.5513 - val_loss: 0.6911 - val_accuracy: 0.5974
Epoch 3/100
313/313 [==============================] - 13s 42ms/step - loss: 0.6890 - accuracy: 0.6032 - val_loss: 0.6862 - val_accuracy: 0.6252
Epoch 4/100
313/313 [==============================] - 12s 40ms/step - loss: 0.6739 - accuracy: 0.6398 - val_loss: 0.6549 - val_accuracy: 0.6452
Epoch 5/100
313/313 [==============================] - 16s 53ms/step - loss: 0.6425 - accuracy: 0.6512 - val_loss: 0.6372 - val_accuracy: 0.6538
Epoch 6/100
313/313 [==============================] - 18s 56ms/step - loss: 0.6307 - accuracy: 0.6633 - val_loss: 0.6271 - val_accuracy: 0.6618
Epoch 7/100
313/313 [==============================] - 16s 51ms/step - loss: 0.6222 - accuracy: 0.6676 - val_loss: 0.6197 - val_accuracy: 0.6704
Epoch 8/100
313/313 [==============================] - 12s 38ms/step - loss: 0.6146 - accuracy: 0.6737 - val_loss: 0.6143 - val_accuracy: 0.6744
Epoch 9/100
313/313 [==============================] - 13s 40ms/step - loss: 0.6072 - accuracy: 0.6817 - val_loss: 0.6078 - val_accuracy: 0.6734
Epoch 10/100
313/313 [==============================] - 12s 40ms/step - loss: 0.6013 - accuracy: 0.6870 - val_loss: 0.6008 - val_accuracy: 0.6832
Epoch 11/100
313/313 [==============================] - 12s 39ms/step - loss: 0.5962 - accuracy: 0.6916 - val_loss: 0.5956 - val_accuracy: 0.6888
Epoch 12/100
313/313 [==============================] - 11s 36ms/step - loss: 0.5916 - accuracy: 0.6941 - val_loss: 0.5925 - val_accuracy: 0.6884
Epoch 13/100
313/313 [==============================] - 11s 35ms/step - loss: 0.5886 - accuracy: 0.6951 - val_loss: 0.5956 - val_accuracy: 0.6858
Epoch 14/100
313/313 [==============================] - 11s 35ms/step - loss: 0.5860 - accuracy: 0.6962 - val_loss: 0.5872 - val_accuracy: 0.6930
Epoch 15/100
313/313 [==============================] - 13s 40ms/step - loss: 0.5841 - accuracy: 0.6992 - val_loss: 0.5882 - val_accuracy: 0.6882
Epoch 16/100
313/313 [==============================] - 14s 44ms/step - loss: 0.5827 - accuracy: 0.6982 - val_loss: 0.5833 - val_accuracy: 0.6918
Epoch 17/100
313/313 [==============================] - 14s 44ms/step - loss: 0.5814 - accuracy: 0.6991 - val_loss: 0.5820 - val_accuracy: 0.6960
Epoch 18/100
313/313 [==============================] - 15s 48ms/step - loss: 0.5808 - accuracy: 0.6992 - val_loss: 0.5863 - val_accuracy: 0.6898
Epoch 19/100
313/313 [==============================] - 17s 54ms/step - loss: 0.5799 - accuracy: 0.6985 - val_loss: 0.5804 - val_accuracy: 0.6938
Epoch 20/100
313/313 [==============================] - 17s 55ms/step - loss: 0.5789 - accuracy: 0.7003 - val_loss: 0.5844 - val_accuracy: 0.6928
Epoch 21/100
313/313 [==============================] - 16s 50ms/step - loss: 0.5785 - accuracy: 0.7005 - val_loss: 0.5794 - val_accuracy: 0.6980
Epoch 22/100
313/313 [==============================] - 12s 39ms/step - loss: 0.5776 - accuracy: 0.7002 - val_loss: 0.5780 - val_accuracy: 0.6998
Epoch 23/100
313/313 [==============================] - 11s 37ms/step - loss: 0.5769 - accuracy: 0.6995 - val_loss: 0.5792 - val_accuracy: 0.6944
Epoch 24/100
313/313 [==============================] - 13s 42ms/step - loss: 0.5768 - accuracy: 0.7013 - val_loss: 0.5771 - val_accuracy: 0.6992
Epoch 25/100
313/313 [==============================] - 13s 42ms/step - loss: 0.5762 - accuracy: 0.6995 - val_loss: 0.5765 - val_accuracy: 0.6988
Epoch 26/100
313/313 [==============================] - 12s 37ms/step - loss: 0.5757 - accuracy: 0.6995 - val_loss: 0.5763 - val_accuracy: 0.6990
Epoch 27/100
313/313 [==============================] - 11s 37ms/step - loss: 0.5751 - accuracy: 0.6999 - val_loss: 0.5762 - val_accuracy: 0.6974
Epoch 28/100
313/313 [==============================] - 12s 37ms/step - loss: 0.5749 - accuracy: 0.7016 - val_loss: 0.5759 - val_accuracy: 0.6952
Epoch 29/100
313/313 [==============================] - 14s 45ms/step - loss: 0.5743 - accuracy: 0.7018 - val_loss: 0.5750 - val_accuracy: 0.6990
Epoch 30/100
313/313 [==============================] - 14s 46ms/step - loss: 0.5738 - accuracy: 0.7005 - val_loss: 0.5769 - val_accuracy: 0.6948
Epoch 31/100
313/313 [==============================] - 14s 44ms/step - loss: 0.5735 - accuracy: 0.7006 - val_loss: 0.5782 - val_accuracy: 0.6984
Epoch 32/100
313/313 [==============================] - 15s 47ms/step - loss: 0.5729 - accuracy: 0.7031 - val_loss: 0.5740 - val_accuracy: 0.6994
Epoch 33/100
313/313 [==============================] - 15s 49ms/step - loss: 0.5725 - accuracy: 0.7010 - val_loss: 0.5733 - val_accuracy: 0.7002
Epoch 34/100
313/313 [==============================] - 15s 48ms/step - loss: 0.5718 - accuracy: 0.7031 - val_loss: 0.5735 - val_accuracy: 0.6970
Epoch 35/100
313/313 [==============================] - 15s 47ms/step - loss: 0.5718 - accuracy: 0.7041 - val_loss: 0.5731 - val_accuracy: 0.7034
Epoch 36/100
313/313 [==============================] - 13s 40ms/step - loss: 0.5714 - accuracy: 0.7019 - val_loss: 0.5733 - val_accuracy: 0.7022
Epoch 37/100
313/313 [==============================] - 15s 48ms/step - loss: 0.5707 - accuracy: 0.7026 - val_loss: 0.5840 - val_accuracy: 0.6878
Epoch 38/100
313/313 [==============================] - 14s 45ms/step - loss: 0.5703 - accuracy: 0.7030 - val_loss: 0.5798 - val_accuracy: 0.6918

In [27]:

import matplotlib.pyplot as plt
plt.plot(history.history["loss"])
plt.plot(history.history["val_loss"])
plt.xlabel("epoch")
plt.ylabel("loss")
plt.legend(["train", "val"])
plt.show()

2개의 층을 연결하기
이때, 마지막 층을 제외하고는 모든 타임스텝의 은닉 상태를 전달해야 한다

In [28]:

model3 = keras.Sequential()
model3.add(keras.layers.Embedding(500, 16, input_length=100))
model3.add(keras.layers.LSTM(8, dropout=0.3, return_sequences=True))
model3.add(keras.layers.LSTM(8, dropout=0.3))
model3.add(keras.layers.Dense(1, activation="sigmoid"))

In [29]:

model3.summary()

Model: "sequential_3"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 embedding_2 (Embedding)     (None, 100, 16)           8000      
                                                                 
 lstm_1 (LSTM)               (None, 100, 8)            800       
                                                                 
 lstm_2 (LSTM)               (None, 8)                 544       
                                                                 
 dense_3 (Dense)             (None, 1)                 9         
                                                                 
=================================================================
Total params: 9,353
Trainable params: 9,353
Non-trainable params: 0
_________________________________________________________________

훈련시키기

In [30]:

rmsprop = keras.optimizers.RMSprop(learning_rate=1e-4)
model3.compile(optimizer=rmsprop, 
               loss="binary_crossentropy",
               metrics=["accuracy"])
checkpoint_cb = keras.callbacks.ModelCheckpoint("best-2rnn-model.h5",
                                                save_best_only=True)
early_stopping_cb = keras.callbacks.EarlyStopping(patience=3, 
                                                  restore_best_weights=True)
history = model3.fit(train_seq, train_target, 
                     epochs=100, batch_size=64, 
                     validation_data=(val_seq, val_target),
                     callbacks=[checkpoint_cb, early_stopping_cb])

Epoch 1/100
313/313 [==============================] - 36s 93ms/step - loss: 0.6930 - accuracy: 0.5062 - val_loss: 0.6930 - val_accuracy: 0.4930
Epoch 2/100
313/313 [==============================] - 25s 79ms/step - loss: 0.6924 - accuracy: 0.5232 - val_loss: 0.6920 - val_accuracy: 0.5218
Epoch 3/100
313/313 [==============================] - 24s 78ms/step - loss: 0.6902 - accuracy: 0.5691 - val_loss: 0.6883 - val_accuracy: 0.5786
Epoch 4/100
313/313 [==============================] - 24s 77ms/step - loss: 0.6810 - accuracy: 0.6102 - val_loss: 0.6695 - val_accuracy: 0.6348
Epoch 5/100
313/313 [==============================] - 28s 90ms/step - loss: 0.6460 - accuracy: 0.6444 - val_loss: 0.6245 - val_accuracy: 0.6586
Epoch 6/100
313/313 [==============================] - 26s 84ms/step - loss: 0.6220 - accuracy: 0.6583 - val_loss: 0.6118 - val_accuracy: 0.6706
Epoch 7/100
313/313 [==============================] - 25s 80ms/step - loss: 0.6121 - accuracy: 0.6658 - val_loss: 0.6046 - val_accuracy: 0.6780
Epoch 8/100
313/313 [==============================] - 31s 99ms/step - loss: 0.6059 - accuracy: 0.6741 - val_loss: 0.5991 - val_accuracy: 0.6788
Epoch 9/100
313/313 [==============================] - 27s 86ms/step - loss: 0.5996 - accuracy: 0.6818 - val_loss: 0.5941 - val_accuracy: 0.6834
Epoch 10/100
313/313 [==============================] - 30s 96ms/step - loss: 0.5986 - accuracy: 0.6811 - val_loss: 0.5907 - val_accuracy: 0.6874
Epoch 11/100
313/313 [==============================] - 24s 77ms/step - loss: 0.5945 - accuracy: 0.6850 - val_loss: 0.5887 - val_accuracy: 0.6872
Epoch 12/100
313/313 [==============================] - 23s 74ms/step - loss: 0.5931 - accuracy: 0.6838 - val_loss: 0.5860 - val_accuracy: 0.6892
Epoch 13/100
313/313 [==============================] - 24s 78ms/step - loss: 0.5913 - accuracy: 0.6870 - val_loss: 0.5840 - val_accuracy: 0.6896
Epoch 14/100
313/313 [==============================] - 29s 94ms/step - loss: 0.5880 - accuracy: 0.6909 - val_loss: 0.5873 - val_accuracy: 0.6844
Epoch 15/100
313/313 [==============================] - 29s 91ms/step - loss: 0.5891 - accuracy: 0.6874 - val_loss: 0.5813 - val_accuracy: 0.6906
Epoch 16/100
313/313 [==============================] - 31s 101ms/step - loss: 0.5876 - accuracy: 0.6914 - val_loss: 0.5802 - val_accuracy: 0.6908
Epoch 17/100
313/313 [==============================] - 29s 94ms/step - loss: 0.5863 - accuracy: 0.6898 - val_loss: 0.5830 - val_accuracy: 0.6922
Epoch 18/100
313/313 [==============================] - 30s 96ms/step - loss: 0.5856 - accuracy: 0.6920 - val_loss: 0.5824 - val_accuracy: 0.6920
Epoch 19/100
313/313 [==============================] - 28s 90ms/step - loss: 0.5852 - accuracy: 0.6911 - val_loss: 0.5779 - val_accuracy: 0.6962
Epoch 20/100
313/313 [==============================] - 24s 75ms/step - loss: 0.5845 - accuracy: 0.6914 - val_loss: 0.5789 - val_accuracy: 0.6922
Epoch 21/100
313/313 [==============================] - 22s 70ms/step - loss: 0.5831 - accuracy: 0.6938 - val_loss: 0.5771 - val_accuracy: 0.7020
Epoch 22/100
313/313 [==============================] - 32s 104ms/step - loss: 0.5821 - accuracy: 0.6927 - val_loss: 0.5851 - val_accuracy: 0.6880
Epoch 23/100
313/313 [==============================] - 30s 95ms/step - loss: 0.5838 - accuracy: 0.6917 - val_loss: 0.5757 - val_accuracy: 0.6970
Epoch 24/100
313/313 [==============================] - 29s 93ms/step - loss: 0.5815 - accuracy: 0.6938 - val_loss: 0.5754 - val_accuracy: 0.6986
Epoch 25/100
313/313 [==============================] - 34s 109ms/step - loss: 0.5819 - accuracy: 0.6933 - val_loss: 0.5763 - val_accuracy: 0.6980
Epoch 26/100
313/313 [==============================] - 31s 100ms/step - loss: 0.5804 - accuracy: 0.6950 - val_loss: 0.5738 - val_accuracy: 0.7028
Epoch 27/100
313/313 [==============================] - 26s 84ms/step - loss: 0.5803 - accuracy: 0.6943 - val_loss: 0.5735 - val_accuracy: 0.7008
Epoch 28/100
313/313 [==============================] - 24s 78ms/step - loss: 0.5786 - accuracy: 0.6939 - val_loss: 0.5734 - val_accuracy: 0.7014
Epoch 29/100
313/313 [==============================] - 24s 77ms/step - loss: 0.5788 - accuracy: 0.6947 - val_loss: 0.5728 - val_accuracy: 0.7010
Epoch 30/100
313/313 [==============================] - 23s 74ms/step - loss: 0.5776 - accuracy: 0.6975 - val_loss: 0.5712 - val_accuracy: 0.7032
Epoch 31/100
313/313 [==============================] - 24s 78ms/step - loss: 0.5767 - accuracy: 0.6950 - val_loss: 0.5714 - val_accuracy: 0.7028
Epoch 32/100
313/313 [==============================] - 24s 75ms/step - loss: 0.5759 - accuracy: 0.6927 - val_loss: 0.5713 - val_accuracy: 0.7036
Epoch 33/100
313/313 [==============================] - 27s 87ms/step - loss: 0.5760 - accuracy: 0.6983 - val_loss: 0.5710 - val_accuracy: 0.7054
Epoch 34/100
313/313 [==============================] - 25s 81ms/step - loss: 0.5763 - accuracy: 0.6973 - val_loss: 0.5701 - val_accuracy: 0.7076
Epoch 35/100
313/313 [==============================] - 28s 89ms/step - loss: 0.5744 - accuracy: 0.6941 - val_loss: 0.5701 - val_accuracy: 0.7056
Epoch 36/100
313/313 [==============================] - 26s 82ms/step - loss: 0.5740 - accuracy: 0.6988 - val_loss: 0.5733 - val_accuracy: 0.6992
Epoch 37/100
313/313 [==============================] - 26s 82ms/step - loss: 0.5740 - accuracy: 0.6973 - val_loss: 0.5716 - val_accuracy: 0.7014

In [31]:

plt.plot(history.history["loss"])
plt.plot(history.history["val_loss"])
plt.xlabel("epoch")
plt.ylabel("loss")
plt.legend(["train", "val"])
plt.show()

GRU Gated Recurrent Unit
LSTM의 간소화된 버전
은닉 상태 하나만 포함한다

In [32]:

model4 = keras.Sequential()
model4.add(keras.layers.Embedding(500, 16, input_length=100))
model4.add(keras.layers.GRU(8))
model4.add(keras.layers.Dense(1, activation="sigmoid"))

In [33]:

model4.summary()

Model: "sequential_4"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 embedding_3 (Embedding)     (None, 100, 16)           8000      
                                                                 
 gru (GRU)                   (None, 8)                 624       
                                                                 
 dense_4 (Dense)             (None, 1)                 9         
                                                                 
=================================================================
Total params: 8,633
Trainable params: 8,633
Non-trainable params: 0
_________________________________________________________________

훈련시키기

In [34]:

rmsprop = keras.optimizers.RMSprop(learning_rate=1e-4)
model4.compile(optimizer=rmsprop, 
               loss="binary_crossentropy",
               metrics=["accuracy"])
checkpoint_cb = keras.callbacks.ModelCheckpoint("best-gru-model.h5",
                                                save_best_only=True)
early_stopping_cb = keras.callbacks.EarlyStopping(patience=3, 
                                                  restore_best_weights=True)
history = model4.fit(train_seq, train_target, 
                     epochs=100, 
                     batch_size=64, 
                     validation_data=(val_seq, val_target),
                     callbacks=[checkpoint_cb, early_stopping_cb])

Epoch 1/100
313/313 [==============================] - 16s 42ms/step - loss: 0.6929 - accuracy: 0.5105 - val_loss: 0.6928 - val_accuracy: 0.5216
Epoch 2/100
313/313 [==============================] - 15s 48ms/step - loss: 0.6924 - accuracy: 0.5258 - val_loss: 0.6923 - val_accuracy: 0.5392
Epoch 3/100
313/313 [==============================] - 14s 45ms/step - loss: 0.6918 - accuracy: 0.5379 - val_loss: 0.6918 - val_accuracy: 0.5304
Epoch 4/100
313/313 [==============================] - 13s 41ms/step - loss: 0.6911 - accuracy: 0.5418 - val_loss: 0.6911 - val_accuracy: 0.5412
Epoch 5/100
313/313 [==============================] - 12s 39ms/step - loss: 0.6901 - accuracy: 0.5505 - val_loss: 0.6900 - val_accuracy: 0.5672
Epoch 6/100
313/313 [==============================] - 13s 42ms/step - loss: 0.6888 - accuracy: 0.5583 - val_loss: 0.6887 - val_accuracy: 0.5706
Epoch 7/100
313/313 [==============================] - 13s 41ms/step - loss: 0.6872 - accuracy: 0.5617 - val_loss: 0.6870 - val_accuracy: 0.5642
Epoch 8/100
313/313 [==============================] - 12s 40ms/step - loss: 0.6850 - accuracy: 0.5659 - val_loss: 0.6845 - val_accuracy: 0.5750
Epoch 9/100
313/313 [==============================] - 13s 40ms/step - loss: 0.6822 - accuracy: 0.5712 - val_loss: 0.6818 - val_accuracy: 0.5764
Epoch 10/100
313/313 [==============================] - 13s 40ms/step - loss: 0.6785 - accuracy: 0.5763 - val_loss: 0.6775 - val_accuracy: 0.5838
Epoch 11/100
313/313 [==============================] - 13s 42ms/step - loss: 0.6735 - accuracy: 0.5858 - val_loss: 0.6716 - val_accuracy: 0.5894
Epoch 12/100
313/313 [==============================] - 14s 46ms/step - loss: 0.6662 - accuracy: 0.5969 - val_loss: 0.6625 - val_accuracy: 0.6046
Epoch 13/100
313/313 [==============================] - 13s 41ms/step - loss: 0.6541 - accuracy: 0.6133 - val_loss: 0.6474 - val_accuracy: 0.6258
Epoch 14/100
313/313 [==============================] - 13s 40ms/step - loss: 0.6313 - accuracy: 0.6424 - val_loss: 0.6196 - val_accuracy: 0.6532
Epoch 15/100
313/313 [==============================] - 14s 45ms/step - loss: 0.6090 - accuracy: 0.6686 - val_loss: 0.6077 - val_accuracy: 0.6660
Epoch 16/100
313/313 [==============================] - 13s 42ms/step - loss: 0.6006 - accuracy: 0.6783 - val_loss: 0.6010 - val_accuracy: 0.6750
Epoch 17/100
313/313 [==============================] - 12s 39ms/step - loss: 0.5943 - accuracy: 0.6831 - val_loss: 0.5966 - val_accuracy: 0.6812
Epoch 18/100
313/313 [==============================] - 13s 40ms/step - loss: 0.5897 - accuracy: 0.6887 - val_loss: 0.5907 - val_accuracy: 0.6866
Epoch 19/100
313/313 [==============================] - 13s 41ms/step - loss: 0.5867 - accuracy: 0.6935 - val_loss: 0.5898 - val_accuracy: 0.6892
Epoch 20/100
313/313 [==============================] - 13s 41ms/step - loss: 0.5842 - accuracy: 0.6964 - val_loss: 0.5867 - val_accuracy: 0.6884
Epoch 21/100
313/313 [==============================] - 13s 41ms/step - loss: 0.5820 - accuracy: 0.6976 - val_loss: 0.5835 - val_accuracy: 0.6922
Epoch 22/100
313/313 [==============================] - 15s 47ms/step - loss: 0.5806 - accuracy: 0.6987 - val_loss: 0.5828 - val_accuracy: 0.6966
Epoch 23/100
313/313 [==============================] - 16s 51ms/step - loss: 0.5792 - accuracy: 0.6991 - val_loss: 0.5811 - val_accuracy: 0.6904
Epoch 24/100
313/313 [==============================] - 17s 53ms/step - loss: 0.5783 - accuracy: 0.6999 - val_loss: 0.5805 - val_accuracy: 0.6918
Epoch 25/100
313/313 [==============================] - 15s 48ms/step - loss: 0.5772 - accuracy: 0.6996 - val_loss: 0.5828 - val_accuracy: 0.6934
Epoch 26/100
313/313 [==============================] - 14s 44ms/step - loss: 0.5765 - accuracy: 0.7019 - val_loss: 0.5783 - val_accuracy: 0.6960
Epoch 27/100
313/313 [==============================] - 12s 40ms/step - loss: 0.5757 - accuracy: 0.7016 - val_loss: 0.5779 - val_accuracy: 0.6948
Epoch 28/100
313/313 [==============================] - 12s 40ms/step - loss: 0.5752 - accuracy: 0.7008 - val_loss: 0.5782 - val_accuracy: 0.6974
Epoch 29/100
313/313 [==============================] - 12s 39ms/step - loss: 0.5747 - accuracy: 0.7036 - val_loss: 0.5781 - val_accuracy: 0.6974
Epoch 30/100
313/313 [==============================] - 13s 42ms/step - loss: 0.5746 - accuracy: 0.7008 - val_loss: 0.5795 - val_accuracy: 0.6962

In [35]:

plt.plot(history.history["loss"])
plt.plot(history.history["val_loss"])
plt.xlabel("epoch")
plt.ylabel("loss")
plt.legend(["train", "val"])
plt.show()

In [ ]:

저작자표시

'Data Science > AI' 카테고리의 다른 글

인공지능 주제 탐색 #0 \| AI 분야 알아보기 (0)	2023.03.23
인공지능 #Fin \| 인공지능 기초 용어 정리 (1)	2023.03.14
인공지능 #8-2 \| CNN - 텐서플로 Conv2D, MaxPooling2D, plot_model (0)	2023.03.13
[에러 해결] UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbd in position 0: invalid start byte (0)	2023.03.11
인공지능 #8-1 \| CNN 합성곱 알아보기(필터(커널), 피처 맵, 패딩, 스트라이드, 풀링) (0)	2023.03.10

현재글인공지능 #9 | 텍스트를 위한 인공 신경망(NLP, IMDB, LSTM, GRU)

HappyTomatoLife

기록하는 토마토

linear DS, binary search, REACT, Deque, 자료구조, OCaml, 함수형 언어, 반복문, Expression evaluation, linked stack, linked Queue, Doubly Linked List, maze problem, DS, SQL, 교육상담, singly linked list, 조건문, react.js, JavaScript,

Today :
Yesterday :

HappyTomatoLife

인공지능 #9 | 텍스트를 위한 인공 신경망(NLP, IMDB, LSTM, GRU)

9. NLP - 텍스트를 위한 인공 신경망¶

9-1. 순차 데이터와 순환 신경망¶

9-2. 순환 신경망으로 IMDB 리뷰 분류하기¶

순환 신경망 만들기¶

9-3. LSTM과 GRU 셀¶

LSTM Long Short-Term Memory¶

'Data Science > AI' 카테고리의 다른 글

'Data Science/AI'의 다른글

티스토리툴바

« 2025/01 »
일	월	화	수	목	금	토
			1	2	3	4
5	6	7	8	9	10	11
12	13	14	15	16	17	18
19	20	21	22	23	24	25
26	27	28	29	30	31

인공지능 #9 | 텍스트를 위한 인공 신경망(NLP, IMDB, LSTM, GRU)

9. NLP - 텍스트를 위한 인공 신경망¶

9-1. 순차 데이터와 순환 신경망¶

9-2. 순환 신경망으로 IMDB 리뷰 분류하기¶

순환 신경망 만들기¶

9-3. LSTM과 GRU 셀¶

LSTM Long Short-Term Memory¶

'Data Science > AI' 카테고리의 다른 글

'Data Science/AI'의 다른글

관련글

티스토리툴바