PyTorch / PyTorch Lightning 모듈 사용법

 

(image source)

PyTorch/PyTorch Lightning 모듈 설명

1. PyTorch 

Containers 

• nn.Module 
• nn.Sequential 
• nn.ModuleList 
• nn.ModuleDict 

Learnable module 

Linear Layers

• nn.Linear - [self-explain] [code]

Convolution Layers

• nn.Conv1d 
• nn.Conv2d - [self-explain]
• nn.Conv3d 
• nn.ConvTranspose1d 
• nn.ConvTranspose2d 
• nn.ConvTranspose3d 

Recurrent Layers

• nn.RNNCell - [self-explain]
• nn.RNN 
• nn.LSTM
• nn.GRU

Transformer Layers

• nn.TransformerEncoder 

Geometric Layers (for graph)

Non-learnable module 

Pooling layers 

• nn.MaxPool2d 
• nn.Maxpool3d
• nn.AdaptiveMaxPool2d 

Non-linear Activations

• nn.Sigmoid 
• nn.Tanh 
• nn.ReLU 
• nn.LeakyReLU

Regularization

Normalization Layers 

• nn.BatchNorm1d - [self-explain]
• nn.BatchNorm2d - 

Dropout Layers 

• nn.Dropout - [self-explain]
• nn.Dropout2d 

Loss Functions 

Binary-class 

• nn.BCELoss 

Multiple-class 

• nn.CrossEntropyLoss 

Customize 

• example 

Metrics 

Distance Functions

• nn.CosineSimilarity 

Optimization 

Optimization Algorithm

• optim.SGD 
• optim.Adam

Learning Rate Scheduler 

• lr_scheduler.CosineAnnealingWarmRestarts 

Gradient Clipping (to prevent exploding gradients)

• Norm clipping: nn.utils.clip_grad_norm_
• Value clipping: nn.utils.clip_grad_value_

Early Stopping 

Tools 

Save & Load 

Torch Hub 

Visualization 

• filters 
• feature maps 
• forward hooks 
• backward hooks 
• layer hooks 

XAI 

• CAM

--------------------------------------------------------

2. PyTorch Lightning 

--------------------------------------------------------

3. W&B

Hyperparameter Sweeps 

--------------------------------------------------------

4. Design Practice 

Data Augmentation 

• image classification 
• image object detection 
• image object segmentation 
• video action classification 
• video spatial-temporal action detection

Architecture for Computer Vision 

• Inception & GoogLeNet 
• Residual connection & ResNet 
• DenseNet 
• ViT 
• GCN

시퀀스 프레임(i.e., 비디오 데이터)를 처리하기 위한 트릭 

• Global Temporal pooling with ResNext50 - [self-explain]

Architecture for NLP 

• BERT 
• Transformer