*Memos:
- My post explains GELU, Mish, SiLU and Softplus.
- My post explains heaviside() and Identity().
- My post explains ReLU() and LeakyReLU().
- My post explains PReLU() and ELU().
- My post explains SELU() and CELU().
- My post explains GELU() and Mish().
- My post explains Tanh() and Softsign().
- My post explains Sigmoid() and Softmax().
SiLU() can get the 0D or more D tensor of the zero or more values computed by SiLU function from the 0D or more D tensor of zero or more elements as shown below:
*Memos:
- The 1st argument for initialization is
inplace(Optional-Default:False-Type:bool): *Memos:- It does in-place operation.
- Keep it
Falsebecause it's problematic withTrue.
- The 1st argument is
input(Required-Type:tensoroffloat).
import torch
from torch import nn
my_tensor = torch.tensor([8., -3., 0., 1., 5., -2., -1., 4.])
silu = nn.SiLU()
silu(input=my_tensor)
# tensor([7.9973, -0.1423, 0.0000, 0.7311, 4.9665, -0.2384, -0.2689, 3.9281])
silu
# SiLU()
silu.inplace
# False
silu = nn.SiLU(inplace=True)
silu(input=my_tensor)
# tensor([7.9973, -0.1423, 0.0000, 0.7311, 4.9665, -0.2384, -0.2689, 3.9281])
my_tensor = torch.tensor([[8., -3., 0., 1.],
[5., -2., -1., 4.]])
silu = nn.SiLU()
silu(input=my_tensor)
# tensor([[7.9973, -0.1423, 0.0000, 0.7311],
# [4.9665, -0.2384, -0.2689, 3.9281]])
my_tensor = torch.tensor([[[8., -3.], [0., 1.]],
[[5., -2.], [-1., 4.]]])
silu = nn.SiLU()
silu(input=my_tensor)
# tensor([[[7.9973, -0.1423], [0.0000, 0.7311]],
# [[4.9665, -0.2384], [-0.2689, 3.9281]]])
Softplus() can get the 0D or more D tensor of the zero or more values computed by Mish function from the 0D or more D tensor of zero or more elements as shown below:
*Memos:
- The 1st argument for initialization is
beta(Optional-Default:1.0-Type:float). *It's applied to Mish function. - The 2nd argument for initialization is
threshold(Optional-Default:20.0-Type:float). *The values above it reverted. - The 1st argument is
input(Required-Type:tensoroffloat).
import torch
from torch import nn
my_tensor = torch.tensor([8., -3., 0., 1., 5., -2., -1., 4.])
mish = nn.Mish()
mish(input=my_tensor)
# tensor([8.0000, -0.1456, 0.0000, 0.8651, 4.9996, -0.2525, -0.3034, 3.9974])
mish
# Mish()
mish.inplace
# False
mish = nn.Mish(inplace=True)
mish(input=my_tensor)
# tensor([8.0000, -0.1456, 0.0000, 0.8651, 4.9996, -0.2525, -0.3034, 3.9974])
my_tensor = torch.tensor([[8., -3., 0., 1.],
[5., -2., -1., 4.]])
mish = nn.Mish()
mish(input=my_tensor)
# tensor([[8.0000, -0.1456, 0.0000, 0.8651],
# [4.9996, -0.2525, -0.3034, 3.9974]])
my_tensor = torch.tensor([[[8., -3.], [0., 1.]],
[[5., -2.], [-1., 4.]]])
mish = nn.Mish()
mish(input=my_tensor)
# tensor([[[8.0000, -0.1456], [0.0000, 0.8651]]
# [[4.9996, -0.2525], [-0.3034, 3.9974]]])
Top comments (0)