Sigmoid, Softmax, Cross entropy

• Sigmoid

With only two classes, performs exactly the same operation as Softmax.

 

• Logits

It refers to the stage before passing the final Sigmoid or Softmax, that is, the Raw Output of the Model.

Sigmoid and Softmax receive the Logits to safely perform the operation.

 

• Softmax

For multi-label available only, where one object can have multiple classes at the same time.

---

1. Softmax

 

case1. 1d array.

 

step1. Create x arbitrarily

x=np.random.randint(0,5,(3,))
x
>>>
array([0, 3, 1])

 

step2. Apply the softmax function to x, it transforms into a matrix with a value between 0 and 1 and a sum of 1.

#tf.nn
#Primitive Neural Net

#tf.nn.softmax(logits, axis=None, name=None)

y=tf.nn.softmax(x.astype(np.float32))

y.numpy()
>>>
array([0.04201007, 0.8437947 , 0.11419519], dtype=float32)

y.numpy().sum()
>>>
0.99999994 #not exactly 1 not like pytorch

---

case2. 2d array.

 

step1. Create 2d array x arbitrarily

x=np.random.rand(3,5)
x
>>>
array([[0.66570611, 0.1626294 , 0.36706266, 0.29843456, 0.01490628],
       [0.96794399, 0.33415951, 0.70469845, 0.93473211, 0.06132272],
       [0.1428281 , 0.54365674, 0.05789707, 0.66223181, 0.37162914]])

 

step2. Apply the softmax function to x.

y=tf.nn.softmax(x)
y
>>>
<tf.Tensor: shape=(3, 5), dtype=float64, numpy=
array([[0.2808404 , 0.16981504, 0.20833411, 0.19451611, 0.14649433],
       [0.27245334, 0.1445583 , 0.20939487, 0.26355326, 0.11004024],
       [0.15748642, 0.23513689, 0.14466319, 0.2647386 , 0.1979749 ]])>
       
y.numpy().sum()
>>>
3.0

y.numpy().sum(axis=1)
>>>
array([1., 1., 1.])

 

 

2. log softmax

 

- way 1.

#tf.math.log
#Computes natural logarithm of x element-wise.

tf.math.log(tf.nn.softmax(x.astype(np.float32)))
>>>
<tf.Tensor: shape=(3,), dtype=float32, numpy=array([-1.3265626 , -0.32656264, -4.3265624 ], dtype=float32)>

---

- way 2.

#tf.nn.log_softmax(logits, axis=None, name=None)
#Computes log softmax activations.

tf.nn.log_softmax(x.astype(np.float32))
>>>
<tf.Tensor: shape=(3,), dtype=float32, numpy=array([-1.3265626 , -0.32656264, -4.326563  ], dtype=float32)>

 

• categorical_crossentropy vs softmax_cross_entropy_with_logits

Loss function of multiple classifications.

---

- way 1. tf.keras.losses.categorical_crossentropy

 

step1. Create one-hot array y same shape as x shape.

y=np.random.randint(0,5,(3,))
y_one_hot =tf.one_hot(y,depth=5, dtype=tf.float32)
y_one_hot
>>>
<tf.Tensor: shape=(3, 5), dtype=float32, numpy=
array([[0., 0., 1., 0., 0.],
       [0., 1., 0., 0., 0.],
       [0., 0., 0., 1., 0.]], dtype=float32)>

 

step2. tf.keras.losses.categorical_crossentropy

tf.keras.losses.categorical_crossentropy(y_one_hot, tf.nn.softmax(x)).numpy()
>>>
array([2.01969494, 1.59450077, 1.80262151])

---

- way 2. tf.nn.softmax_cross_entropy_with_logits

tf.nn.softmax_cross_entropy_with_logits(y_one_hot, x).numpy()
>>>
array([2.01969494, 1.59450077, 1.80262151])

 

https://junstar92.tistory.com/120