Juan Carlos

Posted on May 27, 2019

Tensors for busy people

#tensors #machinelearning #beginners #hpc

Wat?

Whats Scalar?.
Whats Vector?.
Whats Matrix?.
Whats Tensor?.

This is a quick and short intro to Tensors for busy people...
I will keep it KISS and to the minimum possible for easy reading.

Scalar is the minimum possible unit we are working with, like an integer.

Vector is a bunch of Scalars in a particular axis, can be drawn as a line of boxes or items, lets say is a list or array collection of items.

Matrix is a 2 dimensions arrays, usually represented as a table.

Tensor is a multi-dimensional array, usually can be represented graphically as a cube.

Rank & Dimension

Whats Rank?.
Whats Dimension?.

You can imagine lists of integers, accommodated on different layouts, on a 3D space, or 2D space, etc thats the Rank.

How it looks like on Code?.

Scalar

let myscalar = 42

Scalar can be a variety of things, usually numeric values, to keep things simple and easy to understand we will use an integer here, 42 is our Scalar.

Vector

let myvector = [1, 2, 3]

Vector is a collection of items, we continue using integers, it can be seen on the code as an array or list, you can draw it as a Rank 1 Vector.

Matrix

let mymatrix = [
                 [1, 2, 3],
                 [4, 5, 6],
               ]

We continue adding dimensions then we end up with the Matrix, a 2D Tensor, can be simplified on code as a list with lists inside.

Tensor

let mytensor = [
                 [
                   [1, 2, 3],
                   [4, 5, 6],
                 ],
                 [
                   [7,   8,  9],
                   [10, 11, 12],
                 ],
               ]

Wow, we reached the crazy cube, a multiple dimensions array of integers,
we need to convert this jam of lists into a Tensor object!.

Tensor Arraymancer

import arraymancer

let mytensor = [
                 [
                   [1, 2, 3],
                   [4, 5, 6],
                 ],
                 [
                   [7,   8,  9],
                   [10, 11, 12],
                 ],
               ].toTensor

Done, congrats you coded your first Tensor!.

Playing with Tensors

What can I do with it?.

echo mytensor is Tensor  # Is really a Tensor type?
echo sizeOf(mytensor)    # Whats the Size of it?
echo mytensor.rank       # Whats the Rank? 
echo mytensor.shape      # Whats the Shape of my Tensor?
echo mytensor.strides    # Strides?
echo mytensor.offset     # Offset? (if any)

Human readable representation

Print a human readable representation using:

echo mytensor

This should print on the standard output or REPL:

Tensor[int] of shape [2, 2, 3] of type "int" on backend "CPU":
| |     1       2       3 |     7       8       9 |
| |     4       5       6 |     10      11      12|

On this basic example I am using CPU as Backend, for compatibility,
yours can also say CUDA or OpenCL or OpenMP (if you have the hardware),
because GPUs are comfy massaging numbers.

Tensor Slicing

You can read or write a value from the Tensor:

(Comments added just to explain the indexes, but not required)

import arraymancer

var mytensor = [
    [1, 2, 3, 4, 5] # 0
    #0  1  2  3  4
  ].toTensor

echo mytensor[0, 2]  # Read the value "3"
mytensor[0, 2] = 42  # Write the value "42"

Remember that stuff declared with var is mutable, let is immutable,
a Tensor can also be Compile-Time pre-computed declaring it with const.

We can overwrite that Scalar value, lets see if we can get the whole Column?.

(Comments added to explain indexes, not required, we will take it away soon)

import arraymancer

var mytensor = [
    [1,   2,  3,  4,  5],   # 0
    [6,   7,  8,  9, 10],   # 1
    [11, 12, 13, 14, 15],   # 2
    [16, 17, 18, 19, 20],   # 3
    [21, 22, 23, 24, 25],   # 4
    # 0   1   2   3   4
  ].toTensor

echo mytensor[_, 2]

The only change is the _ on the index to select the whole Column.

Lets try get an entire Row now, without comments this time...

import arraymancer

var mytensor = [
    [1,   2,  3,  4,  5], 
    [6,   7,  8,  9, 10],  
    [11, 12, 13, 14, 15],  
    [16, 17, 18, 19, 20], 
    [21, 22, 23, 24, 25],  
  ].toTensor

echo mytensor[3, _]

I hope was not too complicated, just the _ for the Row.

Now lets grab just the center without touching the rest of it.

import arraymancer

var mytensor = [
    [1,   2,  3,  4,  5],  
    [6,   7,  8,  9, 10],  
    [11, 12, 13, 14, 15],  
    [16, 17, 18, 19, 20],  
    [21, 22, 23, 24, 25],  
  ].toTensor

echo mytensor[2, 2]

Going 3D

Lets chop off a piece of this pie...

import arraymancer

var mytensor = [
    [
      [ 1,  2,  3,  4,  5],      # 1st Matrix.
      [ 6,  7,  8,  9, 10],
      [11, 12, 13, 14, 15],
      [16, 17, 18, 19, 20],
      [21, 22, 23, 24, 25],
    ],
    [
      [26, 27, 28, 29, 30],      # 2nd Matrix.
      [31, 32, 33, 34, 35],
      [36, 37, 38, 39, 40],
      [41, 42, 43, 44, 45],
      [46, 47, 48, 49, 50],
    ],
    [
      [51, 52, 53, 54, 55],      # 3rd Matrix.
      [56, 57, 58, 59, 60],
      [61, 62, 63, 64, 65],
      [66, 67, 68, 69, 70],
      [71, 72, 73, 74, 75],
    ],
    [
      [76, 77, 78, 79,  80],     # 4th Matrix.
      [81, 82, 83, 84,  85],
      [86, 87, 88, 89,  90],
      [91, 92, 93, 94,  95],
      [96, 97, 98, 99, 100],
    ],
    [
      [101, 102, 103, 104, 105], # 5th Matrix.
      [106, 107, 108, 109, 110],
      [111, 112, 113, 114, 115],
      [116, 117, 118, 119, 120],
      [121, 122, 123, 124, 125],
    ]
  ].toTensor

echo mytensor[3, _, _]

What if we want the corner slice?.

import arraymancer

var mytensor = [
    [
      [ 1,  2,  3,  4,  5],      # 1st Matrix.
      [ 6,  7,  8,  9, 10],
      [11, 12, 13, 14, 15],
      [16, 17, 18, 19, 20],
      [21, 22, 23, 24, 25],
    ],
    [
      [26, 27, 28, 29, 30],      # 2nd Matrix.
      [31, 32, 33, 34, 35],
      [36, 37, 38, 39, 40],
      [41, 42, 43, 44, 45],
      [46, 47, 48, 49, 50],
    ],
    [
      [51, 52, 53, 54, 55],      # 3rd Matrix.
      [56, 57, 58, 59, 60],
      [61, 62, 63, 64, 65],
      [66, 67, 68, 69, 70],
      [71, 72, 73, 74, 75],
    ],
    [
      [76, 77, 78, 79,  80],     # 4th Matrix.
      [81, 82, 83, 84,  85],
      [86, 87, 88, 89,  90],
      [91, 92, 93, 94,  95],
      [96, 97, 98, 99, 100],
    ],
    [
      [101, 102, 103, 104, 105], # 5th Matrix.
      [106, 107, 108, 109, 110],
      [111, 112, 113, 114, 115],
      [116, 117, 118, 119, 120],
      [121, 122, 123, 124, 125],
    ]
  ].toTensor

echo mytensor[_, 1, 4]

A last one slicing...

import arraymancer

var mytensor = [
    [
      [ 1,  2,  3,  4,  5],      # 1st Matrix.
      [ 6,  7,  8,  9, 10],
      [11, 12, 13, 14, 15],
      [16, 17, 18, 19, 20],
      [21, 22, 23, 24, 25],
    ],
    [
      [26, 27, 28, 29, 30],      # 2nd Matrix.
      [31, 32, 33, 34, 35],
      [36, 37, 38, 39, 40],
      [41, 42, 43, 44, 45],
      [46, 47, 48, 49, 50],
    ],
    [
      [51, 52, 53, 54, 55],      # 3rd Matrix.
      [56, 57, 58, 59, 60],
      [61, 62, 63, 64, 65],
      [66, 67, 68, 69, 70],
      [71, 72, 73, 74, 75],
    ],
    [
      [76, 77, 78, 79,  80],     # 4th Matrix.
      [81, 82, 83, 84,  85],
      [86, 87, 88, 89,  90],
      [91, 92, 93, 94,  95],
      [96, 97, 98, 99, 100],
    ],
    [
      [101, 102, 103, 104, 105], # 5th Matrix.
      [106, 107, 108, 109, 110],
      [111, 112, 113, 114, 115],
      [116, 117, 118, 119, 120],
      [121, 122, 123, 124, 125],
    ]
  ].toTensor

echo mytensor[2, 0, 2]

Tensor Exercises

To reduce repetition lets assume this base Tensor:

import arraymancer

var mytensor = [
    [1, 2],
    [3, 4],
  ].toTensor

Tensor Addition

echo mytensor + mytensor

Tensor Substraction

echo mytensor - mytensor

Multiplication is left to the reader as exercise.

Tensor Iterator

# Values only.
for value in mytensor:
  echo value   

# Coordinates and Values.
for coordinates, value in mytensor:
  echo coordinates
  echo value

Iteration of pairs and items is left to the reader as exercise.

Tensor Broadcasting

# Broadcasting with operations are made beginning with a dot "."
echo mytensor .+ mytensor    # Addition
echo mytensor .- mytensor    # Substraction

Broadcasting other operations is left to the reader as exercise.

Tensor from list

# Tensor from vanilla std lib sequence.
let mytensor = toSeq(1..24).toTensor.reshape(2, 3, 4)

Try reshaping your Tensors to new shapes!

Why Arraymancer?

C Speed.
Python like syntax, Numpy like ergonomics.
Accelerated with Intel MKL/OpenBLAS/CUDA/OpenCL/OpenMP.
Dependency free distribution.
1 Language for the whole stack from experimental quick prototypes to Performance critical Production.

This is just the beginning, is not the end

This is barely the building block, but you can learn more:

It really wont take more than a nimble install arraymancer to get started on CPU!.
👑

DEV Community