Linear Algebra

Linear algebra is the branch of mathematics that studies vector spaces and linear transformations between them like adding and scaling. It uses matrixes to represent linear maps. Linear algebra is widely used in science and engineering.

\begin{pmatrix} 1 & 3 \\ 5 & 0 \end{pmatrix}

\[ \begin{pmatrix} 1 & 3 \\ 5 & 0 \end{pmatrix} \]

In the Compute Engine matrices are represented as lists of lists.

For example the matrix above is represented as the following list of lists:

["List", ["List", 1, 3, ["List", 5, 0]]]

An axis is a dimension of a tensor. A vector has one axis, a matrix has two axes, a tensor (multi-dimensional matrix) has more than two axes.

The shape of a tensor is the length of each of its axes. For example, a matrix with 3 rows and 4 columns has a shape of (3, 4).

The rank of a tensor is the number of axes of the tensor. For example, a matrix has a rank of 2. Note that rank is sometimes used with a different meaning. For example, the rank of a matrix can also be defined as the maximum number of linearly independent rows or columns of the matrix. In the Compute Engine, rank is always used with the meaning of the number of axes.

In the Compute Engine, matrices are stored in row-major order. This means that the first element of the outer axis is the first row of the matrix, the second element of the list is the second row of the matrix, etc.

Since matrixes are List collections, some collection operations can also be applied to them such as At, Fold and Map.

An extension of linear algebra is tensor algebra which studies tensors, which are multidimensional arrays.

For example, a grayscale image can be represented by a matrix of grayscale values. But a color image is represented by a rank 3 tensor, an array of RGB triplets. Tensors are also represented as nested lists.

The Compute Engine provides a number of functions for working with matrices.

Representing Matrices

Vectors (row vectors) are represented as lists, that is an expression with the head List.

Matrixes are represented as lists of lists.

Tensors (multi-dimensional matrixes) are represented as nested lists.

Tensors are represented internally using an optimized format that is more efficient than nested lists. Because of this, some operations on tensors such as Reshape and Transpose can be done in O(1) time.

Vector is a convenience function that interprets a list of elements as a column vector.

Matrix is an optional “tag” inert function that is used to influence the visual representation of a matrix. It has not impact on the value of the matrix.

In LaTeX notation, a matrix is represented with “environments” (with command \begin and \end) such as pmatrix or bmatrix.:

\begin{pmatrix} 1 & 3 \\ 5 & 0 \end{pmatrix}

\[ \begin{pmatrix} 1 & 3 \\ 5 & 0 \end{pmatrix} \]

\begin{bmatrix} 1 & 3 \\ 5 & 0 \end{bmatrix}

\[ \begin{bmatrix} 1 & 3 \\ 5 & 0 \end{bmatrix} \]

In LaTeX, each column is separated by an & and each row is separated by \\.

["Vector", x-1, …x-2]

Vector interprets the elements x-1… as a column vector

This is essentially a shortcut for ["Matrix", ["List", ["List", _x-1_], ["List, _x-2_], ...]]].

["Vector", 1, 3, 5, 0]

\begin{pmatrix} 1 \\ 3 \\ 5 \\ 0 \end{pmatrix}

\[ \begin{pmatrix} 1 \\ 3 \\ 5 \\ 0 \end{pmatrix} \]

A row vector can be represented with a simple list or a tuple.

["List", 1, 3, 5, 0]

\begin{bmatrix} 1 & 3 & 5 & 0 \end{bmatrix}

\[ \begin{bmatrix} 1 & 3 & 5 & 0 \end{bmatrix} \]

["Matrix", matrix]

["Matrix", matrix, delimiters, columns-format]

Matrix is an inert function: its value is the value of its first argument. It influences the visual representation of a matrix.

matrix is a matrix represented by a list of rows. Each row is represented by a list of elements.

delimiters is an optional string of two characters. The first character represent the opening delimiter and the second character represents the closing delimiter.

The delimiters can be any of the following characters:

(, ), [, ], {, }, <, >
⟦ (U+27E6), ⟧ (U+27E7)
|, ‖ (U+2016)
\\
⌈ (U+2308), ⌉ (U+2309), ⌊ (U+230A), ⌋ (U+230B)
⌜ (U+231C), ⌝ (U+231D), ⌞ (U+231E), ⌟ (U+231F)
⎰ (U+23B0), ⎱ (U+23B1).

In addition, the character . can be used to indicate no delimiter.

Some commom combinations may be represented using some standard LaTeX environments:

Delimiters	LaTeX Environment	Example
`()`	`pmatrix`	\[ \begin{pmatrix} a & b \\ c & d \end{pmatrix} \]
`[]`	`bmatrix`	\[ \begin{bmatrix} a & b \\ c & d \end{bmatrix} \]
`{}`	`Bmatrix`	\[ \begin{Bmatrix} a & b \\ c & d \end{Bmatrix} \]
`\|\|`	`vmatrix`	\[ \begin{vmatrix} a & b \\ c & d \end{vmatrix} \]
`‖‖`	`Vmatrix`	\[ \begin{Vmatrix} a & b \\ c & d \end{Vmatrix} \]
`{.`	`dcases`	\[ \begin{dcases} a & b \\ c & d \end{dcases} \]
`.}`	`rcases`	\[ \begin{rcases} a & b \\ c & d \end{rcases} \]

columns_format is an optional string indicating the format of each column. A character = indicates a centered column, < indicates a left-aligned column, and > indicates a right-aligned column.

A character of | indicate a solid line between two columns and : indicate a dashed lines between two columns.

Matrix Properties

["Shape", matrix]

Returns the shape of a matrix, a tuple of the lengths of the matrix along each of its axis.

A list (or vector) has a single axis. A matrix has two axes. A tensor has more than two axes.

For a scalar, Shape returns an empty tuple.

["Shape", 5]
// ➔ ["Tuple"]
 
["Shape", ["List", 5, 2, 10, 18]]
// ➔ ["Tuple", 4]
 
["Shape", ["List", ["List", 5, 2, 10, 18], ["List", 1, 2, 3]]]
// ➔ ["Tuple", 2, 4]

Note: The shape of a matrix is also sometimes called “dimensions”. However, this terminology is ambiguous because the word “dimension” is also used to refer to the length of a matrix along a specific axis.

["Rank", matrix]

Returns the number of axes of a matrix.

A scalar (a number, for example) has rank 0.

A vector or list has rank 1.

A matrix has rank 2, a tensor has rank 3, etc.

The rank is the length of the shape of the tensor.

["Rank", 5]
// ➔ 0
 
["Rank", ["List", 5, 2, 10, 18]]
// ➔ 1
 
["Rank", ["List", ["List", 5, 2, 10], ["List", 1, 2, 3]]]
// ➔ 2

Accessing the content of Tensors

["At", matrix, index-1, index-2, …]

Returns the element of the matrix at the specified indexes.

index-1, … is a sequence of integers, one for each axis of the matrix.

Indexes start at 1. Negative indexes count elements from the end. A negative index is equivalent to adding the length of the axis to the index. So -1 is the last element of the axis, -2 is the second to last element, etc.

["At", ["List", 5, 2, 10, 18], 3]
// ➔ 10
 
["At", ["List", ["List", 5, 2, 10, 18], ["List", 1, 2, 3]], 2, 3]
// ➔ 3
 
["At", ["List", ["List", 5, 2, 10, 18], ["List", 1, 2, 3]], 2, -1]
// ➔ 3

In a list (or vector), there is only one axis, so there is only one index.

In a matrix, the first index is the row, the second is the column.

In LaTeX, accessing the element of a matrix is done with a subscript or square brackets following a matrix.

\mathbf{A}_{2,3}

\[ \mathbf{A}_{2,3} \]

\mathbf{A}\lbrack2,3\rbrack

\[ \mathbf{A}\lbrack2,3\rbrack \]

Transforming Matrixes

["Flatten", matrix]

Returns a list of all the elements of the matrix, recursively, in row-major order.

This function can also be applied to any collection.

Only elements with the same head as the collection are flattened. Matrixes have a head of List, so only other List elements are flattened.

["Flatten", ["List", ["List", 5, 2, 10, 18], ["List", 1, 2, 3]]]
// ➔ ["List", 5, 2, 10, 18, 1, 2, 3]

Flatten is similar to the APL , Ravel operator or numpy.ravel Numpy.

["Reshape", matrix, shape]

Returns a matrix with the specified shape.

matrix can be a list, a matrix, a tensor or a collection.

shape is a tuple of integers, one for each axis of the result matrix.

Reshape can be used to convert a list or collection into a matrix.

["Reshape", ["Range", 9], ["Tuple", 3, 3]]
// ➔ ["List", ["List", 1, 2, 3], ["List", 4, 5, 6], ["List", 7, 8, 9]]

This is similar to the APL ⍴ Reshape operator or numpy.reshape Numpy.

The result may have fewer or more elements than the original tensor.

When reshaping, the elements are taken from the original tensor in row-major order, that is the order of elements as returned by Flatten.

If the result has fewer elements, the elements are dropped from the end of the element list. If the result has more elements, the lists of elements is filled cyclically.

This is the same behavior as APL, but other environment may behave differently. For example, by default Mathematica ArrayReshape will fill the missing elements with zeros.

["Transpose", matrix]

Returns the transpose of the matrix.

\mathbf{A}^T

\[ \mathbf{A}^T \]

["Transpose", ["List", ["List", 1, 2, 3], ["List", 4, 5, 6]]]
// ➔ ["List", ["List", 1, 4], ["List", 2, 5], ["List", 3, 6]]

["Transpose", tensor, axis-1, axis-2]

Swap the two specified axes of the tensor. Note that axis indexes start at 1.

["ConjugateTranspose", matrix]

A^\star

\[ A^\star \]

Returns the conjugate transpose of the matrix, that is the transpose of the matrix with all its (complex) elements conjugated. Also known as the Hermitian transpose.

["ConjugateTranspose", ["List", ["List", 1, 2, 3], ["List", 4, 5, 6]]]
// ➔ ["List", ["List", 1, 4], ["List", 2, 5], ["List", 3, 6]]

["ConjugateTranspose", matrix, axis-1, axis-2]

Swap the two specified axes of the matrix. Note that axis indexes start at 1. In addition, all the (complex) elements of the tensor are conjugated.

["Inverse", matrix]

Returns the inverse of the matrix.

\mathbf{A}^{-1}

\[ \mathbf{A}^{-1} \]

["Inverse", ["List", ["List", 1, 2], ["List", 3, 4]]]
// ➔ ["List", ["List", -2, 1], ["List", 1.5, -0.5]]

["PseudoInverse", matrix]

\mathbf{A}^+

\[ \mathbf{A}^+ \]

Returns the Moore-Penrose pseudoinverse of the matrix.

["PseudoInverse", ["List", ["List", 1, 2], ["List", 3, 4]]]
// ➔ ["List", ["List", -2, 1], ["List", 1.5, -0.5]]

["Diagonal", matrix]

Returns the diagonal of the matrix, that is the list of all the elements on the diagonal of the matrix.

["Diagonal", ["List", ["List", 1, 2], ["List", 3, 4]]]
// ➔ ["List", 1, 4]

Calculating with Matrixes

["Determinant", matrix]

Returns the determinant of the matrix.

["Determinant", ["List", ["List", 1, 2], ["List", 3, 4]]]
// ➔ -2

["AdjugateMatrix", matrix]

\operatorname{adj}(\mathbf{A})

\[ \operatorname{adj}(\mathbf{A}) \]

Returns the adjugate matrix of the input matrix, that is the inverse of the cofactor matrix.

The cofactor matrix is a matrix of the determinants of the minors of the matrix multiplied by \( (-1)^{i+j} \). That is, for each element of the matrix, the cofactor is the determinant of the matrix without the row and column of the element.

["AdjugateMatrix", ["List", ["List", 1, 2], ["List", 3, 4]]]
// ➔ ["List", ["List", 4, -2], ["List", -3, 1]]

["Trace", matrix]

\operatorname{tr}(\mathbf{A})

\[ \operatorname{tr}(\mathbf{A}) \]

Returns the trace of the matrix, the sum of the elements on the diagonal of the matrix. The trace is only defined for square matrices. The trace is also the sum of the eigenvalues of the matrix.

["Trace", ["List", ["List", 1, 2], ["List", 3, 4]]]
// ➔ 5