The Geometry of Linear Equations

The fundamental problem of Linear Algebra is to solve a system of linear equations.

  1. Row Picture
    Picture of one equation at a time. An example is where 2x2 equation where the lines meet.

  2. *Column picture
    Star next to it because it's such an important one. Column at a time.

  3. Matrix form
    Look at the problem in matrix form and calling that matrix A

Example

Say we have two equations, two unknowns:

\begin{equation*} 2x - y = 0 \\ -x + 2y = 3 \end{equation*}



\begin{equation} \begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix} \end{equation}

This is our coefficient matrix A.

\begin{equation} \begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix} \begin{bmatrix} x \\ y \end{bmatrix} \end{equation}

The right hand side (RHS) is our unknown. 2 unknowns which is creates a vector, X, with two components, x and y.

\begin{equation} \begin{bmatrix} 2 & -1 \\ -1 & 2 \end{bmatrix} \begin{bmatrix} x \ y

\end{bmatrix}

\begin{bmatrix} 0 \\ 3 \end{bmatrix}

\end{equation}

And we have two right hand sides that also go into a vector b, 0 and 3.

\begin{equation} A X = b \end{equation}

So our linear equation is Ax = b.

Row Picture

Plot all the points that satisfy the first equation \begin{equation} 2x - y = 0 \end{equation} or y = -2x

and the second equation \begin{equation}-x + 2y = 3\end{equation} or y = (3+x)/2

Now let's grapth these lines (just plug in when x=0 to find y value and vice-versa)

In [131]:
import matplotlib.pyplot as plt
%matplotlib inline
x = [i for i in range(-1,3)]
y1 = [2*i for i in x]
y2 = [(3+i)/2 for i in x]

plt.plot(x,y1)
plt.plot(x,y2)
plt.legend(['First eqtn', 'Second eqtn'], loc=0)

plt.axhline(y=0, color='k')
plt.axvline(x=0, color='k')
plt.grid(b=True, which='major')
plt.plot(1,2, marker='o', markersize=10, color="green")
Out[131]:
[<matplotlib.lines.Line2D at 0x112922d30>]

They intercept at (1,2) which solves both equations.

\begin{equation*} x = 1, y = 2 \end{equation*}

Personal note: it seems like you would have to literally visualize/draw out the row picture method (or brute force?) to figure out the point they meet at. Not exactly efficient and impossible past 3d.

Column Picture

Now onto the more important point.

We want to combine certain amounts of [2 -1] and [-1 2] to get [0 3]

\begin{equation} x \begin{bmatrix} 2 \\ -1 \end{bmatrix} + y \begin{bmatrix} -1 \\ 2 \end{bmatrix} = \begin{bmatrix} 0 \\ 3 \end{bmatrix} \end{equation}

It's asking to find the right __**linear combination**__ of the columns

**Most fundamental operation in the whole course**

The linear combination of the columns. Multiply them by some numbers (x and y) and then add them to produce the [0 3]

And we already know x = 1 and y = 2

How does this look?

\begin{equation} 1 \begin{bmatrix} 2 \\ -1 \end{bmatrix} + 2 \begin{bmatrix} -1 \\ 2 \end{bmatrix} = \begin{bmatrix} 0 \\ 3 \end{bmatrix} \end{equation}\begin{equation} V1 = \begin{bmatrix} -1 \\ 2 \end{bmatrix} V2 = \begin{bmatrix} 2 \\ -1 \end{bmatrix} \end{equation}
In [77]:
plt.quiver([0, 0], [0, 0], [2, -1], [-1, 2], angles='xy', scale_units='xy', color=['b','r'], scale=1)
plt.axhline(y=0, color='k')
plt.axvline(x=0, color='k')
plt.grid(b=True, which='major')
plt.plot([0], [0], color = "red", label="V1")
plt.plot([0], [0], color = "blue", label="V2")
plt.xlim(-4, 4)
plt.ylim(-4, 4)
plt.legend()
plt.show()


plt.quiver([0,2,1], [0,-1,1], [2, -1,-1], [-1, 2,2], angles='xy', linestyle = '-', scale_units='xy', color=['b','r','r'], scale=1)
plt.axhline(y=0, color='k')
plt.axvline(x=0, color='k')
plt.grid(b=True, which='major')
plt.plot([0], [0], color = "red", label="V1")
plt.plot([0], [0], color = "blue", label="V2")
plt.plot([0], [0], color = "green", label="B")
plt.xlim(-4, 4)
plt.ylim(-4, 4)
plt.legend()
plt.plot(0,3, marker='o', markersize=10, color="green")
plt.show()

You're taking V2 and attaching 2 of V1's to get to B.

https://math.stackexchange.com/questions/2121717/linear-algebra-understanding-the-column-picture
Above link is a good read on why I couldn't grasp this idea intuitively. It's as if V1 and V2 have become our new x and y axis (conventional [0,1] and [1,0] in 2d space.

Big Question

What if you took all the x's and all the y's, what would be all the results?

Answer

You can get all the points that would fill the entire plane

3x3 example

15:41

3 equations, 3 unknowns

\begin{equation} 2x - y = 0 \\ -x + 2y - z = -1 \\ -3y + 4z = 4 \end{equation}

We want to understand the equation and then solve them. Row picture is one. Column picture is another VERY important way.

Matrix form

16:40

\begin{equation} A = \begin{bmatrix} 2 & -1 & 0 \\ -1 & 2 & -1 \\ 0 & -3 & 4 \end{bmatrix} x = \begin{bmatrix} x \\ y \\ z \end{bmatrix} b = \begin{bmatrix} 0 \\ -1 \\ 4 \end{bmatrix} \end{equation}

A is 3x3 matrix.

Row picture

17:32

For the second equation: You can set

x= 1, while y anx z are 0.

z = 1, x = 0, y = 0

x = 0, y = 0.5, z = 0

Second equation creates a plane.

Note: For some reason, the professor keeeps saying the 1st and 2nd equation creates a plane in 3d space. I don't think this specific example applies....

Any two planes that intercept creates a line of where they intersect.

And 3 planes will meet at a single point since they are not parallel or special.

That single point is the solution.

It shows that it's hard to use the row example because it's hard to visualize. Obviously it's impossible with the 4+ dimensions.

In [8]:
# inspiration from https://stackoverflow.com/questions/19410733/how-to-draw-planes-from-a-set-of-linear-equations-in-python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

point2  = np.array([-1,0,0]) # If a is not 0 a point on the plane is (-d/a, 0, 0)
normal2 = np.array([-1,2,-1])

point3  = np.array([0,0,1]) # If c is not 0 a point on the plane is (0, 0, -d/c)
normal3 = np.array([0,-3,4])

# a plane is a*x+b*y+c*z+d=0
# [a,b,c] is the normal. Thus, we have to calculate
# d and we're set
#d1 = -np.sum(point1*normal1)# dot product
d2 = -np.sum(point2*normal2)# dot product
d3 = -np.sum(point3*normal3)# dot product

# create x,y
xx, yy = np.meshgrid(range(30), range(30))

# calculate corresponding z
#z1 = (-normal1[0]*xx - normal1[1]*yy - d1)*1./normal1[2]
z2 = (-normal2[0]*xx - normal2[1]*yy - d2)*1./normal2[2]
z3 = (-normal3[0]*xx - normal3[1]*yy - d3)*1./normal3[2]

# plot the surface
plt3d = plt.figure().gca(projection='3d')
#plt3d.plot_surface(xx,yy,z1, color='blue')

x = [i for i in range(-5,30)]
y1 = [2*i for i in x]

point2 = np.array([0, 0, 1])
plt3d.scatter(point2[0] , point2[1] , point2[2],  color='black')

plt3d.plot(x,y1)

plt3d.plot_surface(xx,yy,z2, color='red')
plt3d.plot_surface(xx,yy,z3, color='yellow')
plt.show()

Column Picture

22:00

Next lecture will talk about using elimination to find the x,y,z.

In [9]:
import numpy as np
a = np.array([[2, -1, 0], [-1, 2, -1], [0,-3,4]])
b = np.array([0,-1,4])
x = np.linalg.solve(a, b)
print (x) # [ -2.17647059  53.54411765  56.63235294]

[ 0. -0.  1.]
In [76]:
%matplotlib notebook
%matplotlib inline
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure()
ax = fig.gca(projection='3d')

x,y,z = 0,0,0

solution = [0, -1, 4]

ax.quiver(x, y, z, 2, -1, 0, color = 'red')
ax.quiver(x, y, z, -1, 2, -3, color = 'blue')
ax.quiver(x, y, z, 0, -1, 4, color = 'green')
ax.quiver(x, y, z, solution[0], solution[1], solution[2], color = 'purple')


axisRange = 5
axis = [i for i in range(0,axisRange)]
zeroAxis = [0]*axisRange

plt.plot(axis, zeroAxis, zeroAxis, color = 'black')
plt.plot(zeroAxis, axis, zeroAxis, color = 'black')
plt.plot(zeroAxis, zeroAxis, axis, color = 'black')


axisLimits = (-1,3)

ax.set_xlim3d(axisLimits)
ax.set_ylim3d(axisLimits)
ax.set_zlim3d(axisLimits)

plt.show()

The answer isn't easy to see in the column picture. Next lecture, we will talk about elimination. A systematic that everybody and software (large scale) would save that equation.

Big Picture

26:12

Let's change b by adding the first two columns.

\begin{equation} b = \begin{bmatrix} 1 \\ 1 \\ -3 \end{bmatrix} \end{equation}

So now

\begin{equation*} x = 1, y = 1, z =0 \end{equation*}
In [ ]:
#%matplotlib notebook
%matplotlib inline
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure()
ax = fig.gca(projection='3d')

x, y, z = 0,0,0
# Make the direction data for the arrows
u,v,w = 5,5,5

ax.quiver(x, y, z, u, v, w, color = 'red')

axisRange = 5
axis = [i for i in range(0,axisRange)]
zeroAxis = [0]*axisRange

plt.plot(axis, zeroAxis, zeroAxis, color = 'black')
plt.plot(zeroAxis, axis, zeroAxis, color = 'black')
plt.plot(zeroAxis, zeroAxis, axis, color = 'black')

ax.set_xlim3d(0, 10)
ax.set_ylim3d(0, 10)
ax.set_zlim3d(0, 10)

plt.show()

Now think about all the RHS. Can you solve for all RHS?

Can I solve Ax = b for every b? Is there a solution?

If there is, elimination will find that.

In Linear Combinations words...

28:40

Do the linear combination of the columns fill the 3-D space? Every b means all the b's in 3-D space

When you multiply a matrix with a vector, you get a combination of all the columns.

For this matrix, the answer is yes. A non-singular matrix, An invertible matrix. Those are the ones ones we like best.

There are ones that aren't.

An issue is when the 3 columns of the matrix ALL lie in the same plane. If they do, then all their combinations lie on the same plane. You don't get anything new. You can solve it for some RHS, but not all. Most would be out of the plane, thus unreachable.

9 dimension

32:30

Imagine dealing with colmns with 9 components.

Are they independent?

If the 9 columns are chosen such that they're not independent, such as the 8th column being the same as the 9th column. Then it would not contribute anything new and there would RHS b's that you couldn't get. The 9 columns can fill the 9D space.

However if the 9th column is the same as the 8th column and gave nothing new, then what it would fill out would be.....

It would be like a 8D plane inside 9D space.

Multiply a matrix with a vector

35:30

How do you multiple a matrix by a vector?

Say we have:

\begin{equation} A x = b \end{equation}\begin{equation} \begin{bmatrix} 2 & 5 \\ 1 & 3 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} \end{equation}

2 ways to do it

In [96]:
import numpy as np
a = np.array([[ 2, 5], [ 1,3]])
b = np.array([1, 2])

print (a.dot(b))

[12  7]

Column At a time

36:51

\begin{equation} \begin{bmatrix} 2 & 5 \\ 1 & 3 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} = 1 \begin{bmatrix} 2 \\ 1 \end{bmatrix} + 2 \begin{bmatrix} 5 \\ 3 \end{bmatrix} = \begin{bmatrix} 12 \\ 7 \end{bmatrix} \end{equation}

1 of the 1st column and 2 of the 2nd column.

Ax is a linear combination of columns of A.

Row at a time

Dot product: Each row multiply with x.

\begin{equation} \begin{bmatrix} 2 & 5 \\ 1 & 3 \end{bmatrix} \begin{bmatrix} 1 \\ 2 \end{bmatrix} = \begin{bmatrix} 2 \times 1 + 5 \times 2 \\ 1 \times 1 + 3 \times 2 \end{bmatrix} = \begin{bmatrix} 12 \\ 7 \end{bmatrix} \end{equation}
In [78]:
#%matplotlib notebook
%matplotlib inline
from mpl_toolkits.mplot3d import axes3d
import matplotlib.pyplot as plt
import numpy as np

fig = plt.figure()
ax = fig.gca(projection='3d')

x, y, z = 0,0,0
# Make the direction data for the arrows
u,v,w = 5,5,5

ax.quiver(x, y, z, u, v, w, color = 'red')

axisRange = 5
axis = [i for i in range(0,axisRange)]
zeroAxis = [0]*axisRange

plt.plot(axis, zeroAxis, zeroAxis, color = 'black')
plt.plot(zeroAxis, axis, zeroAxis, color = 'black')
plt.plot(zeroAxis, zeroAxis, axis, color = 'black')

ax.set_xlim3d(0, 10)
ax.set_ylim3d(0, 10)
ax.set_zlim3d(0, 10)

plt.show()
In [ ]: