sympyimport numpy as np
from sympy import * # Import all functions form the module.
# Bokeh for interactive plots
from bokeh.plotting import figure, output_notebook, show
from bokeh.layouts import row, column
from bokeh.models import Span
output_notebook()
def plot(w=700,h=500,title='',x='X',y='f(X)'):
'''Wrapper function to ease starting a new plot.
'''
p = figure(plot_width=w,plot_height=h,title=title,toolbar_location="below")
p.xaxis.axis_label = x
p.yaxis.axis_label = y
return p
Calculating the instantaneous rate of change in a function.
$$ \frac{d f(x)}{d x} = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h} $$
In the other notebook accompanying today's lecture, we developed a simple function that could approximate the derivative for us by taking small changes. We saw that this performed pretty well at accomplishing the basic aims of calculating the derivative.
def deriv(x,func,nudge=.001):
'''
More generic derivative function
'''
return (func(x + nudge) - func(x))/nudge
Let's use this function to "double check" the differentiation rules outlined below.
def f(x):
return 3
x = np.arange(-5,5.1,.1)
p = plot(700,300)
p.line(x,f(x),line_width=3,legend='f(x)=3')
p.line(x,deriv(x,f),line_width=3,
line_dash='dashdot',legend="f'(x)=0",
color="red",alpha=.5)
show(p)
$$\frac{d f(cx)}{d x} $$
$$\lim_{h \to 0} \frac{c(f(x+h)) - c(f(x))}{h}$$
$$ \lim_{h \to 0} c(\frac{f(x+h) - f(x)}{h}) $$
$$ f'(cx) = cf'(x) $$
where $c$ is some arbitrary constant.
def f1(x):
return 3*np.sin(x)
def f2(x):
return 1*np.sin(x)
def f3(x):
return .5*np.sin(x)
x = np.arange(-5,5.1,.1)
p = plot(700,300)
p.line(x,deriv(x,f1),line_width=3,
line_dash='dashdot',legend="f'(x)=3cos(x)",
color="red",alpha=.5)
p.line(x,deriv(x,f2),line_width=3,
line_dash='dashdot',legend="f'(x)=1cos(x)",
color="skyblue",alpha=.5)
p.line(x,deriv(x,f3),line_width=3,
line_dash='dashdot',legend="f'(x)=.5cos(x)",
color="forestgreen",alpha=.5)
show(p)
Let's apply the above to more than one function.
$$ \frac{d(f + g)}{d x} $$
$$\lim_{h \to 0} \frac{(f(x+h) + g(x+h) ) - (f(x) + g(x))}{h}$$
$$ \lim_{h \to 0} \frac{(f(x+h) - f(x)) + (g(x+h) - g(x))}{h}$$
$$ \lim_{h \to 0} (\frac{f(x+h) - f(x)}{h} + \frac{g(x+h) - g(x)}{h})$$
$$ f'(x) + g'(x) $$
Naturally, this holds as well for subtraction.
$$ \frac{d(f(x) - g(x))}{d x} = f'(x) - g'(x) $$
These linear operators should look familiar!
$$ f(x) = x^3 + 2x^2 - 3x$$
$$ \frac{df(x)}{dx} = \frac{d(x^3 + 2x^2 - 3x)}{dx}$$
$$ \frac{df(x)}{dx} = \frac{d(x^3)}{dx} + \frac{d(2x^2)}{dx} - \frac{d(3x)}{dx}$$
$$ \frac{df(x)}{dx} = \frac{d(x^3)}{dx} + 2\frac{d(x^2)}{dx} - 3\frac{d(x)}{dx}$$
$$ \frac{df(x)}{dx} = 3x^2 + 2(2x) - 3(1)$$
$$ \frac{df(x)}{dx} = 3x^2 + 4x - 3$$
Given the function is differentiable, we can differentiate each individual component of the function individually and then add the components together.
# Define our function and its components
def f(x):
return x**3 + 2*x**2 + 3*x
def f_comp1(x):
return x**3
def f_comp2(x):
return 2*x**2
def f_comp3(x):
return 3*x
# Specify a range of x
x = np.arange(-5,5+.1,.1)
# Plot each of the component Functions
p1 = plot(700,400,title="Each Function Individually")
p1.line(x,f_comp1(x),
color='black',line_width=2,alpha=.5,
legend ='f1 = x^3')
p1.line(x,f_comp2(x),
color='orange',line_width=2,alpha=.5,
legend ='f2 = 2x^2')
p1.line(x,f_comp3(x),
color='green',line_width=2,alpha=.5,
legend ='f2 = 3x')
p1.legend.location = 'top_center'
# Plot the combination of the component functions
p2 = plot(700,400,title="Combined Components")
for i in range(-5,5+1):
p2.line([i,i], # X-axis
[f_comp1(i),0], # Y-Axis
color='black',line_width=5,alpha=.4)
p2.line([i,i], # X-axis
[f_comp1(i)+f_comp2(i),f_comp1(i)], # Y-Axis
color='orange',line_width=5,alpha=.4)
p2.line([i,i], # X-axis
[f_comp1(i)+f_comp2(i)+f_comp3(i),f_comp1(i)+f_comp2(i)], # Y-Axis
color='green',line_width=5,alpha=.6)
p2.line(x,f(x),color='steelblue',line_width=3)
# Print the two functions side by side
show(column(p1,p2))
Likewise, this extends to the derivatives as we are are altering the rate of change when altering the form of the function.
# Derivatives of each or our component functions
def df(x):
return 3*x**2 + 4*x + 3
def df_comp1(x):
return 3*x**2
def df_comp2(x):
return 4*x
def df_comp3(x):
return 3
# Plot each of the component Functions
p1 = plot(700,400,title="The Derivative of each Function Individually")
p1.line(x,df_comp1(x),
color='black',line_width=2,alpha=.5,
legend="df1 = 3x^2")
p1.line(x,df_comp2(x),
color='orange',line_width=2,alpha=.5,
legend="df2 = 4x")
p1.line(x,df_comp3(x),
color='green',line_width=2,alpha=.5,
legend="df3 = 3")
p1.legend.location = 'top_center'
# Plot the combination of the component functions
p2 = plot(700,400,title="The Derivatives Combined")
for i in range(-5,5+1):
p2.line([i,i],[df_comp1(i),0],color='black',line_width=5,alpha=.4)
p2.line([i,i],[df_comp1(i)+df_comp2(i),df_comp1(i)],color='orange',line_width=5,alpha=.4)
p2.line([i,i],[df_comp1(i)+df_comp2(i)+df_comp3(i),df_comp1(i)+df_comp2(i)],color='green',line_width=5,alpha=.6)
p2.line(x,df(x),color='steelblue',line_width=3)
# Print the two functions side by side
show(column(p1,p2))
$$ f(x) =x^n $$
$$ f'(x) = nx^{n-1} $$
The underlying derivation of the power rule is best seen in its application to different polynomials. Let's take the derivative of three polynomials and see how the power rule emerges as a consistent pattern.
Example 1
Consider the function $f(x) = x^2$
$$ f'(x) = \lim_{h\to0}\frac{(x + h)^2 - x^2}{h} $$
$$ \frac{x^2 + 2 x (h) + h^2 - x^2}{h}$$
$$ \frac{2 x (h) + h^2}{h}$$
$$ \frac{h(2 x + h)}{h}$$
$$ 2 x + h $$
and as we shrink $h$ to zero (is taken to its limit), we end up with
$$2x$$
Example 2:
Consider the function $ f(x) = x^3 $
$$ f'(x) = \lim_{h \to 0} \frac{(x + h)^3 - x^3}{h}$$
$$\frac{(h)^3 - 3 (h)^2 x + 3(h)x^2}{h}$$
$$(h)^2 - 3 (h) x + 3x^2$$
$$3x^2$$
Example 3:
Consider the function $ f(x) = x^7 $
$$f'(x) = \lim_{h \to 0} \frac{(x + h)^7 - x^7}{h}$$
$$\frac{(h)^7 + 7(h)^6 x + 21 (h)^5 x^2 + 35 (h)^3 x^4 + 21 (h)^2 x^5 + 7(h) x^6}{h}$$
$$\frac{ h(h^6 + 7 h^5 x + 21 h^4 x^2 + 35 h^2 x^4 + 21 h x^5 + 7 x^6)}{h}$$
$$ h^6 + 7 h^5 x + 21 h^4 x^2 + 35 h^2 x^4 + 21 h x^5 + 7 x^6$$
Which ultimately reduces to as $h$ approaches 0.
$$ 7x^6 $$
The emerging pattern reveals a surprisingly simple rule, which is the power rule
$$ n x ^{n-1} $$
What is a square?
$$ x^2$$
$$ x \cdot x$$
$$ area = base \times height $$
What is the change in area when we increase the area of the square a little?
# Parameters
base = 2 # Base
height = 2 # Height
h = .1 # our slight change
# PLOT 1
p1 = plot(475,475,y='x',x='x')
# plot our square
p1.quad(top=[height], bottom=[0], left=[0],
right=[base], color="steelblue",
legend="Initial Area")
# Plot our square with a slight h
p1.quad(top=[height+h], bottom=[0], left=[0],
right=[base+h], color="gold",alpha=.2,
legend="Area + some increase (h)")
p1.legend.location = 'bottom_left'
# PLOT 2
# Lets color the areas being changes
p2 = plot(475,475,x='x',y='x')
# Change in the height as we adjust x.
p2.quad(top=[height + h], bottom=[height], left=[0],
right=[base], color="red",alpha=.2,
legend="x(h)")
# Change in the base as we adjust x.
p2.quad(top=[height], bottom=[0], left=[base],
right=[base + h], color="purple",alpha=.2,
legend="(h)x")
# Change in the base as we adjust x.
p2.quad(top=[height+h], bottom=[height], left=[base],
right=[base+h], color="gold",alpha=.2,
legend="h(h)")
p2.legend.location = 'bottom_left'
show(column(p1,p2))
What is the resulting area of the change?
$$ new~area = xh + xh + h^2 $$
$$ 2xh + h^2 $$
Divide $h$ out
$$ \frac{2xh + h^2}{h} $$
$$ \lim_{h \to 0} 2x + h $$
Again, as we shrink the size of $h$ (the change) down into smaller and smaller bits, we end up with
$$2x$$
Can you see how this would extend to any polynomial?
$$ F'(x) = \lim_{h \to 0} \frac{f(x+h)g(x+h) - f(x)g(x)}{h} $$
We'll need to perform a trick to manipulate this expression algebraically. We can do this by adding and subtracting the same value in the numerator. Note that this doesn't change anything in the expression, but it does allow us a way to manipulate it.
$$ \lim_{h \to 0} \frac{f(x+h)g(x+h) \textbf{- f(x)g(x+h) + f(x)g(x+h)} - f(x)g(x)}{h} $$
Regroup values and use what we know of addition
$$ \lim_{h \to 0} \frac{f(x+h)g(x+h) - f(x)g(x+h)}{h} + \lim_{h \to 0} \frac{f(x)g(x+h) - f(x)g(x)}{h} $$
Factor out values
$$ \lim_{h \to 0} g(x+h)\frac{f(x+h)-f(x)}{h} + \lim_{h \to 0} f(x) \frac{g(x+h)-g(x)}{h}$$
Take the limit of each component
$$ (\lim_{h \to 0} g(x+h)) (\lim_{h \to 0} \frac{f(x+h)-f(x)}{h}) + (\lim_{h \to 0} f(x)) (\lim_{h \to 0}\frac{g(x+h)-g(x)}{h})$$
$$ g(x)f'(x) + f(x)g'(x) $$
More commonly arranged as
$$ f'(x)g(x) + f(x)g'(x) $$
$$ f(x) = 2x + 3 $$
$$ g(x) = x^2 $$
$$ F(x) = f(x)g(x) $$
$$ F(x) = (2x + 3)(x^2) $$
$$ F'(x) = (2)(x^2) + (2x + 3)2x $$
$$ F'(x) = 2x^2 + 4x^2 + 6x $$
$$ F'(x) = 6x^2 + 6x $$
Let's consider the following functions:
$$ f(x) = \sin(x) $$
$$ g(x) = x^2 $$
What is the resulting product of these two functions as we nudge x incrementally? As we've seen, when dealing with products, it's sometimes useful to think in terms of the area when multiplied by one another.
def f(x):
return np.sin(x)
def g(x):
return x**2
# Parameters
base = 1 # Base
height = 1 # Height
area = base*height
h = .1 # our slight change
# PLOT 1
# plot our square
p1 = plot(700,475,x='x^2',y='sin(x)')
p1.quad(top=[f(height)], bottom=[0], left=[0],
right=[g(base)], color="steelblue",
legend='Initial Area')
# Plot our square with a slight nudge
p1.quad(top=[f(height + h)], bottom=[0], left=[0],
right=[g(base + h)], color="gold",alpha=.2,
legend='Initial Area + some increase (h)')
p1.legend.location = 'bottom_left'
# PLOT 2
# Lets color the areas being changes
p2 = plot(700,475,x='x^2',y='sin(x)')
# p2.quad(top=[f(height)], bottom=[0], left=[0],
# right=[g(base)], color="steelblue",alpha=.05,
# legend="x^2 sin(x)")
# Change in the height as we adjust x.
p2.quad(top=[f(height + h)], bottom=[f(height)], left=[0],
right=[g(base)], color="red",alpha=.2,
legend="x^2 sin(x+h)")
# Change in the base as we adjust x.
p2.quad(top=[f(height)], bottom=[0], left=[g(base)],
right=[g(base + h)], color="purple",alpha=.2,
legend="sin(x) (x+h)^2")
# Change in the base as we adjust x.
p2.quad(top=[f(height+h)], bottom=[f(height)], left=[g(base)],
right=[g(base+h)], color="gold",alpha=.2,
legend="(x+h)^2 (sin(x+h))")
p2.legend.location = 'bottom_left'
show(column(p1,p2))
Let's gather these pieces
$$ \frac{\sin(x+h)-sin(x)}{h} x^2 + \sin(x) \frac{(x+h)^2-x^2}{h} + \frac{\sin(x+h)-sin(x)}{h}\frac{(x+h)^2-x^2}{h} $$
We can essentially ignore $\sin(x+h)(x+h)^2$ as that area will shrink to a trivially small size as $h \to 0 $
This leaves us with
$$ \frac{\sin(x+h)-sin(x)}{h} x^2 + \sin(x) \frac{(x+h)^2-x^2}{h}$$
$$ \cos(x) x^2 + \sin(x) 2x $$
We are simply adding the respective area. This is the essence of the product rule.
$$ f'(x) g(x) + f(x) g'(x) $$
The chain rule is dictates how we deal with function compositions. That is, functions within functions.
$$ g: x \mapsto z $$
$$ f: z \mapsto y $$
$$ g \circ f: x \mapsto y $$
$$ f(g(x))= y $$
We saw something similar to this when we took multiple matrix transformations.
$$\textbf{A}\textbf{B}\vec{x} = \vec{y}$$
$$ F'(x) = \lim_{h \to 0}\frac{f(g(x+h)) - f(g(x))}{h} $$
Now let's multiply and divide by $g(x+h) - g(x)$. Recall that this doesn't change underlying value of the function as long as we multiply this value into both the numerator and the denominator.
$$ \lim_{h \to 0}\frac{f(g(x+h)) - f(g(x))}{h} \frac{g(x+h) - g(x)}{g(x+h) - g(x)} $$
$$ \lim_{h \to 0}\frac{f(g(x+h)) - f(g(x))}{g(x+h) - g(x)} \frac{g(x+h) - g(x)}{h} $$
Recall that $h$ is really $(x + h) - x$, that is, it's the difference. When we think in these terms we can see that we have two similar types of statements...
$$ \lim_{h \to 0}\frac{f(g(x+h)) - f(g(x))}{g(x+h) - g(x)}\lim_{h \to 0}\frac{g(x+h) - g(x)}{(x+h) - x} $$
The first product is taking the change in $f(\cdot)$ with respect to the change in $g(x)$ and the second takes the change in $g(\cdot)$ with respect to the change in $x$.
$$ f'(g(x))g'(x) $$
$$ x \mapsto z \mapsto y $$
How does a change in $x$ correspond to a change in $z$ and how does that resulting change in $z$ correspond to a change in $y$?
$$ f(x) = \sin(x) $$
$$ g(x) = x^2 $$
# Define Functions
def f(x):
return np.sin(x)
def g(x):
return x**2
Below is function that will help us analyze the changes discretely.
def chain(x,nudge=.1):
x1 = x
x2 = x + nudge
print(f'''
Changing x from {x1} to {x2}
x1: {x1}\t\t --> g(x1): {round(g(x1),1) }\t\t --> f(g(x1)): {round(f(g(x1)),1)}
x2: {x2}\t\t --> g(x2): {round(g(x2),1) }\t\t --> f(g(x2)): {round(f(g(x2)),1)}
ch: {round(x2-x1,2)}\t\t --> {round(g(x2)-g(x1),1)}\t\t --> {round(f(g(x2))-f(g(x1)),1)}
''')
for i in [1,5,10,20,50]:
chain(i)
# Show graphically
x = np.arange(-5,5 + .01 , .01)
# Plot functional forms
p1 = plot(700,400,y='')
p1.line(x,g(x),color='steelblue',legend='g(x)',line_width=3,alpha=.5)
p1.line(x,f(x),color='red',legend="f(x)",line_width=3,alpha=.5)
p1.line(x,f(g(x)),color='purple',legend='f(g(x))',line_width=3,alpha=.5)
# Define composite function
def F(x):
'''Composite function of f and g'''
return f(g(x))
# Plot derivatives
p2 = plot(700,400,y='')
p2.line(x,deriv(x,func=g,nudge=.001),color='steelblue',
line_dash="dashed",
legend="g'(x)",
line_width=3,alpha=.5)
p2.line(x,deriv(g(x),func=f,nudge=.001),color='purple',
line_dash="dashed",
legend="f'(g(x))",
line_width=3,alpha=.5)
p2.line(x,F(x),color='grey',
line_dash="dashed",
line_width=3,alpha=.5,
legend="F'(x) = f'(g(x))g'(x)")
show(column(p1,p2))
$$ f'(x) = \lim_{h \to0} \frac{e^{x+h} - e^x}{h} $$
$$ \lim_{h \to0} \frac{e^{x}e^{h} - e^x}{h} $$
Factor out $e^{x}$
$$ e^{x} \lim_{h \to0} \frac{e^{h} - 1}{h} $$
$\frac{e^{h} - 1}{h}$ converges to $1$ in the limit (see reading), leaving
$$ e^{x} $$
# Non-rigorous but a computational proof of this
for h in [.01,.001,.00001,.0000001]:
print((np.exp(h)-1)/h)
This isn't as much an intuition as watching the function behave when we nudge it.
def f(x):
return np.exp(x)
x = np.arange(-5,5.1,.1)
p = plot(700,300)
p.line(x,f(x),line_width=3,color="black")
p.line(x,deriv(x,f,nudge=.001),line_width=6,color="red",alpha=.4)
show(p)
We covered the three most important rule of differentiation (of which the other rules can usually be derived). However, it's useful to list off all the rules that we'll encounter most often for future reference.
| Rule Name | Function | Derivative |
|---|---|---|
| Sum | $ f(x) + g(x)$ | $f'(x) + g'(x)$ |
| Difference | $ f(x) - g(x)$ | $f'(x) - g'(x)$ |
| Scalar | $ f(ax) $ | $af'(x)$ |
| Product | $ f(x)g(x)$ | $f'(x)g(x) + f(x)g'(x)$ |
| Quotient | $ \frac{f(x)}{g(x)}$ | $\frac{f'(x)g(x) - f(x)g'(x)}{g(x)^2}$ |
| Chain | $ f(g(x))$ | $f'(g(x))g'(x)$ |
| Constant | $ f(c)$ | $0$ |
| Exponential | $ e^x$ | $ e^x$ |
| Natural Log | $ ln(x)$ | $\frac{1}{x}$ |
| Cosine | $ \cos(x) $ | $-\sin(x)$ |
| Sine | $ \sin(x) $ | $\cos(x)$ |
| Tangent | $ \tan(x) $ | $1 + \tan^2(x)$ |
(Note that I've bolded the rules that we covered today).
Python has a useful library for calculating the derivative and other mathematical derivations. With sympy, we can easily calculate the mathematical derivations outlined above (and much more). For now, I'll put it on our radar as it offers us a tool in python to quickly calculate the derivatives of trick functions.
z = symbols('z') # specify the symbol used in an expression
init_printing(use_unicode=True)
expr = z**3
expr
# Calculate the derivative
expr.diff()
# Implements all the tools and logic outlined above
expr2 = sin(z**2)
expr2.diff()
We can evaluate the symbolic outputs using the evalf and lambdify methods.
expr.evalf(subs={z: 2})
expr.diff().evalf(subs={z: 2})
lambdify allows us to efficiently pass whole arrays of numbers through. We convert each expression into a special "lambda-like" function that we can then pass numerical values generated by numpy.
# Define our expressions
expr = z**3 - 2*z**2 - 1*z
f = lambdify(z, expr, "numpy") # function
df = lambdify(z, expr.diff(), "numpy") # first derivative
expr
Let's plot the resulting values as we've done this entire time using the numerical arrays provided.
a = np.arange(-5,5,.1)
p = plot()
hline = Span(location=0, dimension='width', line_color='grey', line_width=2)
p.line(a,f(a),legend=f'f(x)={str(expr)}',alpha=.5,line_width=3)
p.line(a,df(a),color="red",
legend=f"f'(x)={str(expr.diff())}",alpha=.5,line_width=3,
line_dash = 'dashdot')
p.renderers.extend([hline])
show(p)
expr3 = (z**3+z)*(z*9+z**2)
expr3
expr3.expand()
simplify(expr3)
Express output equations in terms of python or latex.
print(expr3)
latex(expr3)