Lecture 11: Support Vector Machines
TTIC 31020: Introduction to Machine Learning
Instructor: Greg Shakhnarovich
Lecture by Andreas Argyriou
TTI–Chicago
October 20, 2010
Lecture 11: Support Vector Machines TTIC 31020
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Guidelines and tips
Prepare for your exams
Study with the several resources on Docsity
Earn points to download
Earn points by helping other students or get them with a premium plan
Community
Ask the community
Ask the community for help and clear up your study doubts
University Rankings
Discover the best universities in your country according to Docsity users
Free resources
Our save-the-student-ebooks!
Download our free guides on studying techniques, anxiety management strategies, and thesis advice from Docsity tutors
Support Vector Machines-Introduction to Machine Learning-Lecture 11-Computer Science, Lecture notes of Introduction to Machine Learning
Support Vector Machines, Andreas Argyriou, Large Margin Classification, Optimal Separating Hyperplane, Optimal Linear Classifier, Representer Theorem, Regularization, Lagrange Multipliers, Max-Margin Optimization, Quadratic Programming, Margin Decision Boundary, Support Vectors, SVM Classification, Nonlinear Decision Boundaries, Greg Shakhnarovich, Lecture Slides, Introduction to Machine Learning, Computer Science, Toyota Technological Institute at Chicago, United States of America.
Typology: Lecture notes
2011/2012
1 / 37
This page cannot be seen from the preview
Don't miss anything!
Related documents
Partial preview of the text
Download Support Vector Machines-Introduction to Machine Learning-Lecture 11-Computer Science and more Lecture notes Introduction to Machine Learning in PDF only on Docsity!
Lecture 11: Support Vector Machines
TTIC 31020: Introduction to Machine Learning
Instructor: Greg Shakhnarovich
Lecture by Andreas Argyriou
TTI–Chicago
October 20, 2010
Plan for today
Large margin classification; optimal separating hyperplane
Support Vector Machines
Optimal linear classifier
Which decision boundary is better?
Optimal linear classifier
Which decision boundary is better?
Regularization alone does not capture this intuition
Large margin classifier
Distance from a correctly classified (x, y) to the boundary:
‖w‖
y
wT^ x + w 0
Margin of the classifier on X = {(xi, yi)}Ni=1, assuming it
achieves 100% accuracy: the distance to the closest point
min i
‖w‖
yi
w
T xi + w 0
We are interested in a large margin classifier:
argmax w,w 0
‖w‖
min i
yi
w
T xi + w 0
Optimal separating hyperplane
So, we seek argmaxw,w 0
1 ‖w‖ mini^ yi
wT^ xi + w 0
Hard optimization problem... but: we can set
min i
yi
w
T xi + w 0
since can rescale ‖w‖, w 0 appropriately.
Then, the optimization becomes:
argmax w,w 0
‖w‖
s.t. yi
wT^ xi + w 0
≥ 1 , ∀i = 1,... , N.
Representer theorem
Consider the optimization problem
w
∗ = argmin w
‖w‖
2 s.t. yi(w
T xi + w 0 ) ≥ 1 ∀i
Theorem: the solution can be represented as
w∗^ =
∑^ N
i=
αixi
This is the “magic” behind Support Vector Machines!
Representer theorem - proof I
w∗^ = argmin w
‖w‖^2 s.t. yi(wT^ xi + w 0 ) ≥ 1 ∀i ⇒ w∗^ =
∑^ N
i=
αixi
Let w∗^ = wX + w⊥, where wX =
∑N
i=1 αixi^ ∈^ Span(x^1 ,... ,^ xN^ ), w⊥ ∈/ Span(x 1 ,... , xN )
Representer theorem - proof I
w∗^ = argmin w
‖w‖^2 s.t. yi(wT^ xi + w 0 ) ≥ 1 ∀i ⇒ w∗^ =
∑^ N
i=
αixi
Let w∗^ = wX + w⊥, where wX =
∑N
i=1 αixi^ ∈^ Span(x^1 ,... ,^ xN^ ), w⊥ ∈/ Span(x 1 ,... , xN ), i.e., wT ⊥xi = 0 for all i = 1,... , N
For all xi we have
w
∗T xi = w
T X xi^ +^ w
T ⊥xi^ =
Representer theorem - proof I
w∗^ = argmin w
‖w‖^2 s.t. yi(wT^ xi + w 0 ) ≥ 1 ∀i ⇒ w∗^ =
∑^ N
i=
αixi
Let w∗^ = wX + w⊥, where wX =
∑N
i=1 αixi^ ∈^ Span(x^1 ,... ,^ xN^ ), w⊥ ∈/ Span(x 1 ,... , xN ), i.e., wT ⊥xi = 0 for all i = 1,... , N
For all xi we have
w
∗T xi = w
T X xi^ +^ w
T ⊥xi^ =^ w
T X xi
Representer theorem - proof II
w∗^ = argmin w
‖w‖^2 s.t. yi(wT^ xi + w 0 ) ≥ 1 ∀i ⇒ w∗^ =
∑^ N
i=
αixi
Now, we have
‖w∗‖^2 = w∗
T w∗
Representer theorem - proof II
w∗^ = argmin w
‖w‖^2 s.t. yi(wT^ xi + w 0 ) ≥ 1 ∀i ⇒ w∗^ =
∑^ N
i=
αixi
Now, we have
‖w∗‖^2 = w∗
T w∗^ = (wX + w⊥)
T (wX + w⊥)
Representer theorem - proof II
w∗^ = argmin w
‖w‖^2 s.t. yi(wT^ xi + w 0 ) ≥ 1 ∀i ⇒ w∗^ =
∑^ N
i=
αixi
Now, we have
‖w∗‖^2 = w∗
T w∗^ = (wX + w⊥)
T (wX + w⊥) = wX T^ wX ︸ ︷︷ ︸ ‖wX ‖^2
- w ⊥Tw⊥ ︸ ︷︷ ︸ ‖w⊥‖^2
since wTX w⊥ = 0.
Suppose w⊥ 6 = 0. Then, we have a solution wX that satisfies
all the constraints, and for which ‖wX ‖^2 < ‖wX ‖^2 + ‖w⊥‖^2 = ‖w∗‖^2.
Representer theorem - proof II
w∗^ = argmin w
‖w‖^2 s.t. yi(wT^ xi + w 0 ) ≥ 1 ∀i ⇒ w∗^ =
∑^ N
i=
αixi
Now, we have
‖w∗‖^2 = w∗
T w∗^ = (wX + w⊥)
T (wX + w⊥) = wX T^ wX ︸ ︷︷ ︸ ‖wX ‖^2
- w ⊥Tw⊥ ︸ ︷︷ ︸ ‖w⊥‖^2
since wTX w⊥ = 0.
Suppose w⊥ 6 = 0. Then, we have a solution wX that satisfies
all the constraints, and for which ‖wX ‖^2 < ‖wX ‖^2 + ‖w⊥‖^2 = ‖w∗‖^2.
This contradicts optimality of w∗, hence w∗^ = wX. QED