[Sklearn error: ‘SVR’ object has no attribute ‘_impl’] (https://stackoverflow.com/questions/19089386/sklearn-error-svr-object-has-no-attribute-impl)
0.14.1 SVC issue

Mostly, this error is caused by improper version of scikit-learn.
For example, you trained one model using scikit-learn with version of 0.1. However, you updated your packages to 0.2 after training.
When you using joblib.load to load the model file, it will throws out this error since pickle can’t recognize which version.

TODO

I met this error when I tried to use GridSearchCV to find best hyper-parameter.
I checked my configuration and found when I set the parameter n_jobs to -1 which means run parallelly, it will throw out this error.

gs = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=1, cv=cv)

It seems that when using multiprocessing to run estimator parallely, the pickle cannot serialize the object(e.g nested function).

Can’t pickle nested function in FeatureExtracter
Python: pickling nested functions

Cliff & Flat Regions

Cliff and Flat regions in the parameter space of neural network.
θ and γ are two parameters and L(θ, γ) is the loss function. The region with light color is a cliff and the region with dark color is a flat region.

Whiten

在深度学习训练中，白化（Whiten）是加速收敛的一个小Trick，所谓白化是指将图像像素点变化到均值为0，方差为1的正态分布。

Dropout

Batch Normalization

Weight Normalization

Local Minima

局部最小值

Ablation Study

2
消融研究，指通过移除某个模型或者算法的某些特征，来观察这些特征对模型效果的影响
实际上ablation study就是为了研究模型中所提出的一些结构是否有效而设计的实验。比如你提出了某某结构，但是要想确定这个结构是否有利于最终的效果，那就要将去掉该结构的网络与加上该结构的网络所得到的结果进行对比，这就是ablation study

An ablation study typically refers to removing some “feature” of the model or algorithm, and seeing how that affects performance.

Examples:

An LSTM has 4 gates: feature, input, output, forget. We might ask: are all 4 necessary? What if I remove one? Indeed, lots of experimentation has gone into LSTM variants, the GRU being a notable example (which is simpler).
If certain tricks are used to get an algorithm to work, it’s useful to know whether the algorithm is robust to removing these tricks. For example, DeepMind’s original DQN paper reports using (1) only periodically updating the reference network and (2) using a replay buffer rather than updating online. It’s very useful for the research community to know that both these tricks are necessary, in order to build on top of these results.
If an algorithm is a modification of a previous work, and has multiple differences, researchers want to know what the key difference is.
Simpler is better (inductive prior towards simpler model classes). If you can get the same performance with two models, prefer the simpler one.

Sharp Minimal

尖锐最小值
Yoshua Bengio 组在2017年发表文章Sharp Minima Can Generalize For Deep Nets介绍了在深度神经网络中尖锐的最小值也可以让模型具有泛化能力.

Internal Covariate Shift – ICS

继承关系顺序的不同

通过一个例子来说明继承关系顺序的不同.
在两端代码中, 分别使用显示调用父类初始化方法和super机制进行初始化操作.

显示调用父类初始化方法

class A:
    def __init__(self):
        print("Enter A")
        print("Leave A")

class B(A):
    def __init__(self):
        print("Enter B")
        A.__init__(self)
        print("Leave B")

class C(A):
    def __init__(self):
        print("Enter C")
        A.__init__(self)
        print("Leave C")

class D(A):
    def __init__(self):
        print("Enter D")
        A.__init__(self)
        print("Leave D")

class E(B, C, D):
    def __init__(self):
        print("Enter E")
        B.__init__(self)
        C.__init__(self)
        D.__init__(self)
        print("Leave E")

E()

运行上述代码, 得到的输出是

Enter E
Enter B
Enter A
Leave A
Leave B
Enter C
Enter A
Leave A
Leave C
Enter D
Enter A
Leave A
Leave D
Leave E

super机制

class A:
    def __init__(self):
        print("Enter A")
        print("Leave A")

class B(A):
    def __init__(self):
        print("Enter B")
        super(B, self).__init__()
        print("Leave B")

class C(A):
    def __init__(self):
        print("Enter C")
        super(C, self).__init__()
        print("Leave C")

class D(A):
    def __init__(self):
        print("Enter D")
        super(D, self).__init__()
        print("Leave D")

class E(B, C, D):
    def __init__(self):
        print("Enter E")
        super(E, self).__init__()
        print("Leave E")

E()

运行上述代码, 得到的输出是

Enter E
Enter B
Enter C
Enter D
Enter A
Leave A
Leave D
Leave C
Leave B
Leave E

由上述的例子可以得出, 在涉及多继承时, super机制可以保证公共父类仅被执行一次，至于执行的顺序，是按照mro进行的（E.mro).

Python 多重继承mro
文档 : The Python 2.3 Method Resolution Order
Python Tutorial: Understanding Python MRO – Class search path

什么是MRO?

MRO即method resolution order，主要用于在多继承时判断调的属性的路径(来自于哪个类). 在Python2.3之前是基于此算法，但从Python2.3起应用了新算法:C3算法。

为什么采用C3算法

C3算法最早被提出是用于Lisp的，应用在Python中是为了解决原来基于深度优先搜索算法不满足本地优先级，和单调性的问题。
本地优先级：指声明时父类的顺序，比如C(A,B)，如果访问C类对象属性时，应该根据声明顺序，优先查找A类，然后再查找B类。
单调性：如果在C的解析顺序中，A排在B的前面，那么在C的所有子类里，也必须满足这个顺序。
在Python官网的The Python 2.3 Method Resolution Order中作者举了例子，说明这一情况。

F=type(‘Food’, (), {remember2buy:’spam’})
E=type(‘Eggs’, (F,), {remember2buy:’eggs’})
G=type(‘GoodFood’, (F,E), {})
根据本地优先级在调用G类对象属性时应该优先查找F类，而在Python2.3之前的算法给出的顺序是G E F O，而在心得C3算法中通过阻止类层次不清晰的声明来解决这一问题，以上声明在C3算法中就是非法的。

C3算法

判断mro要先确定一个线性序列，然后查找路径由由序列中类的顺序决定。所以C3算法就是生成一个线性序列。
如果继承至一个基类:
class B(A)
这时B的mro序列为[B,A]

如果继承至多个基类
class B(A1,A2,A3 …)
这时B的mro序列 mro(B) = [B] + merge(mro(A1), mro(A2), mro(A3) …, [A1,A2,A3])
merge操作就是C3算法的核心。
遍历执行merge操作的序列，如果一个序列的第一个元素，在其他序列中也是第一个元素，或不在其他序列出现，则从所有执行merge操作序列中删除这个元素，合并到当前的mro中。
merge操作后的序列，继续执行merge操作，直到merge操作的序列为空。
如果merge操作的序列无法为空，则说明不合法。

例子：
class A(O):pass
class B(O):pass
class C(O):pass
class E(A,B):pass
class F(B,C):pass
class G(E,F):pass

A、B、C都继承至一个基类，所以mro序列依次为[A,O]、[B,O]、[C,O]
mro(E) = [E] + merge(mro(A), mro(B), [A,B])
= [E] + merge([A,O], [B,O], [A,B])
执行merge操作的序列为[A,O]、[B,O]、[A,B]
A是序列[A,O]中的第一个元素，在序列[B,O]中不出现，在序列[A,B]中也是第一个元素，所以从执行merge操作的序列([A,O]、[B,O]、[A,B])中删除A，合并到当前mro，[E]中。
mro(E) = [E,A] + merge([O], [B,O], [B])
再执行merge操作，O是序列[O]中的第一个元素，但O在序列[B,O]中出现并且不是其中第一个元素。继续查看[B,O]的第一个元素B，B满足条件，所以从执行merge操作的序列中删除B，合并到[E, A]中。
mro(E) = [E,A,B] + merge([O], [O])
= [E,A,B,O]

同理
mro(F) = [F] + merge(mro(B), mro(C), [B,C])
= [F] + merge([B,O], [C,O], [B,C])
= [F,B] + merge([O], [C,O], [C])
= [F,B,C] + merge([O], [O])
= [F,B,C,O]

mro(G) = [G] + merge(mro[E], mro[F], [E,F])
= [G] + merge([E,A,B,O], [F,B,C,O], [E,F])
= [G,E] + merge([A,B,O], [F,B,C,O], [F])
= [G,E,A] + merge([B,O], [F,B,C,O], [F])
= [G,E,A,F] + merge([B,O], [B,C,O])
= [G,E,A,F,B] + merge([O], [C,O])
= [G,E,A,F,B,C] + merge([O], [O])
= [G,E,A,F,B,C,O]