机器学习：5.4 Stacking

• • • Multi-layer Stacking
    • Overfitting in Multi-layer Stacking

Combine multiple base learners to reduce variance

• Base learners can be different model types

• Linearly combine base learners outputs by learned parameters

Widely used in competitions

bagging VS stacking

• Bagging: bootstrap samples to get diversity

• Stacking: different types of models extract different features

• Can use a different set of base learners at each level

• Concatenating original inputs helps

Train leaners from different levels on different data to alleviate
overfitting

Split training data into A and B, train L 1 L_1 L1​ learners on A, run inference on B to generate training data for L 2 L_2 L2​ learners

Repeated k-fold bagging:

• Train k models as in k-fold cross validation
• Combine predictions of each model on out-of-fold data
• Repeat step 1,2 by n times, average the n predictions of each example for the next level training