Neural networks can overwrite old knowledge when learning new things, a challenge for continual learning.
Manifestations
- Lost capabilities after fine-tuning
- Degraded general knowledge
- Skill regression
Mitigation
- Elastic weight consolidation
- Experience replay
- Multi-task training
- Careful fine-tuning scope