Continual learning (CL) enables machine learning models to learn tasks sequentially while maintaining performance on previously learned tasks. This capability is crucial for developing intelligent systems that adapt to evolving conditions across domains like robotics, recommendation systems, and autonomous vehicles. However, neural networks typically suffer from catastrophic forgetting, where learning new tasks disrupts performance on older ones, often necessitating costly retraining from scratch.We present GKD-ER (Gradient-space Knowledge Distillation with Episodic Replay), a framework that effectively reduces catastrophic forgetting by combining three complementary techniques:

Gradient Projection (GP): Removes gradient components that would harm older tasks, ensuring parameter updates for new tasks remain orthogonal to previously learned knowledge.

Knowledge Distillation (KD): Maintains functional consistency by aligning the current model's outputs with those from a saved reference model on old data.

Episodic Replay (ER): Periodically revisits representative samples from past tasks stored in a memory buffer, reinforcing old decision boundaries and providing stable checkpoints.

Under standard conditions and representative replay assumptions, we theoretically demonstrate that GKD-ER achieves bounded forgetting. Our empirical evaluation on established benchmarks like Permuted MNIST and Split MNIST shows that GKD-ER surpasses strong baselines (Naive, EWC, SI, and ER alone) with higher final accuracies, significantly reduced forgetting, and stable class-level decision boundaries across tasks.By integrating constraints at the gradient, functional, and empirical levels, GKD-ER strikes an effective balance between stability and plasticity. This work advances the development of systems capable of continuous learning while preserving past expertise—a key step toward truly adaptive, lifelong learning agents.