The recent proliferation of so-called open-source large language models (such as LLaMA, Falcon, Mistral) has introduced a broader range of alternatives for AI practitioners and researchers. However, the majority of these models cannot be considered truly open-source, as they often provide only partial artifacts, such as final model weights or inference code. Furthermore, technical documentation accompanying these models tends to focus on high-level architectural decisions and superficial metrics, leaving critical aspects of the training process, including dataset composition, distribution, model checkpoints, and intermediate results, largely undisclosed. This lack of transparency presents a significant barrier to progress in the field, restricting the potential for open, collaborative research. In the absence of access to original datasets, attempts to further train or fine-tune these models by third parties are susceptible to issues such as catastrophic forgetting.In response to this challenge, we propose a method that facilitates more effective supervised fine-tuning of these closed-source models, without requiring access to the original data, while mitigating the risk of catastrophic forgetting.