Global Generalization Injection: Using Generated Sentences in Pre-Training Transformers

When we pre-train transformer models, we typically rely on existing texts to teach the model the intricacies of language. But what if we added a new twist to this process?

Imagine using the very sentences generated by the model itself as part of the training data. Could this act as a form of global generalization injection, adding new layers of complexity and adaptability to the learning process?

The concept raises intriguing questions:

• Is there an existing architecture that utilizes this idea?
• Would this approach enhance the model's robustness, or could it introduce unexpected challenges?

I am waiting for your interesting and insightful ideas.