This research introduces a pioneering framework named TransBERT that capitalizes on the capabilities of two sophisticated language models, TransPolymer and polyBERT, to comprehensively advance the polymer informatics field. TransPolymer, a Transformer-based language model, predicts polymer properties by leveraging self-attention mechanisms. The model employs a polymer tokenizer imbued with chemical awareness, facilitating the extraction of meaningful representations from polymer sequences. Moreover, TransPolymer benefits from rigorous pretraining on extensive unlabeled datasets through Masked Language Modeling, underscoring the pivotal role of self-attention in effectively modeling polymer sequences. In conjunction with TransPolymer, polyBERT contributes a fully automated polymer informatics pipeline designed to expedite the identification of application-specific polymer candidates with heightened speed and accuracy. Drawing inspiration from Natural Language Processing concepts, polyBERT operates as a chemical linguist, treating the chemical structure of polymers as a unique language. The pipeline integrates a polymer chemical fingerprinting capability and a multitask learning approach to map polyBERT fingerprints to diverse polymer properties effectively. Notably, polyBERT outperforms existing polymer property prediction methods based on manually crafted fingerprint schemes by achieving a remarkable two orders of magnitude increase in speed while maintaining high accuracy and integrating TransPolymer and polyBERT results in a robust computational tool poised to propel the fields of polymer design and structure-property relationship understanding. This combined framework strategically harnesses the strengths of Transformer models and machine-driven informatics, offering unparalleled efficiency in the prediction and identification of polymer properties. This synergistic approach holds significant promise for scalable deployment, including applications in cloud infrastructures, thereby making substantial contributions to the advancement of polymer science and informatics.