Image Transformer has recently achieved significant progress for natural image understanding, either using supervised (ViT, DeiT, etc.) or self-supervised (BEiT, MAE, etc.) pre-training techniques. In this paper, we propose HiT, a self-supervised pre-trained Histological Image Transformer model using large-scale unlabeled histological images for medical image processing tasks, which is essential since no supervised counterparts ever exist due to the lack of human-labeled histological images. We leverage HiT as the backbone network in a variety of vision-based histological image processing tasks. Experiment results have illustrated that the self-supervised pre-trained HiT model the new state-of-the-art results on these downstream tasks, e.g. histological image classification on SIPaKMeD database achieved an accuracy of 97.45% and 99.29% for 5-class and 2-class classifications, respectively.