There are two ways to unify gravitational field and gauge field. One is to represent gravitational field as principal bundle connection, and the other is to represent gauge field as affine connection. Poincar\'{e} gauge theory and metric-affine gauge theory adopt the first approach. This paper adopts the second. In this approach: (i) Gauge field and gravitational field can both be represented by affine connection; they can be described by a unified spatial frame. (ii) Time can be regarded as the total metric with respect to all dimensions of internal coordinate space and external coordinate space. On-shell can be regarded as gradient direction. Quantum theory can be regarded as a geometric theory of distribution of gradient directions. Hence, gauge theory, gravitational theory, and quantum theory all reflect intrinsic geometric properties of manifold. (iii) Coupling constants, chiral asymmetry, PMNS mixing and CKM mixing arise spontaneously as geometric properties in affine connection representation, so they are not necessary to be regarded as direct postulates in the Lagrangian anymore. (iv) The unification theory of gauge fields that are represented by affine connection can avoid the problem that a proton decays into a lepton in theories such as $SU(5)$. (v) There exists a geometric interpretation to the color confinement of quarks. In the affine connection representation, we can get better interpretations to the above physical properties, therefore, to represent gauge fields by affine connection is probably a necessary step towards the ultimate theory of physics.