Einstein's theory of general relativity, which Newton's theory of gravity is a part of, is fraught with the problem of singularity that has been established as a theorem by Hawking and Penrose, the later being awarded the Nobel Prize in recent years. The crucial {\it hypothesis} that founds the basis of both Einstein's and Newton's theories of gravity is that bodies with unequal magnitudes of masses fall with the same acceleration under the gravity of a source object. Since, the validity of the Einstein's equations is one of the assumptions based on which Hawking and Penrose have proved the theorem, therefore, the above hypothesis is implicitly one of the founding pillars of the singularity theorem. In this work, I demonstrate how one can possibly write a non-singular theory of gravity which manifests that the above mentioned hypothesis is only valid in an approximate sense in the ``large distance'' scenario. To mention a specific instance, under the gravity of the earth, a $5$ kg and a $500$ kg fall with accelerations which differ by approximately $113.148\times 10^{-32}$ meter/sec$^2$ and the more massive object falls with less acceleration. Further, I demonstrate why the concept of gravitational field is not definable in the ``small distance'' regime which automatically justifies why the Einstein's and Newton's theories fail to provide any ``small distance'' analysis. In course of writing down this theory, I demonstrate why the continuum hypothesis as spelled out by Goedel, is undecidable. The theory has several aspects which provide the following realizations: (i) Descartes' self-skepticism concerning exact representation of numbers by drawing lines (ii) Born's wish of taking into account ``natural uncertainty in all observations'' while describing ``a physical situation'' by means of ``real numbers'' (iii) Klein's vision of having ``a fusion of arithmetic and geometry'' where ``a point is replaced by a small spot'' (iv) Goedel's assertion about ``non-standard analysis, in some version'' being ``the analysis of the future''. A major drawback of this work is that it can easily appear to the authorities of modern science as too simple to believe in. This is, firstly due to the origin of the motivations being rooted to the truthfulness of the language in which physics is written and secondly due to the lucidity of the calculations involved. However, at the same time, this work can also appear as a fresh and non-standard approach to do physics from its roots, where the problem of singularity is not even there to begin with. The credibility of this work depends largely on whether the reader is willing adopt the second mindset.