Unreasonable Effectiveness

In an address to the Prussian Acadamy of Science in Berlin 
on Jan 27, 1921, Einstein was discussing the significance of 
mathematics in the history of scientific thought, and remarked

 "At this point an enigma present itself which in all 
  ages has agitated inquiring minds.  How can it be that
  mathematics, being after all a product of human thought
  which is independent of experience, is so admirably
  appropriate to the objects of reality?  Is human reason,
  then, without experience, merely by taking thought, able
  to fathom the properties of real things?

  In my opinion the answer to this question is breifly
  this: As far as the laws of mathematics refer to
  reality, they are not certain; and as far as they 
  are certain, they do not refer to reality."

This same issue was later addressed by Eugene Wigner in his famous 
essay "The Unreasonable Effectiveness of Mathematics", which appeared 
in the "Communications in Pure and Applied Mathematics", vol 13, 
1960, and is probably the most often cited reference for this notion, 
although it's somewhat odd that the idea became so closely associated 
with the physicist Wigner, considering that Einstein could say in 
1921 that this enigma "in all ages has agitated inquiring minds".  
It isn't as if Wigner provided a compelling explanation for the 
enigma, since his conclusion was basically identical to his 
premise:

   The miracle of appropriateness of the language of
   mathematics for the formulation of the laws of physics
   is a wonderful gift which we neither understand nor
   deserve.  We should be grateful for it, and hope that
   it will remain valid for future research, and that it
   will extend, for better or for worse, to our pleasure
   even though perhaps also to our bafflement, to wide
   branches of learning.

I would say that Einstein's consideration of this enigma (as 
quoted above) was far more insightful, so perhaps all those Wigner 
references don't do justice to Einstein (to say nothing of all those 
other inquiring minds through the ages).  On the other hand, Einstein 
gets quite a few undeserved citations for Kant's epigram about 
comprehensibility, because he happenned to quote it in his famous 
1936 essay "Physics and Reality"

   The very fact that the totality of our sense experiences
   is such that by means of thinking...it can be put in order,
   this fact is one which leaves us in awe, but which we shall
   never understand.  One may say "the eternal mystery of the
   world is its comprehensibility".  It is one of the great
   realizations of Immanual Kant that this setting up of a
   real external world would be senseless without this
   comprehensibility.

Of course, Kant was famously discredited among physicists by his 
pronouncements on "final categories" and necessary modes of thought, 
among which he unluckily listed the framework of Euclidean space.  
With the advent of non-Euclidean geometry Kant fell into disrepute 
among physicists and mathematicians, so it was perhaps inevitable 
that they would find another source for that epigram, which is 
simply too good to discard just because you've decided the author 
was an idiot.  (Of course, notwithstanding this opinion popular
among mathematicians, Kant was far from being an idiot, and Einstein 
had a healthy respect for him and his ideas.)

One of the earlier inquiring mind to be agitated by the link 
between math and nature was Pythagoras, circa 550 BC.  Certainly 
the Pythagorean doctrine that "all things are numbers" is a fairly 
explicit assertion of correspondence between the concepts of 
mathematics and the elements of the physical world and experience.  
On the other hand, this connection didn't originate with the 
Pythagoreans, but was carried over from various traditions in 
the earlier Eastern cultures, particularly Babylonia.  Of course, 
with ancient peoples it's not always easy to distinguish between 
mathematics and numerology, nor between philosophy and mysticism.
Then again, the mystical "hermetic" tradition survived in the 
scientific world at least through the time of Isaac Newton.