Investigating something scientifically usually means trying to make predictions about what is going to happen. But as good as the scientific models are, they only very rarely make perfect predictions. Instead, scientists usually make predictions that are probabilistic in nature. Making valid probabilistic models is important to the inductive reasoning at the heart of scientific thinking about the world.
But the problem with probability - and induction in general, or logical thinking of any kind - is that it requires certain assumptions. In a new article, I discuss the Application of Probability in Physics and what assumptions are applied in different probability approaches, from classical probability theory to Bayesian statistics to the frequentism approach which is usually applied in physics.
Of course, things in probability really get interesting when it comes to the realm of quantum physics. The way probability manifests in quantum physics warrants its own detailed discussion ... but the best starting place is in understanding the less esoteric aspects of probability.