OK, so, the thing about scientists is they typically develop the habit of entertaining even improbable ideas. Because they understand the limitations of human experience and realize that even things that intuitively might seem improbable may actually be quite possible once you dig into the details. So rather than ignoring even outlandish ideas scientists prefer to take the time to do some math to see if it’s reasonable to dig into more, or not. These are often “first order approximations” or “back of the envelope” estimates (bote’s).
The issue at hand here is nuclear fusion. During the Manhattan project the scientists had gotten fairly familiar with fission but much less so with fusion. The specific scenario isn’t the atmosphere “catching on fire” but a self-sustaining fusion reaction taking place. Fusion will occur if you can achieve a combination of density and temperature sufficient to cause a high enough level of fusion reactions that the energy released by such will maintain and/or increase the temperature, leading to a self-perpetuating cycle.
In the case of a nuclear detonation you inject a ton of energy into a small portion of the atmosphere, heating it up tremendously. So you have a very hot expanding (and thus cooling) bubble of gas/plasma. If the gas/plasma was hot enough to cause fusion of the Oxygen or Nitrogen in the atmosphere and if those fusion reactions themselves released enough energy then there’s the potential that the fusion reactions would increase at a rate beyond the rate of dissipation of heat. And then you end up with a thermonuclear fusion “flame front” which would sort of spread around the atmosphere and destroy all life on Earth.
Now, this is a very improbable thing on its face just because of the fact that at ordinary atmospheric gas densities the rate of fusion even in a superheated bubble of atmosphere isn’t going to be very high compared to the enormously rapid process of expansion and cooling that takes place. However, as I mentioned, scientists don’t take “intuitively improbable” as good enough, they make sure to do the math. And indeed they performed their due diligence, did the math, found out that it was basically an impossible eventuality, and so went on with the test (and operational deployment).
Sources:
* [Los Alamos Technical Reports – LA-602: Ignition of the Atmosphere with Nuclear Bombs](https://fas.org/sgp/othergov/doe/lanl/docs1/00329010.pdf)
Whaddaya Say?