I think that the second equation is correct. But the following definitions are important - also because of the units.

1.) The ripple coefficient must be used in the equation as an absolute number - and in contrast to you I call this "a" .
2.) Γ is the reflection coefficient (absolute), when multiplied by 100 >>> in %
3.) a is the "ripple coefficient" (absolut) : a=Amax/Amin (absolut) and with your terminology ρ=20log(a).
4.) This leads to the result Γ=0 if Amax=Amin (no ripple) which makes sense.
5.) Your example: ρ=0.46 dB (positive sign !) >>> a=1.0544
Γ²=1-1/1.0544²=1-1/1.111=0.1005024 >>>Γ=0.317 >>> -10 dB.

I think it suits.
Regards