Chlorine is actually not usually the main issue with tap water, it's more often the fluoride or the total dissolved solids that build up over time. I had a similar thing with my calathea last year, it would crisp up on the edges no matter what I did. I finally tested my tap water and the hardness was off the charts, around 250 ppm. A basic pitcher filter with an ion exchange resin helped way more than just letting the water sit out overnight, which only removes chlorine, not the minerals. The peace lily is super sensitive to salt buildup in the soil from those minerals, so your fix makes total sense.