So the Jeep started making a little more noise than normal. It almost sounded like rod knock, but it was too high up on the motor. I spent a little time troubleshooting things and realized that it was probably the water pump, even though it was completely quiet when I was spinning it by hand and not leaking, it was the only accessory that had the slightest play.
Thankfully, a Cherokee is stupid easy to work on. This only took about two hours to swap out. I am a little concerned by the amount the water pump propeller ate into the cylinder sleeve on piston #1 though. I really wish this wouldn’t have started the day I was driving to New Orleans for a flight for work. Timing is always bad with this kind of thing, but you just deal with it as best you can.