I'd like to imagine there was this ephemeral, fleeting chance that Jimmy could've walked away from this, until he mentioned the peach grove. As we see, it's so connected to the very heart of Eli and the reminder of its disappearance evaporates the last shred of humanity in Eli. He disengages for a moment, mentally sighing and thinking "...okay."