How to safely upgrade Arduino firmware and roll back if needed

When working with Arduino boards, updating the bootloader or sketch firmware can introduce incompatibilities or unexpected behavior. What are the recommended steps to perform a safe upgrade, including how to back up the current firmware, verify the new image, and test functionality before committing? If the upgrade causes issues, what reliable rollback procedures exist to restore the previous version without bricking the device? Please outline any required tools (e.g., ISP programmer, avrdude) and safety checks (e.g., checksum, version comparison) that should be part of the process.