Android: Changing keystore for signing doesn't request new password

1. Set up signing with a keystore on a project
2. Click "Sign package"
3. Input the password
4. Change the keystore to something else

At this point I would expect for a new password to be requested and the alias etc. updated, but I have to uncheck and check the "Sign package" checkbox before it requests it again.