proof of work – Determination of Block Size Limit
What happens if, for instance, a block’s nonce is so large that the block’s size exceeds the limit?
The nonce is of a fixed size, 4 bytes. It can not be any other size.
Is there any systematic way of setting this limit without knowing how to break SHA256?
SHA256 has little to do with the validity of a block, having broken SHA256 in any way would not change the consensus rule of the block size. An invalid block with valid proof of work is still invalid, for example if it was larger than the rule specifying limits.
As an aside, no hash function has ever been broken in a way that would impact Bitcoins proof of work. Other than being too small, MD5 is completely suitable for POW despite being embarrassingly broken for any other task.