IEC 61508 is a standard on the functional safety of electrical and electronic systems and specifically includes software.
So if you want to learn what is necessary I suggest you download the standard.
ISO 26262 is an automotive functional safety standard which again sets out methods of risk assement and how risk concatenates through an automotive function chain. Again it explicitly mentions software.
How you test software for safety is an enormous topic and too beg to address here.
I would just point out that these standards also expect users to test the tools they use to help them create software - such as compilers - to make sure they do not introduce problems.
An example of the kind of thing that these standards do is that IEC 61508 requires that the software is fully tested at the function level and that all possible branches and paths are taken through the software. This is an important step in that having a system do unexpected things that were never tested may be possible to avoid. This, however, cannot necessarily ensure that every unexpected external event will result in the correct software "decision" and outcome.
Your point about software testing not being exhaustive against all possibilities and timings of external events is important.
Formal methods were once thought to be the way to "prove" hardware was correct but idea of formally provable hardware lost momentum when it was realized that formally proving sofware-plus-hardware-plus-interrupts was a much less tractable problem.