We should take a lesson from the real world. There are about 3 supervisor for every worker.
The supervisors fight amongst each other (this is where all the parallelism happens) to see how much of the worker's time they can get. So we need 3 cores for every "working core" - that actually does the non-parallel task. In this way there are no conflicts within the "working core" which does all the actual work. The other 3 cores are just used to sort out all the parallelism.
i like this "My hypothesis is that we can solve [the software crisis in parallel computing], but only if we work from the algorithm down to the hardware -- not the traditional hardware first mentality." Tim Mattson, principal engineer at Intel.
Parallel processing is not that difficult. Software is modelling and the world is concurrent by nature. But what is difficult is to undo the brainswashing of a sloppy education in software engineering that starts bottom up (from the hardware to the software: I call that the von Neumann syndrome), hence the two conflicting quotes from Tim Mattson, Intel.
We have a concurrent programming model that covers for a single multi-tasking processor to a networked system with (theoretically millions of nodes, even heterogeneous) in just 5 KB per node. Difficult? No, because it is based on a formal development and borrows from CSP (the Communicating Sequential Processes process algebra of Hoare). It's actually very natural. But where do they still teach CSP?
"Everybody who learns concurrency thinks they understand it, ends up finding mysterious races they thought weren’t possible, and discovers that they didn’t actually understand it yet after all."
Sums up my experience. You get better at it with time but there is no room for complacency!
I think parallel processing is a bit different to multi-tasking.
The purpose of multi-tasking is to simultaneously enable multiple independent tasks so that they can proceed on one computer. There is a need to protect shared resources etc.
Parallel processing takes a single task and tries to break it into smaller tasks to exploit multiple processors.
As an example, consider the difference in the way make can work.
Multi-tasking allows two seperate makes to progress in parallel building two projects.
Multi-processing allows multiple threads within the building of a single project to allow that build to proceed faster.
Design for multi-processing is completely different than for multi-tasking.