The RasPi SDRAM and SoC both get warm, depending on what the CPU is doing. You can also overclock and overvolt the CPU, which generates a lot more heat and voids the warranty in the case of overvolting.
The SoC conducts heat both to the SDRAM and to the back of the board, so both sides can get hot. The 3.3V linear regulator is right next to the CPU doesn't help. I found that if you don't have any air flow on the back of the board it gets pretty warm. So I mounted my RasPi vertically in an open frame to get natural air flow across both sides of the board. This keeps the chips cool for the kind of things I do with RasPi.
You can also get heat sinks for the SoC and USB/LAN chips.
I am pretty new to Raspberry Pi. Just got my board yesterday and the first thing I wanted to take a look was the processor. All what I could see a single chip which seemed to be the processor, but the marking on it showed the name of the company as Samsung. I read that the SoC was from Broadcom, could not find it. Ultimately "google" cleared my doubt and I was amazed to see the packaging.
I agree with you that this will have heat challenges and hopefully the heat would get conducted through the balls from the top package (SDRAM) to the bottom package (SoC) and to the copper planes...hopefully the ground planes have enough copper. Though in the manual it is mentioned that the board is designed to be operated at room temperature only.
re "At 1.6x2.0 mm2 there is not much space to put balls at a 0.4mm pitch"
Not much room at all. Yet, they keep getting smaller. 0.3mm parts have started showing up. Last year one of our customers had us put a 0.3mm pitch class-D amplifier BGA onto a board. We've seen several more since then.
I think the KL02 part is 2mm x 2mm and the not yet available KL03 part comes in at 1.6mm x 2mm with 0.4mm pitch balls.
I thought we'd be seeing more POP by now, but, with a few exceptions, it still seems to only show up on phone processors. The Texas Instruments OMAP on the beagleboard (not Beaglebone) also uses POP: a 0.4mm pitch processor BGA with a 0.5mm pitch memory chip on top of it.