The Setup: A custom PRGB-to-LVDS daughter card on a DM355 evaluation module (EVM) driving a 1280x768 LCD, it is supplied with a 74.25-MHz clock from the daughter card.
The Question: What is the source of noise when using PRGB output with the TMS320DM355 evaluation module? [Editor: And how does one get rid of it?]
For example, in the photo below, the decode demo is used to display a test pattern that was captured using the encode demo. The displayed image and the captured image are both at 720x480 resolution. The video looks fine on the standard NTSC output, with and without the on screen display.
PRGB image showing noise
However, when switched to PRGB output (using the driver code developed for a TMS320DM6446 project) the image is quite noisy. In the photo, the worst of the noise is in the top half of the image. The decode demo OSD is not displayed, but the OSD is active and squeezed into the top half of the image on purpose for debugging. When the OSD is disabled completely, the noise is much less severe as it is in the bottom half of the photo.
The Answer: This is memory bandwidth saturation which causes the VPBE to be unable to meet real-time deadlines in servicing the display. You can see where individual lines are missing and shifted when the deadline is broken. It ends up looking like random horizontal lines popping up randomly on the screen.
This explanation is congruent with your observation that when the OSD is enabled, it makes things significantly worse. An OSD window enabled means another mass of memory accesses in addition to the video itself making the effect more pronounced. The DM6446 that you had working with good quality before has 2x the DDR bandwidth available (32-bit vs. 16-bit bus) making the probability of such effects actually appearing much lower.
To make this better, the first area as you have suggested improves things already would be to lower the amount of bandwidth being used by the OSD. Any other display windows you have open will be slowing things down and exacerbating this issue. The other step you may want to take is to try adjusting the PBBPR register discussed in section 4.7 of SPRUEH7d. This can help to alleviate these sorts of issues.
As a general note, you will have a hard time supporting true 720p60 or higher output on the DM355 without the higher-frequency 270MHz version (versus the 216-MHz version on the EVM). More details can be found at: http://wiki.davincidsp.com/index.php?title=DM355_HD_Display