Timothy Farrar has some interesting ideas embedded in this recent post describing the design guidelines driving the development of his GPGPU OS. After thinking about it a bit, I've extracted these rules from his post:

Development Environment

1. Code-to-test iteration should be instant
2. Built-in memory, cycle, and bandwidth performance monitoring should be provided
3. Save and restore of current data and execution state should be possible

Language

1. Inputs and outputs should be explicit and naturally limited to valid addresses
2. High performance data structures should have natural representations
3. Parallelism and dataflow should be explicitly supported

Clearly others are thinking on the same lines; functional languages are another approach these rules would work with naturally.