We’ve been looking at the challenges of using good methodology in doing architecture research, which can include benchmark selection, simulator validation, setup randomization, instrumentation effects, software development stack tuning, replicatability in the presence of nondeterminism, and the cost of accurate simulation. Ultimately, the test of a proposed new architecture or feature is to show it working effectively in hardware. Today we see two examples of academic projects that actually made it to fabrication, which is fairly rare, given the cost of such efforts. The most common discovery in building real hardware is that there are many factors and constraints that end up delaying construction and draining off the anticipated performance. In the process, valuable lessons are learned by the participants, which will inform future work. There is no subsitute for this experience. 

Slides are here