Experimentation

While the Internet and more generally network technologies improve our lives in many ways, their increasing presence also highlights their various limits. These limits drive the research and industrial communities in the design and development of new networking technologies. These technologies need to pass various evaluation tests before turning into full fledged products. Simulation tests are often inexpensive and allow rapid first approximations of behaviours such as performance and scalability. However, due to design and computational complexities, simulations always use simplified models, which fail to capture the complexity and dynamics of real environments. Emulation is an alternative, where parts of the test use prototypes and other parts use substitute entities reproducing real-world behaviours. However, these substitutes again often fail to capture the complexity of the real-world. As a result, only experimentations in real testbeds with ideally real users provide evaluation results close to reality. Thus, testbeds are a key component in the development of new networking technologies, and this insight led to many large-scale initiatives around the world with multi-billion dollar budgets.

 Compared to other scientific research fields, such as the life sciences, the networking field so far has not developed a culture of rigorous peer verification of experimental results. This is mostly due to the fact that even if the same or a similar experimental infrastructure is available (i.e. through a federation of world-wide testbeds), there is currently no unambiguous way to describe an experiment enabling others to repeat it. Indeed, many high quality publications provide insufficient descriptions of the performed experiments to allow others a fair and valid comparison of different approaches to specific problem in different settings. Thus, to increase the scientific rigour of our field, there is a need for efficient tools and methodologies that provide complete descriptions of experiments, including resources used, and measurements taken.