March 1, 2011

inside the FP7 evaluation (part 1?)

I am involved in the process of project evaluation for the European Commission for the first time. Some selected remarks:
  • there is an art of writing proposals. Scientists know that the art of writing academic papers has become a key skill in the modern science battlefield. The art of writing proposal is also widely admitted, but I had never faced it. Now I know. The top proposals I reviewed have a lot in common, including the approach and the style. Generally, these projects are about an exciting but obscured concept (vaporware?), supported by a very standard research from high-h-ranked scientists (business as usual), in cooperation with fresh SMEs (old students), and the classic large company (whose role is... hmm, well, to be there). These proposals have probably been powerpoint in a previous life: objectives are presented in a bullet-mode emphatic way, the proposal is un-verbose (so less risk of inconsistency), every page contains a figure or a table. As can be expected, the workpackage organization is perfect with an ideal balance of man-month by workpackage and by partners. It is difficult to know the future of such project. The academic work is probably already under submission. Several web revolutions will occur until the end of the project. The final software will probably fail, because it will be coded by un-managed students in University. However, in the evaluation form of European Commission, such a proposal deserves a "check" for every critical parameter, so at the end, they have good chances to win.
  • it is innovation, it is not about research. For those who had doubts. The best proposals are ambitious. Most of them include some attractive real-world experimentations, which requires committing a lot of developers. As the overall funding is constrained, the research-oriented demand is minimal. Moreover, as previously said, no inconsistency is tolerable, therefore every dozen of claimed man-month should be justified, and related with the remaining of the project, which is very short-term. Therefore, the scientific topic of every participating "researcher" is approximately defined in advance. Is it research? Of course not, it is the so-called innovation by research. I tend to be in favor of such early development project, however the other funding agencies (local area and national) follow the same objective, so is there a way to make un-purposed research? And why the hell are there so few innovations by research from Europe although it is already the seventh similar program?
  • the reviewing process is very short, but well paid. I detected one bad consequence from being a paid reviewer: reviewers have incentives to evaluate more projects, although they have no time to evaluate them carefully. Delays are very tight: I had seven projects to review in less than twelve days, each project being a hundred pages document, which details a three years long study by a consortium ranging from six to twelve partners. Moreover, the quality of the proposals was excellent, even for the worst one. Hence, one half day for a complete reviewing is minimum for a (slow?) young scientist like me. The risk is to produce a quick evaluation based on the strategy of "killing a project for any small detail". My personal reviewing strategy (I saw several people doing the same) was to first have a very quick first pass on the whole document, then to go into details once the overall concept was clear. In this context, no inconsistency is tolerable, it is better to have only one proposal writer, preferentially someone who... now come back to the first point of this post.
I will probably have more to say after my week at Brussels for the final decision.

No comments:

Post a Comment