This is an informal post; as much a reflection as something that may actually be useful to others. This post is on some thoughts on being an NSF panelist for a review panel. It’s written after having served on three panels - one face-to-face, and two online - and feeling like I have something to say, though I’m unsure what, just yet. This is sort of written for someone interested in serving as a panelist or someone serving for the first time looking for some thoughts from someone relatively new to the process (and so aware of the questions/concerns I had). Here goes…
It is common to hear this from others who have previously served on a review panel, but it bears mentioning that it takes a lot of time! I think one rough way to estimate how much time it will take is to consider each proposal that you will review to be comparable in the time it will take to review as one journal article. Thus, the standard of reviewing eight proposals would, by this token, be roughly comparable to reviewing eight journal articles. In some ways, that’s an over-estimate of how long the process of reviewing proposals will take, but when all is said and done (reading the proposals, writing feedback, logistically preparing for and learning about the program/review process, discussing the proposals, and writing summaries), it seems about right. Try to plan accordingly!
Reviewing eight proposals can be overwhelming. Something I’ve found helpful is to first skim all eight and then to read the proposals more carefully, taking notes. After that, I find it helpful to write a rough sketch of a review with general thoughts - just to have something down on the page. Then, I’ll read the proposal more carefully, adding specific observations/questions to the review.
The review does not need to be perfect at the time you submit (24 hours or the day before the panel); you will be discussing the proposal with other panelists, and it is most important to have a record of what you want to raise and discuss with others. You can revise your review up until the end of the panel.
During the panel, you’ll provide a brief overview (not an evaluation) of the proposals for which you’re the primary panelist; then, you’ll provide your comments, then will take notes on others’ comments. The discussion is time to try to argue for the weaknesses of proposals or to advocate for their strengths; it can be a dynamic and engaging conversation and part of the review process, one that can be important (or, to me, it feels that way!), to whether a proposal will be recommended for funding. The result of this discussion is the panel providing a rating of Highly Competitive, Competitive, or Not Competitive different from the (effectively 1-5) ratings you individually provide.
One big take-away about the process is that you have ample time and space to discuss, refine, and come to some kind of consensus on the proposal; this means that your individual review should be incisive and should stand on its own, but does not need to be perfect. It is likely you will notice things your co-panelists do not, and vice versa. This is good.
The program officer generally facilitates and summarizes but doesn’t advise/directly intervene. I am not sure if I’ve been lucky or if most program officers do this well, but this has consistently been the case for the panels I’ve served on.
Getting to work with and know co-panelists is a positive part of the process. It’s a great opportunity to learn about others and their work and to engage with (perhaps well-established scholars) in a productive/ fun way.
You’re paid!* *A little. You’re paid a stipend of $400, which is great, but probably is quite low relative to the time it takes (or, on an hourly basis). It’s a nice part of the process but isn’t in the top two-three reasons for reviewing, I think. I imagine this is especially the case for more senior scholars, for whom $400 is a smaller proportion of their salary/compensation.
You’ll likely have a chance to talk with the program officer about funding opportunities. You also will have a point of contact afterward to request feedback (such as for a one-page summary of a potential project).
It has universally been a good experience, though with some variation across panels in the reasons why (e.g., some took a lot longer than others - we almost missed dinner for one! :)).
What did I miss?
This is less advice, but I find virtual panels to be a lot more practical; while there are doubtlessly benefits to travel to NSF - and I really enjoyed visiting when I did - it makes a big investment of time and energy a huge one. I hope NSF continues to conduct virtual panels.