January 26, 2010 8:50 AM
Should NTIA/RUS Share Scores And Comments With Applicants?
Last week I wrote about how NTIA and RUS were failing stimulus applicants by not communicating with them.
While there has been some good news since then, namely that at least NTIA will be sending out rejection letters, I think we need to be seriously considering an idea I proposed in that post, namely requiring NTIA and RUS to release the scores they gave projects as well as any comments that were written during the review process.
There a number of reasons for doing this.
First, it would provide important feedback to applicants as they figure out what to do for the second round. While first round applicants have been encouraged to reapply, how are they supposed to know what they did wrong in round one without any feedback?
I understand that NTIA/RUS are overwhelmed and don't have time to provide detailed commentary on how applications can improve, but by having them give back the scores and supporting notes that would at least give applicants something to work with. And the result of doing this would presumably be to afford them a greater opportunity to improve their applications for the second round.
While I know that nothing's necessarily easy when you're dealing with hundreds and thousands of applications, I think the relatively small amount of time needed to do this would pay off huge dividends in the end and would be worth the effort.
The second major reason for enforcing this transparency is so that the public can have some sense that the review process is actually working. This would help both by providing a soft audit of this review process to make sure things aren't totally broken, and at the same time it would help with the PR challenge NTIA/RUS face. Right now the only people who believe the process is working is those who've received money. Everyone else thinks it's broken. If they share the scores and notes, they can show that it is in fact working.
Another potential side effect of doing this is that it may help to shame bad actors from reapplying in the second round. Let's be real, there are a lot of bogus projects that applied for money. But if they're shown to have embarrassingly low scores in the first round, especially if the scores are accompanied by notes ridiculing what they're proposing, then that might help keep them away from asking for money again. I know this may not happen as if you applied for money you don't deserve then you likely don't have much of a moral compass to begin with, but this is another example of the positive effect that being transparent can bring.
I honestly don't see much of a downside to doing this. Sure there are questions to answer, like should all scores be made available to everyone, or should they only send applicants their own scores? (I'd love to see the information available to all.)
And the one caveat is that by showing the scores you're likely going to end up with some applicants (rightly or wrongly) questioning their total and making a stink about it.
But, assuming that the review process is working, then these are issues that can be overcome and ultimately I think embracing transparency will be a net positive for NTIA/RUS.
So I throw the gauntlet down to NTIA and RUS. Are you willing to step up and use this as an opportunity to dispel the many doubts around what you're doing?
And to the White House and Congress, you all are starting to talk a big game regarding open government, well here's an opportunity to live up to that ideal. We might not be able to get NTIA and RUS to take this request to be transparent seriously as they have a lot of other things on their plate right now. But if not then we need you all to lean on them to make sure this happens.
If you truly believe that by opening up government we can improve how it serves the people of the United States, then here's a ball for you to take and run with.
Maybe I'm missing something about all this that would make opening up scores and notes a bad idea. If you can think of anything, add a comment below about what it is as I'm always open to being wrong.
But my sense right now is that NTIA/RUS have the opportunity to realize a big win with minimal effort. So hopefully this idea can gain the traction needed to become a reality.
Comments (4)
This is an easy one: of course they should share this information. A rejection letter tells one nothing. The information upon which the decision(s) were based is what the erstwhile applicants, and the People need--and deserve. Otherwise, how will they learn from the experience? They put a lot of time and effort into these applications, under nearly impossible time constraints. A full, by which I mean non-redacted, reporting of the data upon which the decisions were made = the least one should expect from an administration claiming commitment to transparency and accountability in Government.
Posted by Sara Wedeman on January 26, 2010 12:08 PM
The Neighborhood Stabilization Program 2 is doing exactly what you have suggested. Rejected applicants can obtain a review of their applications. Why should this aspect of the stimulus program be different?
I can see the benefit of restricting detailed analysis of individual applications to the applicant but a summary of all application deficiencies and sufficiencies would be valuable to all.
Posted by Gary McGoffin on January 26, 2010 12:10 PM
Hi Geoff,
I am in total agreement with you. As someone who helped on a couple applications, it would be great to get feedback. Doing so would be very helpful and would alleviate my need to try and contact a live person at NTIA to obtain this feedback.
Another approach would be to provide highlights of why the funded projects got funded. What are the characteristics of those projects that stood apart from the rejected applications? Is there a cost-benefit ratio, defining community need, quality of partnership or other factor that raised the level of these applications.
The new NOFA refers to the ranking of applications in a clearer sense than NOFA 1. I chuckled at the notion that applications that meet zero of the program purposes will rank behind those that meet one of seven purposes. Clearly with the competition for these funds, it is unlikely to get funded if you do not meet at least half of the purposes. NTIA could save itself from some work by clearly stating that in the NOFA.
Clearly stating what makes a winning application would motivate many people to seek other funding for those projects.
Posted by Bill Coleman on January 26, 2010 12:12 PM
There should be no reason why proposers should not receive a detailed analysis of their bids, considering the fact that three reviewers were involved on the first NOFA and at least two will be on the second. The process in many cases got complicated because incumbents challenged many of the unserved and underserved claims of proposers. As has been pointed out, the rejection letters are being sent out two weeks before proposers can respond to the second NOFA. I am sure they would be helped by receiving as detailed an account of why they were rejected as possible so this could help in the second round. Yes lets make as much of this as transparent as possible as the Administration has promised, although I am concerned that the same sausage making that goes into laws could appear in the awards process.
Posted by Dave Chaffee on January 26, 2010 12:24 PM