Drafts, reviews and leaks

Here is an IPCC factsheet summarising how the drafting and reviewing of the IPCC report works, and another one giving further details for the case of AR5 WG1. The timing of the review process can be seen here.

The draft reports

The IPCC WG1 authors wrote a First Order Draft (FOD), and this was reviewed by “expert reviewers” from Dec 2011 – Feb 2012. The authors then revised the report, coming up with a Second Order Draft (SOD), that was reviewed by experts and Government representatives from Oct – Nov 2012.  A final draft was then produced, and the SPM of this final draft was reviewed by Governments only during June and July 2013.  The final stage of the revision process takes place at the meeting in Stockholm this week, 23-26 Sept, where the SPM is gone through line by line and agreed.

Who are the “expert reviewers”?

And why are they in quotation marks? Well, some people advertise the fact that they serve as “expert reviewers” for the IPCC as if this gives them increased credibility. But the fact is, virtually anyone can apply to be a reviewer, and almost all applications are accepted (one of the few exceptions, I believe, is that applications from journalists are not accepted).

I have already written about my experience of reviewing AR5 at the Making Science Public blog.   To register as a reviewer, you give your details and say which sections of the report you would like to review. There is a space for relevant publications, but if you don’t have any, you can leave that blank. You are then asked to tick a box saying that you have the necessary expertise to act as a reviewer.  So the “experts” are self-selected! I was confirmed as reviewer number 948. Anthony Watts was given “a shock”  by being accepted as reviewer 1029.  In total, about 1500 reviewers registered for the FOD, and a similar number for the SOD (a second registration process was needed to review the SOD).

The next stage is that you receive an Invitation Letter that explains the process and states that the draft reports are confidential and may not be cited, quoted or distributed. You have to agree to this, by ticking a box, before you can get access to the draft report.  At this stage, some of those who had applied to be reviewers dropped out (including Steve McIntyre), because they were not prepared to agree to this restriction. If you do tick this box, you get access to the draft report (not just the chapters you expressed an interest in) and you are sent an Excel file to write your comments in.

For the FOD, 659 reviewers submitted a total of 21,400 comments, indicating that the majority of those who signed up either were not prepared to agree to the confidentiality restriction, or simply did not get round to submitting comments – perhaps they just wanted to look at the draft report rather than comment on it. Over the entire review process, over 54,000 review comments were made. Each of these comments has to be responded to by the IPCC authors.  Eventually, all comments and author responses will be made publicly available, although it is not yet clear exactly when this will happen. (For the AR4 report, comments and responses are here at Harvard, in a rather un-user-friendly format.)

Since the draft reports cite research papers that have been accepted but not published, reviewers have the right to see these papers. I requested three such papers and received the following response from the IPCC: “Please find attached a copy of the non-published literature you requested. For security reasons, the attached copy is an encrypted version of a pdf. The copy can be viewed by a software (LockLizard) which is provided free of charge and is simple and quick to download. Below you will find instructions on how to download the software, register the license, and view the protected file.” Take a look at the LockLizard website – especially the video at the top. This gives an insight into the secrecy paranoia of the IPCC. These are research papers on climate science, soon to be published, but in the view of the IPCC they are closely guarded secrets. The Windows-only software does not work on my Linux computer, and when I asked for an alternative format the IPCC refused, so I was unable to see these papers.

The leaks

Given the large number of reviewers, it was almost inevitable that there would be leaks. In fact, it could be argued that it’s surprising that there weren’t more leaks.

The first leak was of an earlier rough draft of the report (not mentioned above), known as the “Zero order draft” (ZOD), see here and here, back in December 2011.

As far as I know, there was no leak of the FOD. The most high-profile leak was that of the SOD, in December 2012 (see also here).  The leaker was Alec Rawls (son of the renowned philosopher John Rawls). He explains the reasons for leaking the report, which are essentially  public interest.  Rawls remarks that the timing of his leak, in December 2012, came after the end of the SOD review process, so it cannot be claimed that the leak interfered with the review process.  When looking at the files, be aware of a slight confusion: there was no SPM in the FOD; the SPM associated with the SOD is confusingly labelled “First Order Draft”, even though it is a summary of the material in the SOD!

More recently, in August there were leaks of the near-final version of the SPM.  It has been suggested by some people that these were deliberate leaks from the IPCC to sympathetic journalists who could be relied on to promote alarm.

My comments

I wrote only about 30 comments on the FOD and 20 on the SOD. I have uploaded my
comments on the FOD and comments on the SOD.
Many of my comments on the FOD criticised the common IPCC trick of comparing a short-term trend with a longer-term trend. This is misleading, since for any signal which fluctuates up and down, short term trends are likely to be larger than long-term trends. For example, for a random walk (which is probably not a good model for the climate), the trend is expected to be inversely proportional to the square root of the length of the time interval.
My comments on the SOD were critical of some exaggerated and misleading claims in the SPM, but welcomed the apparent acceptance of some of my FOD comments on trend lines (you don’t get to see a direct response to your comments, you have to infer from the revised version).

Have other people made their review comments public? If so, where are they to be found?

What’s the point?

Some have raised the question of what is the point in bothering with taking part in the IPCC review process. There have been repeated complaints that valid, critical comments have been ignored by the IPCC authors.
Another potential problem is that the IPCC authors are free to do whatever they like with the final version of the report, for example introducing completely new ideas, claims and graphs that were not in the draft versions and therefore have not been subjected to expert review.

8 thoughts on “Drafts, reviews and leaks

  1. Anthony Watts has posted another leak – a version of the SPM dated 7 June 2013 and labelled “Final Draft”. He has also very kindly linked to my blog (see Pingback). The government review of the SPM referred to above took place from 7 June – 2 August, so this version is prior to the final round of comments from governments. Presumably this is the version that was leaked to the press this summer. Some of the quotes from the newspaper articles seem to match exactly.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s