Comparing temperature data sets

Introduction
One of the tactics of climate deniers to cast doubt on climate science is to attack the accuracy of the atmospheric temperature record. See this post for discussion of these types of attack.

There are five main groups that produce global temperature time series. Three of the groups use data from surface weather stations - GISS, Hadley and NOAA - and two of them use microwave measurements from sattelites - UAH and RSS.

The attacks on the temperature record are invariably aimed at the surface temperature groups and rarely or never at the satelite groups. This provides us with a simple method of checking the claims, rather than reading the many links in the previous post.

If the surface measurements are significantly different to the satelite ones then maybe the climate deniers have a case.

All of the five groups freely provide their data on the Internet. They can be found at the following locations: GISS, Hadley, NOAA, UAH and RSS

Charting the data

I downloaded all five sets of data and charted them using the statistical programming language R. The resulting chart can be seen below:


Figure 1: All five atmospheric data sets presented on one chart - with data unadjusted
At first sight this chart gives some support for the denier claim, as the surface measurements show a warmer world than the satelite measurements. On closer analysis though it is clear that the temperature trend lines are quite similar, just offset from eachother. Note that the temperature values (on the Y-axis) are in fractions of a degree. They are temperature anomalies not absolute temperatures.

The following explanation for the way the anomalies are calulated comes from this post, by Tamino:

Anomaly is the difference between temperature at a given time, and the average temperature for the same time of year during some reference period. So temperature anomaly doesn’t really tell you, in absolute terms, how hot or cold it is — it tells you how much hotter or colder it is, than it was (on average) during the reference period.

It is important to realise that the five different temperature groups develop their data using four different reference periods:
* GISS : 1951 to 1980
* Hadley : 1961 to 1990
* NOAA : 1901 to 2000
* UAH and RSS : 1979 to 2000
These diffeent periods had differnet average temperatures. GISS and NOAA are the coolest (roughly the same), followed by Hadley and then the satelite groups (UAH and RSS) the warmest periods. As GISS and NOAA are calculating their anomalies by comparing with a the coldest of the reference periods is would be expected that they would show the greatest warming, followed by Hadley and then the satelite groups. This is exactly what can be seen in the chart above.

The appropriate way of placing all of the temperature groups on the same chart is by computing the anomolies for each group using a common reference period. I chose the reference period of the satelite groups as I would only have to re-calculate the anomolies for the surface groups - ie only three sets of calculations.

The method of computing the new anomolies was to calculate the mean anomoly for each of the temperature series during the chosen reference period and subtract it from each of the monthly values. I saved this data as a new file.

Figure 2 below shows the data charted. Note that the spread is much reduced and that the trend lines are quite close together and are relatively similar.


Figure 2: All five atmospheric data sets presented on one chart - with data adjusted to account for different reference periods

My confidence with the accuracy of figure 2 increased when I compared it with this chart that was produced by Tamino.

Both charts are crowded and therefore a little difficult to read, so I decided to produce charts just comparing two of the data sets at a time. Figure three compares the two satelite data sets - RSS and UAH.


Figure 3: RSS and UAH since 1979

These use the same data so it is not surprising that they are relatively similar - the differences can be attributed to different processing of the data. The main diffeence is in the trends - RSS has a steeper trend than UAH.

One of the claims about the CRU during the email controversy was that the CRU scientists were fudging the data to enhance the warming trend. If this were true you would expect that the Hadley data would have a significantly warmer trend than the satelite data. Figure 4 compares the RSS and Hadley data sets after adjusting them to the same reference period. What is most interesting about this chart is the similarity of the data with virtually identical trends! Note that these two groups use fundamentally different methods of calculating their data sets. If Hadley are fudging their data and processes, RSS must be too!


Figure 4 : RSS and Hadley data sets compared when adusted to the same reference period

Figure 5 shows that Hadley has a stronger warming trend than UAH. This is hardly a surprise in the light of the two previous figures.


Figure 5: compares Hadley with UAH adusted to the same reference period.

Lastly, how does Hadley compare to GISS? Figure 6 shows that GISS has a stronger warming trend. This is largely due to the fact that GISS attampts to take account of the Arctic (which is warming very strongly) while Hadley ignores the Arctic.


Figure 6: hadley compared to GISS (same reference period)

A Skeptic Agrees

Roy Spencer, who with John Chrisite produces the UAH satelite data, agrees that there is very little difference between his data and that of HadleyCRU. In the quote below he discusses that attacks on Phil Jones and the Hadley data:

"He says he's not very organised. I'm not very organised myself," said Professor Spencer. "If you asked me to find original data from 20 years ago I'd have great difficulty too.

"We just didn't realise in those days how important and controversial this would all become - now it would just all be stored on computer. Phil Jones has been looking at climate records for a very long time. Frankly our data set agrees with his, so unless we are all making the same mistake we're not likely to find out anything new from the data anyway."

Conclusion

It is clear from this analysis that the temperature data sets are quite similar, showing warming over the last 30 years. The claims that the surface temperature data sets are manipulated to exaggerate the warming trend are clearly not supported by the evidence.

UPDATE: 16/12/2010
Tamino has written an updated post on this topic at this link.
_______________________________________________________________

Post Script
It is not necessary to download data and begin to learn charting with R to analyse the temperature data sets. Charts can be easily generated at the Wood for Trees website.

Some of my confidence in the charts that I generated for this post comes from the charts that I produced at Wood For Trees, which are virtually identical.

To view my Wood for Trees charts click on the links below:

UAH Vs RSS

Hadley Vs RSS

Hadley Vs UAH

Hadley Vs GISS

Post Post Script

Here are some of the scripts that I wrote to produce the charts.

Figure 2

R Scripts

########### All Temp time series since 1979 adjusted ##############

## STEP 1: SETUP - Source File
par(las=1)
link <- "C:\\Learn_R\\All_temp_groups_ADJUSTED.csv" ## STEP 2: READ DATA my_data <- read.table(link, sep = ",", dec=".", skip = 0, row.names = NULL, header = T, colClasses = rep("numeric", 2), na.strings = c("", "*", "-", -99.99,99.9, 999.9), col.names = c("yr","GISS","HAD","NOAA","UAH","RSS")) ## STEP 3: MANIPULATE DATA Title <- "GISS, RSS, HAD, UAH, NOAAA Temp series since 1979 adjusted" ## Ceate Plot plot(GISS ~ yr, data = my_data, type="l", col = "red", xlab = "year", ylab = "temp", main = Title) points(RSS ~ yr, data = my_data, type="l", col = "darkblue") points(HAD ~ yr, data = my_data, type="l", col = "green") points(UAH ~ yr, data = my_data, type="l", col = "black") points(NOAA ~ yr, data = my_data, type="l", col = "slategrey") ## Insert text annotations text(1980,0.65, "GISS", col="red") text(1980,0.6, "RSS", col="darkblue") text(1980,0.55, "HADLEY", col="darkgreen") text(1980,0.5, "UAH", col="black") text(1980,0.45, "NOAA", col="slategrey") ## Plot trends abline(lm(GISS ~ yr, data = my_data), col = "red") abline(lm(RSS ~ yr, data = my_data), col = "darkblue") abline(lm(HAD ~ yr, data = my_data), col = "darkgreen") abline(lm(UAH ~ yr, data = my_data), col = "black") abline(lm(NOAA ~ yr, data = my_data), col = "slategrey") Figure 4

#################### Hadley and RSS temp calc adjusted #################

## STEP 1: SETUP - Source File
par(las=1)
link <- "C:\\Learn_R\\All_temp_groups_ADJUSTED.csv"

## STEP 2: READ DATA
my_data <- read.table(link,
sep = ",", dec=".", skip = 0,
row.names = NULL, header = T,
colClasses = rep("numeric", 2),
na.strings = c("", "*", "-", -99.99,99.9, 999.9),
col.names = c("yr","GISS","HAD","NOAA","UAH","RSS"))
## STEP 3: MANIPULATE DATA
Title <- "RSS and Hadley Temp series since 1979 (Hadley adjusted)"

## Ceate Plot
plot(GISS ~ yr, data = my_data, type="l", xlim = c(1979,2010 ), ylim = c(-0.4,0.8 ), col = "white", xlab = "year", ylab = "temp", main = Title)
points(RSS ~ yr, data = my_data, type="l", col = "blue")
points(HAD ~ yr, data = my_data, type="l", col = "green")

## Insert text annotations
text(1980,0.4, "HAD", col="green")
text(1980,0.5, "RSS", col="blue")

## Plot trends
abline(lm(RSS ~ yr, data = my_data), col = "blue")
abline(lm(HAD ~ yr, data = my_data), col = "green")

Attacks on the atmospheric temperature record

There have been at least three ways that climate deniers have attempted to invalidate the atmospheric temperature record.

1. Claiming that weather stations are placed in locations that produce warming

The main proponent of this view is Anthony Watts at his surfacestations.org site.

Watts uses photographs in an attempt to demonstrate his claim but two papers have been written analysing the US temperature record that completely demolish Watts' claims. Discussion of the papers can be found here and here.


2. Claiming that the stolen CRU emails prove that climate scientists have deliberately manipulated the temperature record to exaggerate the warming trend.

Here is one of many examples:

Perhaps the most infamous example of this comes from the "hide the decline" email. This email initially garnered widespread media attention, as well as significant disagreement over its implications. In our view, the email, as well as the contextual history behind it, appears to show several scientists eager to present a particular viewpoint-that anthropogenic emissions are largely responsible for global warming-even when the data showed something different.

The climate scientists were actually discussing a technical issue in dendrochronology (study of tree rings)called the divergence problem. See here and here for details. Climate deniers who mistakenly believe that the planet has been cooling over the last decade or so, quickly assumed that the scientists were trying to hide this, nonexistent, temperature decline.

There have been three enquiries resulting from the campaign of the deniers against the CRU scientists. All three have exonerated the scientists involved, see here, here and here. What is the denier response to this thorough debunking of their claims? You might expect that honest people would apologise in such circumstances, but that is not the denier way. The response to the exhonerations has been to claim that the people making the reports are part of the conspiracy, as demonstrated in this post, by Australia's foremost climate denier. Such people believe in conspiracies wrapped up in conspiracies, and people treat such twaddle seriously. Weird!

UPDATE (7/05/2010)
Stephan Lewandowsky has written an interesting article on conspiracy theories including the truthers and AGW deniers. The most interesting thing about Lewandowsky's article is the number of conspiracy nuts who have come out in the comments.

UPDATE (21/05/2010)
Dr Roy Spencer, runs the UAH satelite temperature record. Spencer is a well known climate skeptic. One of the major targets of the attacks on scientists and the temperature record is Phil Jones, who runs the HadleyCRU temperature record. At a recent Heartland Conference Spencer defended Jones in the following terms:

"He says he's not very organised. I'm not very organised myself," said Professor Spencer. "If you asked me to find original data from 20 years ago I'd have great difficulty too.

"We just didn't realise in those days how important and controversial this would all become - now it would just all be stored on computer. Phil Jones has been looking at climate records for a very long time. Frankly our data set agrees with his, so unless we are all making the same mistake we're not likely to find out anything new from the data anyway."

(Source here paragraphs 3 and 4 after the heading Cloud Effect)



UPDATE (09/07/2010)
The final two reports on the climategate "scandal" are now in, see here and here. There have now been five enquiries about the CRU email issue. All five have exonerated the scientists involved and pointed out the untruths and misrepresentations put around by the global warming denial industry. Will we see the climate deniers retract their untrue claims and apologise for their mistakes - I wouldn't hold my breath!

UPDATE (22/11/2010)
There have been many posts on the anniversary of the publishing of the stolen emails; for examples see posts here here.

UPDATE (27/08/2011)
The National Science Foundation (NSF)has conducted a very thorough review of the allegations about the behaviour of Michael Mann (of hockey stick fame). In the report that can be found here he was completely exonerated from any implication of impropriety or research misconduct.

For more evidence on the similarity of the various temperatrue records see this post.

3. Claiming that weather stations in colder locations have been deliberately removed from the temperature record

Watts and D’Aleo make the following claim in this document on page 9.

Around 1990, NOAA lost more than three-quarters of the climate measuring stations around the world.
It can be shown that country by country, they lost stations with a bias towards higher-latitude, higher-altitude and rural locations, all of which had a tendency to be cooler.

This claim has been widely debunked here, here, here, and here.

At this link Roy Spencer states:

... at face value, this plot seems to indicate that the rapid decrease in the number of stations included in the GHCN database in recent years has not caused a spurious warming trend in the Jones dataset — at least not since 1986.


All of these attacks on mainstream climate science show strong elements of paranoid conspiracy theorising. It is clear that the vast majority of people qualified to make a judgement accept that the planet is warming and that most of the warming in the last 50 years has been induced by human activities - anthroprogenic global warming (AGW). For evidence of this claim see this post. The making of paranoid claims that the climate scientists are conspiring to distort reality for left wing political aims, is one of the few responses available to those who want to deny the reality of AGW. The response to the official exhonerations of the CRU scientists shows that deniers not only believe in conspiracies but conspiracies wrapped up in conspiracies.

In more rational eras the crazy claims of the climate deniers would be laughed at by most people, who would then move on to more productive pursuits, but irrationality seems to be rife in the world at the moment. See here and here for some ideas on the current irrationality.

It is clear from an analysis of the climate data that the five main temperature data sets are quite similar, when properly analysed, see this post for details. The Hadley / CRU time series sits squarely in the middle of the range of results. If the CRU scientists had been deliberately manipulating the data to enhance the warming then the Hadley / CRU data would be an outlier. Clearly this is not the case.

A practical Turing Machine

One of my heros is Alan Turing.

Turing played an important role in the deciphering of the German Enigma codes during WWII, and as a result saved the lives of many people directly involved in the prosecution of the war.

He was a mathemetician who played an important role in the development of computers.

One of his seminal ideas was of the Turing Machine, that demonstrated the theoretical possibility of stored program computing.

The video below shows a modern implementation of the Turing Machine.

Climate and charting with R

For many years I have been using Excel to analyse data and produce charts. Excel charts, though, are ugly.

Many of the Climate Science sites that I visit have very clear and attractive charts of temperature trends to illustrate the arguments presented in posts. Upon investigation I have discovered that many of these superior charts are produced in a statistical analysis program called R.

R is open source software and can be downloaded free from this site.

This post discribes:
* the five groups that produce temperature data
* downloading temperature data to a computer
* importing temperature data into R
* manipulating the data in R
* producing charts in R

There are five main groups of researchers that produce global temperature records. Three use temperatures measured at the surface: Giss, Hadley and NOAA; and two use satellites: UAH and RSS

The temperature data can be found at the following locations: Giss data, Hadley data, NOAA data, UAH data and RSS data.

These data sets require some manipulation before they can be imported into R. The spaces have to be removed and the data separated by commas. I remove the spaces in word and insert a tab character between the pieces of data, using the find ... replace function. I import the data into Excel and export it in comma delilited (CSV) format.

Here are the first two years of the Hadley data after this manipulation:
year,anom
1850.042,-0.691
1850.125,-0.357
1850.208,-0.816
1850.292,-0.586
1850.375,-0.385
1850.458,-0.311
1850.542,-0.237
1850.625,-0.34
1850.708,-0.51
1850.792,-0.504
1850.875,-0.259
1850.958,-0.318
1851.042,-0.345
1851.125,-0.394
1851.208,-0.503
1851.292,-0.48
1851.375,-0.391
1851.458,-0.264
1851.542,-0.279
1851.625,-0.175
1851.708,-0.211
1851.792,-0.123
1851.875,-0.141
1851.958,-0.151

The strange decimal numbers represent the months of each year and ensure that the months are equally spaced in a chart.

The surface temperature data go back to the 19^th century. I produced the chart below from the Hadley data:



The red line is the monthly data. The black line is the line that best fits the data (regression line).

Here is the script that generates the chart above:

#################### Hadley temp average calc #################
## STEP 1: SETUP - Source File
par(las=1)
link <- "C:\\Learn_R\\Hadleydata\\Hadley_since_1850.csv"

## STEP 2: READ DATA
my_data <- read.table(link,
sep = ",", dec=".", skip = 0,
row.names = NULL, header = T,
colClasses = rep("numeric", 2),
na.strings = c("", "*", "-", -99.99,99.9, 999.9),
col.names = c("year", "temperature"))

## STEP 3: MANIPULATE DATA
Title <- "Hadley Temperature Chart and Linear Regression since 1850"

## STEP 4: Ceate Plot
# Plot temperature data
plot(temperature ~ year, data = my_data, type="l", col = "red", main = Title)
# Calculate regression
lm_fit <- lm(temperature ~ year, data = my_data)
# Display regression stats in R Console
summary(lm_fit)
# Plot regression line
abline(lm(temperature ~ year, data = my_data ))

############################################################

The par(las=1) command ensures that the Y-axis values are vertically aligned

The link <- "C:\\Learn_R\\Hadleydata\\Hadley_since_1850.csv" assignes file and path to the link vector (variable). I store the data files on C: drive in a sub_folder of the folder Learn_R.

Step 2 reads the data file into a variable (data frame) and performs some formatting.

Step 3: adds the title for the chart.

Step 4:
* Plots the data (with the plot command)
- temperature against year
- indicates the data to be plotted (data = my_data)
- sets the chart type to line (type="l")
- sets the colour of the line to red (col = "red")
- inserts the title from Step 3 (main = Title)
* Calculates the regression statistics
- with the command : lm_fit <- lm(temperature ~ year, data = my_data)
- writes the regression statistics to the R console (summary(lm_fit))
* Uses the regression information to plot a line of best fit
- with the command : abline(lm(temperature ~ year, data = my_data ))

Most of this script comes from D. Kelly O'Day's site at Climate Charts & Graphs. O'Day's site is a very useful introduction to the R language.

The link R Resources on the site provides links to a wide range of useful materials.

Here are some more charts that I plotted:









More charts can be seen at this link.

Sam Harris: Science can answer moral questions

The Danger of Science Denial

We are more prosperous and more educated than ever before. Most of the good life we live comes from technology that is based on scientific discoveries. A rational response would be a continuing respect for the discoveries of science, but the reverse seems to be happening. There are many movements aimed at undermining our undestanding of the world that has been developed by science. Some examples are:
* Climate denial
* Vaccine denial
* Opposition to Genetic modification of food
* HIV denial
* Alternative medicines
* Young Earth creationism
* Opposition to evoltionary theory
* Moonlanding hoax
* Belief in psychics and other paranormal phenomena
* UFOs

The first five examples in the list are causing deaths or will cause deaths in the future. The others, might not be as directly dangerous, but they undermine acceptance of science and rationality which is a more subtle but still serious danger.

The video presentation by Michael Specter discusses some of these issues. A brief bio of Specter can be found below the video. Two of my favourite quotes from the video are:

Be sceptical, ask questions, demand proof. Demand evidence. Don't take anything for granted. But here is the thing: When you get proof, you need to accept the proof. And we're not that good at doing that.

You know, science isn't a company. It's not a country. It's not even an idea. It's a process. It's a process. And sometimes it works, and sometimes it doesn't. But the idea that we should not allow science to do its job because we're afraid, is really very deadening and it prevents millions of people from prospering.

I also laughed with the audience when Specter described the alternative medicine industry as "big placebo".



Here is Specter's bio from the TED site:

Michael Specter's new book, Denialism: How Irrational Thinking Hinders Scientific Progress, Harms the Planet and Threatens Our Lives, dives into a worrisome strain of modern life -- a vocal anti-science bias that may prevent us from making the right choices for our future. Specter studies how the active movements against vaccines, genetically engineered food, science-based medicine and biotechnological solutions to climate change may actually put the world at risk. (For instance, anti-vaccination activists could soon trigger the US return of polio, not to mention the continuing rise of measles.) More insidiously, the chilling effect caused by the new denialism may prevent useful science from being accomplished.

Specter has been a writer for the New Yorker for more than a decade; before that, he was a science writer and then the Moscow bureau chief for the New York Times. He writes about science and politics for the New Yorker, with a fascinating sideline in biographical profiles.

I particularly like this Specter quote, which isn't in the video:

"Denialism is a virus and viruses are contagious."

Strange Beliefs

In this video Michael Shermer gives examples of strange beliefs that people hold.

Is the Earth past the tipping point?

I have written plenty of posts about the causes and danger of climate change. My climate change posts can be found at this link.

There are many other environmental threats that are largely caused by human activities.

The University of Minnesota's Institute on the Environment has listed these environmental threats as follows: Biodiversity loss; Land use; Freshwater use; Nitrogen and phosphorus cycles; Stratospheric ozone; Ocean acidification; Climate change; Chemical Pollution; and Aerosol loading in the atmosphere.

The video below notes a large range of human caused environmental threats:



Below is the introduction to a Scientific American article written by scientists at the University of Minnesota's Institute on the Environment.

* Although climate change gets ample attention, species loss and nitrogen pollution exceed safe limits by greater degrees. Other environmental processes are also headed toward dangerous levels.
* Promptly switching to low-carbon energy sources, curtailing land clearing and revolutionizing agricultural practices are crucial to making human life on Earth more sustainable.

For nearly 10,000 years—since the dawn of civilization and the Holocene era—our world seemed unimaginably large. Vast frontiers of land and ocean offered infinite resources. Humans could pollute freely, and they could avoid any local repercussions simply by moving elsewhere. People built entire empires and economic systems on their ability to exploit what seemed to be inexhaustible riches, never realizing that the privilege would come to an end.

But thanks to advances in public health, the industrial revolution and later the green revolution, population has surged from about one billion in 1800 to nearly seven billion today. In the past 50 years alone, our numbers have more than doubled. Fueled by affluence, our use of resources has also reached staggering levels; in 50 years the global consumption of food and freshwater has more than tripled, and fossil-fuel use has risen fourfold. We now co-opt between one third and one half of all the photosynthesis on the planet.

The article can be found here.

Here is another video by the same scientific group:



The Stockholm Resiliance Centre is also working on the issue of Planetary Boundaries.

Here are two videos of Johan Rockström explaining their approach to Planetary Boundaries:





UPDATE: 5/9/2010

Rockström also gave a talk at TED which can be found here

Tha aim of the research is to define boundaries within which humanity has the flexibility to choose pathways for our future development and well-being. This is a first map of the planet's safe operating zones.

The scientists first identified the Earth System processes and potential biophysical thresholds, which, if crossed, could generate unacceptable environmental change for humanity. They then proposed the boundaries that should be respected in order to reduce the risk of crossing these thresholds.

Nine boundaries identified were climate change, stratospheric ozone, land use change, freshwater use, biological diversity, ocean acidification, nitrogen and phosphorus inputs to the biosphere and oceans, aerosol loading and chemical pollution.

The study suggests that three of these boundaries (climate change, biological diversity and nitrogen input to the biosphere) may already have been transgressed. In addition, it emphasizes that the boundaries are strongly connected — crossing one boundary may seriously threaten the ability to stay within safe levels of the others

.
Source: Stockholm Resiliance Centre

Note that both groups identify the same nine boundaries.

The boundaries are described in more detail here.

The group's article in Nature can be found here.