TWINSPAN in R

TWINSPANTWINSPAN is perhaps one of the most popular clustering methods (at least among vegetation ecologists), which is not implemented in R. R-sig-eco forum has several posts (mostly from Jari Oksanen and Dave Roberts) on the topic of TWINSPAN in R, where they described difficulties with importing the original TWINSPAN code (written in FORTRAN) into R. Seems that both Jari and Dave spent considerable effort trying to implement the method into R, but seems like there is some problem which is not easy to crack.

From my experience (and I guess also from experience of many others, not only vegetation ecologists), TWINSPAN does sometimes give rather nice and ecologically meaningful results, since it is based on cutting the data along the main compositional gradients. There is yet another divisive method, somewhat analogous to TWINSPAN, called DIANA (DIvisive ANAlysis clustering), which was proposed by Macnaughton-Smith et al. (1964), described in detail by Kaufman & Rousseeuw (1990), and made available in series of Fortran written programs with poetic names like AGNES, CLARA, DAISY, DIANA, or FANNY (later implemented into R package cluster). But it seems to me that this method for some reason never gained so much popularity like TWINSPAN did, mostly perhaps due to its sensitivity to outlying samples (which will be separated first into one-item clusters). This is why I think it still would be nice to have TWINSPAN handy in R, also because of its modified version, which we introduced couple years ago (Roleček et al. 2009) and which up to now is available only in JUICE, program for editing and analysis of vegetation data developed by Lubomír Tichý (Tichý 2002). Modified TWINSPAN is basically a sequence of divisions calculated by standard TWINSPAN, each time applied on the most heterogeneous group – here is again a similarity with DIANA, which also in each step divides the group which is the most compositionally heterogeneous.

Recently, I got a simple idea, how to make TWINSPAN work in R, and how to extend this implementation also for modified TWINSPAN. The way I used is somewhat clumsy, but it seems to work. I used the twinspan.exe file, which is an executable program based on original FORTRAN library written by Mark O. Hill (author of the algorithm and the original FORTRAN code) and compiled in 2003 by Stephan M. Hennekens into a stand-alone program for use in MEGATAB, software which before was used together with TURBOVEG for editing and classification of vegetation data (Hennekens & Schaminée 2001, Schaminée & Hennekens 2001). I created R package twinspanR, which includes twinspan.exe, and added bunch of supporting functions to maintain import and export of data to twinspan.exe and back to R. So basically the TWINSPAN is calculated by the original Hill’s algorithm, and R functions in twinspanR package are for handling the whole thing conveniently in R (see the notes below for more technical details how the library exactly works).

For details how to install the library from GitHub see the Readme.md file. Some more examples how to use twinspan package will be (hopefully) made soon available on my website for analysis of community ecology data in R.

The following is an example code, using example dataset danube with Ellenberg’s meadow dataset (used also as an example in the first publication of TWINSPAN and DCA by Hill 1979 and Hill & Gauch 1980, respectively). In this example I used modified TWINSPAN (Roleček et al. 2009) with division into 4 groups and heterogeneity of clusters measured by Bray-Curtis dissimilarity measure. I projected the results into DCA ordination diagram, along to the original tabular classification made manually by Ellenberg (from Mueller-Dombois & Ellenberg 1974):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
## Modified TWINSPAN on traditional Ellenberg's Danube meadow dataset,
## projected on DCA and compared with original classification into
## three vegetation types made by tabular sorting:
library (twinspanR)
library (vegan)
data (danube)
res <- twinspan (danube$spe, modif = TRUE, clusters = 4)
k <- cut (res)
 
dca <- decorana (danube$spe)
par (mfrow = c(1,2))
ordiplot (dca, type = 'n', display = 'si', main = 'Modified TWINSPAN')
points (dca, col = k)
for (i in c(1,2,4)) ordihull (dca, groups = k, show.group = i, col = i,
 draw = 'polygon', label = TRUE)
ordiplot (dca, type = 'n', display = 'si', main = 'Original assignment\n (Ellenberg 1954)')
points (dca, col = danube$env$veg.type)
for (i in c(1:3)) ordihull (dca, groups = danube$env$veg.type,
 show.group = unique (danube$env$veg.type)[i], col = i,
 draw = 'polygon', label = TRUE)

example twinspan figure

Some technical details how the twinspanR package works

Some further details how it works. The executable file, twinspan.exe, is stored in \exec subdirectory of the library. There is also tw.bat file, which launches twinspan.exe and feeds it with data from R (I have shamelessly stolen this idea from the way how Lubomír Tichý executes TWINSPAN in JUICE, and it is also similar to the way how Tom August implemented another Hill’s FORTRAN library, FRESCALO (Hill 2011), into R as a function frescalo in package sparta). I take compositional data in R, transform them to required *.cc! format (luckily there is a function write.CEP in rioja package, written by Steve Juggins) and save them to \exec subdirectory (where also twinspan.exe is located). Then, I create file tw.dat with input parameters for twinspan.exe (using function create.tw.dat), and use the shell function in R to launch the tw.bat file. All calculations are done using original twinspan.exe, R just reads its output from tw.PUN file. It’s in a no way an elegant approach, but works just fine.

The twinspan.exe file used for calculation of TWINSPAN in R is taken from the distribution of JUICE program (Tichý 2002). It is the original FORTRAN code written by Mark O. Hill, compiled (cca in 2002) by Stephan M. Hennekens into twinspan.exe for use in his program MEGATAB (which was used together with Turboveg for analysis of community data). The version of twinspan.exe used here implements the changes by Petr Šmilauer, related mainly to problems with algorithm convergence, which cause the results being dependent on the order of samples in the input data table. Note that this algorithm is slightly different from TWINSPAN implemented in WinTWINS software (Hill & Šmilauer 2005), which implements also other modifications by Birks and ter Braak. In fact, there are at least four different versions of TWINSPAN recently used for analysis, and in certain circumstances they differ in the results how they classify the samples (when I have time, I will elaborate this topic further).

Unfortunately, the presence of twinspan.exe file in the twinspanR library is a problem for its portability – seems like CRAN doesn’t allow packages with executables inside (for understandable security reasons), and R-Forge allows it to be uploaded, but fails to build it. For now, the channel for distribution of this package is GitHub, and it will perhaps remain there until some other solution will show up. Simply, it’s just a quick and dirty way how to get TWINSPAN functionality in R without too much hassle, before somebody manages to write fully functional implementation of TWINSPAN in R.

References

  • Hennekens S.M. & Schaminée J.H.J. (2001): TURBOVEG, a comprehensive data base management system for vegetation data. Journal of Vegetation Science, 12: 589-591.
  • Hill M.O. (1979): TWINSPAN: A FORTRAN Program for Arranging Multivariate Data in an Ordered Two-way Table by Classification of the Individuals and Attributes. Cornell University, Ithaca, NY.
  • Hill M.O. & Šmilauer P. (2005): TWINSPAN for Windows version 2.3. Centre for Ecology and Hydrology & University of South Bohemia, Huntingdon & České Budějovice.
  • Hill M.O. & Gauch H.G. (1980): Detrended correspondence analysis: An improved ordination technique. Vegetatio, 42: 47-58.
  • Kaufman L. & Rousseeuw P. J. (2009): Finding groups in data: an introduction to cluster analysis (Vol. 344). John Wiley & Sons.
  • Macnaughton-Smith P., Williams W. T., Dale M. B. & Mockett L. G. (1964): Dissimilarity analysis: a new technique of hierarchical sub-division. Nature, 202: 1034-1035.
  • Mueller-Dombois D. & Ellenberg H. (1974): Aims and Methods of Vegetation Ecology. John Wiley & Sons, New York, Chichester, Brisbane, Toronto.
  • Roleček J., Tichý L., Zelený D. & Chytrý M. (2009): Modified TWINSPAN classification in which the hierarchy respects cluster heterogeneity. Journal of Vegetation Science, 20:596-602.
  • Schaminée J.H.J & Hennekens S.M. (2001): TURBOVEG, MEGATAB und SYNBIOSYS: neue Entwicklungen in der Pflanzensoziologie. Ber. Reinhold-Tüxen Ges., 13: 21-34.
  • Tichý L. (2002): JUICE, software for vegetation classification. Journal of Vegetation Science, 13: 451-453.

24 thoughts on “TWINSPAN in R

  1. Attila Lengyel

    Hi David,
    it is nice to see Twinspan finally implemented in R! Once I also tried to write a script for it, however, I was unable to follow all the ‘tricks’ hidden in the original Twinspan code but left out from the descriptions, thus I gave up with it. That’s why it’s great that your programme uses the ‘original’ (or at least a trusted) version.
    Congratulations for your especially useful site!
    Cheers,
    Attila

    Reply
    1. David Zelený

      Hi Attila,
      Thanks for a nice comment! I was working on TWINSPAN in R last Christmas, because I just suddently realized the way how it can be done in R and just wanted to try it – and looks like it works. The package is far from ready, but is somewhat functional – there is still couple of things needed to be finished, like plotting the dendrogram, or printing the indicator values for individual splits. Hope to finish these once. I will be happy, if you use it, if you give me a feedback – especially about things which are not working in a way advertised in the package.

      Have a nice day!

      David

      Reply
  2. Sapphire McMullan-Fisher

    I can’t seem to fine your twinspanR library in CRAN – did I miss a link or is it not yet available?

    Best wishes
    Sapphire

    Reply
  3. Sapphire McMullan-Fisher

    Apologies for previous question. I am new to R and didn’t realise you could get other packages from places other than CRAN – managed to get the package out of github.

    Is there a twinspan users or chat group?

    Best wishes
    Sapphire

    Reply
    1. David Zelený

      Hi Sapphire,

      you are right, the package is only on GitHub, and this is perhaps not going to change any time soon – the problem is that the twinspanR library can be used only under Windows (due to it’s internal dependencies, since the TWINSPAN algorithm is calculated externally using twinspan.exe program), so it cannot be uploaded into CRAN (which requires multiplatform usability of packages). Indeed, this is only one reason – the package is in the developmental stage, and was a kind of experiment – hope to finish in future when I have more time, if I see it can be useful. CRAN is not a good place for packages under development, while GitHub is.

      Cheers!

      David

      Reply
      1. Sapphire McMullan-Fisher

        Thanks for you reply. It may have also answered the question I had when I ran twinspan as only half of it worked with the test data and I am using a mac 🙁

        Are you planning on making it mac friendly at some point? I will see if I can find someone with R on a windows machine.

        Thanks
        Sapphire

        Reply
  4. David Zelený

    Hi Sapphire,

    To write a Twinspan package which can be used across platforms would be great, but currently I don’t plan to do it – it’s different story than my simple solution (to wrap twinspan.exe file and send data into and back). The most recent attempt to rewrite twinspan into R I know is from Dave Roberts, but his solution differs from “genuine” TWINSPAN of Mark Hill, so the results will be more or less different. I also heard somewhere that Jari Oksanen tried to port the original Fortran library into R, but didn’t succeed for some reason, but may be if he has motivation, he may give it a try again. That’s it, sorry for that 🙂

    David

    Reply
  5. kieran

    Hi David,

    Very useful to have an R compatible TWINSPAN! Congratulations and thank you for your efforts.
    I’m interesting in reviewing the hierarchical organization of my data (via twinspan), could you tell me if it’s possible to plot a dendrogram directly from the output? And if not, do you have any tips as to how I might (most efficiently) extract and organize the relevant information from the output provided?

    Thanks, Kieran.

    Reply
    1. David Zelený

      Hi Kieran,

      Unfortunately there is no plotting function in the twinspanR package so far, I haven’t found a handy way how to make it. So far the best guess is to extract the hierarchy from the output and draw it somewhere externally. If you use standard Twinspan, just print the two-way sorted table, and at the bottom of the table you will find the 0-1 sequence determining the hierarchy, e.g.:
      tw.res <- twinspan (danube$spe)
      print (tw.res, what = 'table')

      The same 0-1 sequence is also in twinspan output in the variable “class”.

      If you use modified twinspan, the table output is more complex (there is table for each division), but the “class” output should help:
      tw.res.modif <- twinspan (danube$spe, modif = T)
      tw.res.modif

      order plot.no class
      1 1 1 *1111
      2 2 2 *1111
      3 3 3 *1111
      4 4 4 *1111
      5 5 5 *0000
      6 6 6 *0011
      7 7 7 *0011
      ...

      I should make the dendrogram output, just so far I didn’t found a clever way how to do it…

      Hope this helps

      David

      Reply
  6. kieran

    Hi David,
    Thanks for your comments. So it seems exporting and manual construction is the way to go if I want to create the dendrogram itself. Although I guess I can actually review the hierarchical group structure quite easily (in a piecemeal fashion) by interrogating the binary matrix (with site labels), which would serve my current analytical desires just fine….
    Anyways, thanks again for sorting a quick and easy TWINSPAN option(s) for R and for your subsequent help!

    All the best,
    Kieran.

    Reply
  7. Noah Dell

    David,

    I was wondering if you might know (or else be able to find) what the values for the convergence criteria were set as in the Hill and Šimlauer (2005) version of TWINSPAN. Are the convergence criteria in that version of TWINSPAN and that implemented in twinspanR similar to those suggested by Oksanen and Minchin (1997)?

    Thanks,

    Noah

    Reply
    1. David Zelený

      Hi Noah,

      My guess is that the convergence criteria are the same as in Hill & Šmilauer 2005. The reason for my guess is that twinspanR is using twinspan.exe library, which is currently used in JUICE software, and as I know, Petr Šmilauer couple of years ago was fixing this file, since it gave unstable classification results. But there are some differences between twinspan.exe and Hill & Šmilauer’s Windows program, since the later implements also other tweaks by ter Braak and Birks (I guess), which are not implemented in twinspan.exe (this has different evolution – it comes from original Hill’s fortan-based twinspan, and was originally maintained by Stephan Hennekens for his Megatab software).
      I did some testing, though not really comprehensively, but seems like each version is giving slightly different results (in non standard parametric settings). Problem is that only the original Hill’s algorithm is somewhat properly described (in his report from 1979, which is unfortunately hard to get), the later changes are actually not properly described anywhere.

      Not sure if this helps…

      David

      Reply
      1. Noah Dell

        David,

        It does help quite a bit. I am examining different methods for clustering/classification of vegetation data and was glad to find your beta version of twinspanR. I have since been looking into what is different between versions of TWINSPAN.

        Thanks again,

        Noah

        Reply
    1. David Zelený

      Hi Ivy,
      Don’t worry about Github – you don’t need it on your computer, it’s online repository where the twinspanR library is stored. To install the twinspanR library, first install library devtools, and then twinspanR:

      install.packages ('devtools')
      devtools::install_github("zdealveindy/twinspanR")
      library (twinspanR)

      After this, you should be able to use TWINSPAN in R.
      Cheers!
      David

      Reply
  8. Noah Dell

    David,

    I have encountered an error with twinspanR. Sometimes when I go to use it, I get an error that reads:

    “Error in groups01[,1] = cut(tw.temp[[1]],level=1)-1: number of items to replace is not a multiple of replacement length”

    Is this something you have encountered before? I have looked over the source code and cannot exactly figure out what this means. My guess is that there is something happening between the creation of the empty matrix group01 and the empty list tw.temp and the the reference to tw.temp[[1]], and I presume it is happening in the call to twinspan0.

    Thanks in advance,

    Noah

    Reply
    1. David Zelený

      Hi Noah,

      Wouldn’t you post (or send) some small dataset which is causing this error? Just to be able to reproduce… I will have a look what’s going on…

      David

      Reply
      1. Noah Dell

        David,

        I have sent, in an email, an example data set that has been acting up on me, though it is not the only one.

        Thanks again,

        Noah

        Reply
        1. David Zelený

          Hi Noah,

          I think I found the bug, thanks for sending the data. The problem is actually with row names in your community matrix – the rownames should not contain the white spaces, this will cause trouble in ‘cut.tw’ function. The easiest way to fix is to avoid the spaces at all. I modified the twinspanR library (function twinspan) – I released the twinspanR version v0.16 into GitHub which – in case that the rownames contain spaces will convert it into syntactically valid names using ‘make.names’ function (replace spaces with dots).

          Can you check whether it works with the v0.16 version and let me know? When I tried, looks ok, no error.

          Cheers
          David

          Reply
  9. Noah Dell

    David,

    That works perfectly. I can’t believe I had not thought of that. That makes sense why the program seemed to “think” that it had two different numbers of elements at different points. I even had made modifications to print the output from certain functions and was very confused because it seemed to be working internally.

    Thanks again,
    Noah

    Reply
    1. David Zelený

      Great, happy that it works. Thanks for feedback, it’s great to know that somebody is actually really using that 🙂 This library is just my sentiment, once during Christmas I just by the way found a way how to put twinspan.exe into R folder and tailor all those algorithms to feed it by data and take data out, and cannot get rid of the excitements that it may be actually “twinspan in R” until it was finished. What bugs me most at this moment is that there is not good plot function for the dendrogram, I want to add, but it’s a bit complex…
      Cheers!
      David

      Reply
  10. Jolien D

    Hi

    Thank you for creating this package, but I don’t seem to be able to install it. I’ve tried with the directions you gave with devtools and I’ve even tried downloading the zip and then installing it from the local file, but it always seem to uninstall itself for some reason, when trying to install I always get this error:

    Downloading GitHub repo zdealveindy/twinspanR@master
    from URL https://api.github.com/repos/zdealveindy/twinspanR/zipball/master
    Installing twinspanR
    Installing 1 package: vegan
    Installing package into ‘C:/Users/jolientje/Documents/R/win-library/3.2’
    (as ‘lib’ is unspecified)
    Warning: package ‘vegan’ is in use and will not be installed
    “C:/PROGRA~1/R/R-32~1.2/bin/i386/R” –no-site-file –no-environ –no-save –no-restore
    –quiet CMD INSTALL
    “C:/Users/jolientje/AppData/Local/Temp/Rtmpusp9vO/devtools1f947e2b7e4b/zdealveindy-twinspanR-63e085f”
    –library=”C:/Users/jolientje/Documents/R/win-library/3.2″ –install-tests

    * installing *source* package ‘twinspanR’ …
    ** R
    ** data
    ** exec
    ** preparing package for lazy loading
    Warning: package ‘lattice’ was built under R version 3.2.3
    Warning: package ‘rioja’ was built under R version 3.2.5
    Warning: package ‘betapart’ was built under R version 3.2.5
    Error in loadNamespace(j <- i[[1L]], c(lib.loc, .libPaths()), versionCheck = vI[[j]]) :
    there is no package called 'geometry'
    Error : package 'betapart' could not be loaded
    ERROR: lazy loading failed for package 'twinspanR'
    * removing 'C:/Users/jolientje/Documents/R/win-library/3.2/twinspanR'
    Error: Command failed (1)

    Am I doing something wrong or are my settings wrong?
    Do you have any idea how I can solve this?

    Reply
    1. david Post author

      Hi Jolien,

      Sorry, I overlooked this comment, now I see you posted it already in May, sorry for that! I guess the reply is already not relevant for you, but could be for others running into the same problem.

      From the error message you posted it seems like a problem with R package dependencies, not really twinspanR package. Looks like ‘betapart’ package cannot load the dependency ‘geometry’, for reason I cannot see. Could you try to install ‘betapart’ package first, i.e. type

      install.packages (‘betapart’)

      and then run the twinspanR installation again? If you observe similar error messages, just install the problematic library manually as the betapart library above (e.g. install manually the library ‘geometry’, which was causing the problem in your case).

      David

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *