• Editorial
• Open Access

• Received: 22 September 2010
• Accepted: 22 September 2010
• Published: 1 October 2010

Recently, Kuipers et al.[1] reported an exciting development: the first disease-focused, locus-specific mutation database (LSDB), developed through the automated extraction of mutation data from full-text articles taken from the scientific literature. Although this report clearly constituted a landmark in its field, neither the idea nor the methodology is entirely new, since Horn et al.[2] successfully extracted point mutation data reported (mainly in an evolutionary context) in G protein-coupled receptors and nuclear hormone receptors from full-text literature some years ago. Since then a range of different approaches has been attempted, with varying degrees of success [3–6]. Kuipers et al.[1] employed their own search tool, Mutator, however, specifically to extract disease-related missense and nonsense mutation data from the scientific literature, with a view to constructing an LSDB for Fabry disease (FMDB) containing mutations in the α-galactosidase (GLA) gene. Briefly, these authors employed a disease-oriented PubMed keyword search to identify relevant articles from the literature to be downloaded. The full text of relevant publications was then automatically screened for mutation data, and those mutations found to occur in amino acid residues that appeared to match the GLA protein sequence were selected for inclusion in the database. Although the results appear at first glance to be impressive, they warrant closer inspection.

Using Mutator, Kuipers et al.[1] identified 367 'unique GLA gene mutations' (listed in their Supplementary Table), 108 of which they claimed to be absent from the Human Gene Mutation Database (HGMD⁷). Despite the fact that the authors failed to evaluate Mutator with respect to 'standard' performance measures (eg precision and recall), their tool appears to be a significant improvement over previously published methods, particularly those that simply screened PubMed abstracts rather than full text [6, 8–12]. In their comparison with HGMD data, however, Kuipers et al.[1] appear to have used the somewhat outdated (but nevertheless freely available) online version of the database (http://www.hgmd.org) rather than the up-to-date subscription version, HGMD Professional (http://www.biobase-international.com/pages/index.php?id=hgmddatabase). We examined the 108 GLA mutations claimed to be absent from HGMD, and found that 48 were actually present in HGMD Professional. Of the remainder, seven were listed in HGMD under an alternative mutation type (eg small indels), four still remained unresolved (in that the precise nature of the nucleotide change was unclear or ambiguous in the original report), while 24 were non-Fabry disease false positives (see below). The remaining 25 mutations (~7 per cent of the total number of bona fide GLA mutations), reported in 12 different papers, appear to have been inadvertently omitted by HGMD. This is most likely because (i) they were mentioned only briefly within the text of the article concerned and (ii) no hint of the articles' mutation content could have been gleaned from inspection of article titles, abstracts or keywords. False negatives appear to be less of a problem than false positives, with Mutator failing to recognise only nine of the GLA mutations logged by HGMD Professional.

The success of Mutator in identifying these hitherto latent lesions certainly testifies to the future potential utility of automatic tools designed to search for and extract mutation data directly from full-text articles. Indeed, we are currently exploring ways of incorporating automated full-text searching into HGMD's mutation identification strategy. In terms of the currently available version of the Mutator program, however, there would appear to be a significant problem with false positives. As mentioned above, when we carefully examined the 108 identified GLA mutations that were claimed to be absent from HGMD, 24 entries (22 per cent) proved to be non-GLA/non-Fabry disease false positives. Hence, even the insertion of a step into the search program which was designed to check that all identified mutations matched the GLA protein sequence had not prevented Mutator from identifying spurious mutation data (eg mutations in other genes located within article titles cited in the reference list that coincidentally matched the GLA protein sequence etc.). Taken together, it would appear that considerable work remains to be done in order to optimise the performance of the program, even for the missense/nonsense mutation category that constitutes the specific target mutation type for Mutator. As it stands, a significant amount of manual validation and curation would still need to be performed in order for the GLA mutation dataset to be reliable in terms of its content.

Assuming that these initial difficulties can be overcome, what is the potential of this approach in terms of scaling up the extraction of mutation data for the > 3,800 human genes [13] currently known to harbour mutations causing and/or associated with human inherited disease? On the basis of the report of Kuipers et al.,[1] we would say that the long-term prospects are likely to be very good, assuming that the search criteria can be suitably refined, using Boolean parameters, so as to exclude the false positives. As the authors discovered, however, each gene/protein is likely to have its own particular false positives to contend with. Thus, the search for GLA mutations also pulled out mutations in the Gla (γ-carboxyglutamic acid) domains of various coagulation factors. It follows that significant effort must be devoted to avoiding such false positives on a gene-wise basis. In this context, it is pertinent to point out that there are likely to be a number of categories of mutation that will be very difficult to identify correctly (or alternatively to weed out) in an entirely automated fashion. These are likely to include:

For all the above-mentioned reasons, it is, at present, hard to see how the automated collation of mutation data can be wholly accurate and reliable without the subsequent deployment of labour-intensive manual validation and curation steps. Even as it stands, however, it is evident that the automated data extraction tool described by Kuipers et al.[1] is likely to represent a very valuable adjunct to the semi-automated data-collation procedures currently employed by both the LSDBs and HGMD. The authors are therefore to be congratulated for the remarkable degree of success that their Mutator tool has already achieved at the pilot stage of its development. While the prospect of next-generation tools for the extraction and validation of mutation data is awaited with keen interest, for the time being we concur with the view expressed by Winnenburg et al.[14] and Caporaso et al.[15] that the most cost-effective and reliable approach to mutation data collection and annotation is currently for automated text-mining methods to be integrated into the manual annotation process, and for manual and automated approaches to be used in concert to mine the biomedical literature for pathological gene lesions.