0.26 Underpinnings of the social edition  (Page 20/32)

5. moving into full prototype development: new directions

5.1. rebuilding

Our rebuilding process was primarily driven by the questions generated from our earlier proof of concept. The proof-of-concept pointed us toward a web-based user interface to meet the needs of the research community. Building human knowledge into our application also becomes more feasible with a web environment, since we can depend on a centralized storage system and an ability to easily share information. The proof-of-concept also suggested that we rethink our document storage framework, since exponential slow-downs in full-text searching speed quickly render the tool dysfunctional in environments with millions of documents. For long-term scalability a new approach was needed.

In order to move into full prototype development, we were first required to rebuild the foundation of both REKn and PReE applications, as outlined in detail in the previous section. To summarize:

1. We are rebuilding the PReE(?) user interface. A web-based environment allows us to be agile in our development practices and to quickly incorporate emerging ideas and visions.
2. The Ruby programming language has been selected as the new development platform. While it can be considered the “new kid on the block” of web-scripting languages, the benefits it offers (such as the Ruby on Rails application framework) make it an enticing choice to say the least. The use of Ruby on Rails offers a rapid prototyping environment, which cuts huge chunks of development time out of our overhead. Ruby on Rails also provides us with the ability to add “Web 2.0” user interface features to our project simply and easily.
3. We are working on developing a “one-stop” administrative interface for harvesting and processing new documents. Rather than having bits and pieces scattered around, we propose to use an extensible model for adding processing abilities to our application. Once the model has been built, the processing of a new type of document will simply require the addition of a new plug-in to bring the document into the application.
4. We decided to keep the relational database for application-specific data needs (such as user info and user created content) in addition to implementing a dedicated full-text indexing engine to search both the text and the associated metadata. An application that offers time-efficient full-text searchability of documents is greatly valued by its users. To this end we decided to enlist the use of the “granddaddy” of open-source full-text indexing engines, Lucene. Lucene gives us fast, robust and scalable full-text searching. The Solr layer on top of Lucene allows us to “talk” to Lucene from any programming language we choose and give it powerful additions such as basic text analysis and the ability to uniquely identify a document. While Fedora Commons might prove to be a better alternative to Solr, the switch will have to wait until such time as the Fedora GSearch tool has been built into the RubyFedora library.
5. We are working toward centralizing document processing. Until now, a different stand-alone tool processed each style of document. We are planning to pull all of these tools together in one place and to allow new tools to be added easily, with the facility for administrators to go through the process of adding new documents into the knowledgebase attached to PReE.
6. We are rebuilding the interconnections between PReE and other related community tools. From metadata lookup tools to applications providing data analysis, the next development of PReE will be designed with flexibility and long-term scalability in mind.