Time to Budget for a Kete in your community for the new financial year

Joann Ransom, Horowhenua Library Trust
Josh Forde, Rabid Technologies

Background

Rabid and Horowhenua Library Trust kicked off the ‘Kete2’ upgrade project in August last year.  We put out a proposal to raise $200k and manage a very uncertain upgrade in a way that would allow Kete collections to be upgraded and modernised. All parties aim to upgrade the project to be modern, usable and in a state that new libraries and community groups can run digital heritage projects with confidence.

Stage one is an upgrade/rebuild phase.  Future phases will need to address reaching a sustainable arrangement for New Zealand users and libraries, and subsequently efforts to expand the reach and impact of the project.

Scope

The project scope of the first phase was to target the largest risks to the technology and mitigate security concerns. Over time, a lack of maintenance funding has meant that certain open-source components have lost use and the project hasn’t maintained small upgrades.  Coupled with a codebase that has had an iterative approach, the architecture underlying the software was often sprawling and a real challenge to comprehend. Many Kete instances are doing some creative things with the features of the system and it has been challenging dealing with the breadth of that. The short version is that scope has been much larger than anticipated but we’ve been making steady progress. To simplify the requirement, the project is focusing on the simple collection and description features that a typical library would require.

Progress Update

Rabid are committed to delivering an upgrade.  What this large increase in scope means is:

• planned development resource needs a lot more time as well as the financial resource that is involved. We have a core team of 2 developers, Rob Ramsay and Eoin Kelly, doing most of the work in the upgrade. We’ve utilised pair programming to get better progress refactoring and ensuring we don’t get stuck on the work.

• where Rabid are putting in this time, they are doing this with a lot of commercial risk.  Rabid are essentially putting in the investment to be able to offer Kete users the ability to upgrade, in the hope that libraries will come on board.

• to date (end of Feb), Rabid delivered a lot of work.  We did a calculation of work to progress:

• the commercial development bill is approximately $80,000 of work against $35,000 to date.

• this number doesn’t take account of the analysis, community meetings, relationship building and sales time Rabid put into the project. They don’t seek to recoup this time but it is a cost to them.

• our current estimates of work remaining are that there may be 6-8 weeks FTE workload to come, potentially more.

• we need to balance this workload against commercial realities.  The Rabid team will continue to work on this, but need to ration that workload going forward.

• The core ‘risk’ of this phase has been worked through now.  There is still a lot of work to be done, but the unforeseeable risks of the upgrade itself are done albeit the cost has been at the pessimistic end of our estimates.  We have got enough confidence to have built user stories level analysis and have agreed milestones ordering what we will do from here to deliver.

• There is however some uncertainty about how long it will take to deliver a functioning demonstration site and upgrade the Horowhenua Kete to provide completion.

WHERE IS THE TECHNOLOGY RIGHT NOW?

Our desire is to simplify and standardise the codebase.  We won’t have a project we’ll be entirely satisfied with technically, but we’ll have confidence that the core application has integrity.  There are future decisions to be made around delivering and supporting Kete in a manageable way, but the essential code will be of reasonable quality going forward.

A brief overview of the work completed :

1. Ruby on Rails 3 upgrade has been completed.

The upgrade of this version necessitates upgrading the dependencies, gems, libraries that functioned for previous Kete. This is an amorphous body of work incorporating understanding the architecture, upgrading the systems and tracing errors through the code.

2. The search engine Zebra has been pulled out of the system. This search engine didn’t get uptake in the Ruby on Rails community but PostGres search features have been perfectly adequate for Kete to date.

3. The database has been changed from MySQL to PostGreSQL.  We communicated an intention to use MongoDB, which is the database of National Libraries.  When we upgraded, we found that PostGreSQL met our needs without the added complexity of Mongo.  PostGreSQL has recently become a strong de facto database, being resilient, flexible and widely supported.  This doesn’t mean versions of Kete wouldn’t work with MongoDB but there are more pros for PostGreSQL than reasons to use a NoSQL solution.

4. We have migrated data from Kete Horowhenua in our development.  This will show that the first Kete has been upgraded and allow testing and confidence in the deliverable.  There has been prioritisation of some features, and we have gone for a functioning system over exhaustively rebuilding features.  For example, some past migrations from other databases (such as past perfect) haven’t been tested, as is the template creation of extended fields.

Why is Rabid committing resource to this upgrade?   

Joann Ransom and Josh Forde have been speaking for some time about the project.

There is clearly increasing awareness and conversation about the importance of community in digital heritage and collections.  Rabid believe if we can complete this project, Kete will be able to grow again, including beyond New Zealand.

Kete delivers an excellent service and the collections you have are very valuable already. We don’t blame any particular user for the state of the project – but Rabid see small community projects like Pukekura Park or the Otago Memory project doing great work collecting digital heritage items, without the technical services to keep those projects secure for the long-term.

When balanced with the overhead of providing a well controlled, standardised software solution, Rabid think Kete is of great value and can grow towards a vision of any New Zealander’s digital history being able to be stored, described, and ‘storified’ in a digital format that will outlast the physical media that their history/whakapapa hold.

In short, Rabid believe in the underlying values of Kete and are working to a point that we can give you confidence in the software that runs your collections.

What next

Naturally we would like to deal to these problems immediately, but we need to work within our resources and keep steady progress on the technology. Rabid are taking on as much as they are able, and it would be extremely helpful from here for libraries to make commitments to the project (expressions of interest via Te Horowhenua Trust) for either:

a.    a new Kete subsequent to the upgrade including an allowance for software maintenance and upgrades in a collective fashion
b.    willingness to commit to an upgrade of their Kete, in tandem with a support arrangement.

A pricing model will be available in the next week or so.

That said, it’s exciting to have progress, and to be able to browse and explore the new Kete within a new application framework.

You can view a demonstration site using Kete Horowhenua at kete2.herokuapp.com.  Functionality is strictly browsing only (all creation and editing of items is upcoming)