AVH3 public release priority task plan

Actions for getting AVH v3 to public release
Compiled in first instance on 22nd and 23rd May 2008 by Paul Coddington (incl. input at keyboard), Shunde Zhang, Gerson Galang, Bill Barker.

Revised list by Paul Coddington (with help from Bill Barker and Rex Croft) on 5 June 2009. Subsequent updates are noted with the date.

Tasks that still need to be done before AVH3 is released:
 * 1) There should be a deadline for when it will be publicly released, and someone should be allocated as a project manager to be responsible for this happening. Otherwise discussions about what should be done or what needs to be added will continue indefinitely with no sense of urgency. Note that many of the tasks required to bring AVH to release have been action items for over a year.
 * 2) *[19/10/09 PC] We had aimed to get AVH ready for release by end of Sept but several actions are not yet complete. Some of the programming has taken longer than expected, and the request to implement a new interface has also slowed completion.
 * 3) Action on HISCOM/CHAH for policy on creation of registered user accounts - what are the criteria for getting an account, who decides whether someone fits the criteria, what is the process for checking bona fides, who has permissions to add an account?
 * 4) *[19/10/09 PC] We need an approved, documented policy for the registration process that we can implement in AVH3.
 * 5) Allocate people (from HISCOM?) as AVH admins and set up admin accounts for them.
 * 6) *[19/10/09 PC] The code for registration of accounts has been at an old version (where the top level AVH admin approves all accounts - which was the original spec) that hadn't been updated or properly tested while eRSA was waiting on an agreed process. We are in the process of updating and testing this so there is an additional admin level (e.g. HISCOM members) to do approval process. But to finalize the implementation we need criteria for approval - e.g. does anyone at this level get to approve (so all of them get notified)? Or is it done on a state basis (e.g. researchers in a particular state get approved by HISCOM people from their state herbarium)?
 * 7) *[19/10/09 PC] Once the new registration code is working (initial version should be ready by 21 Oct 09) then we can add HISCOMers and test whatever registration process is agreed.
 * 8) Action on Canberra (Jim) to engage web designer to update the look of the web interface. Ben Richardson has added code (way to go Ben!) to make it easier to handle different stylesheets which has improved the design of the interface already.
 * 9) *[19/10/09 PC] Agreement by HISCOM to implement new interface proposed by Niels. This has now been done for the simple query and is available for testing on avhtest. Still needs some bug fixes. Ben's improvements have been integrated into the new version of the code. Incorporating new interface into extended query will take at least 2 weeks additional work at eRSA, and some work on bug-fixing etc from Niels.
 * 10) Action for Brett and Laurence to revise data use agreement needed for public and new data use agreement for registered users.
 * 11) * What is status?
 * 12) New data dump from all herbaria, maybe try to clean up some of the problematic records beforehand.
 * 13) *[19/10/09 PC] eRSA working on system for upload and automated ingest of data dumps. Should be ready for testing 23/10/09.
 * 14) Some help pages still need to be completed.
 * 15) There is a bug in the BioCASE indexing part of the AVH code which meant not all fields in indexed records were being stored properly in the AVH database. This should be fixed (in the code and the database) by mid-June 2009.
 * 16) * [19/10/09 PC] I believe this is fixed but need to check - the programmer is on leave at the moment.
 * 17) Apart from this bug fix, I believe all the required programming has been done (as listed in AVH Programming Progress) except synchronisation of accounts, but this is not required until we get more than one instance of AVH, so not necessary for release.
 * 18) * [19/10/09 PC] We are going to use a database synchronisation mechanism for this, which requires some testing and updating of installation documentation, but shouldn't require much (or any) change in the AVH program enable this.
 * 19) * [19/10/09 PC] Some additional programming was requested, to modify the user interface and to enable automated ingest of data dumps, this is still being implemented.
 * 20) * [19/10/09 PC] Changes to the code for registering users was missing from the previous list of programming tasks, this is also being done now.
 * 21) Add GET (or POST) query syntax, as in previous version, to enable automated links to be generated from member websites
 * 22) * [19/10/09 PC] This is not trivial to do, particularly for the extended query. The first attempt at implementing it for the simple query broke the extended query. It would be a lot easier if we could just implement it (at least initially) for the public query. Is that enough, or do we need an API for the restricted interface also?
 * 23) Testing and QA on new version of AVH
 * 24) * [19/10/09 PC] Current version is available on avhtest, we hope to have a release candidate for testing during the week of Oct 26th, which will include upload and automatic ingest of data dumps, and the new process for user registration.
 * 25) Move new version of AVH from development to production public server
 * 26) * [19/10/09 PC] Depending on how long testing the new version takes, this could be done in early November.

There are a number of things that would be nice to have done before public release, but aren't necessary, and could be added in later updates. CHAH and HISCOM need to decide.

Additional tasks, not necessarily before release:
 * 1) Move installation from eRSA to one or more of the herbaria (doesn't have to be done before public launch, but probably a good idea)
 * 2) * [19/10/09 PC] I expect this would need a significant amount of advance planning and negotiation with the institutions aiming to host the new AVH, particularly state government IT departments (e.g. for SA). I don't believe this has happened yet.
 * 3) Synchronization of user accounts across multiple instances of AVH still needs to be implemented, but this is not required until we get more than one instance of AVH
 * 4) * [19/10/09 PC] We are testing a prototype database synchronization mechanism and should have this ready and documented by early November.
 * 5) Some additional programming work has also been done (with a bit more still to be done) to improve the code and make AVH more customizable, based on input from NZ, but this is not necessary for AVH public release
 * 6) * [19/10/09 PC] This code has been incorporated into the latest release candidate.
 * 7) EWS/WeedTracker completed and public release at the same time as AVH
 * 8) * [19/10/09 PC] There is quite a lot of work to be done on this, which has currently taken a back seat to AVH changes. We have a new developer working on AVH and EWS and it's hard for them to estimate how much work is required and how long it will take.
 * 9) Add option to use a different mapper, in particular a mapper based on Google Maps, e.g. Rex's mapper, Paul Flemons BioMaps, etc. Need to modify AVH code as well as additional development of Rex's mapper.
 * 10) Full set of available fields from all herbaria
 * 11) Records from all countries, or at least from around the broader Australasian region (e.g. PNG, Indonesia, NZ, New Caledonia, Antarctic islands, etc)
 * 12) Have AVH publishing data to GBIF etc, by putting a BioCASE or TAPIR provider in front of the AVH database. That would require quite a lot of additional work, could perhaps be done with funds from ALA if that proposal was accepted.
 * 13) * This work will be funded by ALA, not expected to happen until early 2010.
 * 14) AVH still doesn't support curation fields for loans etc, but currently only AD provides this and will need more funding for programming

Timelines from here
Propose:
 * HISCOM review 26-29 May 2008. Use History to view amendments. Use Discussion to promote debate.
 * CHAH. Raise with Brett Summerell to ensure OK in concept; CHAH teleconference agenda item: 17 June, for info, clarification and for discussion of time for announcing public release and launch
 * Design, Help, Programming commence ASAP.
 * EWS: QA testers approached 26-29 May, testing in June as early as possible.

Design of revised user interface
(Ben Richardson, Siobahn Duffy; Jim Croft coordinator)


 * 1) Simple changes to header, footer, CSS of each page can be done easily. Can easily fix up hyperlinks from each input field to relevant section of a user guide
 * 2) Any other major changes to the design and user interface will require significant JSP programming effort, will not be feasible within the initial 3 weeks effort, will need to be costed as future work.
 * 3) Prompts for queries should specify which ones are not indexed so search on these fields may be slow.
 * 4) Look at suggestions from Chris Brodie about modifying map interface, presentation and output, and decide which of these changes should be made, specify required interface.
 * 5) Specify what should be done to add Google Maps output option.
 * 6) Suggested top priority in development for public release: bring AVH WeedTracker to sufficient acceptable look and feel for Weeds industry testers to utilise with clarity; provide feedback and produce a design upgrade specification (Issue: ideally part of AVH2 public release implementation - if too many tasks then separate and follow hard on AVH2).

Page content - licensing, help, business description
(Alison Vaughan help pages, prompts; Jim Croft business description; Laurence Paine licensing)
 * 1) Consistency of field names in prompts and outputs. Don't use ABCD concept names like unitID, but clear descriptors. Alison to provide.
 * 2) Information about which query fields are supported by which herbaria, and which fields are well populated. Alison to do.
 * 3) Change: name of states and Canberra to Herbarium names; change prompt to State/Territory Herbarium.  HISCOM to provide correct prompts.
 * 4) HISCOM to consider how to address cultivated data and ensure all herbaria are proving information in agreement with HISPID5

Programming
(Gerson Galang, Shunde Zhang, Paul Coddington; Rex Croft, Bill Barker vetting).

Current production version is http://www.eresearchsa.edu.au/avh

Development version for testing is http://www.sapac.edu.au/avhtest

Google code repository for AVH code is http://code.google.com/p/avh/ Much more documentation of the code is needed.


 * AVH Programming Tasks and Progress 2009
 * AVH Programming Requirements 2008
 * AVH Programming Progress 2008
 * Issues of data in AVH

Tasks for HISCOM

 * 1) Full scientfic name should include authorities (and not family).  HISCOM to provide.
 * 2) Date last edited needs to provided for all records by all herbaria.

To be done later

 * 1) Some user interface modifications in the output data specification section.
 * 2) * Estimated Time: 0. Some of this has already been done, some is suggestions that will be considered as part of revised user interface spec.
 * 3) Data consistency for Habit, Habitat, and Notes.  Rex Croft to investigate extending his existing script that loads Melbourne data into Adelaide database so that it can do data loads from all different data providers into the AVH database in a consistent way.
 * 4) Taxon group. This has been reduced to medium priority.

Future wish list

 * A location to record future requests / improvements


 * 1) Rather than have the user select 'sort order' on the query pages, can they re-sort the results in the output (by clicking on the column headers)? This is how these things usually work, so it will be a bit more sophisticated and intuitive for the user. (Alison)
 * 2) * Most versions that I have investigated are commercial javascript packages. I haven't found a good free routine that will sort the rows of a table based on a column heading, locally within the browser. (Rex) - try http://tablesorter.com/docs/ I've used it and it works well (Peter :)

AVH WeedTracker - an Early Warning System for Weeds
(Bill Barker, coordinate).

Design of revised user interface
(AVH2 Design team plus a Weeds person)

Page content, Licensing, Help
(TBA)

Programming
(SAPAC, AD as above)

QA

 * 1) Confirm EWS working appropriately by addition of record to AD data
 * 2) Roll out to Weeds testers - those agreeing two years ago: John Moore WA, John Virtue and David Cooke SA, Kate Blood Vic., Matt Baker and John Thorp Tas., John Hosking NSW, Jane Morton Qld (add Dane Panetta back from sick leave); add Paul Pheloung ACT.

Hosting

 * 1) Consider who hosts. Adelaide willing and IT infrastructure management considers feasible with little difficulty. SA DEH already run tomcat applications and have MySQL installed to support BioCase.

Administration of Weeds industry users

 * 1) Revisit Bill's discussions two years ago, address with Weeds testers as part of QA. Adelaide IT management consider can handle external management.

Next steps
(HISCOM).

Providing an EWS to general users to track any Australian plant taxon
EWS has an implicit assumption that herbaria have implemented a nightly BioCase or Tapir feed to AVH, since it needs new records in order to determine the spread of species. It will only be useful to those States that have achieved this.

Releasing Australian herbarium specimen records to GBIF

 * 1) Is AVH the way to achieve this while meeting CHAH view of what data should be released? Outcome: a recommendation to CHAH