- Large ingestion of 4 gig tifs with derivates and preservation checks (10k works) in ~ 1 hour - 5k batch metadata updates ~5 minutes - Round trip spreadsheet update (5k records in 5 minutes) - Preservation dashboard / verification - Local authority creation and updates This presentation will discuss the process, what we learned, and how it relates to the Samvera community at large. and On St. Patrick's Day NU went live with our new digital collection repository and asset management tool prioritizing speed of ingestion and metadata updates. We reframed the problem by working with end-users to look closely at workflows and prioritize solutions rather than any specific technology. The resulting application ecosystem is extremely budget friendly and the architecture supports
Northwestern University Libraries is currently running Samvera applications in production. Three of these are developed, maintained, and managed by the Repository & Digital Curation workgroup, * Arch, an Institutional Repository, based on Hyrax 2.4.1 * AVR, Northwestern's audiovisual repository, based on Avalon 6.3 * DONUT, the staff-facing ingest interface for the digital object repository, based on Hyrax 2.4.1 In developing and deploying these applications, we have encountered (and mostly overcome) numerous stumbling blocks relating to performance, scalability, customization, and assumptions about the deployment environment and infrastructure on which the apps will run. While we have found it possible to shoehorn the Samvera stack (as it exists today) into our Amazon Web Services cloud-based deployment environment, we have also started to investigate the rewards and compromises involved in taking a cloud-first approach to our next generation of tools. We have identified several basic tenets for this approach so far, * If AWS offers a native Software-as-a-Service (SaaS) solution for a particular problem, use it (e.g., choose ElasticSearch/Cloud Search over Solr) * Avoid virtual server instances that run 24x7 waiting for requests/work * Do not assume there is a local filesystem to work with * Optimize startup time so that units of work can be spawned and killed as needed * Constantly assess and reassess every unit of work for scalability, repeatability, and idempotence * Keep data portable and code adaptable, but don't over-stress about vendor lock-in In this presentation, members of the Repository Development & Administration Team will present on lessons learned from 7 years of working with Samvera, Avalon, and Hyrax, what the future holds for our next round of in-house development, and the opportunities & compromises our cloud-first approach creates regarding our use of and contributions to the larger Samvera community., and A presentation at Samvera Virtual Connect 2019 described thus
Keyword:
Cloud services, Samvera, and Virtual Connect 2019
Subject:
Samvera Community
Creator:
Klein, Michael B
Contributor:
Quinn, Brendan, Schober, David, Arling, Adam, Shaw, Karen, and Northwestern University
Over the past two years, Northwestern University Libraries has moved its repository infrastructure and applications to Amazon Web Services. Our initial solution, presented at Samvera Connect 2017, involved AWS CloudFormation, several different deployment platforms, and a lot of manual intervention. In our second phase, we have adopted a fully automated build/configure/deploy system to stand up Fedora, Solr, PostgreSQL, Redis, a Cantaloupe IIIF server, an Avalon Media System instance, a secure CloudFront streaming media distribution, and two Hyrax applications using Terraform, Docker, AWS Elastic Beanstalk, and a whole bunch of homegrown tools and hacks. This presentation will provide an overview of our current system, and hopefully jumpstart some discussions of how these tools can be adopted, standardized, and reused among other members of the Samvera community. A video recording of this session is available at the 'Related URL' below., A presentation given at Samvera Connect 2018, originally titled "My Life in Ops, and Docker, Terraform, AWS, and Learning As We Go", described thus