- Large ingestion of 4 gig tifs with derivates and preservation checks (10k works) in ~ 1 hour - 5k batch metadata updates ~5 minutes - Round trip spreadsheet update (5k records in 5 minutes) - Preservation dashboard / verification - Local authority creation and updates This presentation will discuss the process, what we learned, and how it relates to the Samvera community at large. and On St. Patrick's Day NU went live with our new digital collection repository and asset management tool prioritizing speed of ingestion and metadata updates. We reframed the problem by working with end-users to look closely at workflows and prioritize solutions rather than any specific technology. The resulting application ecosystem is extremely budget friendly and the architecture supports
A presentation given at Samvera Connect 2020 On-line described thus and After trying to navigate deployment, configuration, performance, and scaling issues of several different image servers and support infrastructure (Cantaloupe, Aware, Riiif, nginx, and SquidCache, to name a few), we decided to see if we could build something less general/configurable but far more suited to our use case and runtime environment. serverless-iiif started out as a bare bones, proof-of-concept demonstration of how a scalable, high-performance IIIF image server could be implemented in a small, inexpensive AWS Lambda function. Just over a year later, the project serves as the basis for high-volume IIIF services running in production at Northwestern University, Princeton University, the University of Notre Dame, and the Royal Pavilion & Museums, Brighton & Hove. This presentation will cover the project from its beginnings (as a small demo repository carved out of Northwestern's cloud repository infrastructure), through a number of forks, merges, performance enhancements, deployment improvements, and into production. We will also include performance benchmarks, current production stats, and some thoughts on future work. The 'Related URL' below links to a video recording of the session. The video has closed captioning.
Keyword:
Cloud services, Connect 2020, International Image Interoperability Framework (IIIF), and Samvera
Subject:
Samvera Community
Creator:
Silverton, Edward, Klein, Michael B, Hartzler, Jonathan, and Pendragon, Trey
Contributor:
Northwestern University Libraries, University of Notre Dame, Mnemoscene, and Princeton University Library
Component Maintenance WG - James Griffin BrowseEverything IG - James Griffin Metadata IG - Anna Goslen Hyrax v3.0 Metadata Application Profile Documentation Review WG - Nora Egloff Repository Management IG - Moira Downey Infrastructure WG - Michael Klein Newspapers IG - Eben English Hyrax Maintenance WG - Tom Johnson Geo Predicates WG - John Huck The video recording of this segment is available at the 'Related URL' below. and Working and Interest Group Updates
Keyword:
Geodata, Newspapers, Screencast, Interest and Working Groups, Virtual Connect 2020, Repository Management, Hyrax, Samvera, and Metadata
Subject:
Samvera Community
Creator:
Egloff, Nora, Klein, Michael B, Goslen, Anna, Johnson, Tom, Griffin, James, Huck, John, Downey, Moira, and English, Eben
Princeton and Northwestern recently underwent a two-week spike to explore a set of new technologies we might use in our respective teams. We looked at ElasticSearch, Elixir, and Phoenix. This presentation will go through our expected outcomes, strategies for a successful collaboration, our eventual output, and a retrospective on how the process went with advice for any others looking to do this kind of exploratory work. The video recording of this segment is available at the 'Related URL' below.
Keyword:
Screencast, Virtual Connect 2020, and Samvera
Subject:
Samvera Community
Creator:
Klein, Michael B and Pendragon, Trey
Contributor:
Princeton University Library and Northwestern University Libraries
Northwestern University Libraries is currently running Samvera applications in production. Three of these are developed, maintained, and managed by the Repository & Digital Curation workgroup, * Arch, an Institutional Repository, based on Hyrax 2.4.1 * AVR, Northwestern's audiovisual repository, based on Avalon 6.3 * DONUT, the staff-facing ingest interface for the digital object repository, based on Hyrax 2.4.1 In developing and deploying these applications, we have encountered (and mostly overcome) numerous stumbling blocks relating to performance, scalability, customization, and assumptions about the deployment environment and infrastructure on which the apps will run. While we have found it possible to shoehorn the Samvera stack (as it exists today) into our Amazon Web Services cloud-based deployment environment, we have also started to investigate the rewards and compromises involved in taking a cloud-first approach to our next generation of tools. We have identified several basic tenets for this approach so far, * If AWS offers a native Software-as-a-Service (SaaS) solution for a particular problem, use it (e.g., choose ElasticSearch/Cloud Search over Solr) * Avoid virtual server instances that run 24x7 waiting for requests/work * Do not assume there is a local filesystem to work with * Optimize startup time so that units of work can be spawned and killed as needed * Constantly assess and reassess every unit of work for scalability, repeatability, and idempotence * Keep data portable and code adaptable, but don't over-stress about vendor lock-in In this presentation, members of the Repository Development & Administration Team will present on lessons learned from 7 years of working with Samvera, Avalon, and Hyrax, what the future holds for our next round of in-house development, and the opportunities & compromises our cloud-first approach creates regarding our use of and contributions to the larger Samvera community., and A presentation at Samvera Virtual Connect 2019 described thus
Keyword:
Cloud services, Samvera, and Virtual Connect 2019
Subject:
Samvera Community
Creator:
Klein, Michael B
Contributor:
Quinn, Brendan, Schober, David, Arling, Adam, Shaw, Karen, and Northwestern University
Over the past two years, Northwestern University Libraries has moved its repository infrastructure and applications to Amazon Web Services. Our initial solution, presented at Samvera Connect 2017, involved AWS CloudFormation, several different deployment platforms, and a lot of manual intervention. In our second phase, we have adopted a fully automated build/configure/deploy system to stand up Fedora, Solr, PostgreSQL, Redis, a Cantaloupe IIIF server, an Avalon Media System instance, a secure CloudFront streaming media distribution, and two Hyrax applications using Terraform, Docker, AWS Elastic Beanstalk, and a whole bunch of homegrown tools and hacks. This presentation will provide an overview of our current system, and hopefully jumpstart some discussions of how these tools can be adopted, standardized, and reused among other members of the Samvera community. A video recording of this session is available at the 'Related URL' below., A presentation given at Samvera Connect 2018, originally titled "My Life in Ops, and Docker, Terraform, AWS, and Learning As We Go", described thus
Using agile for more accurate proposals - Aaron Collier New Tools for NU, A combined slide pack containing the lightning talks presented at Samvera Connect 2019. Deploying a custom Hyrax on Azure with Terraform and Kubernetes from a standing start - Julie Allinson Controlled Vocabulary Options in Hyrax - Julie Hardesty Scholarsphere is Dead, Long Live Scholarsphere! - Adam Wead Work Estimation Records, and Elixir, Broadway, Phoenix, and GraphQL - Michael B Klein Hyku Open Source Institutional Repository Development partnership awarded $1M Arcadia grant to improve open scholarship infrastructure - Brian Hole Showcasing & Connecting Women in Technology - Robin Ruggaber Iterating on a Hiring Process - Jeremy Friesen
Keyword:
Lightning talk, Samvera, and Connect 2019
Subject:
Samvera Community
Creator:
Klein, Michael B, Collier, Aaron, Wead, Adam, Ruggaber, Robin, Hardesty, Juliet L, Allinson, Julie, Friesen, Jeremy, and Hole, Brian