Reimagining Annotations for Multimodal Cultural Heritage

7 février 2025 - 9 février 2025

Résumé

Les progrès rapides des technologies numériques, en particulier grâce à l’essor de l’intelligence artificielle, élargissent constamment les possibilités d’annotation des documents, qu’il s’agisse de textes, d’images, de vidéos, de sons, de pages web ou de codes. La valorisation du patrimoine numérique, notamment des documents audiovisuels et des corpus multimodaux, qu’ils soient numérisés ou nativement numériques, devient un enjeu majeur pour les institutions culturelles et les chercheurs en SHS.

Parmi les réponses proposées, les interfaces d’annotation manuelles et/ou automatiques, grâce à des techniques computationnelles, permettent d’offrir des opportunités de lecture de près comme de loin. Elles engendrent cependant de nombreuses questions épistémologiques. Comment articuler et organiser les hiérarchies informationnelles qui émergent de ces méthodes ? Quelles sont les possibilités et les limites des méthodes de lecture de près et de lecture distante et comment articuler ces deux approches ? Comment encourager un travail cumulatif et collaboratif ? Que devient le document lorsqu’il s’intègre dans un réseau d’annotations, de lectures et de fragments ?

Afin de répondre à ces différentes questions, le colloque Reimagining Annotation for Multimodal Cultural Heritage, organisé par Clarisse Bardiot, professeur en études théâtrales et en humanités numériques, et Jacob Hart, post-doctorant en musicologie et en humanités numériques, se déroule à l’université Rennes 2 pendant trois jours :

  • Une première journée est consacrée à la découverte et à la prise en main de deux environnements numériques lors d’une session de workshop, suivie d’une conférence inaugurale par Scott deLahunta, professeur en danse à l’université de Coventry.
  • Le colloque se poursuit avec deux journées de communications scientifiques, dont la conférence de Melvin Wevers, maître de conférences en histoire numérique à l’université d’Amsterdam. Il se clôturera par une table ronde sur les défis auxquels sont actuellement confrontées les institutions culturelles et les chercheurs en matière d’interopérabilité et de durabilité.

Ce colloque est l’occasion de réunir une communauté internationale d’experts du domaine afin d’explorer les questions relatives aux outils (état de l’art, enjeux de développement informatique et d’interfaces), aux méthodes (méthodologies émergentes, développement épistémologique des humanités numériques) et aux projets (explorer comment ces techniques s’intègrent dans les projets de recherche en sciences humaines).

Cette manifestation rassemble des chercheurs en histoire, en informatique, en histoire de l’art, en arts de la scène, en cinéma, en sciences de l’information et de la communication et en littérature, des archivistes, des artistes, ainsi que des institutions culturelles et patrimoniales. Elle est également l’occasion de lancer le programme de recherche ERC STAGE dirigé par Clarisse Bardiot.

2024 February 7th

  • 12h30 > Registration

13h – 16h30 > Workshops

  • 13h – 14h30 > SCENE by Clarisse Bardiot, Jacob Hart and David Rouquet, Université Rennes 2 (France)
  • 15h – 16h30 > Distant Viewing Toolkit by Lauren Tilton and Taylor Arnold, University of Richmond (USA)
  • 16h30 – 17h > Coffee Break

17h > Conference Opening and Welcome Remarks

  • Vincent Gouëset, President of the Université Rennes 2
  • Michael E. Sinatra, Professor at the Université de Montréal, founding director of the Digital Humanities center CRIHN
  • Nicolas Thély, director of the MSHB
  • Sophie Lucet, director of the lab Arts: pratiques et poétiques, Université Rennes 2
  • Clarisse Bardiot and Jacob Hart, co-chairs of the conference, Université Rennes 2

17h30 – 18h30 > Keynote: Scott deLahunta, University of Coventry (UK), Software for Dancers.

Chair: Clarisse Bardiot

Piecemaker was a video annotation software used by The Forsythe Company in Frankfurt from 2007-2013 to aid the ensemble in recalling material they were working on in the studio in the context of choreographic creation. Most of the annotations were made live, with a static camera, while creative work was happening. These recordings were never intended to be made public. In 2010, the Motion Bank project took on the task of developing this software for wider public use in the dance community. Since then, different versions have been made and used for several projects. This development has been supported with research funding from various sources, and the software is now completely open source. Crucially, the focus has remained on making a digital (documentation) tool that is usable and useful for dance practitioners, both artists and educators. This focus will form the core of this talk, including insights into methods into language-usegained from the Motion Bank research. I will reflect on the nature of dance processes and knowledge to address some critical questions that come up in the context of developing usable and useful software for dancers. I will lightly trace a history of this from the multimedia 1990s to the current data-driven and AI preoccupied present, and speculate on what this might mean for the understanding and valuing of bodily practices.

Opening Reception

2024 February 8th

  • 9h > Registration

9h30 – 11h > Audiovisual Documents Analytics

Chair: Jacob Hart

  • Olivier Aubert, Nantes Université (France) – Advene, a Look Back on 20 Years of Video Annotation Instrumentation.
  • Mark J. Williams, Dartmouth College (USA) – Deep Screens and Evocative Surfaces: New Research from The Media Ecology Project and the DEV Lab at Dartmouth.
  • Michael Rau and Peter Broadwell, Stanford University (USA) – Machine Intelligence for Motion Exegesis (MIME): Applying Pose Estimation and Related Technologies to Analyze Archival Performance Recordings.

11h – 11h30 > Coffee Break

11h30 – 13h > The Temporal Dimensions of Distant Viewing

Chair: Michael E. Sinatra

  • Matteo Treleani, Université Côte d’Azur (France) – Crossing Borders Archives. The Circulation of Stock Shots in Audiovisual Media.
  • Nicola Carboni, Université de Genève (Switzerland) – The Structures of Visual Exchanges.
  • Mila Oiva, Tallinn University (Estonia) – Using Multidimensional Vector Embeddings to Study Temporal Dimensions of Historical Newsreel Data.

13h – 14h > Lunch and visit of the exhibition Archives graphiques de la recherche

14h – 15h30 > Short Papers

Chair: Mila Oiva

  • Tanya Clement, University of Texas at Austin (USA) – AVAnnotate: Creating Scholarly Editions and Exhibits with IIIF and AV Archives.
  • Nabeel Siddiqui, Susquehanna University (USA) – Bipartite Frame Networks in the Analysis of Film: a Case Study Utilizing Commercial Computer Vision APIS.
  • Théo Heugebaert, Université Rennes 2 (France) – Visualizing Rhythms Through Digital Annotations: Challenges and Issues in the Performing Arts.
  • Bérénice Serra and Léna Frei, Institute Digital Communication Environments (Switzerland) – Intuitive Access to Oral History Video (The Pellaton Experience).
  • Diane Jakacki, Bucknell University (USA), Susan Brown, University of Guelph (Canada), Michael Ilovan, University of Alberta (Canada) and Luciano Frizzera, Concordia University (Canada) – The Linked Editing Academic Framework (LEAF) in the Multimodal Annotation Ecosystem.

15h30 – 16h > Coffee Break

16h – 17h30 > IIIF, from Images to Multimodal Corpora Annotations

Chair: Lauren Tilton

  • Régis Robineau, Biblissima +, Campus Condorcet (France) – Overview of the IIIF Initiative for Interoperability of Digital Objects on the Web (Image, Audio/Video, 3D).
  • Jean-Christophe Carius and Chloé Pochon, INHA (France) – From Source Annotation to Scientific Publishing: the PENSE and PerVisum Projects.
  • Clarisse Bardiot and Jacob Hart, Université Rennes 2 (France) – IIIF, a Standard for Multimodal Corpora? The Building of SCENE.

17h30 – 18h > Coffee Break

18h – 19h > Keynote: Melvin Wevers, University of Amsterdam (Netherlands), A Multimodal Turn: Navigating AI Developments in Digital Humanities.

Chair: Jacob Hart

In the ever-evolving landscape of Digital Humanities (DH), research methodologies predominantly centered on textual data. However, the advent of deep learning revolutionized this scope, enabling the automated analysis and labeling of visual materials. Despite their capabilities, these early methods demanded extensive training datasets. The landscape saw another transformation with the rise of multimodal deep learning architectures, such as the Contrastive Language Image Pre-training (CLIP). Such innovations brought about a convergence of GPT-inspired interfaces for visual analysis, broadening the ambit of multimodal research. These technological leaps have now positioned humanists on the cusp of computational visual analysis.

This keynote aims to spotlight these advancements and probe deeper into their alignment with multimodal theory. By doing so, it strives to understand their ramifications on the humanistic engagement with visual mediums. As we journey through this alignment, we find ourselves at a crossroads, grappling with pressing dilemmas of practicality, adaptability, and choice. Can the humanistic community keep pace with these swift technological evolutions? And, more fundamentally, is there an imperative to stay abreast, or should we gravitate towards more established techniques, offering greater control and explainability?

20h > Diner

2024 february 9th

9h > Registration

9h30 – 11h > Annotations for Contextualization and Narratives

Chair: Nicola Carboni

  • Øyvind Eide, Kai Michael Niebes, Nadjim Noori, Vyshantha Simha and Elisabeth Reuhl, University of Cologne (Germany) – 3D Annotations as Multimodal Storytelling.
  • Delfina Sol Martinez Pandiana, Università di Bologna (Italy) – Coding the Encoder: Situating Subjective and Contextual Aspects in High-Level Image Annotations.
  • Marie-Claude Poulin, University of Applied Arts Vienna (Austria) – VR and AR Prototypes for Multi-sensory and Haptic Forms of Documentation and Archiving of Digital Art (LeFo Project).

11h – 11h30 > Coffee Break

11h30 – 13h > Performing and Visual Arts Documentation and Analysis

Chair: Clarisse Bardiot

  • Lauren Tilton, University of Richmond (USA) – Distant Viewing with Libraries: Photography and the Library of Congress.
  • Carla Fernandes, NOVA University Lisbon (Portugal) – Multimodal Video Annotations as Metadata for Performing Arts Documentation.
  • Rime Touil, Bibliothèque nationale de France (France) – Curating Born-Digital Archives at the National Library of France: the Amos Gitai collection’s Case Study.

13h – 14h > Lunch and visit of the exhibition Archives graphiques de la recherche

14h – 15h30 > Designing Tools and Workflows by and for Researchers

Chair: Servanne Monjour

  • Luca Federico Cerra and Sean Takats, Université du Luxembourg (Luxembourg) – Two Historians’ Relationship with Sources in the Digital Age.
  • Julien Schuh, Université Paris Nanterre (France) – AI Toolkits for the Social Sciences and Humanities: A Closer Look at ModOAP, BaOIA, EyCon, and PictorIA.
  • Susan Brown and Kim Martin, University of Guelph (Canada) – The Linked Infrastructure for Networked Cultural Scholarship (LINCS): Bridging the Research/Heritage Collection Gap.

15h30 – 16h > Coffee Break

16h – 17h30 > Perspectives (round table)

Chair: Michael E. Sinatra

  • Nicolas Larrousse, Huma-Num (France)
  • Philippe Effantin, Ouest-Valorisation (France)
  • Arthur Lezer, Le Lab, INA (France)
  • Susan Brown, University of Guelph (Canada)
  • Clarisse Bardiot, Université Rennes 2 (France)

Closure

Throughout the event, the exhibition Archives graphiques de la recherche. De Jacques Bertin à Adrian Frutiger is being held on the campus. The team welcomes you to visit it. Two guided tours by the curator are organised during lunch at 13h20.

The Région and SATT Ouest Valorisation support projects in Human and Social Sciences resulting from public research.

This project has received funding from the European Research Council (ERC) under the European Union’s Horizon Europe research and innovation programme (grant agreement no. 101097091 – STAGE). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or ERC. Neither the European Union nor the granting authority can be held responsible for them.