Rhythmic Trajectories is an artistic research project designed to develop visual and sonic representations of cinematic rhythm. At the basis of this exploration is the understanding of movement as the inherent specificity of cinema — the illusion of coherent movement based on the projected sequence of distinct frames; the illusion of a coherent cinematic time and space assembling separate shots. The synergy of the composed audiovisual elements of cinema results in a rhythmic structure that I call cinematic rhythm. This research aims to examine the elements forming this cinematic rhythm.
This project uses visualization to deconstruct and formalize the elements of cinematic rhythm. Rhythm and, specifically, cinematic rhythm has been addressed by different authors in the context of film studies – Ken Dancyger, Karel Reisz, David Bordwell and Kristin Thompson – just to mention a few. It has been generally tied to the temporal specificity of cinema and especially to the formal categories of film editing such as timing, pacing and frequency discussed and reviewed in depth in Karen Pearlman’s book Cutting Rhythm, Shaping the Film Edit. In the words of another theoretician: “... rhythm is a kind of dialectic of time rather than a continuity whose intermittent variations distort for us the normal flow of time.” (Mitry 2000: 104)
For the purpose of this essay, I will define cinematic rhythm as the formal characteristic of a complex and systematic, temporally and spatially structured audio-visual sequence that contains meaning. Cinematic rhythm and its visualization concerns meaning which can be sensed beyond (but not instead of) the narrative and linguistic realm of a film. Cinematic rhythm contains what we can call embodied, kinesthetic knowledge as an indispensable part of a film experience.
The practical part of the research consists of a series of short études relating to different aspects of cinematic rhythm. I design and add these aspects as a visual and/or auditive layer to the original film sequence. This artistic research project builds on my knowledge as a film editor. It reflects and visually contextualizes the mostly intuitive work done by a film editor, while also tying that work to the cognitive process of experiencing an edited film. The research interest of my project is to define what cinematic rhythm consists of; to identify the formal elements to which cinematic rhythm can be disassembled and finally to develop a visual representation to each of these distinct elements. In order to do so, it is important to note that there are different levels on which cinematic rhythm can be discussed – inside one shot; on the cut, at the juxtaposition of two shots; in a sequence as the connection of several shots; or on the narrative level of the film, where the structural element is a scene (or character). Karen Pearlman talks about physical, emotional and event rhythm (Pearlman 2009: 84) These categories are pointing toward a typologisation of rhythm based on the content of the film sequence and how these different types are experienced by the spectator. (A more detailed discussion follows.) The concept of energy, the ideas of tension and release, the structural meaning of repetition and iteration are all an important aspect of the work of visualization to be presented. It ties back to the questions of what cinematic rhythm consists of, and how movement (be it physical, emotional or event) as the inherent specificity of cinema is able to create a sensuous knowledge.
The études are short experimental studies set to develop and iterate formal tools and aesthetics (such as animation using Adobe After Effects or computational approach using the software Processing) to visualize cinematic rhythm as part of the research process. Using the methodological approach of the étude understood as a simple exercise set to develop in its iteration a complex idea situates it in the tradition and constructivist context of Meyerhold’s biomechanics (see Fig. 1) and the montage experiments of Kuleshov. The études are non-destructive, close readings of film sequences (these concepts will be explored in depth later in the text) with formal relevance in the context of cinematic rhythm. Interestingly, the method of adding a visual and/or auditory layer to the original film has opened another field of inquiry — keeping the original sequence intact turns the added layer into an interpretation that interacts in turn with the original sequence. A friction evolves from the relationship between the original and the added layer that is of special interest for my artistic research. One possible question that arises from the études is: What happens at the contact point of friction between those two layers, what kind of sensuous knowledge arises from the interpretation that originates at the edge of the friction?
As a research project, Rhythmic Trajectories can be situated at the intersection of three loosely related fields that shed light on its methodology and its research question. In the following part, I will provide a short overview of the fields that constitute the theoretical background for my artistic research: 1) information visualization, and more specifically film annotation and visualization, where I will refer to Lev Manovich’s definition of this term; 2) the theory and practice of film editing with a special focus on cinematic rhythm, especially the influential work of Karen Pearlman, Cutting Rhythms, Shaping the Film Edit, which deals with the different forms of rhythm within film that are created and shaped by editing; and 3) visual arts and especially artistic research, pointing toward examples and different approaches in both fields.
The primary goal of Rhythmic Trajectories is to find visual and aural representation for cinematic elements (within the frame and on an intra-frame level as well). This is done to emphasise, formalize and visualize data present in the film sequence. In this context, my research can be situated in the field of information visualization and more specifically film visualization. Lev Manovich defines information visualization as “a mapping between discrete data and a visual representation. […] if we believe that a brain uses a number of distinct representational and cognitive modalities, we can define infovis as a mapping from other cognitive modalities (such as mathematical and propositional) to an image modality” (Manovich 2010). Cognitive modalities can be understood as different models of how we make sense of the world. Each modality has its own structures of representational ‘tools’ and affordances. The term ‘mapping’ used by Lev Manovich points beyond the meaning of the term ‘translation’ where both objects (the translated and the original) stay in the realm of the same representational mode of, for example, language. In the context of visualization, mapping can be used as meaning transposition – moving from one representational structure to another (such as from language to visual, or mathematical to visual). The correspondence between the two modes, as opposed to translation, can be much more complex and indirect.
Film itself is multimodal – it is visual, aural and linguistic all at the same time. Looking at the history of visualizing films, its primary goal is an analytic one and connected to digital humanities, film studies, and film-historical approaches. As these studies have always been text-based at their core, this approach has been limited by its own boundaries. Language turns out to be insufficient when it has to represent complex audiovisual entities. The mere description of categories such as shot size, shot length, content, and sound can serve as a transcript but it is never capable of representing temporality and the complexity of the ways audiovisual entities generate meaning. The necessity of visual correspondence to be used alongside the argument in these studies is not questioned anymore. The work of authors such as Catherine Grant1 and Kogonada2 has legitimised the form of the video essay in academic discourse. Nevertheless, there is still a discussion about the legitimacy of visualizations as a stand-alone (or non-linguistic) argument.
On the other hand, there has been a multitude of analytic approaches that are focusing on visual modes without building on the linguistic meaning in the way the video essay does. The project Cinemetrics3, for example, initiated by Yuri Tsivian is based on a measurement theory relating to shot length and using this as the data point for a further analysis. Frederic Brodbeck’s visualisation project4, titled Cinemetrics as well, utilizes quantitative measurement theory in order to create a so-called visual “fingerprint” of a film. In her forthcoming book Digital Humanities and Film Studies. Visualising Dziga Vertov's Work, Adelheid Heftberger (2019) discusses the research project conducted in cooperation with Lev Manovich, situating it at the intersection of digital humanities, film studies and information visualisation. The visualization combines quantitative and qualitative approaches in visualizing such attributes as shot length on the one hand and conducting a visual shot analysis on the other hand.
In the realm of qualitative analysis, there are at least three significant and partly similar projects. Kevin L. Ferguson has introduced the term of “volumetric cinema”5, where the separate film shots have been remapped spatially in a three-dimensional progression. Similarly, Joachim Sauter created the project “The Invisible Shapes of Things Past” as a parametric translation of movies into space6. Single frames from a film sequence are lined up in space, according to the camera movement with which they were shot. Another research project worth mentioning here is the VAT7, conducted by Virginia Kuhn et al., a large-scale video analytics project, establishing a software workbench for video analysis, annotation, and visualization. Its goal is to enable the user to discover formal/visual cross-references in film sequences based on an automated image analytics based on computer vision algorithms.
And last but not least there are attempts to create a notation to film sequences, either prior to the production of the film, as a pre-visualization, or after the completion of the film, as a universal analytic tool. For the pre-visualization, there are examples already from the beginning of the film history, such as the “partiture” by Eisenstein created to accompany a sequence from the film Aleksandr Nevskii (1938, USSR). (See Fig. 6) Although not meant to be realised as a film, László Moholy-Nagy’s “Sketch for a Score for a Mechanized Eccentric” (1924) is a “synthesis of form, motion, sound, light [colour], and odour” resembling a storyboard or a sheet music. (See Fig. 5) As far as film visualization as an analytic tool, there are more recent examples, such as the Audio-Visual Rhetoric research by Gesche Joost, Sandra Buchmüller and Tom Bieling (2009) or different annotation softwares such as ANVIL8. All these examples emphasize the connection between notation and visualization in which formalized, graphic elements refer back to different attributes (such as movement, dynamics, graphical structure among others) of the audio-visual work.
Summarizing the previous section, visualizing a film (or a part of it) is a process to “excavate” or bring to the surface hidden structures, patterns and correlations. While Lev Manovich refers to information visualization as a mapping, I would rather use the term “modal transposition”, where film as an audio-visual, temporal and constructed entity has been transposed into an abstract and formalized visual representation. The interest for visualization in my research project is, on the one hand, exactly to “excavate” the sensuous knowledge contained in the original sequence and to highlight, analyze and define at the same time what cinematic rhythm is. Although there is a formal similarity to the visual “partitures” created by Eisenstein or Moholy-Nagy, there is no expectation of reverse iterative reconstruction of the film sequence in the way a musical notation is meant to work.
The main research question of my work revolves around the definition and categorisation of cinematic rhythm in the context of film editing. What is at stake in this endeavour is to be able to analyse how a cinematic rhythm is constructed in the process of film editing; but this research also asks the questions how a spectator relates to cinematic rhythm and what the sensuous knowledge is that is created by the spectator experiencing a film sequence.
In the discussion of how cinematic rhythm can be defined, my main source is the work of Karen Pearlman. She uses three concepts to describe the attributes of cinematic rhythm: timing, pacing and trajectory phrasing. Timing has been discussed by Pearlman as the decision made by the editor about the first (and last) frame of the shot, its duration and its placement in the sequence. The term pacing in the context of film editing has been used as an equivalent to the term tempo in music. It refers to “three distinct operations: the rate of cutting, the rate or concentration of movement or change in shots and sequences, and the rate of movement or events over the course of the whole film” (Pearlman 2009: 47). Alongside these two terms, both of which have been used in a larger and traditional context of discussing rhythm, Pearlman introduces the new term “trajectory phrasing”. In her own words: “Trajectory phrasing shapes the energy and spatial organization of movements in shots and across edits into rhythms” (Pearlman 2009: 43).
Phrasing refers to structured and self-contained sequences with a specific arch defined by the contained energy and movement. The latter can be parsed easily because of its visual and spatial formal attributes – to put it simply, it has a direction and velocity. In contrast, energy is a rather abstract concept. As a scientific concept, energy is defined as a quantitative property of the potential to change. In the cinematic realm, energy could be understood as the underlying meaning of or intention behind a visible movement. Pearlman offers the term “effort”, as used by Rudolf Laban and Irmgard Bartenieff, to discuss “the attitude and intention behind movement that informs the way it is done”. (Pearlman 2009:53)
Pearlman (2009: 83) also introduces three categories of rhythm: physical, emotional and event rhythm. In this categorisation, rhythm is interchangeable with the concept of movement: “Movement as a material from which edited rhythms are shaped is not limited to movement of images. It includes movement of sound, emotion, ideas, and stories”. (Pearlman 2009: 83). The three categories – physical, emotional and event movement – are perceived differently by the spectator. Physical movement is experienced by the spectator as a sensory experience (Pearlman 2009: 84). Pearlman introduces here the idea of the energy of the movement that the spectator relates to through kinesthetic empathy. The editing is shaping the physical rhythm of a film sequence relating to the changes in the amplitude and modality of the energy of the movement. The result in the edited film sequence is the so-called physical trajectory. Its meaning is direct and kinesthetic; it is not representing something else. (Pearlman 2009: 84).
The second category is the emotional rhythm. It is, again, shaped by the editor and refers to the kind of cinematic motion that isn’t necessarily different from the physical, while its significance is that it is experienced as an emotion. In this case, motion represents emotional states and as such evokes certain emotions in the spectator. (Pearlman 2009: 84) The third category introduced by Pearlman relates to the movement of events. In her words: “An event is the release of new information or change of direction for characters as they pursue their goals. Each significant change in a story structure is an event.” (Pearlman 2009: 85) In other words, the movement of events happens on the narrative level of the film, on a macro level. In a more traditional approach, this category could be aligned to what we call a scene in a film, although the definition of a scene involves a change in time and space, whereas Pearlman refers to change on the level of the storyline (Pearlman 2009: 8).
In conclusion, all three categories describe different manifestations of cinematic rhythm. Physical, emotional and event motion can be the same phenomena in film, even if experienced differently by the spectator and referring to different aspects of the meaning evoked by the film. As I will describe later, the first étude is visualizing the physical rhythm in the sequence, whereas the second étude is visualizing an emotional rhythm expressed through the gestures of the actors in the sequence. As a superimposed term, brought in by Karen Pearlman, energy is central to the understanding of the underlying intention and at the same time, the conveyed meaning of the movement. My interest lies in the perception and experience of the cinematic rhythm and the forming of sensuous knowledge based on it.
The goal of this project to visualize cinematic rhythm created by editing can be situated at the intersection of different fields. Alongside film visualization and film studies, this project relates in its aesthetic approach to the field of artistic research.
Artistic research understands art as a reflexive practice, as an independent form for the production of aesthetic knowledge. Artistic research can trigger cognition through experience in the form of both sensuous and notional cognition. This knowledge is different from scientific knowledge in its methodology, its goals and its research field. At the same time, artistic research can be interdisciplinary in its inquiry. It uses its own aesthetic practice, as its own field of experience: “[...] the research as conducted by artists today is not characterized by an objective empirical approach, since art, obviously, does not strive for generalization, repeatability and quantification. Rather, artistic research is directed towards unique, particular, local knowledge”. (Slager 2004:13) Looking at the two études, on the one hand, it is clear that in both cases there are elements that can be objectively described, on the other hand, visualization added to each of the film sequences is a particular aesthetic response, specific, unique and subjective in its end form.
Against this background, the phrase "art as research” seems to be not quite accurate, because it is not the art, which evolves into research somehow. What exists, however, is research that becomes artistic - so it should be rather named "Research as Art", with the central question: When is Research Art? (Klein 2012).
In the context of digital scholarship, there is the overarching question of the research mode and its design – what kind of knowledge can be extrapolated from the raw material (in the case of the current project a film sequence)? Where can the inquired knowledge be situated? Is it scientific or artistic? Can it be both at the same time? Does this research fall into a theoretical discourse or does it have its own practice? Can this binary be eliminated? Can a visualization stand alone or is the object of inquiry always needed? All these questions are important in the context of the present research project and will be addressed at several points of the process.
Alongside the openness of the term “artistic research”, the argument for a non-binary research approach has been addressed in the context of video analytics within the field of digital humanities research. As Virginia Kuhn discusses in her text “Images on the Move: Analytics for a Mixed Methods Approach”, “the issue is not strictly a matter of quantitative versus qualitative methods, or even formal analysis versus critical theory”. (Kuhn 2018: 301) Kuhn argues for a “mixed method approach”, which is designed to serve a particular project or a particular set of research questions. I would add that even the opposition between the objective scientific method and the subjective artistic method should be questioned and opened up in the context of artistic research. These two perspectives can be synthesized in one research object, addressing and critically framing the friction accruing in this context.
In the field of visual arts and film, Rhythmic Trajectories has a very strong affiliation with the aesthetics of early experimental films. My work is influenced by artists like Oskar Fischinger, Walther Ruthmann and Viktor Eggeling, who started the movement Absolute Film. Their mission was to free their cinematic work from the representational and to produce abstraction exclusively with cinematic means. Although the original film sequence in this current project follows the traditional cinematic representation, the derivative visual layer, when extracted and presented separately, recalls the aesthetic of Absolute Film.
Another artistic tradition where Rhythmic Trajectories can be situated is Appropriation Art, especially the works of painter and conceptual artist John Baldessari, such as the series Double Bill or Commissioned Paintings. (See Fig. 7) He uses colour forms or simply the image of a hand to point at important elements of the image. The gesture of pointing at or accentuating in visual means can be tied to the approach used in this project.
To summarize this last section, Rhythmic Trajectories is deeply rooted in a purely formal and aesthetic artistic practice that can be traced both in film and visual arts. On the other hand, my project is engaging in a rather qualitative and aesthetic practice that seeks to produce a sensuous knowledge that is particular and retraceable at the same time.
The methodology of Rhythmic Trajectories is primarily to find a visual representation of movement and its manifestation as cinematic rhythm. This visualization acts as an annotation, a layer added to the original film sequence.
The process of creating this layer begins by identifying the visual and aural elements that, edited together, build a specific rhythmic phrase. For the identification of these elements, I use a combination of subjective and objective approaches. The subjective approach can be described as my intuitive knowledge as an editor, trained to look and listen to footage and extract moments, movements or beats as rhythmical modules. These audiovisual pieces are extracted and assembled in a way that, as a sequence, they form a perceivable cinematic rhythm. Accordingly, the first step in the process is to identify these elements in the given sequence based on the embodied perception of the cinematic rhythm. In the second step, the already identified, elemental rhythmic component has to be traced. In the case of a visual element, I use the motion tracking tool of Adobe After Effects to track the movement. In this step, I look for parts of the movement that bear a significance on the one hand and help simplify it, on the other hand. For example, in the complex and choreographed movement of the soldiers in the first étude, where different body parts are moving in different ways, I had to identify the prevalent direction of the movement. In the process of identifying the motion accents, motion beats, or emphasis points (Pearlman 2009: 52), I had to either chose a part of the complex movement that could signify the whole movement or, in other cases, I had to synthesize related movements into one movement trajectory.
My third step was to find a visual representation, a digitally generated geometric form, in a specific colour, to create an overall visual design that has a significant correspondence to the chosen film sequence. This means, for example, that the colour of the shape or line is picked (digitally selected) from the original film sequence. Thus, the colour of the cinematic rhythms’ visualisation is directly connected to the specific visual attributes of the moving element in the original film. Not only does the colour assign different geometric forms and utilize the same approach - but the form and the change of the geometric form also refer directly to the perceived movement in the film sequence. By relating my selection of the colours and geometric forms on colours and forms from the underlying original film, I build a congruent system. Each time the same movement occurs, I use the same shape, adding attributes such as motion, size and colour that correspond to the direction of the underlying film’s movement, its intensity and phrasing in the original sequence.
In the first étude, there is an additional step of sonification where the existing layer of visualization has been transposed to an audio layer. The process of derivation and formalization has been already accomplished so that the sonification corresponds to the visual layer. Each specific graphical element then is matched to a specific sound in order to have an additional audio layer instead of the original soundtrack of the film.
It is important to emphasize that these derivative layers (visual and sonic) are added to the original sequence without altering it. This nondestructive way of visualization enables the viewer to be part of the original perceptual experience, but with an added emphasis. This is similar to the gesture of pointing at something, where the appropriated object stays intact and, at the same time, the process of appropriation is revealed. Furthermore, the two additional layers (visual and sonic) that have been added to the original are able to exist as a stand-alone sequence as well. This option opens up the possibility to create different combinations of the layers.
In the creative work, I can separate four basic layers: the original visual layer with added visualization (A), original sound (B), abstract layer visualization without the original (C), and derived sonic layer (D). There are four combinations possible with these four elements (A+B, A+D, C+B, C+D). The first combination is the closest to the original film sequence and the last combination does not actually contain the original sequence, but only its rhythmic derivation.
Finally, I would like to address the question of choosing a specific film sequence. At the beginning of this artistic research, there were certain ideas about what cinematic rhythm can be and how it can be atomized and categorized. In my first attempt from a heuristic to a more systematic work, I wanted to choose film sequences where I believed I would find a simple rhythmical structure. In the first étude, the rhythm I focus on is in the choreographed movement, while in the second étude, it is in a dialogue scene. hence, in the first sequence, the visualization refers to the physical movement, while in the second sequence, I am only focusing on the gestures of the two actors expressed as a physical movement yet referring to emotions. Additionally, the first film sequence has been edited by me originally, so I could rely on the knowledge I have of constructing the cinematic rhythm within it.
I would argue that there is a “mixed approach”, which is analytical and artistic at the same time, applied during the process of the creation of these études. There is a certain amount of structured categorization of the cinematic rhythm, especially when the visual material is combined with these notes. On the other hand, the two études should be able to exist as a stand-alone visual essay, emphasizing the gesture of pointing at relevant elements. Moreover, in the combination of the different layers (original and derivative), there is a point where the abstract visualization leaves the original film sequence and plays in the tradition of the Absolute Film.
Video 1: From Wagah (Supriyo Sen, 2009, Germany-India, Editor: Szilvia Ruszev), visualization by the author.
The first étude uses a sequence from the short documentary Wagah (Supriyo Sen, 2009, Germany/India), which I edited. The sequence shows the oddly amusing choreography of the flag-lowering ceremony on the Indian-Pakistani border. The film is narrated through the perspective of three young boys who are selling DVDs at the ceremony to pay their school fees. Until recent times, the ceremony used to be held every evening, attracting thousands of people. The ceremony itself has a symbolic character embedded in the context of the history of the violent Partition of India and Pakistan after the British decolonization. Wagah is the only border crossing point along the 3000-km-long line dividing the two countries. The flag-lowering ceremony has a tediously prepared choreography, matched on both sides of the border to be the same, comprised of absurd clothes, movements and roll calls resembling military parades and tribal rituals at the same time.
As described earlier, the rhythmical structure in this étude has been deconstructed and formalized to reach an abstract annotation and visual representation. The ceremony, being a thoroughly designed choreography, consists of well-defined patterns of movement performed on both the Indian and Pakistani side. The way the sequence has been edited also represents this pattern of action and reaction, alternating between the two sides, showing both the soldiers and the viewers. This method of editing is based on a variational repetition, cutting between the two sides and matching duration and movement type so that this alternation can be perceived as a pattern. For example, the sequence starts with excited calls and clapping from both sides of the viewers. This is the warm-up before the core of the ceremony starts. The cheerleaders on both sides are firing up the national pride of the audience. Both the sound and the movement are very rhythmic. I made the decision to visualize the sound, which is more prevalent in this case. Here, and also later in the sequence, the loud calls have been visualized by circular forms, increasing size according to the length and loudness of the original sound. The colour of the circle was picked from the original sequence and stayed unchanged. The colour was also an important tool in the original film editing process and is used to make it easier for the viewer to identify the Indian and Pakistani sides of the border. After viewing the material, it was clear, for example, that green prevails on the Pakistani side since it is part of the clothes and also of the national flag. On the other hand, warm colours such as orange and pink represent the Indian side. They are part of the Indian flag (orange) and represented in the clothing. White has been used as the more neutral colour, in the case of movements and sounds being acted on both sides, such as the steps of the soldiers or the sound of the military trumpet which accompanies the ceremony. Other geometric forms used in the visualization are triangles and rectangles, each corresponding with specific attributes of the movement they visualize. For example, steps are performed as a sharp and distinct gesture represented by small moving triangles, whereas the sound of the trumpet is a long and loud expansion, represented by expanding white rectangular shapes.
The next step in this process was a sonification of the already existing visual layer. I chose the sounds to represent the attributes derived from the visual layer’s motion, with consideration for their lengths or other specificities such as sharpness, expansion, or dynamic. After some iterations, I realized that the sounds I was using were less abstract than the graphical forms. This pushes the sonic layer to a kind of satirical interpretation. The sound of horns and drums amplifies the absurdity of the movements seen in the sequence.
Working through all possible combinations of layers (as described earlier), there were four different versions left at the end, each addressing the original research question of visualizing the cinematic rhythm in a slightly different way. In the first version, only an abstract visual layer has been added to the original film sequence. In the second the original sound has been replaced by a new sound layer. In the third version, the visual layer has been reduced solely to the visual abstraction and combined with the original sound. Lastly, the fourth version is comprised of both newly produced visual and sonic derivatives of the original film sequence. What has been presented here as Étude I. is a combination of all four versions following the process of their production. In the beginning, they are introduced step by step, then later, changing between them to finish on the most abstract fourth version. In the process of the creation of this visualization, my goal was to find a visual representation of the physical movement in the sequence. Although the initial intention was to create a notation referring to specific elements of the sequence without representing its meaning, it was intriguing to observe how this allegedly objective approach shifted during the process. Moving from the representational toward the abstract, the four versions shift from the analytic toward the artistic allowing the original film sequence to transform into an artistic object of appropriation. The different variations and connections of the original and visualized or sonified layers resulted in the different interpretations addressing the question of sensual knowledge about cinematic rhythm in different ways.
Video 2: From A Woman Under Influence (John Cassavetes, 1974, USA, Editor: David Armstrong), visualization by the author.
Étude II focuses on the rhythm of gestures using a scene from the film A Woman Under Influence (John Cassavetes, 1974, USA), edited by David Armstrong. This is a dialogue scene, showing the main characters, Gena Rowlands and Peter Falk, who plays her husband. It is an emotionally charged scene, a fight between two people struggling in love and despair. The scene has a specific arc, where gestures and looks are felt to be more significant than words. After the very quiet beginning where no words are spoken but gestures play against each other, there is a higher pitched phase where the emotions are explicitly acted out. This is followed by the third phase of consolidation getting quieter and slower again.
It was clear that, as opposed to the first étude where the cinematic rhythm consists of the repetition of different visual and sonic patterns, the cinematic rhythm in this sequence has been defined by the gestures of the two actors. Formalizing these gestures was the first step in the process of visualization where I applied the motion tracking tool of Adobe After Effects. In order to use this specific plugin, I had to choose significant visual points to be followed. The complexity of a gesture had to be reduced to data points which can be apprehended by the software. In each shot, I chose a point corresponding to the eyes and to each of the hands of the actors. This resulted in complex movement trajectories for each shot that could be used for a visualization.
At this point, another step of reduction had to be applied because I realized that the complexity of these separate motion tracks made the visualization too self-referential and therefore unsuitable for the purpose of this research. To create synthesis, I manually summed up the three motion-tracking lines and created a new gesture trajectory for each shot. As a next step, a write-on effect has been used to create a brush animation, where the position of the brush is defined by the key frames of the motion tracking.
The colour of each meandering line is derived from the original shot and amplified to correspond to the emotional expressiveness of the scene. It was interesting to realize how much the derived colours (blue and pink) follow certain gender stereotypes. Without being explicit in the original film sequence, these two colour schemes emerged from the visualization referring to the two colours commonly associated with the gender of man (blue) and woman (pink). Another finding of the visualization is that the dynamic and intensity of the gesture recalls another gender stereotype of women being more emotional in this binary.
When creating this étude, it was very important to preserve the original cuts, so the viewers can have their own sensual experience of the original film sequence. An additional formal decision was to gradually desaturate the original film sequence before adding the visualization, in order to reduce the visual elements and let the abstract layer emerge. Furthermore, the original film was also embedded in the frame as a smaller picture. This seemed to be a necessary step in order to allow space for the flow of the gestures that continue off-frame. Creating colourful lines as a visual representation of gestures emphasizes the idea of the movement that creates a trajectory which refers not only to the present moment of the visible frame but to all connected frames in the past and the future. In other words, inscribing an animated line back on to the original film sequence enables visually synthesizing the wholeness of the movement. The decision to let the trajectories of the gestures of each person exist in the next shot relates to the cut that joins the two shuts and creates the experience of the conversation of the two people. It points at the fact that each cut creates a special relationship between adjacent shots, both literally and in an abstract way. In the same way, each gesture is a reaction to the gesture of the other, continuing and disrupting it at the same time. In the progression of the étude, there is a brief moment, where the original visual layer completely dissolves and thus giving space for an abstract version of the continuing choreography of human gestures.
Rhythmic Trajectories can be defined as an artistic research project aiming to find visual representation for what I call cinematic rhythm. One of my research questions was to find a definition of the term cinematic rhythm and to identify the elements it consists of on different levels of a film sequence (in a single shot, as a physical, emotional or event movement). The formal visualization of such data, I would argue, is key in understanding how a complex audio-visual experience has been shaped by the editing and how it is experienced by the spectator as a sensual and embodied experience. The project is aimed to be an open-ended, qualitative research utilizing methodologies from various fields, such as information visualization, the humanities and artistic practices.
During the process of the manual dissection of the two film sequences presented in this paper, the most interesting phenomenon I observed was the allegedly objective exploration turning into an artistic expression. When does formal, non-linguistic description turn from a commentary into a subjective interpretation, and finally into an independent artistic object?
In the first étude, adding a new sound layer shifted the perspective from the scientific to the artistic mode. Replacing the original audio with abstract but recognizable sounds “excavates” both the absurdity of the movements and the choreography seen in the sequence. The graphic elements I used to address the cinematic rhythm resemble a notation, without claiming a reciprocal and generalizable meaning. In the second étude, the aesthetic decision to form the animated lines in a rather organic way, as opposed to the simple graphical forms of the first étude, amplifies the tendency of an artistic interpretation. Both études are created by using digital tools such as automated tracking in Adobe After Effects. Nevertheless, the end form of the visual representation is based on a rather particular and manual approach, where the steps of derivation, reduction and synthesis are conducted by the author.
The next step in the research process will be to adopt a similar methodology to a wider range of film sequences that are representational for other types of cinematic rhythm (such as event rhythm for example). Experimenting with computational tools, such as programming languages and computer vision, could open up the way for a more generalized formal approach, while at the same time, this step would also challenge the notion of artistic research in a new and exciting way.
Szilvia Ruszev
Division of Media Arts + Practice, University of Southern California
1 https://vimeopro.com/filmstudiesff/audiovisual-film-studies-for-free
2 https://vimeo.com/kogonada
3 https://www.cinemetrics.lv
4 http://cinemetrics.fredericbrodbeck.de/
5 https://vimeo.com/kevinlferguson
6 https://artcom.de/en/project/the-invisible-shape-of-things-past/
7 http://thevatproject.org/
8 http://www.anvil-software.org/
Szilvia Ruszev is media practitioner and scholar focusing on the notion of montage. Her artistic work relates to very personal moments, certain states of emotional solitude in relation to the Other. Her professional work as film editor represents a comprehensive approach to independent filmmaking with more than 30 films to her credit. Her broader research interest focuses on nonverbal forms of knowledge acquisition, montage theories, and politics of post-cinema.
Szilvia, born and raised in Hungary and Bulgaria, studied Film Theory at the Eötvös Loránd University (Hungary) and Film Editing at the Film University Babelsberg (Germany), where she worked as a faculty member for six years. As editor, she collaborated with internationally acclaimed directors such as Peter Greenaway, Anders Østergaard, and János Szász. Her award-winning work has been part of numerous international film festivals and exhibitions such as Karlovy Vary IFF, TIFF Toronto, Berlin IFF. Currently, she is pursuing a Ph. D. degree in Media Arts + Practice at the University of Southern California.
Bochow, Jörg. 1997. Das Theater Meyerholds und die Biomechanik. Berlin.
Dancyger, Ken. 1997. The Technique of Film and Video Editing: Theory and Practice (2nd ed.). Boston.
Heftberger, Adelheid. Forthcoming 2019. Digital Humanities and Film Studies. Visualising Dziga Vertov's Work. Basel.
Joost, Gesche, Buchmüller, Sandra, and Bieling, Tom. 2009. “Audio-visual Rhetoric and its Methods of Visualization.” In Undisciplined! Design Research Society Conference 2008, Sheffield Hallam University, 16-19 July 2008. http://shura.shu.ac.uk/556/1/fulltext.pdf
Klein, Julian. 2012. “What Is Artistic Research?”
https://www.researchcatalogue.net/view/15292/15293
Kuhn, Virginia. 2018.”Images on the Move: Analytics for a Mixed Methods Approach.” In The Routledge Companion to Media Studies and Digital Humanities, edited by Sayers, Jentery, 300-310.
Manovich, Lev. 2010. “What is Visualization.” http://manovich.net/index.php/projects/what-is-visualization.
Manovich, Lev. 2013. “Visualizing Vertov.” http://softwarestudies.com/cultural_analytics/Manovich.Visualizing_Vertov.2013.pdf
Mitry, Jean. 2000. The Aesthetics and Psychology of the Cinema. Bloomington.
Moholy-Nagy, László. 1924. Partiturskizze zu einer mechanischen Exzentrik. Munich.
Pearlman, Karen. 2009. Cutting Rhythms. Shaping the Film Edit. Burlington.
Reisz, Karel. 1953. The Technique of Film Editing (1st ed.). Boston.
Slager, Henk. 2004. “Methodocity.” In Artistic Research. Lier en Boog. Series of Philosophy of Art and An Theory, Vol. 18, edited by Annette Balkema and Henk Slager, 13.
Thompson, Kristen and David Bordwell. 2011. Film history: an introduction. New York City.
Sen, Supriyo. 2009. Wagah. Detail Film.
Cassavetes, John. 1974. A Woman Under Influence. Faces.
Ruszev, Szilvia. 2018. “Rhythmic Trajectories – Visualizing Cinematic Rhythms in Film Sequences.” Women Cutting Movies: Editors from East and Central Europe (ed. by Adelheid Heftberger and Ana Grgic). Special Issue of Apparatus. Film, Media and Digital Cultures in Central and Eastern Europe 7. DOI: http://dx.doi.org/10.17892/app.2018.0007.146
URL: http://www.apparatusjournal.net/
Copyright: The text of this article has been published under https://creativecommons.org/licenses/by/4.0/ This license does not apply to the media referenced in the article, which are subject to the individual rights owner's terms.