Until the World Falls Silent
Introduction
This motion capture project was chosen from my inspiration of human-AI contrast. In assessment 1, I designed a story named ‘Divided Presence’, which successfully drew my friends’ attention. After a meeting, we decided to extend my idea and implement it together. So, I got a team that would provide great support for my ideas. In the following sections, I will review the whole process of this artwork from my personal perspective and provide evaluation and summary.
Process & Timeline
Week1-5: Project Initiation
In the initial stage of this project, my concept and inspiration were derived from my reflections and thoughts on the development of artificial intelligence from 2020, which include the replacement and the unemployment issue brought by AI.
The foundational and the most important inspiration comes from an artwork created by Memo Akten and Quayola in 2011 named ‘Forms’ (figure 1). They recorded different athletes’ motion in sport competition and transferred these movements into abstract forms to present them. This artwork explored techniques of extrapolation to shape abstract forms, visualizing hidden relationships – power, balance, grace and conflict – between the body and its surroundings (M, Akten & Quayola 2011).
This project may reveal the cold, data-centric perspective of AI: rational, calculated, and basically removed from the humanity of human beings. At first, the theme was designed to encourage public critical thinking: What does it mean for motions to be "understood" by a computer?
However, after listening to Liu’s presentation with his concept of the four seasons to a human, which symbolised a whole life of a human, we held a meeting and decided to change the narrative focus on the contradictions between AI and humans, the overall path of the development of AI, and the predictions for the future. So, at the end week five, we decided to divide the story into four parts, each reflecting one of the four seasons: spring, summer, autumn, and winter.
Week 6：Group Formation and Early Planning
In week 6, we carried out the group formation and task allocation, then we also made further efforts on the concept design as well as several other details. These included defining the overall artistic and sound style, narrative framework, and drafting the initial version of the script, among other tasks.
As the Wolf’s team’s leader, the first thing after carrying out the group is to give each member a clear task. This may test my ability to be a leader and to assign people well. Our group has seven members. Jie Liu had a background in directing and video editing, so I asked him to write the first draft of the story script and later do the video editing when we finished our motion capture. Considering Sicong Chen had experience in Maya, I asked her to do and lead the work in Maya. The other members also had balanced skills, so I asked them to do more work in MotionBuilder and Maya particle effects.
For the motion capture part and the following adjustment, I had divided the four seasons into two groups. Spring and winter part had complex particle effects but simple character actions. Summer and autumn had simple particles but many potential problems in motion capture. So, I assigned Sicong to be the leader to finish the spring and winter sections. I would participate in and lead the making of the rest parts.
As the leader of the whole team, I believe that I must try my best to participate in all parts, offer constructive feedback, provide essential support to each member, and set work deadlines.
Firstly, I set up a shared zone on Teams to facilitate access the file exchange for everyone. In addition, since all of us are Chinese members, we also have a WeChat group for quick communication. Then we started to discuss about the story itself, which was created by Liu as the first draft.
For the artistic style, we prefer the cartoon characters inspired by the game Human: Fall Flat and adopted a simplified low poly style. This choice allowed our block particles created with the Mesh function in Maya to fit in more naturally, avoiding any sense of weird and inconsistency. Despite the style is not that realistic, it adds playfulness and effectively emphasizes the message we wanted to convey.
To achieve this effect, I sourced free open-sourced assets from Sketchfab, including models of trees and rocks. I also participated in the background design. I guided my teammates on how to build the background in Maya and create a mood board for colour concepts. I provided then with accurate colour references and initially set up a very simple scene and camera for reference, although these were not preserved in the final version.
In addition, I was responsible for arranging the character modelling, which went through three revisions. The first model I found was too angular and not smooth. It did not match the expected visual effect. The second model had major topology issues and weigh painting issues as well. Finally, I found a suitable model, but because the original was composed of triangles, I cleaned it and did retopology to convert it into quads for our use. Then I applied a quick rig to the model in Maya and then created suitable weigh painting to it manually.
Week 7: Motion Capture Recording
The motion capture recording date for our team was on July 17. Because the workload of our story was larger than other groups’, we arranged two rehearsals in UNSW. One rehearsal was held three days before the recording, during which we finalized the storyboard and performed while taking notes and discussing adjustments of details. For example, we increased the amplitude of certain actions and deleted or replaced movements that were too compact to avoid losing sampling points during the mocap recording. Every member contributed many new ideas and suggestions. After extensive adjustments and rehearsals, we finished the actions as we expected. During the morning of July 17, Wei and I performed a reference version, which was recorded to provide guidance for the mocap actors in the official session that afternoon. During the motion capture recording, I was responsible for operating the computer software.
Finally, we got successful motion capture data on that weekends from our teacher. However, when we moved on to the next stage of data processing, especially my part, I found that some parts of the motion capture data contained problems that were difficult to fix.
Week 7: Motion Capture Recording
After getting the original mocap data, all the members devoted themselves into their respective tasks. In this part, I was responsible for fixing the character motion data for summer and autumn. Unfortunately, the mocap data of Emma, the actress for autumn section, contained some issues that were difficult to fix. I will present the overall process of repair in the following paragraphs.
Firstly, when I received the original data sent by my teacher, I opened it in MotionBuilder to find out mistakes frames by frames. Then, I created one character named Emma and another named Human, which was bound to my model. After done that, I followed the standard process to clean up the data. I set Emma’s motion data as the source and connected it to my character, then baked it onto the control rig. In this way, an editable motion animation was prepared and ready for manual adjustments.
The next step I had done was to delete the incorrect keyframes in the animation curves (F-Curves) interface that caused jittering in the character’s detailed parts like hands and feet. Some of the hand and foot trackers seemed to contain errors, which led to noticeable and weird jittering. For this part, I asked the teacher for advice and received very helpful guidance. The solution was to delete the faulty keyframes, then create a new animation layer, make the necessary adjustments on this layer, and change the weight values to repair and smooth the character animation.
After fixing the detailed errors in the character animation, the next step was the most difficult one. In the autumn section, I found that the mocap data of the Emma character showed a misalignment of the hips. After discussing with my teacher, I learned that this might because the actor’s position was too close to the boundary, which could reduce the sensitivity of the cameras and led to misalignment and jitter. This jitter occurred at the hips and spines, so it was too difficult to fix. If I modified this area, they would affect the animation of all other joints. I tried all the possible methods suggested by the teacher, but none of them worked, and some solutions could not be applied due to time constraints.
To solve this problem, I found that the issue only appeared in the last one-fifth of this section. Consequently, I deleted the final portion of the animation and chose another similar motion from the Mixamo open-source website. I then used the Story tool in MotionBuilder to stitch the motions together and refined the transition with an additional animation layer. Fortunately, the stitching was very successful, and it was not easy for audience to notice any difference in the movements, which remained highly coherent. Using the same method, I also improved the ending of the summer section by adding a short three-second fall animation to emphasize the conflict between humans and AI.
With this, all character animation work was completed. The entire process took me about nine days.
Week 9-10: Final Rendering and Compositing
When each member finished their own part and submitted it according to the timeline I had set, I checked all the resources and refined them. After that, I sent the files to Liu, who would replacing the camera and lights in Maya to match his final editing requirements. On August 5, Liu completed all the shot editing and shared the Maya project package with us. After he provided the specific rendering needs, I finished the rendering of the autumn part on August 6. Finally, our final output was edited by liu on August 9, marking the completion of the entire project.
Evaluation
Firstly, our work was highly organized, which benefited from my time planning and task allocation. I accurately recognized each member’s strengths and assigned them tasks that allowed their skills to reach the maximum value. I frequently set stage deadlines to everyone in this group after each Friday’s class, and everyone submitted their work on time. I also communicated with the group almost every day in our online WeChat group, sharing new ideas or technical details.
We also made thorough preparations for future tasks. For example, we built the background and confirmed the overall art style very early and repeated rehearsals of the complex motion capture actions before the official recording. These steps could reduce unexpected issues. In addition, our group was always willing to help each other, especially when some members struggled with technical issues. This prevented the team from getting stuck and greatly improved efficiency. For instance, the image below shows a PDF guide I made and shared with the group, teaching everyone how to convert videos to 25fps while keeping the keyframes aligned.
Screen shot: The guide in Chinese about how to convert videos to 25fps while keeping the keyframes aligned
However, there is still space for improvement. For example, due to my limited experience with motion capture technology, I encountered errors in data recording that were difficult to repair, as there has been no further verification or confirmation. In future practice, I should pay closer attention to confirming the completeness of the data in time to avoid such these issues. In addition, as the team leader, I was indeed involved in every stage and provided guidance, but this to some extent restricted the individual freedom of the members. Moving forward, I should focus more on offering directional guidance while reducing detailed requirements.