Pinxin Liu (Andy)

I am a graduate student in the Computer Science Department at the University of Rochester. I received my B.S. in Computer Science with highest honor distinction from the University of Rochester.

My research interests are multimodal large language models, visual agents, video generative models, and human-behavior understanding and generation.

Email  /  Google Scholar  /  GitHub  /  CV

Seeking Research / Applied Scientist / ML Engineer positions in Multimodal Language Models, Human Video Generation (3D / Diffusion), Video Understanding (MLLM), Audio-Visual Learning, and 3D Perception & Understanding. My master's research focused primarily on language models and human-centered perception/generation.

• LLM-based Motion Understanding & Generation: Integrated 3D body mesh representations into an LLM tokenization pipeline with a diffusion/flow-matching generation head, enabling unified cross-modal understanding and controllable fine-grained body motion generation; unlocking instruction-following avatar animation at scale for interactive digital human experiences on Meta Quest and AR devices.
• Automatic Data Annotation at Scale: Designed an automated annotation pipeline converting raw 3D pose and mesh sequences into structured semantic language descriptions without human labelers; reducing data curation cost by orders of magnitude and enabling training on million-scale motion corpora that was previously infeasible.

• Foundation Human Video Generation: Built a large-scale diffusion-based foundation model with multi-stage training on 100M+ video-text pairs using FSDP across multi-node GPU clusters, establishing the core generative backbone for the company's film production pipeline.
• Speech-to-Gesture Alignment: Developed cross-modal alignment between pixel-level motion and speech semantic embeddings; shipped an upper-body co-speech gesture animation system in production for film post-production clients.
• Pixel-Level Artifact Refinement: Shipped a pixel-space refinement module suppressing boundary artifacts across diverse backbones without retraining, adopted as a plug-and-play quality layer across video generation products.

We introduce Open3DFaceVid and cast facial-parameter modeling as a language problem, enabling bidirectional Motion2Language and Language2Motion for text-conditioned facial animation and understanding.

We propose a context-aware gesture representation for co-speech gesture video generation.

A comprehensive survey on video understanding techniques powered by large language models.

We propose a latent shortcut mechanism for co-speech gesture generation with spatial-temporal modeling.

We present a kinematic-aware approach for human motion understanding and generation.

We introduce MMPerspective, the first benchmark to systematically evaluate MLLMs' understanding of perspective through 10 tasks across perception, reasoning, and robustness (2,711 images, 5,083 QA pairs). Accepted to NeurIPS 2025 DB Track.

We propose GaussianStyle, integrating 3D Gaussian Splatting with StyleGAN for head avatars. The framework preserves expression and pose with Gaussians while projecting implicit volume into StyleGAN for high-frequency detail, achieving state-of-the-art in reenactment, novel view synthesis, and animation.

A comprehensive survey on generative AI for cel-animation.

We present a method to generate a drivable toonified avatar. Given a monocular video and a written instruction about the avatar style, it can generate a toonified avatar that can be animated in real time.

We attach the multi-combined tri-plane structure for monocular photo-realistic volumetric head avatar reconstructions.

We apply the mix-resolution images in one-shot talking head training. The resolution could achieve 512px from 256px in previous.

Template from Jon Barron.  © 2025 Pinxin Liu