Deep Learning & Algorithm Research

Sharing research on Transformers, Large Language Models, Position Encoding, and cutting-edge AI techniques

Misaya Yang

2Articles

1Categories

184Views

IsoHC: How I Solved the Stability Crisis in Hyper-Connections

Feb 6, 202690 reads10 min read

When I first encountered Hyper-Connections, I was excited by its potential. But as I scaled up my experiments, training became unstable at depth. In this post, I share how I developed IsoHC—a new manifold constraint that preserves both global stream mean and ℓ₂ energy along residuals.

Dynamic Frequency RoPE: My Journey to Solving Long-Sequence Transformers

Jan 23, 202694 reads7 min read

When I first started working with Transformers on long sequences, I quickly ran into a fundamental problem: how do we help these models understand position in sequences that are much longer than what they were trained on? In this post, I share how I developed Dynamic Frequency RoPE (DF-RoPE).