Kilian Haefeli

Hi, I'm Kilian. I do Deep Learning Engineering and Research and study at ETH Zurich.

I am interested in Neural Networks, their training dynamics, internal circuitry, and how to scale them.

I currently pretrain Large Language Models at Cohere and previously worked on RAG at Aleph Alpha.

Email  /  GitHub  /  Google Scholar  /  LinkedIn  /  CV

Publications

HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context

Extending the HiPPO framework to demonstrate that continuous SSMs can approximate the next state of a dynamical system from previous state observations provided in-context.

Diffusion Models for Graphs Benefit From Discrete State Spaces

Diffusion Model for Graphs using discrete Bernoulli Perturbations over edge connections. This approach results in maintained sparsity, and sampling with much less steps resulting in new SOTA graph generation.

Experience

I am currently working as an intern of technical staff on pretraining and performance engineering at cohere .

Prior to that I have worked on Retrieval Augmented Generation at aleph alpha .

During my undergrad I co-founded a company called Airica which specialized in indoor air quality sensors and accompanying software to analyze, manage and predict room occupancy using indoor air-quality measurements. The company was acquired by Logitech, where I continued to work on sequence models for time series prediction for a brief time.

Education

I am currently attending the EECS masters at ETH Zurich, where I focus on Deep Learning, Optimization and Parallel Systems.

in spring 2024 I attended University of Toronto specializing in Information Theory, Statistical Learning and Systems for Machine Learning.

Prior to that I have completed my bachelors at ETH in EECS majoring in Signal Processing and Machine Learning.

Projects

Flash Attention in C CUDA

Flash Attenton in ~300 lines of pure Cuda C, outperforming native PyTorch Attention. Manually implemented and profiled methods such as SMEM tiling, SMEM padding for reduced bank conflicts, 2D Blocktiling and vectorized memory access.

Optimizable DeepPoly

Implemented an optimizable Neural DeepPoly Verifier as custom torch modules.

Efficient Neural Representation Learning for Star-Convex Boundaries

Graph neural networks for modelling the evolution of the phase-boundary surface generated by the Stephans equation during 3D laser printing.

Design and source code from Jon Barron's website