Note Efficient Training | Check TSAIL(https://ml.cs.tsinghua.edu.cn/), GaLore(https://github.com/jiaweizzhao/GaLore) for more.

Note Scalibility | Scalability Research Paradigms for Large Models Research on the scalability of large models typically involves several common paradigms: Scaling Laws Studying how model performance improves as parameters, data, and compute increase, often fitting empirical power-law relationships. This helps predict the returns of training larger models. Key papers: Kaplan et al., "Scaling Laws for Neural Language Models" (2020) Henighan et al., "Scaling Laws for Autoregressive Generative Model

Note DeepOps | SLURM skills range from basic job submission to expert-level cluster management. Level 1 covers essential commands like sbatch, srun, squeue, enabling users to run simple jobs. Level 2 focuses on efficient resource requests (--mem, --cpus-per-task), logging, and job history (sacct). Level 3 introduces advanced scheduling techniques such as job dependencies (--dependency), priority management, partitions, and job arrays. Level 4 involves expert skills like diagnosing scheduler behavior

lldacing/flash-attention-windows-wheel

Updated May 31 • 238

Note Speed-up attention | Other work includes xformers, sage attention(https://github.com/thu-ml/SageAttention). NB CUDA complier.

Embodied-CoT/ecot-openvla-7b-oxe

Robotics • 8B • Updated Jul 10, 2024 • 824 • 2

Note Minkowsky style Sparse Tensor, to speed up sparse voxel in space. | recommend of pruning of Large Reconstruction Model - style or NeRF - style work, TensorRF, distillat field of 3D Gaussian. Pretrained LRM could be leveraged for Mesh generation. Some game eigine companies: unity, roblox(https://github.com/Roblox/cube/tree/main/cube3d) Recent impressive works and pre-trained backbones: Hunyuan3D, TRELLIS, InstantMesh, Zero123, ... Also check Dream- style works for more generation