OpenArch – PyTorch implementations of modern LLM architectures
Recorded: Sept. 14, 2026, 10 a.m.
| Original | Summarized |
GitHub - anuj0456/OpenArch: PyTorch implementations of modern open-source LLM architectures (Llama, Qwen, DeepSeek, Gemma, GPT-OSS, Kimi, and more) — written from scratch for readability and learning, based on Sebastian Raschka's LLM Architecture Gallery. · GitHub Skip to content Navigation MenuSign inAppearance settingsPlatformAI CODE CREATIONGitHub CopilotWrite better code with AIGitHub Copilot appDirect agents from issue to mergeMCP RegistryIntegrate external toolsDEVELOPER WORKFLOWSActionsAutomate any workflowCodespacesInstant dev environmentsIssuesPlan and track workCode ReviewManage code changesCode QualityEnforce quality at mergeAPPLICATION SECURITYGitHub Advanced SecurityFind and fix vulnerabilitiesCode securitySecure your code as you buildSecret protectionStop leaks before they startEXPLOREWhy GitHubDocumentationBlogChangelogMarketplaceView all featuresSolutionsBY COMPANY SIZEEnterprisesSmall and medium teamsStartupsNonprofitsBY USE CASEApp ModernizationDevSecOpsDevOpsCI/CDView all use casesBY INDUSTRYHealthcareFinancial servicesManufacturingGovernmentView all industriesView all solutionsResourcesEXPLORE BY TOPICAISoftware DevelopmentDevOpsSecurityView all topicsEXPLORE BY TYPECustomer storiesEvents & webinarsEbooks & reportsBusiness insightsGitHub SkillsSUPPORT & SERVICESDocumentationCustomer supportCommunity forumTrust centerPartnersView all resourcesOpen SourceCOMMUNITYGitHub SponsorsFund open source developersPROGRAMSSecurity LabMaintainer CommunityGitHub StarsArchive ProgramREPOSITORIESTopicsTrendingCollectionsEnterpriseENTERPRISE SOLUTIONSEnterprise platformAI-powered developer platformAVAILABLE ADD-ONSGitHub Advanced SecurityEnterprise-grade security featuresCopilot for BusinessEnterprise-grade AI featuresPremium SupportEnterprise-grade 24/7 supportPricingSearch/Sign inSign upAppearance settings You signed in with another tab or window. Reload to refresh your session. Dismiss alert anuj0456 OpenArch Public
Notifications
Fork
Star Code Issues Pull requests Actions Projects Security and quality Insights
Additional navigation options
Code Issues Pull requests Actions Projects Security and quality Insights
mainBranchesTagsGo to fileCodeOpen more actions menuLatest commit History89 Commits89 CommitsFolders and filesNameNameLast commit messageLast commit dateimageimage multimodalmultimodal texttext .gitignore.gitignore CONTRIBUTING.mdCONTRIBUTING.md LICENSELICENSE README.mdREADME.md requirements.txtrequirements.txt View all filesRepository files navigationREADMEContributingMIT licenseMore itemsOpenArch Python implementations of modern open-source LLM architectures — written from scratch, one model at a time. This repository contains hand-written PyTorch implementations of the model architectures cataloged in Sebastian Raschka's LLM Architecture Gallery. Each model is implemented to the best of my knowledge from the original papers, technical reports, reference config.json files, and the excellent writeups by Sebastian Raschka and Machine Learning Mastery. Attention: MHA, GQA, MQA, MLA, sliding-window, linear/DeltaNet hybrids Reading the official model code can be hard because production repos optimize for speed, sharding, and backward compatibility. This repo optimizes for reading. Implementations marked ✅ are usable for forward passes; those marked 🚧 are under construction. Modality Text Llama 2 Llama 3 OLMo 2 DeepSeek R1 Gemma 3 Mistral 3 Llama 4 Maverick Qwen 3 30B-A3B Kimi K2 GLM 4.5 GPT-OSS Grok-2.5 Multimodal Qwen3 Image The full target list mirrors the 72 architectures in the Architecture Gallery. Contributions toward any of them are welcome. Each model lives in its own folder with respective model.py and a short README.md describing the architectural choices and references used. Pick an unimplemented model from the gallery and add a model.py for it Please open an issue before starting a large piece of work so we can avoid duplicating effort. Implementations should prioritize readability over performance — this is a learning resource first. Sebastian Raschka — for the LLM Architecture Gallery, the Big LLM Architecture Comparison series, and the LLMs From Scratch book and codebase. The architecture diagrams, fact sheets, and side-by-side comparisons in the gallery are the primary reference behind every model in this repo. Any errors in the implementations here are entirely my own. Footer © 2026 GitHub, Inc. Footer navigation Terms Privacy Security Status Community Docs Contact Manage cookies Do not share my personal information You can’t perform that action at this time. |
This repository provides hand-written PyTorch implementations of modern open-source Large Language Model (LLM) architectures, designed primarily as a resource for learning and architectural comparison based on Sebastian Raschka's LLM Architecture Gallery. The core objective of the project is to prioritize readability, allowing users to explicitly see the structural choices made in LLM design, such as attention mechanisms, normalization techniques, positional encodings, and Mixture of Experts (MoE) routing, side-by-side. This approach contrasts with production libraries that often optimize for speed, sharding, and backward compatibility. The repository focuses on making the subtle yet critical differences between various LLM designs explicit. Key architectural components that are systematically implemented and compared include various attention types like Multihead Attention (MHA), Grouped Query Attention (GQA), Multi-Query Attention (MQA), and hybrid methods such as sliding-window attention, as well as different normalization strategies including RMSNorm, post-norm, and QK-Norm. Furthermore, the implementations explore diverse positional encoding methods such as RoPE and NoPE, and various decoder structures, including dense versus sparse MoE configurations. A substantial portion of the repository is dedicated to implementing specific models, offering concrete examples of these architectural choices. Implementations are available for several large and medium-scale models, including GPT-2 XL, Llama 2, Llama 3, OLMo 2, DeepSeek R1, Gemma 3, Mistral 3, Llama 4 Maverick, Qwen 3, and Kimi K2. These implementations detail the specific combinations of the architectural elements mentioned previously, such as which normalization and positional encoding methods are utilized by each model. The project also includes support for multimodal models, demonstrated through the PaliGemma implementation. The repository is structured to house each model within its own dedicated folder, containing the necessary model definition file and documentation describing the architectural decisions and references used for that specific implementation. This modular layout facilitates easy study and comparison of the foundational engineering choices across different families of LLMs. The repository actively seeks contributions aimed at deepening the understanding of LLM construction. Potential contributions include implementing missing architectures from the gallery, documenting the architectural specifics of existing models, developing forward-pass tests to verify weight loading and output matching, and refactoring shared components. The project emphasizes that implementations should prioritize clarity over raw performance, serving fundamentally as a learning resource. The work is built upon significant educational foundations, acknowledging the contributions of Sebastian Raschka for the foundational LLM Architecture Gallery and the LLMs From Scratch book and codebase, as well as Jason Brownlee and the Machine Learning Mastery team for providing accessible tutorials on deep learning and transformer architectures. The work is licensed under the Apache License 2.0, and the disclaimers note that the implementations are intended as educational assets and are not affiliated with or endorsed by the original model creators, advising users to consult official implementations for production use. |