Deep Learning (DL) neural networks
Universal function approximation, backpropagation, stochastic gradient descent
🌿 Natural Language Processing (NLP) text & language
Sequence modeling, attention mechanisms, transformer architectures
🔷 Large Language Models (LLMs)
Foundation models trained on massive text corpora for general language understanding and generation.
Autoregressive modeling: P(x₁,...,xₙ) = ∏P(xᵢ|x₁,...,xᵢ₋₁), self-attention, layer normalization
GPT Family (OpenAI)
- GPT‑1 2018
- GPT‑2 1.5B params
- DistilGPT‑2 compressed
- GPT‑3 175B params
- GPT‑3.5‑turbo
- GPT‑4 / GPT‑4‑turbo
- GPT‑4o multimodal
- GPT‑4.1 latest
- o1 / o1‑mini reasoning
- Decoder-only transformer, causal attention masking
Claude Family (Anthropic)
- Claude 1.0
- Claude 2.0 / 2.1
- Claude 3 Haiku fast
- Claude 3 Sonnet balanced
- Claude 3 Opus most capable
- Claude 3.5 Sonnet enhanced
- Claude 4 Sonnet latest
- Constitutional AI, RLHF training, safety objectives
Gemini Family (Google)
- LaMDA conversation
- PaLM / PaLM 2
- Gemini Pro
- Gemini Ultra
- Gemini 1.5 long context
- Gemini 2.0 multimodal
- Mixture of experts (MoE), Pathways architecture
Meta LLaMA Family
- LLaMA 1 7B-65B
- LLaMA 2 chat variants
- Code Llama programming
- LLaMA 3 / 3.1 405B
- LLaMA 3.2 multimodal
- RMSNorm, SwiGLU activation, rotary positional embeddings
Open Source Alternatives
- Mistral 7B / 8x7B (Mixtral)
- Falcon 7B / 40B / 180B
- Alpaca (Stanford)
- Vicuna (LMSYS)
- MPT (MosaicML)
- Qwen (Alibaba)
- Yi (01.AI)
- Various architectural optimizations, efficiency improvements
🔷 Specialized NLP Models
Understanding Models (Encoders)
Bidirectional attention, masked language modeling objective
- BERT / RoBERTa
- Masked LM: P(xᵢ|x₋ᵢ), bidirectional transformer
- DeBERTa
- Disentangled attention, enhanced mask decoder
- ELECTRA
- Replaced token detection, generator-discriminator training
- DistilBERT compressed
- Knowledge distillation from teacher to student model
Text-to-Text Models
Encoder-decoder architecture, span denoising objectives
- T5 / UL2
- Text-to-text unified framework, span corruption
- BART
- Denoising autoencoder, various noise functions
- Pegasus summarization
- Gap sentence generation pre-training
Embedding Models
Contrastive learning, cosine similarity, semantic vector spaces
- Sentence-BERT
- Siamese networks, triplet loss
- OpenAI text-embedding-ada-002
- E5 / BGE
- Instructor task-specific
- Task-aware embeddings, instruction following
🔷 Code Generation Models
Programming language modeling, syntax-aware attention
- GitHub Copilot (OpenAI Codex)
- CodeT5 / CodeGen
- StarCoder / Code Llama
- Replit Code v1.5
- Amazon CodeWhisperer
- Tabnine
- Abstract syntax tree (AST) modeling, code completion probability
🌿 Computer Vision (CV) images & video
Convolutional operations, spatial hierarchies, visual feature learning
🔷 Object Detection & Segmentation
Non-maximum suppression, intersection over union (IoU), anchor-based/anchor-free methods
YOLO Family
- YOLOv1 → v3 early versions
- YOLOv4 / YOLOv5
- YOLOX / YOLOv6
- YOLOv8 ultralytics
- YOLO11 latest
- Grid-based detection, confidence × class probability
R-CNN Family
- R-CNN
- Fast R-CNN
- Faster R-CNN
- Mask R-CNN segmentation
- Region proposal networks, ROI pooling/align
Modern Detection
- SSD (Single Shot Detector)
- RetinaNet
- Focal loss for class imbalance
- DETR (Detection Transformer)
- Set prediction, Hungarian matching algorithm
- EfficientDet
- Compound scaling, BiFPN architecture
🔷 Image Classification & Backbones
Hierarchical feature extraction, skip connections, batch normalization
Convolutional Networks
- LeNet / AlexNet historical
- Convolution: (f*g)[n] = Σf[m]g[n-m]
- VGG-16 / VGG-19
- Small 3×3 filters, depth over width
- ResNet (18/34/50/101/152)
- Residual connections: F(x) + x, gradient highway
- ResNeXt / Wide ResNet
- DenseNet
- Dense connections, feature reuse
- EfficientNet v1/v2
- Neural architecture search, compound scaling
- RegNet
Vision Transformers
Patch embeddings, positional encoding, self-attention for vision
- Vision Transformer (ViT)
- Image patches as tokens, position embeddings
- DeiT (Data-efficient ViT)
- Distillation token, teacher-student training
- Swin Transformer
- Shifted windows, hierarchical feature maps
- ConvNeXt modernized ConvNet
- ConvNet with transformer design principles
- MaxViT
🔷 Image Generation
Diffusion Models
Forward/reverse diffusion processes, denoising score matching
- DDPM / DDIM
- q(x_t|x₀) = N(√α̅_t x₀, (1-α̅_t)I), reverse process
- Stable Diffusion 1.5 / 2.0
- Stable Diffusion XL (SDXL)
- Stable Diffusion 3 latest
- DALL·E 2 / DALL·E 3
- Midjourney v5 / v6
- Imagen (Google)
- Adobe Firefly
- Latent space diffusion, CLIP guidance, classifier-free guidance
GANs & Other Generative
Minimax game theory, adversarial training, Nash equilibrium
- StyleGAN 2/3
- Style injection, AdaIN normalization
- BigGAN
- VQ-VAE / VQ-GAN
- Vector quantization, codebook learning
- Parti (Google)
🔷 Video Understanding
Spatio-temporal convolutions, temporal modeling, motion estimation
- Video Swin Transformer
- TimeSformer
- Divided space-time attention
- SlowFast Networks
- Dual pathway: slow (spatial) + fast (temporal)
- I3D (Inflated 3D ConvNet)
- OpenAI Sora video generation
- Runway Gen-2
- Stable Video Diffusion
🌿 Speech & Audio AI sound processing
Signal processing, spectrograms, mel-frequency cepstral coefficients (MFCC)
🔷 Speech Recognition (ASR)
Hidden Markov models, connectionist temporal classification (CTC)
- Whisper (OpenAI) multilingual
- Sequence-to-sequence with attention
- Wav2Vec 2.0 (Meta)
- Contrastive learning on raw audio waveforms
- DeepSpeech (Mozilla)
- CTC loss, RNN-based architecture
- Conformer
- Convolution + transformer hybrid
- Google Speech-to-Text
- Amazon Transcribe
🔷 Text-to-Speech (TTS)
Vocoder design, mel-spectrogram prediction, neural audio synthesis
- Tacotron 2
- Encoder-decoder with attention, mel-spectrogram prediction
- WaveNet / WaveGlow
- Dilated causal convolutions, autoregressive generation
- VITS (End-to-end TTS)
- Variational inference, normalizing flows
- Bark (Suno AI) expressive
- ElevenLabs commercial
- Tortoise TTS
🔷 Music & Audio Generation
Audio synthesis, harmonic analysis, rhythm modeling
- MusicLM (Google)
- Suno AI commercial
- Udio commercial
- Jukebox (OpenAI)
- VQ-VAE hierarchical modeling
- AudioCraft (Meta)
- MusiCNN
🔷 Audio Processing
- PANNs (Audio Tagging)
- YAMNet (Audio Event Detection)
- AudioMAE
- Masked autoencoding for audio spectrograms
🌿 Multimodal AI text + image + audio + video
Cross-modal attention, contrastive learning, unified embedding spaces
🔷 Vision-Language Models
Cross-attention between visual and textual features
- GPT‑4o / GPT‑4‑vision OpenAI
- Claude 3 Opus/Sonnet (vision)
- Gemini Pro Vision / Ultra
- LLaVA (Large Language and Vision)
- Visual instruction tuning, cross-modal alignment
- BLIP / BLIP-2
- Bootstrap vision-language pre-training
- Flamingo (DeepMind)
- Few-shot learning with frozen language models
- Kosmos‑1/2 (Microsoft)
- Qwen-VL
🔷 CLIP & Embedding Models
Contrastive learning: maximize similarity of correct pairs, minimize incorrect pairs
- CLIP (OpenAI)
- Contrastive loss: L = -log(exp(sim(x,y)/τ) / Σexp(sim(x,y')/τ))
- ALIGN (Google)
- CoCa (Contrastive Captioners)
- SigLIP
- Sigmoid loss instead of softmax for efficiency
🔷 Multimodal Generation
- GPT-4o text→image→text
- Gemini 2.0 comprehensive
- Any-to-Any (Meta) research
- Unified tokenization across modalities
🌿 Robotics & Embodied AI physical world
Control theory, kinematics, dynamics, trajectory optimization
🔷 Manipulation & Control
Inverse kinematics, PID control, Model Predictive Control (MPC)
- RT-1 / RT-2 (Google)
- Transformer for robotics, vision-language-action
- PaLM-E language + robotics
- RoboCat (DeepMind)
- OpenVLA vision-language-action
- SARA (Stanford)
- Spatial action representations
🔷 Simulation & Training
Physics simulation, contact dynamics, domain randomization
- Isaac Gym (NVIDIA)
- MuJoCo
- Multi-joint dynamics with contact
- PyBullet
- Habitat (Meta)
🔷 Navigation & SLAM
Simultaneous Localization and Mapping, particle filters, graph optimization
- Neural SLAM
- PointNet++
- Hierarchical point set feature learning
- DD-PPO (Distributed RL)
🌿 Reinforcement Learning (RL) decision making
Markov Decision Processes, Bellman equations, policy gradients, temporal difference learning
🔷 Game Playing
Monte Carlo Tree Search, value/policy networks, self-play
- AlphaGo / AlphaZero
- MCTS + neural networks, UCB1 selection
- MuZero
- Model-based planning with learned dynamics
- OpenAI Five (Dota 2)
- AlphaStar (StarCraft II)
- Agent57 (Atari)
- Meta-controller for exploration vs exploitation
🔷 RLHF & Alignment
Human feedback training for AI alignment and safety.
Preference modeling, reward learning from comparisons, KL regularization
- InstructGPT (OpenAI)
- PPO with KL penalty: J = E[r(x,y)] - β KL(π,π_ref)
- Constitutional AI (Anthropic)
- Sparrow (DeepMind)
- LaMDA Safety Training
🔷 Policy Learning
Policy gradient methods, actor-critic algorithms, experience replay
- PPO (Proximal Policy Optimization)
- Clipped surrogate objective, trust region
- SAC (Soft Actor-Critic)
- Maximum entropy RL, entropy regularization
- Rainbow DQN
- Combines 6 DQN improvements
- IMPALA
🌿 Recommendation Systems personalization
Collaborative filtering, matrix factorization, embedding learning
🔷 Deep Recommenders
Neural collaborative filtering, factorization machines, deep CTR prediction
- Neural Collaborative Filtering
- Replace inner product with neural networks
- Wide & Deep (Google)
- Memorization (wide) + generalization (deep)
- DeepFM
- Factorization machines + deep neural networks
- DIN / DIEN (Alibaba)
- Attention-based user interest modeling
- PinSage (Pinterest)
- Graph convolutional networks for recommendations
🔷 Sequential Recommendations
Sequence modeling, temporal dynamics, session-based recommendations
- SASRec (Self-Attention)
- Self-attention for sequential patterns
- BERT4Rec
- Bidirectional encoder for user sequences
- GRU4Rec
- RNN for session-based recommendations
🌿 AI for Science & Drug Discovery scientific AI
Molecular dynamics, protein folding energy landscapes, sequence-structure relationships
🔷 Protein Folding & Structure
Energy minimization, Ramachandran plots, contact prediction, attention over residue pairs
- AlphaFold 2/3 (DeepMind) breakthrough
- MSA processing, attention over residue pairs, distance/angle prediction
- ESMFold (Meta)
- Language model embeddings for folding
- ChimeraX AlphaFold
- ColabFold
🔷 Drug Discovery
Molecular representations, SMILES encoding, molecular property prediction
- AlphaFold DB
- MolGAN
- GANs for molecular generation
- Junction Tree VAE
- Tree-structured molecular representations
- ChemBERTa
- MegaMolBART
🔷 Materials Science
- Crystal Graph Neural Networks
- Crystal structure as graphs, material property prediction
- SchNet
- Continuous-filter convolutional networks
- Materials Project ML
🌿 Autonomous Systems self-driving
Sensor fusion, localization, path planning, control theory
🔷 Self-Driving Cars
SLAM, Kalman filtering, trajectory optimization, behavioral planning
- Tesla FSD (Full Self-Driving)
- Neural network end-to-end driving
- Waymo Driver
- Cruise AV
- Mobileye EyeQ
- NVIDIA DRIVE
🔷 Perception Systems
3D object detection, bird's eye view representations, multi-sensor fusion
- BEVFormer (Bird's Eye View)
- Spatial cross-attention, temporal self-attention
- DETR3D
- PointPillars
- Point cloud to pseudo-image conversion
- CenterPoint
🔷 Drones & UAVs
- DJI AI Flight Control
- AirSim (Microsoft)
- FlightGoggles
- Real-time trajectory optimization, collision avoidance
🌿 Time Series & Forecasting temporal data
Autoregressive models, state space models, spectral analysis, trend decomposition
🔷 Deep Time Series Models
Recurrent architectures, temporal convolutions, attention over time
- LSTM / GRU
- Gating mechanisms, forget gates, cell state updates
- Temporal Convolutional Networks
- Dilated convolutions, causal convolutions
- Transformer for Time Series
- Positional encoding for temporal patterns
- N-BEATS
- Neural basis expansion analysis, interpretable forecasting
- DeepAR (Amazon)
- Autoregressive RNN with probabilistic forecasting
🔷 Foundation Models for Forecasting
Pre-trained models on diverse time series, zero-shot forecasting
- TimeGPT Nixtla
- Transformer pre-trained on 100B+ time series points
- Chronos (Amazon) zero-shot
- Tokenized time series, language model scaling laws
- ForecastPFN
- Prior-data fitted networks, in-context learning
🌿 Graph Neural Networks structured data
Message passing, graph convolutions, spectral graph theory, adjacency matrices
🔷 Core GNN Architectures
Neighborhood aggregation, permutation invariance, graph isomorphism
- Graph Convolutional Networks (GCN)
- H^(l+1) = σ(D^(-1/2)AD^(-1/2)H^(l)W^(l)), spectral approach
- GraphSAGE
- Inductive learning, sampling and aggregating
- Graph Attention Networks (GAT)
- Self-attention over graph neighborhoods
- Graph Transformer
- Global attention with positional encodings
- Message Passing Neural Networks
- m_ij = M(h_i, h_j, e_ij), h_i' = U(h_i, Σ m_ij)
🔷 Applications
- Social Network Analysis
- Knowledge Graph Reasoning
- Molecular Property Prediction
- Traffic Flow Prediction
- Spatio-temporal graph modeling
🌿 AI Agents & Planning autonomous agents
Planning algorithms, state space search, multi-agent coordination, game theory
🔷 LLM-based Agents
Reasoning chains, tool usage, action space modeling
- AutoGPT autonomous
- LangChain Agents
- ReAct (Reasoning + Acting)
- Thought-action-observation loops
- Toolformer
- Self-supervised tool use learning
- WebGPT
- Code Interpreter (OpenAI)
🔷 Multi-Agent Systems
Cooperative game theory, Nash equilibrium, mechanism design
- AutoGen (Microsoft) multi-agent
- CrewAI
- MetaGPT
- ChatDev
- Role assignment, communication protocols
🔷 AI Workflow & Automation Platforms
No-code/low-code platforms for building AI-powered workflows and automations.
Directed acyclic graphs (DAGs), workflow orchestration, event-driven systems
AI-First Platforms
- Dify LLM app builder
- Flowise drag-and-drop LLM
- LangFlow visual LangChain
- Botpress conversational AI
- Rasa open source chatbot
- Voiceflow voice/chat apps
General Automation + AI
- Make (formerly Integromat) visual automation
- n8n open source workflow
- Zapier app integration
- Microsoft Power Automate
- IFTTT simple triggers
- Activepieces open source
- Windmill developer-first
Agent Orchestration
- MCP (Model Context Protocol) Anthropic standard
- OpenAI Assistants API
- LangGraph stateful agents
- State machines, graph-based agent workflows
- Semantic Kernel (Microsoft)
- Haystack deepset
- Crew AI Studio
Enterprise AI Orchestration
- Databricks MLflow
- Kubeflow Pipelines
- Azure ML Pipelines
- AWS Step Functions
- Google Cloud Workflows
- Prefect data workflows
- Airflow Apache
- DAG scheduling, dependency management
🔷 Planning & Reasoning
Search algorithms, constraint satisfaction, logical inference
- Tree of Thoughts
- Breadth-first search over reasoning trees
- Chain-of-Thought Prompting
- Step-by-step reasoning decomposition
- Self-Consistency Decoding
- Sample multiple reasoning paths, majority vote
- Program-aided Language Models
- Code generation for numerical reasoning
🌿 Generative AI Architectures creation
Probabilistic modeling, likelihood maximization, latent variable models
🔷 Transformer Architectures
Self-attention: Attention(Q,K,V) = softmax(QK^T/√d_k)V, positional encoding
- Vanilla Transformer
- Multi-head attention, feed-forward networks
- GPT (Decoder-only)
- Causal masking, autoregressive generation
- BERT (Encoder-only)
- Bidirectional attention, masked language modeling
- T5 (Encoder-Decoder)
- Text-to-text unified framework
- PaLM (Pathways)
- Switch Transformer
- Mixture of experts routing
- Mamba state space
- Selective state spaces, linear attention alternative
🔷 Diffusion Models
Forward process: q(x_t|x_{t-1}) = N(√(1-β_t)x_{t-1}, β_t I), reverse denoising
- DDPM (Denoising Diffusion)
- Markov chain diffusion, score matching
- Stable Diffusion
- DALL·E 2/3
- Imagen / Parti
- ControlNet controllable generation
- Conditional generation with spatial controls
🔷 Variational Models
Evidence Lower BOund (ELBO), KL divergence regularization
- VAE (Variational Autoencoder)
- ELBO = E[log p(x|z)] - KL(q(z|x)||p(z))
- β-VAE
- β-weighted KL term for disentanglement
- VQ-VAE / VQ-VAE-2
- Vector quantization, discrete latent space
🌿 AI Safety & Alignment responsible AI
Robustness optimization, distributional shift, adversarial examples, reward modeling
🔷 Alignment Techniques
Human preference modeling, reward learning, value alignment
- RLHF (Reinforcement Learning from Human Feedback)
- Bradley-Terry preference model, reward model training
- Constitutional AI (Anthropic)
- Self-supervised harmfulness reduction
- AI Safety via Debate
- Two-player zero-sum game for truthfulness
- Iterated Amplification
- Red Teaming Methods
- Adversarial testing, failure mode discovery
🔷 Interpretability & Explainability
Attribution methods, gradient-based explanations, feature importance
- LIME / SHAP
- Local linear approximations, Shapley values
- Attention Visualization
- Concept Activation Vectors
- TCAV: directional derivatives in concept space
- Mechanistic Interpretability
- Anthropic's Circuit Analysis
- Reverse engineering neural network computations
🔷 Robustness & Security
Adversarial perturbations, certified defenses, distributional robustness
- Adversarial Training
- Minimax optimization, PGD attacks
- Certified Defenses
- Lipschitz constraints, interval bound propagation
- Differential Privacy
- ε-differential privacy, noise injection
- Federated Learning
- Distributed learning, privacy preservation
🌿 Edge AI & Model Optimization efficiency
Model compression, quantization, pruning, efficient architectures
🔷 Model Compression
Singular value decomposition, weight magnitude pruning, entropy-based quantization
- Knowledge Distillation
- Student mimics teacher: L = αL_CE + (1-α)L_KD
- Pruning (Magnitude/Structured)
- Weight magnitude thresholding, lottery ticket hypothesis
- Quantization (INT8/INT4)
- Uniform/non-uniform quantization, calibration datasets
- Low-Rank Factorization
- Matrix decomposition: W ≈ UV^T
- LoRA (Low-Rank Adaptation)
- W = W₀ + BA, where B∈R^(d×r), A∈R^(r×k)
🔷 Mobile & Edge Models
Depthwise separable convolutions, neural architecture search
- MobileNet v1/v2/v3
- Depthwise separable convolutions, inverted residuals
- EfficientNet
- Compound scaling: depth/width/resolution
- TinyBERT
- DistilBERT
- TensorFlow Lite
- ONNX Runtime
- Apple Core ML
🌿 Domain-Specific AI specialized applications
Domain adaptation, transfer learning, specialized loss functions
🔷 Healthcare & Medical AI
Medical image analysis, diagnostic classification, survival analysis
- Med-PaLM (Google)
- GPT-4 Medical
- CheXNet (Chest X-ray)
- DenseNet for radiological diagnosis
- DeepMind AlphaFold
- IBM Watson Health
- Radiology AI (Various)
🔷 Finance & Trading
Time series forecasting, risk modeling, portfolio optimization
- BloombergGPT
- FinBERT
- Algorithmic Trading AI
- Market microstructure modeling, order flow prediction
- Risk Management Models
- Value at Risk (VaR), Monte Carlo simulations
- Fraud Detection Systems
- Anomaly detection, graph-based fraud networks
🔷 Legal AI
- Legal BERT variants
- Contract Analysis AI
- Case Law Search
- Patent Analysis
- Legal document similarity, citation networks
🔷 Education & Learning
Adaptive learning, knowledge tracing, educational data mining
- Khan Academy AI Tutor
- Duolingo AI
- Spaced repetition algorithms, forgetting curves
- Adaptive Learning Systems
- Bayesian knowledge tracing, item response theory
- Academic Writing Assistants
🌿 Quantum AI & Computing quantum advantage
Quantum superposition, entanglement, quantum circuit learning, variational quantum algorithms
🔷 Quantum Machine Learning (QML)
Quantum feature maps, parameterized quantum circuits, quantum kernels
- Variational Quantum Circuits (VQC)
- U(θ) = ∏U_i(θ_i), parameterized quantum gates
- Quantum Neural Networks
- Quantum perceptrons, quantum backpropagation
- Quantum Support Vector Machines
- Quantum feature maps, exponential kernel spaces
- Quantum Generative Models
- Born machine, quantum GANs
- IBM Qiskit Machine Learning
- Google Cirq TensorFlow Quantum
- PennyLane (Xanadu)
🔷 Quantum Algorithms
Grover's algorithm, quantum Fourier transform, amplitude amplification
- Variational Quantum Eigensolver (VQE)
- ⟨ψ(θ)|H|ψ(θ)⟩ minimization, NISQ algorithms
- Quantum Approximate Optimization (QAOA)
- Alternating ansatz: e^(-iγH_C)e^(-iβH_M)
- Quantum Principal Component Analysis
- Quantum matrix exponentiation, HHL algorithm
- D-Wave Quantum Annealing
- Ising model optimization, quantum tunneling
🔷 Hybrid Quantum-Classical
Variational hybrid algorithms, classical optimization of quantum circuits
- Quantum-Classical Neural Networks
- Quantum Transfer Learning
- Pre-trained quantum feature extractors
- Quantum Reinforcement Learning
- Quantum policy gradients, quantum Q-learning
- Quantum Federated Learning
🌿 Federated & Distributed Learning privacy-preserving
Distributed optimization, privacy preservation, secure aggregation, communication efficiency
🔷 Federated Learning Algorithms
Federated averaging, non-IID data challenges, client sampling strategies
- FedAvg (Federated Averaging)
- w^{t+1} = w^t - η∇F(w^t), F = Σp_kF_k(w)
- FedProx (Federated Proximal)
- Proximal term: μ/2||w-w^t||², heterogeneity handling
- FedNova (Federated Nova)
- Normalized averaging, varying local steps
- SCAFFOLD
- Control variates for client drift correction
- FedOpt (Adam/AdaGrad variants)
- Server-side adaptive optimization
🔷 Split & Vertical Learning
Model splitting, gradient compression, vertical data partitioning
- Split Learning
- Forward pass splitting, smashed data transmission
- Vertical Federated Learning
- Feature space partitioning, secure computation
- SplitFed (Split + Federated)
- Gradient Compression
- Sparsification, quantization, error feedback
🔷 Privacy & Security
Differential privacy, secure aggregation, homomorphic encryption
- Differential Privacy in FL
- DP-SGD: noise σ = C√(2ln(1.25/δ))/ε
- Secure Multiparty Computation
- Secret sharing, garbled circuits
- Byzantine-Robust Aggregation
- Krum, trimmed mean, geometric median
- Homomorphic Encryption
- Computing on encrypted data, CKKS scheme
🌿 AI Infrastructure & MLOps production systems
Continuous integration/deployment, model monitoring, A/B testing, drift detection
🔷 Model Deployment & Serving
Load balancing, auto-scaling, latency optimization, resource allocation
- Hugging Face Inference Endpoints
- Replicate Cloud Platform
- AWS SageMaker Endpoints
- Google Vertex AI
- Azure ML Endpoints
- Seldon Core
- KServe (Kubernetes)
- TensorFlow Serving
- TorchServe
- ONNX Runtime Server
- Request batching, model caching, GPU utilization
🔷 Monitoring & Observability
Statistical drift detection, performance monitoring, alert systems
- Weights & Biases
- MLflow Tracking
- Neptune AI
- ClearML
- Evidently AI drift detection
- KL divergence drift: D_KL(P_ref||P_curr)
- Arize AI
- Fiddler AI
- WhyLabs
- Population stability index, data quality metrics
🔷 Feature Engineering & Storage
Feature pipelines, versioning, real-time serving, consistency
- Feast (Feature Store)
- Tecton
- Hopsworks
- AWS Feature Store
- Databricks Feature Store
- DVC (Data Version Control)
- Great Expectations
- Feature consistency, point-in-time correctness
🔷 A/B Testing & Experimentation
Statistical significance, power analysis, multi-armed bandits
- Optimizely
- LaunchDarkly
- Split.io
- Google Optimize
- Facebook Planout
- Welch's t-test, sequential testing, false discovery rate
- Multi-Armed Bandits
- Thompson sampling, upper confidence bounds
🌿 Retrieval-Augmented Generation (RAG) knowledge integration
Dense retrieval, vector similarity, knowledge fusion, query decomposition
🔷 Dense Retrieval Systems
Bi-encoder architecture, contrastive learning, hard negative mining
- DPR (Dense Passage Retrieval)
- Similarity: sim(q,p) = E_Q(q)^T E_P(p)
- ColBERT
- Late interaction: Σ max_j E_q_i^T E_p_j
- ANCE (Approximate Nearest Neighbor)
- RetroMAE
- Masked autoencoding for retrieval
- E5 / BGE Embeddings
- Instructor Embeddings
- Task-aware dense retrieval
🔷 Advanced RAG Architectures
Multi-hop reasoning, fusion-in-decoder, retrieval refinement
- FiD (Fusion-in-Decoder)
- Independent encoding, joint decoding
- REALM (Retrieval-Enhanced LM)
- End-to-end retrieval learning
- RAG-Token / RAG-Sequence
- Self-RAG reflection
- Retrieval necessity prediction, self-critique
- Modular RAG
- Graph RAG
- Knowledge graph enhanced retrieval
🔷 Vector Databases & Search
Approximate nearest neighbor search, indexing algorithms, sharding
- Pinecone
- Weaviate
- Qdrant
- Chroma
- Milvus / Zilliz
- FAISS (Meta)
- HNSW: Hierarchical Navigable Small World
- Annoy (Spotify)
- ScaNN (Google)
- Product quantization, inverted file index
🌿 Causal AI & World Models reasoning & simulation
Causal inference, structural equations, counterfactual reasoning, physics simulation
🔷 Causal Inference & Discovery
Directed acyclic graphs, do-calculus, structural causal models
- PC Algorithm
- Constraint-based causal discovery
- GES (Greedy Equivalence Search)
- CausalML (Uber)
- DoWhy (Microsoft)
- do(X=x): P(Y|do(X=x)) causal intervention
- Neural Causal Models
- Structural equation models with neural networks
- CausalNex
- pgmpy (Probabilistic Graphical Models)
🔷 World Models & Simulation
Model-based reinforcement learning, physics simulation, predictive modeling
- DreamerV3 (DeepMind)
- Latent world model: p(s_{t+1}|s_t,a_t)
- MuJoCo (Multi-Joint dynamics)
- Contact dynamics, constraint solving
- Isaac Gym (NVIDIA)
- PyBullet
- Habitat (Meta)
- AI Habitat 2.0
- ThreeDWorld (MIT)
- Physics-informed neural networks (PINNs)
🔷 Digital Twins & Simulation
Real-time synchronization, digital-physical coupling, predictive maintenance
- NVIDIA Omniverse
- Unity ML-Agents
- Unreal Engine AI
- AnyLogic Simulation
- CARLA (Autonomous Driving)
- Sensor simulation, traffic modeling
- Gazebo (Robotics)
- CoppeliaSim
🌿 Creative AI artistic generation
Style transfer, generative adversarial training, artistic style optimization
🔷 Art & Visual Generation
Neural style transfer, GANs for art, aesthetic loss functions
- DALL·E 2/3 text-to-image
- Midjourney v6
- Stable Diffusion Art
- Adobe Firefly
- RunwayML Gen-2
- StyleGAN for Art
- Style loss: L_style = Σ||G(F) - G(S)||²
- Neural Style Transfer
- Gram matrices, content + style loss
- ArtBreeder
- NightCafe Studio
🔷 Music & Audio Generation
Harmonic analysis, rhythm modeling, spectral synthesis
- Suno AI commercial music
- Udio AI music
- MusicLM (Google)
- Jukebox (OpenAI)
- VQ-VAE hierarchical audio modeling
- AIVA (AI Composer)
- Amper Music
- Soundraw
- MuseNet (OpenAI)
- Transformer for symbolic music
🔷 Creative Writing & Content
Language modeling for creativity, style control, narrative generation
- GPT-4 Creative Writing
- Claude Creative Assistant
- Jasper AI
- Copy.ai
- Writesonic
- NovelAI
- Sudowrite
- Controllable generation, style conditioning
🔷 Interactive & Procedural AI
Real-time generation, user interaction modeling, procedural algorithms
- AI Dungeon
- Character.AI
- Replika
- Procedural Content Generation
- L-systems, cellular automata, noise functions
- Unity ML-Agents Creative
- Unreal Engine MetaHuman
🌿 Emerging AI Research cutting-edge
Scaling laws: L(N) = aN^(-α), compute-optimal training, emergent capabilities
🔷 Foundation Model Research
Scaling laws: L(N) = aN^(-α), compute-optimal training
- GPT-5 rumored
- Claude 4+ future
- Gemini Advanced
- Meta LLaMA 4 development
- Mixture of Experts (MoE) scaling
- Sparse expert routing, load balancing
🔷 Novel Architectures
Alternative attention mechanisms, efficient transformers
- Mamba (State Space Models) efficiency
- Selective state spaces, linear complexity
- RetNet (Alternative to Transformers)
- Retention mechanism, parallel/recurrent duality
- Hyena (Subquadratic Attention)
- Convolutional operators, implicit attention
- Kolmogorov-Arnold Networks
- KAN: learnable activation functions on edges
- Liquid Neural Networks
- Time-aware neurons, causal interactions
🔷 AGI Research
Multi-task learning, meta-learning, unified architectures
- OpenAI Q* rumored reasoning
- DeepMind Gato generalist agent
- Multi-modal tokenization, unified architecture
- AdA (Adaptive Agent)
- Artificial General Intelligence projects
🔷 Neuromorphic Computing
Spiking neural networks, event-driven computation, brain-inspired architectures
- Intel Loihi 2
- Spiking neural networks, asynchronous computation
- IBM TrueNorth
- SpiNNaker (Manchester)
- BrainChip Akida
- Integrate-and-fire neurons, spike-timing dependent plasticity
- Memristor Networks
- Resistive memory, in-memory computation