Evolution of AI-Based Fabric Defect Detection (Part 2): Teacher–Student Distillation Methods

Publication Note

This article is Part 2 of AI‑Innovate’s publication of the review paper ‘Evolution of AI‑Based Fabric Defect Detection: From Classical Vision to Foundation Models’. The manuscript has been divided into multiple articles solely to improve readability on the AI‑Innovate website. Apart from this publication note, article navigation, and editorial transitions, the research content remains unchanged.

3. Teacher–Student Distillation Methods

Feature-based methods demonstrated that it was possible to detect defects without ever observing defective samples during training. By learning a robust representation of normal fabrics, these methods transformed anomaly detection from a classification problem into one of measuring deviations from normality. However, this success introduced a new practical challenge. Many feature-based approaches depended on storing large collections of normal feature embeddings or performing computationally expensive comparisons against statistical models during inference. As industrial datasets continued to grow, these storage and search requirements became increasingly difficult to scale.

Researchers therefore began asking a different question:

Instead of memorizing what normal features look like, could a neural network simply learn how a normal fabric should behave?

This idea marked the beginning of the teacher–student distillation paradigm.

Rather than explicitly modeling the distribution of normal features, teacher–student methods transfer knowledge from a powerful, pre-trained neural network, the teacher, to a smaller trainable student network. The teacher remains completely frozen throughout training and serves as a reliable reference that extracts rich feature representations from every input image. The student’s only objective is to reproduce these representations as accurately as possible, but it is trained exclusively using defect-free images.

The underlying assumption is remarkably intuitive.

Because the student is exposed only to normal samples, it gradually learns how normal fabrics should appear in the teacher’s feature space. During deployment, when a defect is encountered for the first time, the teacher faithfully extracts features describing both the normal texture and the abnormal region. The student, however, has never learned to represent such unexpected structures. Consequently, it fails to reproduce the teacher’s response at defective locations. The discrepancy between the teacher and student feature representations naturally becomes an anomaly signal, enabling both defect detection and localization without requiring any defective training examples.

The earliest implementations of this concept focused primarily on feature consistency across multiple network layers.

Rather than comparing only the deepest semantic representations, these methods encouraged the student to imitate the teacher simultaneously at several feature hierarchies. Early layers preserved fine spatial structures such as edges and textures, while deeper layers captured increasingly abstract semantic information. By minimizing discrepancies across multiple resolutions, the student learned a comprehensive representation of normal fabrics that combined detailed texture information with broader structural understanding.

This multi-scale supervision significantly improved localization performance compared with earlier feature-based methods. Because anomaly scores were computed directly from feature differences instead of nearest-neighbor searches or probabilistic distances, inference became both simpler and more memory-efficient. The student network effectively acted as a compact approximation of the entire normal feature distribution.

Despite these advantages, practical deployment soon revealed another limitation.

Although the student network was considerably smaller than the teacher, the teacher itself often remained a large convolutional backbone originally designed for image classification. Running both networks simultaneously during inference increased computational cost and limited deployment on embedded hardware, edge devices, and high-speed manufacturing systems where strict latency constraints are common.

This challenge motivated the next stage in the evolution of teacher–student learning.

Rather than treating computational efficiency as a secondary objective, researchers began designing architectures in which efficiency became an integral part of the learning process. Lightweight backbones, simplified feature extractors, and compact students were carefully optimized to preserve anomaly detection accuracy while dramatically reducing inference time and memory consumption.

These architectural refinements demonstrated that high detection performance did not necessarily require increasingly large neural networks. By carefully selecting informative feature hierarchies and reducing redundant computations, efficient teacher–student frameworks achieved real-time inspection speeds while maintaining competitive localization accuracy. This represented an important milestone for industrial deployment, where processing thousands of images per minute is often more valuable than marginal improvements in benchmark accuracy.

As models became faster, researchers turned their attention to another persistent weakness.

Although multi-scale feature matching effectively highlighted many structural defects, every spatial location within an image continued to receive approximately equal importance during training and inference. In practice, however, industrial images are highly unbalanced. Most image regions contain perfectly normal texture, while defects occupy only a very small fraction of the scene. Treating every patch equally therefore forces the model to spend much of its capacity learning redundant background information instead of concentrating on the subtle irregularities that truly matter.

Recent developments addressed this issue by integrating self-supervised transformer representations into the teacher–student framework.

Vision Transformers pretrained through self-supervised learning provide significantly richer feature representations than conventional convolutional backbones, particularly for highly textured materials. More importantly, their attention mechanisms naturally capture long-range contextual relationships while preserving local structural details. Instead of relying solely on convolutional receptive fields, the teacher can now identify which image regions carry the most informative signals for distinguishing normal patterns from anomalies.

The student, in turn, is trained to reproduce these enhanced representations while implicitly inheriting the teacher’s attention to discriminative regions. As a result, anomaly detection becomes increasingly focused on meaningful structural variations rather than redundant background textures. This combination of transformer-based feature extraction and knowledge distillation substantially improves the detection of subtle textile defects such as broken yarns, weak stains, and fine weaving irregularities that might otherwise remain hidden within repetitive patterns.

Looking back, the evolution of teacher–student methods reflects a gradual refinement of a single central idea.

The first generation demonstrated that normal behavior could be learned instead of stored, eliminating the need for large memory banks. The second generation focused on making this learning process computationally efficient, enabling real-time deployment on industrial hardware. The most recent generation further strengthened the paradigm by incorporating self-supervised transformer representations and attention mechanisms, allowing the student to imitate not only the teacher’s features but also its ability to focus on the most informative regions of an image.

Despite these advances, one important limitation remained unresolved. As teacher–student models became increasingly capable, the student occasionally learned to reproduce abnormal patterns almost as successfully as normal ones. In other words, the very success of knowledge distillation sometimes weakened the anomaly signal itself. This observation led researchers to reconsider the architecture of the distillation process, ultimately giving rise to Reverse Distillation, where the objective was no longer simply to imitate the teacher, but to constrain how information flows through the student so that anomalous patterns could no longer be reconstructed effectively.