Evolution of AI-Based Fabric Defect Detection (Part 3): Reverse Distillation Methods

Publication Note

This article is Part 3 of AI‑Innovare’s publication of the review paper ‘Evolution of AI‑Based Fabric Defect Detection: From Classical Vision to Foundation Models’. The manuscript has been divided into multiple articles solely to improve readability on the AI‑Innovate website. Apart from this publication note, article navigation, and editorial transitions, the research content remains unchanged.

4. Reverse Distillation Methods

Conventional teacher–student distillation represented a major step forward in industrial anomaly detection. By learning to imitate the behavior of a powerful teacher network using only defect-free samples, the student could identify anomalies through discrepancies in feature representations rather than explicit defect classification. This approach eliminated the need for large memory banks and provided an efficient framework for one-class learning.

However, as researchers continued improving these models, an unexpected problem began to emerge.

Ironically, the student was becoming too good at its job.

The original assumption behind knowledge distillation was that the student, having never observed defective samples, would naturally fail to reproduce the teacher’s response whenever an anomaly appeared. In practice, however, increasingly powerful student networks often learned feature representations that generalized beyond the normal data used during training. Instead of failing on abnormal regions, they gradually became capable of reconstructing many anomalous patterns almost as accurately as normal ones.

This phenomenon significantly weakened the anomaly signal.

Since anomaly detection depended entirely on the difference between the teacher’s and student’s feature representations, reducing this discrepancy inevitably made defects more difficult to detect. Small scratches, broken yarns, or weak structural irregularities sometimes produced only negligible reconstruction errors, increasing the risk of missed detections. Rather than lacking sufficient learning capacity, the student now possessed too much representational power.

This realization fundamentally changed the objective of knowledge distillation.

Instead of asking “How can the student imitate the teacher more accurately?”, researchers began asking a very different question:

“How can we deliberately prevent the student from reconstructing abnormal patterns?”

This shift in philosophy gave rise to Reverse Distillation.

Unlike conventional distillation, where the teacher and student operate as parallel feature extractors, reverse distillation reorganizes the entire architecture into an encoder–decoder framework. The teacher acts as a frozen encoder that extracts rich multi-scale representations from the input image, while the student becomes a decoder responsible for reconstructing those representations. Between these two components lies a carefully designed bottleneck that compresses the teacher’s information before it reaches the student.

This bottleneck is the defining characteristic of reverse distillation.

Rather than transmitting every feature extracted by the teacher, the bottleneck filters the information, forcing the network to preserve only the characteristics that consistently describe normal samples. Because it is trained exclusively on defect-free data, the compressed representation gradually becomes a compact model of normality while discarding information that does not conform to the learned distribution.

During inference, this behavior produces a natural asymmetry.

The teacher continues to encode every detail of the input image, including anomalous structures. However, the bottleneck has never learned how to represent these unexpected patterns. Consequently, the student decoder receives an incomplete representation and is unable to faithfully reconstruct the teacher’s features in defective regions. The resulting discrepancy becomes significantly larger than in conventional distillation, strengthening the anomaly signal and improving localization accuracy.

This architectural redesign solved one of the most important weaknesses of earlier teacher–student methods. Instead of relying on the student to fail naturally, the bottleneck explicitly limits what the student is capable of reconstructing. In other words, the inability to reproduce anomalies is no longer an accidental property of the model, it becomes an intentional design objective.

Although reverse distillation substantially improved anomaly discrimination, practical applications soon revealed another limitation.

Compressing all information into a deep semantic bottleneck inevitably sacrifices part of the fine-grained structural information contained in shallow feature layers. For many industrial inspection tasks this trade-off is acceptable. Textile inspection, however, presents a unique challenge. The most important defects are often extremely subtle, consisting of broken fibers, slight yarn misalignments, tiny holes, or weak weaving irregularities that differ from normal fabric by only a few pixels. These defects are primarily characterized by high-frequency texture information rather than high-level semantic content.

As a result, researchers recognized that a purely semantic bottleneck was insufficient for textile inspection.

The next stage in the evolution therefore focused on preserving shallow texture information without sacrificing the anomaly discrimination introduced by reverse distillation.

This objective led to the development of residual reverse distillation architectures.

Instead of allowing information to pass exclusively through the bottleneck, residual connections were introduced between corresponding layers of the teacher and student networks. These shortcut pathways preserve high-resolution texture features while maintaining the restrictive bottleneck responsible for suppressing anomalous representations. Consequently, the student gains access to the detailed structural characteristics of normal fabrics without becoming powerful enough to reconstruct abnormal patterns completely.