| Abstract | Bangla Sign Language (BdSL) recognition presents multifaceted challenges due to signer diversity and spatiotemporal variability. While flat classification pipelines are widely used, they often overlook the underlying phonological relationships among signs that can inform more structured and accurate recognition. To address this gap, we propose a hierarchical recognition framework that integrates phonological clustering into a Bidirectional Long Short-Term Memory (Bi-LSTM)-based sequence modeling pipeline. First, baseline classification—referring to a flat, non-clustered recognition approach—is performed us ing five Bi-LSTM configurations of increasing complexity to assess the trade-off between accuracy and model size. The resulting confusion matrices are analyzed to identify sign pairs with high misclassification rates, revealing underlying phonological similarities. Based on this analysis, a confusion-matrix driven clustering strategy is employed to group visually and phonologically similar signs. Cluster-specific feature engineering is then applied, and the same Bi-LSTM architecture is retrained separately within each cluster. Experiments are conducted on a curated 50-class subset of the SignBD-Word dataset. In the baseline setting, the most complex model (Bi-LSTM-1) achieves 91.10% accuracy with 2.6 million parameters. With the proposed confusion-matrix driven clustering architecture, four out of six clusters employing the lightweight Bi-LSTM classifiers outperform the baseline model, reaching up to 94.58% accuracy. Remarkably, the lightweight Bi-LSTM-4 model—with only 373K parameters (14.4% of Bi LSTM-1)—achieves a weighted average accuracy of 92.60% on cluster-level classification, surpassing the baseline by1.5percentagepoints. Thisreflectsan85.6%reduction in a number of parameters, demonstrating that efficient cluster-specific feature engineering, which allows each model to capture nuanced patterns within each cluster can improve overall predictive accuracy. |