Large language models (LLMs) demonstrate exceptional accuracy in password-guessing tasks, but their massive memory cost, high computational demands, and high duplicate rates severely limit deployment in resource-constrained scenarios. This paper introduces the 1-bit/1.58-bit quantized BitNet architecture to the password guessing domain for the first time, proposing the lightweight PassBitNet framework. PassBitNet adopts a pattern-guided autoregressive generation paradigm with three plug-and-play deduplication modules (SPP, SW-Bloom, and Shard-Dedup). Experiments demonstrate that PassBitNet-16H12-B achieves a 31.78% hit rate under \(10^{9}\) guesses with only 3.40MB–more than double the hit rate of PassGAN while maintaining a performance within 10 percentage points below PassGPT, with merely 1/100th the memory of PassGPT. To address redundancy in autoregressive generation, three strategies were designed: Stepwise Probability Penalty (SPP), Sliding Window-Bloom Filter (SW-Bloom), and Shard-Dedup. These enable source-level duplication suppression and streaming filtering. The deduplication algorithms reduce the duplication rate to 2.01% (SPP) and 6.12% (SW-Bloom). The latter achieves a hit rate of 41.92% with only a 4.6% increase in time overhead.
Large language model training commonly relies on multidimensional parallelism, including data, tensor, pipeline, and context parallelism, to support long-context and large-scale workloads. However, real pretraining corpora consist of highly heterogeneous variable-length samples, which create a complex coupling between the internal structure of packed documents and their actual execution cost. Existing approaches typically optimize upstream packing and downstream context parallelism separately, while paying limited attention to their coupled impact on execution block completion time, local load balance, and communication overhead. To address this issue, we propose ATAC, an anchor-tail aware framework for jointly optimizing packed-document construction and context-parallel sharding in large language model training. ATAC consists of two complementary components: WFAP, which constructs execution-friendly packed documents by jointly considering workload structure and sample fragmentation during packing, and ATP-CP, which performs hierarchical sharding by exploiting the anchor-tail structure of packed documents so as to mitigate intra-block stragglers while controlling additional key-value transfer overhead. In an A800-calibrated pipeline-level evaluation, ATAC improves normalized throughput over the strongest implemented baseline, achieving an overall geometric mean speedup of 1.93\(\times \). Further analysis shows that WFAP substantially reduces runtime dispersion across packed documents, while ATP-CP provides a more effective balance between runtime equalization and communication overhead. Overall, this work demonstrates that input-structure-aware co-design of packing and context parallelism is an effective approach to improving calibrated pipeline-level execution efficiency in large language model training.