Abstract:
The article proposes a fuzzy regulator for calculating sliding window parameters to prepare training data for Large Language Models. The traditional approach sets the stride and context length parameters as fixed constants, uniform for the entire text, and does not account for the linguistic characteristics of individual fragments, such as dense scientific text and monotonous, repetitive text. The proposed method utilizes two automatically computed features of a fragment: lexical diversity and the average BPE token length. Based on the Mamdani algorithm with a base of 9 fuzzy logic rules and defuzzification using the center of gravity method, the fuzzy regulator adaptively calculates the stride and context length values for each fragment. The proposed approach has a cognitive interpretation, as it mimics the mechanism of adaptive human attention during reading, for example, complex fragments are processed more attentively with a small stride size.