YuStemmer

Word stemming is usually applied to query and search systems. It enables them to return related results with similar meaning but slightly different spelling. As an example, the English stemmer returns “write” for “write”, “writes”, “writing”, and “writings”.

Stemmers are available for these languages:

• Arabic, Armenian, Basque, Catalan, Czech, Danish, Dutch, English, Finnish, French, German, German2 (different Umlaut handling), Greek, Hindi, Hungarian, Indonesian, Irish, Italian, Latin, Lithuanian, Lovins (English), Nepali, Norwegian, Porter (English), Portuguese, Romanian, Russian, Serbian, Slovene, Spanish, Swedish, Tamil, Turkish.

YuStemmer is fully algorithmic. No extensive lookup dictionaries are needed. This results in small memory footprint and excellent performance.

YuStemmer was initially developed for the DISQLite3 Full Text Search (FTS) engine which is prepared to use it out of the box. Besides that, YuStemmer fits many other purposes.

YuStemmer is organized into different classes, each of them optimized for a particular string type and text encoding:

• TYuStemmer class:
  • ANSI text, 8-bit, usually in ISO-8859-1, unless otherwise noted.
  • Some stemmers might not be available for this class.
• TYuStemmer_8 class:
  • UTF-8 text, 8-bit.
  • All stemmers are available for this class.
• TYuStemmer_16 class:
  • UTF-16 text, 16-bit. This corresponds to Delphi's WideString / UnicodeString.
  • All stemmers are available for this class.

Make sure to choose the stemmer class matching your string type and character set. Otherwise you will suffer a performance penalty caused by avoidable string conversions. In Delphi, such conversions usually happen implicitly and go unnoticed by most developers. Therefore, pay close attention here to make the most of YuStemmer!

The Stem() method does the work for all of the above classes. It expects a single word and returns its stem. If there is no stem, the original word is returned unchanged.

function StemFrench(const AWord: AnsiString): AnsiString;
var
  Stemmer: TYuStemmer;
begin
  Stemmer := TYuStemmer_French.Create;
  Result := Stemmer.Stem(AWord);
  Stemmer.Free;
end;

To improve performance when stemming a great number of words, it is safe to reuse the same instance of a stemmer class multiple times.

procedure StemItalian(const AWords: TStringList);
var
  i: Integer;
  { TStringList is UTF-16 in Unicode Delphis. }
  Stemmer: {$IFDEF Unicode}TYuStemmer_16{$ELSE}TYuStemmer{$ENDIF};
begin
  {$IFDEF Unicode}
  Stemmer := TYuStemmer_Italian_16.Create;
  {$ELSE Unicode}
  Stemmer := TYuStemmer_Italian.Create;
  {$ENDIF Unicode}
  for i := 0 to AWords.Count - 1 do
    AWords[i] := Stemmer.Stem(AWords[i]);
  Stemmer.Free;
end;