String tokenization is a process by which the "tokens" of a string are split up and stored in a container such as an array between some delimiter.

Take for example the string "Foo Bar"

In this string, if the delimiter was a space character, the tokens would then be "Foo" and "Bar." Similarly, if the delimiter was the capital B character, the tokens would then be "Foo " and "ar."

Tokenizing strings is a useful technique that will help you to read or parse through strings of text and to separate useful tokens. For example, this technique can be used to separate name/value pairs of a parameter list in a URL. name1=Value1&name2=Value2. Setting the delimiter to the ampersand character can separate the pairs into "name1=Value1" and "name2=Value2."

The following method called StringTokenize will do just that:

C_TEXT($1;$2;$stringBuffer;$delim)
C_TEXT($token;$char)
C_LONGINT($0;$curPos;$length;$elemIndex)
C_POINTER($3;$tokenList)

  ` initialize our local variables

$stringBuffer:=$1
$delim:=$2
$tokenList:=$3

$length:=Length($stringBuffer)
$elemIndex:=0

If(Size of array($tokenList->)>0)
  DELETE ELEMENT($tokenList->;1;Size of array($tokenList->))
End if

While (($length>0))  ` for debug
  ` initialize our scope variables
  $token:=""
  $curPos:=1
  $elemIndex:=$elemIndex+1

  ` expand the array as necessary
  INSERT ELEMENT($tokenList->;$elemIndex)

  ` parse the text input one character at a time until either delim
  ` is reached and we have successfuly created a token or length has been reached
  Repeat
   $char:=Substring($stringBuffer;$curPos;1)

   If ($char#$delim)
     $token:=$token+$char
End if

    $curPos:=$curPos+1
  Until ((($char=$delim) & ($token#"")) | ($curPos>$length))

    ` reset the string buffer
  $stringBuffer:=Substring($stringBuffer;$curPos)
  $length:=Length($stringBuffer)

  ` add the token to the list if not null
  If ($token#"")
    $tokenList->{$elemIndex}:=$token
  End if

End while

  ` return true on condition: added elements
$0:=Size of array($tokenList->)

If you run the following line of code after adding the above method...

ARRAY TEXT(myarray;0)
StringTokenize("this is my string";char(Space);->myarray)

...myarray will be filled with the items "this" "is" "my" "string" since space is the delimiter.