Log In    |    Knowledge Base    |    4D Home
Tech Tip: String Tokenizing: Storing a String's "Tokens" in an Array
Published On: July 29, 2004

String tokenization is a process by which the "tokens" of a string are split up and stored in a container such as an array between some delimiter.

Take for example the string "Foo Bar"

In this string, if the delimiter was a space character, the tokens would then be "Foo" and "Bar." Similarly, if the delimiter was the capital B character, the tokens would then be "Foo " and "ar."

Tokenizing strings is a useful technique that will help you to read or parse through strings of text and to separate useful tokens. For example, this technique can be used to separate name/value pairs of a parameter list in a URL. name1=Value1&name2=Value2. Setting the delimiter to the ampersand character can separate the pairs into "name1=Value1" and "name2=Value2."

The following method called StringTokenize will do just that:


  ` initialize our local variables



If(Size of array($tokenList->)>0)
  DELETE ELEMENT($tokenList->;1;Size of array($tokenList->))
End if

While (($length>0))  ` for debug
  ` initialize our scope variables

  ` expand the array as necessary
  INSERT ELEMENT($tokenList->;$elemIndex)

  ` parse the text input one character at a time until either delim
  ` is reached and we have successfuly created a token or length has been reached

   If ($char#$delim)
End if

  Until ((($char=$delim) & ($token#"")) | ($curPos>$length))

    ` reset the string buffer

  ` add the token to the list if not null
  If ($token#"")
  End if

End while

  ` return true on condition: added elements
$0:=Size of array($tokenList->)

If you run the following line of code after adding the above method...

ARRAY TEXT(myarray;0)
StringTokenize("this is my string";char(Space);->myarray)

...myarray will be filled with the items "this" "is" "my" "string" since space is the delimiter.