String tokenization is a process by which the "tokens" of a string are split up and stored in a container such as an array between some delimiter.
Take for example the string "Foo Bar"
In this string, if the delimiter was a space character, the tokens would then be "Foo" and "Bar." Similarly, if the delimiter was the capital B character, the tokens would then be "Foo " and "ar."
Tokenizing strings is a useful technique that will help you to read or parse through strings of text and to separate useful tokens. For example, this technique can be used to separate name/value pairs of a parameter list in a URL. name1=Value1&name2=Value2. Setting the delimiter to the ampersand character can separate the pairs into "name1=Value1" and "name2=Value2."
The following method called StringTokenize will do just that:
` initialize our local variables
If(Size of array($tokenList->)>0)
DELETE ELEMENT($tokenList->;1;Size of array($tokenList->))
While (($length>0)) ` for debug
` initialize our scope variables
` expand the array as necessary
` parse the text input one character at a time until either delim
` is reached and we have successfuly created a token or length has been reached
Until ((($char=$delim) & ($token#"")) | ($curPos>$length))
` reset the string buffer
` add the token to the list if not null
` return true on condition: added elements
$0:=Size of array($tokenList->)
If you run the following line of code after adding the above method...
StringTokenize("this is my string";char(Space);->myarray)
...myarray will be filled with the items "this" "is" "my" "string" since space is the delimiter.