[Falcor]

It looks like you have a grasp of it, but it looks like something is wrong with your description. Tokenize itself will split a string into parts seperated by spaces and commas.

setstring string,hello world;
tokenize #s(string);
and
tokenize hello world;

will both give you 2 tokens

#t(0) = hello
#t(1) = world

tokenize2 will add extra delimeters (delimeters are the letters or characters that the tokens will be split, this is ADDED to the 2 default ones: spaces and commas)

so

tokenize2 -,hello-world,foo=bar;
or
setstring string,hello-world,foo=bar;
tokenize2 -,#s(string);

will both give you 3 tokens.

#t(0): hello (which uses your extra delemiter "-" )
#t(1): world (which is sperated by the default comma delimeter)
#t(2): foo=bar (which is the last token)

In response to adams information that you do not need a #s() for tokenize2. That is a confusing statment. Any valid string parameter in the syntax of any script in graal is viable to use the #s() message code.