tokenize3

[Warcaptain]

Originally posted by G_yoshi


Eh?

I just fail to see the reason to duplicate an existing command. If it fails to work properly, then post it in the bugs forum Maybe Stefan will go back and make sure it works for the next release

tokenize2 +,bob+went to+have+sex;
#t(0)=bob
#t(1)=went
#t(2)=to
#t(3)=have
#t(4)=sex

MY way
tokenize3 +,bob+went to+have+sex;
#t(0)=bob
#t(1)=went to
#t(2)=have
#t(3)=sex

tokenize3 wouldnt do the ADDITIONAL delimeters it would just simply use new delimeters and remove all old forms.

no spaces etc...