Thread: tokenize3
View Single Post
  #7  
Old 08-09-2002, 04:37 AM
Python523 Python523 is offline
Banned
Join Date: Aug 2001
Location: Illinois
Posts: 3,498
Python523 is on a distinguished road
Quote:
Originally posted by Tyhm
Except tokenize "bob" "went to" "the market" returns
"bob"
"went
to"
"the
market"
is his point
Last I checked when you tokened something in quotes, it took everything inside the quotes as 1 token, I'll check again just to make sure later =/
Reply With Quote