Tuesday, July 1, 2008

Tokenization: Part 8, Summary

Have we really been covering tokenizing for seven posts? I thought it has been much longer than that.

Looking back on it, however, we’ve really talked about many things besides tokenizing and stop words:

  • regular expressions;
  • string and number literals;
  • variables;
  • calling functions;
  • creating functions;
  • saving source code;
  • namespaces;
  • calling Java methods;
  • higher-order functions;
  • sequence literals;
  • quoting;
  • sets;
  • sets as functions;
  • function overloading; and
  • recursive functions.

Many of these things we only touched on, and we’ll revisit them later in more depth. However, these are all basic concepts that we’ll use over and over during this series.

In the meantime, feel free to pat yourselves on the back!

For the next posting, we’ll start creating a filter to stem the tokens, which we’ll use as an excuse to explore functional programming in more detail.

No comments: