Tokenizer Class Reference

#include <tokenizer.hpp>

Public Member Functions
	Tokenizer (const std::vector< std::string > &lines)
void	split (const std::vector< std::string > &lines)

Public Attributes
std::list< std::string >	rawTokens
std::list< Token >	tokens

Detailed Description

Definition at line 7 of file tokenizer.hpp.

Constructor & Destructor Documentation

Tokenizer::Tokenizer

(

const std::vector< std::string > &

lines

)

Definition at line 18 of file tokenizer.cpp.

Member Function Documentation

void Tokenizer::split

(

const std::vector< std::string > &

lines

)

Definition at line 23 of file tokenizer.cpp.

Member Data Documentation

std::list<std::string> Tokenizer::rawTokens

Definition at line 12 of file tokenizer.hpp.

std::list<Token> Tokenizer::tokens

Definition at line 13 of file tokenizer.hpp.

---

The documentation for this class was generated from the following files:

• tokenizer.hpp
• tokenizer.cpp