Devel::Tokenizer::C 0.05

Devel::Tokenizer::C is a Perl module that can generate C source for fast keyword tokenizer. SYNOPSIS use Devel::Tokenizer::C;

Developer:   Marcus Holland-Moritz       more software by author → Price:  0.00 License:   Perl Artistic License File size:   11K Language:    OS:    Rating:   0 /5 (0 votes) Your vote:   enlarge screenshot

Devel::Tokenizer::C is a Perl module that can generate C source for fast keyword tokenizer.

SYNOPSIS

use Devel::Tokenizer::C;

$t = new Devel::Tokenizer::C TokenFunc => sub { "return U$_[0];n" };

$t->add_tokens(qw( bar baz ))->add_tokens(['for']);
$t->add_tokens([qw( foo )], 'defined DIRECTIVE');

print $t->generate;

The Devel::Tokenizer::C module provides a small class for creating the essential ANSI C source code for a fast keyword tokenizer.

The generated code is optimized for speed. On the ANSI-C keyword set, it's 2-3 times faster than equivalent code generated with the gprof utility.

The above example would print the following C source code:
switch (tokstr[0])
{
case 'b':
switch (tokstr[1])
{
case 'a':
switch (tokstr[2])
{
case 'r':
if (tokstr[3] == '')
{ /* bar */
return BAR;
}

goto unknown;

case 'z':
if (tokstr[3] == '')
{ /* baz */
return BAZ;
}

goto unknown;

default:
goto unknown;
}

default:
goto unknown;
}

case 'f':
switch (tokstr[1])
{
case 'o':
switch (tokstr[2])
{
#if defined DIRECTIVE
case 'o':
if (tokstr[3] == '')
{ /* foo */
return FOO;
}

goto unknown;
#endif /* defined DIRECTIVE */

case 'r':
if (tokstr[3] == '')
{ /* for */
return FOR;
}

goto unknown;

default:
goto unknown;
}

default:
goto unknown;
}

default:
goto unknown;
}

So the generated code only includes the main switch statement for the tokenizer. You can configure most of the generated code to fit for your application.

Requirements:

Perl
tags goto unknown  switch tokstr  default goto  devel tokenizer  generated code  the generated  defined directive  add tokens  unknown default  source code  fast keyword  keyword tokenizer  

Download Devel::Tokenizer::C 0.05

 http://mirrors.evolva.ro/CPAN/authors/id/M/MH/MHX/Devel-Tokenizer-C-0.05.tar.gz

Authors software

Similar software

Devel::Tokenizer::C 0.05 (by Marcus Holland-Moritz) Devel::Tokenizer::C is a Perl module that can generate C source for fast keyword tokenizer. SYNOPSIS use Devel::Tokenizer::C;

Text-Tokenizer 0.4.0 (by Samuel Behan) Text::Tokenizer is a flex based lexical analyzator perl module. Text-Tokenizer can be used for parsing of text (configuration) fil

Convert::Morse 0.04 (by Convert::Morse Team) Convert::Morse is a package to convert between ASCII text and MORSE alphabet. SYNOPSIS use Convert::Morse qw(as_ascii as_mo

Parrot::OpTrans 0.4.5 (by The Parrot Project Team) Parrot::OpTrans is a Perl module that can transform Ops to C Code. Parrot::OpTrans is the abstract superclass for the Parrot op to

GOto 1.1 (by Cajus Pollmeier) GOto is a LDAP-based diskless client software. GOto provides a set of scripts to create a Debian-based boot image for diskless ope

Getopt::Simple 1.48 (by Ron Savage) Getopt::Simple is a Perl module that provides a simple wrapper around Getopt::Long. SYNOPSIS use Getopt::Simple;

Devel::Carnivore 0.09 (by Malte Ubl) Devel::Carnivore is a Perl module to spy on your hashes (and objects). SYNOPSIS use Devel::Carnivore; sub new_attribut

Class::Std 0.0.8 (by Damian Conway)

Getargs::Long 1.1001 (by Raphael Manfredi) Getargs::Long is a Perl module with named subroutine arguments, with optional type checking. SYNOPSIS use Getargs::Long;

Devel::Profile 1.04 (by Jeff Weisberg) Devel::Profile is a Perl module to tell me why my perl program runs so slowly. SYNOPSIS perl -d:Profile program.pl les

Other software in this category

zlib 1.2.3 (by Jean-loup Gailly) zlib is designed to be a free, general-purpose, legally unencumbered, lossless data-compression library for use on virtually any comp

libjpeg v6b (by Independent JPEG Group) libjpeg is a library for handling the JPEG (JFIF) image format

OpenSSL 0.9.7c (by The OpenSSL Project Team) The OpenSSL Project is a collaborative effort to develop a robust, commercial-grade, full-featured, and Open Source toolkit implement

libxml2 2.6.27 (by DV) Libxml2 is the XML C parser and toolkit developed for the Gnome project (but usable outside of the Gnome platform), libxml2 library i

GNU C library 2.4 (by Andreas Jaeger) GNU C library (glibc) is one of the most important components of GNU Hurd and most modern Linux distributions. GNU C library is us