Search⌘ K
AI Features
Home
Courses
Using Laravel for Advanced String Manipulation in PHP

A Simple PHP Tokenizer

Explore the process of tokenizing PHP code by combining PHP's built-in tokenizer with Laravel's collection features. Learn to implement cursors like VariableCursor, NumberCursor, and OpenTagCursor to iterate and parse PHP variables, numbers, and opening tags. This lesson guides you through creating a simple lexer to produce token arrays similar to PHP's token_get_all function, enhancing your string splitting skills in Laravel.

We'll cover the following...

Tokenizing PHP code

Let’s consider the example in the code below. Our example uses PHP’s token_get_all function to return the results of PHP’s tokenizer on the input string. We then use Laravel’s collection features to provide a friendlier name for each of the returned tokens:

PHP
<?php
$code = <<<'PHP'
<?php
$value = 5 + (532_323) - $total;
PHP;
collect(token_get_all($code))->map(function ($token) {
    if (!is_string($token) && count($token) > 1) {
        $token['name'] = token_name($token[0]);
    }
    return $token;
})->all();

Our example would produce the following output:

PHP
Array
(
    [0] => Array
        (
            [0] => 389
            [1] =>  1
            [name] => T_OPEN_TAG
        )
    [1] => Array
        (
            [0] => 266
            [1] => $value
            [2] => 2
            [name] => T_VARIABLE
        )
    [2] => Array
        (
            [0] => 392
            [1] =>  
            [2] => 2
            [name] => T_WHITESPACE
        )
    [3] => =
    [4] => Array
        (
            [0] => 392
            [1] =>  
            [2] => 2
            [name] => T_WHITESPACE
        )
    [5] => Array
        (
            [0] => 260
            [1] => 5
            [2] => 2
            [name] => T_LNUMBER
        )
    [6] => Array
        (
            [0] => 392
            [1] =>  
            [2] => 2
            [name] => T_WHITESPACE
        )
    [7] => +
    [8] => Array
        (
            [0] => 392
            [1] =>  
            [2] => 2
            [name] => T_WHITESPACE
        )
    [9] => (
    [10] => Array
        (
            [0] => 260
            [1] => 532_323
            [2] => 2
            [name] => T_LNUMBER
        )
    [11] => )
    [12] => Array
        (
            [0] => 392
            [1] =>  
            [2] => 2
            [name] => T_WHITESPACE
        )
    [13] => -
    [14] => Array
        (
            [0] => 392
            [1] =>  
            [2] => 2
            [name] => T_WHITESPACE
        )
    [15] => Array
        (
            [0] => 266
            [1] => $total
            [2] => 2
            [name] => T_VARIABLE
        )
    [16] => ;
)

Each element in our resulting array corresponds to some of our input text. For instance, the second element corresponds to the $value variable on line 2. The first value in all of our nested arrays contains the token identifier, the second value holds the contents of the match, and the third value includes the line number the token is located on. We can also see that values such as the + and = ...