Tests: new AbstractTokenizerTestCase class

Until now, code coverage was not being recorded for the Tokenizer tests due to the way the tests were set up.

This new `AbstractTokenizerTestCase` class is intended to fix this.

**Background info:**
* Code run in a `setUpBeforeClass()` method does not cause code coverage to be recorded.
* Code run in a `setUp()` method _does_ cause code coverage to be recorded.

Now, the `AbstractMethodUnitTest` test case class was originally set up to allow for testing the utility methods in the `File` class.
To do so in the most efficient way and to allow the tests to be fast, the file containing the code being used in the tests, is only tokenized once in a `setUpBeforeClass()` method.

Over time, when Tokenizer tests were being introduced, the `AbstractMethodUnitTest` class also started to be used by these, which was fine as code coverage wasn't being measured or recorded anyway.

However, this last part has been changed recently via PR 144, so now it is time to ensure that the Tokenizer related tests record code coverage too.

This new test case class allows for that by moving the tokenization of the code used in the tests from `setUpBeforeClass()` to a `setUp()` method.
As the tokenized file won't change during the test run, this is still only done once, but as that "once" is now in the `setUp()` method, it should allow for code coverage to be recorded.

Note: while code coverage can now be measured and recorded for Tokenizer related tests, it will still not be very precise as the `Tokenizer\PHP` class contains "monster methods", which means that the `@covers` tags in the Tokenizer related tests cannot be set up to target only a small, specific part of that class, as targeted by the test. But that's a problem for another time.

Loosely related to 146

Author: jrfnl

tests/Core/Tokenizer/AbstractTokenizerTestCase.php

@@ -0,0 +1,105 @@
+<?php
+/**
+ * Base class to use when testing parts of the tokenizer.
+ *
+ * This is a near duplicate of the AbstractMethodUnitTest class, with the
+ * difference being that it allows for recording code coverage for tokenizer tests.
+ *
+ * @author    Juliette Reinders Folmer <phpcs_nospam@adviesenzo.nl>
+ * @copyright 2018-2019 Juliette Reinders Folmer. All rights reserved.
+ * @license   https://github.com/PHPCSStandards/PHP_CodeSniffer/blob/master/licence.txt BSD Licence
+ */
+
+namespace PHP_CodeSniffer\Tests\Core\Tokenizer;
+
+use PHP_CodeSniffer\Ruleset;
+use PHP_CodeSniffer\Files\DummyFile;
+use PHP_CodeSniffer\Tests\ConfigDouble;
+use PHP_CodeSniffer\Tests\Core\AbstractMethodUnitTest;
+use PHPUnit\Framework\TestCase;
+
+abstract class AbstractTokenizerTestCase extends TestCase
+{
+
+    /**
+     * The file extension of the test case file (without leading dot).
+     *
+     * This allows child classes to overrule the default `inc` with, for instance,
+     * `js` or `css` when applicable.
+     *
+     * @var string
+     */
+    protected $fileExtension = 'inc';
+
+    /**
+     * The tab width setting to use when tokenizing the file.
+     *
+     * This allows for test case files to use a different tab width than the default.
+     *
+     * @var integer
+     */
+    protected $tabWidth = 4;
+
+    /**
+     * The \PHP_CodeSniffer\Files\File object containing the parsed contents of the test case file.
+     *
+     * @var \PHP_CodeSniffer\Files\File
+     */
+    protected $phpcsFile;
+
+
+    /**
+     * Initialize & tokenize \PHP_CodeSniffer\Files\File with code from the test case file.
+     *
+     * The test case file for a unit test class has to be in the same directory
+     * directory and use the same file name as the test class, using the .inc extension.
+     *
+     * @before
+     *
+     * @return void
+     */
+    protected function initializeFile()
+    {
+        if (isset($this->phpcsFile) === false) {
+            $config = new ConfigDouble();
+            // Also set a tab-width to enable testing tab-replaced vs `orig_content`.
+            $config->tabWidth = $this->tabWidth;
+
+            $ruleset = new Ruleset($config);
+
+            // Default to a file with the same name as the test class. Extension is property based.
+            $relativeCN     = str_replace(__NAMESPACE__, '', get_called_class());
+            $relativePath   = str_replace('\\', DIRECTORY_SEPARATOR, $relativeCN);
+            $pathToTestFile = realpath(__DIR__).$relativePath.'.'.$this->fileExtension;
+
+            // Make sure the file gets parsed correctly based on the file type.
+            $contents  = 'phpcs_input_file: '.$pathToTestFile.PHP_EOL;
+            $contents .= file_get_contents($pathToTestFile);
+
+            $this->phpcsFile = new DummyFile($contents, $ruleset, $config);
+            $this->phpcsFile->process();
+        }
+
+    }//end initializeFile()
+
+
+    /**
+     * Get the token pointer for a target token based on a specific comment found on the line before.
+     *
+     * Note: the test delimiter comment MUST start with "/* test" to allow this function to
+     * distinguish between comments used *in* a test and test delimiters.
+     *
+     * @param string           $commentString The delimiter comment to look for.
+     * @param int|string|array $tokenType     The type of token(s) to look for.
+     * @param string           $tokenContent  Optional. The token content for the target token.
+     *
+     * @return int
+     */
+    protected function getTargetToken($commentString, $tokenType, $tokenContent=null)
+    {
+        return AbstractMethodUnitTest::getTargetTokenFromFile($this->phpcsFile, $commentString, $tokenType, $tokenContent);
+
+    }//end getTargetToken()
+
+
+}//end class