-
Notifications
You must be signed in to change notification settings - Fork 46
CryptoLib: Testing and Validation
| Version # | Author | Revision Date | Peer Reviewer | Revision Reason |
|---|---|---|---|---|
| Test Plan and Procedures 11/19/2021 | R. Brown, D. Cody Cutright, S. Zemerick | 11/19/2021 | Justin Morris, John Lucas | Initial |
| 1.0 | R. Brown, D. Cody Cutright, S. Zemerick | 12/02/2021 | Justin Morris, John Lucas | Scheduled Release |
| 1.0.2 | R. Brown, D. Cody Cutright, S. Zemerick | 01/31/2022 | Justin Morris, John Lucas | Scheduled Release |
| # | Document | Title | Description |
|---|---|---|---|
| 1 | CryptoLib_FY22_Task_Plan_draft01.docx | CryptoLib Task Plan, Draft 01, 10/29/2021 | Task Plan developed by both ITC and JPL MGSS |
| 2 | 2021-11-05 CryptoLib Engineering Release Draft Docs Final.pdf | CryptoLib: 1st Engineering Release and API Draft Documentation 11/05/2021 | Engineering Release, API Documentation, and Task Plan Management Document |
This Test Plan and Procedures document describes the CryptoLib V&V testing that will be performed by the NASA IV&V Independent Test Capability (ITC) Team. The ITC Team is collaborating with the JPL AMMOS team to provide a telecommand (TC) CCSDS SDLS encryption C library. This plan describes in detail the testing scope, testing methods, frameworks, and how the results will be managed and maintained. This plan is meant to serve as a “living” document and will be updated as the Test Plan changes and matures.
This wiki document maintains and tracks the methods necessary to adequately test CryptoLib. The sections herein will be used to define an approach that will be used to fully test CryptoLib's functionality in different scenarios. These scenarios include but are not limited to, unit testing, validation testing, and system testing.
The CryptoLib library is a C library with public API functions that are meant to be called from a library user. For example, for TC encryption, the user provides an unencrypted transfer frame (TF) to the Crypto_TC_ApplySecurity() function and the function returns an encrypted TF for uplinking to the spacecraft.
The Testing Strategy for each category is described below. Each category will utilize a combination of test strategies that can include: 1) Compatibility Testing, 2) Unit Testing, 3) Validation Testing, 4) System Testing, 5) Regression Testing, and 6) Static Code Analysis using Klocwork.
The API functionality will primarily be tested through Unit Tests. Unit Tests are designed to test the inputs, outputs, and functionality of the API functions. Also, Unit Tests will be executed automatically on the github.com server and serve as first-check regression testing for new functionality. The Table below lists the API functions that are currently being Unit Tested. Please note that this is a current snapshot and Unit Tests are added often.
Each of the unit tests are named in such a manner that what is being tested is mostly distinguishable by their naming convention. Tests can easily be listed by calling the specific unit testing application with the –lists-tests flag. These tests will be listed in set.name fashion. As can be seen above the current tests are being utilized to test nominal paths, and the functionality of TC_ApplySecurity when functions like CryptoInit are not called, or bad information is passed. They are also being utilized to test that encryption is being applied properly, and that libgcrypt is being utilized in a correct manner. Current unit tests are described in the following section.
The codebase must be capable of building within the Ubuntu v 20.04, and CentOS/RedHat v 8.5 Operating Systems.
The testing of CryptoLib within the Ubuntu operating system has all been automated through the use of GitHub’s Continuous Integration and Continuous Deployment (CI/CI) capabilities. When code is pushed to main branches, or pull requests are created, several containers are created that test various builds with differing flags used to enable different features within the software – some of which include debug builds, MySQL capabilities, and the ability to swap between LibGCrypt and KMC capabilities. Each of these automated containers verify the different types of builds and combination of builds, and in addition to this, verify that unit tests and validation tests properly execute and pass. In addition to this, code coverage is also automatically performed to guarantee that all functionality has been properly tested within the codebase. CentOS, however, cannot be automated in this manner, so these tests will need to be run manually by a developer. There is a possibility of adding a container that is CentOS based to the automated build capabilities, but this has not yet been feasible, and the manual effort of testing the CentOS/RedHat environments has been minimal and sufficed to date.
Pass or fail criteria is based on the ability of the codebase to be built within the respective operating systems (cmake / make), as well as the ability to run unit and validation tests (make test). The inability to do so will result in a failure of these scenarios.
Testing begins upon the trigger of new code to a main branch of the CryptoLib repository, or upon a new pull request. Additionally, a new tag or release of the codebase will trigger Test Entry automatically for Ubuntu through the GitHub automated CI/CD, and manual efforts for the CentOS environment.
Testing is not performed in an automated fashion within feature or bug branches within the project repository. This testing is done manually by developers prior to merging branches, or creating pull requests. Automated testing resumes during and after a branch’s merge.
| Item to Test | Test Description |
|---|---|
| Function return values | The return value of every function should be tested for validity. Each piece of source code .c file will need to have a respective ut_xxx.c file to be used with testing. For example, crypto_tc.c would have a unit test of the naming convention ut_crypto_tc.c |
| Logic pathways | All logical pathways of every function should be tested, as well as the return values of the functions given known and expected values |
| Null and Invalid data | All functions should be tested for error handling and the capability to properly handle NULL values |
CRYPTO_C, CALC_CRC16
Test to verify and validate CRC16
CRYPTO_C, BAD_CC_FLAG
Test and validate bad CC flag
CRYPTO_C, PDU_SWITCH
PDU Switch testing
CRYPTO_C, EXT_PROC_PDU
Crypto Extended Procedures PDU Testing
CRYPTO_AOS, APPLY SECURITY
Testing Crypto Init with invalid SADB
CRYPTO_AOS, PROCESS_SECURITY
Testing Crypto AOS Process Security function
CRYPTO_CONFIG, CRYPTO_INIT_WITH_INCOMPLETE_CONFIG
Crypto Init with incomplete configuration
CRYPTO_CONFIG, CRYPTO_INIT_NO_MANAGED_PARAM_CONFIG
Crypto Init with no managed parameters configuration
CRYPTO_CONFIG, CRYPTO_INIT_MARIADB_NULL
Crypto Init with NULL Maria DB
CRYPTO_CONFIG, CRYPTO_INIT_KMCCRYPTO_NULL
Crypto Init with NULL KMC Crypto Configuration
CRYPTO_CONFIG, CRYPTO_INIT_INVALID_INTERFACE
Crypto Init with Invalid Interface
CRYPTO_CONFIG, CRYPTO_INIT_INVALID_SADB
Crypto Init with invalid SADB
CRYPTO_CONFIG, CRYPTO_CONFIG_MDB
Crypto Init with incomplete configuration
CRYPTO_CONFIG, CRYPTO_CONFIG_KMC
Crypto KMC Configuration
CRYPTO_MC, STATUS
Crypto MC Status Test
CRYPTO_MC, DUMP
Crypto MC, Dump Test
CRYPTO_MC, SELFTEST
Crypto MC SelfTest Test
CRYPTO_MC, READARSN
Crypto MC, ReadARSN Test
CRYPTO_MC, PROCESS
Crypto MC Process Security Test
CRYPTO_MC, TMLENGTH
Crypto MC Get TM Length Test
KMC_CRYPTO, HAPPY_PATH_APPLY_SEC_ENC_AND_AUTH
Nominal Encryption with KMC Crypto Service and JPL Unit Test MariaDB
KMC_CRYPTO, HAPPY_PATH_APPLY_SEC_AUTH_ONLY
Nominal Encryption with KMC Crypto Service and JPL Unit Test MariaDB
KMC_CRYPTO, HAPPY_PATH_PROCESS_SEC_ENC_AND_AUTH
Nominal Encryption with KMC Crypto Service and JPL Unit Test MariaDB
KMC_CRYPTO, HAPPY_PATH_PROCESS_SEC_AUTH_ONLY
Nominal Encryption with KMC Crypto Service and JPL Unit Test MariaDB
KMC_CRYPTO, HAPPY_PATH_APPLY_SEC_CMAC_AUTH_ONLY
Nominal Encryption with KMC Crypto Service and JPL Unit Test MariaDB
KMC_CRYPTO, HAPPY_PATH_PROCESS_SEC_CMAC_AUTH_ONLY
Nominal Encryption KMC Service and JPL Unit Test MariaDB
KMC_CRYPTO, HAPPY_PATH_APPLY_SEC_CMAC_LARGE_FRM_AUTH_ONLY
Nominal Encryption KMC Service and JPL Unit Test MariaDB
TC_APPLY_SECURITY, NO_CRYPTO_INIT
Testing to verify that CRYPTO_LIB_ERR_NO_INIT is returned when the Crypto_Init() function is not called before attempting to process any other inputs
TC_APPLY_SECURITY, NO_CONFIG
No Set Configuration
TC_APPLY_SECURITY, HAPPY_PATH_CLEAR
Nominal clear case
TC_APPLY_SECURITY, HAPPY_PATH_ENC
Nominal encryption case
TC_APPLY_SECURITY, HAPPY_PATH_AUTH_ENC
Nominal Authorized Encryption case
TC_APPLY_SECURITY, BAD_SPACE_CRAFT_ID
Asserting bad space craft ID case
TC_APPLY_SECURITY, BAD_VIRTUAL_CHANNEL_ID
Asserting bad virtual channel ID case
TC_APPLY_SECURITY, NULL_BUFFER
Asserting handling of null buffers
Standard AES-256 test vectors are being generated with predefined shared keys and known inputs to ensure that the functions Crypto_TC_ApplySecurity() and Crypto_TC_ProcessSecurity() correctly utilize the libgcrypt library. Known and existing plaintext/ciphertext test vectors will be utilized as inputs and outputs, and a comparison will be performed to verify that the Crypto_TC_ApplySecurity() function is encrypting as expectedn and that Crypto_TC_ProcessSecurity is decryping as expected. Please note that these tests are not designed to test the libgcrypt library, but instead, test ITC’s usage and configuration of the library.
The section below lists the NIST test cases as found at https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/CAVP-TESTING-BLOCK-CIPHER-MODES. The general idea will be to make use of these vectors within a testing framework that utilizes the Crypto_TC_ApplySecurity and Crypto_TC_ProcessSecurity functions. The vector plain text will have an appended header and FECF trailer. The test frame vector will then be digested by these functions and the payload output compared to the cipher text.
| Item to Test | Test Description |
|---|---|
| Function Validity | Each function should be tested for validity. This means less emphasis is given to the return code of the function |
| Algorithm Validity | Each encryption algorithm will need to be tested as appropriate: Authorization Only Encryption Test: Verify MAC Verify payload unchanged Authorization Only Decryption Test: Verify MAC Encryption Only Test: Verify payload output Decryption Only Test: Verify payload output Authorized Encryption Test: Verify MAC Verify payload output Authorized Decryption Test: Verify MAC Verify payload output |
| FECF | Tests that utilize the optional FECF transfer frame should ensure that new FECFs are valid upon completion |
Validation testing has been automated within the project’s GitHub repository for the Ubuntu Operating System (OS). Manual efforts are performed to verify testing within the CentOS and RedHat environments. Tests are run when code is committed to the main branches of the repository, or upon merge/pull request to these branches.
The tests should be verified as having been performed successfully within the Ubuntu, CentOS/RedHat OSes. Manual tests need to be performed within CentOS/RedHat when the code undergoes a release or tag.
Tests pass if all subtests within a test set complete successfully with expected return values. The entire test scenario is considered a failure, should any subtest within any test set has failed to successfully complete.
Tests are performed automatically when merges or commits are made to the main branches of the repository. These automated tests are only for the Ubuntu OS. Other operating system tests are performed manually when there is a tag or release of the codebase.
All automated test results are maintained within the project repository. These are stored within the actions tab. Snapshots of the repository are archived when a release or tag is generated for the repository.
Validation testing is not automatically performed within feature or bug branches of the repository. In order for this to be accomplished, changes to the workflow must be made within the branch to include it within the CI/CD Actions. Otherwise, these tests must be manually performed. All developers should perform due-diligence with running tests and verifying that they successfully pass before creating a pull-request, committing to a main repository, or when merging features or bug fixes.
ET_VALIDATION, AUTH_ENCRYPTION_TEST
Makes use of the Python3 Cryptodome module to verify that TC_APPLY_SECURITY properly utilizes LibGCrypt to produce the same results as an external third party software. This external Python program is internalized.
DT_VALIDATION, AUTH_DECRYPTION_TEST
Reverses the AUTH_ENCRYPTION_TEST to verify that TC_PROCESSSECURITY is functioning properly and makes appropriate use of libgcrypt. Output should be comparable to the original data that is passed into the authorization test.
NIST_ENC_VALIDATION, AES_GCM_256_IV_96_PT_128_TEST_0
AES_GCM 256 Test Vector Encryption: https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip
NIST_DEC_VALIDATION, AEC_GCM_256_IV_PT_128_TEST_0
AES-GCM 256 Test Vector Decryption: https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip
NIST_ENC_VALIDATION, AES_GCM_256_IV_96_PT_128_TEST_1
AES_GCM 256 Test Vector Encryption: https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip
NIST_DEC_VALIDATION, AES_GCM_256_IV_96_PT_128_TEST_1
AES_GCM 256 Test Vector Decryption: https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip
NIST_ENC_VALIDATION, AES_GCM_256_IV_96_PT_128_TEST_2
AES_GCM 256 Test Vector Encryption: https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip
NIST_DEC_VALIDATION, AES_GCM_256_IV_96_PT_128_TEST_2
AES_GCM 256 Test Vector Decryption: https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip
NIST_ENC_VALIDATION, AES_GCM_256_IV_96_PT_128_TEST_3
AES_GCM 256 Test Vector Encryption: https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip
NIST_DEC_VALIDATION, AES_GCM_256_IV_96_PT_128_TEST_3
AES_GCM 256 Test Vector Decryption: https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip
NIST_ENC_VALIDATION, AES_GCM_256_IV_96_PT_128_TEST_4
AES_GCM 256 Test Vector Encryption: https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip
NIST_DEC_VALIDATION, AES_GCM_256_IV_96_PT_128_TEST_4
AES_GCM 256 Test Vector Decryption: https://csrc.nist.gov/CSRC/media/Projects/Cryptographic-Algorithm-Validation-Program/documents/mac/gcmtestvectors.zip
NIST_ENC_MAC_VALIDATION, AES_GCM_256_IV_96_PT_128_TEST_0
MAC Validation Test, bitmask of zeros
NIST_ENC_MAC_VALIDATION, AES_GCM_256_IV_96_PT_128_TEST_1
MAC Validation Test, bitmask of ones
NIST_DEC_MAC_VALIDATION, AES_GCM_256_IV_96_PT_128_TEST_0
MAC Validation, bitmask of ones
NIST_DEC_MAC_VALIDATION, AES_GCM256_IV_96_PT_128_TEST_0_BAD_DATA
Failed MAC Validation, Bad Data
NIST_DEC_MAC_VALIDATION, AES_GCM256_IV_96_PT_128_TEST_0_BAD_MAC
Failed MAC Validation, Bad Mac
NIST_ENC_CMAC_VALIDATION, AES_CMAC_256_PT_128_TEST_0
Test CMAC, Bitmask of zeros
NIST_ENC_CMAC_VALIDATION, AES_CMAC_256_PT_128_TEST_1
Test CMAC, Bitmask of ones
NIST_DEC_CMAC_VALIDATION, AES_CMAC_256_PT_128_TESET_0
Test CMAC Decryption, Bitmask of zeros
NIST_DEC_CMAC_VALIDATION, AES_CMAC_256_PT_128_TESET_1
Test CMAC Decryption, Bitmask of ones
The validation tests make use of NIST vectors with known outputs to verify that TC_APPLYSECURITY and TC_PROCESSSECURITY generate the output that is expected via known output tests through the NIST standard, and referenced from the following page and document: https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/CAVP-TESTING-BLOCK-CIPHER-MODES.
Test Vector 0:
Key Length 256 bits, IV Length 96 bits, Plaintext Length 128 bits, MAC Length 120 bits
Key = ef9f9284cf599eac3b119905a7d18851e7e374cf63aea04358586b0f757670f8
IV = b6ac8e4963f49207ffd6374c
PT = 722ee47da4b77424733546c2d400c4e5
CT = 1224dfefb72a20d49e09256908874979
Tag = 882eafea22adf8dbed06a2265f907b
Test Case(s):
NIST_ENC_VALIDATION.AES_GCM_256_IV_96_PT_128_TEST_0
NIST_DEC_VALIDATION.AES_GCM_256_IV_96_PT_128_TEST_0
Authorization only MAC validation. Making use of both TC_Apply and TC_Process fucntionality.
Test Case(s):
NIST_ENC_MAC_VALIDATION.AES_GCM_256_IV_96_PT_128_TEST_0
NIST_DEC_MAC_VALIDATION.AES_GCM_256_IV_96_PT_128_TEST_0
Test Vector 1:
Key = e9ccd6eef27f740d1d5c70b187734e11e76a8ac0ad1702ff02180c5c1c9e5399
IV = 1af2613c4184dbd101fcedce
PT = 419635e6e12b257a8ecae411f94480ff
CT = 9cd21f414f1f54d5f6f58b1f2f77e5b6
Tag = 5ea75738899abce4c8b7e091054b59
Test Case(s):
NIST_ENC_VALIDATION.AES_GCM_256_IV_96_PT_128_TEST_1, NIST_DEC_VALIDATION.AES_GCM_256_IV_96_PT_128_TEST_1
Test Vector 2:
Key Length 256 bits, IV Length 96 bits, Plaintext Length 128 bits,** MAC Length** 120 bits
Key = 7ecc9dcb3d5b413cadc3af7b7812758bd869295f8aaf611ba9935de76bd87013
IV = 6805be41e983717bf6781052
PT = 73d4d7984ce422ac983797c0526ac6f9
CT = 487211dd440f4d09d00bc5c3158a822c
Tag = e74e74630331350c0d6233d11c554f
Test Case(s):
NIST_ENC_VALIDATION.AES_GCM_256_IV_96_PT_128_TEST_2, NIST_DEC_VALIDATION.AES_GCM_256_IV_96_PT_128_TEST_2
Test Vector 3:
[https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/CAVP-TESTING-BLOCK-CIPHER-MODES](https://csrc.nist.gov/projects/cryptographic- algorithm-validation-program/CAVP-TESTING-BLOCK-CIPHER-MODES)
Key Length 256 bits, IV Length 96 bits, Plaintext Length 128 bits, MAC Length 120 bits
Key = a881373e248615e3d6576f5a5fb68883515ae72d6a2938e3a6f0b8dcb639c9c0
IV = f0b744f157087df4e41818a9
PT = 07d1dc9930e710b1ebe533c81f671101
CT = b65a2878b9dddbd4a0204dae6a6a6fc0
Tag = dc34a5401542745a650c75e693367b
Test Case(s):
NIST_ENC_VALIDATION.AES_GCM_256_IV_96_PT_128_TEST_3, NIST_DEC_VALIDATION.AES_GCM_256_IV_96_PT_128_TEST_3
Test Vector 4:
Key Length 256 bits, IV Length 96 bits, Plaintext Length 128 bits, MAC Length 120 bits
Key = 84c90349539c2a7989cb24dfae5e4182382ae94ba717d385977017f74f0d87d6
IV = eeddeaf4355c826dfd153393
PT = 31c4e1d0ccece6b7a999bfc31f38559a
CT = 5c6cfbdd06c19445ecf500c21aeca173
Tag = c4df572d5c64825d7e8903ad268bc8
Test Case(s):
NIST_ENC_VALIDATION.AES_GCM_256_IV_96_PT_128_TEST_4, NIST_DEC_VALIDATION.AES_GCM_256_IV_96_PT_128_TEST_4
| Item to Test | Test Description |
|---|---|
| Ground to Spacecraft Communication | Ground -> CryptoLib -> TCP/UDP -> CryptoLib -> Spacecraft |
| Spacecraft to Ground Communication | Spacecraft -> CryptoLib -> TCP/UDP -> CryptoLib -> Ground |
This testing approach would utilize two instances of CryptoLib. One side would make use of the library coupled with a ground station, and the second instance would make use of CryptoLib integrated within a spacecraft. The overall test would verify the communication successfully from one instance to the other (and the reverse) through the CryptoLib software. This testing would lay the groundwork for implementing the ability to swap out different SDLS implementations on spacecrafts and ground stations.
The test would be performed on the Ubuntu OS within two instances. One instance being configured for ground station usage, and the second configured for the spacecraft and integrated with representative flight software.
System testing is considered to have passed, if all subtests within test sets complete successfully and return expected values from one instance to the next. Any type of failure within any subtest or test set is considered a test failure.
This testing will need to be manually performed upon each release or tag of the repository once the system testing has been fully implemented.
Test for a specific release will be stored within a test results directory that will be archived within that tag or release within the GitHub repository.
CryptoLib, and its functions all assume valid frames are passed into the system. It is up to the user or calling program calling the API to verify and validate these frames as they are passed into the library.
All testing activities will be tracked on GitHub as Activities/Issues, and test results will be stored locally on GitHub as well. Below are the return codes from Crypto_TC_ApplySecurity() that are utilized within all unit and validation tests:
| Success / Error Code Description | Constant |
|---|---|
| Success / No Error | CRYPTO_LIB_SUCCESS (0) |
| General Error | CRYPTO_LIB_ERROR (-1) |
| No Initialization | CRYPTO_LIB_ERR_NO_INIT (-2) |
| Invalid Transfer Frame Version Number | CRYPTO_LIB_ERR_INVALID_TFVN (-3) |
| Invalid Spacecraft ID | CRYPTO_LIB_ERR_INVALID_SCID (-4) |
| Invalid Virtual Channel ID | CRYPTO_LIB_ERR_INVALID_VCID (-5) |
| Invalid Map ID | CRYPTO_LIB_ERR_INVALID_MAPID (-6) |
| Invalid Command Control Flag | CRYPTO_LIB_ERR_INVALID_CC_FLAG (-7) |
| No Operational Security Association | CRYPTO_LIB_ERR_NO_OPERATIONAL_SA (-8) |
| Null Buffer | CRYPTO_LIB_ERR_NULL_BUFFER (-9) |