unpack throw exception

[Benjamin-Shengming]

I have tried this pkcs11 lib on ubuntu 19.04 with python 3.7.3, run with a vendor HSM(EngageBlack), all works good.

However, when I use it on an Linux from scratch system, using same HSM, I got following error.

bash-4.3# python3
Python 3.7.4 (default, Sep 24 2019, 10:51:40) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkcs11
>>> lib = pkcs11.lib("/usr/lib/libbvpkcs.so")
>>> token = lib.get_token()
>>> session = token.open()
>>> key = session.generate_key(pkcs11.KeyType.AES, 256)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pkcs11/_pkcs11.pyx", line 409, in pkcs11._pkcs11.Session.generate_key
  File "pkcs11/_pkcs11.pyx", line 564, in pkcs11._pkcs11.Object._make
  File "pkcs11/_pkcs11.pyx", line 615, in pkcs11._pkcs11.Object.__getitem__
  File "pkcs11/_utils.pyx", line 35, in pkcs11._pkcs11._unpack_attributes
  File "/usr/lib/python3.7/site-packages/python_pkcs11-0.5.0-py3.7-linux-x86_64.egg/pkcs11/defaults.py", line 129, in <lambda>
    lambda v: type_(unpack(v)))
  File "/usr/lib/python3.7/site-packages/python_pkcs11-0.5.0-py3.7-linux-x86_64.egg/pkcs11/defaults.py", line 114, in <lambda>
    _ulong = (Struct('L').pack, lambda v: Struct('L').unpack(v)[0])
struct.error: unpack requires a buffer of 8 bytes

are there any other dependencies I need to take care of?
Thanks

[danni]

What architecture are you on?

[Benjamin-Shengming]

virtualbox linux 64 bits

bash-4.3# uname -m
x86_64

bash-4.3# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                3
On-line CPU(s) list:   0-2
Thread(s) per core:    1
Core(s) per socket:    3
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Stepping:              3
CPU MHz:               3292.396
BogoMIPS:              6584.79
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              6144K
NUMA node0 CPU(s):     0-2

[danni]

What I'm guessing is that one of the #defines we use to set the sizes of things in pkcs11.h on your LFS system is missing. And so it's ending up as the wrong size. You might have to print out the contents of the #define environment the shared object is being compiled with.

[Benjamin-Shengming]

Do you mean these defines?

#define CK_PTR            *
#define CK_DEFINE_FUNCTION(returnType, name) returnType name
#define CK_DECLARE_FUNCTION(returnType, name) returnType name
#define CK_DECLARE_FUNCTION_POINTER(returnType, name) returnType (* name)
#define CK_CALLBACK_FUNCTION(returnType, name) returnType (* name)

#ifndef NULL_PTR
#define NULL_PTR          0
#endif

I find that cryptoki.h has been added to master branch but not in release 0.5.0

or some other defines used in cython?

[danni]

The defines in 0.5.0 were in setup.py, in master this has been changed. Does it work better in master?

[Benjamin-Shengming]

tried master branch still get same error:

Python 3.7.4 (default, Oct  3 2019, 04:30:14) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkcs11
>>> lib = pkcs11.lib("/opt/bvhsm/usr/lib64/libbvpkcs.so")
>>> token = lib.get_token()
>>> session = token.open()
>>> from pkcs11 import KeyType, Attribute
>>> 
>>> key = session.generate_key(KeyType.AES, 256, template={
...     Attribute.SENSITIVE: False,
...     Attribute.EXTRACTABLE: True,
... })
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "pkcs11/_pkcs11.pyx", line 433, in pkcs11._pkcs11.Session.generate_key
  File "pkcs11/_pkcs11.pyx", line 586, in pkcs11._pkcs11.Object._make
  File "pkcs11/_pkcs11.pyx", line 637, in pkcs11._pkcs11.Object.__getitem__
  File "pkcs11/_utils.pyx", line 35, in pkcs11._pkcs11._unpack_attributes
  File "/usr/lib/python3.7/site-packages/python_pkcs11-0.5.1.dev33+g46ed66a-py3.7-linux-x86_64.egg/pkcs11/defaults.py", line 132, in <lambda>
    lambda v: type_(unpack(v)))
  File "/usr/lib/python3.7/site-packages/python_pkcs11-0.5.1.dev33+g46ed66a-py3.7-linux-x86_64.egg/pkcs11/defaults.py", line 117, in <lambda>
    _ulong = (Struct('L').pack, lambda v: Struct('L').unpack(v)[0])
struct.error: unpack requires a buffer of 8 bytes

[danni]

Something is wrong with the length of that type. I would start by having a look how long the value v is. Also what item v refers to.

[Benjamin-Shengming]

It might be related to how Python was built. I can reproduce the issue by following steps:

1. on ubuntu14.04, download Python-3.7.4 source code and untar it,

2. guarantee libssl-dev installed, if not sudo apt install ligssl-dev

3. enter into python source folder and build python:

    *  ./configure --enable-shared 
    *  make 
    * sudo make install 

4. sudo pip3 install python-pkcs11

Now run python3, import pkcs11, and create one aes key, same error again

Python 3.7.4 (default, Oct  4 2019, 09:08:14) 
[GCC 4.8.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pkcs11
>>> lib = pkcs11.lib("/opt/bvhsm/usr/lib64/libbvpkcs.so")
>>> token = lib.get_token()
>>> session = token.open()
>>> key = session.generate_key(pkcs11.KeyType.AES, 256)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pkcs11/_pkcs11.pyx", line 409, in pkcs11._pkcs11.Session.generate_key
    return Object._make(self, key)
  File "pkcs11/_pkcs11.pyx", line 564, in pkcs11._pkcs11.Object._make
    object_class = self[Attribute.CLASS]
  File "pkcs11/_pkcs11.pyx", line 615, in pkcs11._pkcs11.Object.__getitem__
    return _unpack_attributes(key, value)
  File "pkcs11/_utils.pyx", line 35, in pkcs11._pkcs11._unpack_attributes
    return unpack(bytes(value))
  File "/home/service/python-pkcs11-0.5.0/pkcs11/defaults.py", line 129, in <lambda>
    lambda v: type_(unpack(v)))
  File "/home/service/python-pkcs11-0.5.0/pkcs11/defaults.py", line 114, in <lambda>
    _ulong = (Struct('L').pack, lambda v: Struct('L').unpack(v)[0])
struct.error: unpack requires a buffer of 8 bytes

[danni]

Okay so I'm guessing it's a mismatch between how long Python thinks an L is and how long PKCS#11 thinks an L is. Is one being compiled for a different word length than the other?

[Benjamin-Shengming]

more info

import pkcs11
from pkcs11 import KeyType
lib = pkcs11.lib("/opt/bvhsm/usr/lib64/libbvpkcs.so")
token = lib.get_token()
session = token.open() 
key = session.generate_key(KeyType.AES, 256)
unpack attributes
Attribute.CLASS <MemoryView of 'array' object>
len of value 4
b'\x04\x00\x00\x00'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pkcs11/_pkcs11.pyx", line 409, in pkcs11._pkcs11.Session.generate_key
  File "pkcs11/_pkcs11.pyx", line 564, in pkcs11._pkcs11.Object._make
  File "pkcs11/_pkcs11.pyx", line 615, in pkcs11._pkcs11.Object.__getitem__
  File "pkcs11/_utils.pyx", line 39, in pkcs11._pkcs11._unpack_attributes
  File "/usr/local/lib/python3.7/site-packages/python_pkcs11-0.5.0-py3.7-linux-x86_64.egg/pkcs11/defaults.py", line 129, in <lambda>
    lambda v: type_(unpack(v)))
  File "/usr/local/lib/python3.7/site-packages/python_pkcs11-0.5.0-py3.7-linux-x86_64.egg/pkcs11/defaults.py", line 114, in <lambda>
    _ulong = (Struct('L').pack, lambda v: Struct('L').unpack(v)[0])
struct.error: unpack requires a buffer of 8 bytes

[Benjamin-Shengming]

same os, same pkcs11 lib but python3.4 from ubuntu

>>> import pkcs11
>>> from pkcs11 import KeyType
>>> lib = pkcs11.lib("/opt/bvhsm/usr/lib64/libbvpkcs.so")
>>> token = lib.get_token()
>>> session = token.open()
>>> key = session.generate_key(KeyType.AES, 256)
unpack attributes
Attribute.CLASS <MemoryView of 'array' object>
len of value 8
b'\x04\x00\x00\x00\x00\x00\x00\x00'
unpack attributes
Attribute.ENCRYPT <MemoryView of 'array' object>
len of value 1
b'\x01'
unpack attributes
Attribute.DECRYPT <MemoryView of 'array' object>
len of value 1
b'\x01'
unpack attributes
Attribute.SIGN <MemoryView of 'array' object>
len of value 1
b'\x01'
unpack attributes
Attribute.VERIFY <MemoryView of 'array' object>
len of value 1
b'\x01'
unpack attributes
Attribute.WRAP <MemoryView of 'array' object>
len of value 1
b'\x01'
unpack attributes
Attribute.UNWRAP <MemoryView of 'array' object>
len of value 1
b'\x01'
unpack attributes
Attribute.DERIVE <MemoryView of 'array' object>
len of value 1
b'\x00'

[Benjamin-Shengming]

Okay so I'm guessing it's a mismatch between how long Python thinks an L is and how long PKCS#11 thinks an L is. Is one being compiled for a different word length than the other?

Does pkcs#11 mean pkcs11 module or python-pkcs11?

pkcs11 module is a shared library provided by hsm vendor and it is working with ubuntu python3.4 + python-pkcs11, however same pkcs11 module does not work with python3.7 built from scratch with python-pkcs11.

I hooked my own trace library into vendor hsm, and get some log:

[2019-10-10 08:50:24.564] [PKCS11_Trace] [info] virtual CK_RV PKCS11_Trace::C_GenerateKey(CK_SESSION_HANDLE, CK_MECHANISM_PTR, CK_ATTRIBUTE_PTR, CK_ULONG, CK_OBJECT_HANDLE_PTR)
[2019-10-10 08:50:24.564] [PKCS11_Trace] [info] session 100
[2019-10-10 08:50:24.564] [PKCS11_Trace] [info] MECHANISM type 4224/CKM_AES_KEY_GEN               
[2019-10-10 08:50:24.564] [PKCS11_Trace] [info] MECHANISM pParameter 0x0
[2019-10-10 08:50:24.564] [PKCS11_Trace] [info] MECHANISM ulParameterLen 0
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] ulcount 14
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] type 0 /  CKA_CLASS              
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] pValue 0x2130ec0 
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] ulValueLen 8
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] 0400000000000000
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] type 258 /  CKA_ID                 
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] pValue 0x15d0210 
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] ulValueLen 0
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] 
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] type 3 /  CKA_LABEL              
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] pValue 0x15d0210 
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] ulValueLen 0
[2019-10-10 08:50:24.565] [PKCS11_Trace] [info] 
.....
[2019-10-10 08:50:24.824] [PKCS11_Trace] [info] virtual CK_RV PKCS11_Trace::C_GetAttributeValue(CK_SESSION_HANDLE, CK_OBJECT_HANDLE, CK_ATTRIBUTE_PTR, CK_ULONG)
[2019-10-10 08:50:24.824] [PKCS11_Trace] [info] session 100
[2019-10-10 08:50:24.824] [PKCS11_Trace] [info] hObject 101
[2019-10-10 08:50:24.824] [PKCS11_Trace] [info] ulCount1
[2019-10-10 08:50:24.825] [PKCS11_Trace] [info] pTemplate 0x7ffdcda2d970
[2019-10-10 08:50:24.825] [PKCS11_Trace] [info] type 0 /  CKA_CLASS              
[2019-10-10 08:50:24.825] [PKCS11_Trace] [info] pValue 0x0 
[2019-10-10 08:50:24.826] [PKCS11_Trace] [info] ulValueLen 0
[2019-10-10 08:50:25.070] [PKCS11_Trace] [info]  CKR_OK
[2019-10-10 08:50:25.070] [PKCS11_Trace] [info] type 0 /  CKA_CLASS              
[2019-10-10 08:50:25.073] [PKCS11_Trace] [info] pValue 0x0 
[2019-10-10 08:50:25.073] [PKCS11_Trace] [info] ulValueLen 4

notice that when generate a aes key, the CKA_CLASS's ulValueLen is 8 but ulValueLen returned is 4 when query the key.

if the problem is in pkcs11 module(vendor's share library), we can't explain why the module works with python3.4 + python-pkcs

[danni]

We'll assume the PKCS#11 module is correct. But that our compilation of the headers might be wrong. Especially if the LFS machine is missing some #define (those headers are full of dynamically sized types).

[Benjamin-Shengming]

Especially if the LFS machine is missing some #define (those headers are full of dynamically sized types).

I can reproduce the issue on ubuntu 1404 as well.

1. Build python-3.7.4 from source
2. python-pkcs11-0.5.0 (pip3 install or install from source)

I got same problem. Could you solve this issue?