kinect data logging aka recording and playback?

[ahundt]

Is it possible to record kinect data to a file and play it back?

I'm calling this data logging but don't want to confuse it with the status/error logging components of the library I found in my searching for support.

If it doesn't exist, consider this a request for it! 👍

[xlz]

ROS can do this. I can't think of a use case of playback where there is no ROS.

Record and export to PCL would be a useful addition to examples.

[svddries]

I also think this would be very valuable, even when not using ROS or pcl (I'm currently working on a kinect project in which neither of them are used).

I guess the easiest way would be to store the raw rgb data packages obtained from the device (which are nicely JPG compressed) and either store the depth values for the depth stream directly, or compress them using PNG (which seems to give good compression results, in my experience).

[davudadiguezel]

It is possibe. You can use NiViewer that comes with openni2 to record. You'll get a .oni file that you can playback. But if you want to do skeleton detection with Nite2, you'll have to overcome some bugs in Nite2 and I don't know, if somebody has ever did this on Linux. If there is somebody please let me know. But recording and playback is with openni2 no problem.

[ahundt]

I don't need skeleton detection, just have to log the data stream because not everyone in my group has usb3 support, plus it would be able to test new code away from the physical rig just using a log of the data. I'm not using ROS and hope to remain cross platform and minimize overhead so I don't think I should necessarily start using it either, unfortunately.

[xlz]

Raw JPEG images are 20MB/s. "Raw" 32-bit float depth images are 25MB/s. Do you think you need compression?

[ahundt]

Hmm, at that data rate it could be tough to log to a non-ssd without compression so it would probably be a good option.

[ttsesm]

@ahundt did you manage to store raw data?

[ahundt]

@Theodoret Right now I'm just saving single frame point clouds out with this: https://github.com/ahundt/libfreenect2pclgrabber

A real implementation would likely still prove useful.

[ttsesm]

@ahundt thanks for the feedback, I see. Actually, at the moment I am saving individual frames as well. I am using opencv to convert libfreenect2::Frames into opencv cv::Mat

 listener.waitForNewFrame(frames);
libfreenect2::Frame *rgb = frames[libfreenect2::Frame::Color];
 libfreenect2::Frame *ir = frames[libfreenect2::Frame::Ir];
libfreenect2::Frame *depth = frames[libfreenect2::Frame::Depth];

cv::Mat(rgb->height, rgb->width, CV_8UC4, rgb->data).copyTo(rgbMat);
cv::Mat(ir->height, ir->width, CV_32FC1, ir->data).copyTo(irMat);
cv::Mat(depth->height, depth->width, CV_32FC1, depth->data).copyTo(depthMat);

//! [registration]
registration->apply(rgb, depth, &undistorted, &registered, true, &registeredInv);
//! [registration]

cv::Mat(undistorted.height, undistorted.width, CV_32FC1, undistorted.data).copyTo(undistortedMat);
cv::Mat(registered.height, registered.width, CV_8UC4, registered.data).copyTo(registeredMat);
cv::Mat(registeredInv.height, registeredInv.width, CV_32FC1, registeredInv.data).copyTo(registeredInvMat);

and then either to save as CV_16UC1 .png(for the depth images, CV_8UC4 for the rgb and registered images) files or as CV_32FC1 .exr files.

Mat registeredInv;
registeredInvMat.convertTo(registeredInv, CV_16UC1);
imwrite(filepath + "registeredInv/" + filename +".png", registeredInv);

or

imwrite(filepath + "registeredInv/" + filename +".exr", registeredInvMat);

However, though I prefer the 16bit .png files due to the file size I am facing the problem I am describing at the #509 with the bigdepth image and the white background if I try to restore it.

---

Also I noticed in your source code from the link above in the depth grabber function:

cv::Mat getDepth(){
        listener_.waitForNewFrame(frames_);
        libfreenect2::Frame * depth = frames_[libfreenect2::Frame::Depth];
        cv::Mat tmp(depth->height, depth->width, CV_8UC4, depth->data);
        cv::Mat r;
        if (mirror_ == true) {cv::flip(tmp,r,1);}
        else {r = tmp.clone();}

        listener_.release(frames_);
        return std::move(r);
    }

you are transforming the libfreenect2::Frame to a CV_8UC4 cv::Mat, shouldn't that be a CV_32FC1 cv::Mat?

[ahundt]

you are transforming the libfreenect2::Frame to a CV_8UC4 cv::Mat, shouldn't that be a CV_32FC1 cv::Mat?

You may be right, I guess the depth is 32 bit float values? Since I'm just flipping it and nothing else 4 bytes and a 32bit float are the same size so I guess that would explain why it works even if I'm using the wrong type... I should fix that, thanks :-)

[xlz]

Tentative specification for such a recording tool.

• JPEG re-compression. Raw JPEG images are 20MB/s. Maybe use TurboJPEG to save scale it down to half and then save it with lower quality.
• Depth compression. "Raw" 32-bit float depth images are 25MB/s. Maybe use a 16-bit fixed point number to represent that (with 1mm precision, the range of 65m is more than enough).
• Use format compatible with PCL. The most efficient one is PCLZF, which can be directly converted to pointclouds with no further registration. Storing full point clouds is very inefficient.

May implement it like this:

kinect2_record [-raw | -pclzf] [cl | gl] [serial]

-raw: unsynchronized, raw jpeg, 16-bit depth, 16-bit ir, 40MB/s
-pclzf: PCLZF format with registered color (512x424) and undistorted depth
[cl | gl] [serial]: the same arguments of Protonect.cpp

The -raw format saves frames in a directory:

kinect2_saves_${serial}_${starting_timestamp}/
camera.txt
frame_$epochsecond.microsecond_color_1920x1080.jpg
frame_$epochsecond.microsecond_depth_512x424.u16
frame_$epochsecond.microsecond_ir_512x424.u16

-pclzf:

kinect2_saves_${serial}_${starting_timestamp}/
frame_$epochsecond.microsecond.xml
frame_$epochsecond.microsecond_depth.pclzf
frame_$epochsecond.microsecond_rgb.pclzf

[ahundt]

Will the overhead of opening and closing many files a second be a problem? What about an optional dependency on a library designed for this sort of thing like flatbuffers or hdf5?