Prepare Data for Training
Prepare arbitrary data in HDF5 format
In addition to image classification datasets, Caffe also have
"HDF5Data" layer for arbitrary inputs. This layer requires all training/validation data to be stored in hdf5 format files.
This example shows how to use python
h5py module to construct such hdf5 file and how to setup caffe
"HDF5Data" layer to read that file.
Build the hdf5 binary fileAssuming you have a text file
'train.txt'with each line with an image file name and a single floating point number to be used as regression target.
Once you have all
h5files and the corresponding test files listing them you can add an HDF5 input layer to your
As shown in above, we pass into Caffe a list of HDF5 files. That is because in the current version there's a size limit of 2GB for a single HDF5 data file. So if the training data exceeds 2GB, we'll need to split it into separate files.
If a single HDF5 data file exceeds 2GB we'll get an error message like
If the total amount of data is less than 2GB, shall we split the data into separate files or not?
According to a piece of comment in Caffe's source code, a single file would be better,
If shuffle == true, the ordering of the HDF5 files is shuffled, and the ordering of data within any given HDF5 file is shuffled, but data between different files are not interleaved.
Prepare image dataset for image classification task
Caffe has a build-in input layer tailored for image classification tasks (i.e., single integer label per input image). This input
"Data" layer is built upon an lmdb or leveldb data structure. In order to use
"Data" layer one has to construct the data structure with all training data.
A quick guide to Caffe's
BuildFirst thing you must do is build caffe and caffe's tools (
convert_imagesetis one of these tools).
After installing caffe and
makeing it make sure you ran
make toolsas well.
Verify that a binary file
convert_imagesetis created in
Prepare your dataImages: put all images in a folder (I'll call it here
Labels: create a text file (e.g.,
/path/to/labels/train.txt) with a line per input image <path/to/file> . For example:
In this example the first image is labeled
1 while the other two are labeled
Convert the datasetRun the binary in shell
Command line explained:
GLOG_logtostderrflag is set to 1 before calling
convert_imagesetindicates the logging mechanism to redirect log messages to stderr.
--resize_widthresize all input images to same size
--shufflerandomly change the order of images and does not preserve the order in the
- Following are the path to the images folder, the labels text file and the output name. Note that the output name should not exist prior to calling
convert_imagesetotherwise you'll get a scary error message.
Other flags that might be useful:
--backend- allows you to choose between an
--gray- convert all images to gray scale.
--encoded_type- keep image data in encoded (jpg/png) compressed form in the database.
--help- shows some help, see all relevant flags under Flags from tools/convert_imageset.cpp
You can check out
for an example how to use
see this thread for more information.