【OpenCV】imread函数的简单分析

OpenCV是图像处理领域中一个非常重要的工具库，其中包含了许多传统图像处理算法，还支持深度图像处理的接口，是视频图像领域工程师需要熟悉的内容。这里简单分析一下imread()中C++版本的工作流程

1.imread()

imread()用于从指定文件中读取一张图片，返回一个OpenCV matrix（Mat格式）。读取图片的格式支持bmp，gif，jpeg等等，其中exif信息指的是Exchangeable Image File Format，即可交换图像文件格式，这种格式主要用于数码相机和其它影像设备拍摄的照片和视频中，它能够在jpeg、tiff等图像文件格式中嵌入诸如拍摄时间、设备型号、曝光参数、地理定位等信息，即元信息

由于读取的文件是压缩的格式（如jpeg等），所以在读取过程中需要先进行解码，然后转换成为cv::mat格式，才能够自由的使用。imread()整体框图如下所示，并以读取jpeg图片为例，大致流程为
（1）查找合适的解码器（findDecoder()）
（2）设置图像处理参数
（3）解析头信息（decoder->readHeader()）
  （a）初始化错误信息（jpeg_std_err()）
  （b）创建解压缩对象（jpeg_create_decompress()）
  （c）数据拷贝（jpeg_buffer_src()）
  （e）解析头信息（jpeg_read_header()）
（4）验证图像参数
（5）解析图像内容（decoder->readData()）
  （a）初始化解压缩函数（jpeg_start_decompress()）
  （b）执行解压缩（jpeg_read_scanlines）
  （c）确保数据处理完成并释放（jpeg_finish_decompress()）

1.1 imread()

函数的声明位于sources/modules/imgcodecs/src/loadsave.hpp

/** @brief Loads an image from a file.# 支持的格式，包括bmp，git，jpeg等等
-   Windows bitmaps - \*.bmp, \*.dib (always supported)
-   GIF files - \*.gif (always supported)
-   JPEG files - \*.jpeg, \*.jpg, \*.jpe (see the *Note* section)@note
# 读取时，根据文件的内容来确定格式，而不是文件名后缀
-   The function determines the type of an image by its content, not by the file extension.
# 读取文件之后，会对文件进行解码，并以BGR顺序存储通道
-   In the case of color images, the decoded images will have the channels stored in **B G R** order.@param filename Name of the file to be loaded.
@param flags Flag that can take values of `cv::ImreadModes`.
*/
// 读取的格式 flag 缺省为 BGR
CV_EXPORTS_W Mat imread( const String& filename, int flags = IMREAD_COLOR_BGR );

定义位于sources/modules/imgcodecs/src/loadsave.cpp

/*** Read an image**  This function merely calls the actual implementation above and returns itself.** @param[in] filename File to load* @param[in] flags Flags you wish to set.
*/
Mat imread( const String& filename, int flags )
{// 性能分析和追踪的宏，记录了调用函数名和进入的时间CV_TRACE_FUNCTION();/// create the basic containerMat img;/// load the data// 读取数据入口imread_( filename, flags, img );/// return a reference to the datareturn img;
}

1.2 imread_()

imread_()函数具体执行读取图像文件的任务，主要工作流程是：
（1）根据图像文件来查找可用的解码器（findDecoder）
（2）设置图像处理参数
（3）读取图像文件头（readHeader）
（4）读取及解码图像数据（readData）

static bool
imread_( const String& filename, int flags, OutputArray mat )
{/// Search for the relevant decoder to handle the imagery// 图片解码器ImageDecoder decoder;#ifdef HAVE_GDALif(flags != IMREAD_UNCHANGED && (flags & IMREAD_LOAD_GDAL) == IMREAD_LOAD_GDAL ){decoder = GdalDecoder().newDecoder();}else{
#endif/* 1.根据图像文件来查找可用的解码器 */ decoder = findDecoder( filename ); 	// 用于后续scale、readHeader等操作
#ifdef HAVE_GDAL}
#endif/// if no decoder was found, return nothing.if( !decoder ){return 0;}/* 2.设置图像处理参数 */// 设置缩放因子int scale_denom = 1;if( flags > IMREAD_LOAD_GDAL ){if( flags & IMREAD_REDUCED_GRAYSCALE_2 )scale_denom = 2;else if( flags & IMREAD_REDUCED_GRAYSCALE_4 )scale_denom = 4;else if( flags & IMREAD_REDUCED_GRAYSCALE_8 )scale_denom = 8;}// Try to decode image by RGB instead of BGR.// 尝试用RGB格式来解码图像，而不是BGRif (flags & IMREAD_COLOR_RGB && flags != IMREAD_UNCHANGED){decoder->setRGB(true);}/// set the scale_denom in the driverdecoder->setScale( scale_denom );/// set the filename in the driverdecoder->setSource( filename );try{// read the header to make sure it succeeds/* 3.读取文件头 */ if( !decoder->readHeader() )return 0;}catch (const cv::Exception& e){CV_LOG_ERROR(NULL, "imread_('" << filename << "'): can't read header: " << e.what());return 0;}catch (...){CV_LOG_ERROR(NULL, "imread_('" << filename << "'): can't read header: unknown exception");return 0;}// established the required input image size// 确保输入的图像尺寸有效Size size = validateInputImageSize(Size(decoder->width(), decoder->height()));// grab the decoded type// 获取解码的类型const int type = calcType(decoder->type(), flags);// 矩阵为空，则创建新的矩阵；否则，检查当前矩阵的信息if (mat.empty()){mat.create( size.height, size.width, type );}else{CV_CheckEQ(size, mat.size(), "");CV_CheckTypeEQ(type, mat.type(), "");CV_Assert(mat.isContinuous());}// read the image dataMat real_mat = mat.getMat();const void * original_ptr = real_mat.data;bool success = false;try{	/* 4.读取及解码图像数据 */if (decoder->readData(real_mat)){CV_CheckTrue(original_ptr == real_mat.data, "Internal imread issue");success = true;}}catch (const cv::Exception& e){CV_LOG_ERROR(NULL, "imread_('" << filename << "'): can't read data: " << e.what());}catch (...){CV_LOG_ERROR(NULL, "imread_('" << filename << "'): can't read data: unknown exception");}if (!success){mat.release();return false;}if( decoder->setScale( scale_denom ) > 1 ) // if decoder is JpegDecoder then decoder->setScale always returns 1{resize( mat, mat, Size( size.width / scale_denom, size.height / scale_denom ), 0, 0, INTER_LINEAR_EXACT);}/// optionally rotate the data if EXIF orientation flag says soif (!mat.empty() && (flags & IMREAD_IGNORE_ORIENTATION) == 0 && flags != IMREAD_UNCHANGED ){ApplyExifOrientation(decoder->getExifTag(ORIENTATION), mat);}return true;
}

1.2.1 查找解码器（findDecoder）

查找一个合适的解码器，例如bmp格式或者jpeg格式，使用的是BmpDecoder或JpegDecoder

static ImageDecoder findDecoder( const String& filename ) {size_t i, maxlen = 0;/// iterate through list of registered codecsImageCodecInitializer& codecs = getCodecs();for( i = 0; i < codecs.decoders.size(); i++ ){size_t len = codecs.decoders[i]->signatureLength();maxlen = std::max(maxlen, len);}/// Open the fileFILE* f= fopen( filename.c_str(), "rb" );/// in the event of a failure, return an empty image decoderif( !f ) {CV_LOG_WARNING(NULL, "imread_('" << filename << "'): can't open/read file: check file path/integrity");return ImageDecoder();}// read the file signatureString signature(maxlen, ' ');// 读取图像文件的前几个字符，这几个字符描述了当前文件的格式maxlen = fread( (void*)signature.c_str(), 1, maxlen, f );fclose(f);signature = signature.substr(0, maxlen);/// compare signature against all decoders// 寻找decoder列表中是否有这几个字符for( i = 0; i < codecs.decoders.size(); i++ ){if( codecs.decoders[i]->checkSignature(signature) )return codecs.decoders[i]->newDecoder();}/// If no decoder was found, return base typereturn ImageDecoder();
}

1.2.2 读取数据头（JpegDecoder–>readHeader）

数据头的读取函数由具体的解码器实现，以JpegDecoder为例
（1）初始化错误信息（jpeg_std_error）
（2）创建解码对象（jpeg_create_decompress）
（3）准备接收数据（jpeg_stdio_src）
（4）设置信息保留字段（jpeg_save_markers）
（5）解析头部（jpeg_read_header）

bool  JpegDecoder::readHeader()
{volatile bool result = false;close(); // 清理历史信息JpegState* state = new JpegState;m_state = state;/* 1.初始化错误信息 */state->cinfo.err = jpeg_std_error(&state->jerr.pub);state->jerr.pub.error_exit = error_exit;/* 一种非局部跳转的机制，用来处理异常或者错误情况,首次调用 setjmp 时，它会保存当前的执行环境到 buffer 中，并返回 0,这允许你在代码中设置一个“检查点”，如果后续发生错误或其他条件触发 longjmp,程序控制流将立即返回到这个 setjmp 调用的地方，并恢复当时保存的环境状态*/if( setjmp( state->jerr.setjmp_buffer ) == 0 ){/* 2.创建解码对象 */jpeg_create_decompress( &state->cinfo );// 如果内存中已经存入数据，则从source中拷贝数据到cinfo中;// 否则，从文件中读取数据if( !m_buf.empty() ){jpeg_buffer_src(&state->cinfo, &state->source);state->source.pub.next_input_byte = m_buf.ptr();state->source.pub.bytes_in_buffer = m_buf.cols*m_buf.rows*m_buf.elemSize();}else{/* 3.准备接收数据 */m_f = fopen( m_filename.c_str(), "rb" );if( m_f )jpeg_stdio_src( &state->cinfo, m_f );}if (state->cinfo.src != 0){/*4.设置信息保留字段(1) 保留 JPEG 图像中的 EXIF 数据（EXIF 一般放在 APP1 中）(2) 0xffff 表示最大长度用于在解码过程中保留图像的元信息*/jpeg_save_markers(&state->cinfo, APP1, 0xffff);/*5.读取并解析 JPEG 文件的头部信息(1) TRUE表示允许对jpeg头部进行一些修正，例如不完整的头部执行后，cinfo 结构体中将填充图像的基本信息：(a) 宽度（image_width）(b) 高度（image_height）(c) 颜色通道数（num_components）(d) 数据精度（data_precision）(e) 是否是彩色图像等*/jpeg_read_header( &state->cinfo, TRUE );// 设置 JPEG 解码器的输出尺寸缩放比例state->cinfo.scale_num=1;state->cinfo.scale_denom = m_scale_denom;m_scale_denom=1; // trick! to know which decoder used scale_denom see imread_// 根据当前解码器配置（包括缩放参数）计算最终输出图像的尺寸jpeg_calc_output_dimensions(&state->cinfo);m_width = state->cinfo.output_width;m_height = state->cinfo.output_height;// 通道数m_type = state->cinfo.num_components > 1 ? CV_8UC3 : CV_8UC1;result = true;}}return result;
}

1.2.2.1 初始化错误信息（jpeg_std_error）

错误信息的初始化使用的是函数指针

GLOBAL(struct jpeg_error_mgr *)
jpeg_std_error(struct jpeg_error_mgr *err)
{memset(err, 0, sizeof(struct jpeg_error_mgr));err->error_exit = error_exit;err->emit_message = emit_message;err->output_message = output_message;err->format_message = format_message;err->reset_error_mgr = reset_error_mgr;/* Initialize message table pointers */err->jpeg_message_table = jpeg_std_message_table;err->last_jpeg_message = (int)JMSG_LASTMSGCODE - 1;return err;
}

错误退出函数error_exit()，其中METHODDEF的定义为static，即将函数的返回值定义为static类型。查阅资料得知，这样定义是为了适配不同的编译环境和函数调用约定，暂时还不确定不同环境的区别

1.2.2.2 创建jpeg解压缩对象（jpeg_create_decompress）

jpeg_create_decompress是一个宏定义，调用函数jpeg_CreateDecompress()

#define jpeg_create_decompress(cinfo) \jpeg_CreateDecompress((cinfo), JPEG_LIB_VERSION, \(size_t)sizeof(struct jpeg_decompress_struct))

jpeg_CreateDecompress()定义位于sources\3rdparth\libjpeg\jdapimin.c中

// #define GLOBAL(type)            type
// 指定函数的链接属性（例如，在某些平台上可能被定义为 __declspec(dllexport) 或其他平台特定的关键字）
GLOBAL(void)
jpeg_CreateDecompress(j_decompress_ptr cinfo, int version, size_t structsize)
{// ...// 初始化结构体{struct jpeg_error_mgr *err = cinfo->err;void *client_data = cinfo->client_data; /* ignore Purify complaint here */memset(cinfo, 0, sizeof(struct jpeg_decompress_struct));cinfo->err = err;cinfo->client_data = client_data;}cinfo->is_decompressor = TRUE;/* Initialize a memory manager instance for this object */jinit_memory_mgr((j_common_ptr)cinfo);// ...
}

1.2.2.3 数据拷贝（jpeg_stdio_src）

函数的作用是设置标准输入源的函数

GLOBAL(void)
jpeg_stdio_src(j_decompress_ptr cinfo, FILE *infile)
{my_src_ptr src;/* The source object and input buffer are made permanent so that a series* of JPEG images can be read from the same file by calling jpeg_stdio_src* only before the first one.  (If we discarded the buffer at the end of* one image, we'd likely lose the start of the next one.)*/if (cinfo->src == NULL) {     /* first time for this JPEG object? */cinfo->src = (struct jpeg_source_mgr *)(*cinfo->mem->alloc_small) ((j_common_ptr)cinfo, JPOOL_PERMANENT,sizeof(my_source_mgr));src = (my_src_ptr)cinfo->src;src->buffer = (JOCTET *)(*cinfo->mem->alloc_small) ((j_common_ptr)cinfo, JPOOL_PERMANENT,INPUT_BUF_SIZE * sizeof(JOCTET));} else if (cinfo->src->init_source != init_source) {/* It is unsafe to reuse the existing source manager unless it was created* by this function.  Otherwise, there is no guarantee that the opaque* structure is the right size.  Note that we could just create a new* structure, but the old structure would not be freed until* jpeg_destroy_decompress() was called.*/ERREXIT(cinfo, JERR_BUFFER_SIZE);}src = (my_src_ptr)cinfo->src;src->pub.init_source = init_source;// 从文件中读取数据，填充到buffer中src->pub.fill_input_buffer = fill_input_buffer;src->pub.skip_input_data = skip_input_data;src->pub.resync_to_restart = jpeg_resync_to_restart; /* use default method */src->pub.term_source = term_source;src->infile = infile;src->pub.bytes_in_buffer = 0; /* forces fill_input_buffer on first read */src->pub.next_input_byte = NULL; /* until buffer loaded */
}

1.2.2.4 设置markers（jpeg_save_markers）

该函数的作用是告诉 libjpeg 在解码 JPEG 文件时，是否应该保存某些特定类型的标记段（marker segment），并设置这些标记段的最大保存长度。标记段用marker_code描述，最大长度用length_limit描述

GLOBAL(void)
jpeg_save_markers(j_decompress_ptr cinfo, int marker_code,unsigned int length_limit)
{// .../* Choose processor routine to use.* APP0/APP14 have special requirements.*/if (length_limit) {// 需要保存marker数据processor = save_marker;/* If saving APP0/APP14, save at least enough for our internal use. */if (marker_code == (int)M_APP0 && length_limit < APP0_DATA_LEN)length_limit = APP0_DATA_LEN;else if (marker_code == (int)M_APP14 && length_limit < APP14_DATA_LEN)length_limit = APP14_DATA_LEN;} else {// 不需要保存marker数据processor = skip_variable;/* If discarding APP0/APP14, use our regular on-the-fly processor. */if (marker_code == (int)M_APP0 || marker_code == (int)M_APP14)processor = get_interesting_appn;}// ...
}

1.2.2.5 Jpeg读取头部（jpeg_read_header）

函数是 libjpeg 库的一部分，用于读取 JPEG 数据流中的头部信息

GLOBAL(int)
jpeg_read_header (j_decompress_ptr cinfo, boolean require_image)
{// .../*jpeg_consume_input负责从输入源读取并处理 JPEG 数据，这里是处理头部数据(1) JPEG_REACHED_SOS: 已经读到了扫描开始标记（SOS）(2) JPEG_REACHED_EOI: 已经到达了图像结束标记（EOI）(3) JPEG_SUSPENDED: 输入数据不足，需要更多数据才能继续*/retcode = jpeg_consume_input(cinfo);switch (retcode) {case JPEG_REACHED_SOS: // 成功解析了图像的头信息retcode = JPEG_HEADER_OK;break;case JPEG_REACHED_EOI: // 没有找到图像，找到了EOI符号if (require_image)		/* Complain if application wanted an image */ERREXIT(cinfo, JERR_NO_IMAGE);/* Reset to start state; it would be safer to require the application to* call jpeg_abort, but we can't change it now for compatibility reasons.* A side effect is to free any temporary memory (there shouldn't be any).*/jpeg_abort((j_common_ptr) cinfo); /* sets state = DSTATE_START */retcode = JPEG_HEADER_TABLES_ONLY;break;case JPEG_SUSPENDED: // 输入数据暂时不足，导致操作挂起，则不执行任何额外操作，直接返回/* no work */break;}return retcode;
}

这里的jpeg_consume_input()的作用是从输入源读取并处理 JPEG 头部数据

GLOBAL(int)
jpeg_consume_input(j_decompress_ptr cinfo)
{int retcode = JPEG_SUSPENDED;/* NB: every possible DSTATE value should be listed in this switch */switch (cinfo->global_state) {case DSTATE_START:/* Start-of-datastream actions: reset appropriate modules */// 重置输入控制器，确保其处于初始状态(*cinfo->inputctl->reset_input_controller) (cinfo);/* Initialize application's data source module */// 初始化输入源（例如文件或内存缓冲区），准备读取数据(*cinfo->src->init_source) (cinfo);// 设置状态为 DSTATE_INHEADER，进入读取头部阶段cinfo->global_state = DSTATE_INHEADER;// 继续进入下一个 case 分支FALLTHROUGH                 /*FALLTHROUGH*/case DSTATE_INHEADER:// 处理输入数据的头部retcode = (*cinfo->inputctl->consume_input) (cinfo);// 返回值为 JPEG_REACHED_SOS（找到了 SOS 段），说明已经读到了图像数据的起始位置if (retcode == JPEG_REACHED_SOS) { /* Found SOS, prepare to decompress *//* Set up default parameters based on header data */// 设置默认解码参数（基于头部信息）default_decompress_parms(cinfo);/* Set global state: ready for start_decompress */// 状态更新为 DSTATE_READY，表示可以开始解码图像了cinfo->global_state = DSTATE_READY;}break;case DSTATE_READY:/* Can't advance past first SOS until start_decompress is called */retcode = JPEG_REACHED_SOS;break;// 这些状态表示解码过程正在进行中，例如正在预扫描、主扫描、缓冲图像数据等case DSTATE_PRELOAD:case DSTATE_PRESCAN:case DSTATE_SCANNING:case DSTATE_RAW_OK:case DSTATE_BUFIMAGE:case DSTATE_BUFPOST:case DSTATE_STOPPING:retcode = (*cinfo->inputctl->consume_input) (cinfo);break;default: // 如果当前状态不在预期范围内，则抛出错误，说明程序内部状态异常ERREXIT1(cinfo, JERR_BAD_STATE, cinfo->global_state);}return retcode;
}

reset_input_controller()的定义位于sources/3rdparty/libjpeg/jdinput.c中

METHODDEF(void)
reset_input_controller (j_decompress_ptr cinfo)
{my_inputctl_ptr inputctl = (my_inputctl_ptr) cinfo->inputctl;// 数据处理函数设置为consume_markers，处理标记符inputctl->pub.consume_input = consume_markers;inputctl->pub.has_multiple_scans = FALSE; /* "unknown" would be better */inputctl->pub.eoi_reached = FALSE;inputctl->inheaders = 1;/* Reset other modules */(*cinfo->err->reset_error_mgr) ((j_common_ptr) cinfo);(*cinfo->marker->reset_marker_reader) (cinfo);/* Reset progression state -- would be cleaner if entropy decoder did this */cinfo->coef_bits = NULL;
}

consume_markers()的定义如下，其工作流程大致为

/*consume_markers()↓检查是否已到 EOI？├─ 是 → 返回 JPEG_REACHED_EOI└─ 否 → 进入主循环↓read_markers() 读取下一个 marker↓根据返回值处理：JPEG_REACHED_SOS → 处理 SOS（首次/多次）JPEG_REACHED_EOI → 设置 eoi_reachedJPEG_SUSPENDED → 返回暂停状态其他 → 返回原值
*/
METHODDEF(int)
consume_markers (j_decompress_ptr cinfo)
{my_inputctl_ptr inputctl = (my_inputctl_ptr) cinfo->inputctl;int val;if (inputctl->pub.eoi_reached) /* After hitting EOI, read no further */return JPEG_REACHED_EOI;for (;;) {			/* Loop to pass pseudo SOS marker */// 不断读取markerval = (*cinfo->marker->read_markers) (cinfo);switch (val) {case JPEG_REACHED_SOS:	/* Found SOS */if (inputctl->inheaders) { /* 1st SOS 第一个SOS标识符 */if (inputctl->inheaders == 1)initial_setup(cinfo); // 初始设置（如分配内存、初始化 DCT 参数等）if (cinfo->comps_in_scan == 0) { /* pseudo SOS marker */ /* 假 SOS（伪标记）*/inputctl->inheaders = 2; // 等待下一次真正的 SOSbreak;}inputctl->inheaders = 0;/* Note: start_input_pass must be called by jdmaster.c* before any more input can be consumed.  jdapimin.c is* responsible for enforcing this sequencing.*/} else {			/* 2nd or later SOS marker */if (! inputctl->pub.has_multiple_scans)ERREXIT(cinfo, JERR_EOI_EXPECTED); /* Oops, I wasn't expecting this! */if (cinfo->comps_in_scan == 0) /* unexpected pseudo SOS marker */break;start_input_pass(cinfo); // 开始新的扫描}return val;case JPEG_REACHED_EOI:	/* Found EOI */inputctl->pub.eoi_reached = TRUE;if (inputctl->inheaders) { /* Tables-only datastream, apparently */ /* 只有表头，没有图像数据 */if (cinfo->marker->saw_SOF)ERREXIT(cinfo, JERR_SOF_NO_SOS);} else {/* Prevent infinite loop in coef ctlr's decompress_data routine* if user set output_scan_number larger than number of scans.*/if (cinfo->output_scan_number > cinfo->input_scan_number)cinfo->output_scan_number = cinfo->input_scan_number;}return val;case JPEG_SUSPENDED:return val;default:return val;}}
}

1.2.3 读取数据（JpegDecoder–>readData）

前面正确解析了图像数据的头部，现在读取图像数据并解析，主要有4个步骤：
（1）分析信息，包括通道数、Exif信息
（2）解码初始化（jpeg_start_decompress）
（3）读取数据，进行解码（jpeg_read_scanlines）
（4）结束解压缩（jpeg_finish_decompress）

bool  JpegDecoder::readData( Mat& img )
{volatile bool result = false;const bool color = img.channels() > 1;if( m_state && m_width && m_height ){jpeg_decompress_struct* cinfo = &((JpegState*)m_state)->cinfo;JpegErrorMgr* jerr = &((JpegState*)m_state)->jerr;if( setjmp( jerr->setjmp_buffer ) == 0 ){// ...// See https://github.com/opencv/opencv/issues/25274// Conversion CMYK->BGR is not supported in libjpeg-turbo.// So supporting both directly and indirectly is necessary.bool doDirectRead = false;/* 1.分析信息 */if( color ) /* 彩色图像 (多通道) */{if( cinfo->num_components != 4 ) // 不是4通道{
#ifdef JCS_EXTENSIONS // default 1cinfo->out_color_space = m_use_rgb ? JCS_EXT_RGB : JCS_EXT_BGR;cinfo->out_color_components = 3;doDirectRead = true; // BGR -> BGR
#elsecinfo->out_color_space = JCS_RGB;cinfo->out_color_components = 3;doDirectRead = m_use_rgb ? true : false; // RGB -> BGR
#endif}else{cinfo->out_color_space = JCS_CMYK; // CMYK格式cinfo->out_color_components = 4;doDirectRead = false; // CMYK -> BGR}}else /* 灰度图像 */{if( cinfo->num_components != 4 ){cinfo->out_color_space = JCS_GRAYSCALE;cinfo->out_color_components = 1;doDirectRead = true; // GRAY -> GRAY}else{cinfo->out_color_space = JCS_CMYK;cinfo->out_color_components = 4;doDirectRead = false; // CMYK -> GRAY}}// Check for Exif marker APP1/* 寻找 Exif 信息所在的 APP1 标记 */jpeg_saved_marker_ptr exif_marker = NULL;jpeg_saved_marker_ptr cmarker = cinfo->marker_list;while( cmarker && exif_marker == NULL ){if (cmarker->marker == APP1)exif_marker = cmarker;cmarker = cmarker->next;}// Parse Exif dataif( exif_marker ){const std::streamsize offsetToTiffHeader = 6; //bytes from Exif size field to the first TIFF headerif (exif_marker->data_length > offsetToTiffHeader){m_exif.parseExif(exif_marker->data + offsetToTiffHeader, exif_marker->data_length - offsetToTiffHeader);}}/* 2.解码初始化 */ jpeg_start_decompress( cinfo );if( doDirectRead){for( int iy = 0 ; iy < m_height; iy ++ ){uchar* data = img.ptr<uchar>(iy);/* 3.读取数据，进行解码 */if (jpeg_read_scanlines( cinfo, &data, 1 ) != 1) return false;}}else{JSAMPARRAY buffer = (*cinfo->mem->alloc_sarray)((j_common_ptr)cinfo,JPOOL_IMAGE, m_width*4, 1 );for( int iy = 0 ; iy < m_height; iy ++ ){uchar* data = img.ptr<uchar>(iy);if (jpeg_read_scanlines( cinfo, buffer, 1 ) != 1) return false;// ...}}result = true;/* 4.结束解压缩 */ jpeg_finish_decompress( cinfo );}}return result;
}

1.2.3.1 解压缩初始化（jpeg_start_decompress）

jpeg_start_decompress()是libjpeg中的一个函数，用于解压缩jpeg图像前的初始化。如果只读取一张jpeg图片，函数只会被调用一次，此时cinfo->global_state应该是DSTATE_READY状态，会调用jinit_master_decompress()执行主控制器的初始化

GLOBAL(boolean)
jpeg_start_decompress (j_decompress_ptr cinfo)
{if (cinfo->global_state == DSTATE_READY) {/* First call: initialize master control, select active modules */jinit_master_decompress(cinfo);if (cinfo->buffered_image) { // 使用图像缓冲模式，返回true，后续工作交给jpeg_start_output完成/* No more work here; expecting jpeg_start_output next */cinfo->global_state = DSTATE_BUFIMAGE;return TRUE;}cinfo->global_state = DSTATE_PRELOAD;}if (cinfo->global_state == DSTATE_PRELOAD) {// ...cinfo->output_scan_number = cinfo->input_scan_number;} else if (cinfo->global_state != DSTATE_PRESCAN)ERREXIT1(cinfo, JERR_BAD_STATE, cinfo->global_state);/* Perform any dummy output passes, and set up for the final pass */return output_pass_setup(cinfo);
}

1.2.3.1.1 主控制器初始化（jinit_master_decompress）

jinit_master_decompress()定义位于sources\3rdparty\libjpeg\jdmaster.c中，会根据上下文信息初始化主控制器

GLOBAL(void)
jinit_master_decompress (j_decompress_ptr cinfo)
{my_master_ptr master;master = (my_master_ptr) (*cinfo->mem->alloc_small)((j_common_ptr) cinfo, JPOOL_IMAGE, SIZEOF(my_decomp_master));cinfo->master = &master->pub;// 在输出图像数据之前初始化各个解码模块master->pub.prepare_for_output_pass = prepare_for_output_pass;// 结束输出master->pub.finish_output_pass = finish_output_pass;master->pub.is_dummy_pass = FALSE;// 据图像格式和用户配置，选择并初始化所有需要用到的解码模块master_selection(cinfo);
}

master_selection()初始化一系列模块，包括熵解码、IDCT、缓冲区控制器等

LOCAL(void)
master_selection (j_decompress_ptr cinfo)
{// ... /* Post-processing: in particular, color conversion first */if (! cinfo->raw_data_out) {if (master->using_merged_upsample) {
#ifdef UPSAMPLE_MERGING_SUPPORTEDjinit_merged_upsampler(cinfo); /* does color conversion too */
#elseERREXIT(cinfo, JERR_NOT_COMPILED);
#endif} else {jinit_color_deconverter(cinfo);jinit_upsampler(cinfo);}jinit_d_post_controller(cinfo, cinfo->enable_2pass_quant);}/* Inverse DCT */jinit_inverse_dct(cinfo); // 初始化IDCT/* Entropy decoding: either Huffman or arithmetic coding. */if (cinfo->arith_code)jinit_arith_decoder(cinfo); // 初始化算术解码器else {jinit_huff_decoder(cinfo); // 初始化huffman解码器}/* Initialize principal buffer controllers. */use_c_buffer = cinfo->inputctl->has_multiple_scans || cinfo->buffered_image;jinit_d_coef_controller(cinfo, use_c_buffer); // 初始化缓冲区控制器// ...
#endif /* D_MULTISCAN_FILES_SUPPORTED */
}

1.2.3.1.2 输出初始化（output_pass_setup）

LOCAL(boolean)
output_pass_setup (j_decompress_ptr cinfo)
{if (cinfo->global_state != DSTATE_PRESCAN) {/* First call: do pass setup */(*cinfo->master->prepare_for_output_pass) (cinfo); // 首次调用，初始化output passcinfo->output_scanline = 0;cinfo->global_state = DSTATE_PRESCAN;}/* Loop over any required dummy passes */while (cinfo->master->is_dummy_pass) {
#ifdef QUANT_2PASS_SUPPORTED/* Crank through the dummy pass */while (cinfo->output_scanline < cinfo->output_height) {JDIMENSION last_scanline;/* Call progress monitor hook if present */if (cinfo->progress != NULL) {cinfo->progress->pass_counter = (long) cinfo->output_scanline;cinfo->progress->pass_limit = (long) cinfo->output_height;(*cinfo->progress->progress_monitor) ((j_common_ptr) cinfo);}/* Process some data */last_scanline = cinfo->output_scanline;(*cinfo->main->process_data) (cinfo, (JSAMPARRAY) NULL,&cinfo->output_scanline, (JDIMENSION) 0);if (cinfo->output_scanline == last_scanline)return FALSE;		/* No progress made, must suspend */}/* Finish up dummy pass, and set up for another one */// 初始化输出(*cinfo->master->finish_output_pass) (cinfo);(*cinfo->master->prepare_for_output_pass) (cinfo);cinfo->output_scanline = 0;
#elseERREXIT(cinfo, JERR_NOT_COMPILED);
#endif /* QUANT_2PASS_SUPPORTED */}/* Ready for application to drive output pass through* jpeg_read_scanlines or jpeg_read_raw_data.*/cinfo->global_state = cinfo->raw_data_out ? DSTATE_RAW_OK : DSTATE_SCANNING;return TRUE;
}

prepare_for_output_pass()会初始化一些解码模块，包括反离散余弦变换（IDCT），系数读取/解码，cconvert 颜色空间转换（如 YCbCr → RGB），upsample上采样（chroma 上采样），cquantize 颜色量化（用于减少颜色数量），post 后处理缓冲区控制，main主输出控制器

METHODDEF(void)
prepare_for_output_pass (j_decompress_ptr cinfo)
{my_master_ptr master = (my_master_ptr) cinfo->master;if (master->pub.is_dummy_pass) { /* 虚拟输出（无输出） */
#ifdef QUANT_2PASS_SUPPORTED/* Final pass of 2-pass quantization */master->pub.is_dummy_pass = FALSE;(*cinfo->cquantize->start_pass) (cinfo, FALSE);(*cinfo->post->start_pass) (cinfo, JBUF_CRANK_DEST);(*cinfo->main->start_pass) (cinfo, JBUF_CRANK_DEST);
#elseERREXIT(cinfo, JERR_NOT_COMPILED);
#endif /* QUANT_2PASS_SUPPORTED */} else {if (cinfo->quantize_colors && cinfo->colormap == NULL) { // 启用了颜色量化但还没有颜色表/* Select new quantization method */if (cinfo->two_pass_quantize && cinfo->enable_2pass_quant) { // 检查是1遍量化还是2遍cinfo->cquantize = master->quantizer_2pass;master->pub.is_dummy_pass = TRUE;} else if (cinfo->enable_1pass_quant) {cinfo->cquantize = master->quantizer_1pass;} else {ERREXIT(cinfo, JERR_MODE_CHANGE);}}// 初始化反离散余弦变换(*cinfo->idct->start_pass) (cinfo);// 初始化系数解码(*cinfo->coef->start_output_pass) (cinfo);if (! cinfo->raw_data_out) {if (! master->using_merged_upsample)(*cinfo->cconvert->start_pass) (cinfo); // 初始化颜色空间转换(*cinfo->upsample->start_pass) (cinfo); // 初始化上采样if (cinfo->quantize_colors) (*cinfo->cquantize->start_pass) (cinfo, master->pub.is_dummy_pass); // 初始化颜色量化(*cinfo->post->start_pass) (cinfo,(master->pub.is_dummy_pass ? JBUF_SAVE_AND_PASS : JBUF_PASS_THRU)); // 后处理缓冲区控制(*cinfo->main->start_pass) (cinfo, JBUF_PASS_THRU); // 主输出控制器}}// ...
}

finish_output_pass()中会增加一个pass的计数器

METHODDEF(void)
finish_output_pass (j_decompress_ptr cinfo)
{my_master_ptr master = (my_master_ptr) cinfo->master;if (cinfo->quantize_colors)(*cinfo->cquantize->finish_pass) (cinfo);master->pass_number++;
}

1.2.3.2 解压缩（jpeg_read_scanlines）

/*(1) scanlines表示处理的一行数据(2) max_lines表示要处理多少行，调用时通常为1
*/
GLOBAL(JDIMENSION)
jpeg_read_scanlines (j_decompress_ptr cinfo, JSAMPARRAY scanlines,JDIMENSION max_lines)
{JDIMENSION row_ctr;if (cinfo->global_state != DSTATE_SCANNING)ERREXIT1(cinfo, JERR_BAD_STATE, cinfo->global_state);/*(1) output_height在jpeg_read_header()中确定，解压缩之后的图像为800x600，那么output_height=600(2) output_scanline表示扫描行数，不能超过output_height*/if (cinfo->output_scanline >= cinfo->output_height) {WARNMS(cinfo, JWRN_TOO_MUCH_DATA);return 0;}/* Call progress monitor hook if present */if (cinfo->progress != NULL) {cinfo->progress->pass_counter = (long) cinfo->output_scanline;cinfo->progress->pass_limit = (long) cinfo->output_height;(*cinfo->progress->progress_monitor) ((j_common_ptr) cinfo);}/* Process some data */row_ctr = 0;(*cinfo->main->process_data) (cinfo, scanlines, &row_ctr, max_lines); // 数据处理cinfo->output_scanline += row_ctr;return row_ctr;
}

cinfo->main->process_data通常情况下调用的是process_data_context_main()，进入了主解码函数当中，随后调用coef->decompress_data进行解码（在coef模块中），最后进行post->post_process_data后处理（在post模块中）

METHODDEF(void)
process_data_context_main(j_decompress_ptr cinfo, _JSAMPARRAY output_buf,JDIMENSION *out_row_ctr, JDIMENSION out_rows_avail)
{my_main_ptr main_ptr = (my_main_ptr)cinfo->main;/* Read input data if we haven't filled the main buffer yet */// 如果main buffer未满，先解压缩数据，填充bufferif (!main_ptr->buffer_full) {if (!(*cinfo->coef->_decompress_data) (cinfo,main_ptr->xbuffer[main_ptr->whichptr]))return;                   /* suspension forced, can do nothing more */main_ptr->buffer_full = TRUE;       /* OK, we have an iMCU row to work with */main_ptr->iMCU_row_ctr++;   /* count rows received */}/* Postprocessor typically will not swallow all the input data it is handed* in one call (due to filling the output buffer first).  Must be prepared* to exit and restart.  This switch lets us keep track of how far we got.* Note that each case falls through to the next on successful completion.*/switch (main_ptr->context_state) {case CTX_POSTPONED_ROW: /* 处理被推迟的行 *//* Call postprocessor using previously set pointers for postponed row */(*cinfo->post->_post_process_data) (cinfo,main_ptr->xbuffer[main_ptr->whichptr],&main_ptr->rowgroup_ctr,main_ptr->rowgroups_avail, output_buf,out_row_ctr, out_rows_avail);if (main_ptr->rowgroup_ctr < main_ptr->rowgroups_avail)return;                   /* Need to suspend */main_ptr->context_state = CTX_PREPARE_FOR_IMCU;if (*out_row_ctr >= out_rows_avail)return;                   /* Postprocessor exactly filled output buf */FALLTHROUGH                 /*FALLTHROUGH*/case CTX_PREPARE_FOR_IMCU: /* 准备处理下一个iMCU行 *//* Prepare to process first M-1 row groups of this iMCU row */main_ptr->rowgroup_ctr = 0;main_ptr->rowgroups_avail = (JDIMENSION)(cinfo->_min_DCT_scaled_size - 1);/* Check for bottom of image: if so, tweak pointers to "duplicate"* the last sample row, and adjust rowgroups_avail to ignore padding rows.*/if (main_ptr->iMCU_row_ctr == cinfo->total_iMCU_rows)set_bottom_pointers(cinfo);main_ptr->context_state = CTX_PROCESS_IMCU;FALLTHROUGH                 /*FALLTHROUGH*/case CTX_PROCESS_IMCU: /* 实际处理iMCU行 *//* Call postprocessor using previously set pointers */(*cinfo->post->_post_process_data) (cinfo,main_ptr->xbuffer[main_ptr->whichptr],&main_ptr->rowgroup_ctr,main_ptr->rowgroups_avail, output_buf,out_row_ctr, out_rows_avail);if (main_ptr->rowgroup_ctr < main_ptr->rowgroups_avail)return;                   /* Need to suspend *//* After the first iMCU, change wraparound pointers to normal state */if (main_ptr->iMCU_row_ctr == 1)set_wraparound_pointers(cinfo);/* Prepare to load new iMCU row using other xbuffer list */main_ptr->whichptr ^= 1;    /* 0=>1 or 1=>0 */main_ptr->buffer_full = FALSE;/* Still need to process last row group of this iMCU row, *//* which is saved at index M+1 of the other xbuffer */main_ptr->rowgroup_ctr = (JDIMENSION)(cinfo->_min_DCT_scaled_size + 1);main_ptr->rowgroups_avail = (JDIMENSION)(cinfo->_min_DCT_scaled_size + 2);main_ptr->context_state = CTX_POSTPONED_ROW;}
}

解码数据使用的是sources\3rdparty\libjpeg\jdcoefct.c下decompress_onepass()函数，进行了熵解码（entropy->decode_mcu）和反变换量化（inverse_DCT）的工作

METHODDEF(int)
decompress_onepass(j_decompress_ptr cinfo, _JSAMPIMAGE output_buf)
{// .../* Loop to process as much as one whole iMCU row */for (yoffset = coef->MCU_vert_offset; yoffset < coef->MCU_rows_per_iMCU_row;yoffset++) {for (MCU_col_num = coef->MCU_ctr; MCU_col_num <= last_MCU_col;MCU_col_num++) {/* Try to fetch an MCU.  Entropy decoder expects buffer to be zeroed. */jzero_far((void *)coef->MCU_buffer[0],(size_t)(cinfo->blocks_in_MCU * sizeof(JBLOCK)));if (!cinfo->entropy->insufficient_data)cinfo->master->last_good_iMCU_row = cinfo->input_iMCU_row;if (!(*cinfo->entropy->decode_mcu) (cinfo, coef->MCU_buffer)) { // 熵解码/* Suspension forced; update state counters and exit */coef->MCU_vert_offset = yoffset;coef->MCU_ctr = MCU_col_num;return JPEG_SUSPENDED;}/* Only perform the IDCT on blocks that are contained within the desired* cropping region.*/if (MCU_col_num >= cinfo->master->first_iMCU_col &&MCU_col_num <= cinfo->master->last_iMCU_col) {/* Determine where data should go in output_buf and do the IDCT thing.* We skip dummy blocks at the right and bottom edges (but blkn gets* incremented past them!).  Note the inner loop relies on having* allocated the MCU_buffer[] blocks sequentially.*/blkn = 0;               /* index of current DCT block within MCU */for (ci = 0; ci < cinfo->comps_in_scan; ci++) {// ...for (yindex = 0; yindex < compptr->MCU_height; yindex++) {if (cinfo->input_iMCU_row < last_iMCU_row ||yoffset + yindex < compptr->last_row_height) {output_col = start_col;for (xindex = 0; xindex < useful_width; xindex++) { // 反变换量化(*inverse_DCT) (cinfo, compptr,(JCOEFPTR)coef->MCU_buffer[blkn + xindex],output_ptr, output_col);output_col += compptr->_DCT_scaled_size;}}blkn += compptr->MCU_width;output_ptr += compptr->_DCT_scaled_size;}}}}/* Completed an MCU row, but perhaps not an iMCU row */coef->MCU_ctr = 0;}// ...
}

1.2.3.2.1 熵解码（decode_mcu）

jpeg格式编码时的步骤为变换量化和熵编码，解码时首先进行熵解码，decode_mcu()调用的是jdhuff.c中的decode_mcu()，更底层的不再深入

1.2.3.2.2 反变换量化（inverse_DCT）

inverse_DCT()调用了jddctint.c中的函数，支持多种DCT变换格式，例如非8x8的变换jpeg_idct_8x16、jpeg_idct_16x16等等，也支持不同的速度，例如jpeg_idct_islow、jpeg_idct_ifast，支持浮点运算，例如jpeg_idct_float。默认情况下，使用的是_jpeg_idct_islow()，不再深入分析

1.2.3.2.3 后处理（post_process_data）

post->post_process_data()对图像进行后处理，我在调试的时候发现会跳转到jdsample.c的sep_upsample()函数中，进行上采样，其中会调用fullsize_upsample()，随后调用jdcolor.c的ycc_rgb_convert()函数进行上采样数据的格式转换

METHODDEF(void)
sep_upsample(j_decompress_ptr cinfo, _JSAMPIMAGE input_buf,JDIMENSION *in_row_group_ctr, JDIMENSION in_row_groups_avail,_JSAMPARRAY output_buf, JDIMENSION *out_row_ctr,JDIMENSION out_rows_avail)
{// .../* Fill the conversion buffer, if it's empty */if (upsample->next_row_out >= cinfo->max_v_samp_factor) {for (ci = 0, compptr = cinfo->comp_info; ci < cinfo->num_components;ci++, compptr++) {/* Invoke per-component upsample method.  Notice we pass a POINTER* to color_buf[ci], so that fullsize_upsample can change it.*/// 上采样，调用了fullsize_upsample()(*upsample->methods[ci]) (cinfo, compptr,input_buf[ci] + (*in_row_group_ctr * upsample->rowgroup_height[ci]),upsample->color_buf + ci);}upsample->next_row_out = 0;}// ...// 颜色转换(*cinfo->cconvert->_color_convert) (cinfo, upsample->color_buf,(JDIMENSION)upsample->next_row_out,output_buf + *out_row_ctr,(int)num_rows);// ...
}

1.2.3.3 解压缩结束（jpeg_finish_decompress）

jpeg_finish_decompress 函数是 libjpeg 库中用于完成 JPEG 图像解压缩过程的一个关键函数。它的主要职责是确保所有剩余的数据都被正确处理，并且资源被适当地释放

GLOBAL(boolean)
jpeg_finish_decompress (j_decompress_ptr cinfo)
{if ((cinfo->global_state == DSTATE_SCANNING ||cinfo->global_state == DSTATE_RAW_OK) && ! cinfo->buffered_image) {/* Terminate final pass of non-buffered mode */if (cinfo->output_scanline < cinfo->output_height)ERREXIT(cinfo, JERR_TOO_LITTLE_DATA);(*cinfo->master->finish_output_pass) (cinfo);cinfo->global_state = DSTATE_STOPPING;} else if (cinfo->global_state == DSTATE_BUFIMAGE) {/* Finishing after a buffered-image operation */cinfo->global_state = DSTATE_STOPPING;} else if (cinfo->global_state != DSTATE_STOPPING) {/* STOPPING = repeat call after a suspension, anything else is error */ERREXIT1(cinfo, JERR_BAD_STATE, cinfo->global_state);}/* Read until EOI */while (! cinfo->inputctl->eoi_reached) {if ((*cinfo->inputctl->consume_input) (cinfo) == JPEG_SUSPENDED)return FALSE;		/* Suspend, come back later */}/* Do final cleanup */(*cinfo->src->term_source) (cinfo); // 终止数据源传输数据/* We can use jpeg_abort to release memory and reset global_state */jpeg_abort((j_common_ptr) cinfo);return TRUE;
}