hive的shims相关类分析

在hive的源码中经常可以看到shims相关的类,shims相关类是用来兼容不同的hadoop和hive版本的,以HadoopShims为例org.apache.hadoop.hive.shims.HadoopShims是一个接口,,具体的实现类为

org.apache.hadoop.hive.shims.Hadoop20Shimsorg.apache.hadoop.hive.shims.Hadoop20SShimsorg.apache.hadoop.hive.shims.Hadoop23Shims

具体对应的那个shims类在ShimLoader类(org.apache.hadoop.hive.shims.ShimLoader)中有所定义,以HADOOP_SHIM_CLASSES为例:

/***ThenamesoftheclassesforshimmingHadoopforeachmajorversion.*/privatestaticfinalHashMap<String,String>HADOOP_SHIM_CLASSES=newHashMap<String,String>();//首先定义了一个hashmapstatic{HADOOP_SHIM_CLASSES.put(“0.20″,”org.apache.hadoop.hive.shims.Hadoop20Shims”);//0.20HADOOP_SHIM_CLASSES.put(“0.20S”,”org.apache.hadoop.hive.shims.Hadoop20SShims”);//1.x和0.20security版本的hadoopHADOOP_SHIM_CLASSES.put(“0.23″,”org.apache.hadoop.hive.shims.Hadoop23Shims”);//2.x版本的hadoop}//向hashmap中插入对应的项,key为version信息,value为实际对应的类

ShimLoader类通过getHadoopShims方法返回具体的HadoopShims

privatestaticHadoopShimshadoopShims;……publicstaticsynchronizedHadoopShimsgetHadoopShims(){if(hadoopShims==null){hadoopShims=loadShims(HADOOP_SHIM_CLASSES,HadoopShims.class);//调用loadShims方法}returnhadoopShims;}

而loadShims方法通过判断hadoop的版本,来获取hashmap中对应的设置类

privatestatic<T>TloadShims(Map<String,String>classMap,Class<T>xface){Stringvers=getMajorVersion();//获取hadoop的版本StringclassName=classMap.get(vers);returncreateShim(className,xface);}

具体哪个hadoop版本是由getMajorVersion获取,比如我们线上使用的是2.0.0-cdh4.6.0,那么getMajorVersion返回的值为0.23,对应的HadoopShims实现类为org.apache.hadoop.hive.shims.Hadoop23Shims

publicstaticStringgetMajorVersion(){Stringvers=VersionInfo.getVersion();//org.apache.hadoop.util.VersionInfo类的getVersion方法比如2.0.0-cdh4.6.0String[]parts=vers.split(“\\.”);if(parts.length<2){thrownewRuntimeException(“IllegalHadoopVersion:”+vers+”(expectedA.B.*format)”);}//SpecialhandlingforHadoop1.xand2.xswitch(Integer.parseInt(parts[0])){case0:break;case1:return”0.20S”;case2:return”0.23″;default:thrownewIllegalArgumentException(“UnrecognizedHadoopmajorversionnumber:”+vers);}StringmajorVersion=parts[0]+”.”+parts[1];//Ifwearerunningasecurityrelease,wewon’thaveUnixUserGroupInformation//(removedbyHADOOP-6299whenswitchingtoJAASforLogin)try{Class.forName(“org.apache.hadoop.security.UnixUserGroupInformation”);}catch(ClassNotFoundExceptioncnf){if(“0.20″.equals(majorVersion)){majorVersion+=”S”;}}returnmajorVersion;}

而不同的shims中规定了不同版本下的属性和方法等,比如判断是否localmode:

吃水不忘挖井人。

hive的shims相关类分析

相关文章:

你感兴趣的文章:

标签云: