问题描述

在通过impala执行hive sql语句时, 抛出以下异常:

impala.error.OperationalError: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

问题分析

导致该问题的原因是因为hive sql在远程hive服务器上执行报错.

Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

经过网上查找和调试, 发现问题在于impala中利用hiveserver2建立链接后, 在创建cursor时可以传入参configuration用于配置执行环境.

    def cursor(self, user=None, configuration=None, convert_types=True,
               dictify=False, fetch_error=True):
        """Get a cursor from the HiveServer2 (HS2) connection.
        Parameters
        ----------
        user : str, optional
        **configuration : dict of str keys and values, optional
            Configuration overlay for the HS2 session.**
        ...
        """

解决方案

在本地的测试环境中, 通过添加对应的配置即可.

# 配置
example_conf = {
    'mapreduce.job.queuename':'QUEUE_TEST',
    'hive.exec.compress.intermediate':'true',
    'hive.auto.convert.join':'true',
    'hive.exec.parallel':'true'
}
# 执行
cursor.execute(sample_sql, configuration=example_conf)

References

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapredLocalTask

HIVE Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask

Hive jdbc执行seelct 语句时报 return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask