TL;DR to be found at the end
Recently I came into a situation that I "needed" Hadoop native libraries. Well, when I say "needed", I mean I was jus...
For further actions, you may consider blocking this person and/or reporting abuse
Depending on which Hadoop version you want to install, you may need to use an earlier Java version to package it. This can be done by temporarily changing the
JAVA_HOME
environment variable before runningmvn package
.In my case, instead of
I had to run
JDK11 is still in progress cwiki.apache.org/confluence/displa...
Yes, you are right. I didn't think of that use case, I just assumed people would have a compliant Java version already installed or set as the main one.
Thanks for this article. I get this error when compiling to generate package using this below command:
mvn package -Pdist,native -DskipTests -Dtar
branch: branch-3.2
[INFO] Apache Hadoop MapReduce NativeTask ................. FAILURE [ 1.766 s]
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.2.2-SNAPSHOT:cmake-compile (cmake-compile) on project hadoop-mapreduce-client-nativetask: make failed with error code 2 -> [Help 1]
It's because hadoop doesn't support macOS native building now.
See github.com/apache/hadoop/blob/trun...
Hello, I don't see enough of the message. Do you have java 8? Or maybe more of the error (line with
cause by
)?I have tried building this now and it works for me.
Yes I have java 8 and that's my java home. Hadoop, hive setup and working though. I read some of the compression codecs works only with native libraries and jobs will fail with java libraries.
I tried this on mac Mojave OS 10.14.6
What version of OpenSSL did you use?
I used OpenSSL 1.1 from Brew and I have this error
This seems more like an issue with the OpenSSL installation than the version you are using. Anyway, this is my current PC (not sure with which I have built it, but libs work)
According to
issues.apache.org/jira/browse/HADO...
OpenSSL 1.1 broke the compilation. They patched it but didn't include it to the version 2 build. Your tutorial used OpenSSL 1.0 (Open SSL 1.1 will have
OpenSSL@1.1
on the path)Too bad that Homebrew already deprecated OpenSSL 1.0
Thank you very much! Will update the post.
This issue discusses how to forcefully install OpenSSL 1.0 using Homebrew
github.com/Homebrew/homebrew-core/...
Hi Sasa,
Thank you for this ! I'm a bit new on hadoop and I'm trying to fix the "native library" thing based on your article. Everything is ok until "Apache Hadoop Common" build :
Mac OSX highSierra
Hadoop version 2.9.2
~/hadoop/tmp/hadoop branch-2.9.2 brew list
autoconf cmake gettext go libidn2 libunistring openjdk pcre2 pyenv-virtualenv sshpass wget
automake direnv git gzip libmpc maven openssl@1.1 pkg-config readline telnet zlib
bzip2 gcc gmp isl libtool mpfr pandoc pyenv snappy tree
openssl : ok
protoc : 2.5.0 ok
Error message :
[INFO] Apache Hadoop Common ............................... FAILURE [02:08 min]
I have :
[WARNING] /Users/ccompain/hadoop/tmp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:256:14: error: incomplete definition of type 'struct evp_cipher_ctx_st'
[WARNING] if (context->flags & EVP_CIPH_NO_PADDING) {
[WARNING] ~~~~~~~^
[WARNING] /usr/local/include/openssl/ossl_typ.h:90:16: note: forward declaration of 'struct evp_cipher_ctx_st'
[WARNING] typedef struct evp_cipher_ctx_st EVP_CIPHER_CTX;
[WARNING] ^
[WARNING] /Users/ccompain/hadoop/tmp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:262:20: error: incomplete definition of type 'struct evp_cipher_ctx_st'
[WARNING] int b = context->cipher->block_size;
[WARNING] ~~~~~~~^
[WARNING] /usr/local/include/openssl/ossl_typ.h:90:16: note: forward declaration of 'struct evp_cipher_ctx_st'
[WARNING] typedef struct evp_cipher_ctx_st EVP_CIPHER_CTX;
[WARNING] ^
[WARNING] /Users/ccompain/hadoop/tmp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:263:16: error: incomplete definition of type 'struct evp_cipher_ctx_st'
[WARNING] if (context->encrypt) {
[WARNING] ~~~~~~~^
[WARNING] /usr/local/include/openssl/ossl_typ.h:90:16: note: forward declaration of 'struct evp_cipher_ctx_st'
[WARNING] typedef struct evp_cipher_ctx_st EVP_CIPHER_CTX;
[WARNING] ^
[WARNING] /Users/ccompain/hadoop/tmp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:310:14: error: incomplete definition of type 'struct evp_cipher_ctx_st'
[WARNING] if (context->flags & EVP_CIPH_NO_PADDING) {
[WARNING] ~~~~~~~^
[WARNING] /usr/local/include/openssl/ossl_typ.h:90:16: note: forward declaration of 'struct evp_cipher_ctx_st'
[WARNING] typedef struct evp_cipher_ctx_st EVP_CIPHER_CTX;
[WARNING] ^
[WARNING] /Users/ccompain/hadoop/tmp/hadoop/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c:313:20: error: incomplete definition of type 'struct evp_cipher_ctx_st'
[WARNING] int b = context->cipher->block_size;
[WARNING] ~~~~~~~^
[WARNING] /usr/local/include/openssl/ossl_typ.h:90:16: note: forward declaration of 'struct evp_cipher_ctx_st'
[WARNING] typedef struct evp_cipher_ctx_st EVP_CIPHER_CTX;
[WARNING] ^
[WARNING] 5 errors generated.
[WARNING] 5 errors generated.
[WARNING] make[2]: *** [CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o] Error 1
[WARNING] make[2]: *** Waiting for unfinished jobs....
[WARNING] make[1]: *** [CMakeFiles/hadoop_static.dir/all] Error 2
[WARNING] make[1]: *** Waiting for unfinished jobs....
[WARNING] make[1]: *** [CMakeFiles/hadoop.dir/all] Error 2
[WARNING] make: *** [all] Error 2
Any idea ?
Thks,
Christophe
Hello, in all honesty, I don't remember having this problem, but seems like you aren't alone. Apparently it is an OpenSSL feature. This apache.org Jira seems to be talking precisely about your problem. Seems like they even provide places you need to fix in the C code.
When is $HADOOP_HOME defined?
when i run line 29 in script it fails and prints out the usage text for cp
$HADOOP_HOME
is not defined by me. It is part of proper Hadoop installation and I am not doing that in this article.Hello, I am trying to install hadoop native library on mac but I have some problems here I tried to solve it but I can’t. Please if u have any idea.
My problem looks like:
Thanks for writing this. Helped me a lot