Monday, August 15, 2016

Cassandra 3.x - Could not initialize class com.sun.jna.Native

Recently I installed datastax cassandra 3.7 (datastax-ddc) in our environment, after installation and configuation, the Cassandra process refused to start with the folloiwng error message:
ERROR [main] 2016-08-12 12:48:06,053 CassandraDaemon.java:731 - Exception encountered during startup
java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:384) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.SystemKeyspace.forceBlockingFlush(SystemKeyspace.java:787) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.SystemKeyspace.removeTruncationRecord(SystemKeyspace.java:638) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilyStore.java:534) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.ColumnFamilyStore.invalidate(ColumnFamilyStore.java:510) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.schema.LegacySchemaMigrator.lambda$unloadLegacySchemaTables$327(LegacySchemaMigrator.java:136) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.schema.LegacySchemaMigrator$$Lambda$95/1799102760.accept(Unknown Source) ~[na:na]
        at java.lang.Iterable.forEach(Iterable.java:75) ~[na:1.8.0_45]
        at org.apache.cassandra.schema.LegacySchemaMigrator.unloadLegacySchemaTables(LegacySchemaMigrator.java:136) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.schema.LegacySchemaMigrator.migrate(LegacySchemaMigrator.java:82) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:245) [apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:585) [apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:714) [apache-cassandra-3.7.0.jar:3.7.0]
Caused by: java.util.concurrent.ExecutionException: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_45]
        at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_45]
        at org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:380) ~[apache-cassandra-3.7.0.jar:3.7.0]
        ... 12 common frames omitted
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.sun.jna.Native
        at org.apache.cassandra.utils.memory.MemoryUtil.allocate(MemoryUtil.java:98) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.io.util.Memory.<init>(Memory.java:74) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.io.util.SafeMemory.<init>(SafeMemory.java:32) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.io.compress.CompressionMetadata$Writer.<init>(CompressionMetadata.java:316) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.io.compress.CompressionMetadata$Writer.open(CompressionMetadata.java:330) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.io.compress.CompressedSequentialWriter.<init>(CompressedSequentialWriter.java:76) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.io.util.SequentialWriter.open(SequentialWriter.java:163) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.io.sstable.format.big.BigTableWriter.<init>(BigTableWriter.java:81) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.io.sstable.format.big.BigFormat$WriterFactory.open(BigFormat.java:92) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.io.sstable.format.SSTableWriter.create(SSTableWriter.java:101) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.io.sstable.SimpleSSTableMultiWriter.create(SimpleSSTableMultiWriter.java:119) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.compaction.AbstractCompactionStrategy.createSSTableMultiWriter(AbstractCompactionStrategy.java:550) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.compaction.CompactionStrategyManager.createSSTableMultiWriter(CompactionStrategyManager.java:902) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.ColumnFamilyStore.createSSTableMultiWriter(ColumnFamilyStore.java:499) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.Memtable$FlushRunnable.createFlushWriter(Memtable.java:466) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.Memtable$FlushRunnable.<init>(Memtable.java:405) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.Memtable$FlushRunnable.<init>(Memtable.java:382) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.Memtable.flushRunnables(Memtable.java:269) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.ColumnFamilyStore$Flush.flushMemtable(ColumnFamilyStore.java:1125) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at org.apache.cassandra.db.ColumnFamilyStore$Flush.run(ColumnFamilyStore.java:1098) ~[apache-cassandra-3.7.0.jar:3.7.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_45]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_45]

I do have the "/usr/share/cassandra/lib/jna-4.0.0.jar" in place (jna-4.0.0.jar comes with Cassandra installation). From the Cassandra log file, I see this warning:
WARN  [main] 2016-08-15 11:00:19,618 CLibrary.java:75 - JNA link failure, one or more native method will be unavailable.

and from the "utils/CLibrary.java"
try
{
    Native.register("c");
}
catch (NoClassDefFoundError e)
{
    logger.warn("JNA not found. Native methods will be disabled.");
    jnaAvailable = false;
}
catch (UnsatisfiedLinkError e)
{
    logger.warn("JNA link failure, one or more native method will be unavailable.");
    logger.trace("JNA link failure details: {}", e.getMessage());
}
catch (NoSuchMethodError e)
{
    logger.warn("Obsolete version of JNA present; unable to register C library. Upgrade to JNA 3.2.7 or later");
    jnaAvailable = false;
}

So JNA is installed for sure, it is just somehow system don't know how to link it properly. From command line, do a:
$ java -jar /usr/share/cassandra/lib/jna-4.0.0.jar

Exception in thread "main" java.lang.UnsatisfiedLinkError: /tmp/jna-115313/jna2403833344876007918.tmp: /tmp/jna-115313/jna2403833344876007918.tmp: failed to map segment from shared object: Operation not permitted
       at java.lang.ClassLoader$NativeLibrary.load(Native Method)
       at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1937)
       at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1822)
       at java.lang.Runtime.load0(Runtime.java:809)
       at java.lang.System.load(System.java:1086)
       >at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:761)
       at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:736)
       at com.sun.jna.Native.<clinit>(Native.java:131)

Bingo! Something to do with "/tmp", and a "cat /etc/fstab" shows us:
tmpfs                   /dev/shm                tmpfs   defaults,nosuid,nodev,noexec        0 0

"noexec" is the cause!

To fix this, you can either remount "tmpfs" without the "noexec", or you can use a different "tmp" directory. I choose the second path becase I don't wnat to comproimise the security. To do this, simply add:
JVM_OPTS="$JVM_OPTS -Djava.io.tmpdir=/var/lib/cassandra/tmp"
into "cassandra-env.sh"

No comments: