Containerize & Execute Apache NIFI-MINIFI Docker on RaspberryPI4 ARM64 in Less Than 15 minutes.

APACHE NIFI-MINIFI

Apache Minifi is a super light-weight version of NIFI made for the Edge Devices.

Despite the fact that Apache Minifi was made for the edge but till now people were unable to unleash the power of Minifi on ARM64 based systems like Raspberry PI4 due to the absence of ARM64 docker in the Apache Minfi.

So looking at this disadvantage I decided on the month of February to build ARM64 docker for MINFI so that MINIFI can be used to automate pipelines in systems like RaspberryPI4.

I created and contributed the ARM64 docker for Apache MINFI :

apache/nifi-minifi

Now my newly contributed module to Apache minfi can be leveraged to execute MiniFi on ARM64 based Edge systems like RaspberryPI4 and that too in a containerized manner.

Setting Up the RaspberryPI4:

1. Buy a RaspberryPi 4 with 4GB RAM and a 16 GB MicroSD Chip
2. Place the MicroSD Card in a USB based MicroSD reader.
3. Flash the MicroSD with image ubuntu-20.04-preinstalled-server-arm64+raspi.img.xz from https://ubuntu.com/download/raspberry-pi and download the 64 bit OS image from the link: https://ubuntu.com/download/raspberry-pi/thank-you?version=20.04&architecture=arm64+raspi [At the time if this code testing UBUNTU 19.10.1 as 20 was not released but for future do in 20 as it has suport till 2025.]
4. Take out the MicroSD card and put it to the RaspberryPI 4 MicroSD card slot and power on the RaspBerryPI4
5. Connect the RaspBerry PI 4 to a LAN cable or Connect To WiFi
6. Git Clone the Repository :

apache/nifi-minifi

Building The NiFi Docker Image For ARM64

# user @ puter in ~/Development/code/apache/nifi-minifi/minifi-docker/dockerhub_ARM64
$ docker images
REPOSITORY               TAG                 IMAGE ID            CREATED                 SIZE
apache/nifi-minifi-arm64 latest                 ddc9dce6019e        About a minute ago   734MB

Note: The default version of NiFi specified by the Dockerfile is typically that of one that is unreleased if working from source. To build an image for a prior released version, one can override the NIFI_VERSION build-arg with the following command:

docker build --build-arg=MINIFI_VERSION={Desired MiNiFi Version} -t apache/nifi-minifi-arm64:latest .

Running The MiniFi ARM64 container:

Supplying configuration to a container

The primary means by which a MiNiFi instance is configured is via the config.yml or the bootstrap.conf.

This can be accomplished through:

• the use of volumes, and
• overlaying the base image

Using volumes to provide configuration

The following example shows the usage of two volumes to provide both a config.yml and a bootstrap.conf to the container instance. This makes use of configuration files on the host and maps them to be used by the MiNiFi instance. This is helpful in scenarios where a single image is used for a variety of configurations.

docker run -d \
    -v ~/minifi-conf/config.yml:/opt/minifi/minifi-0.5.0/conf/config.yml \
    -v ~/minifi-conf/bootstrap.conf:/opt/minifi/minifi-0.5.0/conf/bootstrap.conf \
    apache/nifi-minifi-arm64:latest

MiniFi OutPut on Succesful Execution

MiNiFi running with PID 8.
tail: cannot open '/opt/minifi/minifi-current/logs/minifi-app.log' for reading: No such file or directory

Bootstrap Classpath: /opt/minifi/minifi-0.5.0/conf:/opt/minifi/minifi-0.5.0/lib/bootstrap/*:/opt/minifi/minifi-0.5.0/lib/*
Java home: /docker-java-home
MiNiFi home: /opt/minifi/minifi-0.5.0

Bootstrap Config File: /opt/minifi/minifi-0.5.0/conf/bootstrap.conf

tail: '/opt/minifi/minifi-current/logs/minifi-app.log' has appeared;  following new file
2020-06-05 10:18:42,044 INFO [main] org.apache.nifi.minifi.bootstrap.Command Starting Apache MiNiFi...
2020-06-05 10:18:42,060 INFO [main] org.apache.nifi.minifi.bootstrap.Command Working Directory: /opt/minifi/minifi-0.5.0

Test The Minifi Process:

ubuntu@RPI-NODE1:~$ ps -eaf|grep minifi
ubuntu    9861  9498  0 10:18 pts/0    00:00:00 docker run apache/nifi-minifi:0.5.0
systemd+  9994  9909  0 10:18 ?        00:00:00 /bin/sh -e /opt/minifi/scripts/start.sh
systemd+  9995  9994  0 10:18 ?        00:00:00 tail -F /opt/minifi/minifi-current/logs/minifi-app.log
systemd+  9996  9994  0 10:18 ?        00:00:00 /bin/sh /opt/minifi/minifi-current/bin/minifi.sh run
systemd+ 10009  9996  0 10:18 ?        00:00:00 /bin/sh /opt/minifi/minifi-current/bin/minifi.sh run
systemd+ 10010 10009  3 10:18 ?        00:00:07 /docker-java-home/bin/java -cp /opt/minifi/minifi-0.5.0/conf:/opt/minifi/minifi-0.5.0/lib/bootstrap/*:/opt/minifi/minifi-0.5.0/lib/* -Xms12m -Xmx24m -Dorg.apache.nifi.minifi.bootstrap.config.log.dir=/opt/minifi/minifi-0.5.0/logs -Dorg.apache.nifi.minifi.bootstrap.config.pid.dir=/opt/minifi/minifi-0.5.0/run -Dorg.apache.nifi.minifi.bootstrap.config.file=/opt/minifi/minifi-0.5.0/conf/bootstrap.conf org.apache.nifi.minifi.bootstrap.RunMiNiFi run
systemd+ 10049 10010 25 10:18 ?        00:00:53 /docker-java-home/bin/java -classpath /opt/minifi/minifi-0.5.0/./conf:/opt/minifi/minifi-0.5.0/./lib/hk2-locator-2.5.0-b42.jar:/opt/minifi/minifi-0.5.0/./lib/javax.annotation-api-1.2.jar:/opt/minifi/minifi-0.5.0/./lib/aopalliance-repackaged-2.5.0-b42.jar:/opt/minifi/minifi-0.5.0/./lib/accessors-smart-1.2.jar:/opt/minifi/minifi-0.5.0/./lib/antlr-runtime-3.5.2.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-client-dto-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-registry-client-0.2.0.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-logging-utils-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-administration-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-registry-flow-diff-0.2.0.jar:/opt/minifi/minifi-0.5.0/./lib/org.eclipse.jdt.core-3.8.2.v20130121.jar:/opt/minifi/minifi-0.5.0/./lib/minifi-utils-0.5.0.jar:/opt/minifi/minifi-0.5.0/./lib/osgi-resource-locator-1.0.1.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-site-to-site-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-persistent-provenance-repository-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/javax.servlet.jsp-2.3.2.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-utils-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/httpclient-4.5.3.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-authorizer-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/apache-el-8.0.33.jar:/opt/minifi/minifi-0.5.0/./lib/javax.inject-2.5.0-b42.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-site-to-site-client-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/jersey-media-json-jackson-2.26.jar:/opt/minifi/minifi-0.5.0/./lib/javax.ws.rs-api-2.1.jar:/opt/minifi/minifi-0.5.0/./lib/lucene-core-4.10.4.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-framework-authorization-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-framework-cluster-protocol-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/jetty-jsp-jdt-2.3.3.jar:/opt/minifi/minifi-0.5.0/./lib/jetty-webapp-9.4.3.v20170317.jar:/opt/minifi/minifi-0.5.0/./lib/jackson-mapper-asl-1.9.13.jar:/opt/minifi/minifi-0.5.0/./lib/asm-5.0.4.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-write-ahead-log-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/logback-classic-1.2.3.jar:/opt/minifi/minifi-0.5.0/./lib/javax.servlet-api-3.1.0.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-user-actions-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/commons-collections4-4.0.jar:/opt/minifi/minifi-0.5.0/./lib/jetty-servlets-9.4.3.v20170317.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-web-utils-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/httpcore-4.4.4.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-flowfile-repo-serialization-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/jetty-xml-9.4.3.v20170317.jar:/opt/minifi/minifi-0.5.0/./lib/lucene-analyzers-common-4.10.4.jar:/opt/minifi/minifi-0.5.0/./lib/minifi-nar-utils-0.5.0.jar:/opt/minifi/minifi-0.5.0/./lib/httpcore-nio-4.4.5.jar:/opt/minifi/minifi-0.5.0/./lib/quartz-2.2.1.jar:/opt/minifi/minifi-0.5.0/./lib/validation-api-2.0.0.Final.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-schema-utils-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/javax.servlet.jsp.jstl-api-1.2.1.jar:/opt/minifi/minifi-0.5.0/./lib/jersey-common-2.26.jar:/opt/minifi/minifi-0.5.0/./lib/javax.el-3.0.1-b10.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-properties-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/javax.el-api-3.0.0.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-framework-nar-utils-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/jackson-annotations-2.9.0.jar:/opt/minifi/minifi-0.5.0/./lib/hk2-api-2.5.0-b42.jar:/opt/minifi/minifi-0.5.0/./lib/javax.inject-1.jar:/opt/minifi/minifi-0.5.0/./lib/spring-core-4.2.4.RELEASE.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-socket-utils-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/jetty-schemas-3.1.jar:/opt/minifi/minifi-0.5.0/./lib/logback-core-1.2.3.jar:/opt/minifi/minifi-0.5.0/./lib/apache-jsp-8.0.33.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-api-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/commons-io-2.5.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-security-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/jersey-client-2.26.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-framework-core-api-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/ecj-4.4.2.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-framework-core-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/jersey-hk2-2.26.jar:/opt/minifi/minifi-0.5.0/./lib/jackson-core-2.9.1.jar:/opt/minifi/minifi-0.5.0/./lib/jersey-entity-filtering-2.26.jar:/opt/minifi/minifi-0.5.0/./lib/jetty-io-9.4.3.v20170317.jar:/opt/minifi/minifi-0.5.0/./lib/javax.servlet.jsp-api-2.3.1.jar:/opt/minifi/minifi-0.5.0/./lib/jetty-servlet-9.4.3.v20170317.jar:/opt/minifi/minifi-0.5.0/./lib/httpasyncclient-4.1.1.jar:/opt/minifi/minifi-0.5.0/./lib/jetty-http-9.4.3.v20170317.jar:/opt/minifi/minifi-0.5.0/./lib/guava-18.0.jar:/opt/minifi/minifi-0.5.0/./lib/jackson-core-asl-1.9.13.jar:/opt/minifi/minifi-0.5.0/./lib/json-smart-2.3.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-repository-models-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-registry-security-utils-0.2.0.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-framework-api-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/jackson-module-jaxb-annotations-2.8.4.jar:/opt/minifi/minifi-0.5.0/./lib/jcl-over-slf4j-1.7.25.jar:/opt/minifi/minifi-0.5.0/./lib/minifi-runtime-0.5.0.jar:/opt/minifi/minifi-0.5.0/./lib/jetty-util-9.4.3.v20170317.jar:/opt/minifi/minifi-0.5.0/./lib/netty-3.7.0.Final.jar:/opt/minifi/minifi-0.5.0/./lib/zookeeper-3.4.6.jar:/opt/minifi/minifi-0.5.0/./lib/apache-jsp-9.3.9.v20160517.jar:/opt/minifi/minifi-0.5.0/./lib/jline-0.9.94.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-security-utils-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/json-path-2.4.0.jar:/opt/minifi/minifi-0.5.0/./lib/jetty-continuation-9.4.3.v20170317.jar:/opt/minifi/minifi-0.5.0/./lib/curator-client-2.11.0.jar:/opt/minifi/minifi-0.5.0/./lib/jul-to-slf4j-1.7.25.jar:/opt/minifi/minifi-0.5.0/./lib/jetty-security-9.4.3.v20170317.jar:/opt/minifi/minifi-0.5.0/./lib/curator-framework-2.11.0.jar:/opt/minifi/minifi-0.5.0/./lib/curator-recipes-2.11.0.jar:/opt/minifi/minifi-0.5.0/./lib/slf4j-api-1.7.25.jar:/opt/minifi/minifi-0.5.0/./lib/commons-lang3-3.4.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-expression-language-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-registry-data-model-0.2.0.jar:/opt/minifi/minifi-0.5.0/./lib/minifi-framework-core-0.5.0.jar:/opt/minifi/minifi-0.5.0/./lib/bcpkix-jdk15on-1.55.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-data-provenance-utils-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/bcprov-jdk15on-1.55.jar:/opt/minifi/minifi-0.5.0/./lib/javassist-3.22.0-CR2.jar:/opt/minifi/minifi-0.5.0/./lib/jackson-databind-2.9.1.jar:/opt/minifi/minifi-0.5.0/./lib/jetty-server-9.4.3.v20170317.jar:/opt/minifi/minifi-0.5.0/./lib/hk2-utils-2.5.0-b42.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-properties-loader-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/commons-codec-1.10.jar:/opt/minifi/minifi-0.5.0/./lib/nifi-runtime-1.7.0.jar:/opt/minifi/minifi-0.5.0/./lib/commons-logging-1.2.jar -Dorg.apache.jasper.compiler.disablejsr199=true -Xmx256m -Xms256m -Dsun.net.http.allowRestrictedHeaders=true -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -Djava.security.egd=file:/dev/urandom -Djava.protocol.handler.pkgs=sun.net.www.protocol -Dnifi.properties.file.path=/opt/minifi/minifi-0.5.0/./conf/nifi.properties -Dnifi.bootstrap.listen.port=40353 -Dapp=MiNiFi -Dorg.apache.nifi.minifi.bootstrap.config.log.dir=/opt/minifi/minifi-0.5.0/logs org.apache.nifi.minifi.MiNiFi
ubuntu   11430 11150  0 10:22 pts/1    00:00:00 grep --color=auto minifi

Using volumes to provide configuration

Alternatively, it is possible to create a custom image inheriting from the published image. Creating a Dockerfile extending from the Apache NiFi MiNiFi base image allows users to overlay the configuration permanently into a newly built and custom image. A simple example follows:

cBuilding this Dockerfile will result in a custom image with the specified configuration files incorporated into the new image. This is best for applications where configuration is well defined and relatively static.