apache-nifi

name	apache-nifi
description	Expert guidance for Apache NiFi data integration platform including flow design, processors, controller services, process groups, NiFi Registry integration, and cluster configuration. Use this when working with data flows, processors, or NiFi configuration.
tags	["nifi","dataflow","data-integration"]
color	orange

Apache NiFi Expert Skill

You are an expert in Apache NiFi, a powerful data integration and distribution platform.

Core Concepts

Architecture

• FlowFiles: Data packages flowing through NiFi (content + attributes)
• Processors: Building blocks for data transformation and routing
• Connections: Queues between processors with backpressure
• Process Groups: Containers for organizing flows
• Controller Services: Shared services (DB connections, SSL contexts, etc.)
• Reporting Tasks: Monitoring and metrics collection

Key Processors

• Data Ingestion: GetFile, ListenHTTP, ConsumeKafka, GetFTP
• Data Egress: PutFile, InvokeHTTP, PublishKafka, PutFTP
• Routing & Transformation: RouteOnAttribute, UpdateAttribute, JoltTransformJSON
• Content: ExtractText, ReplaceText, SplitText, MergeContent
• Database: ExecuteSQL, PutSQL, QueryDatabaseTable
• Scripting: ExecuteScript (Groovy, Python, JavaScript)

Flow Design Best Practices

Organization

Process Group Structure:
├── Input Group (data ingestion)
├── Validation Group (data quality)
├── Transformation Group (ETL)
├── Enrichment Group (lookups)
└── Output Group (data egress)

Performance

• Use backpressure thresholds (object count + data size)
• Enable connection load balancing for clusters
• Set appropriate concurrent tasks per processor
• Use batching for high-volume flows

Error Handling

• Always configure failure relationships
• Use funnels to consolidate error handling
• Implement retry logic with penalization
• Log errors to provenance repository

NiFi Registry Integration

Version Control

# Connect to Registry
1. Add Registry Client in NiFi UI (Controller Settings)
2. Right-click Process Group → Version → Start version control
3. Select Registry, Bucket, and Flow name
4. Commit changes with descriptive messages

Best Practices

• Version control all production flows
• Use meaningful commit messages
• Create separate buckets per environment (dev/test/prod)
• Tag stable releases

Expression Language

Common Expressions

# Attributes
${filename}                           # Get attribute value
${filename:isEmpty()}                 # Check if empty
${filename:replace('.txt', '.csv')}   # String manipulation

# Date/Time
${now():format('yyyy-MM-dd')}         # Current date
${created:toDate('yyyy-MM-dd'):format('MM/dd/yyyy')} # Convert format

# Conditional
${attribute:equals('value'):ifElse('yes', 'no')} # If-else logic

# Content
${file.size}                          # FlowFile size
${file.size:gt(1000000)}             # Size comparison

Configuration Files

nifi.properties (Key Settings)

# Web UI
nifi.web.http.host=0.0.0.0
nifi.web.http.port=8080

# Clustering
nifi.cluster.is.node=true
nifi.cluster.node.address=nifi-node1
nifi.cluster.node.protocol.port=11443

# State Management
nifi.state.management.embedded.zookeeper.start=true
nifi.zookeeper.connect.string=localhost:2181

# Repository Settings
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository

Docker Deployment

Single Node

services:
  nifi:
    image: apache/nifi:latest
    ports:
      - "8443:8443"
    environment:
      - SINGLE_USER_CREDENTIALS_USERNAME=admin
      - SINGLE_USER_CREDENTIALS_PASSWORD=ctsBtRBKHRAx69EqUghvvgEvjnaLjFEB
    volumes:
      - ./nifi/conf:/opt/nifi/nifi-current/conf
      - ./nifi/content_repository:/opt/nifi/nifi-current/content_repository
      - ./nifi/database_repository:/opt/nifi/nifi-current/database_repository
      - ./nifi/flowfile_repository:/opt/nifi/nifi-current/flowfile_repository
      - ./nifi/provenance_repository:/opt/nifi/nifi-current/provenance_repository
      - ./nifi/state:/opt/nifi/nifi-current/state

Cluster Configuration

services:
  zookeeper:
    image: zookeeper:3.8
    environment:
      ZOO_MY_ID: 1
      ZOO_SERVERS: server.1=zookeeper:2888:3888;2181

  nifi-1:
    image: apache/nifi:latest
    environment:
      - NIFI_CLUSTER_IS_NODE=true
      - NIFI_CLUSTER_NODE_PROTOCOL_PORT=11443
      - NIFI_ZK_CONNECT_STRING=zookeeper:2181
      - NIFI_ELECTION_MAX_WAIT=1 min
    depends_on:
      - zookeeper

Security

Authentication

• Single-User: Basic username/password (dev/test)
• LDAP: Enterprise directory integration
• OIDC: OAuth2/OpenID Connect (Keycloak)
• Certificate: mTLS with client certificates

Authorization

• File-based: policies.xml
• Ranger: Apache Ranger integration
• OIDC groups: Map OIDC groups to NiFi policies

Monitoring & Troubleshooting

Key Metrics

• FlowFile counts and sizes in queues
• Processor task durations
• System diagnostics (CPU, memory, disk)
• Bulletin board for errors

Logs

# Main logs
logs/nifi-app.log          # Application log
logs/nifi-user.log         # User actions
logs/nifi-bootstrap.log    # Bootstrap process

# Enable DEBUG logging
conf/logback.xml → Set logger level to DEBUG

Common Issues

Issue	Solution
Out of Memory	Increase heap in bootstrap.conf
Disk full	Clean content/provenance repos
Connection timeout	Check network/firewall rules
FlowFile stuck	Check backpressure, processor logs

PLC4X Integration

PLC4X Processors

• GetPLC: Read data from PLCs (Siemens S7, Modbus, OPC-UA, etc.)
• PutPLC: Write data to PLCs

Example Configuration

Connection String: s7://192.168.1.100
Fields: DB1.DBW0:INT,DB1.DBD4:REAL,DB1.DBX8.0:BOOL
Polling Interval: 1 sec

API & CLI

NiFi REST API

# Get process groups
curl -u admin:password https://localhost:8443/nifi-api/process-groups/root

# Get processor status
curl -u admin:password https://localhost:8443/nifi-api/processors/{id}/status

NiFi CLI (Toolkit)

# Export flow
nifi-toolkit/cli.sh nifi export-flow -u http://localhost:8080/nifi -b /tmp/backup.json

# Import flow
nifi-toolkit/cli.sh nifi import-flow -u http://localhost:8080/nifi -i /tmp/backup.json

Resources

• Apache NiFi Docs
• Expression Language Guide
• Processor Documentation