JSoft

https://medium.com//apache-nifi-big-data-handle-dbc75f9733e9

APACHE NIFI BIG DATA HANDLE # Splitting Records with Apache NiFi: Making Big Data More Manageable

# Splitting Records with Apache NiFi: Making Big Data More Manageable

In the world of data processing and ETL (Extract, Transform, Load) operations, Apache NiFi has become a household name. Its ability to efficiently handle large volumes of data from various sources and apply transformations makes it a valuable tool in the data engineer's arsenal. In this article, we'll explore how to use Apache NiFi's SplitRecord processor to break down a massive dataset into smaller, more manageable chunks.

# # The Challenge of Big Data

Imagine you have a dataset with a whopping 50,000 records, and you need to process it efficiently. The default behavior of Apache NiFi's GenerateFlowFile processor is to generate a specified number of records every 0 seconds, which can quickly lead to a flood of data. However, there's a simple solution to tackle this challenge - the SplitRecord processor.

# # Introducing the SplitRecord Processor

The SplitRecord processor is a powerful tool within Apache NiFi that allows you to split large data records into smaller, more manageable chunks. This is particularly useful when dealing with data that needs to be processed in parallel or when you want to break down a massive dataset into smaller pieces for easier analysis.

# # Step-by-Step Splitting

Let's walk through the process of splitting a dataset of 50,000 records into smaller chunks.

1. **Count Your Source Data Records:** First, you need to know the total number of records in your dataset. In this case, you have 50,000 records.

2. **Calculate the Split Size:** To determine how many records should be in each chunk, divide the total number of records by the desired chunk size. For instance, if you want chunks of 10,000 records each, divide 50,000 by 10,000, which equals 5.

3. **Configure the SplitRecord Processor:** In Apache NiFi, configure the SplitRecord processor to split the records based on your calculated chunk size. For example, set it to split every 10,000 records.

4. **Process Your Data:** As the data flows through Apache NiFi, the SplitRecord processor will break it into smaller, more manageable chunks of 10,000 records each.

By following these steps, you can efficiently manage and process large datasets without overwhelming your ETL pipeline.

# # Visual Aid

For a better understanding, here's a screenshot of a flow file configuration in Apache NiFi:

This screenshot demonstrates how the SplitRecord processor is set up to divide the data into smaller chunks based on the calculated split size.

# # Conclusion

Apache NiFi's SplitRecord processor is a valuable tool for data engineers and analysts dealing with large datasets. By breaking down data into smaller, more manageable chunks, it becomes easier to process, analyze, and gain insights from your data.

If you're interested in exploring more Apache NiFi tutorials and tips, be sure to visit [jsoft.live](https://jsoft.live), where you'll find a wealth of resources to enhance your data processing skills.

Feel free to ask any questions or share your experiences with Apache NiFi in the comments below. Happy data processing!

https://jsoft.live

jsoft.live

https://jsoft.live/

jsoft.live

https://www.youtube.com/watch?v=PGyhBwLyK2U

DevOps with GitLab CI Course - Build Pipelines and Deploy to AWS This course will teach you how to use GitLab CI to create CI/CD pipelines for building and deploying software to AWS.🎥 Course created by Valentin Despa.📚 C...

https://youtu.be/9RazJop8tjk

Next Ecommerce Advance Fast part 1 Next Ecommerce -Part 1

https://github.com/thapatechnical/reactjs-interview-questions

GitHub - thapatechnical/reactjs-interview-questions: List of top 500 ReactJS Interview Questions & Answers....Coding exercise questions are coming soon!! List of top 500 ReactJS Interview Questions & Answers....Coding exercise questions are coming soon!! - GitHub - thapatechnical/reactjs-interview-questions: List of top 500 ReactJS Interview Que...

Next js - Training
Topic: getStaticProps

export async function getStaticProps(context) {
try {
const res = await fetch('https://pokeapi.co/api/v2/pokemon?limit=150');
const { results } = await res.json();

//Map function is used to retrieve data .It return data and image
const pokemon = results.map((pokeman, index) => {
const paddedId = ('00' + (index + 1)).slice(-3);

const image = `https://assets.pokemon.com/assets/cms2/img/pokedex/detail/${paddedId}.png`;
return { ...pokeman, image };
});

return {
props: { pokemon },
};
}

catch (err) {
console.error(err);
}
}
Free Pokeman API for Testing

A list of pokeman name and url
https://pokeapi.co/api/v2/pokemon?limit=150
A list of pokemon images
https://assets.pokemon.com/assets/cms2/img/pokedex/detail/898.png

pokeapi.co {"count":1126,"next":"https://pokeapi.co/api/v2/pokemon?offset=150&limit=150","previous":null,"results":[{"name":"bulbasaur","url":"https://pokeapi.co/api/v2/pokemon/1/"},{"name":"ivysaur","url":"https://pokeapi.co/api/v2/pokemon/2/"},{"name":"venusaur","url":"https://pokeapi.co/api/v2/pokemon/3/"},...

Customize Django Admin

from djnago.contrib import admin
from . import models

class BlogAdminArea(admin.AdminSite):

site_header = 'Blog Admin Area'

blog_site = BlogAdminArea(name='BlogAdmin')
admin.site.register(models.post)

How to upload files in React using Axios
import React from 'react';
import axios from 'axios';

const Form = () => {
// a local state to store the currently selected file.
const [selectedFile, setSelectedFile] = React.useState(null);

const handleSubmit = (event) => {
event.preventDefault()
const formData = new FormData();
formData.append("selectedFile", selectedFile);
try {
const response = await axios({
method: "post",
url: "/api/upload/file",
data: formData,
headers: { "Content-Type": "multipart/form-data" },
});
} catch(error) {
console.log(error)
}
}

const handleFileSelect = (event) => {
setSelectedFile(event.target.files[0])
}

return (




)
};

export default Form;

How to set HTTP headers in Axios

const axios = require('axios').default;

// or
// import axios from 'axios';

axios({
url: ‘/user’,
method: 'post',
Headers: {
Authorization: `Bearer ${yourToken}`
Content-Type: ‘application/json’
}).then((response)=>{
console.log(response.data)
});

How to submit form data in post request using axios

import React from 'react';
import axios from 'axios';

const LoginForm = () => {

const [formValue, setformValue] = React.useState({
email: '',
password: ''
});

const handleSubmit = (event) => {
// we will fill this in the coming paragraph
}

const handleChange = (event) => {
setformValue({
...formValue,
[event.target.name]: event.target.value
});
}

return (

Login Form



Login


)
};

export default LoginForm;

To create a form-data we will use FormData Web API, which stores fields and its values as key-value pairs.

Next, make a HTTP POST request in axios with loginFormData passed as a data property value in the axios request object.

const handleSubmit = async() => {
// store the states in the form data
const loginFormData = new FormData();
loginFormData.append("username", formValue.email)
loginFormData.append("password", formValue.password)

try {
// make axios post request
const response = await axios({
method: "post",
url: "/api/login",
data: loginFormData,
headers: { "Content-Type": "multipart/form-data" },
});
} catch(error) {
console.log(error)
}
}

How to submit form data in post request using axios Learn to submit form data in HTTP post request using axios

https://youtu.be/9-Eu9t437M4

Wordpress react Tutorial simple but helpful.Abdul Al Mamun[email protected]/johnbangla" rel="ugc" target="_blank">https://www.jsoft.website/[email protected]/johnbangla

If anyone would like to start learning to develop a WordPress plugin. You can start from here. It is a very basic but effective example. It allows you to create custom posts and widget facilities.

Instruction: Step1:Just copy the below code and name it "jsoft.php"
Step2: put in the plugin folder.
Happy coding......

jsoft – jsoft jsoft

Project: Live Car's nameplate recognition using python and deep learning

https://www.codewithharry.com/blogpost/mongodb-cheatsheet

All MongoDb commands you will ever need (MongoDb Cheatsheet) | CodeWithHarry In this post, we will see a comprehensive list of all the mongodb commands you will ever need as a mongoDb beginner. This list covers almost all the mostly used commands in mongoDb.

Why Industry will choose Mongo Db (NOSQl) over RDMS.

-Very fast because searching system is quite faster using json object. \
- Can insert any field any time
Django Rest Api with MongoDb is a good Example
See: https://art-of-engineer.blogspot.com/2021/06/python-django-mongodb-crud-api-tutorial.html

Python Django + MongoDB CRUD API tutorial In this post we will see how to create CRUD APIs using Python Django as backend and MongoDB for database. Lets install the necessary modu...

use inventory --for creating database
db.dropDatabase() --drop
show dbs -- show all database

--For insert
db.inventory.insertMany( [
{ item: "journal", qty: 25, size: { h: 14, w: 21, uom: "cm" }, status: "A" },
{ item: "notebook", qty: 50, size: { h: 8.5, w: 11, uom: "in" }, status: "A" },
{ item: "paper", qty: 100, size: { h: 8.5, w: 11, uom: "in" }, status: "D" },
{ item: "planner", qty: 75, size: { h: 22.85, w: 30, uom: "cm" }, status: "D" },
{ item: "postcard", qty: 45, size: { h: 10, w: 15.25, uom: "cm" }, status: "A" }
]);

For find:

db.inventory.find( { size: { w: 21, h: 14, uom: "cm" } } )

Advantages of MongoDB
Flexible document schemas.
Code-native data access.
Change-friendly design.
Powerful querying and analytics.
Easy horizontal scale-out.

Current Project: A complete Ionic-based e-commerce mobile Application .

IONIC + ANgular + woo commerce API -A complete Mobile APP Training by jsoft.

project: https://shahajadaprinting.com/ -done

Shahajada Printing Press – Shahajada Printing Press SHAHAJADA PRINTING PRESS was founded Muhammad Mobarak Ali, the father of the present owner many years ago. On 28/09/2011, the present owner, Muhammad Mozammel Hosen, acquired 100% ownership from his father and has been conducting the business with a reputation.

JSoft Online study software solution For Training center.

For the Advance programmers who are searching pyAudio package problems.

Package Name: PyAudio
https://www.lfd.uci.edu/~gohlke/pythonlibs/

Python Extension Packages for Windows - Christoph Gohlke This page provides 32- and 64-bit Windows binaries of many scientific open-source extension packages for the official CPython distribution of the Python programming language. A few binaries are available for the PyPy distribution.

Eid Mubarak. May Allah accept our good deeds. May Allah grant us peace and prosperity in the holy Eid-ul-Adha. May Allah protect us all from the pandemic.

Virtual Doctor AI project ...

api: https://disease-info-api.herokuapp.com/diseases

disease-info-api.herokuapp.com { "diseases": [ { "id": 107, "name": "Rabies - WHO", "data_updated_at": "March 2016", "facts": [ "Rabies is a vaccine-preventable viral disease which occurs in more than 150 countries and territories.", "Dogs are the source of the vast majority of human rabies deaths, contributing up to 99% of all r...

https://www.youtube.com/watch?v=XDaQAmkDFX4&t=436s

Creating Django App on Docker Django Tutorial for Beginners : https://www.youtube.com/watch?v=OTmQOjsl0egSupport by becoming a Member : https://www.youtube.com/channel/UC59K-uG2A5ogwIrHw4...