Speed optimization of Big (Flow) Data Processing (using JAVA or Python or C programming Language)

Essential Technology

- Programming Language: JAVA or Python or C programming

- Required Experience: Network Big Data Processing

- Open Source Management System: InfluxDB, MariaDB, MongoDB (You can choose one of them for your database But, InfluxDB is preferred)

- Library: nfcapd & nfdump ( [login to view URL] ),

- Protocol: netflow, jflow, netstream, sflow

- OS: Linux, CentOS7

- Server requirement:

 Basically, an assigned developer must build own server and need a confirmation of OS version, programming language version, and DB version with Next-core Technology.

 Specification of our server being currently used is below. This specification may be improved. Thus, if an assigned developer has a suggestion regarding improvement (optimization) in server’s specification, we are open to consider the suggestion.

CPU: Intel® Xeon® CPU 2.3 GHz

2P24C/MEM: 128G/HDD: [login to view URL]

Development Period: 3 weeks (=15 business days but negotiable before the project begins)

Inspection Period: 4 business days (included in development period)

Development Cost: 1st requested way: USD $ 1,840 / 2nd requested way: USD $ need discussion

Project Objective: To optimize (or minimize) speed (from 5-7mins to within 2 mins) of present Big (Flow) Data processing (2 ways: 1) csv file to DB & 2) binary file to DB) via MariaDB or MongoDB, or InfluxDB using JAVA or Python or C programming

Project Summary

During data processing for about 5 millions of flow data, we are seeking to optimizing (or minimizing) current speed (5~7 minutes) of data processing from Server and database. The current speed must be within 2 minutes. Please see attachment called ""[login to view URL]" for your understanding.

As you see the workflow and the project objective above, our project is to optimize current data processing between Server and Database into 2 mins or less. When the objective is accomplished, Next-core Technology must retrieve source codes and other applicable references for the objective. Next are information of sample data and detailed course instructions in our project.

Data Given

1. Meta data (Field information) and sample data masked (1.5GB csv file) will be provided. We just attached data field information and examples of sample data first. Later, we will provide masked data. Please see the attached file for your development.

2. For this project, 5 millions of flow data will be standardized amounts. As you know, our objective is to make duration time of data processing between Server and Database into 2 minutes or less. Thus, we must make this 2 minutes or less with 5 million of flow data. To be specific, if you run 2.5 million data, your speed of data processing must be within 1 minute.

3. All masked data given will be flow data so that you should be familiar with their characteristics. For the 1st way (csv file to DB), you can refer to field information PDF file and csv file for your understanding. For the 2nd way (binary file to DB), we will discuss about which data to use after 1st way is figured out.

Please read carefully detailed courses of the development by opening the attached file called "Project [login to view URL]" before you approach to us. This guides you how to complete the project's objective.

If you have any other question, please let us know via chat @ Freelancer.

Java Python Programmation C Linux Programmation de bases de données

Nº du projet : #28642015

À propos du projet