Speed optimization of Big (Flow) Data Processing (using JAVA or Python or C programming Language)

Essential Technology

- Programming Language: JAVA or Python or C programming

- Required Experience: Network Big Data Processing

- Open Source Management System: InfluxDB, MariaDB, MongoDB (You can choose one of them for your database But, InfluxDB is preferred)

- Library: nfcapd & nfdump ( [login to view URL] ),

- Protocol: netflow, jflow, netstream, sflow

- OS: Linux, CentOS7

- Server requirement:

 Basically, an assigned developer must build own server and need a confirmation of OS version, programming language version, and DB version with Next-core Technology.

 Specification of our server being currently used is below. This specification may be improved. Thus, if an assigned developer has a suggestion regarding improvement (optimization) in server’s specification, we are open to consider the suggestion.

CPU: Intel® Xeon® CPU 2.3 GHz

2P24C/MEM: 128G/HDD: [login to view URL]

Development Period: 3 weeks (=15 business days but negotiable before the project begins)

Inspection Period: 4 business days (included in development period)

Development Cost: 1st requested way: USD $ 1,840 / 2nd requested way: USD $ need discussion

Project Objective: To optimize (or minimize) speed (from 5-7mins to within 2 mins) of present Big (Flow) Data processing (2 ways: 1) csv file to DB & 2) binary file to DB) via MariaDB or MongoDB, or InfluxDB using JAVA or Python or C programming

Project Summary

During data processing for about 5 millions of flow data, we are seeking to optimizing (or minimizing) current speed (5~7 minutes) of data processing from Server and database. The current speed must be within 2 minutes. Please see attachment called ""[login to view URL]" for your understanding.

As you see the workflow and the project objective above, our project is to optimize current data processing between Server and Database into 2 mins or less. When the objective is accomplished, Next-core Technology must retrieve source codes and other applicable references for the objective. Next are information of sample data and detailed course instructions in our project.

Data Given

1. Meta data (Field information) and sample data masked (1.5GB csv file) will be provided. We just attached data field information and examples of sample data first. Later, we will provide masked data. Please see the attached file for your development.

2. For this project, 5 millions of flow data will be standardized amounts. As you know, our objective is to make duration time of data processing between Server and Database into 2 minutes or less. Thus, we must make this 2 minutes or less with 5 million of flow data. To be specific, if you run 2.5 million data, your speed of data processing must be within 1 minute.

3. All masked data given will be flow data so that you should be familiar with their characteristics. For the 1st way (csv file to DB), you can refer to field information PDF file and csv file for your understanding. For the 2nd way (binary file to DB), we will discuss about which data to use after 1st way is figured out.

Please read carefully detailed courses of the development by opening the attached file called "Project [login to view URL]" before you approach to us. This guides you how to complete the project's objective.

If you have any other question, please let us know via chat @ Freelancer.

Compétences : Java, Python, Programmation C, Linux, Programmation de bases de données

En voir plus : data extraction using java, projects data processing using visual basic, work flow component created using java tibco, download option price data yahoo using java, market research data processing using quantum, image processing using java, projects data structures using java, video frame processing using java, online quickbooks data read using java code, extract data website using java, aggregate data websites using java, extract data site using java, data processing using excel pdf, classification data mining using java, vehicle number plate identification using image processing using java or c platform, vehicle number plate identification using image processing using java or c++ platform, big data processing using hadoop, big data analytics using java

Concernant l'employeur :
( 5 commentaires ) Seoul, Korea, Republic of

Nº du projet : #28642015

13 freelances font une offre moyenne de 2498 $ pour ce travail

(27 Commentaires)
(81 Commentaires)

Hi, I am interested, gone through the docs. Hire me /////////////////////////////////////////////////////////////

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 21 jours
(11 Commentaires)

Hi I have experience in big data processing and so am very interested in your job. I'll complete your job PERFECTLY. I am ready to start right now. Best Regards.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 4 jours
(8 Commentaires)
(3 Commentaires)

Hi My name is Monil Patel and I am Senior Systems Programmer. I have experience creating script and automates processes like this for over 5 years. I have a very strong background in software development and console a Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(1 Évaluation)

Hi, I will customize application using the Oracle intelligent advisor(Oracle policy automation) and sort the data and offer the fast data processing for [login to view URL] let me know do i need to fill the form as per the repl Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 30 jours
(3 Commentaires)

Hi Sir, I can make your website so fast than many websites. I am full-stack developer and expert in Wordpress, HTML, CSS, Web design, Graphic design, MySQL and PHP I have more than 7 years of experience in develop Plus

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% USD en 1 jour
(0 Commentaires)

I have read your requirements, i am an experienced Wordpress developer and expert Wordpress website speed optimization, I can get score of 85+ to 90 btw with average loading time of 3s to 2s on gtmetrix or google page Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 3 jours
(0 Commentaires)

Hello,your project is very motivated for me. As a senior developer 8+ years of experience python and Software Architecture.I have great experience to work on API,s and full stack development. It would be amazing experi Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 15 jours
(0 Commentaires)

Narinder Alliance Technologies LLC An IT Consulting and Software Development company. We have a team expert in Web Designing, Application Development and Databases. Major offerings include Digital Transformation, Appl Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 30 jours
(1 Évaluation)

Kindest regards from the viitorcloud!! Much appreciated for the great opportunity. We are innovation driven IT development and consulting services company. Inline notes about us: Serving Since: 10+ Years Team Size Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 30 jours
(0 Commentaires)

Hi, We are a team of data science and ML/AI experts who excel across multiple areas with more than twenty five years of combined experience. We hold expertise in Python, Backend Architecture (micro-services, Kubernete Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 25 jours
(0 Commentaires)