Fermé

Improvement of Existing Project (Ruby, Golang, JS, PostgreSQL, ClickHouse)

We have some improvement plan for our old website/metrics collector (written in Ruby for the metrics collector, Go backend, standard JS-jQuery for frontend, PostgreSQL for databases)

The existing tables:

- users

- groups

- stations

- station_logs_x --> metrics per station (splitted in the past from station_logs table)

- station_hours_x --> metrics per station (splitted in the past from station_minutes table)

- station_minutes_x --> metrics per station (splitted in the past from station_hours table)

attachments are existing code examples:

- [login to view URL] -> model

- [login to view URL] -> old collector

- [login to view URL] -> controller

- *.html -> view

The plan was, adding a new type of station but with same/similar metrics, rewrite tcpserver using golang, and migrate station_* table to ClickHouse.

priorities for improvements are:

1. Create a UI for newly MQTT inputs

database columns still the same, the MQTT server already implemented in [login to view URL] (it still inserts to stations, station_logs_x, station_hours_x, station_minutes_x.

see [login to view URL]

subtasks:

1.1 add a flag on stations that mark that this station have different UI than others, if flag present, show this UI, else use old graph UI, the metadata can use something like this (stored as one kv in data column in stations table): [

{"label":...,"sensor_column":"accel_x","color":"#xxxxxx"},

{"label":...,"sensor_column":"accel_y","color":"#xxxxxx"},

{"label":...,"sensor_column":"accel_z","color":"#xxxxxx"},

]

1.2 implement new UI

2. Fix the bug in previous [login to view URL] changes [new imei received doesn't automatically create logs_x/hours_x/minutes_x tables], by rewriting into Go

subtasks:

2.1 rewrite [login to view URL] to [login to view URL], add the creating a new table when station never exists before

2.2 write a unit test (connect, disconnect, receive, response) to make sure it works properly without memory leaks (not holding disconnected socket)

3. Migrate some part of database to ClickHouse (see previous code [login to view URL])

change all queries related to logs/minutes/hours from PostgreSQL to ClickHouse

for the hours and minutes, no need to create trigger anymore, ClickHouse materialized view is automatically updated from source table.

the purpose of this action is that ClickHouse can save space (88 million records = 600MB in clickhouse, vs 19GB in normal RDBMS), have really fast insert performance (600k recs/s vs normal rdbms that only 1500 recs/s) so it would be faster to backup, also it should be faster for retrieval

subtasks:

3.1 write a code to initialize ClickHouse tables when service is up and when new station inserted, this also solves problem 2.1

3.2 add a code to double write to ClickHouse (not just to PostgreSQL), with 2 feature flag/toggle: 1 for disabling writing to the PostgreSQL, 1 for enabling writing to ClickHouse. Replace the old code that writes/queries to stations, station_hours, station_minus with ClickHouse version.

3.3 write a code to migrate data from PostgreSQL to ClickHouse (called after 3,1, make it resumable, if you want to upsert use ReplacingMergeTree, but if not, use other table engine)

3.4 write a script to backup ClickHouse and Restore, test the toggle feature flag by disabling PostgreSQL or enabling ClickHouse

If you are interested and want to do this job, please reply with estimated number of hours you think you will need to solve each subtask you can and the bid for each subtasks/milestone

1.1 db flag for ui: xxx hours, xxx $

1.2 implement new graph: xxx hours, xxx $

2.1 rewrite [login to view URL]: xxx hours, xxx $

2.2 unit test tcpserver: xxx hours, xxx $

3.1 table initializer: xxx hours, xxx $

3.2 toggle and double write: xxx hours, xxx $

3.3 migrator: xxx hours, xxx $

3.4 backup/restore script: xxx hours, xxx $

total: xxx hours, xxx $

note, you must bid/send proposal with formats above (replace the xxx with your estimation)

there's a previous project manager/developer that could guide you to learn the previous source code.

Compétences : Golang, PostgreSQL, JavaScript, Ruby, Linux

en voir plus : integrating wordpress existing project, dunamic web site existing project vbnet, flex php existing project, rails upgrade existing project version 123, aspnet adding silverlight video existing project, upgrade rails existing project, microsoft access existing project, crystal reports add existing project vs05, existing project best game puzzle, converting existing project maven build, change rails version existing project, integrate substruct existing project ror, updating rails existing project, add vue js to an existing project, swagger node js existing project, add next js to existing project

Concernant l'employeur :
( 20 commentaires ) Surabaya, Indonesia

Nº du projet : #29894701

6 freelances font une offre moyenne de 583 $ pour ce travail

techplusintl

Hi there, ★★★ Ruby on Rails (ROR) Expert ★★★ 10+ Years of Experience ★★★ I've read requirements and ready to work on your project. Here's estimate that you require: 1.1 db flag for ui: 11 hours, 110 $ 1.2 implement n Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(66 Commentaires)
6.9
sapnathakur14

Hello, Warm Greetings!!     I read your job description completely and I found my skills relevant to your project. I am a professional web developer with 5 years of programming experience for a multitude of different t Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(19 Commentaires)
6.5
einnovention

Hi there! I have read your project of improvement of existing project. I am a top freelancer and I will do it in a professional way! WAITING FOR YOU IN CHAT BOX! Wajid

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(21 Commentaires)
5.8
aleksandrskvor

Hi. I am senior web designer with 6+ years experience. I am very interested in this project(Improvement of Existing Project (Ruby, Golang, JS, PostgreSQL, ClickHouse)) and want to discuss with you about the project det Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 5 jours
(3 Commentaires)
3.3
nizamfarhas

Hi there I just noticed the project you posted (Improvement of Existing Project (Ruby, Golang, JS, PostgreSQL, ClickHouse)) and very interested in job details. I can give you the perfect result with my skills and dedi Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(7 Commentaires)
4.4
andersonfree231

Hello, Thank you for your care. I've just checked your job description carefully. I am a Full-Stack developer of PHP with 7+ years experiences with web developing. Javascript and PHP are my major and all Javascript, PH Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% USD en 7 jours
(2 Commentaires)
2.6