Terminé

AWS S3 text files to Relational Database

A little background - I am using AWS to manage a mountain bike timing system. Competitors tag on and off a number of different stages and the tag off time minus the tag on time is the stage racetime and the lowest cumulative time over all stages wins.

Out on the racecourse, I am using Raspberry Pi's with RFID readers which collect and store data in text files which automatically upload to the AWS S3 bucket. I currently have two files in there: Stage 1 [login to view URL] and Stage 1 Finish.txt.

I need to get these files into a database to join with a competitor registration database so I can ultimately compute racetimes in AWS, extract some analytics and allow competitors to search and compare times using their surname or racing number to find their race times.

In short, I need help converting these text files into a relational database and this process is automated so that when new items are appended to the text files that they update the database.

I think a Data Pipeline is the way to go but happy to listen to alternative recommendations to achieve my ultimate goals as described above.

The text file item/row is semi-colon delimited as follows:

[login to view URL];1s;2016/12/30;13:56:23.235;00235689

Meaning log information (not needed), 1s (Stage 1 Start), Date, Time to milliseconds and finally the RFID code.

Any help with the Data Pipeline (or another alternative) is required as soon as possible.

Thanks

Steve

Compétences : Services Web d'Amazon, Administration de Bases de données, MySQL

en voir plus : translate audio files text, auduio files text, exporting ning database files, relational database video rent, psd examples files text, relational database project, transcribe sound files text, project database files mega retail store, type picture files text money, perl search files text, php mysql relational database frontend, transcribe audio files text, mysql database hold text files, create html files text files, audio files text

Concernant l'employeur :
( 12 commentaires ) Mandurah, Australia

Nº du projet : #12676953

Décerné à:

jksnetwork

I am highly skilled in the AWS environment. I have been working with AWS for 5 years now and run multiple production workloads across many AWS services. I have both the AWS Devops Engineer and AWS Certified Solutions Plus

%selectedBids___i_sum_sub_7% %project_currencyDetails_sign_sub_8% AUD en 3 jours
(23 Commentaires)
5.0

18 freelance font une offre moyenne de $172 pour ce travail

poornachand

I have 9 years experience in JAVA, J2EE, WebServices and enough experience in AWS webservices. I am not sure if we can dine the pipeline such that it extracts te file and fills the required columns of a relational dat Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 5 jours
(25 Commentaires)
5.5
mak94pk

Hi, Firstly, Thank you for reviewing my proposal. I have gone through your requirements and came to know that you need an expert web developer to build your website. I have strong knowledge and 3+ years experience i Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 3 jours
(14 Commentaires)
4.8
talkwaqar

Please come online for detail. My Skype Id is [login to view URL] I am a highly skilled senior developer with extensive development experience in system design, .NET technology stack including C#, ASP.NET, MVC, Web Api, SQL Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 3 jours
(15 Commentaires)
4.5
bhawin91

I can write a php script which can load data from text file to database. I would love ot work on something new and good thing ..

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 3 jours
(14 Commentaires)
3.9
whites11

Hello there. I think lambda could be the solution for you. I can take care of this task if you wish I am an experienced developer with strong aws knowledge

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 3 jours
(1 Évaluation)
3.7
DevvTime

Hello there i have read your description, i will recommend a web application for this, can you please contact to discuss in details? i am waiting for your pm thanks

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 3 jours
(7 Commentaires)
3.5
%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 3 jours
(3 Commentaires)
3.6
enterpriseit1

Hi, I have over 7 years of experience as a database developer. I have completed three recent projects. All clients were 100% satisfied as you can see from their reviews. One of the projects I completed recently was Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 3 jours
(7 Commentaires)
3.3
titukhan13

sir,this is easy task for me.i will complete your work as soon as possible with out any [login to view URL] give me the chance to work with [login to view URL] for your reply. Thanks

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% AUD en 1 jour
(5 Commentaires)
3.8
%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 3 jours
(8 Commentaires)
3.5
ahmad5857

Hi sir, We assure you the quality of service through various checks that are employed to avoid any compromise on quality of work. Let me introduce my team to you and generally what services we offer to our client Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 8 jours
(1 Évaluation)
3.2
explore1

i have similar experience working with S3 and mysql db.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 4 jours
(2 Commentaires)
2.6
%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 3 jours
(0 Commentaires)
0.0
jupitor

you don't have to load your data to different database unless you have compelling reasons. Now amazon athena supports query the s3 data like sql queries. Much less expensive and less maintenance than using different da Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 3 jours
(0 Commentaires)
0.0
IonRod

Hello! I can do the job u need. I think Python is the best solution for you as it's covering all your needs. On the other hand maybe it makes sense to use logstash to transform and send data to your database and do Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 10 jours
(0 Commentaires)
0.0
ebinthomas

Have 4.5 years of experience in Java, REST, Data Structures and Algorithms, SQL and NoSQL(MongoDB, Redis), Spring, Hibernate hence I will be a good help for this project. I had recently worked with Rajhees, C.E.O, MD , Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 3 jours
(0 Commentaires)
0.0
%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% AUD en 5 jours
(0 Commentaires)
0.0