Build Hadoop/Hive System

We want to find a partner to help us get started in building a big data / datawarehousing system in Hadoop and Hive (or suggested alternative) to run alongside our operational system. This new big data store will provide api, reporting and datawarehousing functions, the latter to drive a tool like Tableau. This data store will then develop to receive streamed and historical batch data and generate metrics from map-reduce calls.

We are a real time vehicle tracking solutions provider collecting information about vehicle positions and their adherence to scheduled journeys.

We are new to the Hadoop world and want to get up to speed ourselves during this development. Therefore we want a full end to end development including initial system setup, development of a data loading/streaming method and provision of a small set of data output methods (RESTful API calls, reports and an initial datawarehouse structure ).

We envisage the following tasks :-

1. Deploy Hadoop/Hive (or suggested alternative) on a virtual Debian server which we will provide access to. We want this performed in such a way that it can easily be expanded into new nodes and would want to see data distributed across more than 1 system/node.

2. Develop a data load process to pull information from our transactional system and load into Hadoop/Hive. This initial data set will be a block of data per day containing a couple of metrics but with quite a few decriptive fields as a vehicle code, location_code. Timestamp, customer code. If this was a traditional star schema then there would be about dimensions. This data has a time based aspect and has a geospatial aspect. We would be able to provide this in Csv format or from an API call. In

3. Develop some map reduce functions to generate some useful metrics and agregations which we can agree.

4. Make some of the metrics available using the existing HADOOP/Hive/RESTful technologies in order to provide an API.

5. It would be nice from us to access the datastore from PHP using perhaps a Hive/ODBC driver not sure if this is possible but it would be good to try this.

6. Organize the data so that an OLAP tool can be used against it. For example, use Pentaho or Tableau to generate some queries to be able to pivot ad drill down. Especially important is to be able to show aggregate data for say a year and drill into month, day etc. Also would be good to be able to show geographical data.

We are interested in working with someone who can recommend the paths to take to make this system expandible, fast and easily accessible and to help us make the best choices. For example help in deciding which database to use would be welcome.

Please bid only if yu have experience of this in the past. If you interested in bidding for this week we would like to hear about your experience in similar projects and your views on whether this is a sensible approach.

Compétences : Big Data Sales, Stockage de données, Hadoop, Hive, PHP

en voir plus : restful find, real time example for data structure, real time data solutions, pentaho 5, olap database example, olap database, node data structure, node csv, full time positions available, example olap, data structure node, c map data structure, big data development php, best olap database, hadoop email system, groovy hadoop hive, build drupal theme existing theme, build mobile website existing website, make logo existing joomla template, make duplicate existing site web, integrate phpbb3 existing login system, build content management system newspaper website using dreamwaever, using nutch hadoop website, nutch hadoop help

Concernant l'employeur :
( 4 commentaires ) Crawley, United Kingdom

Nº du projet : #8477793

Décerné à:


Dear Client, We, Fragma Data Systems, are a Big Data Consulting and Solutions startup based out of Bangalore. Our team comprises of Big Data Architects who have more than 10+yrs hands on experience in designing and Plus

%selectedBids___i_sum_sub_7% %project_currencyDetails_sign_sub_8% GBP en 12 jours
(0 Commentaires)

40 freelance font une offre moyenne de £2693 pour ce travail


Let's discuss over freelancer Personal Message Box for the proper estimation of cost and time. I am myself developer so you will directly work with me. No mediators. No managers. No subcontractors. see my recent Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(121 Commentaires)

Hi! I am TOP 6th freelancer. I am interest your project and hope to work with you. I have enough skills to carry out any sort of work load. I will complete your project perfectly in your deadline. Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(278 Commentaires)

1. FeedbackFive Clone Amazone MWS API's In these days we are near to finalize another API's based feedback management system for sellers on [login to view URL] by using the MWS API's of amazon whose details are currently confi Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 45 jours
(97 Commentaires)

Hello Sri Technocrat will provide fully interactive website for your project. As per the detail, Sri Technocrat will provide three template functional schemes and sample pages to make your choice for layout. It will Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(62 Commentaires)

Hi, Mark here, I would be interested in discussing this project with you. Thanks for the consideration, I hope to hear from you soon Thanks, Mark

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 35 jours
(21 Commentaires)

A proposal has not yet been provided

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(118 Commentaires)

Hi, We are professional web development company. We have team of web designer and developer who developed 100+ web site with different technology. Our developer and designer will give you satisfactory solution bas Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(164 Commentaires)

Hello There, Greetings From Synapse! Hope this finds you well. Coming straight to point: Firstly, I would like to share that we are having our own Cloud foundry. We have Hadoop Experts in our team and they have Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(11 Commentaires)

Hi, Yes,i can help you to build a big data / datawarehousing system in Hadoop and Hive to be more modern as you [login to view URL] would be great if you can give me some time to discuss this project. We offer custom web s Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 55 jours
(46 Commentaires)

Dear Hiring Manager, Greetings !! I hope your day is going well and all is good with you. We would like to discuss the project in details before confirming the bid, so kindly let me know when you are available. Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(50 Commentaires)

Dear Sir, [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] ht Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 300 jours
(61 Commentaires)

Hello, My experiences with performance testing / tuning of Mapreduce or Hive jobs before 01) "For a Company with innovative medical solutions" My Job Functions: Developed MapReduce programs to parse the raw da Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(1 Évaluation)

Hello! We are a US company with a 13 year history of developing cloud-based custom software solutions for a diverse client list. Our rapid development platform is able to save our clients significant time and money Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(1 Évaluation)

Dear Client, Hope you are doing well..!! I have carefully gone through your posted job description and understood your requirements (mentioned tasks) very well. I will do the complete development to make t Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(22 Commentaires)

Hello, I will be honest with you. I run a company which is into data warehousing solutions. We have worked with various clients across US in building DW and Hadoop systems. You can visit thinklayer[dot]com for more det Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(73 Commentaires)

Hello, my name is Olya. I represent Ukrainian IT Company «Webbook». We provide website design and web development services for organizations, public and government institutions, company or private web-pages. We got Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(11 Commentaires)

US PROVIDER - NO PAYMENT RELEASE TILL PROJECT IS FINISHED (for projects estimated to complete within 20 days or less): Hi! I’m Nick from MadGrizly a US based Digital Studio specializing in Web and Mobile innovative de Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(12 Commentaires)

1. Setup and configure a Hadoop cluster on Debian. ETL the batch data to HDFS using Pentaho DI or SSIS 2. Map reduce the files to create aggregates. Store the aggregates in Hive tables. Optionally we can store them Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 45 jours
(4 Commentaires)

Hi, I have read your requirement and i am too much interested in this job ,I have an expert team , Only 10 people and working since 5 years and developed many websites , web applications e commerce websites , plugin d Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 45 jours
(13 Commentaires)

Hi, I am a senior developer with the skills that you need for the job. I can partner with you in this project. I hope we can talk better soon. Best regards, Norberto

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 30 jours
(23 Commentaires)