Data collection and connection

Hi, I'm very interested in working on your project. I have experience working with data sets of this size from previous projects, and I am able to deliver quickly. The first task I would solve by fetching the list from [login to view URL]<USERNAME>/<PASSWORD>/ and then parsing that using standard Linux CLI tools to generate the text file. I estimate around 4 hours for this task. In general I would be using a mix of Python and Linux CLI tools, and if needed, PostgreSQL for storage. But I am also flexible regarding specific tools and languages, these are just my personal preferences. Hope to hear from you, Mathias M. Hansen

hi there, I just checked your proposal. I am Senior PHP and Laravel Developer with 9+ years strong practical experience in building various kinds of web projects. Especially I have good experience with PHP, Laravel, CI, Cakephp, Mysql, Linux and familiar with several JS frameworks such as Angular.js, React.js and Vue.js. I developed number of projects so far and I can provide perfect results to you. How about work with me ? Please note that I am top 6th developer at freelancer.com with around 600 reviews. My ultimate focus is customer and team satisfaction by "shipping" the best work possible in as timely a manner as possible. I can guarantee high quality of code and high availability as I worked at home. I want to build trustworthy, reasonable contract with you. Best Regards, Heng

Hi plusserver, I have gone through your project "Data collection and connection" and could assure you the best of the solution. I am taking it as an opportunity and ready to deliver my best. please check our freelancer portfolio Also do have similar experience doing something similar to yours therefore I am looking forward to discuss and move ahead. Relevant Skills and Experience I have 10+ years of experience in Linux, MySQL, PHP, Python, Software Architecture I can do this within your budget too €18 - €36 EUR / hour Look forward to your response Thanks Mmadi We are looking for a freelancer that can support us in requesting huge datasets (up to 500 Mio lines) from APIs, enrich

Hello There, Greeting of the day! As per your project requirement, would be able to scrap data from a database. I can use PHP that would be suitable for Linux. Technically I would be able to resolve the issue and extarct all data in the format domain | tld in a text file. I need to run a custom script for it. I have been working as developer last 9 yrs. In My past, I have worked in India based MNC, Techmahindra Ltd. and also got an opportunity to work in el Segundo California and San Jose,USA onsite client location for development support. I was working with British Telecom on client support. I have extensive experience as the Full-stack developer. This is Ambikesh having experience of 8+ years in Software and web development. I have extensive experience in , PHP/WordPress,Joomla,Magento,Cake php, codeigniter,Laravel,Yii2.0,,HTML/HTML5,Bootstrap3.0,CSS3,Angular.JS, JQuery, SQL, My SQL, REST API's and SOAP API's I value consistent and continuous communication with you to understand your project concept and our experts do EXACTLY what you have envisioned. So it’s time to turn your ideas into reality with the help of experienced developer. So let’s get started on your dream project. We look forward to hearing from you. Thanks, Best Regards! Ambikesh

Hello, After reading your project details I believe I'm suitable for this project. As I'm expert on this kind of project with more than 5 years experience. Please feel free to contact me. I am looking forward to hear from you. Best regards

Hi, We can use Python for grabing data from API, and use mongo DB as destination, the combination give high performance in large data management tasks, it also run in linux server we are an IT company from India, more than 10 years experience in ndutry, come in chat we can discuss more

I have good team for website design and development, I am terribly excited for the chance to figure with you in accomplishing your goals. We have developed thousands of websites in many programming languages for clients all over the world. plusserver might you be able to please inbox me more insight about this venture

Hello! This is REST API, so there can't any special optimizations. I think the best choice will be using a simple map-reduce algorithm. So basic idea will be: 1. Create multiple threads. In this kind of task, it should be the same as hardware cores. 2. Create multiple async requests for API per each thread 3. Save it to file/database. Usually, the longest part is network communications. Especially, infinity reconnecting for each request, because REST API has following architecture: Connect --> Send Request --> Read response --> Disconnect. IP recognizing will be much faster. Because you can make a request without disconnecting. Directly to google DNS service. But if you want a REALLY fast solution then we should discuss the cluster system. Where you can use multiple PC for API tasks. About technology. The task doesn't require any "hard" tasks, except filesystem writing. So python is the best choice, maybe rust also too.

Hi I have have done it before , Kindly connect with your requirement. I have years of experience and expertise in a gamut of technologies that include: • Php, Magento, Laravel, Wordpress, Android & ios etc • Web (Html, Laravel & Core Php, Wodpress, Joomla) • Linux, MySQL, PHP, Python, Software Architecture Let’s connect and kick start the project !!! For any query, feel free to connect me.

Hi. Pro account is needed for api access (50/month). I do this kind of stuff on aws so it depends on you how much are you willing to pay for bandwidth, storage, etc.

Hi, as an expert in software development, I am very interested in your project since I have rich experiences in database management. Almost all of apps use its own database to store data which is needed in the program. Through my programming history I found that Oracle database is very effective in huge database management system. Since the database contains huge number of lines, selection of database engine is not a perfect solution. The DB design is also very important to reduce the duplication. If you hire me, I will do my best to design the database using with 3NF (this is the recommended database normal form). Also, in my opinion, both Oracle and MySQL can be used. Let's talk about the project in more details. I am looking forward your response. See you later. Thanks.

I am an Web Scrapper doing for many websites as freelancer. delivering into excel database through spider or phyton i can do your project.