Terminé

Bash script to read text values in files recursively and prepare a new file ready for MySQL

We have a series of text files which contain lists of log file information in this format ->

value1:value2

value1:value2

There is no delimitation around the fields other than the full colon ":" separating the 2 values and obviously a new line at the end of the 2nd value. There are however some errors in these files, some of the lines don't follow the correct format above and we need these removing otherwise we wont be able to use LOAD DATA into MySQL when we come to import them.

These files are stored in multiple nested directories, some of which are 5 or 6 levels deep and each directory can contain between 50 and 200 text files.

We are looking for someone to write a simple BASH script to run on ubuntu 16.04 in the root directory of all these files. The script will traverse its way down each file and directory and cleanse the data. The script will need to open each file, read it one line at a time and check that each line conforms to the above format. If it is in the correct format, add it to a new "cleansed data" text file in the same (value1:value2) format in the current directory being checked. When its finished reading all the files in that current directory, it will save the cleansed data text file and move onto the next dorectory.

We have 98 directories of this data and hundreds of files within them. The output of the script will be 98 "cleansed" text files, one in each directory containing all the correctly formatted lines from all the files in that directory.

We'll then use each new cleansed data file with the LOAD DATA facility to upload the new value1:value2 checked pairs.

Compétences : Linux, MySQL, Script Shell

en voir plus : shell script to read a file line by line and write to another file, how to read a file line by line in unix using for loop, shell script to read a file line by line in unix, bash while read, shell script to read data from text file, bash read file line by line, bash read first line of file, bash script read file into variable, script unzip files recursively, script read files, script read text pdf, script read text file, bash script read logs, bash script read columns spreadsheet, script read text sentence, vba script read text file, script read text files, vbs script read text file export html file, lotus script read text files, script find text xml files

Concernant l'employeur :
( 1 commentaire ) Lincoln, United Kingdom

Nº du projet : #18159808

Décerné à:

ukgolang

HI, I have ten years commercial Linux experience as a systems administrator. I can write a script to filter your files. Hope to speak with you soon.

%selectedBids___i_sum_sub_4% %project_currencyDetails_sign_sub_5% GBP en 1 jour
(14 Commentaires)
4.5

11 freelance font une offre moyenne de £57 pour ce travail

kaloyan13

Bash is one of my best skills. I use it every day. All I would need is a few sample files, and some nodes about data types in the file.

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% GBP en 1 jour
(245 Commentaires)
6.6
jbosseu

I have worked as server admin for big names like [login to view URL] and servercentre.net. Over 8yrs of working with cpanel, directadmin and plesk servers. Can help you with this. Let me know.

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% GBP en 1 jour
(124 Commentaires)
5.7
TakaAlex

Hello I can do it immediately in python Thanks ...............................................................

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% GBP en 1 jour
(2 Commentaires)
4.3
simrankaurc

Hello Sir, I am agile safe 4.0 certified I have 7 year of experience in Software development including technologies java, j2ee, spring , hibernate, ,Rest apis, php unix shell scripts, apache tomcat, jboss, server co Plus

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% GBP en 1 jour
(3 Commentaires)
3.6
stanislav103

Hi, there. I am happy to apply in your project. Your project is interesting to me. An experienced Bash/Shell script expert here. If you want to review me, you can go through my feedbacks and git repos. https://git Plus

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% GBP en 1 jour
(7 Commentaires)
2.8
glidelabs

Hi, I can write a python script for this that will full fill the purpose. As you will require data cleanse, so I feel no perfect script can do. There will always scenarios that are not present in current data but can b Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 2 jours
(3 Commentaires)
1.6
otrix

Hello, I'm interested in the project. Right now I have a small web development company with 2-3 web developers. We are currently working on the back-end of our website and when we fully launch our front-end will be com Plus

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% GBP en 1 jour
(0 Commentaires)
0.0
Tristan95

Hi, I wrote many such nifty content aware scripts. This is doable with shellscript, ruby, perl and almost any language. I will choose the one that will get the job done best. I hope you decide for me and be amazed!

%bids___i_sum_sub_32% %project_currencyDetails_sign_sub_33% GBP en 1 jour
(0 Commentaires)
0.0
osamaalvi

6 Years of Experience, Expert in all kind of applications, Laravel 5.1, 5.5, Spring Mysql, SQL Server.

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 2 jours
(0 Commentaires)
0.0
bouwerp

As your description of what the script should do sounds quite complete, as I understand it, the script should: - take as a parameter the root directory of where the log files are - recursively traverse the entire tree Plus

%bids___i_sum_sub_35% %project_currencyDetails_sign_sub_36% GBP en 2 jours
(0 Commentaires)
0.0