Federal Agency Rule Reports -- Data Scraping

I need data on United States federal agency Major Rule Reports scraped from the US Government Accountability Office Congressional Review Act website ([login to view URL]) and put into a spreadsheet. This would require a search of the Major Rules Reports for 113 agencies and recording information from each report. There are about 1,150 reports for the 113 agencies. Each report in the database will be one row (i.e. 1,500 rows). The columns with data for each entry should include data on Agency, sub-agency, subject line of the letter to committees, date of the letter, the committees to which the letter was addressed, the RIN as reported in the enclosure, and the statutory authorization for the rule as reported in the enclosure. The scraping is somewhat involved. We will provide a list of agencies. A search on the GAO site above will have to be conducted for each agency on the list. The searches should be for the exact agency name on the list surrounded by quotation marks (e.g., “Department of Agriculture”). Otherwise the searches will pick up reports that are not associated with that agency. This means the scraper will have to go to the website above and click on the “Search Major Rules Reports” link or go directly to: [login to view URL]:2:{s:4:%22site%22;s:12:%22Publications%22;s:7:%22subsite%22;s:25:%22Federal+Agency+Major+Rule%22;}. The searches will produce anywhere from 0 to about 100 reports per agency. Each report will have to be clicked on and the scraper will have to go to the “View Decision” tab for each report. This will lead the scraper to a letter from the GAO to two or more committees in the U.S. Congress (in most cases only two). These letters will include all of the information I need for each report. The agency name is self explanatory. The sub-agency, if there is one, will be listed after the comma after the agency name in the title or in the “Subject” line of the letter. The subject line is after the addresses in the letter. It begins with the word “Subject.” The date of the letter is at the top of the letter. The committees are listed as part of the address of the letter at the top. They begin with the word “Committee” (e.g., Committee on Agriculture, Nutrition, and Forestry). There are normally two committees listed. On rare occasions there may be a third or fourth. There should be one column for the first committee listed, one for the second, one for the third, etc. It probably makes sense to have the columns for committees at the end of the spreadsheet. The RIN number is usually included in a couple of places in each letter, often in the first paragraph of the letter and later as part of the letter called “ENCLOSURE”. It is normally in parentheses in the following format: (RIN: 0560-AH92). The statutory authorization for each rule is listed after a section of the enclosure called “Statutory authorization for the rule.” The language usually says something like “The final rule is authorized by the ADA Amendments Act of 2008, Pub. L. No. 110-325.” We do not need the “The final rule is authorized by the.” If possible, we only need “ADA Amendments Act of 2008, Pub. L. No. 110-325.”