SEC API requires User Agent, please help

[nora-census]

Hello,
This is my first time posting an issue on GitHub, please excuse the formatting.

I use the 'edgar' R package daily for my job. Recently the package's functions have not been working. After reviewing the SEC developers website and various online forums, it appears the SEC now requires users to include their User Agent (e.g. email address) to successfully make API calls. The functions in the 'edgar' package did at one point include the User Agent as a parameter, but currently do not. Is there any way you can please re-introduce the User Agent parameter back into these functions so the API calls will successfully complete?
Thank you!

[grtax1gh]

Hello,
I'm also facing the same issue. Did you find a solution for this topic?

[Gunratan]

Please make sure to install the current version "2.0.7" of edgar package. It still works and does not need a user agent.

[reillysteel]

I have the latest version (2.0.7) installed, and I'm also having this issue, even when just trying to call getMasterIndex(). The downloaded year.master dataframe gives me no filings and instead says "Your Request Originates from an Undeclared Automated Tool" and "Request Rate Threshold Exceeded," among other things.

I'm guessing that the SEC wants a specific user agent declared (see this link, https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data, which asks for the user agent to be declared in a specific way), and the way the package is doing it isn't cutting it. Looking the code for the getMasterIndex() function from CRAN, I see you're tapping into httr's GET() function and specifying the user agent as "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0". If you made UA a parameter in the getMasterIndex() function so that users can specify their own (e.g., "Your Name email@domain.com" per the SEC's guidelines at the earlier link), maybe that would work?

[reillysteel]

I have the latest version (2.0.7) installed, and I'm also having this issue, even when just trying to call getMasterIndex(). The downloaded year.master dataframe gives me no filings and instead says "Your Request Originates from an Undeclared Automated Tool" and "Request Rate Threshold Exceeded," among other things.

I'm guessing that the SEC wants a specific user agent declared (see this link, https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data, which asks for the user agent to be declared in a specific way), and the way the package is doing it isn't cutting it. Looking the code for the getMasterIndex() function from CRAN, I see you're tapping into httr's GET() function and specifying the user agent as "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0". If you made UA a parameter in the getMasterIndex() function so that users can specify their own (e.g., "Your Name email@domain.com" per the SEC's guidelines at the earlier link), maybe that would work?

I've fixed the problem in the R code for getMasterIndex() by changing the "UA" value to what the SEC asks for (name and email) and adding headers for Accept-Encoding and Host:

UA <- "John Smith jsmith@myuniversity.edu"

[...]

r <- httr::GET(link, 
                     httr::add_headers(`Connection` = "keep-alive", `User-Agent` = UA,
                                       `Accept-Encoding` = "deflate, gzip",
                                       `Host` = "www.sec.gov"),
                     httr::write_disk(dfile, overwrite=TRUE)

It would be great if the package on CRAN could be modified along these lines and with the useragent argument added back in.

[nora-census]

[heart] Nora Trow Shaw (CENSUS/EID FED) reacted to your message:

…

________________________________ From: reillysteel ***@***.***> Sent: Friday, January 17, 2025 9:59:06 PM To: Gunratan/edgar ***@***.***> Cc: Nora Trow Shaw (CENSUS/EID FED) ***@***.***>; Author ***@***.***> Subject: Re: [Gunratan/edgar] SEC API requires User Agent, please help (Issue #10) You don't often get email from ***@***.*** Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> I have the latest version (2.0.7) installed, and I'm also having this issue, even when just trying to call getMasterIndex(). The downloaded year.master dataframe gives me no filings and instead says "Your Request Originates from an Undeclared Automated Tool" and "Request Rate Threshold Exceeded," among other things. I'm guessing that the SEC wants a specific user agent declared (see this link, https://www.sec.gov/search-filings/edgar-search-assistance/accessing-edgar-data, which asks for the user agent to be declared in a specific way), and the way the package is doing it isn't cutting it. Looking the code for the getMasterIndex() function from CRAN, I see you're tapping into httr's GET() function and specifying the user agent as "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:93.0) Gecko/20100101 Firefox/93.0". If you made UA a parameter in the getMasterIndex() function so that users can specify their own (e.g., "Your Name ***@***.******@***.***>" per the SEC's guidelines at the earlier link), maybe that would work? I've fixed the problem in the R code for getMasterIndex() by changing the "UA" value to what the SEC asks for (name and email) and adding headers for Accept-Encoding and Host: UA <- "John Smith ***@***.***" [...] r <- httr::GET(link, httr::add_headers(`Connection` = "keep-alive", `User-Agent` = UA, `Accept-Encoding` = "deflate, gzip", `Host` = "www.sec.gov"), httr::write_disk(dfile, overwrite=TRUE) It would be great if the package on CRAN could be modified along these lines and with the useragent argument added back in. — Reply to this email directly, view it on GitHub<#10 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BJNOICSUGJRTH6PBTYLKTPD2LF4KVAVCNFSM6AAAAABVKZIUUKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKOJZGI4DENJYG4>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[fibrou]

@Gunratan I was able to download a few years of 10-Ks, and then the system starting giving me the problem others indicate - bad bot. I am not sure how to make the adjustment indicated by @reillysteel, but it would be great if we could get an update, or slight modification in the code to include updated User Agent requirements in either a development version, or on CRAN. What do you think?