strip base64 in s3

[deepanshululla]

Title

Strips base64 file contents in s3.

Also a small bug fix for sqs, it was not skipping stripping of base64 files on failure events

Relevant issues

(litellm-py3.12) (base) ➜  litellm git:(feature/s3_strip_base64) poetry run pytest tests/test_litellm/integrations/test_s3_v2.py --tb=short -vv --maxfail=10 -n 4 
=========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.12.2, pytest-7.4.4, pluggy-1.5.0 -- /Users/deepanshu.deepanshu/Library/Caches/pypoetry/virtualenvs/litellm-ohMn1BPm-py3.12/bin/python
cachedir: .pytest_cache
rootdir: /Users/deepanshu.deepanshu/code/code/litellm/deepanshu/litellm
configfile: pyproject.toml
plugins: respx-0.22.0, xdist-3.8.0, anyio-4.5.2, mock-3.14.1, asyncio-0.21.2, retry-1.6.3, requests-mock-1.12.1
asyncio: mode=Mode.AUTO
4 workers [7 items]     
scheduling tests via LoadScheduling

tests/test_litellm/integrations/test_s3_v2.py::test_strip_base64_keeps_non_file_content 
tests/test_litellm/integrations/test_s3_v2.py::test_strip_base64_handles_empty_or_missing_messages 
tests/test_litellm/integrations/test_s3_v2.py::TestS3V2UnitTests::test_s3_v2_source_code_analysis 
tests/test_litellm/integrations/test_s3_v2.py::TestS3V2UnitTests::test_s3_v2_endpoint_url 
[gw1] [ 14%] PASSED tests/test_litellm/integrations/test_s3_v2.py::TestS3V2UnitTests::test_s3_v2_source_code_analysis 
tests/test_litellm/integrations/test_s3_v2.py::test_strip_base64_recursive_redaction 
[gw2] [ 28%] PASSED tests/test_litellm/integrations/test_s3_v2.py::test_strip_base64_handles_empty_or_missing_messages 
[gw3] [ 42%] PASSED tests/test_litellm/integrations/test_s3_v2.py::test_strip_base64_keeps_non_file_content 
tests/test_litellm/integrations/test_s3_v2.py::test_strip_base64_removes_file_and_nontext_entries 
[gw2] [ 57%] PASSED tests/test_litellm/integrations/test_s3_v2.py::test_strip_base64_removes_file_and_nontext_entries 
[gw1] [ 71%] PASSED tests/test_litellm/integrations/test_s3_v2.py::test_strip_base64_recursive_redaction 
[gw0] [ 85%] PASSED tests/test_litellm/integrations/test_s3_v2.py::TestS3V2UnitTests::test_s3_v2_endpoint_url 
tests/test_litellm/integrations/test_s3_v2.py::test_strip_base64_mixed_nested_objects 
[gw0] [100%] PASSED tests/test_litellm/integrations/test_s3_v2.py::test_strip_base64_mixed_nested_objects Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<CustomBatchLogger.periodic_flush() done, defined at /Users/deepanshu.deepanshu/code/code/litellm/deepanshu/litellm/litellm/integrations/custom_batch_logger.py:36> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-4' coro=<CustomBatchLogger.periodic_flush() done, defined at /Users/deepanshu.deepanshu/code/code/litellm/deepanshu/litellm/litellm/integrations/custom_batch_logger.py:36> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<CustomBatchLogger.periodic_flush() done, defined at /Users/deepanshu.deepanshu/code/code/litellm/deepanshu/litellm/litellm/integrations/custom_batch_logger.py:36> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-2' coro=<CustomBatchLogger.periodic_flush() done, defined at /Users/deepanshu.deepanshu/code/code/litellm/deepanshu/litellm/litellm/integrations/custom_batch_logger.py:36> wait_for=<Future pending cb=[Task.task_wakeup()]>>
Task was destroyed but it is pending!
task: <Task pending name='Task-14' coro=<CustomBatchLogger.periodic_flush() done, defined at /Users/deepanshu.deepanshu/code/code/litellm/deepanshu/litellm/litellm/integrations/custom_batch_logger.py:36> wait_for=<Future pending cb=[Task.task_wakeup()]>>


============================================================================================ warnings summary =============================================================================================
litellm/litellm_core_utils/get_model_cost_map.py:41
litellm/litellm_core_utils/get_model_cost_map.py:41
litellm/litellm_core_utils/get_model_cost_map.py:41
litellm/litellm_core_utils/get_model_cost_map.py:41
litellm/litellm_core_utils/get_model_cost_map.py:41
tests/test_litellm/integrations/test_s3_v2.py::TestS3V2UnitTests::test_s3_v2_source_code_analysis
tests/test_litellm/integrations/test_s3_v2.py::test_strip_base64_handles_empty_or_missing_messages
tests/test_litellm/integrations/test_s3_v2.py::test_strip_base64_keeps_non_file_content
tests/test_litellm/integrations/test_s3_v2.py::TestS3V2UnitTests::test_s3_v2_endpoint_url
  /Users/deepanshu.deepanshu/code/code/litellm/deepanshu/litellm/litellm/litellm_core_utils/get_model_cost_map.py:41: DeprecationWarning: open_text is deprecated. Use files() instead. Refer to https://importlib-resources.readthedocs.io/en/latest/using.html#migrating-from-legacy for migration advice.
    with importlib.resources.open_text(

tests/test_litellm/integrations/test_s3_v2.py::TestS3V2UnitTests::test_s3_v2_endpoint_url
tests/test_litellm/integrations/test_s3_v2.py::TestS3V2UnitTests::test_s3_v2_endpoint_url
tests/test_litellm/integrations/test_s3_v2.py::TestS3V2UnitTests::test_s3_v2_endpoint_url
tests/test_litellm/integrations/test_s3_v2.py::TestS3V2UnitTests::test_s3_v2_endpoint_url
tests/test_litellm/integrations/test_s3_v2.py::TestS3V2UnitTests::test_s3_v2_endpoint_url
  /Users/deepanshu.deepanshu/Library/Caches/pypoetry/virtualenvs/litellm-ohMn1BPm-py3.12/lib/python3.12/site-packages/botocore/auth.py:425: DeprecationWarning: datetime.datetime.utcnow() is deprecated and scheduled for removal in a future version. Use timezone-aware objects to represent datetimes in UTC: datetime.datetime.now(datetime.UTC).
    datetime_now = datetime.datetime.utcnow()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== 7 passed, 14 warnings in 2.06s ======================================================================================
(litellm-py3.12) (base) ➜  litellm git:(feature/s3_strip_base64) 

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have Added testing in the tests/litellm/ directory, Adding at least 1 test is a hard requirement - see details
• I have added a screenshot of my new test passing locally
• My PR passes all unit tests on make test-unit
• My PR's scope is as isolated as possible, it only solves 1 specific problem

Type

🆕 New Feature
🐛 Bug Fix

Changes

[vercel]

Someone is attempting to deploy a commit to the CLERKIEAI Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

deepanshu seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}