Guide: How to use open safety models for Trust & Safety

[andrewmchang]

The RMC's first community project sprint is to draft a short guide on how to use open safety models in a T&S context. This guide should include the following:

Context Setting

• Brief overview of the DIRE framework for T&S capabilities (for reference, see ROOST's roadmap)
• Where AI models fit in that framework

Deep Dive

• How AI models fit into specific T&S architectures, with examples / diagrams
• Why and when T&S teams might use open safety models instead of closed / generalized models

Specific Tips & Tricks

• How to not break the bank using the wrong models / how to use with other model types (e.g., how to use a general classifier that's really cheap to run, then send off to specialized models or humans for more review)
• How to run and/or integrate different models (e.g., step by step guide to converting a HF link to something you can run locally / call)

My plan is to finish a draft guide by the end of the month! Ideally, I will draft one section per week, wrapping by 3/27. I'll share a Google Doc for any contributors who want to weigh in!

[andrewmchang]

Starting a draft in this Google Doc, cc @cassidyjames !

[andrewmchang]

Other considerations:
What should be in a policy? What is good policy?
Out of the box classifiers are not customized to specific policies. How can you balance low costs and a customized classifier, or how should you think through that trade off?
To what extent should we consider use cases outside of content moderation – especially moving into account level evaluation or broader definitions of safety (e.g. privacy, fraud, etc.)?

[andrewmchang]

Made some more edits this week, but we're still missing a few things, including architecture diagrams + more details in the tips and tricks section. cc @cassidyjames