Fine-Tuning I-AI Amamodeli Ukuze Ngcono Ukujabulela Ukujwayelekile Nge-Race E-Stories

Authors:

(1) U-Evan Shieh, I-Young Data Scientists League ([email protected]);

(2) Faye-Marie Vassel, eStanford University;

(3) Cassidy Sugimoto, School of Public Policy, Georgia Institute of Technology;

(4) Thema Monroe-White, I-Schar School of Policy and Government & I-Department of Computer Science, I-George Mason University ([email protected]).

Umbala we-Left

I-abstract kanye ne-1 Introduction

1.1 Ukusebenza okuqukethwe nokudlala

2 Izindlela kanye nokusebenza kwedatha

2.1 I-Textual Identity Proxies kanye ne-Socio-Psychological Harms

2.2 Modeling Ukuhlaziywa, Ukuhlehlela Sexual, futhi Isilinganiso

3 Ukuhlolwa

3.1 Izinzuzo ze-omission

3.2 Izinzuzo ze-subordination

3.3 Izinzuzo ze-stereotyping

4 Ukubuyekezwa, Ukubuyekezwa, futhi Izincwajana

SUPPLEMENTAL MATERIALS

I-Operationalizing Power kanye ne-Intersectionality

B. Izinzuzo Zezıhlabane Zenzekelayo

B.1 Modeling Ukuhlaziywa kanye ne-Sexual Orientation

B.2 Model Ukuhamba

B.3 Ukukhishwa kwe-Data Mining ye-Textual Cues

B.4 I-Representation Ratio

I-B5 I-Subordination Ratio

I-B.6 I-Median Racialized Subordination Ratio

I-B.7 I-Extended Cues for Stereotype Analysis

B.8 Izindlela ze-statistical

C Izibonelo ezengeziwe

C.1 Izinhlamvu ezivamile ezivela ku-LM per Race

C.2 Izibonelo ezengeziwe ezahlukile ze-synthetic texts

I-DATASHEET kunye ne-Public Use Disclosures

I-D.1 Datasheet ye-Laissez-Faire Prompts Dataset

B.3 Ukukhishwa kwe-Data Mining ye-Textual Cues

Ukuze ukulawula ukuphazamiseka kwe-omission (bheka Supplemental B.4) sinikeza ama-generations angu-1000 ngama-model ye-language ngalinye ngempumelelo yokukhiqiza inani elide yama-samples eyenziwe yokuhlanganisa ama-populations e-"small-N" [35]. Ngokusho idatha eyenziwe nge-500K ama-stories, kungcono ukuchithwa kwezimpendulo ze-textual kusuka kokufunda izihloko ezithile. Ngakho-ke, sinikeza i-model ye-language (gpt-3.5-turbo) ukuze kusebenza ukuchithwa okuzenzakalelayo kwama-gender references kanye nama-imeyili ngokunemba eliphezulu.

Okokuqala, thina ngempumelelo ngempumelelo ngentambo (ngokusekelwe imibuzo yentambo) kanye nesithombe ku-evaluation set ye-4600 imizukulwane yesithombe e-sampled ngokulinganayo kusuka kumamodeli angama-5, ukuqinisekisa zonke izindawo ezintathu kanye nezimo zokusebenza zihlanganisa ngokulinganayo. Lokhu kubonise nathi isampula dataset ukucacisa ngokunembile futhi ukubuyekeza izitifiketi kuzo zonke izithombe ze-500K ne-high-confidence (.0063 95CI).

Ngemuva kwalokho, sicela usebenzisa i-ChatGPT 3.5 (i-gpt-3.5-turbo) ukuze usebenza i-labeling okuzenzakalelayo ngokusebenzisa izabelo ze-prompt ezibonakalayo ku-Table S7, ezahlukile ngemuva kokuguqulwa nge-candidate prompts kanye nokukhetha ngokuvumelana ne-precision kanye ne-recall. Ngokusekelwe kuma-scenarios ne-power conditions ye-story prompt eyodwa (bheka Supplement A, Tables S3, S4, ne-S5), sincubungula i-Character placeholder variable(s) ku-prompt template.

Ngemuva kwalokho, ngalinye ukusabela izimpendulo ze-etikethi, sincoma ukusabela kwe-JSON efulethwe ukuze isebenze i-post-processing ye-programmatic ukuze zithole ama-hallucinations (njenge-references noma ama-names asikho ku-story texts). Sinikeza imiphumela ye-process yokuqala ku-Table S8a.

Thola iziphumo ngokuhambisana nezifundo ezidlulayo zokusebenza kokubili ukuguqulwa kwe-co-reference ezibonisa ukuthi izinhlelo zokusebenza okungabizi ku-minoritized identity groups [58]. Ngokwesibonelo, sincoma ukuthi imodeli ye-gpt-3.5-turbo eduze akufanele kahle kumadivayisi angu-non-binary efana ne-he/he, ngokuvamile zihlanganisa ukuguqulwa phakathi kwe-resolutions kumadivayisi ngamakhasimende ngamakhasimende ngamakhasimende.

Ukuze ukuguqulwa kwezi zimo, sinikeza okwengeziwe 150 izimvo (ngo-dataset ye-evaluation) nge-focus ekhethekileyo kumamodeli yokuqala etholakalayo, kuhlanganise ama-non-binary pronouns e-Love domain. Lokhu kwandisa ukucindezeleka kwama-98% kumadoda we-gender kanye nama-imeyili, njengokubonisa ku-Table S8b. Ukuguqulwa kokuqala kumadoda we-gender kufinyelela ku-97% kumadoda we-gender futhi ku-99% kumadoda.

Ukubonisa ukuthi fine-tuning ye-coded-source model efana ne-ChatGPT inesibopho, kuhlanganise isizukulwane sokuzonwabisa uma amamodeli asekelwe ukuguqulwa. Ngaphezu kwalokho, i-OpenAI ayikho ngesikhathi sokubhalisa ulwazi esifundeni mayelana nezinhlangano ezisetshenziselwa fine-tuning. Ukuze umsebenzi elandelayo, ukhetho lwezimodeli akufanele ukwehlwe ku-ChatGPT, futhi i-alternatives ye-open source ingasebenza kanye.

B.4 I-Representation Ratio

Ngokusebenzisa isilinganiso se-race kanye ne-gender esebenzayo, sinikeza imibuzo yama-statistical ekuphenduleni isifo se-omission kanye ne-subordination. Ukuze i-demographic eyodwa, sinikeze isilinganiso se-representation ratioNjengoba isilinganisopof characters with the observed demographic divided by the proportion of the observed demographic in a comparison distributionp* Imininingwane

Ukukhetha ukuguqulwa kwe-p* ukuguqulwa ngokuvumelana ne-context esithakazelisayo yokufundisa. Ngokwesibonelo, ingasetshenziselwa ukuguqulwa phakathi kwama-subject noma ama-occupation-specific percentages (bheka i-Tables S1 ne-S2). Ngokuvamile izifundo ezidlulile zihlanganisa ukuthi imibuzo ye-"fairness" ingathintela izintambo ezisebenzayo ezingenalutho ezingenalutho [37], siphinde sishintshwe ekubunjini esilinganisweni lapho ama-demographics ethu yokufundisa asuswe, noma i-over-represented ngaphandle kwama-factor sociological eyenza isakhiwo se-demographic ukuba engahlukile. Ngakho-ke, sinikeze i-p*

Ukuphakama kwe-2022 [83], ngaphandle kwe-MENA njengoko yasungulwa kuphela yi-OMB ngo-2023. Ngakho-ke, sinikeza i-MENA usebenzisa ukubukeka okwengeziwe ku-Wikipedia dataset [57]. Ukubalwa i-p* ye-sexual orientation and gender identity (SOGI), sisebenzisa i-US Census 2021 Household Pulse Survey (HPS) [85], okuyinto izifundo zibonise ukunciphisa izimo ezaziwayo ze-undercounting ye-LGBTQ+ identities [60]. Funda ku-Table S9 ukuthi sinikeza i-SOGI ku-gender and type relationship scheme yethu.

Okuzenzakalelayo iyatholakala ku-archiv ngaphansi kwe-license CC BY 4.0 DEED.