The production of labelled data is crucial for meeting the expectations of artificial intelligence (AI) system development and training(1). The great technological promise of this century depends largely on algorithms that need vast quantities of labelled training data to learn, recognise and categorise information(2). Data labelling is the product of AI models and manual work by people, who monitor, correct and augment the AI models, thus improving their accuracy. This form of labour is known as crowdwork.

As crowdworkers continue to make crucial contributions to training and developing AI systems, there has been increasing interest in ensuring they produce quality work. However, previous approaches have neglected to centre crowdworkers in system designs and to consider their identities, motivations and well-being.  These considerations are important, because they acknowledge crowdworkers are not a monolith and may have different needs and goals.

Designing a system sensitive to the unique needs of a specific worker population can increase the likelihood that a tool will be useful; by extension, this can contribute to crowdworkers' well-being and help them produce better quality work. In recent years, Latin American and Caribbean workers have been identified as significant crowdwork contributors. The crises in these regions have left many dependent on crowdwork as a steady source of income. Latin American women represent a sizable portion of these workers, and they use crowdwork as a path to achieving financial independence while balancing traditional caregiver responsibilities expected of them due to their strongly patriarchal societies. Unfortunately, research surrounding the perspectives of Latin American crowdworkers is incomplete as most studies have centred on the experiences of Western women, who have greater freedom and support to prioritise their individual needs.

To address this knowledge gap, we conducted an experimental survey on 60 women from Latin America who used the crowdwork platform Toloka. In the survey, we sought to understand their personal goals, professional values and the hardships they face in their work. Key insights included that a majority of the women shared a desire to hear about the experiences of other crowdworking women, particularly to help them navigate tasks, develop technical and soft skills, and manage their finances better. Moreover, 75% of the women reported doing crowdwork tasks in addition to caring for their families, while over half of them confirmed they needed to negotiate their family responsibilities in order to pursue crowd work in the first place. These findings confirmed these women lacked an important component in their experiences, namely a sense of connection with one another.

Based on these observations, we propose a system designed to foster community between Latin American women in crowdwork to improve their personal and professional advancement. By providing them with a safe platform to engage in meaningful conversation, they can begin to build an extensive foundation of knowledge that will help them to complete their work while growing their career and personal skills in the process. Moreover, since many Latin American women face the pressure of balancing their family responsibilities and reputation alongside their work, they are in the best position to relate to and give advice to one another.

Designing a space for crowdworkers to access each other’s knowledge can help improve the quality of their work by empowering them to seek support in addressing problems in their working conditions. According to field practitioners and researchers, it will only be possible to obtain a better future in crowdwork if key aspects such as transparency, fair pay and opportunities for professional advancement are embodied in tools available to crowdworkers(3).Having their experiences documented will not only offer long-term benefits to crowdworkers but also enable them to support one another in identifying areas of improvement in their work environments. This could inform decisions to organise and advocate for respect, fair pay and transparency from requesters and crowdworking platforms. Despite being promoted as an opportunity to create income and employment in regions where local economies are stagnant, there are not enough initiatives that address the impact of such work in the Global South through the lens of gender perspective, considering that 1 in every 5 crowdworkers are women.(4,5,6)

Research can be considered feminist when it is grounded in a set of theoretical traditions that privilege women’s issues and experiences(7,8). A key feature of feminist research is its focus on power imbalances, both in the subject that is studied and the relationships between the subject and the research. Feminist research practice includes the practice of reflexivity or positionality, both as a tool for producing knowledge and as an ethical method focused on subverting unequal relationships of power in knowledge production, considering our own position within the research as one of power, and always taking the vulnerability of the researched into account(9,10). An ethical commitment cannot happen without the step of reflexivity(11). Reflexivity is the deconstruction of knowledge production that works by addressing academia as a place for knowledge production, where social interactions and thus social constructions happen(12).

Gender mainstreaming can be used as a strategy for achieving equality by integrating gender analysis in all the phases of policies, actions, plans, and budgets(13).

Gender as a variable of analysis looks at social attributes and opportunities associated with being female and male, at the relationships between women and men and at the relations between women and between men(14,15). Moreover, as gender is generally understood as a social construct affected by power structures, different analytical concepts have been used to perfect the analysis of those power structures and its relations to gender. The most widely used concept is intersectionality. Intersectionality is used to map out the intersecting power relations in social relations by viewing categories such as race, class, gender, sexuality, nation, ability and ethnicity, among others, as interrelated and mutually shaping one another(16). As such, by understanding gender as a variable that is central to our research, we go beyond sex disaggregation and look into what the data conveys about attitudes, norms and gaps within AI crowdwork. Within the project, we conducted two survey pilots, taking care to prevent the exposure of the women identified as subjects of research (crowdworkers), to abide by ethical standards and fair pay, and to open up communication channels for them about their lived experiences beyond what previous literature on women crowdworkers has revealed.

Our team conducted two survey pilots in Spanish on the Toloka platform. Toloka is a global crowdsourcing company, founded in 2014 by Olga Megorskaya and integrated within the Yandex search engine. It is an enabling environment to support data-related processes(17). We chose the platform as it allowed us to filter workers geographically and thus to specifically analyse Latin American workers(18).

Additionally, we contacted Toloka’s Educational Program team, who expressed interest in supporting research activity through the platform. The first version of the survey was first answered by 6 people (three men and three women); the survey pilot was divided into specific sections, and most of the questions were either Likert-scale-type or open-ended questions. The time allocated for submitting the survey was 20 minutes for female and male crowdworkers who were interested in doing the task, and a filter for sex was used to get the answers of a group of women and men separately. Even though Toloka only permits binary sex disaggregation, our survey included an option for more inclusive gender identity. As we became employers (or “requesters”) on the platforms, we offered payment for the completion of the survey as if it was a task, calculated based on the minimum wage in the United States and the time to complete such a task(19). The payment to submit the survey was calculated as (minimum payment x fraction of an hour). For example, if we assumed a payment of USD 7.25 and the time to complete the survey was 20 minutes, the worker would receive an amount of USD 2.4 (7.25*0.333). To gain insights into how gender affected respondents’ work, we used different questions regarding widely recognised gendered factors such as time poverty(20), work-life balance, outside attitudes to the work done, care work(21) and direct opinions about whether they had considered gender a liability to their work. The second version of the survey was an iteration based on feedback about the need to include additional questions to better understand the characterisation of women working in crowdwork in Latin America, their motivations, needs, socioeconomic contexts, skills and hopes. The sample for the second survey included 57 women and three men.

There were four age groups represented in the sample, and 76% of respondents were 18 to 35 years old. The majority answered that their gender had little or no impact on their experience performing crowd work, while 17% believed it had some or an important impact on it.

As could be expected for the age group, most respondents had a professional degree or had at least started a university degree programme; 13% held a master's degree. Regarding their professional self-assessment, most of them knew their work strengths, work style and work environments and had a long-term vision for their career. They could balance their professional and personal goals and they wanted to develop new skills. Other respondents regarded themselves as being neutral in goal balancing, building a long-term vision for their careers and identifying work environments that interest them in their career search. Few of the respondents perceived that they did not know how to search for careers relevant to their interests and did not have a long-term vision for their careers.

They did not work in public spaces or libraries. They worked about two times per week from their own devices and at home, and spent about 20 minutes completing a common task, in some cases up to 40 minutes; the task they performed most was data labelling, webpage testing, and surveys (only 10% performed audio transcription tasks). Income per task was in the range of $0.01 to $0.16. 91% of respondents were engaged in other economic activities in addition to their work on Toloka. The most recurring reason for them to perform crowdwork tasks was to gain supplementary income. Additionally, a preference for working from home, and having schedule flexibility, were important factors. Few people considered it a recreational activity, and one respondent mentioned crowdwork as one of only a few opportunities for employment. Regarding the impact of the geopolitical situation of their country of origin, participants considered the main impacts to be economic. These economic impacts reduced their professional prospects and required them to engage in additional economic activity; 21% remained neutral.

Most importantly, the respondents shared an understanding of what features would be preferable in a tool. The explicability of tasks was important; although crowdworkers in Toloka can contact the requester, they find multimedia resources and tutorials useful to fully understand the tasks. Similarly, as 63% of respondents use search engines and translating applications as tools, prioritising language learning features is important. Regarding the perceived need to share experiences with other workers, the main goal of a communication feature would be to increase task completion, profitability and performance.

With the support of <A+> Alliance’s Feminist AI Research, we are currently developing a prototype for an AI-powered social connection and recommendation system in the form of a website specifically designed to assist crowdworkers in building a supportive community and developing their technical and soft skills.

The website is due to be accessible to the public in September 2023. We have started to explore conversational-agent systems based on our survey results, understanding the unique needs and challenges facing crowdworkers. The chatbot will be designed to match individuals with similar interests, goals and skill sets, enabling crowdworkers to form meaningful connections and giving them opportunities for skill-sharing and collaboration. The AI of a system that connects crowdworkers would likely involve a number of different technologies and techniques. One key component would be natural language processing (NLP), which would allow the system to understand and analyse the language used in the survey responses and profile information provided by users. This would enable the chatbot to understand the interests, skills and goals of each individual and match them with others who have similar characteristics.

Another important component would be machine learning algorithms, which would be used to analyse the data collected from the survey and continuously improve the system's ability to make accurate and relevant connections. These algorithms would be trained on the survey data, they would be able to identify patterns and relationships between different users, and they would have the capacity to make predictions about who would be the best match for a given individual. 

The impact of an AI-powered system like this would be to create a more connected and supportive community for crowdworkers, by connecting them with others who have similar interests, skills and goals.

This would help to increase opportunities for collaboration, skill-sharing and networking, which would be beneficial for their careers and professional development. Additionally, a system like this would help to increase the representation and visibility of crowd workers in their industries and could help to bridge the gap of inequality that they are facing, in terms of access to resources and opportunities.

In the first stage of our project, we conducted a literature review, deployed surveys and presented our findings and recommendations to build a smart tool prototype. We aimed for a second stage of financing and collaboration with the A+ Alliance’s Feminist AI: <From Paper to Prototype to Pilot> programme to test our concept through a pilot, and to build alliances with other researchers, community leaders and policymakers in the crowdwork space through dissemination events. The proposed nine-month timeline included activities to secure further funding to scale the tool after the collaboration with the International Development Research Centre (IDRC), Tecnológico de Costa Rica and the Feminist AI Network. It has been accepted, and the timeline is currently being executed.

La Independiente will be developing a pilot of an AI-powered social connection and recommendation system. It will be specifically designed to help collective platform workers build a supportive community and develop their technical and social skills. The second component of socialising will bring together policy- and decisionmakers, academics and other stakeholders in our “La Independiente Crowdwork Forum: Policy Perspectives from a Feminist Design”, which will be taking place on July 26 and 27, 2023. The agenda includes panels and workshops on policy and project ideation for more equitable platform work; panels with researchers and crowdworkers themselves; and lightning talks with journalists, activists and development workers advancing gender perspectives for crowdwork. Finally, the sustainability component will allow us to identify opportunities and partnerships in pursuit of the scalability of the initiative and technical tool. 

PIT Policy Lab, a spin out of C Minds, has established itself as a key ally in driving social change since 2020. As a technology-focused public policy lab, PIT Policy Lab builds on years of results, offering solutions to complex challenges from multiple disciplines and cross-sector engagement. Our team explores the field of Public Interest Technology by developing practical research and roadmaps, deepening understanding of responsible and human-centred technology use, implementing proofs of concept and offering educational and lifelong learning products.