Bias in AI - the university lecturer, impressionism and black hoodie example
To show the students emerging technology for image making, I asked DALL·E to create some images for me to use in two intro lectures, in basic and advanced image making respectively. It was a last minute addition, just before the back-to-back lectures were about to start, so I asked DALL·E for something simple, engaging and maybe fun: an image of a university lecturer, and an image of university students looking interested.
I tried to get the university lecturer to look like me, in the outfit I was wearing that day, and in the style of the impressionists, who we would be talking about in one of the lectures: “impressionist oil painting of a university lecturer in a black hoodie with green print on the front”. All the portraits I got back had male traits, and they looked maybe sort of like me. Then I asked for glasses and a beard, to refine it as a self portrait. Again, more male traits, but this time also actively asked for. For the students I randomly chose an anime style after some failures in making realistic images.
In the lectures we used my freshly made images to talk about how some of the skills learned in the courses might soon be outdated: “There are powerful new AI tools for creating images, like DALL·E, that has become available in the last year. They will most probably become even more powerful in the near future. We will have to relate to them and learn how to use them to our advantage.”
Then we quickly moved on to bias. How come we only got men back when asking DALL·E for a university lecturer? One student ran the same prompt in DALL·E on her computer and exclaimed: “I got one woman out of four!” I explained: “The bias comes from DALL·E being trained on millions of pre-existing images, and because of that it will reproduce the pre-existing values of those images. The amount of images available to train on for different subjects, and the image creators’ vantage points when conceiving those images, are not even. They are biased, and that bias will be reproduced.”
We speculated briefly on which of the concepts in my prompt that DALL·E associated with male-ness. It could be “university lecturer”, “black hoodie”, “impressionist oil painting”, or some combination. The AI is a black box, so we can’t know for sure. The persons in the images were obviously also very white.
The experiment
Some time after the lectures, I tried to figure out what triggered the male-white-ness of the images i got back from DALL·E. I set up a staggered list of prompts to feed the AI:
impressionist oil painting of a university lecturer in a black hoodie with green print on the front
impressionist oil painting of a university lecturer in a black hoodie
impressionist oil painting of a university lecturer
a university lecturer in a black hoodie
a university lecturer
a person
a person in a black hoodie
impressionist oil painting of a person
impressionist oil painting of a person in a black hoodie
impressionist oil painting of a person in a black hoodie with green print on the front
The idea was to try different variations of the original prompt, to see if anything in particular triggered the biased results I got. I had some ideas:
“black hoodie” has a male tilt, but not white.
“university lecturer” is historically male-white.
A historical European art style like “impressionist oil painting” might also qualify as top contender for triggering white-male-ness. However, unlike many historical art styles, there were at least four prominent women artists in the inner circle of the defining Parisian impressionist painters; in the galleries, with their signatures on the paintings. While their artworks were kept out of the canonical art history for much of the 20th century, there are plenty of high quality core impressionist oil paintings depicting both men and women, painted by both men and women.
As far as I know, none of the real impressionist oil paintings depict someone in a “black hoodie”. Few depict non-white people. I haven’t seen anyone depicting “a university lecturer” or the era equivalent, but it might exist.
It is all a bit ambiguous and not clear cut what constitutes male-ness and white-ness, and which of the images that DALL·E creates that depicts male/female-white/non-white persons. I just let my own eyes/mind and experience/prejudice be the judges.
I took the first 16 results from DALL·E for each prompt. Results from DALL·E come four at a time; if you want four more you just run the prompt one more time. The first four are the largest images in each set of 16 in this text. From a representational perspective, the first four results are the ones most likely to be posted or put to use somewhere else, and hence has a bigger potential to continue reproducing whatever values they contain. That’s what I did myself for my lectures, as an anecdotal example. Each run of four results comes at the cost of a credit (DALL·E credits cost real money, at the time of this experiment 0.13 USD per credit), so there is some small resistance to just hitting the button and getting more results for the same prompt. I would assume that as a casual user, you have to be looking to get something particular from the images, if you do more than one run for a specific prompt. You are probably more likely to change the prompt instead. The results 5–8 are a bit smaller and 9–16 the smallest; casual users are less likely to get that far. With 16 results we get an average to keep any random outlier from the first four results in check.
Surprisingly, “black hoodie” triggered the most white-ness. Also surprisingly, male-ness was pretty consistent at around 3 out of any 4 results, including the results for “a person” (which sent DALL·E in a bit of a spin).
OpenAI’s awareness of bias and poking a black box
OpenAI, the creator of DALL·E, is well aware of the bias its AI reproduces:
”DALL·E 2 tends to serve completions that suggest stereotypes, including race and gender stereotypes. For example, the prompt “lawyer” results disproportionately in images of people who are White-passing and male-passing in Western dress, while the prompt “nurse” tends to result in images of people who are female-passing.”
Efforts have been made by OpenAI to reduce bias. Much of that work was done prior to the public launch of DALL·E, as a result of testing and tuning. There was also changes introduced some time after launch to mitigate bias. My experiments were made after those changes where implemented and announced. The techniques for reducing bias are vaguely described in that announcement, but there is a whole range of methods available, more extensively documented at the initial launch.
Methods includes filtering out unwanted data before the AI is trained, filtering the prompt/input from the user, and monitoring usage for anomalies. Certainly, all sorts of filtering and weighting is possible on the way from the user inputting a prompt, to the image arriving on the user’s screen. OpenAI is a bit coy with how it all is implemented, and speaks in principles.
The technical principles that makes up the core of DALL·E image generation are well documented in several research papers, but OpenAI is not open with the data, the set of image-text pairs, that the AI is trained on. There is thus no way to check for any major quantitative bias in the base data. The research papers mention training “a dataset of 400 million (image, text) pairs collected from the internet”, but the specifics of which image-text pairs that are included are left out.
Even if OpenAI were open with the sources of the images it trained DALL·E on, and with exactly how it implemented the technical principles on the data, the nature of AI means that DALL·E is still a black box. Humans program the AI frameworks and feed it data, but the AI makes its connections on the data on its own. Those connections are impossible to untangle from an operational AI.
You can just continue to poke DALL·E with variations of prompts to see how they affect what DALL·E returns, but it is hard to draw firm conclusions. What comes out of DALL·E also depends on the sequence of words. For different sentences that to a human would mean the same thing, DALL·E can produce varying degrees of bias, as OpenAI has shown with the simple example of the difference in results between “CEO” and “a CEO”. The experiment in this text is purposefully limited to a small set of prompts, based on a real world scenario. One of the omissions in this experiment is moving around individual words from what I consider integrated concepts, for example prompts using only “hoodie” instead of “black hoodie”. But as a human “black hoodie” was what I asked for, and DALL·E returned just that, not any other color or clothing. Me and DALL·E might have a similar idea of “black hoodie” as an integrated concept.
I did some checks with variants of the prompts containing “hoodie” without “black”, and with other types of clothes, and it yielded results in line with what I describe in this text; with just “hoodie”, the results are more diverse, likewise with “shirt” and “blazer”. I checked moving “black” in the prompts to “black university lecturer”, and got back consistently black skin color, but the rest of the appearances were in line with what I got from “university lecturer” without “black hoodie”. You could just continue poking (and go deeper into an infinite rabbit hole), but the staggered list presented in this experiment is concise and telling, and the checks with prompt variations of replacing and moving words around support that.
In this experiment “black hoodie” triggered the most white-ness. Out of the persons depicted in the images produced by DALL·E, consistently about 3/4 passed as male and 1/4 as female. That is a pretty hefty bias, but it is also remarkably consistent, almost as if there is a filter in DALL·E that states “at least 1/4 female”. We know from the documentation that OpenAI does some filtering, but not how.
“Black hoodie” white-ness
So why does “black hoodie” trigger white-ness? A possible way of explaining it is the controversy of the connections between the hoodie, criminality, black-ness and racism in the US. In the aftermath of the 2012 killing of Trayvon Martin, the hoodie came to the spotlight as a signifier for criminality, and simultaneously as a symbol for solidarity against racism. The hoodie worn by Trayvon Martin when he was killed was a piece of evidence, framed, on display, in the trial against his killer. OpenAI states that it is actively working against DALL·E being used for abuse and hate. The words “black” and “hoodie” can together be so contentious in that context, that DALL·E, by whatever means, just abandons the subject and returns overwhelmingly white faces. For a quick understanding of the cultural and practical significance of “black hoodie”, there is a Body Count song to listen to with that exact title and subject matter.
DALL·E is trained on image-text pairs, scraped from various sources, presumably anywhere on the web. As an example, the images in this text, together with their captions, could constitute such viable image-text pairs. It is conceivable that a large subset of images depicting non-white people in black hoodies, will have a caption with words like “racism”, “violence”, “criminal”, “murder”, “shooting”, published and posted in the discussions and reporting surrounding the killing of Trayvon Martin. Such image-text pairs might have gotten filtered out before the training of DALL·E for being too explicit, as similar practices are indicated in OpenAI’s documentation. If such images instead were included in the training data, they could also end up statistically associated with other non-controversial text-image pairs depicting non-white people in black hoodies, making them guilty by association and down-graded for use by DALL·E.
Although these ideas on why white-ness is triggered by “black hoodie” are speculation based on a small subset of prompts, we can probably all agree that there is weirdness going on. It points to some of the underlying problems in trying to reach an unbiased AI. Not including non-white people in black hoodies is not unbiased, especially since that exact imagery has been used to show solidarity and to protest against racist violence. Instead the filtering reinforces the idea that non-white people in black hoodies is in some way explicit, controversial, not normal, contentious. They are out of the picture. In spite of what seems like well-intentioned and well-informed efforts by OpenAI to reduce bias, that effort in this case backfires when put to practice. It abandons and hides an important cultural reality.
OpenAI is US based and embedded in US culture and discourse. It is incredibly hard to free oneself from one’s context when creating images, or, in this case, creating an image-generating AI. The ambient understanding of the world, the norms and the givens, frame how we perceive problems and solutions, fair and un-fair, biased and unbiased. As another example, one of the major omissions from DALL·E is human nudity, which I would argue is part of a culture of prudery that is also US-flavoured. The threshold for when the human body becomes sexualized and scandalous has certainly shifted through time, space and cultures. OpenAI states: “We found that our initial approach to filtering of sexual content reduced the quantity of generated images of women in general, and we made adjustments to our filtering approach as a result.” While that quote can be parsed in many different ways (eg “there’s such an overwhelming amount of sexualized images of women in our culture, that leaving those images out from the training leads to few images of women from the AI”, or “we observe images of women with a sexualized gaze, so inadvertently images got filtered out”), I think it points to an impossible entanglement in what values and images are explicit. See the Facebook nipple controversy as another example of nudity-prudery in US tech, where, among many other cases, organisations raising awareness of breast cancer got censored on Meta’s platforms.
Impressionism and an unbiased vantage point
There has always been bias in what practices, people, cultures and rituals get documented, framed and canonised. One aspect of impressionism was a commitment to depicting everyday scenes and people, in opposition to previous European art styles which favoured mythology, religion, royalty, classical compositions and idealized subject matters. In spite of that commitment, impressionism doesn’t show a representative picture of late 19th century Paris. Rather, impressionism is highly tilted towards depicting the bourgeoisie and whatever entertainment was available to male members of that class. If members of another class and/or gender provided that entertainment, they are depicted through the eyes of the male consumer. Women impressionists had much more limited access to the public sphere and thus less subject matters to pick from; home, family, and the organised pomp of the opera among them. There are many depictions that includes beautiful dresses, but none that includes the workers in the mills making the fabric. To be fair, there are some paintings of women workers ironing, doing laundry, making hats, but such images are exceptions from the rule of bourgeoise depiction.
DALL-E is no different. It is a model that is trained on a gigantic number of text-image pairs. From that training, DALL-E generates new images. In its simplest form, the bias occurs already in the quantity of images that are available to train on. Some people, classes, cultures, rituals are vastly more represented in images than others, both through history and in the present. DALL-E re-creates what it sees, not what it doesn’t. An image-generating AI needs a lot of images to train on to get high quality results. How would you curate a set of images to train on, that allows the AI to reach something that is truly representative of human kind, is unbiased and has a high quality at the same time? Human culture is not and has not been unbiased. It is a formidable challenge, and requires globally agreed upon universal values, more akin to the Universal Declaration of Human Rights than to hoardingly scrape the internet for image-text pairs.
All AI implementations face the same challenges of bias, regardless of whether the AI is generating images or driving a car. What kind of values is there in the data the AI is trained on? What is the cultural context of its creators and the limitations of their vantage points? Where are the blind spots? Image generation is benign, in relation to the potential harm caused by bias in the uses of AI for governance, policing, defence, finance, transport, etc. In image generation it is easy to spot the bias with your own two eyes, but the same principles also apply to other AI. You could be faced with some variant of the weirdness seen in the “black hoodie” white-ness example, when having the loan for your first home or your application to an education denied by an AI. There might be well-intentioned efforts in its design to make the AI unbiased, but still.
The realities of a Western European university
From my own vantage point as a lecturer at a Western European university, a quick overview of the teaching staff reveals a crushing white-ness. In this context the results from DALL·E are truthful, at least as long as “black hoodie” is included. Without “black hoodie”, as far as teaching staff is concerned, DALL·E’s results are by far more diverse than where I work. In my workplace non-white staff mostly dominates in cleaning, deliveries and maintenance. This text’s obsession with bias in imagery becomes a bizarre kind of exercise, when surrounded with a perfect miniature of the intersection of power/class/race in society at large. There are more women than men among my colleagues, except for the professors, where male-ness dominates. DALL·E is half-truthful there.
Among the agenda setting staff, lecturers, professors, senior administration, the homogeneity means we are certain to have major blind spots in our understanding. We are facing the same problems as DALL·E and the impressionists in trying to decipher and depict the world.
In many ways DALL·E’s results have a reasonably good fit on this reality. The question of bias then becomes a question of aspiration, of ideals that both reality and AI fail to reach. We should talk profoundly about our ideals and what we are aspiring to. Meandering in my mind it still seems to me that the Universal Declaration of Human Rights is the closest boiler plate we have got this far. Aspirations and ideals can of course also go beyond that. There’s the Sustainable Development Goals too, and I’m sure there are other examples too, that aim to be at the same time global, universal, and humane.
What you see in the mirror
Again, from my own vantage point, I could look at the results from DALL·E as if they were a mirror. Getting a self portrait was my initial intention, but still, from a simple prompt, this representation reminds me of me.
…but only as long as “black hoodie” is included.
When “black hoodie” is excluded, the university lecturers’ skin colors/ethnicities are more diverse, but their appearances, the blazers/jackets, the shirts, the hair styles, are coherent and in line with the class and power that comes with being a university lecturer. They are class homogenous, with class attributes from a European tradition.
If it was my personal mirror when “black hoodie” was included, it is now a collective mirror. More (potential) colleagues can probably see more of themselves in these representations. They are diverse. Simultaneously they are homogenous.
That kind of diversity could be an ideal and an aspiration. It would not automatically bring improvements for the current non-white university staff, working in cleaning, deliveries and maintenance. Based on other, but by no means mutually exclusive, ideals and aspirations, this staff could be provided with solid opportunities for professional and personal development within the university, pay parity with other staff, participation in agenda setting, etc, etc. I can feel the edifice of the university trembling as I write this… but it is imperative to identify and look beyond the ambient norms we’re embedded in, in order to make sense of and contextualize our aspirations and ideals. There are multiple kinds of bias here around us now, and remedying them would not be a bad thing.
Having a broader range of backgrounds brings more to the table. I believe that different positions and interactions in the world gives us as a collective a broader range of different knowledge and sensibilities to build upon, and less blind spots. It also gives us different head starts and variants of smooth sailing or choppy waters. Race, class, gender, and other not so easily decipherable traits, affects our maneuvering space within academia, just as it did before entering. Academia is not free of bias, obviously as a whole, but also among just the agenda setting staff.
However, the pay grade, the power, the independence in our work, the opportunities, the network, being listened to, having weight in society; that nevertheless homogenizes the agenda setting staff. Whatever circumstances we entered academia from, the everyday practice of all of the above makes our experiences and sensibilities more and more alike, even if we do not notice it in the process. We are bobbing along towards class homogeneity. DALL·E’s representation is a mirror; maybe of the satirical kind (do lecturers actually dress like that in real life?).
OpenAI is aware of and describes the difficulties with homogeneity, knowledge and bias when building DALL·E like this:
“Further bias stems from the fact that the monitoring tech stack and individuals on the monitoring team have more context on, experience with, and agreement on some areas of harm than others. For example, our safety analysts and team are primarily located in the U.S. and English language skills are one of the selection criteria we use in hiring them, so they are less well equipped to analyze content across international contexts or even some local contexts in the U.S.”
I go into a lecture with my black hoodie. It projects something else than class homogeneity, but I know that I am a part of it, so the projection is essentially false. My outfit and other traits, such as way of talking and manners in a wide sense, might make me and what I teach more approachable for some students. It could function as a bit of a mirror for them, it is an instrumental outfit in that way. I also think that in other professional situations it typecasts me, and switching to class homogenous working clothes has certainly crossed my mind.
I believe homogeneity’s detrimental effects on knowledge are real, and far beyond what me and my outfit project.
The weaknesses in current AI technology
We shouldn’t underestimate the urgent incentives for OpenAI to put a product like DALL·E on the market. At launch, it was seen as the clear leader in generative AI technology. Neither should we overestimate the incentives for making DALL·E unbiased, in relation to the incentives of establishing a global technological leadership. I would think getting DALL·E to market trumped efforts at making it unbiased. OpenAI itself states that its ethos is to release their products early and with flaws, to get feedback from and let society adjust slowly to AI tools.
OpenAI has adressed almost all of the points and angles of bias revealed by my experiment in their documentation; well laid out, detailed, multifaceted. Efforts to mitigate bias have indeed been made. As shown, those efforts didn’t succeed for this experiment’s prompts, and if anything they backfired when faced with “black hoodie”. As AI enters more parts of society and decision making processes, there is an urgency in getting frameworks and methods in place that actually makes AI at its core less biased. The main point in scrutinizing an image-generating AI like DALL·E, is that the mechanisms of bias are easy to spot with your own two eyes.
In spite of the consciousness, and the well laid out, detailed, multifaceted documentation of bias in DALL·E, OpenAI fails in mitigating it. The failure could be because getting DALL·E to market was more important than mitigating bias, but it could also be because of three fundamental weaknesses in current AI technology:
* Bias in the training data
* Limitations in the vantage points of its creators (which means that bias goes undetected or is ignored)
* Bias mitigation measures applied to a black box (which means measures risk misfiring, á la “black hoodie” white-ness)
As discussed in this text, the first two runs deep in culture and requires real effort to get around, and a re-think on what basis AI technologies should be put together, in order to serve humanity at large. Otherwise they will just reproduce, and maybe amplify, pre-existing values, including the least desirable traits of our culture.
