An Open Letter About Why and How We Use AI at Bark

Use of machine learning, natural language processing, and artificial intelligence (AI) has exploded over the past decade, pushing huge advancements across lots of industries. As with any technology, there are tradeoffs for those advancements. With AI, there is of course a long history of people musing about machines becoming sentient and deciding to act against the wishes of their human programmers, but I am encouraged to see more and more discussions being had around the garden-variety tradeoffs stemming from the use of AI. 

With great power comes great responsibility, and only through ongoing discussions can we as a society settle on a shared set of ethics around how AI is used.

At Bark, we employ a variety of machine learning techniques to provide our services, and we lean most heavily on deep learning in neural networks. This allows us to train our algorithms to more effectively detect the wide variety of issues that kids face, such as bullying, hate speech, online predation, and suicidal thoughts. Specifically, we aim to utilize the context in a discussion to help classify the issue. There is a massive difference in meaning between “I hate my life because I burned my Pop-Tart” and “I hate my life because no one likes me.” Despite sharing the same exact phrase, the context highlights only one as a potential issue.

To be clear, the terms “AI” and “machine learning” are misapplied and overused all the time. As words, they’ve become fuzzy because so many people use them to refer to things as simple as basic keyword matching or a list of rules. But those naive approaches are ineffective in the vast majority of examples, as in the example above — simple keyword matching can’t distinguish between those two examples and would flag both. This is why we’ve spent an enormous amount of time and effort developing algorithms that utilize context to make decisions.

All of that sounds great, and, well … it is! The levels of precision we can now achieve wouldn’t have been possible even ten years ago. However, nothing in life is perfect, and we’ve always been upfront that this concept applies to our algorithms as well. It’s completely reasonable to therefore ask what the tradeoffs are versus the positive benefits of using Bark, just like we should of any technology.

To illustrate, as a society we accept the negative tradeoff of roughly 40,000 fatalities from car crashes each year in the U.S. in order to get the value that automobiles bring us. It’s certainly uncomfortable to talk about 40,000 deaths a year as an “acceptable tradeoff,” but implicitly that’s exactly what we all do by choosing to drive cars and relying on the benefits that they provide.

A primary benefit of using Bark is the dramatically increased awareness of a wide variety of issues, many of which are very severe. Over the past five years, we’ve escalated nearly 1,000 child predator cases to law enforcement, played a small-but-important role in preventing numerous school shootings, and made tens of thousands of parents aware of imminent suicidal situations. These are reflected in the heart-wrenching testimonials sent to us (or posted online publicly) by parents, schools, and law enforcement. But again, we’ll be the first to emphasize that we are not perfect, and despite seeing massive benefits to children’s safety from our work, we are cognizant of the fact that we always need to evaluate and consider any potential tradeoffs.

Every day, we work hard to thread a needle — trying to find the exact right balance between safety and privacy. There’s a lot wrapped up in this, and I’d love to share a little about our core philosophy and mission, and how we always try to consider edge cases and any unintended consequences of what we do.

Fundamentally, we have built Bark on several concepts: 

First, awareness that many, if not most, issues that kids face are buried in their digital communications. Usage by children of internet-connected devices is nearly ubiquitous in the U.S., and smartphones have become the primary communication tools that children use.

Second, kids are still kids. As a society we draw all kinds of lines about what kids can and can’t do — we don’t let them buy alcohol or cigarettes, we force them to go to school and restrict their ability to work, and we don’t let them vote. Presumably, this is because the decision-making part of their brains is not yet fully formed, and they have less life experience to help them make right decisions. They will make mistakes as they learn who they want to be and what they believe. Caregivers play a vital role in helping children develop and are responsible for keeping them safe. 

Third, despite the need for caregiver involvement and guidance, we believe strongly that children should be respected, and where possible should be included in discussions about things that affect them. We actively encourage (and supply ways to facilitate) open and honest conversations between caregivers and children around online safety, including the use of Bark. We’ve intentionally chosen to build our product in a way that doesn’t give caregivers unfettered access to their child’s messages. We only show them the very small chunk of a conversation that has been flagged as concerning.

Our goal is to provide caregivers with drastically increased awareness of problems that they definitely should be alerted to, with as minimal an increase in awareness to non-issues as possible. Children should have the freedom to explore and use technology in healthy ways, but they should also have a back-stop when situations arise for which they are unprepared or unable to handle. While preemptive education about online dangers is absolutely necessary, it is too tall an ask of our children to expect that they will know how to handle a suicidal situation, recognize a child predator, or know how to handle being bullied without any help from a caregiver.

Because any system is imperfect, the most basic challenge we (or any machine learning system would) face is generating false positives. We go to great lengths to minimize these, but no system will ever have zero. In these cases, a caregiver would unintentionally see some information that could be private to the child and which the caregiver did not actually need to know. It’s also possible that in some instances, the alert we send may be completely correct, but is wrapped in context that the child may prefer the caregiver to not know. For instance, if a child was being bullied because of their sexual identity, the caregiver would get an alert from us about that bullying, and because of the specifics of the conversation could learn of the child’s sexual identity. We work hard to train our algorithms to dissociate statements about one’s sexual identity from insults that use similar terms, and we would not send alerts because of sexual identity.

The question becomes, what should we do in that situation? Certainly, it is heartbreaking to know that some caregivers will not be supportive of their children’s sexual identity, but that of course is not true for all caregivers. And, if the child is being bullied and needs support, does that need supersede the child’s desire for privacy around their sexual identity, especially when many risk factors (rates of bullying, depression, and suicidal ideation) for LGTBQ+ youth are unfortunately substantially higher?

Every situation is unique, and different in ways in which our system could never know, and this makes answering these questions very difficult. The only way to ensure that a child’s sexual identity was never unintentionally revealed to their caregiver would be to never send any alerts for any reason — which means the 85 alerts per day we send for imminent suicidal situations would also not be sent, nor would the tens of thousands of alerts where a child is being bullied, groomed by a sexual predator, or expressing signs of depression or anxiety. The situation is further complicated by the fact that these alerts aren’t actually giving the caregiver any information they can’t access otherwise. Typically when caregivers use Bark, they stop manually “spot-checking” their child’s phone, which actually decreases the chances that a child’s sexual identity would be exposed to the caregiver. Ultimately, we’ve landed on the side of expecting the best from caregivers, and giving them the tools they need to provide support for their children.

We’re in a real-life trolley problem. So while we don’t claim to be perfect, or to have all of these complexities completely figured out, as a company we are working to provide a service that can save lives, or simply make lives better for children and families. We get up every morning and work hard to refine what we do to help drive the best outcomes for kids, in a similar way to how the continual advancements in the automotive industry (seatbelts, airbags, lane assist) have helped dramatically reduce the tradeoffs for using that technology. We know the stakes are high — and that given the skyrocketing rates of teen/tween depression and suicidal ideation, it is incredibly hard to be a kid today, and also incredibly hard to be a caregiver. 

The solution isn’t for us to abandon technology, but rather to continue to work together to refine both the educational and technical tools to drive the best possible outcomes.