The spy tech that followed kids home for remote learning—and won’t leave

A week after the pande­mic forced Minne­a­po­lis students to attend clas­ses online, the city school district’s top secu­rity chief got an urgent email, its subject line in all caps, aler­ting him to poten­tial trou­ble. Just 12 seconds later, he got a second ping. And two minu­tes after that, a third.

In each instance, the emails warning Jason Matlock of “QUES­TI­O­NA­BLE CONTENT” poin­ted to a single culprit: Kids were watching cartoon porn.

Over the next six months, Matlock got nearly 1,300 simi­lar emails from Gaggle, a survei­llance company that moni­tors students’ school-issued Google and Micro­soft accounts. Through arti­fi­cial inte­lli­gence and a team of content mode­ra­tors, Gaggle tracks the online beha­vi­ors of milli­ons of students across the U.S. every day. The sheer volume of reports was overw­hel­ming at first, Matlock acknow­led­ged, and many inci­dents were utterly harm­less. About 100 were rela­ted to anima­ted porno­graphy and, on one occa­sion, a member of Gaggle’s remote survei­llance team flag­ged a ficti­o­nal story that refe­ren­ced “under­wear.”

Hundreds of others, howe­ver, sugges­ted immi­nent danger.

In emails and chat messa­ges, students discus­sed violent impul­ses, eating disor­ders, abuse at home, bouts of depres­sion and, as one student put it, “ending my life.” At a moment of heigh­te­ned social isola­tion and eleva­ted concern over students’ mental health, refe­ren­ces to self-harm stood out, accoun­ting for nearly a third of inci­dent reports over a six-month period. In a docu­ment titled, “My Educa­ti­o­nal Auto­bi­o­graphy, ” students at Roose­velt High School on the south side of Minne­a­po­lis discus­sed bullying, drug over­do­ses, and suicide. “Kill me, ” one student wrote in a docu­ment titled “goodbye.”

Nearly a year after The 74 submit­ted public records requests to unders­tand the Minne­a­po­lis district’s use of Gaggle during the pande­mic, a trove of docu­ments offer an unpre­ce­den­ted look into how one school system deploys a contro­ver­sial secu­rity tool that grew rapidly during COVID-19,  but carries signi­fi­cant civil rights and privacy impli­ca­ti­ons.

The data, glea­ned from those 1,300 inci­dent reports in the first six months of the crisis, high­light how Gaggle’s team of content mode­ra­tors subject chil­dren to relent­less digi­tal survei­llance long after clas­ses end for the day, inclu­ding on weekends, holi­days, late at night and, over the summer. In fact, only about a quar­ter of inci­dents were repor­ted to district offi­ci­als on school days between 8 a.m. and 4 p.m., brin­ging into sharp relief how the service extends scho­ols’ autho­rity far beyond their tradi­ti­o­nal powers to regu­late student speech and beha­vior, inclu­ding at home.

Now, as COVID-era restric­ti­ons subside and Minne­a­po­lis students return to in-person lear­ning this fall, a tool that was pitched as a remote lear­ning neces­sity isn’t going away anytime soon. Minne­a­po­lis offi­ci­als reac­ted swiftly when the pande­mic engul­fed the nation and forced students to learn from the confi­nes of their bedro­oms, paying more than $355,000—inclu­ding nearly $64,000 in fede­ral emer­gency relief money—to part­ner with Gaggle until 2023. Faced with a public health emer­gency, the district circum­ven­ted normal procu­re­ment rules, a reality that preven­ted concer­ned parents from raising objec­ti­ons until after it was too late.

A mental health dilemma

With each alert, Matlock and other district offi­ci­als were given a vivid look into students’ most inti­mate thoughts and online beha­vi­ors, raising signi­fi­cant privacy concerns. It’s unclear, howe­ver, if any of them made kids safer. Inde­pen­dent rese­arch on the effi­cacy of Gaggle and simi­lar servi­ces is all but none­xis­tent.

When students’ mental health comes into play, a compli­ca­ted equa­tion emer­ges. In recent years, scho­ols have ramped up efforts to iden­tify and provide inter­ven­ti­ons to chil­dren at risk of harming them­sel­ves or others. Gaggle execu­ti­ves see their tool as a key to iden­tify youth who are lamen­ting over hards­hips or discus­sing violent plans. On average, Gaggle noti­fies school offi­ci­als within 17 minu­tes after zeroing in on student content rela­ted to suicide and self-harm, accor­ding to the company, and offi­ci­als claim they saved more than 1,400 lives during the 2020–21 school year.

“As a parent you have no idea what’s going on in your kid’s head, but if you don’t know you can’t help them, ” said Jeff Patter­son, Gaggle’s foun­der and CEO. “And I would always want to err on trying to iden­tify kids who need help.”

Critics, howe­ver, have ques­ti­o­ned Gaggle’s effec­ti­ve­ness and worry that rumma­ging through students perso­nal files and conver­sa­ti­ons—and in some cases outing students for exhi­bi­ting signs of mental health issues inclu­ding depres­si­on—­could back­fire.

Using survei­llance to iden­tify chil­dren in distress could exacer­bate feelings of stigma and shame and could ulti­ma­tely make students less likely to ask for help, said  Jenni­fer Mathis, the direc­tor of policy and legal advo­cacy at The Baze­lon Center for Mental Health Law in Washing­ton, D.C.

“Most kids in that situ­a­tion are not going to share anyt­hing anymore and are going to suffer for that, ” she said. “It suggests that anyt­hing you write or say or do in scho­ol—or out of scho­ol—­may be found and held against you and used in ways that you had not envi­si­o­ned.”

Minne­a­po­lis parent Holly Kragt­horpe-Shir­ley had a simi­lar concern and ques­ti­o­ned whet­her kids “actu­ally have a safe space to raise some of their issues in a safe way” if they’re stifled by survei­llance.

In Minne­a­po­lis, for instance, Gaggle flag­ged the keywords “feel depres­sed” in a docu­ment titled “SEL Jour­nal, ” a refe­rence to social-emoti­o­nal lear­ning. In anot­her instance, Gaggle flag­ged “suici­dal” in a docu­ment titled “mental health problems work­book.”

District offi­ci­als acknow­led­ged that Gaggle had captu­red student assign­ments and other perso­nal files, an issue that civil rights groups have long been warning about. The docu­ments obtai­ned by The 74 put hard evidence behind those concerns, said Amelia Vance, the direc­tor of Youth and Educa­tion Privacy at The Future of Privacy Forum, a Washing­ton, D.C.-based think tank.

“The hypot­he­ti­cals we’ve been talking about for a few years have come to frui­tion, ” she said. “It is highly likely to under­cut the trust of students, not only in their school gene­rally, but in their teacher, in their coun­se­lor—in the mental health problems work­book.”

Patter­son shook off any privacy reser­va­ti­ons, inclu­ding those rela­ted to moni­to­ring sensi­tive mate­ri­als like jour­nal entries, which he charac­te­ri­zed as “cries for help.”

“Some­ti­mes when we inter­vene we might cause some challen­ges, but more often than not the kids want to be helped, ” he said. Though Gaggle only moni­tors student files tied to school accounts, he cited a middle school girl’s private jour­nal in a success story. He said the girl wrote in a digi­tal jour­nal that she suffe­red with self-esteem issues and guilt after getting raped.

“No one in her life knew about this inci­dent, and because she jour­na­led about it, ” Gaggle was able to notify school offi­ci­als about what they’d lear­ned, he said. “They were able to inter­vene and get this girl help for things that she couldn’t have dealt with on her own.”

‘Need­les in hays­tacks’

Tools like Gaggle have become ubiqui­tous in class­ro­oms across the country, accor­ding to forth­co­ming rese­arch by the D.C.-based Center for Demo­cracy & Tech­no­logy. In a recent survey, 81% of teachers repor­ted having such soft­ware in place in their scho­ols. Though most students said they’re comfor­ta­ble being moni­to­red, 58% said they don’t share their “true thoughts or ideas” as a result, and 80% said they’re more care­ful about what they search online.

Such data suggest that youth are being primed to accept survei­llance as an inevi­ta­ble reality, said Eliza­beth Laird, the center’s direc­tor of equity in civic tech­no­logy. In return, she said, they’re giving up the ability to explore new ideas and learn from mista­kes.

Gaggle, in busi­ness since 1999 and recently relo­ca­ted to Dallas, moni­tors the digi­tal files of more than 5 million students across the country each year, with the pande­mic being very good for its bottom line. Since the onset of the crisis, the number of students survei­lled by the priva­tely held company, which does not report its yearly reve­nue, has grown by more than 20%. Through arti­fi­cial inte­lli­gence, Gaggle scans students’ emails, chat messa­ges, and other mate­ri­als uplo­a­ded to students’ Google or Micro­soft accounts in search of keywords, images, or videos that could indi­cate self-harm, violence, or sexual beha­vior. Mode­ra­tors evalu­ate flag­ged mate­rial and notify school offi­ci­als about content they find trou­bling—a bar that Matlock acknow­led­ged is quite low as “the system is always going to err on the side of caution” and requi­res district admi­nis­tra­tors to evalu­ate mate­ri­als’ context.

In Minne­a­po­lis, Gaggle offi­ci­als disco­ve­red a majo­rity of offen­ses in files within students’ Google Drive, inclu­ding in word docu­ments and spre­ads­he­ets. More than half of inci­dents origi­na­ted on the Drive. Meanw­hile, 22% origi­na­ted in emails and 23% came from Google Hangouts, the chat feature.

School offi­ci­als are aler­ted to only a tiny frac­tion of student commu­ni­ca­ti­ons caught up in Gaggle’s drag­net. Last school year, Gaggle collec­ted more than 10 billion items nati­o­nally, but just 360,000 inci­dents resul­ted in noti­fi­ca­ti­ons to district offi­ci­als, accor­ding to the company. Nati­o­nally, 41% of inci­dents during the 2020–21 school year rela­ted to suicide and self-harm, accor­ding to Gaggle, and a quar­ter cente­red on violence.

“We are looking for need­les in hays­tacks to basi­cally save kids, ” Patter­son said.

The move came with steep conse­quen­ces. Though Spea­kUP was used just three times during the six-month window inclu­ded in The 74’s data, Gaggle’s survei­llance tool flag­ged students nearly 1,300 times.

During that time, which coin­ci­ded with the switch to remote lear­ning, the largest share of inci­dents — 38 percent — were porno­grap­hic or sexual in nature, inclu­ding refe­ren­ces to “sexual acti­vity invol­ving a student, ” profes­si­o­nal videos and expli­cit, student-produ­ced selfies which trig­ger alerts to the Nati­o­nal Center for Missing and Exploi­ted Chil­dren.

An addi­ti­o­nal 30 percent were rela­ted to suicide and self-harm, inclu­ding inci­dents that were trig­ge­red by keywords inclu­ding “cutting, ” “feeling depres­sed, ” “want to die, ” and “end it all.” an addi­ti­o­nal 18 percent were rela­ted to violence, inclu­ding thre­ats, physi­cal alter­ca­ti­ons, refe­ren­ces to weapons and suspec­ted child abuse. Such inci­dents were trig­ge­red by keywords inclu­ding “Bomb, ” “Glock, ” “going to fight, ” and “beat her.” About a fifth of inci­dents were trig­ge­red by profa­nity.

Concerns over Gaggle’s reach during the pande­mic weren’t limi­ted to Minne­a­po­lis. In Decem­ber 2020, a group of civil rights orga­ni­za­ti­ons inclu­ding the Ameri­can Civil Liber­ties Union of Northern Cali­for­nia argued in a letter that by using Gaggle, the Fresno Unified School District had viola­ted the Cali­for­nia Elec­tro­nic Commu­ni­ca­ti­ons Privacy Act, which requi­res offi­ci­als to obtain search warrants before acces­sing elec­tro­nic infor­ma­tion. Such moni­to­ring, the groups contend, infringe on students’ free-speech and privacy rights with little ability to opt out.

Shaver, whose students used Google Hangouts to the point of it beco­ming a distrac­tion, was alar­med to learn that those commu­ni­ca­ti­ons were being analy­zed by arti­fi­cial inte­lli­gence and poured over by a remote team of people he didn’t even know.

“I’m trying to imagine finding out about this as a high scho­o­ler, that every single word I’ve writ­ten on a Google Hangout or whate­ver is being moni­to­red, ” he said. “There is, of course, some lesson in this, obvi­ously like, ‘Be care­ful of what you put online.’ But we live in a country with laws around unre­a­so­na­ble search and seizu­re—and survei­llance is just a really slip­pery slope.”

The poten­tial to save lives

To Matlock, Gaggle is a life­sa­ver—­li­te­rally. When the tool flag­ged a Minne­a­po­lis student’s suidice note in the middle of the night, Matlock said he rushed to inter­vene. In a late-night phone call, the secu­rity chief said he warned the unna­med parents, who knew their child was strug­gling but didn’t fully recog­nize how bad things had become. Because of Gaggle, school offi­ci­als were able to get the student help. To Matlock, the possi­bi­lity that he saved a student’s life offers a feeling he “can’t even measure in words.”

“If it saved one kid, if it suppor­ted one care­gi­ver, if it suppor­ted one family, I’ll take it, ” he said. “That’s the bottom line.”

Despite heigh­te­ned concern over youth mental health issues during the pande­mic, its effect on youth suicide rates remains fuzzy. Preli­mi­nary data from the Minne­sota health depart­ment show a signi­fi­cant decline in suici­des state­wide during the pande­mic. Between 2019 and 2020, suici­des among people 24 years old and youn­ger decre­a­sed by more than 20% state­wide. Nati­o­nally, the propor­tion of youth emer­gency-room visits rela­ted to suspec­ted suicide attempts has surged during the pande­mic, accor­ding to the Centers for Dise­ase Control and Preven­tion, but preli­mi­nary morta­lity data for people of all ages show a 5.6% decline in self-inflic­ted fata­li­ties in 2020 compa­red to 2019.

Meanw­hile, Gaggle repor­ted that it iden­ti­fied a signi­fi­cant incre­ase of thre­ats rela­ted to suicide, self-harm, and violence nati­on­wide between March 2020 and March 2021. During that period, Gaggle obser­ved a 31% incre­ase in flag­ged content overall, inclu­ding a 35% incre­ase in mate­ri­als rela­ted to suicide and self-harm. Gaggle offi­ci­als said the data high­light a mental health crisis among youth during the pande­mic. But other factors could be at play. Among them is a 50% surge in students’ screen time during the pande­mic, crea­ting addi­ti­o­nal oppor­tu­ni­ties for Gaggle to tag youth beha­vior. Meanw­hile, the number of students moni­to­red by Gaggle nati­o­nally grew markedly during the pande­mic.

But that hasn’t stop­ped Gaggle from citing pande­mic-era mental illness in sales pitches as it markets a new service: Gaggle Therapy. In school districts that sign up for the service, students who are flag­ged by Gaggle’s digi­tal moni­to­ring tool are matched with coun­se­lors for weekly telet­he­rapy sessi­ons. Thera­pists avai­la­ble through the service are inde­pen­dent contrac­tors for Gaggle, and districts can either pay Gaggle for “blan­ket cove­rage, ” which makes all students eligi­ble, or a “retai­ner fee, ” which allows them to “use the service as you need it, ” accor­ding to the company. Under the second scena­rio, Gaggle would have a finan­cial incen­tive to iden­tify more students in need of telet­he­rapy.

In Minne­a­po­lis, Matlock said that school-based social workers and coun­se­lors lead inter­ven­tion efforts when students are iden­ti­fied for mate­ri­als rela­ted to self-harm. “The initial moment may be a shock” when students are confron­ted by school staff about their online beha­vi­ors, he said, but provi­ding them with help “is much better in the long run.”

 

As the district rolled out the service, many parents and students were out of the loop. Among them was Natha­niel Genene, a recent gradu­ate who served as the Minne­a­po­lis school board’s student repre­sen­ta­tive at the time. He said that class­ma­tes contac­ted him after initial news of the Gaggle contract was rele­a­sed.

 

“I had a couple of friends texting me like ‘Nat­ha­niel, is this true?'” he said. “It was kind of inter­es­ting because I had no idea it was even a thing.”

Yet as students gained a grea­ter aware­ness that their commu­ni­ca­ti­ons were being moni­to­red, Matlock said they began to test Gaggle’s para­me­ters using poten­tial keywords, “and then say ‘Hi’ to us while they put it in there.”

As students became condi­ti­o­ned to Gaggle, “the shock is probably a little bit less, ” said Roche­lle Cox, an asso­ci­ate super­in­ten­dent at the Minne­a­po­lis school district. Now, she said students have an outlet to get help without having to expli­citly ask. Instead, they can express their concerns online with an unders­tan­ding that school offi­ci­als are liste­ning. As a result, school-based mental health profes­si­o­nals are able to provide the care students need, she said.

Mathis, with The Baze­lon Center for Mental Health Law, called that argu­ment “ridi­cu­lous.” Offi­ci­als should make sure that students know about avai­la­ble mental health servi­ces and ensure that they feel comfor­ta­ble reaching out for help, she said.

“That’s very diffe­rent than deci­ding that we’re going to catch people by having them write into the ether and that’s how we’re going to find the students who need help, ” she said. “We can be a lot more direct in commu­ni­ca­ting than that, and we should be a lot more direct and a lot more posi­tive.”

In fact, subjec­ting students to survei­llance could push them further into isola­tion, and condi­tion them to lie when offi­ci­als reach out to inquire about their digi­tal commu­ni­ca­ti­ons, argued Vance of the Future of Privacy Forum.

“Effec­tive inter­ven­ti­ons are rarely going to be built on that, you know, ‘I saw what you were typing into a Google search last night’ or ‘wri­ting a jour­nal entry for your English class, '” Vance said. “That doesn’t feel like it builds a trus­ting rela­ti­ons­hip. It feels creepy.”

This arti­cle was also publis­hed at The74 Mi­llion.org, a nonpro­fit educa­tion news site.