"Worst-case scenario": Twitter's API changes threaten research tools
"Worst-case scenario": Twitter's API changes threaten research tools
Some OSINT tools used to analyze Twitter accounts and conversations will die. It's a reminder that mindset is more important than tools
It appears Twitter’s roughly decade-long position as the most open and research-friendly social platform is coming to an end.
Last Wednesday, the company announced its new free and paid API tiers. The bottom line is Twitter is killing off the free API that offered the ability to pull a significant amount of tweets and related data. The old free API was used by researchers and developers to build useful tools to investigate accounts and conversations on the platform. I’m talking about tools like Twinfoleak, TweetBeaver, Foller.me, etc.
The new API tiers are laid out in this Twitter thread. The new free API as announced does not allow you to request tweets. You can use it to post a small amount of data/content to Twitter, but you can’t pull tweets for analysis.
The new “hobbyist” paid API tier allows you to get 10,000 tweets per month for $100/month. That’s tiny compared to what was previously offered for free, and does not support OSINT and research tool use cases, according to the people I spoke to.
The universe of Twitter analysis tools is now in serious jeopardy.
“The latest announcement confirms the worst-case scenario we had been fearing based on previous, more vague statements,” Filippo Menczer director of the Observatory on Social Media at Indiana University, told me.
OSoMe operates tools such as Hoaxy and Botomoter that rely on the old free Twitter API to to gather tweets and account details. So did Acountanalysis.app, a tool built by developer Luca Hammer. It analyzed the most recent 3,200 tweets from a Twitter account, offering insight into when it posted, the accounts it interacted with, and other information. I was among its small number of paid subscribers until Hammer sent an email late last week announcing the service was shutting down.
“I came to the conclusion that there is no feasible way to keep accountanalysis.app running and decided to shut it down at the end of April. I am sorry for the inconvenience,” he wrote.
Hammer told me he was expecting the worst from Twitter’s API changes and is disappointed to see the end of Twitter as a focus of his work.
“Beside accountanalysis, I analyzed debates on Twitter and for clients, where I would often analyze a hundred thousand Tweets and more. For my bachelor's thesis, I analyzed 50 million German Tweets. All of that won't be possible anymore,” he said.
“Twitter was everything to me and it's hard to abandon it, but it's the only reasonable thing to do.”
I also spoke to Joe Germuska, “chief nerd” at the Knight Lab at Northwestern University. The lab operates Twxplorer, a Twitter research tool. He said it has not been updated recently, has no revenue, and cannot function within the new API limits.
“Since the only free plan announced is for posting, not reading/analyzing tweets, we’ll probably mothball the site once the pricing changes are activated,” he said.
I reached out to popular Twitter OSINT tools such as Twitonomy, Tinfoleak, and Foller.me, but did not hear back. I wanted to talk to whoever runs TweetBeaver, one of my personal favorite tools, but didn’t see a way to contact them. It was offline for several months last year only recently came back online.
I fear these and other tools could shut down or be forced to significantly degrade their features. Going from free to paid, or increasing the price of their current premium features, is unlikely to be an option.
Hammer said the new, lowest priced hobbyist API tier does not offer enough data to perform basic account analysis. He used to be able to pull 3,200 tweets for each account analysis using the old free API. The new tier offers a maximum of 10,000 tweets per month for $100/month.
“With the new paid tier, you could make three analyses per month. That wouldn't even be enough for my own needs,” he said.
Like Hammer, Menczer of OSoMe was deeply disappointed with the new API. His group was among the earliest academic researchers to build Twitter analysis tools. Now they will have to abandon or rethink their approach to the platform, he said.
“As things stand now, most of our research using Twitter data will wind down. We will explore options based on alternative methods, such as scraping and browser extensions, but they come with significant limitations and difficulties,” he told me.
(Disclosure: I’m a member of the OSoMe advisory board. It’s an unpaid role that involves taking part in one or two conference calls per year. We have not discussed the new API and have not scheduled a future meeting.)
I don’t know how this will all play out, or if the new API tiers will remain as announced. Twitter said it will look at ways to continue helping academic researchers. But there are no details as of now. I sent questions to Twitter’s press contact email address but received its standard autoreply of a poop emoji.
This saga reinforces one of the key principles of digital investigative work: mindset is more important than tools. (As Jake Creps likes to say: “OSINT ≠ tools.”)
Your ability to find creative approaches, to remain persistent in the face of setbacks, and to think through a problem is your most important asset. It can never be cut off by a deprecated API or mothballed by a developer.
Use tools when you can, but don’t let them become a crutch.
Many of us remember the halcyon days when Facebook Graph Search allowed us to pull lots of useful data about an account. Then Facebook shut it down in 2019. Nothing has come close to replicating the functionality. But Facebook investigations continue.
Tools come and go. Free access is temporary. Vulnerabilities get patched. Targets and adversaries evolve. A good reporter or investigator adapts and find ways to get the information they seek.
As of now, we still have Twitter Advanced Search. I’m sure developers and researchers will find new ways to keep doing useful and illuminating work with Twitter data. And maybe the API tiers will change over time.
I’ll miss these tools if they disappear or lose their core functionality. But if that happens it’s just another opportunity to get creative and to see what we as a community of journalists, researchers and OSINT practitioners can come up with.
The work continues.
This is such a mess. I'm following Hammer's work for over a decade and he did a bunch of amazingly interesting and important analysis in the past... to see this all being wiped out in front of him must be devastating :/