I want to extract all the tweets I've ever written and convert them to small markdown files so they show up as "posts" on this website.

Posts are organized by date, in the traditional blogging format. But so are tweets, kind of? They're listed chronologically anyway. Maybe I could make one post for each day, and then have all the tweets listed on that page. Each one could have its tweet ID as a header, thus having an internal link. Tweets can link to the tweets that precede them, and maybe even backlink to tweets that follow.

It's not quite block references, but as a way of keeping my second brain under my ownership it should work. And this way if anyone wants to cancel me they'll have a convenient search box and permalinks for it. Even if my account gets deleted, my bad takes can stay up.

from pathlib import Path
out_dir = Path('../_posts/tweets/')
posts = [o.name for o in out_dir.iterdir()]

last_date = sorted(posts)[-2][:10]

last_date
'2021-09-12'

This :point_up: is a workaround. Sometime after I started archiving my tweets, Twitter changed the API for search. This means that unofficial scrapers like twintstopped being able to see tweets from more than a few months ago. Even the in-app search was broken for a while.

Fortunately I had already scraped and archived my old tweets, so now I just check to see when the last time I ran this script was and find tweets since that date. This also helps protect me from getting rate-limited by Twitter (they tend to do that if you download 30,000 tweets a few times in a row lol).

import twint
import nest_asyncio
nest_asyncio.apply()


c = twint.Config()
c.Username = 'deepfates'
tweets = []
c.Store_object = True
c.Store_object_tweets_list = tweets
c.Since = last_date
c.Hide_output = True
twint.run.Profile(c)
<twint.run.Twint at 0x7fce23268390>

This is all the configuration necessary to grab my tweets. Have to use Hide_output = True, or it will print every single tweet in the output.

I now have a dataset of tweets. Let's explore one here, and see some of its metadata.

Check data

len(tweets)
3247
t = tweets[-1]
t.conversation_id, t.datestamp, t.datetime, t.id, t.likes_count, t.link, t.mentions, t.photos, t.quote_url, t.replies_count, t.reply_to, t.retweet, t.retweet_date, t.retweet_id, t.retweets_count, t.source, t.thumbnail, t.timestamp, t.timezone, t.tweet, t.urls, t.user_id, t.user_id_str, t.user_rt, t.user_rt_id, t.username, t.video
('1439019641205055496',
 '2021-09-17',
 '2021-09-17 18:28:06 MDT',
 1439023391512670210,
 4,
 'https://twitter.com/deepfates/status/1439023391512670210',
 [],
 [],
 '',
 0,
 [{'screen_name': 'IgorBrigadir', 'name': 'Igor Brigadir', 'id': '495430242'}],
 False,
 '',
 '',
 0,
 '',
 '',
 '18:28:06',
 '-0600',
 '@IgorBrigadir And it said that one day you will wrestle the very world snake itself! And win',
 [],
 3315205122,
 '3315205122',
 '',
 '',
 'deepfates',
 0)

Tweet layout functions

Many of my tweets have pictures or memes attached, but Twitter only includes the pic.twitter.com URL for these. Here I build a few functions to download the image and squeeze it into a Markdown template for display on my site.

import requests
import shutil
def dl_image(url):
    filename = '../images/from_twitter/' + url.split('/')[-1]
    r = requests.get(url, stream = True)
    if r.status_code == 200:
        r.raw.decode_content = True
        with open(filename,'wb') as f:
            shutil.copyfileobj(r.raw, f)
        return(filename)
    else:
        return(None)
    
# hacky thing uses [1:] to shave the first '.' off the filename
def image_template(filename):
    return(f'![image from twitter]({filename[2:]})\n')

    
def get_tweet(t):
    if t.photos == []:
        img_md = ''
    else:
        img_list = [dl_image(url) for url in t.photos]
        img_md = '\n'.join([image_template(o) for o in img_list])

    return(f'''
#### <a href = "{t.link}">*{t.timestamp}*</a>

<font size="5">{t.tweet}</font>

{img_md}

🗨️ {t.replies_count}{t.retweets_count} 🤍  {t.likes_count}   

---
    ''')

def get_md(tweets, date):
    tweets_text = ''.join(t for t in tweets)
    return(f'''---
title: deepfates log {date}
layout: post
toc: true
comments: false
search_exclude: false
hide: true
categories: [tweets]
---

{tweets_text}
            ''')
from IPython.display import Markdown
yesterday = t.datestamp
y_tweets = [tw for tw in tweets if tw.datestamp == yesterday]
len(y_tweets)
4
Markdown(get_tweet([tw for tw in tweets if tw.datestamp == yesterday][-1]))

*18:28:06*

@IgorBrigadir And it said that one day you will wrestle the very world snake itself! And win

🗨️ 0 ♺ 0 🤍 4


y_sorted = sorted(y_tweets, key=lambda x: x.datetime)
# [tweet.tweet for tweet in y_sorted]

Too many replies! Let's limit to just mine for now

y_md = get_md([get_tweet(t) for t in y_sorted if "@" not in t.tweet], yesterday)
len(y_md)
147
with open(f'../_posts/tweets/{yesterday}-tweets.md', 'w') as f:
    print(y_md, file=f)

Do the work

Okay, that'll do for now. It prints a chronological page of tweets for each day.

I'll wrap that behavior in a function and pass it my tweets and a set of dates when i have tweeted.

def write_day_page(day, tweets):
    tweets = [tw for tw in tweets if tw.datestamp == day]
    sorted_tweets = sorted(tweets, key=lambda x: x.datetime)
    md = get_md([get_tweet(t) for t in sorted_tweets], day)
    with open(f'../_posts/tweets/{day}-tweets.md', 'w') as f:
        print(md, file=f)
self_tweets = [t for t in tweets if "@" not in t.tweet]

len(self_tweets)
661
days = set([t.datestamp for t in self_tweets])

len(days)
33
from tqdm import tqdm
for day in tqdm(days):
    write_day_page(day, self_tweets)
100%|██████████| 33/33 [00:15<00:00,  2.12it/s]

I would also like to do analysis to see how often I tweet, and other facts. And maybe make a big list of links. Maybe next time.

For now you can find these secret tweet archives by searching in the Explore page. The days archived this time are as follows.

days
{'2021-09-18',
 '2021-09-19',
 '2021-09-20',
 '2021-09-21',
 '2021-09-22',
 '2021-09-23',
 '2021-09-24',
 '2021-09-25',
 '2021-09-26',
 '2021-09-27',
 '2021-09-28',
 '2021-09-29',
 '2021-09-30',
 '2021-10-01',
 '2021-10-02',
 '2021-10-03',
 '2021-10-04',
 '2021-10-05',
 '2021-10-06',
 '2021-10-07',
 '2021-10-08',
 '2021-10-09',
 '2021-10-10',
 '2021-10-11',
 '2021-10-12',
 '2021-10-13',
 '2021-10-14',
 '2021-10-15',
 '2021-10-16',
 '2021-10-17',
 '2021-10-18',
 '2021-10-19',
 '2021-10-20'}