I analysed 20 years of my chats
Recorded: May 28, 2026, 6:03 a.m.
| Original | Summarized |
1.2M Messages to Obsidian - Building a Relationship Map from 20 Years of Chat History Skip to content Vadim Drobinin Home Am I a Bad Friend? 27 May 2026 My digital history Data archives Drowning in noise Which Sasha Parsing them all Training the prompt Directional sentiment What the data shows Wait but why? On this page My digital history Data archives Drowning in noise Which Sasha Parsing them all Training the prompt Directional sentiment What the data shows Wait but why? In 2014, Tim Urban of WaitButWhy published Your Life in Weeks - a grid where each square is one week of one's life, and most of the grid is already filled. The image bothered me for years. I started tracking things partly because of it - I wanted the grid to mean something, not just count down. But the biometric data is an odd representation of how fulfilling my life has been. The grid suggests it's the events that matter - jobs, trips, schools, marriages - and those are easy to mark. But they hardly tell how I felt during those weeks, or what I was like to the people around me. That was what I wanted to measure. My notes and their connections growing over the years. Tired of being bad at maintaining relationships[1]1. Not bad per se - I just procrastinate a lot. Once I learnt to shoot and stalk deer because I wanted to cook a steak - and cooking is way easier than human interactions. and wanting the data to compensate, I set off on a quest to build a personal CRM of sorts, built from the record rather than from memory - thanks to the trail left by my prolific time-wasting on the Internet for the past few decades. ICQ, IRC, DC++ in 2000s: midnight channels for script kiddies and banter - all gone, and probably for the best. The ten-year-old I was in those chats doesn't need a structured archive. Armed with GDPR and data access laws, I got myself archives with all my messages, reactions, and social graphs. Bars: new unique words per year (never used before). Line: those new words as a share of that year's total vocabulary. 2016 has the most new words but a low novelty rate because the total vocabulary that year was enormous - I guess I was very social. With the noise filtered, the cleaned messages need classification: what's a life event, what's banter, who's being mentioned, what's the emotional temperature. But before any of that, there's a more basic problem. Need a mobile engineer who's this obsessed with data? Directional sentiment ¶ 5,695 conversation-days. Each bubble = one sentiment tag. Bubble size = number of days. Given the dataset that's mostly friends and family, 66% M:warm was expected[8]8. What I didn't expect is finding out that on average, 12.9% of my conversations each month are transactional - but in March it's 17%. I have the UK tax-year-end to blame.. The interesting data is the change over time. A friendship shifting from M:playful to M:transactional across 18 months is drifting, and I don't think that's something I could notice one conversation at a time. Monthly messages with close friend, 2013-2020. Interestingly enough, the amount of messages might drop, but their average length could increase too - so the friendship doesn't die, it just changes shape. Average message length per sender across three relationships. If not messages and their length, then what? Consider vocabulary overlap - in some of my relationships it went from 69.5% of our most-used words in common to 8.7%[9]9. Jaccard similarity of each person's top-100 words with mine, measured yearly. The divergence tracks the relationship cooling - we stopped talking about the same things.. We now use almost entirely non-overlapping vocabularies. A random message from either of us could be trivially attributed by word choice alone. Vocabulary convergence across six relationships. Does it mean we're not friends anymore? Not necessarily, but it does mean our interests differ way more than before - which might be a good thing as well. Session count and messages per session, partner vs close friend. I also had a look at response times, but they tell you more about someone's phone habits than their feelings. The delta across chats is small and grows with reply length - a better metric would have been time from receiving a message to reading it, but none of the exports give me that. Median response time across four relationships. Ironically, on average I reply to the first message in the session with a large delay, but then reply to all follow up messages faster than every other person in my dataset replies to me - do I have too much time or nothing else to do? The gap is negligible for my partner and closest friends and grows for peripheral contacts. Question rate per relationship. The partner line gently declines as Pierson predicted. Every thinning relationship shows the opposite. Many people who tried analysing their chats before built cloud tags - something I hope I won't have to see ever again - so I focused on endearment frequency instead. It works well with the partner chat - there are three clear eras of vocabulary, each tracking certain stages of relationships and living arrangements: Endearment frequency in the partner chat, 2016-2026. Longing and love declarations give way to dimunitives and pet language, and then give way to approval signals and jokes. When you're apart, text carries the emotion, but when you're together, it mainly captures the logistics as the emotional things are said in person. People whose last-ever message was in that year. The median friendship in the dataset lasts three years, but 41 people went silent for 3+ years then reappeared. Dunbar's layers measured longitudinally. My support clique and active network shrink over years. But the total conversation-days per year stayed nearly flat at ~360 for six years straight - the pandemic didn't change it either, although it felt like they'd go up given the amount of coordination for binge-drinking over Zoom. Even though over years I lost 75% of my network, I didn't free up a single conversation-day, just kept redistributing the same ~360 days across fewer people. Each dot is a person I've talked to. Bubble size = conversation volume. I would have described myself as "the supportive friend." The data says I'm equally "the advice friend" - mentoring (50 asymmetric days) nearly matches supportive (59). When someone needs me, my reflex is to explain, not to listen. I didn't know this. Sentiment on travel days versus at-home days. I can also clearly tell when I had a 9-to-5 period of my life: my peak messaging hour migrated from midnight to midday. Message timing by hour, 2013-2026. Evening-active people tend to have more friends but of lower quality[11]11. Roy et al. (2021), "Chronotype of social interaction", Scientific Reports. They classified chronotypes from call-record data and found evening-active users have wider but thinner social networks.. During my "midnight" years I had ~300 active contacts, but throughout the "midday" years it dropped to ~60. Clusters of memories have a shape, so I figured I should walk through itConstellationWalk through your notes in 3DView app. This research didn't change how I talk to people. I still default to advice when someone needs listening, still show different people different versions of myself, still leave half my contacts on unread for days. But I went down a nostalgia rabbit hole I wasn't expecting and learnt things about my relationships I couldn't have seen one conversation at a time. I now know what my patterns look like from the outside - assuming someone else would bother running their chats through the same ten-step pipeline and looking me up. Not bad per se - I just procrastinate a lot. Once I learnt to shoot and stalk deer because I wanted to cook a steak - and cooking is way easier than human interactions. ↩︎ A now-obscure social network, popular in the post-Soviet space in the noughties. I haven't been to Russia for a decade or so, but the archives going back to 2008 are still there. Gotta love totalitarian states, eh? ↩︎ Slavic languages often use a "-sha" suffix to create endearing diminutives, e.g Paul = Pavel = Pasha, Maria = Masha, Innokentiy = Kesha. ↩︎ Fine-tune a BERT model on labelled name-resolution pairs, predict which "Sasha" based on surrounding topics. ↩︎ BERT tops out at 75.6% F1 on event detection (Xi et al, MUSIED 2022) with a professionally annotated corpus in a single domain - I suspect multilingual banter with a small hand-labelled training set would do much worse. ↩︎ In total I ran 200+ sessions, roughly 15-20 billion tokens including context. On Opus, that's around $15k. On an M5 Pro 32 GB running Qwen3-30B-A3B locally via MLX, it's around 10-15 weeks of continuous inference. Pick your poison. ↩︎ Poria et al. (2019) showed you can't assign emotion to a conversation without tracking who said what. A message from Person A might read as angry in isolation but is sarcasm in context of their usual tone with Person B. ↩︎ What I didn't expect is finding out that on average, 12.9% of my conversations each month are transactional - but in March it's 17%. I have the UK tax-year-end to blame. ↩︎ Jaccard similarity of each person's top-100 words with mine, measured yearly. The divergence tracks the relationship cooling - we stopped talking about the same things. ↩︎ Emma Pierson analysed 5,500 emails in a long-distance relationship and noticed that questions declined as the relationship matured. My data confirms this for deepening relationships but shows the opposite for thinning ones. ↩︎ Roy et al. (2021), "Chronotype of social interaction", Scientific Reports. They classified chronotypes from call-record data and found evening-active users have wider but thinner social networks. ↩︎ ← My Phone Replaced a Brass Plug GitHub © Drobinin Limited, registered in Scotland (SC786293). |
The process of analyzing two decades of personal chat history into a structured vault represents an effort to understand emotional bandwidth, endearment cycles, and friendship dynamics. The motivation stemmed from a desire to give meaning to life events, which led to tracking physical milestones, but the author realized this approach failed to capture the nuances of interpersonal interactions. Initially, journaling provided some record, but it proved insufficient for capturing forgotten conversations and subtle relational patterns. Consequently, the author sought to build a personal Customer Relationship Management system based on digital records. The digital history gathered spanned various online platforms, including ICQ, VK, Twitter, Facebook, Instagram, and Telegram, necessitating parsing heterogenous data formats. The process of extracting meaningful information from these archives involved dealing with significant noise, as a large portion of the messages consisted of filler words, emojis, and links. This required sophisticated filtering techniques, involving sampling from offset positions, frequency counting short tokens, and manual review, resulting in a cleaned corpus. The novelty rate of vocabulary showed a plateau, indicating that most of the author's vocabulary was established early in life. A major challenge in structuring the data was resolving identities, particularly nicknames and diminutives, which varied across platforms and languages. The author noted that names like "Sasha" could refer to multiple individuals depending on context, requiring advanced methods beyond simple heuristic or name entity recognition models. To establish a comprehensive profile for each known individual, the author considered fine-tuning large language models, such as BERT, on labeled name-resolution pairs, although the necessity of hand-labeling data presented a significant hurdle. For extracting life events and relational data, the author initially attempted keyword matching with named entity recognition but found it produced numerous false positives. This led to experimenting with large language models for classification and entity resolution. The method involved running numerous sessions and processing chunks of messages to generate structured JSON manifests containing daily notes, sentiment tags, entity profiles, and life timeline events, alongside tracking unresolved ambiguities. A deterministic script then used this output to populate the final vault, ensuring that all data provenance was meticulously tracked back to the original source messages via an SQLite store, allowing for surgical rollback if necessary. Shifting focus to emotional aspects, the author recognized that standard sentiment analysis is inadequate for understanding relationships, as emotion is context-dependent. Simple polarity tagging fails to capture the asymmetry between individuals. Therefore, the author developed a more complex tagging system involving eighteen sentiment tags and directional prefixes reflecting the emotional state of the sender, the counterpart, and the mutual feeling. This analysis revealed that while the expected relationship sentiment among friends and family was mostly warm, the analysis uncovered shifts in transactional versus relational dynamics over time. Further analysis focused on communication patterns. Examining message volume and length across different relationships revealed that the quantity of messages did not necessarily correlate with relationship longevity; instead, message volume might decrease while average message length increases, indicating a change in the shape of the friendship. Vocabulary overlap measurements, such as Jaccard similarity of top words, demonstrated a divergence in interests and shared language as relationships evolved, suggesting a cooling of the connection. The analysis of interaction metrics also provided insight into relational changes. Interactions with a partner were characterized by thousands of micro-check-ins, whereas chats with close friends involved fewer, more thoughtful sessions. Response times were noted as a less reliable metric for emotion, but the author observed that reply patterns varied significantly across relationships. A key finding was that while emotional contagion did not cross threads, the rate of questioning served as an inverse proxy for relationship bandwidth; this rate increased in thinning relationships, suggesting that less frequent communication correlates with increased information-seeking behavior. The study also highlighted social chronotypes, finding that evening-active individuals tended to have wider but thinner social networks. Furthermore, the author observed a shift in social behavior correlating with daily routines, noting that peak messaging hours migrated from midnight to midday. Ultimately, the experience underscored that while the process was an exercise in data science, the result provided a powerful, observable understanding of interpersonal patterns, suggesting that a better understanding of these dynamics is essential for being a supportive friend. |