TinyDB September 2023 Update

As we are nearing the end of Summer 2023, some of you might have been hoping for some news about TinyDB, a ScalaJS-based relational database in your web browser with spreadsheet-like usability. There is not too much to talk about today, but to give you an idea of what to expect I am hoping to clarify some scheduling updates as time has marched on.

Where is TinyDB?

Worry not; TinyDB is still coming, and you will all be able to use it later this year. Originally I had expected TinyDB to be available in a limited beta format by this time, but there have been some delays due to various factors. Nothing complicated with the code, actually, but it would be more accurate to say that I simply have not had the time to work on the project as much as I would like. The past several months at my day job have been quite hectic and things show little sign of slowing down, so at the end of each work day doing even more coding staring at my computer screen is often the last thing I want to do. So for most of the past few months I have been taking some mental health time, spending some quality time with my four beautiful pets (two of which are Australian Shepherd dogs, known for their impossibly high energy levels and demanding personalities). Besides this, I’m sure you can sympathize with how the latest and greatest video games have been coming out on a near weekly basis this summer and also eating up what little free time I have on the weekends as well. Baldur’s Gate 3 and Starfield will eventually come for us all.

So, recognizing this, I am going to start ensuring that I at least have something to show for the project moving forward up to twice a month. Starting in October you will see an update on everything I have been able to accomplish during the month of September, and there might be a second update as well later in the month as more developments and plans are made.

TinyDB Roadmap

So youy might be wondering, what should you expect from these updates? What features have yet to be developed, and what milestones are there right now? I have put together this loose outline of the features I will be working on, and approximate levels of priority for when I hope to get done with them.

• File Format Specification - September 2023
• Early Mobile UI Support - September/October 2023
• Early Beta Access - October 2023
• Additional chart types - 2023
• Grid Layout Mode - 2023
• Cloud Storage Support - Late 2023 / Early 2024
• Public Beta Access - Early / Mid 2024
• Triggers - 2024
• More Functions & Custom Hooks - 2024
• Funding & Sustainability Plans - 2024?

As you can see, most of these features I’ll try to implement some time this year, and the early Beta access will actually start before almost all of these features and more will be implemented. This early version will be mostly equivalent to the version developed for ScalaDays Seattle 2023, to start, with some minor enhancements to make sure it works at a basic, feature-complete level for normal database tasks. You will have tables, some basic functions and queries, JOINs, very basic chart support, basic mobile UI, and the ability to save your database to a standard file format. In fact, the file format right now is basically a dump of the memory contents, which is not optimal for file size or future feature support. Getting that file format standardized is currently my highest priority and the main blocker for getting a version of this app available for people to use.

What about that donation?

Oh, yes, of course. I have not forgotten about my pledge to donate on behalf of any and all early supporters who have subscribed to the TinyDB email list with a special secret code. A few of you were a little colorful with your choice of code and not everyone inputted the code exactly the same, but rounding out the list and fudging the numbers a bit to account for those inexact codes, it looks like we have a total of fifteen pledges, meaning I have donated $150 to The Trevor Project.

Thanks to your interest in my project this money can go towards helping prevent LGBT+ suicide, in a time when so much uncertainty, bigotry and hatred out there for people who want to peacefully be themselves. I and the TinyDB project are happy to put our money where our mouth is and be a proud supporter of LGBT+ lives whenever possible.

How can I hear more?

Go to TinyDB.app today and enter your email address into the form to get early beta access within the next month (or two, if there’s another delay). You will also get email notifications every time there is a feature or development update. You are free to unsubscribe at any time.

Thank you everyone for your understanding and support. I look forward to getting TinyDB into your hands very soon!

Introducing TinyDB

TinyDB is an application I have been developing in various forms for a long time now, that I am finally able to start showing off and announcing plans for public availability. This month at Scala Days Seattle 2023 I was pleased to be able to demo the application and present about how the development using Scala.js was going along. Due to the focus of the talk being primarily about the tech used, and how useful it has been to me, I would like to take some time to actually kick-start the marketing hype cycle for this small project and let it be further known.

What is TinyDB?

TinyDB is a browser-based database application that is as easy to use as a spreadsheet. You do not need to worry about preconfiguring data schemas or types to get started, so you can use it as a basic, text-based data entry program without any configuration. With TinyDB, I hope to allow you to opt into the level of control that you need over your data, while doing so in a way that is portable and easy-to-use from any device, whether it’s your desktop computer, laptop, tablet, phone, or any other device with a modern web browser.

Some (super unfinished) images

To give you a quick idea of what this thing will be capable of at a glance, I’ve taken some development screenshots. Unfortunately I have not put a lot of effort into the CSS, so it’s a bit ugly right now but I will produce more videos and blog entries as this gets developed.

Table Display

Each table is a grid of buttons that, when clicked, lets you edit its contents much like a spreadsheet cell. You can edit column information by clicking the column, including the type of column, custom formatting, allowed data values, title, and so on.

Custom Enumerations

You can set columns to use custom enumerations of values, which let you ensure that invalid values are impossible. This example shows configuring an enumeration type for video game platforms, i.e. PlayStation and PC.

Queries

This is a very-in-progress dialog window for constructing a query. You are able to select a host table in the FROM section, JOIN it to another table, aggregate, group by columns (UI not implemented yet), select values based on a custom expression language, and filter your queries just like you would using SQL.

Charts

This is an example chart showing the ability to turn any table you generate into a visualization. You are able to select which columns represent which axes or data inputs. For this example, I have a pie chart showing the number of copies I own of three video games.

What can TinyDB do?

TinyDB can do all of the typical tasks you would associate with either a spreadsheet or a database. For those familiar with relational SQL databases, this includes:

• Queries with a basic expression language
• Aggregations & grouping
• Table joins

This will also support a number of features designed to make it easy to control your data, including:

• Custom formatting options for columns
• Enumeration support (fixed sets of “valid values” you select in a drop-down box, i.e. “days of the week”)
• CSV data import
• Custom column rules

Visually, TinyDB resembles a spreadsheet, so you should be able to do things that a spreadsheet will let you do. While it is not as freeform as, say, Excel, TinyDB will support reactive programming much the same way as a spreadsheet, by enabling you to compute the values of both columns and entire tables through reactive queries. This will allow you to have tables where, say, column “D” is the sum of the values in columns “A” + “B” + “C”. It will also allow you to aggregate and produce results from tables that frequently change. For example, lets say you have a table with all of your monthly expenses in it. You can have a “reactive query” that represents an aggregation of that table, and every time you add a new expense it will sum it up into the total for you to see.

TinyDB will also support various kinds of charts with customization options for the kind of chart, how you source its data, and how it should be displayed.

In a future release I plan on supporting more features that I am presently laying the groundwork for, including:

• Trigger functions (actions to execute whenever a row is inserted into a table)
• JavaScript-based “hooks” for your own custom logic
• More types of charts
• Grid-like interface for seeing multiple tables, queries, and charts at once
• Toggleable “foreign keys” to help with relational data entry
• Custom data entry interface support
• …And more!!!

Example use case & a brief history

Starting back around 2017, I wished to create something to help manage my multimedia backlog – games, movies, books, TV shows, you name it – while also approaching the complexity of something like IMDB, TMDB, or TheTVDB.

This proved to be quite an investment, and not something I was able to commit myself to for a long time without burnout.

I decided to try using an Excel spreadsheet instead, which, for the most part, worked fairly well. I was able to organize my media in as many tables as I wanted, with the columns and data types that made sense to me, and creating something that I could edit from my desktop as well as my phone or tablet. Unfortunately, I ran into a serious snag, and that was that I expected to be able to join tables together in my browser. Neither Excel nor Google Sheets really support that feature for the web-based spreadsheets they offer, or the features they did offer were otherwise inadequate for my needs. I wanted to be able to define multiple tables that I could cross-reference and combine with each other in interesting ways to categorize and calculate statistics about my backlogs, and this simply was not possible with Excel.

Lets say that I would like to keep track of how many games I own, per-year, per-gaming platform. The trouble with storing this data in tabular form is that to do it manually, it requires a lot of cross-referencing that your computer should be able to do for you. So, lets conjure up some normalized data to illustrate my point:

This table, lets call it table “A”, has two games and their original release year. You could say that this table is the “original release year for a game”. Now, lets say we also have this table:

This table, table “B”, contains the platforms that a game is released for. The reason these are two separate tables is because the data in each table scales differently per-type. Every game only has one “initial release year”, but not every game has the same number of platforms.

I would like to query this data and find out, for each game, how many releases of it I own, and the initial year it came out. In SQL, you would write a query something like this:

SELECT A.Title AS Title, A.Year AS Year, SUM(*) AS Count
FROM A
LEFT JOIN B ON A.Title == B.Title
GROUP BY A.Title, A.Year

And it would produce the following results:

Trying to do this kind of aggregation and combining from multiple tables is simply not possible without leveraging relational data. Otherwise, you could get inconsistencies, or require complex application logic in order to ensure the data looks the way you expect. When dealing with manual data entry, with reasonably small data sets (as in, not on the order of zillions of rows, something that can fit in memory of a single machine) then relational data techniques are extremely powerful, and they allow you to turn what is otherwise a painful, error-prone process into something that your database can do for you.

When I was using spreadsheets, I kept running into this limitation all the time when using my phone to edit my tables with data on my media collection and backlog. My phone is powerful; it is 2023 after all. I should be able to, with a data set as reasonably-sized as mine, do queries with JOINs and aggregations and other sorts of expressions without needing to use a full-fledged database hosted on a server somewhere.

That is the promise of TinyDB.

When can I use TinyDB?

I am planning for beta access to start by some time in August 2023. Right now, the app is undergoing some major facelifts and does not support certain core features yet that are essential to the experience, so I am working on finishing those up getting this polished as soon as possible.

If you would like to get notified for the instant this is available, and possibly even get early beta access, you can go to tinydb.app right now and fill out the email subscription form.

What if I also want to subtract $10 from your wallet?

Luckily for you, I am running a promotion in honor of it currently being June, aka “Pride Month”. If you go to the above-mentioned email subscription form, you will see a field named “secret code” that is marked optional. If you specify “transrights” or some close-enough variation in the text box when you sign up, I will donate $10 to The Trevor Project in your honor as a way of saying thanks, and to help support trans kids. I have set a soft limit of $500 since I’m not made of money, so hurry up and add your name to the list by the end of June (or however long I want to honor the promotion for) and get it all sorted out.

Does TinyDB cost anything?

I would like all of the basic features I described of TinyDB here today to be free for all users, forever. I do not ever plan on limiting access or shoving intrusive ads into the experience. I would like to be able to offer certain features that go beyond simple database tasks in a browser, however, that might need a cost attached in order to do them sustainably. Depending on how popular this gets, I might also look into crowdfunding options for prioritizing support requests and enabling early access to features, for example. We will see.

Will it be free/open-source software?

Parts of it, yes, probably, maybe. Right now it is fully closed-source but I do intend to give parts of this an applicable license that permits free use and integrating with other software. I am still doing research into which license is best for my needs at this time.

Please share, get the word out, and I hope to start showing off new features and functionality as I develop them over the coming weeks!

Women Belong In Tech

Women belong in tech.

It’s such an obvious statement. Of course women belong in tech. Who would dare to disagree? And yet, here I am, at around ~7:30am on a Friday, writing about this. I should not be here. My first meeting of the day is not for another 2.5 hours and most of my coworkers are in a different time zone so I should have been able to get a little extra sleep in, but I am unexpectedly awake, and utterly furious.

I have not written a proper blog post in over three years. Originally, when I started this blog, I told myself, “I will write a new post every month this year” over and over, but nothing would ever come out. So after all this time if I’m actually writing about something, you know it’s actually important enough to get me off my ass.

So, women. Why are we talking about women? Because women have to regularly deal with such mountains of absolute disrespect that I’ve never had to face. Because women have to be better than men to even stand a chance in this industry, whether that’s being paid appropriate wages, being taken seriously at work, not being sexually harassed or objectified, or not having obscene double-standards applied to them every which way they go, because they are women. As if being a woman should have anything to do with one’s technical ability. Ada Lovelace, a woman, is commonly considered the world’s first programmer, and yet by and large this largely male-dominated industry fails to honor that legacy daily, by subjecting women to the most heinous and utterly indefensible situations that no person should ever have to deal with.

Women have not been silent about this. For the better part of a century now, you can find countless examples of women recounting their horrible experiences with bosses, coworkers, other men in their industry, and of course, the educational system. It’s not a surprise that only twenty percent of computer science graduates are women, despite women making up the majority of college graduates in general, when you consider just how unfairly treated women are in this space. From my own anecdotal experience, the vast majority of my classes at Purdue University Northwest were pretty evenly split between demographics, except for my computer science courses. Not only were the vast majority of my classmates male, but they exhibited common toxic rhetoric and behavior of the day as much as you could reasonably expect. I’ll even admit, I was occasionally there alongside them, saying things such as, “maybe men are just more predisposed towards these kinds of careers than women,” or even using my education and career choices as a way to feel smugly superior to others (i.e. liberal arts majors).

This was wrong then. I was wrong then. I was also only in my late teens and early twenties. Now, nearing my thirties, I have nothing but disdain for that kind of person and those popular talking points of the day. Still, these injustices persist, even amongst people I would otherwise respect or admire in this industry. You do not have to look far to find the average “tech-bro” figure, you know the kind. Maybe they developed some library or program many years ago that people still use to this day, or maybe they wrote a book or two that people recommend without a second thought. Maybe they founded or currently lead some company, or sit in a prestigious position, regularly giving talks and being an icon in their small corner of the industry. So you follow them online, hoping for more nuggets of wisdom, only to find that they exhibit some unusually regressive politics. On occasion, you might hear them say something small that sits wrong with you, or see some kind of decision they make in a position of authority that upsets you. But you tell yourself, “this person is well respected. People love this person. Surely, I should be giving them the benefit of the doubt, and not cause a scene. I should know my place and leave the experts to it.”

In the midst of this, you might see some people decide they’ve had enough. Maybe they are women, or people of color. Maybe they are white men who are upset on behalf of these marginalized groups. You don’t fully understand it. You have not been in their position, with their life experiences, and you could not possibly imagine the sheer scale of what years of microaggressions and other toxic behaviors will do to your ability to unquestionably tolerate others. All you ever see is maybe “some questionable opinions” at worst. And when these people start cutting ties, or they dare to hold these people accountable, you wonder why people are trying to “cancel” this figure. Surely, we should all just get along, and these rabble-rousers are the true source of the problem, yes? It’s all those people who choose to be upset in the first place who are the problem. Not I, a man who has never lived these lives. A man who has been happily coasting through most of my career, never worrying too much about whether I was being taken seriously, except during perhaps those early months or years when you’re still new. “See,” you say, “it happens to everybody. Women do not have a monopoly on feeling unappreciated in tech.” And so on, and so on, as you begin to rationalize for yourself why it’s totally okay how you are not nearly as angry as these other people.

So, “of course women belong in tech”, you might say. “But…”

This line of thinking, this toxic line of thinking, is exactly the problem and what needs to be addressed if we are ever going to hope for a more equal workplace for women.

We Need To Talk About John

At the risk of making this article sound like it’s exclusively about a single person and a single recent event, I’ve already been tweeting publicly about this so it’s no use pretending that anything else was the inspiration for today’s blog post.

John De Goes is a notable figure in the greater “Scala programming language community”, which is to say he has been around for quite some time, contributing to various projects of much renown, as well as being the co-founder of his own company doing Scala consultancy and development: Ziverge. They help coordinate, among other things, the development of the increasingly notable “ZIO” ecosystem of software libraries, as well as provide their own consultancy services and run conferences.

And as fortune would have it, John has been so kind as to give us all a brilliant demonstration of the sort of issues I’ve outlined above.

I am not going to bore you with paragraph after paragraph about what exactly the entire deal with this person is. Every single minor or major quabble over the years, every disagreement or questionable decision or rivalry. Even if you do not use the Scala programming language, however, you probably know “a John” in your own communities. Nearly every community of sufficient size has at least one notable figure who seems unusually divisive from a distance. Maybe you don’t pay too much attention to fine details. Maybe you just want to grill, for God’s sake. You don’t have time for community drama, and I respect that. There are more important things in life, after all, than wasting every precious moment by obsessing over specific actors in specific communities and their actions. It is not a healthy way to live.

Reader, I tell you: nearly every single thing I have ever learned about people along these lines has been against my will, and likely you might feel the same way about other people who you would much rather forget about. Not that I have been tied down and force-fed any of this, mind. I have had every opportunity to just “log off” and forget about the more problematic occurrences related to this sort of community drama, regardless of who is responsible or otherwise at the center of it. I am burdened with the gift of this knowledge, the understanding of the actions and words of specific people at specific places and times, and life would be so much easier if I could simply forget. If I could solely focus on doing my job, spending time with my family, and worrying about nothing else for the rest of my life.

Sometimes, however, against my desire to be free from this hell, I am brought back in, kicking and screaming, to observe once more the latest controversy related to some person or other, whether that is John or somebody else. Today’s specific situation, is the subject of what I believe are outright sexist comments made against a woman in tech, and the kinds of people who see nothing wrong with that.

The (Relevant) Timeline

To make this as easy to digest as possible I’ll save you from needing an entire college course worth of Scala community history and drama, and we will approach this situation on its own merits. (Though, I am happy to discuss other details privately, or on future occasions, as the need arises.)

In Scala, like many programming languages, there are several companies and organizations that collaborate on community-focused projects such as tooling or core libraries. Scala has a bit of an academic background, including the language’s creator Martin Odersky, who currently heads programming research at EPFL (in English: “Swiss Federal Institute of Technology Lausanne”). At EPFL, they have a non-profit center for coordination of the Scala language named the Scala Center. Here, various tools and programs are developed and organized by some extremely well-respected and talented contributors.

I do not have a personal relationship with anybody working at the Scala Center, for the record. Maybe I follow some of them on Twitter, and maybe a couple of them follow me back as mutuals, but we never directly communicate. I’ve only ever observed them from a distance, and perhaps people at my current place-of-business do work with them more frequently (for example, planning conferences and the like). Regardless, I should not have to be familiar with them personally, or professionally, to see the problem with the events I am about to describe.

Similarly, I am not personally or professionally familiar (to the best of my knowledge) with anyone at SoftwareMill, a Scala and Java focused consultancy that provides contractual services for clients such as software development, dev-ops, and so on. SoftwareMill is hosting a conference later this month in Europe called “Scalar Conference”, a place for Scala-focused industry professionals and academics alike to gather, present talks, and socialize.

John De Goes was set to present at Scalar Conf, and in the lead up to the conference he posted a series of tweets critical of Scala Center’s direction and resource allocation. Of note is the fact that Ziverge, his company, sought to sponsor the Scala Center, but they refused this sponsorship for their own private reasons.

Here is where the sexism finally hits: John shares a clip, originally missing context, of the current executive director of the Scala Center having a brief outburst of profanity during a talk. John says the following:

The Scala Center’s internal org chart raises many questions: the Executive Director of the Scala Center has no background in either business or engineering (essential for this job!), and their communication skills have substantial room for improvement.

The audio of the clip is transcribed as follows:

So, whoever it is, I challenge that person to grow some fucking balls and come speak to me. So thank you very much for those who understood, and those little pussies, you are just fucked up.

Following this, Scalar Conf announced that John’s talk will no longer appear, as they felt his recent public activity crossed a line of personal attack that should not be crossed.

Breaking it down

Right. So. Apparently it has to be explained to people exactly what is wrong with what John is doing here, as there are several people both in the replies and quote-tweets to Scalar Conf as well as on John’s twitter timeline who are arguing about this.

Lets break it down, piece by piece, by gradually adding context as necessary.

On Meritocracy

First, we have John questioning this woman’s credentials. As I made clear in the introductory segment, women repeatedly face issues of toxicity in the workplace every day, especially in programming and general tech which is a male-dominated field. Regardless of whether someone is a woman or anything else, I find questioning someone’s credentials to be particularly tactless in most cases. There are situations where I feel it is totally appropriate to question someone’s ability to do their job, but usually you should do not do that sort of thing by looking at their educational or business background, except perhaps if you are hiring. This person is, presumably, well-respected and trusted by her peers, or at least I have zero reason to assume otherwise. She achieved her current position, which is admirable, and absent evidence to the contrary she is likely doing a perfectly fine job at the stated goals of her position, at the very least. Barring, of course, any of the more specific criticisms of the Scala Center that people can leverage against them, which are not themselves personal attacks, but did lead into them.

Reaching back a bit into the sort of toxic dialogue I was exposed to back in college, I distinctly remember there being an increase in hatred towards “diversity hires” or the very idea of it. If a person in any position was not a white straight male, and anybody had any criticisms of their work whatsoever, people would jump to the horrible conclusion of saying this person was only hired “because they are a woman”, or “because they are black” and so on. In tech especially, it is generally assumed, as it is a male dominated field, that the average man knows what they’re doing. Men respect each other and by default they see each other as equals. If a white man in tech is not exactly excelling at his job, he is not ever called a “diversity hire”. He is the “default”. He is the “status quo”. He does not need to be questioned nearly as hard. Hell, the average man would likely have sympathy for someone having difficulties at their job, rather than assuming straight away they are incompetent. A woman, however, has much more to prove. People are inherently skeptical of their abilities and understanding of what’s necessary for the job. Even among women and people of color, these habits are so pervasive that we might not even recognize when we are exercising them, or letting them subconsciously bias us towards certain conclusions.

Is John literally saying she is a diversity hire? Not with words, no. But the sentiment is clear. She is a woman, who is in a position without the “right credentials”, so she is inherently suspect. Ask any woman in tech if they have ever been grilled about their know-how, their credentials, their ability to perform basic tasks any person should be capable of. Even with the “right degrees” or the “right background”, women still do not get taken seriously, and that is even besides the point that the background does not matter one bit. If John were more honest, he would recognize that there are numerous talented and skilled professionals in nearly every field who are either self-taught or pivoted from another career or educational background. He frequently mentors and instructs people lacking those qualifications as well, so on many levels, he should know better.

To judge a person for their background at all in situations as low-stakes as this, is to question whether or not they belong here. It denies the herculean efforts that people of all kinds, particularly women, have to make in order to be even considered for the same sort of roles and positions that the average man is. Recall I said earlier that women have to be even better than the average man in order to be taken seriously. Now consider that, if you are a woman and you are doing everything you possibly can to succeed in this industry, that you will still be questioned as an outsider, as someone who needs to “prove themselves” no matter what.

We have a stylistic preference for men as a society, not based on any good reasoning, but based on institutionalized sexism. A relevant quote:

We live in a sham meritocracy, where we pretend to pick the best person for each job, while simply picking those we prefer: and when the jobs pay well, they are still overwhelming male. Our preferences are based on style rather than substance, so we pick individuals for leadership on the basis of their confidence rather than competence, charisma rather than humility, and narcissism rather than integrity. For every Angela Merkel, there are many Silvio Berlusconis, Jair Bolsonaros, and Donald Trumps. Not just in politics, but also in business, the typical leader is not known for their humility or competence, but arrogance and incompetence.

Is it ever wrong to criticize someone who happens to be a woman? Absolutely not. I probably do it every other day on social media if I had to guess, or would if I couldn’t stop myself by finding better uses of my time. That is not what is at issue here, however. A woman exists in a position where she is trying to make a difference, and that criticism crossed a very clear line that questioned whether or not she belongs in this boys club, by dogwhistling the sort of diversity hire rhetoric you frequently find in hateful right-wing circles. Regardless of whether or not that was the intent, it certainly comes across that way to people who have been paying attention to these sorts of issues for the better part of the past decade.

On Profanity and “Professionalism”

If you follow me on Twitter you might now I fucking swear quite a fucking bit. I’m not afraid of it. I feel it is occasionally quite relevant to certain discussions at hand, and it is a succinct way to express excitement and anger all the same. Even still, there is a time and place for everything, so while you might not swear in church or while conducting an important meeting, that switch might flip in less “professional” settings.

So what exactly is “professionalism” and why does it matter here? John is accusing this woman of being unfit for her job, partially due to this outburst, as it comes across as “unprofessional”. Again, without going too deep into the weeds here, John in particular has a particular affinity for this idea of “professionalism”, even naming one particularly poorly-thought-out “code” after the concept. The general idea behind a lot of John’s public-facing statements and interactions has been to reinforce this idea of being a “professional”, as in, someone who does not give in to “politics” and infighting, slinging insults between parties. He positions himself as trying to best exemplify these ideals, though it would be all too easy to call him a hypocrite on that front (more on that later).

Respectfully to anyone who might be so inclined to be offended by this particular profanity outburst that we are discussing: your feelings here do not matter, and they are besides the point. Trying to tone-police women for a singular outburst of profanity, regardless of context, comes across as if you are judging people (women) for the “crime” of being human and having emotions. You are not her boss. You are not her friend. You do not need to comment. Situations like these are dealt with by the appropriate parties, at the time they occur, and don’t need your input.

It goes without saying that you are not supposed to say those kind of things in a professional setting without good reason. So you might be asking, what is the reason? The continued objectification of women by men who should know better.

The clip that was linked originally did not have context posted, but it was shared before being taken offline for privacy. When faced with the sort of harassment that these situations usually result in, I cannot blame anyone for locking down their accounts to ride out the storm, or removing/making private videos or content that is relevant so it does not receive dozens of hateful comments.

The outburst was in response to a female presenter being told, during this talk, “You are pretty!” by someone anonymous. Here is one attendee of the talk in question who raised concerns that very day, correctly pointing out that, “…[this] may be part of the reason why women don’t feel as welcome in IT as men do…”

(UPDATE: The above paragraph was edited to clarify that another woman, not the now-director of the Scala Center, was the recipient of objectifying remarks. For the record I was not present at the event in question so I have had to slowly piece together information from anecdotes and first-hand witness accounts.)

Now, I know what some of you are thinking: “That comment does not seem so bad”, or, “It is bad but it is not nearly as bad as the outburst that came after”, perhaps. I know why you feel this way. Maybe you are a man who has not had to deal with men cat-calling you, or trying to get their grimy hands all over you and your body by any means necessary. Contrary to what you might have been told as a small child, it is not, actually, always the best idea to say these kind of things to a woman. Women deal with objectification on a daily basis in ways both large and small. Imagine being a grown adult woman, having dealt with creepy relatives, “nice guy” friends who only wanted you for sex but don’t respect when you want to be left alone, sexualizing comments from even the youngest of ages, all continuing into and throughout most of the rest of your life. It can feel like you are treated less like a real, human being with your own thoughts and emotions, and more as an object, something designed to titilate those with no sense of self-control rather than being properly respected.

I am a married man. I love my wife, and I could not imagine my life without her. I can call her beautiful or pretty. I have a consensual relationship with her, and we talk about any of our issues that come up if we ever cross any boundaries. You are not married to random women you see on the street, or in conferences. You are not even in a general relationship with them. And even then, every person has their own boundaries, and their own understanding of what is acceptable behavior towards them. By default, why would you ever assume those comments are okay to make to someone who has not agreed to hear them?

That covers the objectification angle, but for some reason you might be hung up on the choice of colorful language. The thing about language like this is, absent any sort of divine punishment for speaking it, none of it actually matters. People give words meaning and power based on their context, not whether they are inherently good or not. It’s why you can watch movies with actors that curse like a sailor, objectify women, or even speak racial slurs if the story calls for it. It’s such an easy target to focus on the words without focusing on why they were spoken.

It is so tempting to look at this situation in a bottle and think that this woman, who to you might as well have been willed into existence just for this talk if you do not know them or have not grown up with them, is acting “irresponsibly” out of nowhere, that there is no possible context that would make this okay. By refusing to even pretend to understand the source of the frustration, that sometimes a single straw can break a camel’s back after so many years of this, it shows a distinct lack of empathy and understanding.

Also, it was nearly four years ago

Don’t think this has been mentioned by John or almost anybody else, but for the record, this talk occurred on or around October 7, 2019.

John’s comments, by comparison, were made on March 9th, 2023, and today is March 10, 2023, about three and a half years apart.

When we’re talking about the competency of someone to do their job, or to be an effective leader, linking to a single (justified, in my opinion) outburst of profanity from nearly four years ago (which was some time before she got the job) just makes it seem like you are digging for dirt. You are looking for evidence to use in your righteous quest to criticize someone, and not actually holding this against them in any meaningfully honest capacity. Did that outburst affect you? Have you not been able to sleep soundly for four years? Really? (UPDATE: I was able to confirm this was, indeed, before she got the job, and has maintained it without incident for the past 3+ years.)

Lets be reasonable. Who among us has not, at any point in time, ever gotten “unreasonably angry” about some situation? Maybe you cursed at someone. Maybe it was your boss, or your friend, or your family. Being angry is not an unforgivable sin. By contrast, I would argue that refusing to understand why someone would be angry or refusing to apologize if your own anger was unjustified, is actually a problem worth caring about.

Or, perhaps, this is such a small, singular incident, that even if it were a bad look (which it is not) it was so long ago and comes with so many caveats that I would be willing to overlook it on the principle that each person’s character is a rolling average of their continuous actions. If you do something bad once, and never approvingly acknowledge it ever again, for certain kinds of actions that seems like a fair sort of thing to waive. Now, does that mean I have to make nice with everyone who was personally mean to me several years ago? Absolutely not. That is my personal, private decision. Not only was this not directed at John, so he has no right to feel personally offended, but it is also so small in scale and scope that any reasonable person should be able to forgive and look the other way as if it never happened.

Do you want to know what happened the last time I had an outburst in a professional setting? I was in California for an internship. I was thrilled to have the opportunity, being from the midwest, as I could use this to bolster my resume and get a head start on my career. Unfortunately for me, the pay was lousy, and literally the only place I could afford to stay (while dipping into my savings) was an Airbnb with a dozen or so other beds. I slept in a closet all summer, I had to deal with living around over a dozen other people (each of which had their own issues and conflicts), I commuted to work in the hot San Jose sun, and after a while it started getting to me. I became angry. Not just at myself for taking this obviously bad deal of a job, but at the business for ever thinking it was acceptable to offer in the first place.

I had a meeting with the boss of this company, where I laid out my concerns, and how it had affected my mental health, hoping to find some kind of understanding. They threw the entire thing back in my face, at which point I said, and I quote, they were paying me a “fucking starvation wage”. This greatly upset my boss. I’ll never forget how he responded to me, not by defusing the situation but by telling me, “first of all, do not swear at me”. He followed this up with, I kid you not, a book recommendation. I forget what it was called, (something maybe involving the phrase “Yes, sir!”) but looking into it, I felt so offended I was without words. The premise was basically a way to condition you to be uncritically accepting of whatever your boss is giving you. Don’t stand up for yourself, it seemed to say. You are being given an opportunity and you should be thankful! Never mind the fact that $15/hr in the San Francisco Bay Area of all places is literally too poor to afford the most basic accommodations. Never mind the fact that the other intern there was also in a similar living situation as me, so they could have done their due diligence upfront instead of waiting for someone to complain.

When I think of people being overly critical of profanity, acting like it is the most unspeakable sin a person can commit, I think of that old boss. I think of how me swearing at him that one time, on the verge of tears after several days of nervous breakdowns, was more offensive than their terrible business proposition. I think of being told to shut up and take what I was given, because I suppose I “should be thankful I even have a job at all”, and should not complain about the conditions I was forced to live in if I wanted to keep the job.

And when I think of women who complain about their working conditions, or react with hostility towards even the smallest-seeming aggressions, I think of how I felt back then. I took that job because I felt it would help me in my career. How many times have you ever done something you felt uncomfortable with because it would help you in some way? Now imagine that is basically your entire life. You need to deal with your sexist boss, or your sexist coworkers, or your sexist family because it’s for the greater good. You’re not allowed to be upset. You’re not allowed to speak out, to scream, to curse. Because that is not what a “proper lady” would do. Not what an “educated woman” should do. You should know your place. You should only ever be fully professional, and you should expect to take these aggressions on the chin for your entire life, because if you speak up in any capacity, you’ll be judged for it. For causing division. For raising a fuss. For not having an “appropriate tone”.

Think of that the next time you want to so publicly judge someone for something. Think about whether or not you are capable of putting yourself in their shoes and finding common ground. Don’t make this about yourself and about how you, personally, would “not have reacted that way” or some “holier than thou” nonsense. Don’t do it, especially if you’re basically telling a woman to know her place.

Cancel Culture Hypocrisy

I mentioned I’d touch on this, so briefly, John and people like him are frequent critics of “cancel culture”, this idea that society could come to “cancel” you at any moment, for any infraction no matter how small, demanding you lose your job and whatever prestige you may have left, all because of a couple simple mistakes. A small thing that may have happened once or twice, that does not constitute the entirety of your character, and should not be held over your head for eternity no matter how long ago it was.

Sound familiar?

I do not believe that this exists in the capacity these people describe, for a wide variety of reasons, but lets use John as the perfect example of why this concept is nothing but a hot, steaming load.

John has been “cancelled” by his own mentioning or endorsing of opinions expressing this on social media, I want to say, roughly 4-6 times over the past six-ish years, at least, including this one. Each incident usually involved some conference deciding he should no longer be allowed to speak, or him not being welcome within certain private groups in the community, or some person no longer wanting to associate with code or projects related to him or his company. These are private decisions, made by private individuals and the organizations that they make up, and all stem from freedom of association.

Not every country has “freedom of speech” per se, but in most places you absolutely have the right to choose who you associate with. If I decide I no longer like a family member, I am totally free to cut them out of my life. If someone says something horribly offensive that crosses a line, and I run an event that they are invited to, it is within my rights to bar them from that event, for the safety and comfort of the event attendees.

So what is the result of John being “cancelled”? He can no longer attend a few events. He is no longer welcome in some organizations. Some people have independently concluded that they do not want to associate with him. Others do. He is still welcome in other events. He is still welcome in other organizations. He even runs his own. His company, by my understanding, has contracts with some big names. He has several employees who would presumably stand by him even if the moon fell.

That does not sound like being “cancelled” to me.

What does it mean if you can be “cancelled” and still be financially and even socially successful? Still run your own business, still run your own conferences that I assume have reasonable attendance, get enthusiastic employees and customers alike. It’s not like the internet banded together to, say, “take the wife and kids”, metaphorically speaking. Scala is far from a large community, but it is an active one. Everyone kind of knows each other, except the people who don’t need to. I’ve worked with dozens upon dozens of people who either do not care about any of this, or actively support the man. So, what exactly was taken from him that he feels he deserves?

The implication here is that if you are “cancelled” and actually upset about it, it’s solely a frustration about people choosing to disagree with your words and actions. That’s it. If a conference doesn’t want you there, they have made their case. They do not want to associate with you. Are you denying their right to say so? Well, of course not, because John has his own conferences, he has his own company, clients, and so on. If people don’t like you, you have exactly two options:

• Attempt to convince them otherwise, either by apologizing and demonstrating changes
• Accept the situation, and continue on with your life

That is not to say he has no right to criticize people who would otherwise criticize him. This happens in all directions, all the time, in every community. It’s a fundamental human right. But he is not being “cancelled” for it. He’s just upset that select groups of people have drawn a line that he is on the other side of.

…

Oh, yeah, remember when I said this:

Sound familiar?

What exactly is John doing, if not trying to “cancel” someone by his own definition? He is publicly trying to hold someone accountable for actions that occurred in the distant past, actions that many would agree are “not really a big deal” at worst, and he is implicitly stating, through criticizing her position, that she does not deserve (and therefore, should lose) her job.

“Cancel culture” for me, but not for thee, I suppose.

If John were being honest, he would recognize this and admit as much: that he is trying to “cancel” someone. John would have you believe that this is different, and not the same thing at all, however, because this time, it’s justified. This time, it’s not directed at him, but at someone else. Someone who allegedly does not like him. They deserve it, you see, for ever doing this one thing “wrong” in the first place. It’s totally normal to bring up these small things years later for no good reason other than dragging someone’s name through the mud.

Now see, here’s where John and I agree: it is, actually, totally okay and normal to bring up the bad things people have done, whenever it is relevant or couth to do so, especially if it is part of making a larger point that actually has a meaningful impact on people, such as establishing a long-term pattern of awful behavior, or questioning why people would associate with someone for what constitutes some good reason. I am heavily critical of Elon Musk for example, and I will be until the day I die. I will absolutely bring up things he has said and done within the past several years, not simply because he has failed to grow and improve, but more to the point: he continues to get worse with each passing day, saying and doing such insane and terrible things that it makes me wonder why anyone ever liked him to begin with.

On the contrary, the woman in question is not exactly guilty of any sort of crime, legally or morally. Even if you argue “maybe they shouldn’t have cursed that one time”, you have to admit that it was not only so long ago, but it was a singular event. There is no evidence to suggest that she regularly exhibits seriously problematic and unprofessional behaviors. My closest understanding of the situation is simply that this is the best thing John could find in an attempt to discredit the Scala Center, which is pathetic.

Closing & Resolutions

I would like to say something as easy and simple as, “I think John should apologize to this woman, and the Scala Center for attacking one of their own”. I would like to, but it seems futile at this stage. People such as John, not specifically but an entire group of people, these men who feel justified in wedging division between people along sexual boundaries and then recoiling in horror at the suggestion that they might have done or said something wrong, are an entire societal problem. I do not expect an apology, because as I alluded to earlier, John and people like him have long patterns of behavior. Many people, myself and others included, independently come to conclusions about people such as him, based on the way they interact with others. Based on the actions and words they say at appropriate times. I have seen nothing, not one single ray of hope from that crowd, in so many years.

It is so easy to pretend this is as simple as a respectable difference of opinion or whatever. Or even go on a rant about how some form of justice is being done by being “unafraid of the truth” or however these sorts of people want to phrase it.

I am just so tired. I am tired, I am angry, and it does not come from a place of hatred for any one person. It comes from a lifetime of frustration. Frustration for not being able to truly ever understand what it is that other people deal with. Frustration from ever being wrong about any of this at any point earlier in my life. Frustration from the continued acceptance and tolerance of this sort of behavior in technical communities.

And John, if you’re reading this, please grow and change as a person.

Be good to each other out there, folks, but never let your guard down. Keep standing up for what is right, and be willing to speak your truth, unless of course your “truth” involves smearing the reputation of a woman for your own ends.

The Journey of Moving to Pure Functional Scala

Today, January 8th, I am leaving my position at Rewards Network, a Chicago-based company that primarily develops in Scala, to work with 47 Degrees, a functional programming consultancy. During my time at Rewards Network for the past few years I’ve worked on several different styles of projects and been introduced to innumerable libraries and frameworks. These tools all take advantage of the Scala language’s many advanced features, some more than others, and in many cases can enable entirely separate models of programming. One model that particularly fascinates me is that of effectful, pure-functional programming. Since starting the Scala portion of my career at Rewards Network, I’ve spearheaded an effort within my team to promote this style of programming internally. Today, as a result, every single new application that we develop is now written in this style, by group consensus, something that I wouldn’t have dreamed was possible when I was just a new hire at this company.

I would like to write about my experiences being a part of this effort, what it meant for us, and what lessons I’ve learned from trying to switch out our previous stack for a pure-functional one, in Scala.

The purpose of this article is not to convince you that pure functional programming, specifically, is a good idea, or that you should be doing it (you should be, and I could argue this for days, but I won’t belabor that point). Instead, I would like to illustrate exactly how we were able to succeed in the efforts to convert my team’s entire programming methodology. In the event that you are interested in introducing some other paradigm, technique, or programming practice to the company you work for, I hope my experiences will help make your own endeavors a success, or at least give you a reference point to figure out where things can go wrong if you are not careful.

ELI5: Pure Functional Programming in Scala

Before going forward, there’s a chance you’ve stumbled upon this article without knowing much about either “pure functional programming” or the Scala language. As mentioned just above, I am not trying to convert you to coding in this style, much less the specific language of choice, but to talk about how I went from convincing my team of an idea, and how we followed through. For reference, here is a brief description of pure functional programming (henceforth “pure FP”) and Scala both:

Scala is a programming language that borrows the object-oriented style from Java, and the functional style from languages like ML. This means that, like Java, all values are “objects”, or instances of a “class” that defines certain “methods”. It also means that functions, like in all functional programming languages, are first-class citizens of the language, able to be defined and passed around as arguments to other functions without ceremony. Scala makes a number of innovations and design decisions on top of these to make both styles of programming more approachable, including features such as “case classes” that are a simplified way to make low-boilerplate, data-oriented classes with some standard functionality. Another feature you might see referenced frequently is the notion of “implicit parameters”, which the compiler is able to pass to functions or class constructors as-requested, in a manner similar to that of “dependency injection” but at compile-time. Many advanced features of Scala use implicits, such as “extension methods” on objects by having an “implicit (wrapper) class” in scope that the compiler can transparently use without needing to wrap objects in this class yourself, or by allowing you to write “typeclasses” (a feature found in many other functional programming languages) by defining standalone “instances” of them for certain types that can be implicitly summoned when they are in scope.

Pure functional programming is a discipline of functional programming whereby you eliminate the possibility of “side-effects” in your code. If your code does anything other than what the type signature indicates, such as accepting input from the user, talking to the network, or producing some form of output as a file or logging, then that code is said to be side-effecting. In pure FP, Instead of immediately performing these actions, you instead construct a data structure that contains the intention to perform these actions, encoded somehow. You can do this indirectly, using (for example) a list or tree of specially-typed values where each item in the data structure represents an action you wish to perform. This is then passed to an interpreter that looks at the instructions in the order they are listed, and actually performs your side-effects. More commonly these days, specific data types will directly encode your side-effecting functions into a value that can be chained into other values of the same wrapping type. In pure functional Scala, and some other languages like Haskell, this is referred to as the type IO. IO is some kind of “monad”, meaning (among many other things) that you can sequence multiple IO values together to form a single “program” as a value. You can then run this value directly to execute all of your stored up steps, or pass it to the runtime or some upper layer to run your program for you.

Some languages are themselves pure, and do not allow you to compile or run “impure” code, but Scala is impure by design. This means that interacting with the pure and impure parts of Scala code often requires either “wrapping” impure code so that it becomes pure, or explicitly running pure code at the moments you wish it to be evaluated when used in impure code. In practice, this is not so difficult to reason about, and even if you have never done pure functional programming you are probably already familiar with similar “patterns” in imperative code, such as the “builder pattern” where you have to explicitly build something before it can be used. It does create a bit of an inconsistency that will force you to prefer one or the other style as much as realistically possible, however, so while you can adopt a library written in this style, you will not see the full benefits until most of your code is written this way.

Our Initial Stack

Now that I’ve gotten a lot of the formalities out of the way, lets get into the meat of why we’re here: moving from one stack to another, and the problems we hoped to solve. Our stack at first was primarily using a tool in Scala called “Akka”, which is an implementation of the “actor model” of computing. Essentially, instead of manually managing threads and concurrent tasks, you modeled them as “actors” which are distinct entities that can send and receive messages, as well as maintain their own state. Upon receiving a message, and actor can act on it by performing some action and/or modifying its internal state. This model of programming is, I should note, not incompatible with a pure functional style in the slightest, and in fact you can implement your own actor system fairly easily with modern pure-FP Scala libraries. I should also note, for the record, that Akka is also a very widely-used platform for developing distributed applications that can scale very easily in a cluster. For these cases, I would highly recommend it, but from the start I knew it was not a good fit for our team or what we were intending to use it for.

When I say that Akka can “scale” easily, please take a moment to internalize that I am talking specifically about cluster computing, and very large workloads such as online multiplayer gaming where availability is paramount. This is because the actor model itself, not specifically Akka, makes it possible to abstract your code away from the particulars of any one machine or thread, as each actor will be running “somewhere on your cluster”, on a thread you cannot control, and at times that are naturally unpredictable. The notion of “eventual consistency”, a concept observed in large distributed systems where atomic operations are not always possible, is baked-in to the style of programming here. Sending and receiving a message is entirely asynchronous, meaning that you cannot guarantee when and where something occurs, only that if something happens, your actors will behave a certain way in response. It follows that, if your system requires eventual consistency (which most do at a large enough scale), you have programming primitives that work that way by-design. The downside of this is that if you don’t need that, or your only points of eventual consistency are between different applications and not between asynchronous tasks in a cluster, you can end up over-complicating what are otherwise fairly straightforward tasks.

The business problems that my team was trying to solve, and, chances are, the business problems of the team you may find yourself on at this moment, not only did not require designing with this in mind, but it actively made it more complicated every single step of the way.

Imagine you are writing an application that does some simple, single-purpose function. For example, an app that reads from a database table or file once a day, validates that data, and publishes the results somewhere like maybe another table or an Apache Kafka topic. Or, lets even make this more complicated and say you are doing a lot of simple things at once, but in a way that is easily partitioned across multiple instances of an application; say, subscribing to a partitioned Kafka topic in a shared consumer group (if you’re familiar with that) or processing uploads to a service that is partitioned by-key to balance the load. In my experiences, the vast majority of anything anybody is doing in software is going to be written along these lines, and in these cases I would say that actors are not only overkill but they overcomplicate the solution to each problem. Instead of writing a function that, when evaluated, writes the contents of a file as a single logical operation, you might be tempted to write a FileWriterActor that receives messages containing the file contents and then writes them. This is significantly more complicated, has more boilerplate, and is, quite frankly, a misuse of the actor model that does nothing to help you solve business problems effectively.

Over time, this resulted in code that I personally found difficult to read, test, and reason about. The feeling was also shared amongst many of my fellow team members, and nearly every other day I would hear about some new, strange way that something was broken, or certain operations were not occurring when we expected them to. This also extended, to an extent, to Akka Streams, a library that builds on top of Akka to provide a robust push-based streaming solution with adapters for several common integrations available as part of the “Alpakka” project (a pun name, meant to evoke Apache Camel, another integrations library for Java). None of these libraries had any notion of controlling side effects, and it was generally seen as a feature of the platform that your code may not be very predictable at runtime, given its elasticity requirements.

At the end of the day, it was a fundamental issue of software architecture, and the tools that we used shaped the way that we wrote a lot of our code. It shaped our expectations of “what Scala development was” in a sense, but the language is full of opportunity, and I intended to see if things would be better for us a different way.

The Big Migration

My first real exposure to the pure functional side of Scala was with an excellent library called Monix by Alexandru Nedelcu. I was attracted to it because I was a fan of “reactive streaming”, and Monix was a straightforward and easy-to-understand implementation of that model of programming. To make this work, and as part of its implementation, it defined something similar to IO named Task, which is like a computation that is suspended as a value, not yet executed. While I did not pay much attention to this part of the library at the time, instead focusing more on the streaming capabilities the library offered, this type would later be my formal introduction to the world of pure Scala.

Some time within a few months of being brought on, I pitched Observable, the streaming type of the library (which should be familiar to those of you who have used a Reactive Extensions library such as RxJS or RxJava) as an alternative to Akka Streams. I personally found it much less intimidating to get started with, and I felt that it had a much nicer set of operators available in its public API. While there was some limited interest in using better tooling, the main concern amongst team members was that we had already invested a lot of time and energy into Akka Streams, and rewriting code or otherwise supporting two different sets of applications that use their own stacks, was just not something we were comfortable doing at the time. While disappointing, this effectively illustrates the first major takeaway of my experiences:

A Clear Path to Buy-In

Over time and as my personal interest in the more pure-functional side of Scala increased, I started using projects from the community organization that Monix belongs to, Typelevel, in my own personal work. I was able to experiment with different libraries, find out which ones seemed to work the best for the type of work I wanted to do, and get more of a handle on the underlying concepts behind them. This is also when I got more familiar with Cats Effect, the library that provided the foundation for the Task portion of Monix. This library also provided its own IO monad, usable everywhere Task was, and I was hooked. No longer was I just interested in getting my team to use a different streaming library, but I wanted my team to experience the full benefits of pure functional programming for themselves.

The biggest hurdle to convincing my team to switch from one stack of software to another was not going to be solved by evangelizing its virtues relentlessly as if my team were a bunch of unconverted heathens. No, to get people on board, the most effective way to do that was to find some way that clearly illustrates the problems faced, with a solution that, ideally, wows people. I say “ideally” because realistically you may get a response along the lines of, “that’s nice but things are working fine and we don’t want to rethink our entire programming style for benefits we cannot directly observe.” If I was going to propose writing our applications in a pure-functional style, I would need to not only make the case as to why you should write in that style in the first place, but I should do so with clear, directed examples of the benefits we would see. As I mentioned earlier, we had a serious problem with overly complicated code that did simple things, and this would be a prime candidate problem to focus my attention on as I tried to promote pure FP.

To attack the problems my team had been facing, I decided to use a three-pronged approach:

1. Disavow the “actors for everything” model that we had been up-to-that-point been using in every application. Instead, favor simple, linear function composition and only break out the concurrency toolbox when absolutely necessary.
2. Stress the dependability, testability, and assurance that pure FP gives you. By removing side-effects from the equation, and building your entire programs this way, it makes it so much easier to both model and control the effects of different parts of your programs.
3. Focus on the libraries, what they are good at, and what they give us besides the ability to write pure functional code. So many libraries that we ended up taking advantage of had such nice, user-friendly APIs that felt understandable, and less “magical” than some other libraries. If the virtues of whatever dogmatic programming philosophy I was spewing at them were not going to make any converts, I at least wanted to make sure that the actual libraries that follow these concepts are impressive and immediately valuable.

Over the next several months, I developed an internal presentation that went over the Cats Effect library and its immediate ecosystem. By the way, if you learn any single library in the Scala ecosystem, the single most all-encompasingly-useful library there is, is Cats, followed very closely by Cats Effect (which is a separate submodule specifically targeting pure-FP). The Cats family of projects implements a host of useful typeclasses for Scala and other implicit syntax enhancements. Many libraries are not only built with them, but of the libraries that don’t that implement similar functionality, it is usually trivial to interoperate with them and get the best of both worlds.

Anyway, for this presentation, I wanted to prepare a set of examples based on actual problems we faced using design patterns that I felt were clear improvements on what we had been using up to that point. I did not write full applications, of course, but I wrote the basic skeleton of about two or three, and I walked my teammates through on the decision process and what benefits each decision gives you. The most important thing I tried to keep in mind during this process, was to make it seem just as easy, if not easier, compared to what we were currently doing, so I made sure to leave more advanced techniques that required significant explanation at home. The presentation was an amazing success and, within a couple months and after several internal discussions afterward, I got cleared to use Cats Effect on a new project that was starting up as a way of test-running this style of programming.

Of course, I got rather lucky, and even if you go through all the effort to make demos and presentations for your own teams, there’s a good chance you might not get much response. Once, at a Java job, I wrote my own Either monad (called ErrorOrSuccess) for a config-template-parser app, and I gave a lengthy presentation on the virtues of monadic error handling. To my team at the time, this just flew way over their heads, not because they didn’t understand it as-explained but because the problem that it solved was so far removed from their immediate vision. Functional programmers tend to know the benefits of error handling in this way by heart, and sing its praises often, but if the “Java way” is to just throw and check exceptions everywhere, and it “works fine”, what are you really gaining? The solutions that I hoped to provide at the time were so far removed from the types of problems that they were having, and that instance turned into a bit of an unconvincing failure.

So when trying to sell your ideas to your team, however large or small they are, you not only need to be convincing that it solves a problem, but you need to present it as something that has a direct connection to the stressful programming problems they deal with on the job daily. Of course, you can’t just hop right over. Any series of changes to the entire way your team writes software can’t, nor shouldn’t, be done overnight. What you need is a plan, even if it’s a bit ad-hoc, to introduce elements of this new style in a way that makes everybody feel comfortable.

Swapping the Stack

So at this point we had a general idea of where we wanted to go, which was away from our old Akka-based stack and onto the Typelevel stack, which is the sort of organization that houses libraries like Cats. This included, but was not limited to:

• Scala Standard Library w/ Akka -> Cats and Cats Effect as the “baseline” of all programs
• Akka Streams -> FS2, a different streaming library built on top of Cats Effect
• Slick -> Doobie for database access
• Akka HTTP -> http4s, for hosting HTTP servers
• Lightbend/Typesafe Config -> Decline for config and argument loading
• Json4s -> Circe, an excellent functional JSON library

You would be right to assume that is a big set of changes for any team to go through. It took a long time for everybody to get to grips with each layer in this new stack, some more than others. On each project, we would try out a new library, one or two at a time, figure out the quirks and things to be aware of relative to our old solutions, and figure out if it was worth the switch. On older projects that were already written in our older stack, it became a matter of necessity. “Do I need to rewrite all of this code this new way? Does it need to be rewritten anyway for some other reason? Is the old way of doing things actively causing us a problem? Is there some way to quarantine our older code and only allow new code to be made with our newer styles and libraries?”

Moving over to these new libraries, as a gradual process, was mostly straightforward. There were many changes and investigations into what libraries we should use that fit with this style over time. After a while of working in this style and using these libraries, it became natural for most team members to not only use these tools but also to help teach each other how to use them. Before we could get to that point however, we had to first overcome the initial difficulties with changing our programming style. For some team members, this was not so hard, but over time it became much easier.

But in the beginning, things were not always so easy.

Sometimes Technology Problems Are People Problems

On my team we had people along the full range you would expect as a result of major changes like this, from outright skepticism-turned-antagonism to evangelizing it alongside me. Most people were just about as cautiously positive as you would expect, but the biggest difficulty I had was dealing with conflicts as they popped up.

My most significant conflict was dealing with an engineer significantly more senior than I, who had enthusiastically latched onto the stack and went off immediately to write his next application in that style. Every team member was new to writing code in this style, and I had only written code like this on personal projects at the time. So when this senior engineer ran into problems that nobody else on the team could solve, it became a point of contention for the entire team and it carried a huge risk of making me look bad. As a less experienced engineer who had just convinced his team, full of people much more senior, to completely change their programming style going forward on the promise of more maintainable code, I was suddenly staring a huge possible career failure in the face.

Reaching this point, I think, is inevitable if you make it this far. Rare is the significant technical overhaul that goes off without a serious hitch. Somebody is going to disagree, possibly very loudly and directed at the people who originally championed these decisions, and you need a way of dealing with them and their particular problems. Or worse, people will want to agree and even enthusiastically so, but if they keep running into difficulties that nobody can help with, that’s a huge net productivity loss that also damages team morale. In my case, if I wanted to make this migration succeed, somebody had to do everything they could in order to assuage the worries and concerns of my team members. That meant that someone, in this case myself, had to clock into overdrive and help lift the rest of my team members up.

An Expert Is You

There are so many more people who not only know more than I do about pure FP, but they have a much better grasp on the tools and techniques that pure functional programmers employ. What I realized very quickly was that even if I was not an expert at the time, somebody had to be, or at least good enough with all of these tools to be able to assist my fellow team members. If you are introducing some concept or library to your team, you’ll probably find that you have to do what I ended up doing for the better part of two years: filling the shoes of the hypothetical expert you wanted on your team.

What that meant in practice, was not just being someone to give advice and do the research for people, but you have to be willing to sit down with people, work through their problems, sometimes on your own time, and figure things out along with them. It’s a lot of work, and I spent a good chunk of my time making sure that everybody who had questions, everybody who was new and unfamiliar, had not just examples to follow or documentation to read, but also somebody to sit down with and talk things through. You can’t be distant with your team, only focusing on your work, if you are trying to get them to adopt practices or tools that you’re promoting. Along the way, you also need to find out what your weaknesses are, what parts of the language, tooling, and other features are foreign to you, so you can be more well-equipped the next time “that one problem” comes up. In due time, whether you like it or not, you’ll end up becoming the “expert” that your team needs.

Because I had to be the main point of contact for all sorts of problems we ran into over the past two years, this naturally exposed me to all sorts of new ways things can go wrong. By documenting and getting exposure to all of these problems, it will only reinforce your status as the person on the team with the most familiarity and knowledge with the problem domain.

(For an example of the specific, technical kinds of problems faced, see the section at the bottom of this article.)

Some problems, like the one described at the end of this article, had solutions that helped automate away the pain and enforce good programming discipline. Others required more project-specific expertise, and that meant that I had to really dig into how these tools all worked together. For example, FS2 as a streaming library is not just about streaming data, but about composing IO programs that do several operations repeatedly, have multiple sub-programs inside of them, or do various tasks concurrently with safe resource acquisition. In the end, FS2 became a kind of cornerstone of our tech stack, and almost every app we wrote benefitted from using it somewhere. This meant that in all of our applications, certain patterns would emerge, and what better way to encode those than to write them down?

Scala’s primary build tool, SBT, allows you to create project templates using a tool called Gitter8, which is a sort of template language. While helping my fellow team members with pair programming or them generally asking me for advice was enjoyable, it was not a very efficient use of my time, so I used this Gitter8 tool to create special templates for starting new projects. As we started new projects a lot, this greatly reduced the amount of boilerplate necessary as certain common things like an HTTP health check, common config variables, and general project structure setup were just a single command away. As we had different projects with different needs, these templates established the “baseline” of common libraries used, such as Cats, Cats Effect, Decline, and FS2. Other libraries with more specific usages, such as fd4s/fs2-kafka for our apps that uses Apache Kafka, were able to be “toggled” during the template init process, and with that toggle came some example code for getting started with that library.

In addition to using templates to share example code and project structure, I wrote a large list of “best practices” into a document that I would reference to my coworkers. I made it a personal rule that every single time I gave somebody pointed advice about writing pure functional code, or anything with these libraries in particular, I had to write it down in here. It served not only as a central repository for advice for my team, but also a great way to remind myself of things I’ve said, or given out as advice. While it’s not the point of this article to tell you every single one of them, there’s a possibility some version of that will be published here on my personal website, or as a GitHub Gist someday.

But of course, I never would have been able to compile these things together, or gain the project knowledge that I had, without a little help from my friends, so-to-speak.

Never Hurts to Ask For Help

None of anything I am discussing here was solely on me. While it’s true that, if you are going to sell ideas to your team, you need to be willing to put in the effort, it is not a one-person job. Sometimes you are able to have other team members be just as interested as you are, and they are able to take some of the burden off your shoulders. While I was not quite so lucky, I was able to find a lot of help in your friendly neighborhood Scala community.

Each of these projects we worked with, from the official Scala-affiliated projects like the language itself and our build tool SBT, to the Typelevel-stack projects we worked with daily, has a Gitter chatroom that you can join. Joining these, even if you are not outright saying anything yourself frequently, is a great way to increase the surface area of your own knowledge by exposing yourself to new concepts and ideas daily. If you are interested in a particular project, one thing you can also do is seek out the maintainers, follow them around online, and look at the talks, blog posts, gists, or other materials they are associated with, if any.

As much as I wanted to always be there to help my coworkers out, sometimes the best thing to do for both you and them is to refer them elsewhere. Point them at the community, and make sure that everybody knows this is an acceptable thing to do. You get a break for the good of your mental health, and they get more exposure to the problems that other people in the community are also having, same as you. I can tell you from experience that every person I’ve worked with who has taken this advice to seek out project communities has gone on to become so much more proficient with them on a daily basis than before. There are practically no downsides, only making sure to invest a little time here and there.

Finally, it just so happened that a lot of what I was going for was not only desirable within my team, but the outside help that we brought in was just itching for an opportunity to dig into pure functional programming as well. We had hired several consultants over the years, and two consultants that we hired (named below in the Acknowledgements section) were instrumental in providing additional assistance wherever possible. There are many consultants in the Scala world who are dying to get to work on projects with this type of stack, and they took the opportunity to not only help us build out our apps in a pure-functional style beyond what I was capable of myself, but they also wrote some training documentation for us to go through to make sure everybody was on the same page.

As mentioned at the very beginning of this article, and worth repeating to make clear my biases, I, too, will be taking my experiences learned here and elsewhere to my new position with 47 Degrees, which offers consulting services for Scala and many other languages. I want to help enable people to make the best Scala software they possibly can, and being familiar with most major areas in the Scala ecosystem including working on a transition of this size, I hope that my services will be useful to the people I end up working with in any area.

If I have a single take-away from my experiences listed here, it’s that any kind of programming technology change, especially one wher you’re swapping out your entire paradigm, is a lot of effort, and it is not a one-man job. You not only need buy-in from your team, but you need a community full of people who have put in effort of their own to make things easy. We should not be afraid to reach out and ask for help, from the small things like fixing and understanding bugs, to the larger things like training materials, availability for answering design or implementation-level questions, and being able to listen and take people and their problems seriously.

In Conclusion

• Have a clear goal, no matter how large, and make sure you have a path to get there. Try to focus on a real problem that your team is having, such as testability, code flexibility, or being able to depend on the software you write without feeling like there’s a layer of “magic” underneath it all.
• Always have more than one reason for why a change should be made. For me, the libraries and frameworks were ergonomic and very usable out of the box, in addition to solving underlying problems that were hard to illustrate in presentations. Cover your weaknesses and it will be much easier to convince people.
• You need to put the work into materials. Presentations, working code, documentation, templates, you name it. If it doesn’t exist, you’re asking the rest of your team to pick up your slack, and that doesn’t look good on you when you’re trying to “sell” your team on something.
• Always try to be available. I was passionate about this, so I always tried to respond to questions and concerns from my coworkers even at odd hours. I would not say that is a requirement, but sometimes even just being there during working hours is more than enough. Listen to people, their problems, and take them seriously. Work to overcome them together.
• Write everything down. All of the tips, tricks, techniques, patterns, you-name-it, that you come across and implement as part of your journey. You’ll not only thank yourself later for cataloging it, but you’ll cement the knowledge in your head, only making you more effective at helping your team succeed.
• Ask for help. This is not a one-person job, nor should it be. Get involved with the community of people making the tools you’re using, and ask them any questions you come up with. Sometimes this also means investing in training for your team, whether through internal or external means.

Most importantly, watch out for your own mental health. Any kind of shift of this magnitude on your team requires being able to do a lot of work to accomodate everybody, their fears and confusions alike, but at the end of the day you are a person with your own needs. Don’t be afraid to say no, or refer people who need help to other resources when you can’t be there.

Overall, my time working on all of these projects has been an immense learning experience, not just in how to write effective pure functional code, but as a case-study in the kind of things that need to happen to pull off a transition like this over the years. I am happy that it has not only gone so well, but I believe that it will last a long time as the seeds of properly believing in this style of programming have been planted across my team. It’s no longer just some fringe theory of software development that I happen to employ on the weekends for personal projects, but through lots of help and perseverence we were all able to make this a reality for our professional work as well.

If you find yourself in these shoes, with big ideas and an eye for change at your workplace, you may not always succeed but do know it is possible with the right attitude, approach, and application of discipline.

Acknowledgements

I would like to acknowledge my amazing team that I’ve worked with until now at Rewards Network in Chicago, IL. If you are looking for a full-time Scala position, and perhaps want to get your hands on projects using these fancy libraries, I can wholeheartedly recommend them. They are remote-friendly (not just from the COVID-19 pandemic) and are very accomodating to any person’s unique needs in my experience. They are also a very safe, friendly, and accommodating place to work for developers with a generally progressive outlook, so you can be sure that if you don’t fit the stereotypical bill of a “white cis-het male developer” that you will be welcomed and fit right in.

Special thanks also goes to all of the Gitter channels I frequented for most of this time, including all of the Typelevel projects and their maintainers. I could have never figured out as much as I have without the constant (really, constant) guidance and assistance I have received from every one of them.

We would not have accomplished everything if it weren’t also for the amazing assistance from Francis Toth and Calvin Lee Fernandes, two consultants who we worked with during parts of the transitionary phase. They put together training materials as well as developed applications with us in a pure functional style, and overall I was very impressed with their quality. If you would like to contact either of them, here are their websites:

• Calvin Lee Fernandes - https://www.kaizen-solutions.io/
• Francis Toth - https://contramap.dev/

And of course, if you are reading this, this is my personal website. You can access the sidebar of this site to see the most up-to-date information on contacting me personally, or as mentioned you can also get to me through 47 Degrees.

Further Reading

If you are actually interested in pure functional programming in Scala, besides offering my own services

• Functional Programming with Effects, presentation by Rob Norris (aka tpolecat, author of Doobie).
  • Understanding pure-functional programming means understanding the concept of “effects”, and this is explained excellently here.
• The FS2 Docs
  • I firmly believe that FS2 is one of the most powerful and most generally useful libraries for composing programs in Scala. The fact that it’s pure-functional is an amazing bonus, and it has a thriving ecosystem.
• Practical Functional Programming in Scala by Gabriel Volpe
• “The Red Book” - Functional Programming in Scala by Paul Chiusano and Runar Bjarnason
  • Side-note: this is the book that got me into Scala in the first place and demystified a lot of what functional programming is, in a nutshell.
  • Also, the authors of this book are now developing the Unison programming language. It looks fascinating and might be good for you to look into in your spare time.

That One Example Problem in Pure FP

One of the common problems we ran into very early on was that certain operations were not happening when we expected them to, or at all. This was also a problem we ran into with Akka, due to its eventually consistent nature, but the problem here was much simpler to solve and was simply a matter of programming discipline rather than being an eventual consistency problem. You see, in pure FP, you never execute anything at all until the “end of the world”, which is the location at which your code needs to be ran. So for most programs, this is the end of your “main method”, or if you are writing code that is passed to a callback in some Java library, you might need to manually run your code inside of there. The problem we ran into was simply a matter of making sure that, instead of trying to “run” code like before (e.g. calling a method and doing nothing with the result), we had to make sure that every single operation chained into another operation, and the result was returned.

Here is a small code sample of this type of problem, for reference:

import cats.effect.IO

//In Scala, Unit is what many other languages call "void".
//It is used when your result value is useless, such as when printing to your console.
//This function is supposed to upload some data to a repository.
//Then, it is supposed to post a success message using the client.
//Here is generally how you would write it using blocking APIs in a non-pure-functional style:
def myBuggyFunction(client: MyClient[IO], repo: MyRepository[IO]): IO[Unit] = {
  val exampleData = List("Hello", "World", "!")
  repo.uploadSomeData(exampleData) //Result: IO[Unit]
  client.postSuccess //Result: IO[Unit]
}

//The above function has a bug, can you spot it?
//The only thing that happens is the client posts "success".
//The database is never accessed at all. Why is that?
//This is because the operations, IO values, are not executed until the "end of the world"
//To properly execute them, we need to return the right IO value.
//Here is a version where we make sure that our operations are properly sequenced
//flatMap here means, chain this first IO value into the second one
//The underscore _ means "ignore the result of the uploadData operation"
//Normally you would use flatMap to create "dependent" IO values that use the result of the first one.
//Because we don't care about the result, it is disregarded with the underscore.
def myFunctionWithoutBugs(client: MyClient[IO], repo: MyRepository[IO]): IO[Unit] = {
  val exampleData = List("Hello", "World", "!")
  val uploadData = repo.uploadSomeData(exampleData) //Result: IO[Unit]
  val postSuccess = client.postSuccess //Result: IO[Unit]

  uploadData.flatMap(_ => postSuccess) //Result: IO[Unit]
}

As the comments above explain, this particular problem can be a bit tricky to realize you are doing unless the compiler is helping you out. So, to get into the habit, we enabled a compiler warning for “discarded values”, meaning that every single (non-Unit) value in a function has to be either used to compute some other value or returned from the function. This single change eliminated the source of the error and helped enforce good programming practices. To use this yourself, I would recommend the “sbt-tpolecat” plugin which enables this and many other useful (I’d say necessary) compiler warnings that enforce a good programming style.