When we write programs in a conventional programming language, what we're doing with the data is shown front and centre, and the data itself is sidelined or often not even present. In a spreadsheet, this is reversed: the data is prioritised, and the formulae are hidden.
There's an immediacy to a spreadsheet: the user can start with very literal actions on data, and slowly introduce abstractions like formulae. In conventional programming, the programmer has nothing but abstractions with which to work.
It's reminiscent of one of Fred Brooks' remarks about how showing the data structures makes the algorithms obvious, but showing the algorithms reveals little about the data structures.
I've had a lot of experience with dealing with exactly this problem in ML.
There is no replacement for visualizing the data and intermediates. You have to see what is going on, it's the only way to catch bugs and issues and make good performance improvements.
Like actually looking at the gradients and weights and activations and predictions and stuff often shows you that something janky is going on, its great for adjusting architectures. Looking into specific high loss test samples and mispredictioms and stuff will show you that there are problems with your data or normalization and whatnot.
The issue is that there are basically an infinite number of intermediates you could potentially look at. So ignoring almost all of them is the only thing that scales and you have to be extremely deliberate.
That is an interesting take on spreadsheets. I wonder if it is possible to establish some way to switch back and forth between 'programming view' vs 'spreadsheet view'.
It’s pretty simple. Everyone knows how to use it effectively and the cost of training every accountant or finance person is far greater than paying Microsoft a few bucks included in your email anyways. Plus you know it will be compatible with external files.
I worked at a place that used Google Workspace and the most effective people simply supplied their own Excel, because it's just way more efficient than spending hours learning how to use Google's clunky pivot table function, when you already know how to do it well with Microsoft.
Yeah don’t think it’s going anywhere any time soon in big corporate - at least in finance. The network effects are just too powerful - compatibility, training, integration, skill availability etc.
Every single attempt to migrate to something else has been a comical failure. On the plus side they tend to be rapid failures rather than SAP style multi years
As someone that was part of such project, the main issue is that Excel is really programming, plus there are macros, VBA, AddIns, lambda, PowerQuery, and now Python on top.
All those migration projects tend to fail on the details, there is always that one customisation point that is easily done in Excel, but requires a whole team with architects and such for getting the same capability on the replacement tool.
Nice thing is, Microsoft themselves are undoing that with the push to Office as a web-first application. While making it more ubiquitous (they think), they are at the same time cutting loose some of its strongest anchors
Spreadsheets are to Finance folk what WordPress is to Marketing folk and what tools like v0 and other LLM-powered web app builders are to Product folk. The promise of having some fancy schmancy technological tool let you do your job by yourself and without needing to collaborate with other people is basically the strongest value proposition that you can make to working professionals.
The fact that all these tools eventually fall apart at scale is basically irrelevant. You can only take away control from their cold, dead hands, and they will never learn the programming skills to build something more scalable themselves. All solutions almost invariably trend towards shifting scaling problems away from the desired frontend that keeps control (or at least the illusion of control) in the hands of business stakeholders (one of the reasons why companies love building all manner of integrations on top of Jira).
And let's be clear, it's not like the alternatives here (i.e. databases) are so easy. Finance folk are used to creating new spreadsheets on their local computer for free and at will; most databases require setting up a server, DNS, certificates, firewalls, most of which have real costs. SQLite naively sounds like a reasonable approach, but by default a table is limited to 2,000 columns while an Excel spreadsheet is by default limited to about 16,000 columns, and yes, stuff like this really matters when you're talking about trying to uproot a favored tool. At the end of the day, most of Excel's limitations are due to attempting to cram a spreadsheet into a single file (same as SQLite); if Microsoft were smart, they'd offer a cloud-only "spreadsheet" (really a database over a full filesystem, or maybe over object storage) without the limitations of ordinary Excel spreadsheets, where Excel-the-desktop-app only downloads to the local client the relevant cells that are actually in view, while adding more options to mirror/load data from external sources into other sheets attached to the same cloud-only "spreadsheet".
Spreadsheets are the killer apps. Since I became pointy haired sheets and documents are the fuel of most what I do. Even things like Jira we use them only for the bare minimum and we refer to sheets to see how things are going. Not that I think it's sane, but it's what works.
I even have done this first day's advent of code, and the first part of day two, in a Google sheet. Formulas only, so no scripting needed.
It's just so much easier to stay and keep paying Microsoft than switching. But why hasn't any startup found a way yet to disrupt it by giving the users something they actually love more, so they can start lobbying businesses to switch to it, I wonder.
Excel is a behemoth of functionality which is constantly growing and improving. It's not as simple as building "just another spreadsheet"; and attempts are made, dozens of them. But it seems they all are either not really understand the strength of Excel, or simply admit defeat from the beginning and try to find a niche where they can compete, by looking at specific usecases and target groups. Kinda like all those companies who build hyper-specialized cooking-tools, but still can't beat the versatility of a good knife, so they mainly sell to normal people, not the real experts.
Because the power of Excel comes from the huge amount of features it has slowly accumulated over the decades of its lifetime. You can pretty easily make an Excel 97 replacement today, probably. But people won't be able to use that in these business contexts - they each need some of those obscure 1% features that got added.
I think when you learn to code, and I mean coding "real" programs, you forget how powerful Excel actually is.
Yes, it's awful in many ways, but it is very accessible to people who don't consider themselves programmers. "I just want to do my thing" is very easily done in a mashup of the spreadsheet and VBA.
We also forget the pain of learning a new technology. People whose first experience is excel also go through this. Shit doesn't do what you wanted. After a while, they can build stuff in excel, but they don't want to learn python, because there's more pain coming.
Most senior devs have transcended particular languages or technologies, so they don't see it. They pay a very small cost for picking up a new tool, so they scratch their heads when they see a guy who wants to run a trading book on a spreadsheet.
Excel is where it is due to a kind of mild bait and switch. In the DOS and OS/2 days before Windows 95, Microsoft told Lotus and other software companies OS2 was the future, not windows. So Lotus when all in on OS/2 with its spreadsheet.
Then came 1995, M/S released windows 95 and at the same time they had a full software suit running on W95. It took Lotus a while to get a W95 version out. I think the same happened to Word Perfect.
So here we are, too bad Libreoffice was unable to do the same with Linux. But now people are so entrenched I doubt they will only change if forced.
Also, years ago IBM tried to get people on LibreOffice and off M/S, it failed miserably. Many Orgs. in Europe is trying now, I hope they succeed.
BTW I wonder if we are heading down the same path with github ? I hope not but seems we could be.
When we write programs in a conventional programming language, what we're doing with the data is shown front and centre, and the data itself is sidelined or often not even present. In a spreadsheet, this is reversed: the data is prioritised, and the formulae are hidden.
There's an immediacy to a spreadsheet: the user can start with very literal actions on data, and slowly introduce abstractions like formulae. In conventional programming, the programmer has nothing but abstractions with which to work.
It's reminiscent of one of Fred Brooks' remarks about how showing the data structures makes the algorithms obvious, but showing the algorithms reveals little about the data structures.
I've had a lot of experience with dealing with exactly this problem in ML.
There is no replacement for visualizing the data and intermediates. You have to see what is going on, it's the only way to catch bugs and issues and make good performance improvements.
Like actually looking at the gradients and weights and activations and predictions and stuff often shows you that something janky is going on, its great for adjusting architectures. Looking into specific high loss test samples and mispredictioms and stuff will show you that there are problems with your data or normalization and whatnot.
The issue is that there are basically an infinite number of intermediates you could potentially look at. So ignoring almost all of them is the only thing that scales and you have to be extremely deliberate.
It has the appeal of Smalltalk or Lisp inspired live coding environments, gone mainstream with an approach that people can better understand.
Note all the audio programming tools getting embraced by music folks as well, it is coding, but made approachable.
That is an interesting take on spreadsheets. I wonder if it is possible to establish some way to switch back and forth between 'programming view' vs 'spreadsheet view'.
I have always had vague ideas about this with pandas but don't have the aptitude or motivation to take on a project of that size.
I hate Microsoft but love Excel. I was hoping python in excel would be this but I don't think it quite is.
It’s pretty simple. Everyone knows how to use it effectively and the cost of training every accountant or finance person is far greater than paying Microsoft a few bucks included in your email anyways. Plus you know it will be compatible with external files.
I worked at a place that used Google Workspace and the most effective people simply supplied their own Excel, because it's just way more efficient than spending hours learning how to use Google's clunky pivot table function, when you already know how to do it well with Microsoft.
Yeah don’t think it’s going anywhere any time soon in big corporate - at least in finance. The network effects are just too powerful - compatibility, training, integration, skill availability etc.
Every single attempt to migrate to something else has been a comical failure. On the plus side they tend to be rapid failures rather than SAP style multi years
As someone that was part of such project, the main issue is that Excel is really programming, plus there are macros, VBA, AddIns, lambda, PowerQuery, and now Python on top.
All those migration projects tend to fail on the details, there is always that one customisation point that is easily done in Excel, but requires a whole team with architects and such for getting the same capability on the replacement tool.
Nice thing is, Microsoft themselves are undoing that with the push to Office as a web-first application. While making it more ubiquitous (they think), they are at the same time cutting loose some of its strongest anchors
Excel is the glue holding a lot of corporate workflows together. Like Perl or Python.
I have seen it used for everything from configuring aircraft for sale to quoting bond prices for voice traders.
There's even that famous Japanese artist who uses it for painting.
It's sort of like the OG normies Jupyter notebook.
Spreadsheets are to Finance folk what WordPress is to Marketing folk and what tools like v0 and other LLM-powered web app builders are to Product folk. The promise of having some fancy schmancy technological tool let you do your job by yourself and without needing to collaborate with other people is basically the strongest value proposition that you can make to working professionals.
The fact that all these tools eventually fall apart at scale is basically irrelevant. You can only take away control from their cold, dead hands, and they will never learn the programming skills to build something more scalable themselves. All solutions almost invariably trend towards shifting scaling problems away from the desired frontend that keeps control (or at least the illusion of control) in the hands of business stakeholders (one of the reasons why companies love building all manner of integrations on top of Jira).
And let's be clear, it's not like the alternatives here (i.e. databases) are so easy. Finance folk are used to creating new spreadsheets on their local computer for free and at will; most databases require setting up a server, DNS, certificates, firewalls, most of which have real costs. SQLite naively sounds like a reasonable approach, but by default a table is limited to 2,000 columns while an Excel spreadsheet is by default limited to about 16,000 columns, and yes, stuff like this really matters when you're talking about trying to uproot a favored tool. At the end of the day, most of Excel's limitations are due to attempting to cram a spreadsheet into a single file (same as SQLite); if Microsoft were smart, they'd offer a cloud-only "spreadsheet" (really a database over a full filesystem, or maybe over object storage) without the limitations of ordinary Excel spreadsheets, where Excel-the-desktop-app only downloads to the local client the relevant cells that are actually in view, while adding more options to mirror/load data from external sources into other sheets attached to the same cloud-only "spreadsheet".
https://archive.is/84dmE
Spreadsheets are the killer apps. Since I became pointy haired sheets and documents are the fuel of most what I do. Even things like Jira we use them only for the bare minimum and we refer to sheets to see how things are going. Not that I think it's sane, but it's what works.
I even have done this first day's advent of code, and the first part of day two, in a Google sheet. Formulas only, so no scripting needed.
Wish it had interesting alternatives like all the C++ alternatives today. Imagine a spreadsheet that could be 'compiled' into a full stack program.
It's just so much easier to stay and keep paying Microsoft than switching. But why hasn't any startup found a way yet to disrupt it by giving the users something they actually love more, so they can start lobbying businesses to switch to it, I wonder.
Excel is a behemoth of functionality which is constantly growing and improving. It's not as simple as building "just another spreadsheet"; and attempts are made, dozens of them. But it seems they all are either not really understand the strength of Excel, or simply admit defeat from the beginning and try to find a niche where they can compete, by looking at specific usecases and target groups. Kinda like all those companies who build hyper-specialized cooking-tools, but still can't beat the versatility of a good knife, so they mainly sell to normal people, not the real experts.
Because every department has a different use of Excel.
LibreOffice and Google Sheets are still quite far from everything that Excel is capabale of.
I was part of a project to replace Excel, on a lifesciences department that was using it, as you might think about R, Pandas and similar.
Tableau was even a more appealing candidate to them.
Because the power of Excel comes from the huge amount of features it has slowly accumulated over the decades of its lifetime. You can pretty easily make an Excel 97 replacement today, probably. But people won't be able to use that in these business contexts - they each need some of those obscure 1% features that got added.
I think when you learn to code, and I mean coding "real" programs, you forget how powerful Excel actually is.
Yes, it's awful in many ways, but it is very accessible to people who don't consider themselves programmers. "I just want to do my thing" is very easily done in a mashup of the spreadsheet and VBA.
We also forget the pain of learning a new technology. People whose first experience is excel also go through this. Shit doesn't do what you wanted. After a while, they can build stuff in excel, but they don't want to learn python, because there's more pain coming.
Most senior devs have transcended particular languages or technologies, so they don't see it. They pay a very small cost for picking up a new tool, so they scratch their heads when they see a guy who wants to run a trading book on a spreadsheet.
Excel is where it is due to a kind of mild bait and switch. In the DOS and OS/2 days before Windows 95, Microsoft told Lotus and other software companies OS2 was the future, not windows. So Lotus when all in on OS/2 with its spreadsheet.
Then came 1995, M/S released windows 95 and at the same time they had a full software suit running on W95. It took Lotus a while to get a W95 version out. I think the same happened to Word Perfect.
So here we are, too bad Libreoffice was unable to do the same with Linux. But now people are so entrenched I doubt they will only change if forced.
Also, years ago IBM tried to get people on LibreOffice and off M/S, it failed miserably. Many Orgs. in Europe is trying now, I hope they succeed.
BTW I wonder if we are heading down the same path with github ? I hope not but seems we could be.
For many of the people I see stuck on spreadsheets, Excel is at the limit of what they can comprehend.
[dead]
[dead]