Earlier in the sub, I saw a post about packages or modules that Python users and developers were glad to have used and are now in their toolkit.
But how about the opposite? What are packages that you like what it achieves but you struggle with syntactically or in terms of end goal? Maybe other developers on the sub can provide alternatives and suggestions?
I was about to learn this the hard way.
I loved using rq for this use case when using redis
I investigated rq too. Redis isn't the best message broker, but if it's already in your tech stack it can be a good option.
Rabbitmq is better if you need a heavy duty broker, but it's more complex to use and operate. It might be overkill depending on the use case.
We had a lot of problems with jobs getting mysteriously stuck with RQ.
Having this problem with deferred jobs too. For some reason they just don’t run sometimes.
Celery has been a nightmare for me. Such a black box sometimes
What are some problems with it? I use it at work but not too much, just some basic task scheduling together with redis.
I mostly just hate the way it’s maintained. I’ve found that it’s incredibly buggy and unreliable, but the maintainers will close issues evident and valid bug issues for no reason.
Dramatiq is a similar library in spirit, but I’ve found to be much more reliable and simple. Also its source is relatively easy to grok and contribute to
Haha. The times I've used Celery, I end up wishing I'd instead used something much simpler (like Huey) or something external to Python, robust and well documented (like RabbitMQ) - but Celery seems to have just the right amount of both magic and inflexibility to make me regret that I used it.
Oh thank god I dodged that bullet. I went simply with rabbitmq and pika and it does the job well. Of course I had to do some low level tuning, but there’s so much resources around to help, that it wasn’t that difficult.
Also does not uses the same terminology as RabbitMQ, which makes the initial setup a pain to configure / debug
Pysimplegui. They started as probably the most user friendly GUI framework, and then they went subscription model out of nowhere.
Yeah this is the one. It's not even that good.
FreeSimpleGUI is a good enough hack, but like damn
NiceGUI to the rescue 😌
He went to a subscription model? I liked that library, but the guy who made it was once DMing me his emotional problems on reddit when someone criticized his library and I defended it briefly
huh. Interesting. Glad I migrated away from it to NiceGUI before the change to subscription model.
Llangchain. A garbage heap of a codebase that makes you jump through 5 abstractions to do the simplest thing. They litterly created a class for prompts that’s a wrapper on an f-string.
Starting to feel this way about llamaindex as well.
I can do chunking and hit the embeddings API just fine and get exactly the behavior I actually want without wading through their shit
Also all the hype new vector databases like chroma seem kinda useless. There are extensions to postgres to get vector types, distance and cosine angle.
So much this.. After this 8 month old gist I droped all langchain/chroma/weviate from my stack and go hit postgres and the llm api servers directly https://gist.github.com/chuyqa/e122155055c1e74fbdc0a47f0d5e9c72
Same, chromadb was painful to work with. Especially the way it returns the query result. Just confusing.
I'm still using LlamaIndex but between the learning curve and the fact it had inproper async support the gain was minimal
Those prompt templates are exactly why I couldn't jump on the bandwagon.
I was turned down from a job that wanted 5 years of LangChain experience. Sounds like I dodged a bullet?
You couldn’t ask for a better red flag
5 years langchain
8 years LLM finetuning
2 PhDs
Authentic asian
Job: Filling out Excel sheets
“Work hard; play hard!” $50k salary, 5 days pto, 80 hr/wk minimum
Twas coming to say this. It's awful
Any decent alternative? or just working with the raw apis?
Haystack! Simple, extensible, mature. I feel like I work for them given I comment so much about Haystack. But I just want fellow devs to enjoy their work and have an easy life. It abstracts just the right amount.
Am using the textsplitter as I havent found a better one but the rest I rather not touch - I like to understand what is happening
And the documentation is too hard for me - it pushes me to read their code which is a waste of time
Serial, not pyserial, Serial.
Oh yeah, I've been confused multiple times installing serial and getting weird errors for code that worked just fine elsewhere only to discover I had the wrong package.
GStreamer is the big one. I love it. There's basically nothing like it. But when it goes wrong it can be hard to debug, and sometimes you have to do hacky nonsense to make it do what you want.
Like, my app detects what virtualenv it's running from, and adds a symlink to the global package, because you cannot install it with Pip, and system-site-packages is not ideal.
That sounds... Bad
It's pretty much unmatched for capabilities, but it can be rather hard to debug.
The API bindings are autogenerated and not very pythonic, and most everything is a plugin that it discovers at runtime, so there's not really any autocomplete or anything, you're creating elements by passing types as strings, not calling classes.
Error messages are not great by default, if some element doesn't think it's ready to start, it doesn't tell you why or what. A lot of the time you won't be able to link something because the ports don't exist till runtime, like you'll have a decoder that doesn't know what it's going to decode, so the output port won't exist till it knows the sample rate and channel count and all that.
My two projects with it are an NVR that saves CPU by only decoding keyframes, and a web based audio mixer, and it does it all very well, once you get it working. I'd prefer a much higher level interface, and I work with it mostly through my IceMedia wrapper that makes it a bit higher level.
If I was going to rewrite it all, I would probably consider using Rust or Cython and doing my own low level media work, but I suspect that would be a bad idea and probably not work on platforms I don't actually have.
I'm hoping Rustimport gets an automatic "Cross compile for all the big platforms" option, and then other approaches to media might be a bit more practical.
Like, my app detects what virtualenv it's running from, and adds a symlink to the global package, because you cannot install it with Pip, and system-site-packages is not ideal.
I was reacting to this in particular. Now I realize I misread it as saying that GStreamer escapes the venv on its own. That sounds less bad.
Streamlit. NiceGUI or Dash are far nicer to work with, and support "normal" programming (instead of the "full re-execution on load" thing that Streamlit does). I didn't find out about them though bc Streamlit was "good enough"
I came to comment that. streamlit markets itself as a way to speed up prototyping. But as soon as you hit the first adjustment, you will start paying heavy technical debt.
Totally agree. I’ve got a love-hate relationship with it - good at getting something up quickly, but you need to do mental gymnastics with the top-down reloads to make any half-complex app work. Then any custom UI requests from clients become a never-ending clusterfuck of iframes in iframes…
Is Shiny for python better?
(anyone who tested both in a big enough project that went to production, preferably)
I've been using Shiny for Python for my job. I find it less complicated than Dash and easier to scale up compared to Streamlit. However when a Shiny app gets complicated, it can get very difficult to track where exactly there might be an issue. Last week I spent 6 hours troubleshooting why a plot on my UI panel wasn't showing when it was basically due to me calling an input id that didn't exist 😅 my own fault but I wish I got notification of that when the app was running. Also you can't use the same id across different UI elements.
Shiny dev here. Great feedback, thank you! That’s definitely a common footgun that we need to make easier to detect.
Panel+Voila is all you need
I've been using streamlit regularly. Mainly because I don't know the front end stuff. It allows me to build a simple frontend very easily. The other thing is that I can deploy it to cloud very easily.
Poetry is starting to feel this way for me. Running into issues with the .toml file when installing some packages, it’s on their roadmap at least to upgrade to PEP 621.
I love the package about 95% of the time. Sometimes I have to go back to conda though.
One thing that took me a while to figure out with poetry is what happens if you unwittingly try to install a package that’s already installed by another package as a dependency.
You don’t get told “hey, you already have this, are you sure you want to explicitly install it?”
Instead, poetry just tries to install it. But it doesn’t go through dependency resolution in a way that would produce the version that you have installed - it tries to install the newest version that is supported by all of your other explicit named package dependencies in pyproject.toml. At least, this is what I think it is doing.
What can result is some really confusing dependency conflict errors, which take you down a rabbit hole of trying to pin specific dependencies of sub packages. All because you didn’t run poetry show <package-name> to see you already have the package and don’t need to install it.
After having used poetry for several years now, I’ve seen this issue crop up for other developers several times. In fact, it’s pretty much the only issue that devs at my job have with poetry, and invariably i find out about it after someone has wasted often a couple of days trying to untangle a problem that wasn’t even there in the first place.
It’s ultimately an easy fix, but I don’t understand why poetry doesn’t add a step to check if a package is already present. Especially if you are just running poetry add <package-name> with no version specified. It seems to be making pretty huge assumptions about what you mean by that command which are almost always going to be wrong if the package is already present as a sub dependency.
You shouldn't rely on a transient dependency of one of your dependencies. If you import it, it should be in your direct dependencies.
If you don't do it, you are one refactoring by a third party away from your code not working.
That's fine, but it still seems strange that Poetry doesn't detect that the pre-existing version meets the requirements.
What it does is kind of reasonable imo, poetry add {package} is implicitly poetry add {package}=={latest}. Then it just tries to resolve those requirements.
That’s really good to know and thanks for typing all that out
Yup, I seem to need to step on this landmine every 6 months or so. Thats about enough time to forget about this, for me.
I did this one time when I was awake too long. Spent like half the day troubleshooting installing 800 things. Normally I just move everything and create a new environment. Granted it might not be the best approach, but I liken it to a hard reboot
I find pdm to be favorite package manager over poetry
rye.
I looked into rye, albeit this was a few months ago, and unfortunately it was still too premature. I love that it's been taken up by the same team as ruff, and I think it will improve to get to a better maturity level. But, for me, it's just not ready yet ¯\_(ツ)_/¯
Well, we are about to use it in production soon
I also prefer PDM. I've been using it for about a year and it's kind of the most all around "perfect" app for this. Fast dependency resolution, I run into far fewer errors than I did with Poetry, the [tool.pdm.scripts] section of my pyproject.toml is incredibly useful (fuck poe the poet, I known that's the poetry version, but it's so much extra lifting compared to PDM), etc.
It doesn't do much more than any other package manager, but it does everything very well and that's all I can ask for haha.
I'm watching rye closely though, I love Ruff and am excited about a package manager that uses uv.
Can’t beat a requirements.txt, pip and virtual envs
I've been using hatch envs for a while and let me tell you, in my book they very decisively beat doing things manually.
I’m glad! I’ll have to try it out. I just have ptsd from trying all these different python solutions. None of which beat requirements simplicity for me 🫠
Add `uv` to that for speed and it's just perfect.
Yeah it just works. I don't care about what the new fancy method like poetry or conda. I use whatever python.org says.
If ain't broke
It is tho
Pycharm makes virtual envs so easy that i havent even bothered with the other solutions. If i mess up that badly i can just make a new one in the gui with a few clicks
thank you for saying this, I want to like poetry....but...
I've had one big poetry related annoyance, and that was the fact that it doesn't give you a way to make a wheel with pinned libraries.
freeze-wheel solved that, but for some reason it's very hard to find. Google will tell you about several other packages before it even mentions the freeze wheel plugin.
What is a wheel with pinned libraries?
When you do poetry publish, it thinks you're publishing a library, and will allow any version of the dependencies that matches what you have in the pyproject, even if it's not what's in the lockfile.
Freeze wheel just makes the wheel depends on exact fixed versions.
Oh...by narrowing down the install requires.
Of course, it'll make it virtually impossible to use with any other libraries, so seems of limited use
It's definitely not appropriate for almost any library, but very useful for an end-user app
Yeah this wd why I ended up writing my own. Well partially, I didn't know poetry existed back then. Felt like an idiot then not so much.
I love poetry, but it can be a pain in the ass sometimes.
P.E., some docker images (Nvidia) come with pre installed packages that poetry tries to override and I have to remove them from the project before breaking the image (No CUDA support).
Fuck I feel the same way. And I was the one who pushed poetry to all my developer friends after reading hypermodern python.
It is kind of great when it works, but once you have some problems it is just one more tool to learn and fix. I'm back to virtualenv and requirements.txt.
I've been using uv lately and love it. SOOOOO fast
The situation with uv and rye seems confusing. They're both maintained by the same team, and I think they both aim to do the same thing. Not sure which one I should be looking at, right now.
Tensor flow, it's just so bad
Ugh for some reason I have dependency issues EVERY TIME I try to do something with tensorflow, it can do cool stuff but it sucks that its so stupidly hard to set up on different devices
I CAN NOT make it run on a GPU it’s a nightmare
Why do you think Keras exists?
Altough I prefer Pytorch / lighttorch.
Win32com. Useful as a tool but it's essentially VBA in python
Are there any alternatives though? I would love to have a better tool for COM objects, but I’m not aware of any.
honestly, the better alternative is to learn how the thing you're trying to do with COM is done in linux and do it like that instead.
I can’t even figure out how to find all the ways to interact with COM objects. If anyone has a good source/tutorial to be able to learn to debug/reverse engineer any objects when developing I’m here for it.
I could just be an idiot or not researched it enough.
Omg i hate it and love it as the same time, like if you’re windows only then it’s great everyone in my team started using it for excel and now it’s a burden
Anaconda/Conda, then Poetry.
I want build and dependency tooling to achieve a few things. 1. I almost never fight with it. 2. It almost never violates the principle of least surprise. 3. It is as fast as is reasonable given what it is being asked to do. 4. It doesn't constrain my options for other tools I use unless I invest stupid amounts of time to overcome whatever the limitation is.
As a result I stick to a few simple things as much as possible: pip, pip-tools, setup-scm, twine, wheel, and something to work as a local pypi. Once in awhile I have to coax a package build that has an O/S library dependency or requires a compile, but only once ever has conda done that for me in a situation I couldn't quickly fix myself (stan on windows).
I tried going into poetry in an attempt to jump from setuptools via setup.py to a more "current" packaging format
Heard that poetry was the go-to for containerization and its "Nix-like" approach thing, so I gave it a try
Nope, didnt work, not only did it not work, it made my packaging far more convoluted than it needed to be and it broke installation lmao
So i went for the next best classic packaging, non-containerized method which was via pyproject.toml, that allowed you to choose setuptools as well as other backends
Stuck with that, but damn, poetry was a nightmare
This is strange to me. I started using poetry a little over a year ago, and I haven't had these kinds of issues. The only time I've had an issue with it was when my system Python version didn't match the Python version being used in the project and it wouldn't make the lock file. Outside of that, poetry has been pretty easy and intuitive for me
Poetry solves real problems, but also reinvents many aspects of packaging in an almost-but-not-quite compatible way.
I've spent an unreasonable amount of time hunting down Poetry bugs or figuring out ways around Poetry limitations. But not using Poetry isn't an option either if you need some of its more unique features (Poetry offers the only widely used lockfile format in the Python ecosystem and has much better support for private package repos than pip-based tools). The integrated venv management also provides really good developer experience.
I used pipenv at my last position, and that one was a pain
I know it’s an unpopular opinion, I like pipenv. Simple. Works fast enough, easy to use in docker.
Poetry. Slow, uses a weird pyproject.toml structure, flaky
pyproject.toml is a PEP. https://peps.python.org/pep-0518/
I used to think it was poetry too. I found this article enlightening. https://realpython.com/pypi-publish-python-package/
I haven't tried flit yet. I will.
Poetry's use of pyproject.toml to specify project metadata is non-standard, as it uses a custom [tool.poetry] table instead of the standard [project] table. Furthermore, its dependency specifiers also fall outside the standard.
This all comes together to produce a file that is completely tied to the build backend - thereby defeating the whole purpose of pyproject.toml being a declaration of project metadata that can be used to build a package with interchangeable build backends.
If I start out with Setuptools, I can switch to Hatch down the line without changing anything but my build backend to Hatchling in pyproject.toml. The same cannot be said for a pyproject.toml file created by Poetry.
The file itself is a PEP, but compared to setuptools/scikit-build/other tools that use it, I still find the keys and structure poetry uses to be less readable
I was trying to make a program compare two pdf files and i just kept finding a bunch of shit
It's pdf, it's all shit
I tend to use poppler utils pdftotext Linux utility to dump to text and then compare the text. If they are scans good luck to you
Plotly Dash. It uses React under the hood, but it’s much easier to just write your own front end in React than it is to deal with the abstraction of React that is Dash. Feels like trying to knit with oven gloves on. Things that are super simple to implement in React become a big song and dance in Dash. Baffles me why anyone still uses it
pandas
Pandas API is kind of shit in many ways, up there with matplotlib. That said idk if id be able to write something better.
Now that i know it its very useful but its definitely something you have to get used to
Polars is something better. And plotly instead of matplotlib.
I feel matplotlib is much more intuitive and easy to use than plotly (granted I've been using matplotlib first and only recently discovered plotly). Doing things in plotly feels so cumbersome/complicated with so much nested dictionaries to change a parameter. For example to change the axis limits in matplotlib i just do plt.xlim([[0,100]), in plotly it is fig.update_layout( xaxis=dict(range=[0, 100])), just so much more complicated.
Try plotnine instead. You will love it.
But matplotlib is just a port of MatLab.
Depends on which api you’re using with plotly—they also support direct manipulation via dot notation. That fig.update_layout method is really for when you have a set of defaults or templates or something. If you’re just changing one parameter, I can see why you’re mad.
Polars is fantastic. And for someone learning R first, I rea;y like the syntax of plot nine, which is the ggplot2 equivalent.
Pandas and matplotlib are like Tammy I and Tammy II
I have had my own wrapper for pandas that I've been using for years.
Please publish 🤣
Try polars. It's way better.
I’ve never seen anyone badmouth polars. It’s the perfect storm of replacing a shitty, cumbersome package and having a really good dev community. All my homies love polars.
I've fallen back to writing my own unit tests for even single pandas functions because I don't trust them, and my fears are constantly confirmed when I find weird corners with hidden compound dtype issues that break functions and make pandas behave in ways other than expected. It could really use some work to make it more consistent.
I basically always read everything as a string. Then create a list of columns for each datatype (numeric, dates, etc). Applies to polars too. That way they’re not guessing wrong data types 😂
What don’t you like about it? It has a bit of a learning curve, but makes transforming data super fast once you get used to it! I did stuff manually before, oh dear, how many hours did I waste on that…
"What don’t you like about it?"
You forget the multi index. Seriously does anyone use that?
And I want to use type hints but the linter always complain when using pandas.
If you've used tidyverse in R then pandas feels incredibly cumbersome.
People in my company that use that just for CSV make me very sad.
When you convert the data frame back to JSON and have NaNs 🤬🤬
Agreed, but not because it's bad. I think it is really good.
The problem I have with it is that so many developers use it completely unnecessarily. I have seen too many projects use pandas to do something as simple as sum a list, or create a CSV. It is such a unnecessarily large dependency to have completely unnecessarily.
I had a coworker who loved pandas and he'd sometimes have scripts that were unreasonably slow. I'd say "it's probably pandas" and he'd laugh, and then id inherit the code, remove pandas, and the execution time drops from like 5 minutes to a couple seconds.
A performance hit of 100x is very common if you're iterating over rows or otherwise using pandas but not using numpy/pandas idioms
The number footguns in PynamoDB makes it very annoying to use at times.
And the shit part is that a lot of Python MQ/Kafka libraries either don’t support Kafka or don’t have any type stubs and wrong/zero documentation
tensorflow
Glad it's almost dead now.
Tkinter. When I first started making GUIs.
Then I abandoned Python for GUI apps, and switched to HTML, CSS and vanilla JS. So much better on every front :)
1) Separation of styling, markup, and code 2) stylesheets- not everything is an inline style 3) CSS has easy media queries, flexbox, grid. All great for responsive design 4) Much better docs 5) Devtools!
In my efforts to avoid Tkinter, I accidentally became fullstack
Why let users do what they want to do in 5 seconds when it might take 5 minutes instead?
If you love html, CSS and vanilla JS, you gonna love svelte.
I feel the same way about Pandas, numpy, matplotlib but I have to come to the defense: What these packages do is really hard. I also use R for statistics and data dicing/slicing tasks, and things that "kinda work" in R need to be made much more explicit in Python. Which I actually appreciate but still, ... it's hard. Sometimes I try to change something in Pandas code I wrote years ago and I can't wrap my head around how it works. Occasinaly there are also unhelpful comments by my former self along the lines of # I don't know why this works, but it does, so don't touch it!!!
Ray
Please tell more!
That looks like one of those libraries that handles complicated stuff in such an appealing way that it's immediately obvious it's a lie.
Oh oh. Yea this one is complicated...
Oh I've been wrangling with Ray the past 4 months. Needless to say, the deadline for my project was yesterday, now I'm hoping I'll meet the 3 month extension... 75% of the things I had to debug weren't even my own code, but the library itself. Right now, things are sort of running, but they eventually crash. Why? I couldn't tell you. So instead of asking why, I decided to just let it crash and recover from there when it does instead. It doesn't work. Why? I couldn't tell you.
matplotlib, I HATE IT SO SO SO MUCH.
json - all these years and the docs are still a good example of what not to do
Oh hey, I use it all the time !
Could you tell me what I shouldn't do with it ?
It could very much be useful to me down the line.
I believe they are talking about the docs only, not the module itself.
Improve them! Contributing to CPython isn't hard
This is one I can't do without!
solace-pubsubplus
Ignoring the ungodly amount of OOP abstraction for everything, my braking point was when for the first time in my life I got java.nullpoint.exception from running a python program.
Databricks. Apaarently, they changed it, so you can't use it in your IDE without a premium feature (Unity Catalog), forcing you to develop in their god-awful notebook """environment""" instead. It has zero conventional tooling, a barely working debugger and close to no version control. Have fun convincing your management that you actually need premium.
Databricks notebooks are a travesty of software.
Their entire data processing flow is a nightmare.
They are the new Oracle.
Airflow
I can't describe how good airflow is. Im moving the company I work for to airflow
Really? I think airflow is pretty great!
Well that's a pretty strong opinion. I'm willing to test an alternative. What do you recommend? Dagster?
I highly recommend Dagster. We use it a lot at work and although it has a high learning curve, the docs are pretty good and the functionality is amazing
I was a user when it was 1.x . It was a nightmare. Maybe its better now?
2.9.1 ? You bet it's much better.
I've been using since 1.9.X and I'm really happy with the changes.
Nice to hear, was a choice when I was choosing an orchestration tool back in the day. Ended up going with Prefect.
What do you think of Prefect? I have tried it and am trying to get my .org to implement it, but I have not heard much about people's experience using it long term / in prod.
Just picked it up again last year after 5 years of not needing to orchestrate anything. As a glorified cron, I think it’s great for orchestrating all my etl and training. And their docs have gotten way better. Dashboard and tagging, exactly what I need them to be.
Haven’t run it in an org setting but Reddit had the most reviews when I was asking myself the same question. My takeaway was that there’s nothing wrong with prefect exactly, but the functional programming paradigm that makes it so easy to get off the ground can get squirrelly to manage when a project grows.
Docopt
Click. It’s great at first but if your project grows it will eventually get in the way, and then it takes a lot of work to refactor out of it.
Honestly pandas. Polars makes more sense to me.
Have you used polars for work?
I have tried and failed multiple times. Bugs are everywhere and breaking changes everywhere. The docs is still pretty bad too, sometimes has to read the code. Might got better after they just released 1.0.0 yesterday though.
Do you happen to remember which bugs you encountered?
Were they "this raises when it shouldn't" kind of bugs, or "this is just a wrong result" ones?
I think it's now much better. And it is more in style of SQL for dataframe transformations than pandas. That helped me to get around the headdache of moving away from pandas.
yeah its way overhyped, useful only for relatively niche use cases. pandas isn't going anywhere.
IMO not really overhype, it is progressing at an impressive speed, just not mature enough.
Big problem of polars Is the missing of a community, it's vert difficult figure out how to do things.
May I suggest the project's Discord? https://discord.gg/4UfP5cfBE7
And yet here’s me (admittedly an amateur) struggling with Polars error handling and having to fall back to Pandas
It's because a lot of polars apologists don't actually use it that much for serious work. They just really like the idea of it.
I can't speak for everyone, but there's several examples of companies using it for serious work, e.g. G-Researc https://pola.rs/posts/case-gresearch/
It's bandwagon hype for the shiny new thing. Still lacking pretty basic features, at least the last time I tried it.
Tried to do something relatively simple, at least in pandas, and the polars workaround was some convoluted mess.
Pandas isn't going anywhere anytime soon. Just add duckdb if you hate the syntax so much lol
Which basic features did you find missing?
Sounds like `matplotlib`, cannot even get rid of it.
Plotly all the way
pip-tools… because what if we made dependency management even harder and more convoluted than anything else?
Interesting, what's complicated about it?
https://github.com/mfesiem/msiempy
no documentation, just a "demo script" that creates and .. then deletes via api all the rules of you production siem.... NICE.
kivymd, package support is dropped the second a new version is introduced, stable packages are nowhere stable, buggy as fuck, bloated,slower than it should be, kivy langluage makes no sense to use, should have learned kotlin instead. i was stupid. fucked around with it, released an app, had fun. 8/10 i do not recommend.
This is a very old decision.
4-Suite
It was a powerful, performant XML library, providing a Python interface to a C implementation. I built a fairly big static site generator, using it heavily. But it stopped being updated and with it having binary components, became incompatible with later Python 2 versions, nevermind 3.
I hate pandas. It’s course, and rough, and gets everywhere.
dask.
When i first found it, i was amazed, thought it would solve all my Problems. But after a while… it just made them Worse.
Not a Problem with dask, rather a Problem with my own expectations.
Polars
Is it just me or is Polars vs Pandas becoming the next "vim vs emacs"?
You mean internet users are going to arbitrarily pick a side and get overly defensive about it? No way…
Polars is the new cool kid. Pandas is the old friend that’s always been there but it’s time to move on 😂.
Why don't you like Polars? I migrated some on my code from Pandas to Polars and really happy with it.
How? Why?
im also curious. polars just hit 1.0 and is gaining a ton of momentum. syntactically i much prefer it over pandas and it is much more performant
I just hate pandas syntax. Every good library should name their join function JOIN as sql is just the standard. But no pandas must be different and must use merge, because pandas join is joining on index (defaults to row number)
I am seconding this one. The idea and API organization of polars is top notch. The fact they just had a 1.0 release terrifies me though. There are so many sharp edges within the library still.
Lets take a simple case: load some data out of a database and into a polars dataframe, do some transformations, save the dataframe to a parquet file.
Oops you had a sparse column in the database (many nulls). You just blew up because the streaming inference window was too small. So you set the datatype on the column ahead of time.
Oops you tried to do your transforms lazily? You know that column you just added types to? Well now that data type is lost from the streaming data type inference baked into the lazy evaluation.
Oops you tried to add types to the output of the transformation? That type is ignored. Pump your inference window up. To what? Guess it just has to be the length of the returned results to avoid further problems.
Finally you get your data transformed and you want to save it out. They provide a nice polars.DataFrame.write_parquet method. We are home fre- NO WAIT BOOM. Their serializer for parquet does not support all their own data types that can be represented in a dataframe. After some digging around you figure out it is the UUID row ids causing the issue. These get represented in the dataframe as a pl.Object. Ok no problem we will just cast them to pl.String an- BOOM. You cannot use pl.Expr.cast on an object. So now you are forced to use the self-proclaimed slow .map_elements() API with this gem.
df.with_columns(pl.col(pl.Object).map_elements(lambda x: str(x) if hasattr(x, "__str__") else x))
You got fed up and wrapped your polars transformations in a except Exception? Oops. Polars throws a bunch of pyo3_runtime.PanicException all over the place and it inherits from BaseException, not Exception. Polars provides a polars.exceptions.PolarsPanicExceptions alias you can catch, but this behavior took a bit to track down and is not what I could consider normal behavior for python application code.
I wanted to like polars so much, but I have had much more luck with duckdb for these types of tasks. The sacrifice of polar's nice clean native python API was worth the consistency of behavior I got from duckdb.
fastapi - tried it and its fun. now im stuck with an old version that im having hard time to upgrade. pydantic is now v2 and im using libs that are not async compatible. boto3 n stripe.
Not much a FastAPI issue then.
I never have time to upgrade my projects, but recently, I discovered Renovate.
I strongly recommend you to use renovate or similar tools, because it’s automatically bump your packages. If you are confident enough to your tests, you can also auto-approve/auto-merge renovate PR.
Major updates are kinda Easter’s to upgrade when renovate did 75% of the job !
There's an unofficial version of async boto which has been popular for years
Built some hacked together connector with fastapi a while ago and it runs way too well, so now our application core relies on that api to get data from an obscure data source
tried hatch, but man poetry just is a lot more simpler and fits a lot more use cases. I get what the hatch devs were going for tho
also, any api wrapper that’s auto generated or any api wrapper that just returns dicts. at that point I just want to make the request myself, most of the time api resources are state dependent anyways so why not make atleast a dataclass
ibis api is also not for me, not a big fan of R dataframes and using R like syntax in python also makes me cringe
pre-v1 openai wrapper was pretty bad, a beautiful piece of over-engineering that was a neat example of what happens if you have a design pattern in mind but apply it without any regard to actual developer experience. You ended up with objects on which you had to hasattr or catch AttributeError as a normal, recommended way to use them. And it was all just worse representations of the plain json payloads - it's just a rest API, having wrappers is barely necessary in the first place. If you used the wrapper then it's even more difficult to catch "out of credit" errors or get the amount of consumed token - just because, it ended up not being exported in the weird objects the lib gave to you.
They've moved on to pydantic wrappers since then. Not a bad move but I don't find the dependency to pydantic necessary either.
Celery
A lot of code and a lot of behind the scenes magic. Abstracts away from the message broker which makes it really hard to use the broker's own observability and monitoring tooling.
Wish I had directly used RabbitMQ with pika.