In SAP, tax rules are spread across customizing, tables and custom Z code, and the AI can't see any of that just from the task description. In companies, what's useful is having the AI review and explain, and leaving the decision to whoever knows the process. At least that's how I've been using it in my own work, with great results: going through change requests, trying to understand the impact they might have, and so on.

As a proof-of-concept, I've been using extra tokens to formalize the IRS tax forms and worksheets, mostly just to prove this can be done. The idea is that something like this could be adapted into a publicly maintained version of the IRS Direct File program (RIP). I'm up to an estimated 60% coverage of US federal tax situations so far.

https://github.com/emmilco/us-tax-model

I prepared my taxes this year using Astra + Codex. It did a thorough job on a complicated return, and I sent it to my accountant to fill out the tax form, with everything laid out in a spreadsheet. They sent it back for final review, and I had Codex check it; it was riddled with errors. The refund was nearly $2k lower than it should have been, and they put a bunch of stuff into the wrong fields. Had to go back and forth multiple times to get it 100% correct. Made me wonder how many previous years' returns were wrong.

I feel like we're going to get to a point where it's mandatory to use AI for certain tasks - humans just can't be trusted to get things right that require a level of precision.

This is about wild west accounting, though, when the user has no idea what they're doing.

When launching my financial business, I spent half a year learning accounting to be sure to understand everything, then I implemented my accounting software, specialized for my activity. And as luck had it, I made it be a CLI. This last few months, I've been using Codex(lately with GPT-6.1 Sol) to do the annoying work - attaching proofs to entries, fixing the dates and amounts issues, etc. It works on on a database copy, generate a script of commands using my CLI to fix and close the previous month, that I then review before executing on my prod database, sometimes asking questions to the agent about why this or how that. … And it's been flawless so far, both on my professional accounting and my personal one. Knowing a bit about what we ask the model to do can go a long way.

Same story here as many others: I now do my personal and business taxes myself using an application I had claude cook up for me, doing it 100% correct, for South Africa's tax system. Previously I paid thousands to a small accounting firm to do it for me as it can be slightly nerve wrecking, but I had claude explain to me exactly the mechanisms on how everything works, including the tax portal that the government provides to do submissions. Absolutely amazing. Oh and the filing took me 5 minutes to do myself and cost me near zero once it all clicked into place.

I have a non-trivial tax situation, as a US citizen living abroad. I prepare my own tax returns, never having found a cross-border tax preparer I trust, and have for the past few years, fed my draft return along with all the supporting paperwork into Claude. It has:

1. found errors, e.g. where I used the incorrect exchange rate (and managed to work out how I had made that error, which I was impressed by)

2. found items which would appear to be errors except for context it did not know

3. flagged areas where it disagreed with me as to the correct tax treatment, but accepted that the law is ambiguous (and in some cases I preferred its interpretation after additional research and in other cases I did not)

4. flagged areas where it disagreed with IRS guidance, and only acknowledged said guidance after I provided links to it

5. provided highly inaccurate estimated tax calculations, and not been able to fix its estimates even when challenged

So my takeaway is that it is a very useful tool for tax, but perhaps unsurprisingly, it is not good enough to dump all your paperwork into and expect a shiny return you don't need to check. (Possibly it is if you just have a W-2 and no other substantial income, but I can't speak for that scenario.)

My tax accountant mis-filed my taxes a couple years ago, which cost me about $20K (which I learned after I switched to another accountant later). Don't know if Claude / GPT would make the same mistake or not given SOTA today, but feels like there's an opportunity for models to be saving me thousands or tens of thousands of dollars a year.

Everybody is saying their tax accountant messed up and the models caught it which makes me think, why aren't tax accountants doing the same thing lot of us are doing - ask LLM to do the prep and they just review and finalize?

Can't be good getting caught with mistakes and losing customers?

Is this just giving a general-purpose agent a workspace full of documents and an assignment, then seeing what happens? I feel this doesn't really say anything. It appears they mostly dumped all of this in base models with little to nothing in the way of infrastructure to support it. If you just randomly grab some person off the street and say 'do taxes' they would probably mess up just as badly. The interesting question would be what the models could do in a well engineered harness focused on taxes. That would be reflective of the performance boundary and how serious we would have to take AI in that field.

I find TurboTax is easy enough to use, except that data collection and entering is a huge hassle (to me, merely logging onto each website and getting the corresponding tax doc is painful chore). What would be a good way to collect the data easily? If I solve that problem, tax filing will become an easy task.

I'm surprised Sonnet 5.5 did noticeably better than Opus and Fable. Same pattern in their other benchmarks.

There is currently a bill in Congress to finally allow regulation of paid tax preparers. Not sure if addresses AI, but probably not.

https://www.accountingtoday.com/news/new-bill-calls-for-irs-...

There is also an IRS form (14157) to report a return preparer complaint, which of course does not include an option to report a faulty AI.

There's also: https://github.com/column-tax/tax-calc-bench

I'm not sure LLMs will be good at doing taxes any time soon primarily because they're very bad at even simple math unless it's explained in words like in proofs. A good illustration of this is asking an LLM something like this: "If I put a container on a food scale, tare it, add 100g of peanuts, tare it, weigh 100g of almonds and then remove the container from the scale, what does the scale read?" LLMs completely puke all over themselves on this one. They do not account for not taring after the almonds and they do not see the weight of the container needs to be subtracted. The correct answer is minus (100g + container weight). Although this particular question also exploits another LLM weakness which is physical world concepts. LLM are just guessing. For an LLM to do taxes well, it would need to be trained on many scenarios and it would need a math service that it can use as a support service.

I wasn't able to quickly see what tools/system-prompts were provided in the tasks. [0]

These types of detail-oriented professional tasks seem like they would perform better with task-specific harnesses. The example they give, of Astra missing the property excluded gains requirement, seems easily fixed by scaffolding ensuring that requirements are compiled first, then checked.

If the scaffolding is absent, is this benchmark testing that a model can self-scaffold, or track requirements in-context?

0 - https://huggingface.co/datasets/surgeai/DAYJOB-tax/tree/main...

I had Gemini help me with RMD calculations for an estate I manage. Advice was mostly good. Wrong about withholding for both US and CA. Missed deductions on US. Was accurate on MFJ and spousal death impacts.

I've found it so far more effective for preparation than for planning/advice.

I've got a plan to feed the laws and rules and instructions into the machine before the prompts - is RAG still cool? Maybe I should small-language this?

Not related but I hope QuickBooks learn something and make their software better.

I downloaded 1700 of my transactions and use AI to build an app for it, so I can analyze the transactions and make sure that everything is properly categorized. It used to take me whole day because of the Quickbooks. It took me less than an hour to go through all the transactions.

Sharing all my financial info with LLM companies is not something that I like.

EDIT: good point that paying accountant does not ensure avoiding it

wonderful, you made a good point that make me dislike situation even more

wonder if we should bring back https://taxheaven3000.com/ in the age of ai

What is human baseline? How it stacks against 10% LLM accuracy?..

i want to always have a fast way of reviewing my tax returns without having to hassle my accountant or hire another one.

As a counterpoint, I ask the big llms every month the same set of tax questions about the last US tax bill and they still can't give me the correct answers.

I started my career fixing the mistakes that my clients' former tax advisers made. It looks like the next generation of tax lawyers will get to do the same except this time the former advisers will be llms.

Isn't writing out the example in the beginning that Astra missed basically ensuring that every future model will get this task correct?

I guess my point is these benchmarks unless they are constantly refreshed with new scenarios and not ever having their outcomes written about publicly or available on a public server, will always be prone to being gamed

Useless benchmark as it seems to be evaluating literally only the weights.

Ah, yes, the 'dude trust us' bench.

Kinda points to another thing... that our tax code is insanely bad.

Article I hoped to read: Evaluating AI agents on 50 professional tax evasions

I used Grok Bot and found out that I could offset some stock value going down and get back ~$300. I gave it my bank statements, email access (which had my stocks monthly report as attachment) and a few other things.

I'd have honestly never been able to do it myself - its just too tedious.

So are the accountants completely fucked yet?

[flagged]

[dead]

[flagged]

[flagged]

[flagged]

[flagged]

[flagged]

[flagged]

[dead]

[flagged]