[Chatbot Copilot] - We Built a FeatureScript AI Assistant That Outperforms ChatGPT

guillauem_d · December 2024

Hey everyone,

I’m a mechanical engineer, and I’ve been working with FeatureScript at my company for about a year now. Honestly, it’s been tough. Tools like ChatGPT suck at FeatureScript, which is frustrating when you know how useful they are for more familiar languages like JavaScript or Python. Unless you're a FeatureScript expert, I think we can all agree that productivity takes a big hit compared to more common languages because there’s no good LLM-powered copilot to help out.

So, we fixed that ! We built our own AI assistant, specifically for FeatureScript !

It’s not perfect, but it’s way better than ChatGPT in this area. This isn’t just a basic GPT model with the Onshape Standard Library as input—two of our machine learning engineers spent six months building it. They used a technique based on a paper called “Large language model multi-agent collaborations”. I’m no, expert but basically, they created a system with reasoning steps, multiple agents working together to analyze questions, generate code, cross-check the documentation, test, and iterate until the output works. It uses RAG, agent-based reasoning, and multiple LLM calls to get the job done. Internally, it’s been a game-changer for how we work with FeatureScript. And while this tool clearly doesn't make us better than a featurescript expert, it does make us infinitely more productive.

Don’t get me wrong, if we had 10 million dollars to label tons of FeatureScript data and fine-tune a big model, that would be even better. But for now, this approach is giving us a really good productivity boost, and we’re pretty excited about it.

Now we’re wondering if we should share this with the outside world. There are a couple of big questions:

Would people pay for it? Let’s be real: running all these models and doing this whole chain-of-thought process isn’t cheap. Plus, they put so much work into this. They’d have to charge a monthly fee. For professional work, it’s a no-brainer—just a small productivity boost pays for itself almost immediately. But do you feel the same way?
Is there enough interest out there? Are we looking at a niche tool that only a handful of people would find useful, or is there a real market for something like this?

I’d love to hear what you think. Would a specialized FeatureScript assistant be worth the cost to you or your company? Are we barking up the wrong tree, or does this seem like something that could fill a real need?

Thanks for any thoughts you can share !

S1mon · December 2024

I'd love to see some YouTube demos before commenting too much.

That said, I would prefer that PTC/Onshape develop a first party solution to this problem. The problem isn't so much that we need an LLM powered assistant, it's that there are people who would like to build something like a feature that could be re-used, but is easier to do than fighting with queries and sketches in FeatureScript. Solving the FeatureScript usability issues with an LLM may be helpful, and I would certainly be willing to try it, but I'm not sure that's solving the right problem.

I feel like FeatureScript represents a huge, mostly untapped potential to change the way parametric mechanical CAD is done. Despite spending a fair amount of time going through tutorials and following the discussions on the Forum here, I find myself very frustrated by my progress or lack thereof. I haven't touched Grasshopper (parametric visual coding for Rhino) in a long time, but my memory of it was that it was so much easier to just get started and make something useful. I wish there was something that would perhaps live above FeatureScript which would be that simple to use, or put another way, I wish there were a way to to connect regular Onshape features into meta-features which could be re-used.

EvanReese · December 2024

It's definitely interesting, and feels like an inevitable thing that will exist. I think I share Simon's feelings though. If it really worked it would be valuable. Even more so if it could get Onshape users from non-FeatureScript users to FeatureScript users. That initial learning curve is steep (or was for me anyway). I also use Grasshopper from time to time, and agree that some kind of node-based FS builder could be huge for enabling people who are currently intimidated by FeatureScript. The two would complement one another, not compete.

jelte_steur814 · December 2024

I think something like this is inevitable. It's just a matter of when…

I think many non-featurescript coding Onshape users are technical enough to create a featurescript without understanding the syntax if it could be coded with natural language. And LLM's are the intermediary layer to make this happen.

Agree with Simon and Evan that this will eventually need to be integrated into Onshape.

for the non-FS-Coding lot you're looking at problem urgency (Coelen scale) two or three. not an easy sell.

for professional coders; IF their productivity would indeed go up a lot, it would be an easy sell indeed. question is how many are out there. probably urgency 5.

I think a freemium model (so people can experience the benefit) with limited tokens, a few subscription tiers with more tokens each and the ability to buy tokens separately for irregular users would be a beautiful model (sorta like OnScale).

MichaelPascoe · December 2024

Congrats! Video demonstration?

The standard rate for integrated AI is around 10 to 15 USD / month. This is the cost of git copilot for visual studio which is a must have for quick questions and repetitive work or code clean up.

Assuming it is as useful as git copilot, I would pay 15 to 30 USD / month for a tool like this. It would need to be an integrated app in the Onshape app side panel, I wouldn't want to have to go to gpt's website for every question. Not sure Onshape allows apps for feature studios though, so it might need to be a web extension that can read the featurescript on the current page then insert or replace code.

guillauem_d · January 7

Thank you everyone for your precious feedback and support! My company is excited to continue working on this project and actually launch something! There's still quite a bit of work ahead, but we're making progress. We're focusing on 2 key steps :

Currently, the inference process is really expensive—around €30/month barely covers the calculation costs with medium frequency usage. We're actively working on optimizing everything to make it more affordable and scalable.
We need to decide on the best way to deliver this tool—whether as a chatbot website, browser extension, or an Onshape app side panel. This is still under discussion and will take time to develop.

Once these challenges are resolved (hopefully within a month or two), we plan to launch a free beta to gather valuable feedback from the community. If you're interested in being a beta tester, feel free to send me a private message with your email address.

Thanks again for your thoughts and encouragement—this community has been incredibly helpful!

guillauem_d · February 14

Hey everyone,

We’ve been working hard on this, and now it’s finally here! Our FeatureScript AI assistant is live, and you can try it out for free at:
👉 https://mecagent.com/login

To get an account and join the beta program, head over to our Discord:
👉 https://discord.gg/teWeTp3G

A couple of things to keep in mind:

🔹 This is a beta. While the chatbot on mecagent.com is already better than ChatGPT at FeatureScript, it’s not as advanced as our internal AI expert model—simply because running these models is expensive. We’re rolling out improvements step by step, and that’s where we need your help.

🔹 Feedback is key. Your input is crucial for making this tool better. Please share your thoughts on Discord and use the thumbs-up/down buttons in the chatbot to help us refine the responses.

🔹 Three specialized chatbots. On mecagent.com, you’ll find three different AI assistants, each specialized in a specific area. Make sure to choose the right one for the task at hand.

The goal is to make this assistant as useful as possible for FeatureScript development. As we refine the model, beta testers in Discord will gradually get access to more advanced versions.

Looking forward to hearing what you think!

(Be gentle, we pay the tokens😅)

Caden_Armstrong · February 14

~~first feedback: there's no signup page…~~

spoke too soon, sign up is through discord.

Caden_Armstrong · February 14

I've spent ~15 minutes playing around with it, and here are my first impressions:

I tried asking for a few simple features, but as expected - its not able to produce anything working, even given the lowest hanging fruit. I've naysayed this enough times that the forums don't need another rant from me on the topic.

But I will be nice and evaluate it on the other more reasonable use cases described: documentation and question answering.
I asked the chat a number of easy enough questions, including a few real questions asked on the forum.

The good -

Responses have citations.
I tried to see if any data had been stolen to train the model - and nothing obvious comes up.

The bad -
It invents new featurescript functions that don't exist (or syntax, or doesn't use them correctly, etc)
It gets answers to questions wrong more often than it is correct.
- It gets even the simplest questions wrong, like "how do I join two lists"
The citations don't match the answer. I'm seeing "opBooelan" as the source for joining two lists, and nearly every answer i get.

I think there is use and value in having an AI that can answer questions about featurescript. Many questions can be answered by reading the documentation, which can be a bit overwhelming if you don't know the correct words to search. But at this time, your AI would be more frustrating than useful to a new featurescripter.

I'm curious what your plans are for training data to improve your model?

MichaelPascoe · February 17

Tested it out with my generic FS test prompt test for FeatureScript. Failed to respond due to hallucinations. I'll wait to continue testing until the hallucinations are fixed. Those are a deal breaker.

Feedback:

Hallucinations will keep it from producing usable FS
Sign up process was a bit painful
Requires standalone window or tab which means lots of copy / paste and back and forth. Not sure of a good solution for this since Onshape doesn't allow side panel apps for feature studio. You might be able to have your chat as a web extension which could then see your code selections and insert / replace.

markus_cox · February 18

I've just trained gemini ai to write fully working featurscript in only a few days and several deep research dives its now able to do pretty complex things. Errors are an issue, it doesn't make many mistakes if any but it'll forget what its working on and think its working on a earlier project, I have to remind it what were doing then it gets it abd goes from where it left off. First few days it struggled but now it understand how to code so it dosnt product errors it can do a fully tested working script in about 20 minutes, it'll create the whole script then I spend the next 20 min copy and pasting each revised code in to onshape, confirming its working then repet untill its done. I'm seriously surprised how well its working, II'm Just having issues using it in onshsoe because onshapes not showing it as available

[Chatbot Copilot] - We Built a FeatureScript AI Assistant That Outperforms ChatGPT

Comments

Howdy, Stranger!

Quick Links

Categories