Best Of

I've spent ~15 minutes playing around with it, and here are my first impressions:

I tried asking for a few simple features, but as expected - its not able to produce anything working, even given the lowest hanging fruit. I've naysayed this enough times that the forums don't need another rant from me on the topic.

But I will be nice and evaluate it on the other more reasonable use cases described: documentation and question answering.
I asked the chat a number of easy enough questions, including a few real questions asked on the forum.

The good -

• Responses have citations.
• I tried to see if any data had been stolen to train the model - and nothing obvious comes up.
  
  The bad -
• It invents new featurescript functions that don't exist (or syntax, or doesn't use them correctly, etc)
• It gets answers to questions wrong more often than it is correct.
  • It gets even the simplest questions wrong, like "how do I join two lists"
• The citations don't match the answer. I'm seeing "opBooelan" as the source for joining two lists, and nearly every answer i get.
  

I think there is use and value in having an AI that can answer questions about featurescript. Many questions can be answered by reading the documentation, which can be a bit overwhelming if you don't know the correct words to search. But at this time, your AI would be more frustrating than useful to a new featurescripter.

I'm curious what your plans are for training data to improve your model?

Hey everyone,

We’ve been working hard on this, and now it’s finally here! Our FeatureScript AI assistant is live, and you can try it out for free at:
👉 https://mecagent.com/login

To get an account and join the beta program, head over to our Discord:
👉 https://discord.gg/teWeTp3G

A couple of things to keep in mind:

🔹 This is a beta. While the chatbot on mecagent.com is already better than ChatGPT at FeatureScript, it’s not as advanced as our internal AI expert model—simply because running these models is expensive. We’re rolling out improvements step by step, and that’s where we need your help.

🔹 Feedback is key. Your input is crucial for making this tool better. Please share your thoughts on Discord and use the thumbs-up/down buttons in the chatbot to help us refine the responses.

🔹 Three specialized chatbots. On mecagent.com, you’ll find three different AI assistants, each specialized in a specific area. Make sure to choose the right one for the task at hand.

The goal is to make this assistant as useful as possible for FeatureScript development. As we refine the model, beta testers in Discord will gradually get access to more advanced versions.

Looking forward to hearing what you think!

(Be gentle, we pay the tokens😅)