Evaluation System

Follow

Comments

15 comments

  • Avatar
    ismail ersoy

    Hi isabella,
    I wish you luck for your new evaluation system. However, ı need to state of my view about it and you may take it as a suggestion. Well, this system is too new to be given point but it sounds good. One thing has attracted my attention. You say your in house linguists will evaluate the Tasks that may means these linguists are turkish or have Turkish talent( Turkish is an example as İts my native one.)since they must be. And I would like to ask you that if you have linguists who can give rate to Turkish task, then why didn't you ask them to help us? I mean, they should have been alerted when someone disputed the rate or submist a request about getting unfair rates, so this linguist could have decided if it wasa unfair rate or not. That s All I WANna provide you. Thanks. Bye.

  • Avatar
    Marian Rodu (Edited )

    While I can see why a new and more objective system for increasing quality was needed, it doesn't seem all that fair. We get tasks at all hours of the day and night. They are also very infrequent and diverse so we might feel obliged to take some tasks when we are in a hurry or late at night or losing out completely. That means that most of the time we actually get the tasks we're most likely not in an ideal moment to do them and we probably never will. That means that we are prone to making mistakes and if we get unlucky and lose our paid editor status, then there will be no purpose to keeping an eye on unbabel.
    As previously mentioned, tasks can be very infrequent and if beside this we will have to wait months or more for a random reevaluation that might reinstate us or not, then why use the site at all?

  • Avatar
    ismail ersoy

    I completely agree with marian. How can we be sure that your linguist are fair enough and have the adequate knowledge about our native languages. Because it is beyond of machine translation. Otherwise, to whom will this site service? For which aim? Moreover, that is the most important aspect that it is not really certain how muchtime we will wait for getting our statues back even if we are professionals in our real lives. We wont be able reject or etc. So as marian said we are just obliged to do so.

  • Avatar
    Pablo Ortega

    I completely agree with Marian too. That new system is very unfair.

  • Avatar
    Isabella de Brito

    Dear Editors,

    The Evaluation consist on 5 of your latest 20 tasks done, that can be either Paid or Training Tasks - while the first on depends on the demand, Training Tasks can be done anytime. Also, we trust Editors to deliver *quality* in every task they do, so if you perceive that internal/external factors - lack of time, somnolence, tiredness, or you just feel today isn't your day (it's ok, we are humans!) - might influence negatively your performance editing, we advise you leave it for a while and get back when you are able to give your best.

    Our Linguists check the accuracy of Evaluations based on this criteria (http://help.unbabel.com/hc/en-us/articles/206658027). We trust them and they own Unbabel's opinion and the final word about the results given. Unbabel's Linguists' Team needs enough time to go through each evaluation carefully, because every Editor deserves to be evaluated with the same attention. We hope to cut the time spent on evaluations soon and therefore be able to run more evaluations.

  • Avatar
    Alan Maffey

    Today I got notified of this evaluation and after 20 minutes was given the results (didn't pass, got a 2.2...) and my confidence went down with my score.... haha.

    I know that the evaluation results are final and can't be disputed and all.... but I think this was not properly evaluated.

    For example, I translated this text from English to Spanish: "We use a computer to translate your message into English, and it is not always accurate">"Nosotros usamos una computadora para traducir tu mensaje al Inglés, y no siempre es exacta."

    The linguist said that «computadora» and «y no siempre es exacta», are not properly translated, but for me it's properly translated.

    What am I supposed to write?

    On top of that, the mail said "the last 20 tasks in that language pair" but the ones that were evaluated were not my lasts 20 tasks in that language pair.

    I already know that it's kind of useless to argue this, but I wanted to do it anyway.

  • Avatar
    Isabella de Brito

    Dear @Alan,

    For languages that has two variants (like Spanish, with a Latin American and European variant), is very common to find words or expressions that aren't suitable for both - and we usually advise our Editors to only translate into the variant they feel most familiar and comfortable with.

    These evaluations are run in a frequent manner and in the meanwhile you can work improve your translation skills by doing more tasks.

    Regards,
    Isabella @ Unbabel Support Team

  • Avatar
    Alan Maffey

    Yes, but what about the last 20 tasks part? because the tasks that were evaluated were between the last 200, not 20.

  • Avatar
    Isabella de Brito

    Dear @Alan,

    The 5-tasks evaluation set is aleatory chosen on a range of the latest 20 tasks done in the language pair, containing a reasonable amount of words in order to evaluate the Editors in the best way possible. Tasks with 10, 15 words aren't considered in a evaluation set, for example.

    Regards,
    Isabella @ Unbabel Support Team

  • Avatar
    Alan Maffey

    Lately I've been skipping the tasks that are short so I am really sure that the tasks that have 1 to 20 words are... 6 or 7 in 40. While the ones with 30 to 300 words are at least 33-34 in 40
    .... plus I double-checked on my the "My Tasks" tab to be sure.... and i've only done 5-6 tasks since yesterday.

    http://prntscr.com/apb4dt
    http://prntscr.com/apb4j4

  • Avatar
    Alan Maffey

    And of the 5 tasks that were evaluated only 2 appear here.

  • Avatar
    Pablo Ortega (Edited )

    Hello. The 5-tasks evaluation set is for language pair each one, or is a only evaluation for all the pairs? Is possible that I will approve some pairs and other pairs won´t be approve?

  • Avatar
    Martin Hoffmann

    Yesterday I received three E-Mails that 3 tasks in my language pair we're rated (EN->DE), my mean score is something like 4.4. Can I expect the next 2 tasks to be reviewed that I need to be eligible for paid tasks soon? Thanks

  • Avatar
    Sebastian Ott

    Hey Isabella ^_^/)

    I have a question that needs clarification:

    What happens to translators who already had a license to do paid tasks before the evaluation system launched ? Can they still do paid tasks in these pairs, while being evaluated at some point or is their paid license frozen and they do have to wait until they win the evaluation lottery ? :P

    Several translators - including me - have been wondering about that, because they don't receive tasks anymore.

    Thank you for your time !

  • Avatar
    Martin Hoffmann

    implying they still answer questions in this thread!

Article is closed for comments.

Powered by Zendesk