Question 1

What are adaptive machine learning models?

Accepted Answer

Adaptive machine learning models are implemented in production with a process that automatically samples and re-labels data that is sent to the model for inference. This data is used for regularly evaluating model performance online. When model metrics start to decay on the new data, model tuning is triggered automatically. This feature is continually running in the background to adapt Toloka models to your data.

Question 2

How can I interact with adaptive ML models?

Accepted Answer

Toloka offers adaptive ML models as a remote API with methods available in the model interface. Models are available with two modes of interaction: online (real-time) inference and batch inference.

Online inference is when the model provides a synchronous response for each data item sent for inference. This mode is appropriate for use cases when the response is needed within seconds, like customer support automation or content moderation.
Batch inference is when the model posts results after processing an entire batch of data. This mode works better for use cases that involve gathering a batch of data over time and evaluating it, like social media monitoring and analysis.

Our off-the-shelf models support only online inference mode in the closed beta, but custom models can be set up with batch inference support.

Question 3

What infrastructure does Toloka use to deploy and serve adaptive ML models?

Accepted Answer

Toloka has the capability to deploy and monitor models in any cloud environment of the customer&#x27;s choice, or on the customer&#x27;s premises in some cases. By default, Toloka uses Microsoft Azure infrastructure to provide resilient, compliant, and geographically distributed service hosting.

Question 4

What data do Toloka models train on?

Accepted Answer

Toloka models are trained on very large datasets with multiple sources of data. Some of these include open-source data and public data curated with the help of the Toloka crowd. Toloka also uses proprietary non-public data sources that are gathered and labeled via crowdsourcing with the expertise of our R&D team. We use state-of-the-art technologies to generate high-quality non-standard datasets for search relevance evaluation, e-commerce product categorization, evaluation of semantic similarity, and more.

Question 5

What data can be used for fine tuning and optimization of adaptive models?

Accepted Answer

There are 3 ways to use Toloka’s collection of pre-trained models:

Use the models as-is for standard classification, recognition, or generation tasks.
Fine-tune a model on a pre-labeled dataset that you provide.
Deploy a model with the Adaptivity parameter, which triggers the process to collect and label the fine-tuning dataset on a regular basis.

If you are interested in fine-tuning a model right before deployment, you can label your historical data first on the Toloka data labeling platform.

Question 6

How often do models need to be retrained?

Accepted Answer

Toloka adaptive models support schedule-based or event-based triggers for re-training and fine-tuning. The best approach depends on the scenario. This could include re-training once a day, when model accuracy drops below a threshold for new data, or when the "adaptivity" process collects a certain amount of data. We recommend using the default settings designed for each model by our machine learning engineers to best fit the use case and domain.

How it works

Data types

By Industry

By AI domain

Resources

Knowledge Base

In the industry

In science

Adaptive ML models

Why you need
adaptive ML models

How adaptive models work