Powered by AI

Turbocharge data ingestion with Osmos AI-powered homepage

Written by 
Vijay Sarad
July 18, 2024
Visual of the new Osmos AI-powered homepage with the main Osmos navigation bar along the right. An astronaut icon flies across the center of the page above text boxes, prompting the user to create a pipeline or create an uploader. A chat bar sits at the bottom of the screen, asking the question, “How can Osmos help you today?”

Introducing the Osmos AI-powered Homepage

We are thrilled to announce a significant upgrade to the Osmos experience with the launch of our AI-Powered Homepage. Gone are the days of one-size-fits-all home page experiences;  Osmos Chat is a sophisticated tool that integrates deeply with Osmos core functionality.

Whether you want details about your project or are learning how to use Osmos for the first time, this new, incredibly useful feature gives you access to all the information you need fast. Unlike chatbots that are just a thin wrapper around the latest Generative AI model, our AI-Powered Homepage taps into Osmos APIs and a deep well of documentation, serving rich, actionable help and insights.

  • Find problems quickly by querying the system about pipeline/uploader health. Ask questions like “Show me all pipelines that ran in the last 30 days with more than 5 errors”.
  • Get speedy access to the things that matter. When you ask questions like the one above, Osmos displays a list of clickable cards that allow you to see details for each pipeline. Clicking on a card takes you to that pipeline’s page for deeper exploration.
  • Turn answers into reports. Any time Osmos Chat responds with a table full of data (eg: “Which uploaders have had uploads in the last 24 hours?”), you can immediately sort and filter the resulting table, and download the raw data as a CSV. Turn your weekly reporting exercise into a single question you ask Osmos.
  • Discover a deeper learning experience. Ask questions to strengthen your understanding of Osmos. More than just searching our documentation, the Osmos Chat implementation has full access to our help documentation and advanced internal material used by our Customer Success team, which is used to synthesize meaningful responses to your queries. For example, you can ask, “When would I use pipelines instead of uploaders?”
  • Build API clients faster. Osmos Chat understands the ins and outs of our powerful API, helping you build your use-case faster, from simple job tracking all the way to programmatically querying our powerful Datasets Query Engine.

For managers, the new Homepage improves organizational efficiency and enables them to create a high-level view of the metrics they need most. Osmos Homepage Chat lets any user access the information they need to get the job done.  With prompts like “show me all pipelines that have run in the last week,” managers gain a bird's eye view across all of their pipelines and uploaders, giving them the big-picture perspective on the data being ingested into their system without a dependency on team members to create and run complex queries on their behalf.

Osmos AI chat interface showing a user request for pipelines that have run in the last week. The response lists four pipelines: Daily Acme import, Fee data ingestion, NW Sales, and Email ingest, with options for further details or actions. The interface includes a conversational AI assistant ready to assist with data pipeline management

The new Osmos Homepage is a learning hub for all team members, whether they are in data operations and responsible for uploading data or software developers responsible for system configuration or they’re in implementation or sales and need to upload a CSV fast. By using Osmos Chat, they can access detailed, Osmos-specific guidance that goes beyond simple search. Data Operations personnel can now learn how to use Osmos more effectively by asking questions like “How do I  fix errors after uploading a file?” This allows them to extract maximum value from Osmos’s powerful tools quickly and easily, which results in faster onboarding and significantly increased productivity.

Image displaying the Osmos Chat interface. The user inputs the query, “How do I automatically move a file from an FTP site, map it, clean it, and send it to an S3 bucket?” Osmos returns step-by-step directions for how to complete this previously complicated task.

Build API integration faster


Osmos Chat turbocharges developer productivity. Just provide a description of the code that you need in the chat box and leave the rest to Osmos! The result is a code snippet that stitches together Osmos APIs so that you can quickly write code without having to sift through complex API documentation and figure out how to cobble it together. It’s like having your own personal software developer at your beck and call.

For example, a developer could ask for code that enumerates Osmos uploaders and then enumerate jobs associated with each of those uploaders. Osmos Chat will return a code snippet that uses multiple Osmos APIs with valid parameters to show all uploaders in a project and jobs for each of those projects.

const fetch = require('node-fetch');
const apiKey = 'YOUR_API_KEY_HERE';

/**
* Fetches uploaders for the given project.
* @param {string} projectId - The project ID to fetch uploaders for.
* @returns {Promise<Array>} List of uploaders.
*/

async function fetchUploaders(projectId) {
   const url = `https://api.osmos.io/v1/projects/${projectId}/uploaders`;
   try {
       const response = await fetch(url, {
           method: 'GET',
           headers: {
               'Content-Type': 'application/json',
               'Authorization': `Bearer ${apiKey}`
           }
       });

       if (!response.ok) {
          throw new Error(`HTTP error! status: ${response.status}`);
       }

       const data = await response.json();
       return data.uploaders;

   } catch (error) {
       console.error('Error fetching uploaders:', error);
       throw error;
   }
}

/**
* Fetches jobs for a given uploader.
* @param {string} uploaderId - The uploader ID to fetch jobs for.
* @returns {Promise<Array>} List of jobs.
*/

async function fetchJobsForUploader(uploaderId) {
   const url = `https://api.osmos.io/v1/uploaders/${uploaderId}/jobs`;
   try {‍

const response = await fetch(url, {
           method: 'GET',
           headers: {
               'Content-Type': 'application/json',
               'Authorization': `Bearer ${apiKey}`
           }
       });

       if (!response.ok) {
           throw new Error(`HTTP error! status: ${response.status}`);
       }

       const data = await response.json();
       return data.jobs;

   } catch (error) {
       console.error(`Error fetching jobs for uploader ${uploaderId}:`, error);
       throw error;
   }
}

// Main function to execute fetching uploaders and their jobs

async function main() {
   const projectId = 'your_project_id_here';

   try {
       const uploaders = await fetchUploaders(projectId);
       for (const uploader of uploaders) {
           const jobs = await fetchJobsForUploader(uploader.id);
           console.log(`Uploader ${uploader.name} (${uploader.id}) has the following jobs:`, jobs);
       }

   } catch (error) {
       console.error('An error occurred:', error);
   }
}

main();

The new Osmos home page is available to all users when they log in today. If you’re new to Osmos and would like to learn more, please contact an Osmos expert today.

Should You Build or Buy a Data Importer?

But before you jump headfirst into building your own solution make sure you consider these eleven often overlooked and underestimated variables.

view the GUIDE

Vijay Sarad