Convert data into 3D assets using Dynamics 365 Guides, AI Builder, and Power Automate

As digital transformation drives innovation and transforms the way we work, we see business operations and traditional internal processes being disrupted daily. To adapt and scale changes across their business, many organizations are overwhelmed and faced with digitizing thousands of manual checklists, paper forms, documents, and more. The good news is today’s Microsoft solutions make this transition easier than ever before. With Microsoft’s suite of low-code development solutions (e.g., Microsoft Power Platform, Power Automate, and AI Builder), employees of all skill levels can create and customize business applications and processes. By combining these solutions with Microsoft Dynamics 365 Guides, a mixed reality application that offers step-by-step, hands-free work instructions and collaboration, you can convert your existing 2D manuals into 3D holograms, rather than transferring manually. Once converted, organizations can use Dynamics 365 Guides to provide immersive, on-the-job guidance that will increase efficiency, increase knowledge retention, and enable your organization to adapt at the speed of change.

Better together: Dynamics 365 Guides, AI Builder, and Power Automate

With Dynamics 365 Guides, authoring instructions is as simple as creating a presentation in Microsoft PowerPoint. Using the Guides PC and HoloLens apps, you create and edit instructions by dragging and dropping 2D and 3D content into your real-world environment to show users how and where to complete tasks. However, converting an existing multi-page manual by copying the information from scratch makes the authoring process feel time-consuming and daunting. This is where Microsoft Power Platform AI Builder comes in. AI Builder empowers anyone to build, train, and publish AI models, enabling you to infuse AI into your regular processes. You can create and train a custom AI model to extract text information from a variety of paper manuals in minutes, or use one of the pre-built models.

Using Power Automate, you save the extracted information to Microsoft Dataverse (an enterprise-grade intelligent scalable data platform that powers business applications like Dynamics 365 Guides). By storing the data in a format readable by Dynamics 365 Guides, users can simply open the Dynamics 365 Guides PC and HoloLens apps and add 3D content on top of the text instructions extracted from existing manuals.

In this blog, we’ll focus on how to use the AI Builder form processing model to extract and convert information and create a guide. Every organization has work instructions that come in different formats, such as checklists for inspections, training documentation, and standard operating procedures. By using the form processing model, you can read and save information from existing documents such as images or PDF files.

When you automate this process, you save valuable time by reviewing, extracting, organizing, and saving the data automatically by using Power Automate and Power Apps.

Overall process

Creating a guide using a custom AI model and Power Automate requires three simple steps:

1. Train an AI model with AI Builder to extract text from a .PDF file or image.
2. Take extracted information from the AI model and create a guide using Power Automate.
3. Enhance the guide further and author in the Dynamics 365 Guides PC and HoloLens apps.

For a closer look at each step below, follow along with a team at Toyota Motors North America in this video, at the 2:40 min mark.

Step one: Train an AI model with AI Builder to extract text from a .PDF file or image

1. Create your AI model: Choose “form processing” (under the “Documents” area of the AI Builder homepage) and go through the step-by-step wizard.
2. Define the data you want to extract: Define the fields, tables, and checkboxes you want to teach your model to extract. See the create a form processing custom model documentation for more details.
3. Upload your documents: Upload at least five different versions of the same basic document. AI Builder also supports collections of documents that have different layouts. See the AI Builder form processing models blog for more details.
4. Select important areas, and then train your AI model: Teach your model areas it should learn aboutand then train it. Training is an automatic process where AI Builder “teaches” your AI model to understand a specific instruction type, such as a PDF checklist.
5. Publish and use your new AI model: Now you can automate document processing in apps and flows.  

Step two: Take extracted information from the AI model and create a guide using Power Automate

1. Extract the data: Create a Power Automate flow to take the extracted information from AI Builder and create a guide with step-by-step instructions. You can do this in minutes and no coding skills are required.
2. Create a Power Automate flow and convert data into steps: Build each piece of data as a “step” for your instructions with easy-to-follow drag-and-drop functionalities. This is where you reference the Dynamics 365 Guides data in Microsoft Dataverse. By harnessing Dataverse, we can connect data so that your instructions are now editable from the Dynamics 365 Guides PC app.

Step three: Author in the Dynamics 365 Guides PC and HoloLens apps

1. Make edits in the PC and HoloLens apps: Once content for your guide is in the right spot, you can author in the Dynamics 365 Guides PC and HoloLens apps to make edits and changes.
2. Optimize for 3D experience: Customize the steps to better suit hands-free needs based on a 3D, mixed-reality experience. Add links, photos, reports, 3D models, and other resources.

Whether creating a single manual or hundreds, AI Builder and Dynamics 365 Guides can transform instructions and training, making your processes easier, smarter, and faster. Seamlessly capture, translate, and extract information to unlock a new way of working and save your teams hours of time.

“Customers have been able to save months of deployment time by using AI Builder and Guides, allowing them to quickly integrate mixed-reality work instructions into their daily workflows without any development expertise. They’re able to update content they already have into 3D holographic instructions for a variety of scenarios, including assembly, changeovers, training, and service and maintenance, resulting in improvements to safety, efficiency, and reduced onboarding time.”Shirley Ho, Senior Designer, Dynamics 365 Guides.

Getting started

Ready to design a guide using AI Builder, Power Automate, and Dynamics 365 Guides today? Here’s how to get started.

• Power Apps: A Power Apps or Power Automate license is required to start using AI Builder. For more details about Power Apps and Power Automate licensing, see the licensing guide.
• Dynamics 365 Guides: Try the 60-day free trial of Dynamics 365 Guides. Learn about licensing requirements.
• AI Builder: Models in preview do not require an AI Builder trial or license nor consume credits. Anybody can test AI Builder and start a free trial today.

Other resources

AI Builder in Power Automate

• Learn more about AI Builder and get your questions answered using AI Builder documentation. Make sure to see these topics:
  • Form processing model overview
  • Object detection model overview
  • List of AI solutions

Dynamics 365 Guides

• Watch our Toyota North America customer video that showcases Dynamics 365 Guides and AI Builder in Power Automate.
• Read more about Dynamics 365 Guides capabilities and get your questions answered using our Dynamics 365 Guides documentation.
• Take the Work with Dynamics 365 Guides learning path.

The post Convert data into 3D assets using Dynamics 365 Guides, AI Builder, and Power Automate appeared first on Microsoft Dynamics 365 Blog.