Developing artificial intelligence using .NET
Samuele Resca
・1 min read
Originally posted on samueleresca.net
Microsoft has recently released two important AI services:
- Microsoft Cognitive Services are APIs which lets you tap into an ever-growing collection of powerful AI algorithms developed by experts in the fields of computer vision, speech, natural language processing, knowledge extraction and web search;
- Bot framework is a very useful framework to build and connect intelligent bots to interact with your users naturally wherever they are, from Telegram to Skype, Slack, Facebook and other popular services;
This article discovers this services and shows how developing artificial intelligence using .NET.
Demo
The demo shows how to build a Bot that can automatically caption photos sent by users.
Requirements
- Visual studio 2015 community edition , download here;
- Bot conversation emulator, click here to download;
- Microsoft cognitive services subscription, sign in here;
- Visual studio Bot Framework .NET template, which can be downloaded here. To install, save the zip file to your Visual Studio 2015 templates directory which is traditionally in
"%USERPROFILE%\Documents\Visual Studio 2015\Templates\ProjectTemplates\Visual C#\"
Setup the bot project
Create a new C# project using the new Bot Application template:
Visual studio generates a new controller, by default, called MessageController
which is the main entry point of your bot:
using System;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using System.Web.Http;
using System.Web.Http.Description;
using Microsoft.Bot.Connector;
using Newtonsoft.Json;
namespace Bot_Application1
{
[BotAuthentication]
public class MessagesController : ApiController
{
/// <summary>
/// POST: api/Messages
/// Receive a message from a user and reply to it
/// </summary>
public async Task<HttpResponseMessage> Post([FromBody]Activity activity)
{
if (activity.Type == ActivityTypes.Message)
{
ConnectorClient connector = new ConnectorClient(new Uri(activity.ServiceUrl));
// calculate something for us to return
int length = (activity.Text ?? string.Empty).Length;
// return our reply to the user
Activity reply = activity.CreateReply($"You sent {activity.Text} which was {length} characters");
await connector.Conversations.ReplyToActivityAsync(reply);
}
else
{
HandleSystemMessage(activity);
}
var response = Request.CreateResponse(HttpStatusCode.OK);
return response;
}
private Activity HandleSystemMessage(Activity message)
{
if (message.Type == ActivityTypes.DeleteUserData)
{
// Implement user deletion here
// If we handle user deletion, return a real message
}
else if (message.Type == ActivityTypes.ConversationUpdate)
{
// Handle conversation state changes, like members being added and removed
// Use Activity.MembersAdded and Activity.MembersRemoved and Activity.Action for info
// Not available in all channels
}
else if (message.Type == ActivityTypes.ContactRelationUpdate)
{
// Handle add/remove from contact lists
// Activity.From + Activity.Action represent what happened
}
else if (message.Type == ActivityTypes.Typing)
{
// Handle knowing tha the user is typing
}
else if (message.Type == ActivityTypes.Ping)
{
}
return null;
}
}
}
Setup the bot emulator
The emulator gives the possibility to test your bots locally. You can download the emulator here.
The emulator requires 3 params:
- The Url for your bot set the localhost: pulled from the last step. You will need to add the path "/api/messages" to your URL when using the Bot Application template;
- MicrosoftAppId field is NOT required to test bots locally;
- MicrosoftAppPassword field is NOT required to test bots locally;
Important: you need to run your bot project in Visual Studio, to use the bot emulator.
Setup images recognition (Microsoft cognitive services: Vision)
Microsoft Cognitive Services APIs offer the possibility to implement AI inside our applications. There are different services: Speech, Language, Knowledge, Vision and Search. This demo uses the Vision service to caption photos sent by users.
Class schema
Code
Firstly, add the IVisionConnector.cs
interface and VisionConnector.cs
class to your project:
using Microsoft.Bot.Connector;
using Microsoft.ProjectOxford.Vision;
using Microsoft.ProjectOxford.Vision.Contract;
using System.Threading.Tasks;
namespace Blog.BotApplicationSample.Vision
{
public interface IVisionConnector
{
Task<AnalysisResult> AnalizeImage(Activity activity);
VisualFeature[] getVisualFeatures();
VisionServiceClient getVisionClient();
}
}
The IVisionConnector
interface is referenced by MessagesController
and describes main methods which are used by VisionConnector
. API Token is required by VisionServiceClient
to consume APIs. You can get the APIs token here.
using Microsoft.Bot.Connector;
using Microsoft.ProjectOxford.Vision;
using Microsoft.ProjectOxford.Vision.Contract;
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Threading.Tasks;
using System.Web;
namespace Blog.BotApplicationSample.Vision
{
public class VisionConnector : IVisionConnector
{
private VisualFeature[] visualFeatures = new VisualFeature[] {
VisualFeature.Adult, //recognize adult content
VisualFeature.Categories, //recognize image features
VisualFeature.Description //generate image caption
};
private VisionServiceClient visionClient = new VisionServiceClient("<YOUR API KEY HERE> https://www.microsoft.com/cognitive-services/en-us/sign-up");
public async Task<AnalysisResult> AnalizeImage(Activity activity) {
//If the user uploaded an image, read it, and send it to the Vision API
if (activity.Attachments.Any() && activity.Attachments.First().ContentType.Contains("image"))
{
//stores image url (parsed from attachment or message)
string uploadedImageUrl = activity.Attachments.First().ContentUrl; ;
uploadedImageUrl = HttpUtility.UrlDecode(uploadedImageUrl.Substring(uploadedImageUrl.IndexOf("file=") + 5));
using (Stream imageFileStream = File.OpenRead(uploadedImageUrl))
{
try
{
return await this.visionClient.AnalyzeImageAsync(imageFileStream, visualFeatures);
}
catch (Exception e)
{
return null; //on error, reset analysis result to null
}
}
}
//Else, if the user did not upload an image, determine if the message contains a url, and send it to the Vision API
else
{
try
{
return await visionClient.AnalyzeImageAsync(activity.Text, visualFeatures);
}
catch (Exception e)
{
return null; //on error, reset analysis result to null
}
}
}
public VisualFeature[] getVisualFeatures() {
return visualFeatures;
}
public VisionServiceClient getVisionClient()
{
return visionClient;
}
}
}
VisionConnector
implements methods to communicate with Microsoft Cognitive API. Next, go to MessagesController.cs
class file and replace the following code:
using System;
using System.Linq;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using System.Web.Http;
using System.Web.Http.Description;
using Microsoft.Bot.Connector;
using Newtonsoft.Json;
using Blog.BotApplicationSample.Vision;
namespace Blog.BotApplicationSample
{
[BotAuthentication]
public class MessagesController : ApiController
{
public IVisionConnector visionConnector;
public MessagesController() {
visionConnector = new VisionConnector();
}
/// <summary>
/// POST: api/Messages
/// Receive a message from a user and reply to it
/// </summary>
public async Task<HttpResponseMessage> Post([FromBody]Activity activity)
{
ConnectorClient connector = new ConnectorClient(new Uri(activity.ServiceUrl));
if (activity.Type == ActivityTypes.Message)
{
var analysisResult =await visionConnector.AnalizeImage(activity);
Activity reply = activity.CreateReply("Did you upload an image? I'm more of a visual person. " +
"Try sending me an image or an image url"); //default reply
if (analysisResult != null)
{
string imageCaption = analysisResult.Description.Captions[0].Text;
reply = activity.CreateReply("I think it's " + imageCaption);
}
await connector.Conversations.ReplyToActivityAsync(reply);
return new HttpResponseMessage(HttpStatusCode.Accepted);
}
else
{
HandleSystemMessage(activity);
}
var response = Request.CreateResponse(HttpStatusCode.OK);
return response;
}
private Activity HandleSystemMessage(Activity message)
{
if (message.Type == ActivityTypes.DeleteUserData)
{
// Implement user deletion here
// If we handle user deletion, return a real message
}
else if (message.Type == ActivityTypes.ConversationUpdate)
{
// Handle conversation state changes, like members being added and removed
// Use Activity.MembersAdded and Activity.MembersRemoved and Activity.Action for info
// Not available in all channels
}
else if (message.Type == ActivityTypes.ContactRelationUpdate)
{
// Handle add/remove from contact lists
// Activity.From + Activity.Action represent what happened
}
else if (message.Type == ActivityTypes.Typing)
{
// Handle knowing tha the user is typing
}
else if (message.Type == ActivityTypes.Ping)
{
}
return null;
}
}
}
MessageController
class watches all incoming images and returns the detected caption.
All together now!
Finally, run your Visual studio project and set the bot URL on Bot emulator:
Final thoughts
It's very easy developing artificial intelligence using .NET, Microsoft Cognitive Services and Bot framework . They let you build cross-platform apps with powerful algorithms using just a few lines of code. Bot framework is compatibile with the most famous chats: Facebook, Telegram, Skype and Whatsapp. The example is available on GitHub.
Cheers :)