In my previous post, I described how I used React and the Amplify CLI to implement an initial front-end for the Bearcam Companion. This time I will write about
- UI improvements (especially the bounding boxes)
- Adding authentication, sign-up and sign-in
- Implementing a method for users to identify bears
UI Improvements
Last time I mentioned I was not happy with using <canvas> elements for drawing bounding boxes around the bears. I set out to use <div> and CSS instead, as inspired by the Amazon Rekognition demo interface:
I wrapped my <img> element with a relatively positioned <div>. I created a Boxes component, and used the map() function to instantiate each box in the boxList:
<div style={{position:'relative', margin:'auto', display: 'block'}}>
<img id="refImage" ref={inputEl} src={imagePath} alt="bearcam frame" />
{
boxList.map( (box) =>
<Boxes key={box.id} box={box} />
)}
</div>
In Boxes.js, I get the box information: top, left, height and width from the respective box fields. I use these to set the location of an absolutely positioned <div>. I add the label text in another <div> along with the confidence (converted to a percentage by multiplying by 100 and truncating). The code snippet looks like this:
const boxTop = `${box.top*100}%`
const boxLeft = `${box.left*100}%`
const boxHeight = `${box.height*100}%`
const boxWidth = `${box.width*100}%`
return(
<div className="bbox tooltip" key={box.id}
style={{top: boxTop, left: boxLeft, height: boxHeight, width: boxWidth }} >
<div className="identname">{box.label} ({Math.trunc(box.confidence*100)})</div>
</div>
)
Using CSS, I control the bbox and identname styles and locations. I use the :hover properties to control the color of the bbox and the visibility of the text. With this implementation, I have a much better bounding box experience (note the blue, default box on the left and the red, hover box on the right):
Authentication
Before allowing the user to identify the bears, I want to set up authentication. My main motivation is to associate identifications with users. This will ensure I only get one identification per user and may also come in handy for future functionality.
I used Amplify Studio to enable authentication, select a Username based login mechanism and configure the sign up options. Back on my developer machine, I performed an amplify pull to get the authentication changes. Enabling the built in sign in and sign up flow is as simple as wrapping App in withAuthenticator. I can now access the user information from user:
import { withAuthenticator } from '@aws-amplify/ui-react';
function App({ signOut, user }) {
return (
<div className="App">
<header className="App-header">
<div className="headerImage">
<img width={200} height={65} src="/BearID-Project-Logo-PNG_inverse.png" alt="BearID Logo" />
</div>
<Heading level={5} color="white">Hello, {user.username} </Heading>
<Button onClick={signOut} color="gray">Sign out</Button>
</header>
<Heading level={4}>Bearcam Companion</Heading>
<FrameView user={user} />
<footer className="App-footer">
<h2>©2022 BearID Project</h2>
</footer>
</div>
);
}
export default withAuthenticator(App);
The default sign in screen looks like this:
Identifications
Now that the user is logged in, I want them to be able to identify the bears in the images. I created a new data model, Identifications. This model includes the name of the bear, name, and username of the user that made the identification, user. Since each bear can be identified by multiple users, I need to create a 1:n relationship between Objects and Identifications. I called this field objectsID. The model in Amplify Studio looks like this:
After an amplify pull I can start using the new data model in my front end. Now I can get all the Identifications for the current box with a call like this:
const idents = await DataStore.query(Identifications, c => c.objectsID("eq", box.id));
This gives me all the individual Identifications for the box. What I really want is a tabulation of votes for each bear name. Then I can show the top voted name (and percentage) in the default box view, like this:
DataStore doesn't provide this sort of aggregation (nor does DynamoDB behind it). I found a bit of code using .reduce to group my idents from above by a key, and a count for each key:
function groupIdents(list, key) {
return list.reduce(function(rv, x) {
rv[x[key]] = rv[x[key]] ? ++rv[x[key]] : 1;
return rv;
}, {});
};
I call groupIdents with idents and a key of name, which is the bear name. I then sort the results by the count.
const gIdents = groupIdents(idents,"name");
pairIdents = Object.entries(gIdents).sort((a,b) => b[1]-a[1]);
I want to use idents in a new component, BoxIDs, which will render the sorted list of bear names and counts/percentages. I want this content to to show for each box and update when new identifications are added. To manage this, I made use of useState() and useEffect() hooks. I created a useState() hooks for my sorted list of names/counts (identAgg) and total count (identCount):
const [identAgg, setIdentAgg] = useState([["Unknown", 1]]);
const [identCount, setIdentCount] = useState(1);
As you can see, I set the default identAgg to have the name "Unknown" with a count of 1. I also set the default identCount to 1. I will use these values when no identifications have been made.
The useEffect() hook lets me run code on certain lifecycle events or when things change. I wrapped the previous code in the useEffect() so that it runs when box.id changes:
useEffect(() => {
async function getIdents() {
var idents = await DataStore.query(Identifications, c => c.objectsID("eq", box.id));
var pairIdents = [["Unknown", 1]];
var count = 1;
if (idents.length) {
const gIdents = groupIdents(idents,"name");
pairIdents = Object.entries(gIdents).sort((a,b) => b[1]-a[1]);
count = idents.length;
}
setIdentList(idents);
setIdentCount(count);
setIdentAgg(pairIdents);
}
getIdents();
DataStore.observe(Identifications).subscribe(getIdents);
}, [box.id]);
I can display the top identification and count/percent information by adding the following to my render:
<div className="identname">{identAgg[0][0]} ({identAgg[0][1]}/{identCount} = {Math.trunc(identAgg[0][1]*100/identCount)}%)
That takes care of the default view I showed previously. When the user hovers over the box, I want to show more details like this:
In this case I choose to show the sorted list of top identifications and their respective counts. The new BoxIDs component renders the name and count for each aggregated identification:
import React from 'react'
export default function BoxIDs({ ident }) {
return(
<div >{ident[0]} ({ident[1]})</div>
)
}
I added it to Boxes by inserting the following into the render:
<div className="identdetails">
{
identAgg.map( (ident) =>
<BoxIDs key={box.id + "-" + ident[0]} ident={ident} />
)
}
<SetID boxID={box.id} curList={identList} username={username} />
</div>
You may have noticed SetID above. This component shows the user's current selection and implements a drop down list of all the possible identifications. The user's current selection is found by searching the list of identifications for one where the user matches the current user. When the user selects an identification from the drop down, it creates a new Identification for the user. If the user has previously made an identification, it modifies the existing one instead. The UI looks like this:
Conclusion
That wraps up the latest round of changes. This is getting close to something users can test. I still need to implement a way to pull in new images and automatically find the bears and there are always UI improvements to be made. It's also about time to put everything in a code repository.
I'll cover these topics next time...
Top comments (0)