Update

These issues are in the process of being addressed where appropriate.

See this Twitter post for the first set.

Introduction

Hi! I'm Ashley. I'm a software tester who used to work in marketing and communications so I keep swapping hats back and forth. I'm recently back on the job hunt after a 6 month sabbatical which I was very lucky to be able to take. But it's a very different market to the last two that I have experienced as a QE, so I'm trying new things. I've been more than a little bit inspired by the indie hacker movement, so I want to help y'all out while I'm interviewing.

Every fortnight, I'm going to choose one early-stage project to #testinpublic with a real-world application to keep my testing skills sharp. You can nominate your product, or suggest a friend's, and I'll take a look over a couple days, and send you my thoughts.

For my first week, I'm looking at Charty App, an app where you can create animated charts. This one is just soon to come out of alpha so I expect it to be usable, but not polished. Did anyone say "Minimum Lovable Product"?

Executive Summary

• 2-day functional testing exercise
• Tested key features: user account, payment, one type of chart, import, export
• Most issues were around user experience and error handling

Summary

Most issues I found were in two categories: user experience, and error handling.

It came down to two questions

• How do you avoid overwhelming users when you want to give them all the possible options?
• How do you handle errors for minimum negative impact on customer experience?

With user experience, there are some issues I’d like to call out.

• The main issue is that there was premature optimisation of categories, with settings for several aspects scattered across several categories. It added a lot of scrolling and short-term memory recollection to do what I wanted.
• I was also expecting some categories and settings to have different names, as it was what I had come to expect from using other charting software.
• The next, is that many settings were missing default values and/or units of measurement. Give the user a reference point!
• The last, is that some subcategory titles were missing the title CSS styling, and they got a bit lost in that big list. For styling, with this many options available, it becomes really important to be able to separate feature from feature aspects at first glance

I’d rank the importance of the error-handling issues in the below order

• 500 errors that led to crashes and therefore lost work
• Silent failures that would lead to confusion and lost time
• Enabling inputs outside of the allowed range

Approach

When I start looking at a new piece of software, I always try and gain as much understanding of the product as possible by wandering around it for a while by myself without any deliberate provision of context from the creators. Then I'll dig in when it gets interesting. This means that I don't have any explanations of how it works to make my experience easier, and I use it much closer to how an actual new user would.

When I take this approach, I deliberately DON'T ask questions, just like most users wouldn't, and I see if I get disoriented or confused or overwhelmed, and so on. My context is instead similar products, and UX design patterns. These create unwritten expectations.

For this exercise, I'm going to limit my testing to a small part of the app. I chose The User Account, Line Charts, Import, Export, and Payment.

I have not prioritised (what I feel may be) the key issues, because while a QA may and should engage in bug advocacy they are not the ultimate decision maker. I have chosen instead key features to test, and thrown everything that I think could be of concern into an organised pile. Occasionally I will put my marketing hat back on and make some suggestions. I will always point out when that is.

This is also not exhaustive, as I set a time limit of 2 days to test. It also cannot be exhaustive. There are always bugs, just hopefully as few as possible.

For the sake of brevity, I will only be functional testing, and only on my home device. My observations will only cover possible defects, concerns, and questions.

I tested with the latest version of Chrome on Windows 10.

General Observations

• I felt like the settings options were a bit scattered and the namings and labels a little inconsistent. Some titles are missing title CSS styling as well. While some settings have on-hover tooltips and others don't. I'll provide examples for each further down.

• There is no unit of measurement provided for some of the different settings. Is it px? I would assume it's px.

• I was able to enter numbers above the maximum set in the slider for most features using sliders to set values. This should not be possible.

Naming consistency

• Placement dropdown naming was inconsistent. For Common Text Properties it used used "top, bottom". For Ticks it used "start, middle, end". But these both referred to label positions.

• Box Radius is titled Angle for Background Plate Border and Box Radius for Watermark Box. Unless I am misunderstanding something.

• There are some Points settings in Graph Settings and other Points settings in Points Labels. Personally, I found it annoying going to two different places to set all the Points settings.

• Animations settings are in three places
  -- Animation Timings
  -- Graph Animation
  -- Particles, which appear to be celebratory animations (unless you chose the sad or angry face emoji)

• Most Ticks settings are in Ticks, but to set Tick label size, I need to go to Common Text Properties.

• 500s could be handled better. When they happen, the whole application crashes and I see the start of the stack-trace. I will not provide screenshots of that here.

• There seem to be two ways to add and remove Grid Lines

Ticks -> Horizontal Ticks
Ticks -> Vertical Ticks

Grid -> Enable Grid X
Grid -> Enable Grid Y

However, the Grid Line settings over-rule the Ticks Line settings. This could get confusing for the user (and it's more code that the developer has to maintain), especially as the default setting for the Starter Line Chart has the Grids turned off in Grid.

Marketing hat on

The Animation options sound very mathematical and I wasn't sure what the difference was between most of them without trying them out! Perhaps a section on your features page with animations demonstrating all the options.

• There are a LOT of options. The user can basically customise everything! There's celebratory graphics for social media, and slick animations for everything. For that reason, I would
  consider

• some renaming of some categories.

• some rearranging of some categories.

• clustering the animations options together.

• subcategories

• I personally was unsure about how to add x and y labels for a while!

• ALL settings are Graph Settings. That category is probably a high candidate for renaming.

How to get to there

1) on-hover tooltips explaining what each category means (just temporarily. in the long term the new short titles should be enough)

2) running analytics to see which categories and settings are most often used and least often used for a more effective clustering.

3) talking to customers to understand why

Perhaps: Least often used - at the bottom and accordion shut by default. Most often used - At the top and accordion shut by default.

Marketing hat off

~The User Account~ (Log In, User Profile)

• I believe it's a standard of web experience that the user should never lose their work. I would expect that if you click "Log In" from a page where you are editing a project, it would ask you if you want to save the project currently in progress as a JSON file (as there is not currently an option to save projects in progress on the user profile). This does not happen. Additionally, I was unable to use the Back button to rescue my work.

• I can't close the User Profile modal. I have to click Open Editor to close it indirectly.

Payment

• I think I was given a crypto wallet to use for DePlan when I signed up for an account but I can't be sure as it was labeled "Your UID" in my User Profile. I don't know what it's for.

• Not necessarily a bug but why is it not an option to pay for Pay-As-You-Go Access from the User Profile page? Probably something about DePlan I do not understand.

• Why is the dollar sign on the left for the Pay-As-You-Go option, and on the right for the Monthly Premium option? It's not clear what currency you pay in for either option. I think it's USD.

File/Import

• If the user attempts to import non-compliant JSON it silently fails.

• If the user attempts to import the project without the project settings it silently fails.

• If the user removes a setting from the JSON they are still able to import it. The relevant setting just returns to the default.

• If the user brute forces a setting to be higher than the maximum or lower than the minimum, unless it's Animation Framerate, it is allowed. It can still be successfully uploaded.

• It was unclear until I tried it that Open referred to importing project files.

Marketing hat on

• The purpose of File -> Open is a little unclear until you look at the icon. Then it's obvious that it means 'Import'. I would maybe change the titles like File -> Project Open -> Import Save -> Save.

Marketing hat off

Export

• Why is there a + icon for "Export Image" but not "Export Animation"

Play button

• No on-hover tooltip. Interesting that it's not using the start/stop video icons.

The Starter Line Chart

Animation Timings

• What is the maximum animation framerate? It's not clear but it appears to be 300 because when I try a number higher than that, it changes to 300.

Background Properties

• Plate Border is missing some title CSS styling

Watermark

• Watermark means something else to me. I would be expecting to Free version to have the ChartyApp logo watermark in the bottom right-hand corner, but it didn't.

Marketing hat on

• The watermark on the premium version seems more like somewhere to put the customer's branding. I'd suggest, if I may, an option to upload an image instead of adding text.

Marketing hat off

• At the default X and Y offset, the Watermark is falling off the chart for characters like "q, j, p, g" when they are lower case for fonts with both lower and upper case, like Raleway and Montserrat.

Changing the Box Padding should probably push the box up so all the text remains in the chart, but it stays in the same place horizontally.

Common Text Properties

• There was no character limit for the Graph Header.

• The on-hover tooltip for the Enable Graph Header slider to show the tooltip, is not correct.

• If you hid the Graph Header, the chart did not increase in size to fill up the space no longer used by it.

• There was a very high range of options for the Ticks Label Text Size. It currently doesn't detect collision and set a limit when that is reached, and so they can currently overlap.

Particles

• There is some sort of bug with Particles -> Graphics -> Custom Emojis at the moment generating a 500 error, and when I tried to select it, the whole application crashes and I see the start of the stacktrace.

• Emoji Set 1 -> 9 didn't really mean anything to me as I don't know what's in the set before I try it. They are not descriptive.

Points Labels

• It is interesting that the default is to not leave a space between the point value and point unit of measurement. It does give the user more options.

• Non-numeric characters are removed for the Y-axis when adding the Label Suffix to it. For the X-axis, the NaN error appears.

Graph Animation

• The dropdown options for Animation Type should be Capital Case to make the rest.

Graph Settings

• The sliders are missing on-hover tooltips.

• Line Curve is missing a sub heading, and the dropdown settings are still in camelCase.

Legends

• The labels appear to have multi-lingual capabilities. That's good. I tried Cyrillic.

Ticks

• Horizontal Ticks is missing the title CSS styling.

• I found it a little confusing initially without a reference to x-axis and y-axis.

Editor

• The x-axis values allow non-numeric characters. I suppose so that the user can do the time series any direction they want?

Simple Editor

For 'Import Your Data', when CSV has header, the title 'Group' Values Column (can be empty) implies it can be empty, but there were no NULL options provided. I had to select a column to choose as the Group values column.

Raw Data

• It would be good to have some kind of written alert when the JSON file is no longer JSON compliant.

• If you remove the x-axis value on Raw Data for one of the Groups, it disappears for all on the chart.

• If you deleted the name of the group in the Raw Data editor, that group would disappear from the chart. It wouldn't just remove the name from the label, as you might expect.

• Once the Raw Data JSON was no longer JSON compliant, the chart "froze" in its last compliant state (and red underlines would appear on the editor) but there was no other indicator to the user that the data was not compliant.

Conclusion

I'd like to give Naz a shout-out for some amazing work! And thank you for being a good sport about my publishing my findings of your alpha-stage software on the internet!