Recently, I contributed to the open source project Chatcraft, and participated in the development of 10 release versions from v1.1.0 to v2.0.0. I would like to write a blog post to summarize my contributions.
New Feature contributions
1. Enables users to upload images to ChatCraft and work with vision-related models (related release: v1.2.0, v1.3.0, v1.4.0, v1.6.0, v1.9.0, v2.0.0)
Release note:
v1.2.0: first release
v1.3.0: supports mobile
v1.4.0: adds image compression
v1.6.0: supports multiple image compression options
v1.9.0: removes workaround hardcode
v2.0.0: supports GPT-4 Turbo
Demonstration
(Please click on the GIF file to trigger the animation if it doesn't start automatically.)
Note, although the message time displayed is current, here is the v1.2.0 commit branch being used to show the initial version functionality.
The latest release v2.0.0 is as follows, since the new model of gpt-4-turbo as of 04-09 also supports "Vision", I have adapted it accordingly.
(Please click on the GIF file to trigger the animation if it doesn't start automatically.)
Users can also add images by using drag and drop and copy-paste methods.
| drag and drop | copy-paste |
|---|---|
 |
 |
(Please click on the GIF file to trigger the animation if it doesn't start automatically.)
To implement this feature, I studied the newly emerged vision-related API from the openai-node project and integrated it into the codebase, added new UI components to display user images, and made modifications to the model, database, and optimization operations such as image compression. For example, the integration of new API can be found in the changes from original type CreateChatCompletionRequestMessage to new type ChatCompletionMessageParam as below:
@@ -90,16 +96,28 @@ export class ChatCraftMessage {
date: this.date.toISOString(),
type: this.type,
text: this.text,
+ imageUrls: this.imageUrls,
};
}
- toOpenAiMessage(): OpenAI.Chat.Completions.CreateChatCompletionRequestMessage {
+ toOpenAiMessage(model: ChatCraftModel): OpenAI.Chat.Completions.ChatCompletionMessageParam {
const text = this.text;
+
+ const content: OpenAI.Chat.Completions.ChatCompletionContentPart[] = [{ type: "text", text }];
+ if (model.supportsImages && this.imageUrls.length > 0) {
+ this.imageUrls.forEach((imageUrl) => {
+ content.push({
+ type: "image_url",
+ image_url: { url: imageUrl },
+ });
+ });
+ }
+
switch (this.type) {
case "ai":
return { role: "assistant", content: text };
case "human":
- return { role: "user", content: text };
+ return { role: "user", content };
case "system":
return { role: "system", content: text };
case "function":
@@ -119,6 +137,7 @@ export class ChatCraftMessage {
chatId,
type: this.type,
text: this.text,
+ imageUrls: this.imageUrls,
};
}
This new type suggests a more flexible structure that can accommodate multiple types of content within a single message. This is crucial for integrating capabilities for the image processing and handling, which are essential for models that support vision features.
The image compression function is completed client-side, which can reduce the cost of the user's API token. Users can make related settings in the settings modal.
As a side effect, I use it to do quick image compress.
This feature has been the biggest challenge I've faced in the open-source community so far. In my previous post, 'A 55-commit PR,' I also wrote about my feelings after releasing. Thanks again to the ChatCraft community for their help and support.
2. Enables users to generate images with the DALL·E 3 model (related release:v1.5.0, v1.7.0, v1.8.0, v2.0.0)
Since the image generation model kind of independent than the language models, my approach is to use ChatCraft's command interaction, and I added an /image command, usage:
/image [layout-option]<prompt>
layout=[l\|landscape\|p\|portrait]
Release note:
v1.5.0: first release
v1.7.0: add loading info
v1.8.0: fix for issue: generated image not available after some time
v2.0.0: fix for issue: the loading spinner does not disappear when image generation fails
Demonstration
(Please click on the GIF file to trigger the animation if it doesn't start automatically.)
3. Supports rendering of mathematical expressions (LaTeX) (v1.1.0)
Demonstration
(a night mode example)
Improved UI contribution
Allows for the hiding and showing of long chat. (v1.1.0, v1.2.0)
It works great when the message is super long. During the development, I encountered a Minified React error issue. If you are interested, you can check my post 'Minified React error #426' to see a possible solution. An interesting update in v1.2.0 is we don't fold super long final message to give the user better experience.
Demonstration
(Please click on the GIF file to trigger the animation if it doesn't start automatically.)
Bug fixes contributions
1.dollar symbol render conflict vs Latex rendering issue (v1.1.0)
| Bug | Fix |
|---|---|
 |
 |
2.no label code block rendering issue (v1.1.0)
| Bug | Fix |
|---|---|
 |
 |
3.Html no syntax highlight issue (v1.2.0)
| Bug | Fix |
|---|---|
 |
 |
4.encode & in Markdown URLs issue (v1.2.0)
| Bug | Fix |
|---|---|
command failed, converting that to &
|
 |
5.non-OpenAI model uses OpenAI logo (v1.8.0)
| Bug | Fix |
|---|---|
 |
 |
6. Disable message editing on shared chats (v1.9.0)
| Bug | Fix |
|---|---|
 |
 |
Reflections
My contributions to ChatCraft from versions v1.1.0 to v2.0.0 have been both challenging and rewarding. Throughout this journey, I have been deeply involved in enhancing the functionality and user experience of ChatCraft. These contributions have not only improved the platform but have also provided me with invaluable learning experiences.
My work on ChatCraft has been a profound learning curve and a testament to the power of community and collaboration in the open-source space. I am grateful for the support from the ChatCraft community, which has been instrumental in overcoming challenges and achieving milestones. I look forward to engaging more with the community and continuing to contribute to the evolution of ChatCraft.
Top comments (0)