Run COVID-Net - an open source software to diagnoses COVID-19 from chest x-ray images

This article has been translated with the help of Deepl. The original Japanese version of the article is here.

Introduction.

Coronavirus infection is prevalent. In this article, I write about how I ran it using Docker. I write about how to run COVID-Net using Docker.(I'm not a member of COVID-Net team. I'm just a user.)

What is COVID-Net?

It is a classifier to classify the three categories of "normal," "pneumonia," and "COVID-19" from chest x-ray mages.

Github: https://github.com/lindawangg/COVID-Net

Paper https://arxiv.org/abs/2003.09871

Developers

• Linda Wang
• University of Waterloo, Canada.
• DarwinAI, Inc.

Try to run it using Docker

　In order to encourage more people to try the software, I created a Dockerfile with the help of members of the Japanese local ruby community.

Github: https://github.com/kojix2/covid-net-docker
DockerHub: https://hub.docker.com/r/kojix2/covid-net

Download the Docker image

Download the Docker image from the Docker Hub.

docker pull kojix2/covid-net

Create and launch the container

Here we mount the current directory to /tmp/share. You can now access the images from the container.

sudo docker run -it -v $(pwd):/tmp/share kojix2/covid-net bash

When you launch it, the TensorFlow logo will appear.

________                               _______________                
___  __/__________________________________  ____/__  /________      __
__  /  _  _ \_  __ \_  ___/  __ \_  ___/_  /_   __  /_  __ \_ | /| / /
_  /   /  __/  / / /(__  )/ /_/ /  /   _  __/   _  / / /_/ /_ |/ |/ / 
/_/    \___//_/ /_//____/ \____//_/    /_/      /_/  \____/____/|__/


WARNING: You are running this container as root, which can cause new files in
mounted volumes to be created as the root user on your host machine.

To avoid this, run the container by specifying your user's userid:

$ docker run -u $(id -u):$(id -g) args...

Inference of the example image

Let's judge the example chest x-ray image assets/ex-covid.jpeg.

python inference.py \
    --weightspath models/COVIDNet-CXR4-A \
    --metaname model.meta \
    --ckptname model-18540 \
    --imagepath assets/ex-covid.jpeg

You will see the following results.

Prediction: COVID-19
Confidence
Normal: 0.031, Pneumonia: 0.189, COVID-19: 0.780

The result is a high probability of COVID-19.

Inference of self-prepared images

Next, let's judge the image we have prepared.
Rewrite imagepath.

Download a normal x-ray image from Wikipedia. I saved it here under the name chest.jpg.

python inference.py \
    --weightspath models/COVIDNet-CXR4-A \
    --metaname model.meta \
    --ckptname model-18540 \
    --imagepath /tmp/share/chest.jpg

Results.

Prediction: COVID-19
Confidence
Normal: 0.000, Pneumonia: 0.000, COVID-19: 1.000

Oops, what a result, 100% COVID-19!

Perhaps there were areas that looked like frosted glass shadows.

As you can see, the accuracy of this model is not very high. As far as I've tried it, it tends to misclassify as COVID-19 at a fairly high rate, which means that deep learning-based image classifiers aren't a magic and there's a lot that can go wrong. Image classification AI is just a tool, so You don't have to blindly trust it or be scared of it.

Inference on the severity of COVID-19

The sample chest x-ray image.

python inference_severity.py \
    --weightspath_geo models/COVIDNet-SEV-GEO \
    --weightspath_opc models/COVIDNet-SEV-OPC \
    --metaname model.meta \
    --ckptname model \
    --imagepath assets/ex-covid.jpeg

Result.

Geographic severity: 0.519
Geographic extent score for right + left lung (0 - 8): 4.155
For each lung: 0 = no involvement; 1 = <25%; 2 = 25-50%; 3 = 50-75%; 4 = >75% involvement.
Opacity severity: 0.388
Opacity extent score for right + left lung (0 - 6): 2.329
For each lung: 0 = no opacity; 1 = ground glass opacity; 2 =consolidation; 3 = white-out.

Inference of the severity of COVID-19 on your images.

python inference_severity.py \
    --weightspath_geo models/COVIDNet-SEV-GEO \
    --weightspath_opc models/COVIDNet-SEV-OPC \
    --metaname model.meta \
    --ckptname model \
    --imagepath /tmp/share/chest.jpg

Wikipedia's normal image.

Geographic severity: 0.293
Geographic extent score for right + left lung (0 - 8): 2.341
For each lung: 0 = no involvement; 1 = <25%; 2 = 25-50%; 3 = 50-75%; 4 = >75% involvement.
Opacity severity: 0.242
Opacity extent score for right + left lung (0 - 6): 1.450
For each lung: 0 = no opacity; 1 = ground glass opacity; 2 =consolidation; 3 = white-out.

The COVID-19 decision was completely wrong, but the severity of the disease seems to be quite low. The severity of the disease seems to have been determined correctly.

Impressions of COVID-Net

The first thing I want to confirm is that it is difficult to accurately diagnose coronavirus infections from X-rays.

We should not assume that artificial intelligence can detect COVID-19 from X-rays when humans cannot detect COVID-19 from X-rays. According to some article, only 14.7% of COVID-19 cases were found to be abnormal on chest x-rays. This means that even with advanced technology, it is almost impossible to accurately diagnose COVID-19 on x-rays.

Secondly, I would like to ensure that COVID-Net is not a product intended for clinical use. This point is highlighted by the authors of the software in their README.

Third, there is an active discussion on whether the test to determine COVID-19 should be expanded. or not. Our goal is to save patients' lives, prevent the spread of infection, and distribute medical resources. As many experts have discussed carefully, increasing PCR testing does not necessarily lead to the desired social outcome. I do not have sufficient knowledge into this very difficult issue to comment on it. However, it's important to remember that it's not as simple as adding one less accurate test tool will improve the social outcome.

(Note in English: Many experts have been conservative in expanding PCR testing in Japan. The main reason for this is probably the fear that the stable supply of hospital beds for the most serious patients will fail due to the abuse of PCR testing.)

As such, there are a number of limitations. but I'm going to write briefly about my personal impressions of the project.

First of all, I would like to address the weaknesses of COVID-Net; I think COVID-Net is not accurate, especially the "sensitivity" is too high. (Please note that this comment is not based on a specific dataset. It's just my personal impressions, so it may contain errors.) I tried to load some chest X-ray images available on the Internet, and found that there were quite a few cases that COVID-Net misclassified as COVID-19.

If COVID-19 is a rare disease, hight sensitivity may be may be desirable for screening. But in reality, upper respiratory tract inflammation or pneumonia are very common symptoms.
　Many people come to the hospital with symptoms such as "cough," "runny nose," "sputum" and "sore throat". We need a way to distinguish between COVID-19 and infections that are associated with such frequent symptoms. If COVID-Net is too sensitive, it may also test positive for upper respiratory tract and pneumonia unrelated to coronaviruses. If this is the case, it will likely be a long time before it becomes useful in clinical level.

On the other hand, if the case has progressed to pneumonia, COVID-Net may be able to distinguish between bacterial pneumonia and pneumonia of COVID-19 origin. However, in such advanced cases of pneumonia, CT may be more useful than X-rays. CT provides much more information than an x-ray, and if the expert observes the CT image, he or she can identify with high precision the characteristics of the coronavirus - such as the frosted shading just below the pleura. In advanced cases, CT imaging is performed, which limits the scope of the use of chest x-ray diagnostic software.

Next, the excellent thing about COVID-Net. It is an x-ray COVID-19 detector and is effective in the situation where CT is not available and only x-rays are used to identify coronavirus infections. This is what many medical practitioners and clinics are looking for.

This is my personal opinion, but one of the major problems with the current development of medical imaging AI is that large companies and large hospitals are at the center of its development, so developers' attention is focused on "realizing advanced medical care" and it is difficult to focus on the on the potential societal needs.

(Note in English: This section was written for a situation in the context of a local Japanese society where AI technology has not made much progress. However, the situation may be the same in other bigger countries, I think...)

In this case, large hospitals perform CT and PCR, so motivation to develop diagnostic software for X-rays is unlikely to occur. In this respect, this project is focused on areas that are likely to be useful in the real world, and I think that's a great point.

The developers of COVID-Net provide pre-trained models on the Internet. This means that anyone is free to try out the models. Needless to say, this is a great thing.

The developers of COVID-Net are planning to release a model for CT soon.

Finally, I am not an expert in infectious diseases or x-rays, much less actually involved in the practice of COVID-19. Therefore, some of the above comments and impressions are just my personal opinion based on my imagination.

Especially, the part about how to assess "severity of illness" and how it should be used in the real bedside is beyond me. Therefore, I would be happy if more people could try this kind of model and think about these aspects.

I have respect with everyone who works in the hospital in this difficult situation.

Conclusion

With the spread of new coronavirus infections, everyone is anxious and careful. But we must have hope.

COVID19 is a unique case of machine learning having the potential to save lives. Protecting personal information is important. However, if are willing to provide their own image data and create effective software, then maybe, just maybe, some lives can be saved from it.

On the other hand, increasing the diagnosis accuracy does not necessarily mean that the final outcome (e.g., life-saving rate) will be improved. This is where complex and difficult social issues come into play.

Computerized imaging is an old and new technology. It is my strong hope that as more people are exposed to this new technology, it will drive improvements and lead to real results in saving patients' lives, preventing the spread of infectious diseases, and reducing the burden on medical practices.

Thank you for your reading.