What is Telescope
Telescope is an example of planet: a tool for
aggregating blog feeds. It reminds me of google reader which I used for a long time to read the news.
Setup Environment
I am working on a fedora33 with KDE Plasma Desktop. Docker still not support CgroupsV2. It takes a little time to install docker and I have written a blog post for the installation https://dev.to/yzwdroid/fedora-33-docker-445k.
For some reason, redis has a permission error when docker-compose
Starting telescope_redis_1 ... done
Starting telescope_elasticsearch_1 ... done
Attaching to telescope_redis_1, telescope_elasticsearch_1
redis_1 | find: '.': Permission denied
redis_1 | chown: changing ownership of '.': Permission denied
telescope_redis_1 exited with code 1
So I installed Redis by sudo dnf install redis, and start it with
redis-server
All other part went well, I just follow the environment-setup.md
Modify goURL
To work with the REST API of telescope, I added a new flag -u --url, fetch
the json data, convert the data to a slice of post, get each post's contents by
the url. I just added one new function to make this work. Below is the diff I
made.
git diff HEAD main.go utils.go > lab6.diff
The diff
diff --git c/main.go w/main.go
index 4679a1a..fd3d249 100644
--- c/main.go
+++ w/main.go
@@ -25,6 +25,7 @@ var js = flag.BoolP("json", "j", false, "output json format to stdout")
var fp = flag.StringP("file", "f", "", "file name to check")
var ignore = flag.BoolP("ignore", "i", false, "ignore url patterns")
var failOnly = flag.Bool("fails", false, "show only urls that failed")
+var urlFlag = flag.BoolP("url", "u", false, "read telescope restful API")
func main() {
flag.Parse()
@@ -43,14 +44,20 @@ go run main.go -v or --version check version.
os.Exit(-1)
}
- dat, err := ioutil.ReadFile(*fp)
- check(err)
-
+ var urls []string
+ var dat []byte
+ if *urlFlag {
+ dat = dataTelscope()
+ } else {
+ var err error
+ dat, err = ioutil.ReadFile(*fp)
+ check(err)
+ }
// use xurls tool to exact links from file. Strict mod only match http://
// and https:// schema
rxStrict := xurls.Strict()
// urls is a slice of strings
- urls := rxStrict.FindAllString(string(dat), -1)
+ urls = rxStrict.FindAllString(string(dat), -1)
urls = removeDuplicate(urls)
if *ignore {
diff --git c/utils.go w/utils.go
index 5c27fc7..aee4859 100644
--- c/utils.go
+++ w/utils.go
@@ -2,7 +2,9 @@ package main
import (
"bufio"
+ "encoding/json"
"fmt"
+ "io/ioutil"
"log"
"net/http"
"os"
@@ -18,6 +20,43 @@ type urlStatus struct {
Status int
}
+type post struct {
+ ID string
+ URL string
+}
+
+func dataTelscope() []byte {
+ var data []byte
+ var posts []post
+ resp, err := http.Get("http://localhost:3000/posts")
+ check(err)
+ defer resp.Body.Close()
+
+ if resp.StatusCode == http.StatusOK {
+ body, err := ioutil.ReadAll(resp.Body)
+ check(err)
+ if err := json.Unmarshal(body, &posts); err != nil {
+ panic(err)
+ }
+ for _, p := range posts {
+ resp, err := http.Get("http://localhost:3000" + p.URL)
+ if err != nil {
+ fmt.Println(err)
+ }
+ defer resp.Body.Close()
+ if resp.StatusCode == http.StatusOK {
+ bodyData, err := ioutil.ReadAll(resp.Body)
+ if err != nil {
+ fmt.Println(err)
+ }
+ data = append(data, bodyData...)
+ }
+ }
+ }
+
+ return data
+}
+
func removeDuplicate(urls []string) []string {
result := make([]string, 0, len(urls))
temp := map[string]struct{}{}
The gist version is here https://gist.github.com/yzwdroid/26b6cb71f75f3799dfccd5fb0ed9ec79
Top comments (0)